Investigating the recent massive oil spill in the Gulf of Mexico is a tall order.  There are many contributing causes and a multitude of creative solutions are going to be needed to restore the environment.

During any investigation of this magnitude, there are guaranteed to be a few surprises.  And the Deep Horizon oil spill is no exception.

Scientists have discovered a previously unknown type of oil-eating bacteria feasting on oil from the spill.

This microbe is unique from previously studied varieties because it doesn’t consume large quantities of oxygen along with the oil.  Oxygen consumption is a concern because oxygen is needed in the sea to support life.

This microbe also thrives in cold water temperatures associated with the deep ocean, which might explain why it hasn’t been seen before.  Some scientists are theorizing that the microbe adapted in the deep ocean to consume the oil that naturally seeped from the ocean floor.  Since the huge influx of oil to the water, the bacteria populations have exploded.

Scientists are in a disagreement over how much oil remains in the Gulf, but there is no doubt that less is better. 

This serendipitous solution is a welcome addition to the clean up efforts.  Obviously, there are many other solutions that will needed, but anything that safely reduces the overall amount of oil is a positive development.  Hopefully, with some additional research this microbe could be a potential solution to future incidents.

When performing an investigation, the unexpected sometimes happens.  The better understood the problem is, the easier it is to adapt to any new information. The Cause Mapping method of root cause analysis is an effective way to organize all information needed during an investigation.  Clearly understanding the causes that contribute to an incident will allow an organization to adapt as new information comes available and make sure that resources are used in the most efficient ways when implementing solutions.

(This week, we are proud to announce a Cause Map by a guest blogger, Bill Graham.  Thanks, Bill!)

While completing household chores in the spring of 2010, a Housewife found her front load washing machine stopped with water standing in the clothing.  Inspection of the machine uncovered the washing machine’s drain pump had failed.  Because the washer is less than two years old, it was decided to attempt repair of the machine instead of replacing it.  A replacement pump was not locally available, so the family finds and orders a pump from an Internet dealer.  Delivery time for the pump is approximately one week, during which time the household laundry chore cannot be completed and some of the family’s favorite clothing cannot be worn because it is has not been laundered.  On receiving the new pump, Dad immediately removes the broken pump and finds, to his chagrin, a small, thin guitar pick in the suction of the old pump.  Upon discovery of the guitar pick, the family’s children report that the pick had been left in the pocket of the pants that where being washed at the time of the pump’s failure.  The new pump was installed and the laundry chore resumed for the household.

While most cause analysis programs would identify the guitar pick as the root cause to the washing machine’s failure, Cause Mapping unveils all of the event’s contributing factors and what most efficient / cost effective measures might be taken to avert a similar failure.  For example, if all the family’s children aspire to be guitar players, then a top load washer may better suit their lifestyle while also averting the same mishap.  Or, maybe the family should consider wearing pocket-less clothing.  Or, maybe all family members should assume bigger role in completing the household laundry chore.  Whichever solution is chosen, the impact of these and all contributing causes is easily understood when the event is Cause Mapped.

The objective of a root cause analysis investigation is to prevention.  The causes of an incident are investigated, so that solutions can be developed and implemented, to reduce the risk of the same or a similar problem from occurring.  The process sounds easy, but in practice it can become more involved.  For example, what do you do when one of the identified causes is “lack of safety culture”?  How exactly do you solve that? 

This is the issue that the Washington DC Metrorail (Metro) is currently facing.  The National Transportation and Safety Board (NSTB) recently released findings from the investigation of a DC metro train crash that killed nine last June.  (See our previous blog for more details). Predictably, the NSTB findings include several technical issues including failed track circuits and lack of adequate testing, but the list of causes also includes items like lack of safety culture and ineffective oversight. 

Fortunately, the NSTB also provided recommendations such as developing a non-punitive safety reporting program, establishment of periodic inspections and maintenance procedures for the equipment that failed during this accident, and reviewing the process used to pass along safety and technical information.  One of the important things to notice in this example is that the recommendations are fairly specific, even if the stated cause is a little vague.  Specific solutions are necessary if they are going to be effectively implemented.

 If you find yourself at a point in your organization where a cause is identified as “lack of safety culture”, it’s a good idea to keep asking why questions until you identify the specific problems that are causing the issue.  Is it the safety information that is lacking or incorrect?  Is the process that provides the information confusing?  Do the workers need better safety equipment?  Knowing all the details involved will allow better solutions to be developed.  And better solutions result in lower risks in the future.  Culture is the shared values and practices of the people in an organization.  The Cause Mapping method of root cause analysis has an effective way for an organization to identify “culture gaps” by thoroughly dissecting just one of its incidents.

Astronauts at the International Space Station ran into problems during a planned replacement of a broken ammonia cooling pump on August 7, 2010.  In order to replace the pump, four ammonia hoses and five electrical cables needed to be disconnected to remove the broken pump.  One of the hoses could not be removed because of a jammed fitting.  When an astronaut was able to disconnect it by hitting the fitting with a hammer, it caused an ammonia leak. 

Ammonia is toxic, so the leak impacted both the safety and environmental goals.  Because the broken pump kept one cooling system from working, there was a risk of having to evacuate the space station, should the other system (which was the same age) fail.  This can be considered an impact to the customer service goal.   The repair had to be delayed, which is an impact to the production/schedule goal.  The loss of a redundant system is an impact to the property/equipment goal.     The extended spacewalk is an impact to the labor/time goal. 

Once we fill out the outline with the impact to the goals and information regarding the problem, we can go on to the Cause Map.   The ammonia leak was caused by an unknown leak path and the fitting being removed by a hammer.  The fitting was removed with a hammer because it was jammed and had to be disconnected in order for the broken pump to be replaced.  As we’re not aware of what caused the pump to break (this information will likely be discovered now that the pump has been removed), we leave a question mark on the map, to fill in later.

The failed cooling pump also caused the loss of one cooling system.  If the other system, which is near the end of its expected life, were to fail, this would require evacuation from the station.  

To aid in our understanding of this incident, we can create a very simple process map of the pump replacement.  The red firework shows the step in the replacement that didn’t go well.  To view the outline, Cause Map and Process Map, click on “Download PDF” above.

The Therac-25 is a radiation therapy machine used during the mid-80s.  It delivered two types of radiation beams, a low-power electron beam and a high-power x-ray.  This provided the economic advantage of delivering two kinds of therapeutic radiation with one machine.  From June 1985 to January 1987, the Therac-25 delivered massive radiation overdoses to 6 people around the country.  We can look at the causes of these overdoses in a root cause analysis performed as a Cause Map. 

The radiation overdoses were caused by delivery of the high-powered electron beam without attenuation.  In order for this to happen, the high-powered beam was delivered, and the attenuation was not present.  The lower-powered beam did not require attenuation provided by the beam spreader, so it was possible to operate the machine without it.  The machine did register an error when the high-powered beam was turned on without attenuation.  However, it was possible to operate the the beam with the error and the warning was overridden by the operators.

The Therac-25 had two different responses to errors.  One was to pause the treatment, which allowed the operators to resume without any changes to settings, and another was to reset the machine settings.  The error resulting in this case, having the high-power beam without attenuation, resulted only in a treatment pause, allowing the operator to resume treatment with an override, without changing any of the settings.  Researchers talking to the operators found that the Therac-25 frequently resulted in errors and so operators were accustomed to overriding them.  In this case, the error that resulted (“Malfunction 54″) was ambiguous and not defined in any of the operating manuals.  (This code was apparently only to be used for the manufacturing company, not healthcare users.) 

The Therac-25 allowed the beam to be turned on without error (minus the overridden warning) in this circumstance.  The Therac-25 had no hardware protective circuits and depended solely on software for protection.  The safety analysis of the Therac-25 considered only hardware failures, not software errors, and thus did not discover the need for any sort of hardware protection.  The reasoning given for not including software errors was the “extensive testing” of the Therac-25, the fact that software, unlike hardware, does not degrade, and the general assumption that software is error-proof.  Software errors were assumed to be caused by hardware errors, and residual software errors were not included in the analysis.

Unfortunately the coding used in the Therac-25 was in part borrowed from a previous machine and contained a residual error.  This error was not noticed in previous versions because hardware protective circuits prevented a similar error from occurring.  The residual error was a software error known as a “race condition”.  In short, the output of the coding was dependent on the order the variables were entered.  If an operator were to enter the variables for the treatment very quickly and not in the normal order (such as going back to correct a mistake), the machine would accept the settings before the change from the default setting had registered.  In some of these cases, it resulted in the error described here.  This error was not caught before the overdoses happened because software failures were not considered in the safety analysis (as described above), the code was reused from a previous system that had hardware interlocks (and so had not had these problems) and the review of the software was inadequate.  The coding was not independently reviewed, the design of the software did not include failure modes and the software was not tested with the hardware until installation.

This incident can teach us a lot about over-reliance on one part of a system and re-using designs in a new way with inadequate testing and verification (as well as many other issues).  If we can learn from the mistakes of others, we are less likely to make those mistakes ourselves.  For more detail on this (extremely complicated) issue, please see Nancy Leverson and Clark Turner’s “An Investigation of the Therac-25 Incidents.”

Research is been suspended at a prominent brain-imaging center associated with Columbia University.  Food and Drug Administration investigations found that the Kreitchman PET (positron emission tomography) Center has injected mental patients with drugs that contained potentially harmful impurities repeatedly over the past four years.

Investigations by the lab determined that no patients were harmed from the impurities, but this is still a significant issue in a nationally renown laboratory.

 How did this happen?

This issue can be investigated by building a root cause analysis as a Cause Map.  To start a Cause Map, the impact to the organization goals is determined.  In this example, this issue is obviously an impact to safety because there was potential to harm patients.  It is also an impact to the production-schedule goal because research has been suspended.  Additionally, this problem is an impact to the customer service goal because this issue raises questions about the validity of research results.

To build a Cause Map, select one goal and start asking “why” questions to add causes.  In this case, the first goal considered will be the safety goal.  There was a potential for injury.  Why?  Because impure injections were given to patients.  Why?  Because the injections are necessary for research, because the labs typically prepare the compounds themselves and because the lab prepared the compounds incorrectly.  When there is more than one causes that contributed, the causes are added vertically with an “and” between them.

Each impacted goal needs to eventually connect to the same Cause Map.  If they do not, the impacted goal may not be caused by the same problem and the goals should be revisited.

To continue building the Cause Map, keep asking “why” questions for each added cause until the level of detail is sufficient.

A Cause Map can be as high level or as detailed as needed.  The more significant the impact to the goals, the more likely a detailed Cause Map will be warranted.  Once the Cause Map is completed, it can be used to develop solutions to help prevent the problem from reoccurring.

In this example, the lab is currently changing management and reorganizing procedures to help prevent the similar problems in the future. 

To view an initial Cause Map for this issue, please click the “Download PDF” button above.

Last Wednesday, another set back occurred in the attempt to stem the flow of oil in the Gulf of Mexico from the a well head that was damaged when the Deepwater Horizon Oil Rig exploded on April 20 and sank 36 hours later .

The containment cap used to siphon oil from the damaged well head for the last three weeks had to be temporarily removed for more than 11 hours.  Before being removed, the containment system was sucking up about 29,000 gallons an hour.

So what happened?  Why remove a containment cap that had been working successful?

A root cause analysis of this problem can be built as a Cause Map.  A Cause Map is started by considering the impact to the goals and asking “why” questions to add Causes.  In this example, the first goal we will consider is the Environmental Goal.  Obviously, the environmental goal is impacted because there was additional oil released to the environment because the cap was removed. 

Continuing to ask “why” questions we can add additional causes.  The cap was removed because the ship connected to the containment cap system needed to be moved away from the well because there a safety concern because of the potential for an explosion.

There was an explosion concern because there was evidence that flammable gas was flowing up from the well head because liquid was being pushed out of a valve in the containment system.  This gas was getting into the containment cap system because an underwater vent was bumped by one of the remote-controlled submersible robots being used to monitor the damaged well.

More detail could be added to the Cause Map by continuing to ask why questions.  The detailed Cause Map could then be used to develop solutions that could be implemented to help prevent the problem from reoccurring.

Click on the “Download PDF” button above to view an initial Cause Map.

The containment cap was put back into place around 9 pm on June 23.  The efforts to contain and clean up the oil spill will continue for months and possibly years to come, but at least this small issue has been fixed.

A coal mine explosion in Amaga, Colombia on June 16, 2010 has left at least 18 dead, 1 injured and at least 53 people unaccounted for, and presumed dead.  The deaths and injuries resulted from a fireball caused by an explosion.  

Every explosion is caused by four factors: heat, fuel, oxygen and confinement.  In this case, the fuel was methane gas that had built up in the mine.  Methane is naturally produced as a byproduct of coal mining.  The methane was not removed from the mine because the mine lacked a methane ventilation pipe.  Additionally, the workers at the mine did not realize that methane levels were high because there was no gas detection system at the mine.

The number of dead and missing is so high because more people than usual were at the mine – the explosion happened during shift change.  Rescue efforts have been delayed by the high levels of gas in the mine, further increasing the number of deaths. 

By clicking “Download PDF” above, you can view the thorough root cause analysis built as a Cause Map in a simple, intuitive format that fits on one page.

Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.

On May 25, 2010, the National Highway Traffic Safety Administration (NHTSA) released new data about Toyota’s unintended acceleration issues, increasing the number of deaths potentially linked to the issue to 89.  Additionally, the NHTSA stated that nearly 6,200 complaints regarding acceleration issues in Toyotas have been received since 2000.

The acceleration issues have already resulted in massive recalls of Toyota vehicles in the US.  Nearly 5.4 million vehicles were recalled to fix issues with floor mats that could potentially shift out of position and an addition 2.3 million vehicles were recalled to repair sticking accelerator pedals.  No additional causes have been found for the acceleration issues at this time, but there are a wide range of theories that include electronic issues and solar flares.  Toyota denies that there are any additional causes of the acceleration at this time.

The US government is continuing to investigate the claims of unintended acceleration in Toyotas and an independent 15-month study by the National Academy of Sciences will begin in July.

A recent Wall Street Journal article discussed one of the stranger trends that have been found in the Toyota car crash data.  There have been an unusual number of accidents at beauty salons. 

Why beauty salons?

Just like any problem, this issue can be investigated using a root cause analysis built as a Cause Map.  In this case, the Safety goal would be impacted because there is a potential for injury for both the driver and people inside the salon.  Additional causes can be added to the Cause Map by, asking “why” questions and adding boxes to the right.

In this example, the article speculates that the some of the potential causes may be the age of the drivers involved (older women tend to visit salons more frequently), location of the salons (many are in strip malls near parking lots) or the architecture of salons (many have large glass windows that might distract drivers).  No formal investigation has been done to determine the actual causes of this strange trend, but it is interesting to lay out the potential causes and see what factors might be contributing to the hair salon car crashes.

Click on the “Download PDF” button above to view the initial Cause Map.

On April 20, 2010 about 10 pm a huge explosion rocked a semi-submersible drilling oil rig about 40 miles off the coast of Louisiana in the Gulf of Mexico. The oil rig was called the Deepwater Horizon and was owned by Transocean Ltd and leased to the British Petroleum Company through September 2013.

The oil rig burned for about 36 hours before sinking.  126 people were on the oil rig at the time of the explosion.  Eleven are missing and presumed dead and 4 were critically injured. Oil continues to leak from the wellhead more than a mile underwater on the ocean floor at an estimated rate of 42,000 gallons a day.

Remotely operated submersible vehicles were used to examine the wellhead.  The vehicles were also used in an effort to manually trigger the blowout preventer, which would close the wellhead and prevent any farther release of oil.  The blowout preventer is a 450-ton valve installed at the wellhead that is designed to automatically shut to prevent oil leaks in the event of an accident.  Attempts to manually close the blowout preventer have not been successful.

The other containment options being explored are drilling a separate well nearby to plug the flow at a location below the blowout preventer and building underwater domes that would contain the oil until it could be safely pumped to the surface for disposal.  Both of these alternatives are being actively worked and will take months to complete.  It is estimated that 4.2 million gallons of oil will be released if the blowout preventer is not able to be closed.

The cause of the explosion is unknown at this time.  An investigation is underway by the Coast Guard and the Minerals Management Service. 

A preliminary root cause analysis can be started using the information that is known and details can be added as they become available.  The analysis can be documented using a Cause Map which is a simple, intuitive format that visually lays out all known causes for an incident.  The first step in building a Cause Map is to determine how the organizational goals were impacted by the incident.  Causes for each impacted goal are determined to begin building the Cause Map.

In this case, the safety goal was impacted because 11 people were killed and several injured.  The environmental goal was impacted because there was a significant oil release.  The materials goal was impacted because the $700 million oil rig is a complete loss and the production/schedule goal was impacted because the oil drilling operation is shut down. 

Click on the “Download PDF” button above to view an initial Cause Map.