Root Cause Analysis

Loss of Mars Climate Orbitor

The Mars Climate Orbiter (MCO) was launched atop a Delta II launch vehicle on December 11, 1998.  Nine and a half months after launch, the MCO was scheduled to begin the process of establishing an orbit around Mars.  The plan was to use a technique called aerobraking to reduce the MCO velocity and slowly move the MCO from a 14 hour orbit to a 2 hour orbit.  On September 23, the $125 million dollar MCO was lost during the attempt to establish orbit around Mars.  Investigation into the accident revealed that the orbiter had entered the Martian atmosphere traveling too quickly with too low a trajectory.  The heat produced by friction from hitting the thicker atmosphere present at the lower trajectory at high velocity destroyed the orbiter.  The loss of the MCO cost NASA more than the $125 million dollars spent building the MCO.  In addition, NASA lost a substantial amount of time, lost all potentially gathered data, and lost some of the public support for the NASA program.

NASA investigation revealed many causes of the loss of the orbiter.  One of the most obvious causes is a unit error in the software used to help predict the velocity of the MCO, which in turn is used to predict the trajectory the MCO would enter Martian atmosphere. A little background is needed to understand how an error in the software causes errors in the predicted velocity.   Software called “Small Forces” is used to predict how the MCO’s velocity changed after a angular momentum desturation maneuver.  A angular momentum desturation maneuver is performed when one of the momentum wheels used to help the orbiter maintain orientation in space starts spinning too quickly.  During an angular momentum desturation maneuver, a wheel is deliberately slowed down (which would normally turn the spacecraft) while at the same time a jet is fired to counteract this force and keep the orientation relatively constant.  This whole process affects the speed the spacecraft is traveling and affects the trajectory of entry in the Mars atmosphere.  The error in Small Forces was simple one.  The results were in pound force and the program that predicted velocity expected them to be in Newtons.

The attached PDF file contains an intermediate level root cause analysis of the loss of the MCO.  It was built using  facts from media reports and the NASA investigation reports. The map can be expanded using all the known data to create a detailed Cause Map.

UPDATE: FDA releases revised death count from heparin contamination

The Food & Drug Adminstration (FDA) recently reviewed adverse events related to heparin, which has been the subject of much scrutiny after 19 deaths were reported due to allergic reactions from contaminated vials.  Since January 2007, the FDA has received reports of 103 deaths from people taking heparin, 62 of which involved allergic reactions or hypotension (dangerously low blood pressure).  These deaths include people who were taking all brands of heparin, not just the brand affected by the contamination and recall.  The manufacturers of the brand that was contaminated and recalled says that they know of only 4 deaths assocciated with their contaminated product.  The FDA has stated that this does not mean that the deaths were necessarily caused by the allergic reactions and low blood pressure.  Although allergic reactions and low blood pressure were the cause of death of those who have died from the contaminated vials, it’s not clear that all 62 deaths are associated with contaminated heparin.    In fact, heparin carries a warning detailing risk of low blood pressure.  However, in the year 2006, only 55 deaths were reported from heparin, and only 3 were due to allergic reactions.  So there is obviously something that is increasing the number of allergic reactions to heparin.  Hopefully the increase in deaths is due to the contaminated heparin that has already been recalled from the market, but it’s possible that there are other issues, or other brands that are also contaminated.  The FDA continues to investigate, and hopefully can provide answers soon, especially to the people who depend on heparin for their well-being. 

The previous blog entry shows an intermediate level Cause Map (root cause analysis) as a downloadable pdf.

UPDATE: Grounded Flights for American Airlines

American Airlines resumed a normal flight schedule Saturday afternoon, ending a period of widespread flight cancellations.  Between April 8 and 12, 3,300 flights were canceled when all MD-80 jetliners in the American Airlines fleet were grounded.    More than a quarter of a million passengers were affected by the widespread flight cancellations.  As discussed in a previous blog, these drastic measures were taken when a large percentage of inspected MD-80s failed to meet FAA regulations on wiring from the airframe to a pump in the wheel well.  The wiring can be a fire hazard and affect power distribution. An intermediate level Cause Map showing the causes of the cancellations can be seen in the previous blog posted on April 10.

The cancellations may be over, but the effects will continue to linger.  The cost to the American Airline is estimated to be in the tens of millions of dollars.  In addition to lost revenue, American Airlines gave many inconvenienced passengers $500 travel vouchers and paid to put stranded travelers in hotels.  It is also difficult to put a financial cost on the huge amount of negative publicity that the airline has received as a result of these cancellations, but it is guaranteed to affect their business.  In addition to the financial burden of these cancellations, the entire airline industry is faced with raising fuel costs and this is going to put even more pressure on American Airlines.  Already, American Airlines announced on Friday (ironically on a day when nearly 600 flights were canceled) that it will be raising prices by as much as $30 a round trip tickets to help compensate for high fuel costs.  These dual blows to the bottom line are going to affect the health of the American Airline company for the foreseeable future.

It is also likely that many other airlines will be similarlly affected.  Doing a root cause analysis, it is clear that one of the causes of these cancellations is a new focus by the FAA on “zero tolerance” for any deviations from their detailed regulations.  As airlines struggle to understand the new inspection criteria, it is likely that other airlines will face cancellations.  The airline industry as a whole is facing some high hurdles in the upcoming months.  Four discount carriers have already declared bankruptcy in the last month and it is likely others will follow suit.  Even the established, traditional carriers are seeking changes to stay competitive.  For example, rumors are circulating about a possible Northwest and Delta merger.  This is going to be a turbulent time for Airlines and passengers.

Grounded Flights for American Airlines

Starting April 8, 2008, American Airlines grounded nearly half of its fleet when it pulled all 300 McDonell Douglas jets (MD-80s) from service.  At least 2,400 flights were canceled.  It is estimated that 100 passengers would have been on each of the canceled flights, bringing the total of affected passengers to nearly a quarter of a million people.  The MD-80s were grounded because 15 of 19 inspected aircraft failed FAA inspection this week.  The issue is with the installation of wiring connecting the airframe to a hydraulic pump in the wheel well.  The regulations are written to prevent rubbing and chafing of the wiring, which can lead to exposed wiring.  Exposed wiring is a concern because it can to power issues and shorts, and it is a potential fire hazard.

The most alarming part of the story is that American Airlines grounded these same planes for the exact same issue on March 26 and 27.  Over 350 flights were canceled while the planes were inspected and repaired if necessary to compile with the FAA wiring regulations.  All planes were back in service on March 28 after American Airlines asserted they satisfied the regulation.  Little information is available on what went wrong two weeks ago.   There are a number of questions that would need to be answered to perform a thorough investigation.  Are the FAA regulations confusing?  Do the AA mechanics need additional training?  Did the airline fail to internally check the wiring prior to putting the planes back into service?   If an inspection did occur, did the inspectors understand what they were looking for?   It may not be clear exactly what went wrong, but it is clear that something failed in the system to cause this second round of cancellations.

The attached PDF file contains an intermediate level root cause analysis of the cancellation of American Airline flights on April 8-9.  It was built using the facts that were available in media report.  There are many details still missing, that could be added as more details are known.

Root Cause Analysis: Monte Carlo Hotel Fire - Las Vegas, NV

Just before 11 am on January 25, 2008, a fire started on the roof of the 32 story Monte Carlo Hotel in Las Vegas.  The fire spread quickly along the outside of the building, fueled by the highly flammable foam like material, Exterior Insulation Finishing System (EIFS), used to construct the hotel façade.  A spark from a hand held cutting torch being used on the roof of the hotel hit the EIFS and started the fire.  6,000 guests and workers were evacuated from the hotel.  The hotel remained closed until February 15.   Considering both the damage to the hotel and lost business, the total cost of the fire is approximately $100 million dollars.  Luckily, no major injuries resulted from the fire.

A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  The Cause Map shows that the fire started because a spark from a hand held torch hit a flammable material.  The Cause Map can also be used to identify possible solutions that would prevent another fire.  In this case, two areas that would merit farther investigation would be the use of highly flammable material on buildings and the lack of protective measures taken to protect the EIFS from the sparks.  For example, there were no mats in place to protect the EIFS from being hit by sparks.  From the information available, it isn’t clear why no protective measures were taken to protect the EIFS, but it is known that the contractor failed to obtain the correct permit (which involves getting information on appropriate safety procedures). It is reported in an Associated Press article on the fire that Las Vegas city officials are currently evaluating whether restrictions should be placed on the use of EIFS.

The attached PDF file contains an intermediate level root cause analysis of the hotel fire.  It was built using the facts that were available in media reports on the fire.  As more details are known, the Cause Map can be expanded.

Heparin Contamination - 19 Lives Lost

Heparin, which is widely used as an anticoagulant (blood thinner) has been in the news lately and the news is scary.  19 people have died, and 785 have experienced adverse reactions due to contaminated heparin.  The heparin in question has been found to contain up to 50% oversulfated chondroitin sulfate, which mimics heparin so closely it can not be distinguished in basic tests but provides no anticoagulant activity.  The adverse effects are caused by severe allergic reactions, including low blood pressure which can occasionally lead to fatal stroke.

Whether or not the chondroitin sulfate is to blame for the allergic reactions, it also has the potential to cause serious harm by negatively affecting the blood thinning properties of Heparin.  People who take heparin because they require its anticoagulant properties may have serious difficulties with a dose that is only 50% effective.  Because of these concerns, the Heparin in question is taken off the market.  But serious consumer concerns remain about the system that allowed the contamination to happen in the first place.  Due to the potential for fatal side effects, lots of heparin (the total amount is unclear) have been recalled from 6 countries (at last count).

A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  Click on the pdf document below for a more detailed analysis.

What do we miss by focusing on “THE ROOT CAUSE”?

Many organizations focus on trying to boil down their problems, even extremely complex ones, into the one “root cause”.  One of the problems with this is the overgeneralization that results.  This overgeneralization may allow organizations to feel that they are “off the hook” if, for example, the “root cause” ends up being “human error.”  Because human error is unavoidable, there may be no steps taken to prevent or mitigate further occurences.  Overgeneralization can also lead to a warped perspective of the problem in question, based on the desire to find the one true “root cause” of the event.  If you ask people what the cause of the EXXON VALDEZ oil spill was, many people will say that the Captain was drunk (overgeneralization of a complex issue into human error, specifically pointed at the man in charge).  However, not only was the Captain not present on the bridge at the time of the grounding which resulted in the oil spill, he was found not guilty of operating a vessel under the influence of alcohol. 

Another issue with attempting to find the “root cause” is all of the other contributing causes that will be missed.  This is especially important when the solution for the “root cause” is not 100% effective.  TWA Flight 800 went down for many reasons, but according to the National Transportation Safety Board, the airlines sole focus on preventing fuel tank explosions is preventing ignition energy from entering the tank.  However, that solution is not foolproof - ignition sources can be minimized but not entirely removed.  That is why some are turning their focus towards solutions for the other causes - namely the flammability of the fuel tanks and the presence of oxygen that would allow an explosion to occur.

So if finding the “root cause” isn’t the answer, what is?  Well, in order to effectively combat a problem, we have to find the best solution.  In order to find the best solution, we have to find all the solutions, and in order to find all the solutions, we have to find all the causes.  We do this by making a Cause Map, a visual root cause analysis.  This Cause Map asks “why” until all possible and contributing causes have been identified.  The next step is to identify any potential solutions for each cause.  Once all potential solutions have been found, an organization needs to determine which solution, or solutions, is best based on the severity of the issue, the effectiveness of the solution(s) and the availability of resources to implement the solution(s). 

When a Cause isn’t a Cause: The Failure of Vytorin

Vytoria is a drug intended to improve heart disease.  There are already millions taking it, or one of its parts.  Full results of its trial were released Sunday, March 30th.  Although Vytorin successfully reduced three key risk factors, it did not improve heart disease, because it had no effect on reducing plaque.  The three risk factors improved by Vytorin, and thought to lead to plaque buildup, which leads to heart disease, were LDL (low-density lipoprotein, or bad cholesterol), triglycerides (a form of fat made in the blood), and  artery inflammation as measured by CRP (C-reactive protein, which is released into the blood due to inflammation).  So, if we look at the root cause analysis, we have:

But if this is our Cause Map, and we reduce all three causes, we should reduce the result - plaque formation, which should reduce the occurrence of heart disease.  If we end up with the results we have here, which is no effect on plaque buildup despite proof that the three causes (called “key risk factors” in the medical world) have been reduced, it means there’s a problem with our root cause analysis.  This particular analysis gets even more confusing.  Some drugs, like statins, lower LDL and successfully reduce heart disease.  This implies that the cause-and-effect relationship of LDL and heart disease is valid.  But there was a drug that is no longer being advanced that successfully reduced cholesterol, but actually raised heart risks.  What does all this mean?  It means back to the drawing board on our cause map.  I don’t pretend to have the answers - I don’t think anybody does, or there would be a new drug out there right now - but it means that as you’re reading this, the smart folks developing new drugs are donning their lab coats and trying to figure out what went wrong.

Lexington Plane Crash 2006

Incident Date: August 27, 2006

Root cause analysis can be a very effective technique to analyze a problem.  But what if the evidence trail goes cold?  Is creating a Cause Map still useful when unanswered questions remain after a thorough investigation?  The crash of a Comair jet in Lexington Kentucky on August 27, 2006 is a good example of this situation.  The plane crashed during takeoff, killing 49 people . The flight crew mistakenly attempted to takeoff on the wrong runway, which was too short for the plane to reach the necessary speed for lift off.  Even after a detailed investigation by the National Transportation and Safety Board, it still is not clear why the flight crew used the wrong runway.   As an aside, the pilot and the first officer were competent professionals from all accounts and there is no history of either making errors of this magnitude.

Plane crashes are unique in the fact that there is a lot of data available to investigators.  The cockpit voice recorder (CVR) records all conversations in the cockpit and the flight data recorder (FDR) records instrument readings.  Usually the reason behind plane crashes can be determined using all this data.  In this case, the information did provide some useful insight, but no clear reasons why the mistake occurred. 

Building a Cause Map of this accident does make one thing very clear.  There are many events that had to occur for this mistake to happen.  One of the causes of the plane crash is clearly the error on the part of the flight crew, but another cause is the failure of the traffic controller to catch and correct the error.   There were two separate windows of time where the controller had an opportunity to prevent the plane crash, but didn’t for a variety of reasons.

It’s tempting to say the plane crashed because the crew used the wrong runway and leave it at that. The main problem with this line of reasoning is that this conclusion doesn’t help prevent future crashes, especially since the error isn’t well understood.  If all the focus is placed on why the wrong runway was used, an opportunity to improve the process and prevent future accidents is lost.  In a case where there is missing information, building a cause map can be useful because it helps the investigation to explore all the causes and potential solutions.  Only one cause needs to be eliminated to prevent the accident. For instances, the crew could had lined up at the runway and the accident could have still been prevented if the controller had caught the mistake.  Focusing on a solution to eliminate the better understood causes provides a useful place to start.

A high-level Cause Map of the problem is below:

Levee Break - Fernley, NV

Incident Date: January 5, 2008 

Just after 4 a.m. on January 5th, 2008 about 600 homes began flooding in Fernley, Nevada, about 25 miles East of Reno.  A 50 foot section of a canal embankment failed flooding the adjacent area.  The 32-mile canal carried water from the Truckee River south to Fallon area farms.  There were no injuries in the flooding but it easily could have been very serious.  The complete estimates for repairing the canal and the homes are not available at this time.

A report issued by the U.S. Bureau of Reclamation released March 20th concluded that the century-old irrigation canal failed due to burrowing rodents.  A simple root cause analysis for this incident using the Cause Mapping method captures the tunneled holes in the embankment as one of the causes.  Another one of the causes is the increased water flow in the canal caused by the nearly 2 inches of rain that fell the day before.  The annual rainfall for the area is about 5 inches.

The Cause Map shows that the canal obviously failed because the stress on the embankment was greater than the strength of the embankment.  The increased water flow added to the stress on the embankment while the holes tunneled by the rodents reduced the strength.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.

Since the canal is almost 100 years old tunneling muskrats are not a surprise.  If the holes would have been identified earlier and filled, the risk of the breach would have been reduced significantly.  The evidence that the inspection and maintenance of the canals was ineffective is the fact that the canal failed due to holes.  An effective inspection program would have found the holes and addressed them - that’s the purpose of inspection and maintenance.  Past inspections may have been conducted exactly as required, which simply means the previous inspection requirements were inadequate.  Ineffective inspections is one of the causes of the canal failure that would need to be investitgated further.

The attached PDF file contains an intermediate level root cause analysis of the canal failure.  It includes causes that were considered in the Bureau of Reclamation report as well as some of the evidence and solutions.  A more detailed Cause Map can be created from the specific information in the bureau’s report