Root Cause Analysis

Commuter Rail/Freight Train Collision

On the afternoon of September 12, 2008, a Metrolink commuter train collided head-on with a Union Pacific freight train.  This tragic accident resulted in the deaths of 25, and injured 135, one of the worst train collisions in the country.  The National Transportation Safety Board (NTSB) is investigating the collision, but from primary information we can make a basic cause map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.

We’ll examine the impacts to the goals from the point of view of Metrolink, who operates the commuter rail.  There was an impact to the safety goal because 25 (20 passengers, 5 crew) were killed, and 135 were injured.  The customer service and production goals were both impacted because rail service has been suspended.  Additionally, there was severe damage to both trains, though the total cost is not yet known.

The suspension of service is due to the damage to the trains.  The deaths, injuries and damage to the trains were caused by the impact force.  The impact force is a result of the head-on collision of the two trains, which were both estimated to be traveling at 40 m.p.h.  (Whether or not that is a typical or accepted speed is not yet clear.)  The head-on collision occurred because the two trains were sharing the same tracks.  There is only a single track in this area because of a narrow tunnel, and the commuter train did not pull over onto siding (as occurs nearly almost every day so the freight train can pass).  The train did not pull over because the engineer failed to stop.  Whether that is because he didn’t follow protocol, didn’t notice the signal because of glare, fatigue, or other distractions, or if the signal malfunctioned is not yet known.  (A preliminary investigation by Metrolink indicated that the signal was functioning properly.)

As the NTSB completes its investigation, we will be able to add more detail to this map and remove potential causes that have been shown by investigation to be inappropriate.  As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.

Heart-Lung Transplant Blood Type Mismatch

On February 22nd, 2003, a patient at Duke University Medical Center died after receiving her second heart-lung transplant.  The first transplant she received was rejected by her body due to a blood type incompatibility (she was Type O, the organs were Type A).  The loss of her life was tragic enough, but it was compounded by the fact that the two rare heart-lung block donations she received could have saved the lives of others as well.

We can perform a thorough root cause analysis built as a Cause Map that can capture all of the causes in a simple, intuitive format that fits on one page.  Download this one-page document to see the midlevel Cause Map and write-up.  The death of a patient was an impact to the hospital’s patient safety goal.  The loss of the organs was an impact to the patient services goal. 

The mismatched blood type organs were transplanted because the procuring surgeon (sent to pick up the organs) was not told of the blood type, so he could not perform an effective blood matching.  Donor services offered organs with the incorrect blood type because they didn’t ask what that was, possibly because the surgeon had specifically asked for the organs for this patient, and they assumed that a surgeon wouldn’t ask unless the blood type was correct.  The surgeon didn’t verify the blood type of the organs because he assumed that donor services wouldn’t offer an organ of the wrong blood type (which is normally the case, per their regulations).  The mismatch was discovered in the laboratory, but not until too late in the procedure, because the surgery must begin while the organs are en-route, due to limited viability of the organs.

This tragic incident demonstrates the problem in making assumptions, and it shows us some areas where transplant safety can be improved.  Although this was a very rare case, both hospitals and the donor services are making improvements to their systems to ensure this never happens again.

Hubble Focusing Issues

The Hubble Space Telescope was launched on April 24, 1990.  Once in orbit, it was quickly discovered that the images from Hubble were blurred.  An investigation into the issue revealed that Hubble’s primary mirror was not built to specification and couldn’t properly focus the light.  Specifically, the mirror was flattened too much away from the center and caused the light reflected from the edge of the mirror to focus on a slightly different location than the light reflected from the center.   The primary mirror in Hubble was only off specification by 2.3 micrometers, but the result to the $1.5 billion dollar project was disastrous. 

Solving Hubble’s focus issues was no small feat.  How do you repair a mirror that can’t be replaced on orbit when it is cost prohibitive to bring it back to earth for repair?  The answer was to modify the lens (which met specifications) to work with the off specification mirror.  COSTAR (Corrective Optics Space Telescope Axial Replacement) was added to Hubble during the first servicing mission in December 1993.  COSTAR is essentially eyeglasses for Hubble, additional lens built with the same error as the mirror, but in the opposite direction so that the effects of the off specification mirror shape are canceled out.  With the addition of COSTAR, Hubble met original design goals.

The primary mirror was constructed with a flaw because the tool, called a null corrector, used to create the template to guide the shaping of the mirror was itself flawed.  Null correctors use precisely located mirrors and lens to determine the shape of a mirror.  In order to assemble null correctors, reflected light is used to measure the distance between the mirror and the lens inside the tool.  When the null corrector used to shape the Hubble’s primary mirror was assembled a measurement error was made.  A small amount of reflective coating had fallen off an internal piece of the instrument and the laser used to perform the measurement reflected off the wrong location, resulting in a lens being 1.3 mm to far from the mirror.  Null correctors are extremely precise and do not change once assembled so the Hubble team used a single instrument to guide the mirror shape.  A single flawed tool and inadequate quality controls resulted in a flawed mirror.

 A visual representation of root cause analysis has been created as a Cause Map that can be downloaded.

Pet Food Contamination - UPDATE

On May 22, 2008, Menu Foods and other pet food manufacturers agreed to a settlement on the class action lawsuit resulting from last year’s pet food contamination.  As part of the settlement, they will set up a $24 million fund to reimburse owners for expenses relating to pet deaths or injuries, screenings, and as compensation for food purchases.  This is in addition to $8 million that has already been paid to owners.  Also, they are required to screen for melamine, which owners say they are already doing.

The pet food manufacturers are bearing the brunt of the expense relating to the contamination issue.  But a root cause analysis shows that a significant portion of the blame lies in the regulatory process and dishonest raw material suppliers.  After all, the pet food manufacturers made pet food using raw materials that had been certified as meeting their requirements (which called for no foreign material contamination) and had not been flagged by the FDA. 

It has become increasingly clear that the FDA is not able to properly due its job in the increasingly global nature of U.S. foods and drugs.  The contaminated heparin found earlier this year shows that changes are too slow being made.  And, there is new evidence that private laboratory testing companies in the United States do the bidding of foreign importers who hire them, not the FDA.  These labs have stated that testing results for food entering the United States, no matter what kind of contamination they show, belong to the company.  This means that the results may only be released to the FDA once the company desires - or once a positive result has been obtained - no matter how many rounds of testing that requires.  Some labs have also claimed that importers “lab shop” - sending samples to lab after lab until they get the result they want.  Labs are not required to submit samples to the FDA.  So, the FDA may be in the dark about companies that repeatedly have contamination in their food products.

Dr. David Acheson, the FDA’s assistant commissioner for food protection supports congressional proposals that private labs be accredited by the FDA.  Hopefully action will be taken soon, before more tragedies occur.

General Slocum Steamship Fire

On June 15, 1904, a church group headed out for an excursion through New York City’s East River on the Steamship General Slocum.  Approximately half an hour after the ship left the pier, it caught fire.  Despite being only hundreds of yards from shore, the Captain continued to go full speed ahead in hopes of beaching at North Brother Island, where a hospital was located.  This served to fan the flames quickly over the entire highly flammable ship, killing many in the inferno.  Most of those who were not killed by the fire drowned, even though the Captain did successfully beach the ship at North Brother Island, due to the depth of the water and lack of safety equipment.

To perform a root cause analysis of the General Slocum tragedy, we can use a cause map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  First we look at the impact to the goals.  On the General Slocum there were at least 1,021 fatalities of the passengers and crew that were aboard.  (However, only two of the crew were killed.)  There were other goals that were affected but the magnitude of the loss of life makes any other goals less significant.  The deaths and injuries were the impact to the safety goals.

Passengers drowned because they were in water over their heads with inadequate help or safety equipment.  Passengers were either in the water because they fell when the deck collapsed, or because they jumped into the water trying to avoid the fire.  The water was too deep to stand because only the bow was in shallow water and the passengers could not reach the bow.  This was due to a poor decision on the Master’s part (namely his decision to beach the ship at a severe angle, with the bow in towards the island, instead of parallel to the island, where passengers would have been able to wade to shore.)  Note that the Master himself (and most of the crew) were on the bow side of the ship and were able to (and did) jump off and wade to shore.  The safety equipment, including life preservers, life boats, and life rafts, was mostly unusable due to inadequate upkeep and inadequate inspections.

Passengers (and two crewmembers) were also killed by fire.  Once the fire was started, it spread rapidly and was not put out.    The fire spread rapidly because the ship was highly flammable.  When this ship was constructed, there was no consideration of flammability.  Additionally, the current of air created by the vessel speeding ahead drove the fire across the ship.  The fact that an experienced Master would have allowed this situation was considered misconduct, negligence and inattention to duty - the charges for which the Master was later convicted.   The fire was not put out because of inadequate crew effort and insufficient fire-fighting equipment.  The crew effort was inadequate of a lack of training.  The fire-fighting equipment was insufficient because of inadequate upkeep and inadequate inspections.  (Possibly you are noticing a theme here?)

Although many people have not heard of the General Slocum tragedy, many of its lessons learned have been implemented to make ship travel safer today, although many of the solutions were not implemented widely enough or in time to prevent the Titanic disaster from occurring eight years later.  (Although there were actually more people killed on the General Slocum, it is believed that the Titanic disaster is more well known because the passengers on Titanic were wealthy, as opposed to the working class passengers on General Slocum.  It is also surmised that sympathy for the highly German population aboard General Slocum was diminished as World War I began.)

In a macabre ending to a gruesome story, ships began replacing their outdated, decrepit life preservers after the investigation on General Slocum.  It was later found that the company selling these new life preservers had hidden iron bars within the buoyant material, in a dastardly attempt to raise their apparent weight.  Unfortunately there were no adequate laws (then) against selling defective life-saving equipment.

Blood Substitute Risk

A study recently published by the Journal of the American Medical Association presented a review of clinical trials of hemoglobin-based blood substitutes.  This study showed that the clinical trials resulted in increased risk of heart attack and death for the patients being studied with no clinical benefit. 

We will examine this issue using the Cause Mapping process.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.

In clinical trials one of the overall goals is to have zero injuries. The blood substitute trials led to a 30% increased risk of death, and a 2.7-fold increase in heart attack, causing increased risk with no clinical benefit.  The two goals that are impacted in the blood substitute example are the safety goal and the customer service goal.

In this example all of our impacts to the goals are caused by the increased risk of heart attack (myocardial infarction).  Additionally, there was no clinical benefit shown because the use of blood substitutes did not limit blood transfusions. 

Why was there an increased risk of heart attack?  The increased risk of heart attack is caused by decreased blood flow, which is caused by blood vessel contraction (vasoconstriction).  This occurs because nitric oxide is responsible for blood dilation, hemoglobin molecules scavenge nitric oxide, and a patient receives an infusion of hemoglobin.

The patient receives an infusion of hemoglobin because the patients are unaware of the risk, and because of ongoing clinical trials of hemoglobin-based blood substitutes.  These trials are ongoing because hemoglobin-based blood substitutes have been developed and because clinical trials are being performed.

The hemoglobin-based blood substitutes have been developed because blood substitutes are being developed and most of the blood substitutes are hemoglobin-based, because hemoglobin is seen as the most promising substitute.  The blood substitutes are being developed because they would be better in remote areas or for portability, to help deal with the shortage of blood, and to reduce problems from blood transfusions.

The clinical trials were performed because they were approved by the FDA;  there was no checking by scientists, review boards, or the public; and the companies continued clinical trials.  There was no checking, and the companies continued the trials, because there was a lack of information available.

The FDA and the blood companies are still trying to figure out how to go forward based on these new results.  Because of the potential usefulness of blood substitutes, especially in military applications, it’s likely we’ll continue to see progress on this issue.

Reactor Vessel Head Degradation - Davis-Besse Nuclear Power Station

On March 7, 2002, during refueling, a cavity measuring approximately 4 x 6 inches was discovered that had completely eaten through the more than 6″ thick reactor pressure vessel head of Unit #1 reactor at Davis-Besse Nuclear Power Station.  Fortunately, the thin stainless steel cladding layer had held the reactor pressure, although it was not designed to do so.  The loss of the vessel head was also a loss of a principal fission product barrier (one of the three responsible for ensuring radiaoctive fission products remain within the pressure boundary).  This was an impact to the safety goal.  The loss of a principal fission product barrier is also considered a “significant precursor to core damage” by the NRC, which is another impact to the safety goals.  All told, the cavity resulted in nearly $300 million worth in repairs and upgrades, and a two-year closure of the plant, during electricty production at Davis-Besse was severely reduced.  These were impacts to the material, production, and customer service goals. 

Let’s examine a high level root cause analysis and review some of the causes of the cavity.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.

The cavity was caused by continued boric acid corrosion.  The corrosion occurred when leaking coolant evaporated into boric acid.  This occurred because of a through-wall crack in a nozzle caused by primary water stress corrosion cracking that was undetected.  The corrosion also occured because the leakage was undetected, due to delayed inspections and an ineffective leakage detection methods. 

The boric acid was not removed because it was not viewed as a safety concern.

The corrosion occurred due to inadequate corrosion control, where the corrosion was not detected because of a lack of a full inspection of the head,  and because early signs of corrosion were ignored, or missed.  The oil corrosion products were not completely removed, because they were difficult to remove and their removal was on a “best-effort” basis.  Additionally, the control was inadequate because the rate was higher than expected.

Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.

DeHavilland Comet Accidents - 1954

Sir Geoffrey de Havilland built the first commercial jet that reached production, the Comet.  The Comet design was finalized in 1945, as the British aircraft industry was attempting to establish a commercial aircraft industry post-World War II.  Prior to 1954, there had been some problems (a collision at take-off and a mid-air breakup) and some fixes to the hydraulic control system.  Then, on January 10, 1954 a Comet broke up in mid-air.  Flights were temporarily voluntarily suspended, then resumed.  On April 8, 1954, another Comet broke up in air.  (Both flights were taking off from Rome.)  The lives of 56 passengers and crew were lost in these two incidents, as well as two planes.  Additionally, the prestige of the British aviation industry suffered a blow.  (I’ll consider the lost prestige of British aviation a customer service impact.)

Let’s look at this incident in a Cause Map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  Although there were two separate plane breakups, the Cause Maps are the same (based on the analysis and investigation performed after the accidents).  Essentially, the two planes were lost due to a structural failure of the cabin, caused by fatigue growth of a crack beyond the critical crack length (in essence, the crack length at which crack propagation is so rapid as to be uncontrollable). 

The fatigue cracking of the cabin occurred because the actual pressure cycles exerted on the cabin were more than the allowable (or where cracking would occur).  This was because the allowable pressure cycles were miscalculated.  The allowable pressure cycles were miscalculated for several reasons.  First, the inadequate test program.  There was no prototype, and the fatigue tests were misleading.  One test used a section that was effectively pre-conditioned, extending its life.  In another test, the section tested was so small that the test results were influenced by boundary conditions.

Next, the actual stress was above the predicted stress.  This occurred because 1) the square shaped windows caused pressure stresses to be distributed unevenly and 2) because the actual stress increased in localized areas.  The local stress at rivet holes is far above general stress (usually along the order of three times general stress) and two rows of rivets were used to attach the window frame.

While the comet was being developed, there was a general lack of knowledge about fatigue.  Many designers (de Havilland included) thought that fatigue was associated with vibration, which did not affect jet engines.  Additionally, the spread in fatigue results is large (some experts quote as high as 9:1), meaning that one plane could fail nine times faster (or more slowly) than another.  You can see how this is a problem with a small test sample.

A last problem was that the design of the Comet stretched the bounds of experience.  The comet was designed to fly at twice the speed of other airliners, at twice the height, and at twice the cabin pressure (for passenger comfort).  As such, the design was a great extension of the existing body of knowledge in not just one, but three dimensions.

Probably the most important lesson to come from the de Havilland Comet accidents is the importance of proper testing.  Once the cause was discovered, the Comet was redesigned and flew successfully, although by then Boeing had mostly taken over the market share.  It’s tragic that these accidents had to occur before the problem was solved.

Brooklyn Bridge Turns 125

Brooklyn Bridge marks its 125th birthday on May 24, 2008.  When performing a root cause analysis it is easy to spend a large amount of time focused on failures, but today engineers should take a moment to appreciate the accomplishment of this truly amazing feat.  The bridge has been refurbished many times, but the towers, main cables, and main beams are original and are now 125 years old.

At the time the Brooklyn Bridge was constructed the 6,000 ft long bridge was roughly six times as long as the longest bridge of the type that had previously been built.  The Brooklyn Bridge is one of the nation’s oldest and most treasured suspension bridges.  It has shaped the development of New York City.  At the time it was constructed Brooklyn was largely rural and the bridge helped sparked a growth spurt that dramatically changed the face of Brooklyn.  Brooklyn’s population grew by 42 percent between 1880 and 1890.  At last count in 2006, the bridge carried 126,000 cars per day.

Recent inspections have revealed some deterioration of the bridge, primarily with the newer approach ramps.  In a recent survey, state inspections ranked its condition as “poor”.  New York City plans to spend $250 million to 300 million to fix and repaint the bridge.  Hopefully these updates will return the bridge to good condition and it will continue to safely serve the citizen of New York City for many decades to come.