Root Cause Analysis

DeHavilland Comet Accidents - 1954

Sir Geoffrey de Havilland built the first commercial jet that reached production, the Comet.  The Comet design was finalized in 1945, as the British aircraft industry was attempting to establish a commercial aircraft industry post-World War II.  Prior to 1954, there had been some problems (a collision at take-off and a mid-air breakup) and some fixes to the hydraulic control system.  Then, on January 10, 1954 a Comet broke up in mid-air.  Flights were temporarily voluntarily suspended, then resumed.  On April 8, 1954, another Comet broke up in air.  (Both flights were taking off from Rome.)  The lives of 56 passengers and crew were lost in these two incidents, as well as two planes.  Additionally, the prestige of the British aviation industry suffered a blow.  (I’ll consider the lost prestige of British aviation a customer service impact.)

Let’s look at this incident in a Cause Map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  Although there were two separate plane breakups, the Cause Maps are the same (based on the analysis and investigation performed after the accidents).  Essentially, the two planes were lost due to a structural failure of the cabin, caused by fatigue growth of a crack beyond the critical crack length (in essence, the crack length at which crack propagation is so rapid as to be uncontrollable). 

The fatigue cracking of the cabin occurred because the actual pressure cycles exerted on the cabin were more than the allowable (or where cracking would occur).  This was because the allowable pressure cycles were miscalculated.  The allowable pressure cycles were miscalculated for several reasons.  First, the inadequate test program.  There was no prototype, and the fatigue tests were misleading.  One test used a section that was effectively pre-conditioned, extending its life.  In another test, the section tested was so small that the test results were influenced by boundary conditions.

Next, the actual stress was above the predicted stress.  This occurred because 1) the square shaped windows caused pressure stresses to be distributed unevenly and 2) because the actual stress increased in localized areas.  The local stress at rivet holes is far above general stress (usually along the order of three times general stress) and two rows of rivets were used to attach the window frame.

While the comet was being developed, there was a general lack of knowledge about fatigue.  Many designers (de Havilland included) thought that fatigue was associated with vibration, which did not affect jet engines.  Additionally, the spread in fatigue results is large (some experts quote as high as 9:1), meaning that one plane could fail nine times faster (or more slowly) than another.  You can see how this is a problem with a small test sample.

A last problem was that the design of the Comet stretched the bounds of experience.  The comet was designed to fly at twice the speed of other airliners, at twice the height, and at twice the cabin pressure (for passenger comfort).  As such, the design was a great extension of the existing body of knowledge in not just one, but three dimensions.

Probably the most important lesson to come from the de Havilland Comet accidents is the importance of proper testing.  Once the cause was discovered, the Comet was redesigned and flew successfully, although by then Boeing had mostly taken over the market share.  It’s tragic that these accidents had to occur before the problem was solved.

Brooklyn Bridge Turns 125

Brooklyn Bridge marks its 125th birthday on May 24, 2008.  When performing a root cause analysis it is easy to spend a large amount of time focused on failures, but today engineers should take a moment to appreciate the accomplishment of this truly amazing feat.  The bridge has been refurbished many times, but the towers, main cables, and main beams are original and are now 125 years old.

At the time the Brooklyn Bridge was constructed the 6,000 ft long bridge was roughly six times as long as the longest bridge of the type that had previously been built.  The Brooklyn Bridge is one of the nation’s oldest and most treasured suspension bridges.  It has shaped the development of New York City.  At the time it was constructed Brooklyn was largely rural and the bridge helped sparked a growth spurt that dramatically changed the face of Brooklyn.  Brooklyn’s population grew by 42 percent between 1880 and 1890.  At last count in 2006, the bridge carried 126,000 cars per day.

Recent inspections have revealed some deterioration of the bridge, primarily with the newer approach ramps.  In a recent survey, state inspections ranked its condition as “poor”.  New York City plans to spend $250 million to 300 million to fix and repaint the bridge.  Hopefully these updates will return the bridge to good condition and it will continue to safely serve the citizen of New York City for many decades to come.

Slips, Trips and Falls - A Root Cause Analysis Primer

Slips, trips and falls happen every day.  Falls are responsible for tens of thousands of deaths each year.  (Slips and trips are considered a subset of falls, and are included in these numbers.)  Falls on the job account for 12-15% of all worker’s comp costs.  The direct and indirect costs of workers injured and killed on the job are estimated to be billions of dollars each year, both in worker’s comp claims and in lost productivity.  In 1999, as an example, 5,100 workers were killed by falls and over 570,000 injuries were reported.  However, there are many things that can be done to prevent and lessen the impact of falls.  Creating a Cause Map - a visual root cause analysis - will allow us to identify all the potential causes of falls.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  Once we’ve done that, we can identify the best solutions.

A worker is injured during a fall because the worker strikes the floor, or another object, and the object contacted is hard, and the worker hits in a way that causes injury.  When I say that workers are injured because they hit an object in a way that causes injury, what I am really talking about is factors that worsen a fall, and make injury more likely. The worker could land on a part of his or her body that is more easily injured.  Another way that injuries can be worsened is if a worker falls farther than his or her height (i.e., not a same-level fall).

The worker strikes the floor or other object because he or she falls, and there is no other support for the body, such as a handrail, or a harness.   There are four different ways to fall: slips, trips, the “step and fall” - where a person gets off-balance while stepping - and becoming unbalanced on moving equipment.

A worker slips when there is inadequate traction, either because the force of stepping off is too high, or the coefficient of friction is too low.  The force of stepping off can be higher than average if the worker is walking quickly or running, making a sudden change in direction, or if he or she has an awkward gait, from injury or old age, for example.  The coefficient of friction is a function of the traction provided by the shoes the worker is wearing and the “slipperiness” of the walking surface.  The coefficient of friction is too low if the traction of the worker’s shoes is inadequate and if the floor is slippery, because the surface is wet, icy and/or oily and does not have a non-skid coating.  Of course, for this to be an issue at all, the worker has to step into the slippery area. 

A worker can become off-balance by encountering an unexpected height difference (known as the “step and fall”).  This occurs in one of two ways.  Either the front foot lands on a surface lower than expected, or the ankle turns due to one side of the foot ending up higher than the other side, with footwear that inadequately supports the ankle.  These are both due to an unexpected height difference.

When a worker trips, it is because his or her toe is stopped, but his or her upper body is not stopped.  The upper body is moving because the worker is moving and he toe is topped because it encounters an object in the walking path, a rise in the walking path, or a difference in height of subsequent stairs. 

Last but not least, falls can be caused by workers who become unbalanced on moving equipment.  For this to occur, the worker must be inadequately secured to the equipment while the equipment changes motion, either by turning, decelerating or stopping, or accelerating or starting to move.

Once we have finished our Cause Map and found all the potential causes, we can assign potential solutions to all appropriate causes.  The solutions that I have come up with are in green boxes, near the cause(s) they “solve”.   You can see that some of the solutions are the responsibility of the company, and some are the responsibility of the worker, and some are both.   Although many of the responsibilties lie with the worker, it is in a company’s best interest to provide training on how to prevent, manage and mitigate falls.  Falls may seem like everyday, ordinary minor occurences.  While falls are everyday occurences, the consequences can be anything but minor.

The attached PDF document shows the visual root cause analysis as a Cause Map.

SL-1 Explosion - January 3, 1961

The only fatal reactor accident in the United States occurred on January 3, 1961, when an Army prototype known as SL-1 (for stationary, low power reactor, unit 1) exploded, killing the 3 operators who were present.  We’ll use the SL-1 tragedy as an example of how the Cause Mapping process can be applied to a specific incident.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.

The SL-1 tragedy killed the three operators present, which is an impact to the safety goal.  Another goal is that there be no damage to the vessel. In the case of SL-1, the  vessel sustained extensive damage.

The loss of life and vessel damage were both caused by the reactor exploding.  The reactor exploded because it went prompt critical (an uncontrollable, exponentially increasing fission reaction).  The reactor went prompt critical because withdrawal of the central rod can cause prompt criticality and because the rod was rapidly, manually lifted 26.4″ out of the core.

Withdrawal of the central rod can cause prompt criticality due to a lack of shutdown margin in the core, and inadequate safety criteria.

Because most of the evidence was so effectively destroyed, nobody really knows why the control rod was lifted out of the core.  There are two theories (disregarding the bizarre and improbable murder/suicide theory): 1) the control rod got stuck while being lifted to be attached to the drive mechanism, and, as the operator was exerting greater force on it, suddenly came free, resulting in a lift far greater than intended, or that an rod drop testing/exercising was performed improperly.

The control rod was stuck, and came free while being attached because it was required to be lifted 4″ out of the core and because control rods had been sticking.  The control rods had been sticking for one or more of the following reasons: 1) reduced clearances due to radiation damage (which can cause structural material to swell), 2) the passage was blocked due to loss of poison strips in the channel, caused by poor design and inadequate testing, or 3) lifting equipment not working properly due to inadequate lifting capacity of the lifting equipment.

Exercising/testing was potentially improperly performed.  This could have occurred because the operators chose to exercise/test the rods, attempting to ensure that they would perform properly, and because they didn’t realize what would happen. This is because of inadequate training and inadequate work instructions.  The testing was also potentially done improperly due to inadequate work instructions.

On a positive note, the SL-1 incident did initiate some positive changes in the nuclear industry.  Most notably, reactor design has improved and incorporated a “one-rod stuck” criteria which specifies that a reactor can NOT go critical by the removal of any one control rod.  Additionally, procedures and training have gotten more intense and more formal, and planning for emergencies has increased.

The attached pdf gives a visual representation of the intermediate level root cause analysis, the Cause Map.  It can be printed out to fit on one page.

Train Derailment - Lafayette, Louisiana

About 1:40 am on May 17, six rail cars derailed and overturned near Lafayette, Louisiana.  One of the cars was damaged and leaked about 11,000 gallons of hydrochloric acid.  Authorities evaluated people with 1 mile of the accident.  Approximately 3,000 people were affected, including a few small businesses and a nursing home.  Five people, including two rail workers, were sent to a hospital and treated for eye and skin irritation.  All affect people are being reimbursed for food and hotel costs by the railway company that operated the train, BNSF Railway.

 The was potential for farther release of chemicals because one of other rail cars involved in the accident carried ethylene oxide, a flammable and dangerous chemical, and two of the remaining cars also carried hydrochloric acid. 

The Louisiana State Police’s hazardous materials unit is overseeing clean-up of the accident sit.  The spill is being neutralized with lime and the contaminated material will be removed and disposed of.  The rail car containing ethylene oxide was removed from the site as quickly to remove the potential for additional problems.

The cause of the derailment is not known at this time.  The Federal Railroad Administration will conduct an investigation of the accident.

The attached PDF file contains an intermediate level root cause analysis of the train derailment.  It was built using the facts that were available in media reports on the accident.  As more details are known, the Cause Map can be expanded.

Prudhoe Bay Pipeline Corrosion

In 2006, a British Petroleum (BP) worker in Prudhoe Bay, Alaska discovered a leak in its transit, or feeder pipelines (which deliver the crude oil drilled by BP to the main Trans-Alaskan Pipeline, which transports the oil to Valdez, in the southern part of Alaska.  The oil is taken from there in ships to the lower 48.)  Approximately 5,000 barrels (more than 200,000 gallons) of oil were spilled, adversely affecting almost 2 acres of permafrost (continually frozen soil).  During inspections performed as a result of the spill, severe corrosion (and another, smaller spill) was discovered in 16 miles of pipeline.  BP decided to replace all 16 miles of affected pipeline, at a cost of $260 million.  Additionally, BP paid $20 million in fines, restitution to the State of Alaska, and a payment for environmental research.  Some people believe this is the largest fine ever paid in the state for what was legally considered an “environmental misdemeanor.”

A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page (see attached pdf).  First we will look at the impact to the goals.  For the BP pipeline the environmental goal was impacted because 5,000 barrels of crude oil spilled, which affect on 1.9 acres of permafrost.  The impact on the two environmental goals was caused by the leak of crude oil.   The customer service goal was impact due to an increase in oil prices to consumers.  This occurred because  of the loss of barrels during the production shutdown, which is an impact to our production goal.  The loss of oil is due to the shutdown, which occurred in order to replace the affected lines, which is also an impact to the material goals.  The lines had to be replaced because a loss of pipe integrity was discovered, which led to the fine and restitution, which is also an impact to the material goal.

The loss of pipe integrity was due to severe corrosion product buildup.  The corrosion product buildup also resulted in a hole in the pipeline, which caused the leak.

The permafrost that was affected by the oil because of the leak, but also because the leak was not contained promptly. The next question is “Why was the leak not contained promptly?”  (And also, “Why did the leak occur?” which was due to a hole, but we’ll get to that later.)  The leak was not contained promptly because the the leaked oil was not visible, the location of the leak was inaccessible, and the leak detection program was ineffective. 

The severe corrosion product buildup resulted from three things.  First, there was corrosion in the pipe.  Second, the corrosion went undetected (we’ll go into both of these in more detail).  And third, the pipes were used beyond their design life (25 years vs. the 29 years they had been in service.

There was corrosion in the pipes because there were microbes in the pipe protected by a layer of sediment, and microbes produce corrosive substances (this is known as internal microbiological corrosion).  This layer of sediment was due to an ineffective maintenance program.  It settled to the bottom of the pipe because there were low spots in the pipe, and because the velocity of the oil was too low to remove sediment because the pipe diameter was too large.

The corrosion went undetected because of an ineffective inspection program.  The inspection progrm was ineffective because there was not a regular internal inspection schedule.  The ultrasonic testing used was not effective because it did not cover 100% of the line and the damage was very localized (thus the ultrasonic testing was missing the spots with the worst corrosion).  Additionally, a “smart pig”, which is used internally to measure the wall thickness of a line, was never run through the line, because BP did not believe it was necessary as they performed ultrasonic testing.

Once the Cause Map is build to a sufficient level of detail with supporting evidence the solutions step can be started. The Cause Map is used to identify all the possible solutions for given issue so that the best solutions can be selected. On the Cause Map you can see some solutions derived from the causes (in the green boxes).  Looking through news reports or BP press releases regarding their Prudhoe Bay pipeline, you’ll see that almost all of the actions listed have been or are being taken to prevent this problem from happening again.

Pet Food Contamination - March 2007

On March 15, 2007, the Food and Drug Adminstration (FDA) was notified that ten animals had died from eating pet food.   This began an investigation into a problem that would result in the recall of 150 brands of pet food, and would kill many animals - some veterinarians suggest up to 1,000.  We can illustrate what happened in a Cause Map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page. 

First, we examine the impacts to the goals.  For a food manufacturer, one of the overall goals is to have zero injuries.  Some veterinarians suggest that up to 1,000 dogs and cats were killed in the U.S.  One of the other goals impacted is the customer service goal. In the case of the contaminated pet food, 150 brands (with 60 million containers of Menu Foods pet food, the most affected brand) were recalled.  This was the largest recall in FDA history, and was estimated to cost Menu Foods $54 million.

The loss of pets was caused by renal failure.  The renal failure in dogs and cats occurred because the dogs and cats ate contaminated pet food.  The dogs and cats ate contaminated pet food because it was in the food supply.  This also led to the recall.

Why was the contaminated pet food in the food supply?  The food was contaminated with up to 6% melamine and cyanuric acid (CA), and the contaminants were not detected.  The melamine and cyanuric acid (CA) were found in the food because they were added to the raw ingredients to increase the apparent content of the wheat gluten.  This reduced the cost for the manufacturer because melamine and cyanuric acid are cheaper than wheat gluten.  It increased the apparent protein content because melamine and cyanuric acid mimic the protein response in protein testing. 

The contaminants were not detected because standard tests did not detect them, and because of inadequate insepctions and inaccurate paperwork.  Standard tests did not detect the contaminants because melamine and cyanuric acid mimic protein response in protein testing, and because they were not tested for.  Inspections were inadequate because the material did not receive export inspections in China, because the exports were improperly labeled as non-food, and only food items are subject to mandatory inspection.  The inspections were also inadequate because FDA officials do not have ready access to Chinese plants because there is no binding agreement between China and the FDA.  The paperwork was inaccurate because the broker certified that the material specification was met, and the material specification forbid foreign material.

Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.  See the attached pdf for a visual representation of the cause and effect relationships.

Mission to Hubble Telescope Delay

Early in March 2008, NASA announced that the shuttle mission to the Hubble telescope would take place in the fall rather than in August as originally scheduled.  A trip to Hubble is necessary to replace gyroscopes and batteries that failing.  Additionally, the mission will also be sued to install instruments that will increase the range of the telescope.   The changing schedule itself is not a cause for alarm, but the reasons between the slip are interesting.  The changing schedule shows that NASA is still struggling to recover from the tragic loss of the Columbia in many ways. 

The shuttle mission is delayed because new design fuel tanks will not be manufactured in time to support the original schedule.  In 2003, Columbia and her crew were lost when external foam fell off the fuel tank during ascent and struck the wing of the orbiter creating a plate size hole.  Initially, NASA managed the foam issue by modifying existing fuel tanks.  The last of these pre-existing fuel tanks will fly with Discovery when the shuttle launches for a space station assembly mission May 31.  The fuel tanks for future launches are being built with design modification to prevent foam loss.  This manufacturing process is taking four to five weeks longer than originally planned.  No information is available in media reports explaining why the manufacturing schedule is longer than expected.

The mission to the Hubble telescope is also the only shuttle mission planned that will not go to the international space station.  This fact is relevant because it means that two shuttles have to be prepared for launch, not just one.  Two shuttles means double the work needed to get the new fuel tanks ready for launch.  A second shuttle will be prepared in the event a rescue mission is needed. Trips to the space station are less risky because the astronauts could seek shelter in the space station if the orbiter was damaged, providing a much longer window for potential rescue.

The attached PDF file contains an intermediate level root cause analysis of the delay of the Hubble shuttle mission.  It was built using the facts that were available in media reports.  As more details are known, the Cause Map can be expanded.

Sinkhole - Daisetta, TX

On May 7, 2008, a sinkhole formed in Daisetta, Texas, near the Deloach Vacuum Compnay.  The sinkhole quickly grew to approximately 900′ x 600′ x 260′.  Fortunately, no one was injured.  But it did have a severe impact, both on the Deloach company and on the town. 

We can analyze this incident in a Cause Map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page. 

See attached PDF document of the detailed level of the Cause Map.  First, on the left we begin with the impact to the goals.  There were many goals impacted in this case: safety (possibility of injury); environment (a crude oil pipeline leaked and tankers and storage tanks fell into the sinkhole., which is also an impact to the material goal, because goods were lost); customer service (in this case, residents, who were affected by the main power line severed, the main street blocked off, and the potential of anevacuation; and production (Deloach Vacuum company shut down).

It is believed that the sinkhole occurred because of the collapse of a salt dome underneath the town.  The salt dome collapsed because a portion of the salt dome dissolved due to exposure to water.  It is believed that the exposure to water was from one or more of three possible sources.  The first is that it was the natural path of groundwater, due to geological features.  The second is that the water leaked through holes drilled through the surface, either wells or drill holes for salt water disposal or oil & gas production.  The third possibility is that salt water is injected underground dissolved part of the salt dome.  Salt water is injected underground because salt water is disposed in the dome (see above) and salt water waste exists because it is taken from crude oil.

Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.

The investigation is continuing, as Texas officials try to figure out how to prevent further damage to the sinkhole.  For now, the expansion appears to be slowing down, and hopefully soon life can get back to normal in this Texas town.

UPDATE: Heparin Contaminant Identified

Earlier this year, contamination of the U.S. supply of heparin was brought to light.  A significant portion of the U.S. supply of heparin was recalled, and the death toll potentially associated with the contamination has now climbed to 81, with hundreds of adverse events also reported.  Additionally, prior to the recall there was concern for deaths and injuries associated with the contaminated drug not fulfilling its expected purpose - preventing blood clots during surgeries and kidney dialysis - because the contaminant has no blood thinning properties.  So far, the contaminated drug has been found in 10 countries thus far, increasing concern about the drug supply chain.

Researchers have verified that the contaminant in the recalled heparin is oversulfated chondroitin sulfate (OSCS) and that they have discovered a mechanism by which the contaminant can cause the adverse effects (falling blood pressure and severe allergic reactions).  Additionally, the researchers have provided a test for regulators to screen heparin for this contaminant.  

They have determined that the OSCS was present at the active ingredient supplier plant in China.  Because OSCS does not occur in nature and mimics the chemical structure of heparin so closely, it is believed that the (mostly unregulated) crude heparin suppliers in China added OSCS to increase their profit, as OSCS is many times less expensive than heparin.  The OSCS was not detected by standard impurity tests, due to its similarity with heparin.  In Congressional hearings since the event, the Food and Drug Adminstration (FDA) has said that the inspections of the Chinese plant (as well as those of most foreign plants) were inadequate due to lack of funding for the FDA mission.

The attached pdf Cause Map shows that the heparin got into the drug supply after being placed in the raw ingredients.  It was not discovered by regulators, due to the lack of a commonly used, effective test.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  As more information is released about the failings of the supply chain in this instance, we can add more details to the cause map.