Root Cause Analysis

Slips, Trips and Falls - A Root Cause Analysis Primer

Slips, trips and falls happen every day.  Falls are responsible for tens of thousands of deaths each year.  (Slips and trips are considered a subset of falls, and are included in these numbers.)  Falls on the job account for 12-15% of all worker’s comp costs.  The direct and indirect costs of workers injured and killed on the job are estimated to be billions of dollars each year, both in worker’s comp claims and in lost productivity.  In 1999, as an example, 5,100 workers were killed by falls and over 570,000 injuries were reported.  However, there are many things that can be done to prevent and lessen the impact of falls.  Creating a Cause Map - a visual root cause analysis - will allow us to identify all the potential causes of falls.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  Once we’ve done that, we can identify the best solutions.

A worker is injured during a fall because the worker strikes the floor, or another object, and the object contacted is hard, and the worker hits in a way that causes injury.  When I say that workers are injured because they hit an object in a way that causes injury, what I am really talking about is factors that worsen a fall, and make injury more likely. The worker could land on a part of his or her body that is more easily injured.  Another way that injuries can be worsened is if a worker falls farther than his or her height (i.e., not a same-level fall).

The worker strikes the floor or other object because he or she falls, and there is no other support for the body, such as a handrail, or a harness.   There are four different ways to fall: slips, trips, the “step and fall” - where a person gets off-balance while stepping - and becoming unbalanced on moving equipment.

A worker slips when there is inadequate traction, either because the force of stepping off is too high, or the coefficient of friction is too low.  The force of stepping off can be higher than average if the worker is walking quickly or running, making a sudden change in direction, or if he or she has an awkward gait, from injury or old age, for example.  The coefficient of friction is a function of the traction provided by the shoes the worker is wearing and the “slipperiness” of the walking surface.  The coefficient of friction is too low if the traction of the worker’s shoes is inadequate and if the floor is slippery, because the surface is wet, icy and/or oily and does not have a non-skid coating.  Of course, for this to be an issue at all, the worker has to step into the slippery area. 

A worker can become off-balance by encountering an unexpected height difference (known as the “step and fall”).  This occurs in one of two ways.  Either the front foot lands on a surface lower than expected, or the ankle turns due to one side of the foot ending up higher than the other side, with footwear that inadequately supports the ankle.  These are both due to an unexpected height difference.

When a worker trips, it is because his or her toe is stopped, but his or her upper body is not stopped.  The upper body is moving because the worker is moving and he toe is topped because it encounters an object in the walking path, a rise in the walking path, or a difference in height of subsequent stairs. 

Last but not least, falls can be caused by workers who become unbalanced on moving equipment.  For this to occur, the worker must be inadequately secured to the equipment while the equipment changes motion, either by turning, decelerating or stopping, or accelerating or starting to move.

Once we have finished our Cause Map and found all the potential causes, we can assign potential solutions to all appropriate causes.  The solutions that I have come up with are in green boxes, near the cause(s) they “solve”.   You can see that some of the solutions are the responsibility of the company, and some are the responsibility of the worker, and some are both.   Although many of the responsibilties lie with the worker, it is in a company’s best interest to provide training on how to prevent, manage and mitigate falls.  Falls may seem like everyday, ordinary minor occurences.  While falls are everyday occurences, the consequences can be anything but minor.

The attached PDF document shows the visual root cause analysis as a Cause Map.

SL-1 Explosion - January 3, 1961

The only fatal reactor accident in the United States occurred on January 3, 1961, when an Army prototype known as SL-1 (for stationary, low power reactor, unit 1) exploded, killing the 3 operators who were present.  We’ll use the SL-1 tragedy as an example of how the Cause Mapping process can be applied to a specific incident.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.

The SL-1 tragedy killed the three operators present, which is an impact to the safety goal.  Another goal is that there be no damage to the vessel. In the case of SL-1, the  vessel sustained extensive damage.

The loss of life and vessel damage were both caused by the reactor exploding.  The reactor exploded because it went prompt critical (an uncontrollable, exponentially increasing fission reaction).  The reactor went prompt critical because withdrawal of the central rod can cause prompt criticality and because the rod was rapidly, manually lifted 26.4″ out of the core.

Withdrawal of the central rod can cause prompt criticality due to a lack of shutdown margin in the core, and inadequate safety criteria.

Because most of the evidence was so effectively destroyed, nobody really knows why the control rod was lifted out of the core.  There are two theories (disregarding the bizarre and improbable murder/suicide theory): 1) the control rod got stuck while being lifted to be attached to the drive mechanism, and, as the operator was exerting greater force on it, suddenly came free, resulting in a lift far greater than intended, or that an rod drop testing/exercising was performed improperly.

The control rod was stuck, and came free while being attached because it was required to be lifted 4″ out of the core and because control rods had been sticking.  The control rods had been sticking for one or more of the following reasons: 1) reduced clearances due to radiation damage (which can cause structural material to swell), 2) the passage was blocked due to loss of poison strips in the channel, caused by poor design and inadequate testing, or 3) lifting equipment not working properly due to inadequate lifting capacity of the lifting equipment.

Exercising/testing was potentially improperly performed.  This could have occurred because the operators chose to exercise/test the rods, attempting to ensure that they would perform properly, and because they didn’t realize what would happen. This is because of inadequate training and inadequate work instructions.  The testing was also potentially done improperly due to inadequate work instructions.

On a positive note, the SL-1 incident did initiate some positive changes in the nuclear industry.  Most notably, reactor design has improved and incorporated a “one-rod stuck” criteria which specifies that a reactor can NOT go critical by the removal of any one control rod.  Additionally, procedures and training have gotten more intense and more formal, and planning for emergencies has increased.

The attached pdf gives a visual representation of the intermediate level root cause analysis, the Cause Map.  It can be printed out to fit on one page.

Train Derailment - Lafayette, Louisiana

About 1:40 am on May 17, six rail cars derailed and overturned near Lafayette, Louisiana.  One of the cars was damaged and leaked about 11,000 gallons of hydrochloric acid.  Authorities evaluated people with 1 mile of the accident.  Approximately 3,000 people were affected, including a few small businesses and a nursing home.  Five people, including two rail workers, were sent to a hospital and treated for eye and skin irritation.  All affect people are being reimbursed for food and hotel costs by the railway company that operated the train, BNSF Railway.

 The was potential for farther release of chemicals because one of other rail cars involved in the accident carried ethylene oxide, a flammable and dangerous chemical, and two of the remaining cars also carried hydrochloric acid. 

The Louisiana State Police’s hazardous materials unit is overseeing clean-up of the accident sit.  The spill is being neutralized with lime and the contaminated material will be removed and disposed of.  The rail car containing ethylene oxide was removed from the site as quickly to remove the potential for additional problems.

The cause of the derailment is not known at this time.  The Federal Railroad Administration will conduct an investigation of the accident.

The attached PDF file contains an intermediate level root cause analysis of the train derailment.  It was built using the facts that were available in media reports on the accident.  As more details are known, the Cause Map can be expanded.

Prudhoe Bay Pipeline Corrosion

In 2006, a British Petroleum (BP) worker in Prudhoe Bay, Alaska discovered a leak in its transit, or feeder pipelines (which deliver the crude oil drilled by BP to the main Trans-Alaskan Pipeline, which transports the oil to Valdez, in the southern part of Alaska.  The oil is taken from there in ships to the lower 48.)  Approximately 5,000 barrels (more than 200,000 gallons) of oil were spilled, adversely affecting almost 2 acres of permafrost (continually frozen soil).  During inspections performed as a result of the spill, severe corrosion (and another, smaller spill) was discovered in 16 miles of pipeline.  BP decided to replace all 16 miles of affected pipeline, at a cost of $260 million.  Additionally, BP paid $20 million in fines, restitution to the State of Alaska, and a payment for environmental research.  Some people believe this is the largest fine ever paid in the state for what was legally considered an “environmental misdemeanor.”

A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page (see attached pdf).  First we will look at the impact to the goals.  For the BP pipeline the environmental goal was impacted because 5,000 barrels of crude oil spilled, which affect on 1.9 acres of permafrost.  The impact on the two environmental goals was caused by the leak of crude oil.   The customer service goal was impact due to an increase in oil prices to consumers.  This occurred because  of the loss of barrels during the production shutdown, which is an impact to our production goal.  The loss of oil is due to the shutdown, which occurred in order to replace the affected lines, which is also an impact to the material goals.  The lines had to be replaced because a loss of pipe integrity was discovered, which led to the fine and restitution, which is also an impact to the material goal.

The loss of pipe integrity was due to severe corrosion product buildup.  The corrosion product buildup also resulted in a hole in the pipeline, which caused the leak.

The permafrost that was affected by the oil because of the leak, but also because the leak was not contained promptly. The next question is “Why was the leak not contained promptly?”  (And also, “Why did the leak occur?” which was due to a hole, but we’ll get to that later.)  The leak was not contained promptly because the the leaked oil was not visible, the location of the leak was inaccessible, and the leak detection program was ineffective. 

The severe corrosion product buildup resulted from three things.  First, there was corrosion in the pipe.  Second, the corrosion went undetected (we’ll go into both of these in more detail).  And third, the pipes were used beyond their design life (25 years vs. the 29 years they had been in service.

There was corrosion in the pipes because there were microbes in the pipe protected by a layer of sediment, and microbes produce corrosive substances (this is known as internal microbiological corrosion).  This layer of sediment was due to an ineffective maintenance program.  It settled to the bottom of the pipe because there were low spots in the pipe, and because the velocity of the oil was too low to remove sediment because the pipe diameter was too large.

The corrosion went undetected because of an ineffective inspection program.  The inspection progrm was ineffective because there was not a regular internal inspection schedule.  The ultrasonic testing used was not effective because it did not cover 100% of the line and the damage was very localized (thus the ultrasonic testing was missing the spots with the worst corrosion).  Additionally, a “smart pig”, which is used internally to measure the wall thickness of a line, was never run through the line, because BP did not believe it was necessary as they performed ultrasonic testing.

Once the Cause Map is build to a sufficient level of detail with supporting evidence the solutions step can be started. The Cause Map is used to identify all the possible solutions for given issue so that the best solutions can be selected. On the Cause Map you can see some solutions derived from the causes (in the green boxes).  Looking through news reports or BP press releases regarding their Prudhoe Bay pipeline, you’ll see that almost all of the actions listed have been or are being taken to prevent this problem from happening again.

Pet Food Contamination - March 2007

On March 15, 2007, the Food and Drug Adminstration (FDA) was notified that ten animals had died from eating pet food.   This began an investigation into a problem that would result in the recall of 150 brands of pet food, and would kill many animals - some veterinarians suggest up to 1,000.  We can illustrate what happened in a Cause Map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page. 

First, we examine the impacts to the goals.  For a food manufacturer, one of the overall goals is to have zero injuries.  Some veterinarians suggest that up to 1,000 dogs and cats were killed in the U.S.  One of the other goals impacted is the customer service goal. In the case of the contaminated pet food, 150 brands (with 60 million containers of Menu Foods pet food, the most affected brand) were recalled.  This was the largest recall in FDA history, and was estimated to cost Menu Foods $54 million.

The loss of pets was caused by renal failure.  The renal failure in dogs and cats occurred because the dogs and cats ate contaminated pet food.  The dogs and cats ate contaminated pet food because it was in the food supply.  This also led to the recall.

Why was the contaminated pet food in the food supply?  The food was contaminated with up to 6% melamine and cyanuric acid (CA), and the contaminants were not detected.  The melamine and cyanuric acid (CA) were found in the food because they were added to the raw ingredients to increase the apparent content of the wheat gluten.  This reduced the cost for the manufacturer because melamine and cyanuric acid are cheaper than wheat gluten.  It increased the apparent protein content because melamine and cyanuric acid mimic the protein response in protein testing. 

The contaminants were not detected because standard tests did not detect them, and because of inadequate insepctions and inaccurate paperwork.  Standard tests did not detect the contaminants because melamine and cyanuric acid mimic protein response in protein testing, and because they were not tested for.  Inspections were inadequate because the material did not receive export inspections in China, because the exports were improperly labeled as non-food, and only food items are subject to mandatory inspection.  The inspections were also inadequate because FDA officials do not have ready access to Chinese plants because there is no binding agreement between China and the FDA.  The paperwork was inaccurate because the broker certified that the material specification was met, and the material specification forbid foreign material.

Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.  See the attached pdf for a visual representation of the cause and effect relationships.

Mission to Hubble Telescope Delay

Early in March 2008, NASA announced that the shuttle mission to the Hubble telescope would take place in the fall rather than in August as originally scheduled.  A trip to Hubble is necessary to replace gyroscopes and batteries that failing.  Additionally, the mission will also be sued to install instruments that will increase the range of the telescope.   The changing schedule itself is not a cause for alarm, but the reasons between the slip are interesting.  The changing schedule shows that NASA is still struggling to recover from the tragic loss of the Columbia in many ways. 

The shuttle mission is delayed because new design fuel tanks will not be manufactured in time to support the original schedule.  In 2003, Columbia and her crew were lost when external foam fell off the fuel tank during ascent and struck the wing of the orbiter creating a plate size hole.  Initially, NASA managed the foam issue by modifying existing fuel tanks.  The last of these pre-existing fuel tanks will fly with Discovery when the shuttle launches for a space station assembly mission May 31.  The fuel tanks for future launches are being built with design modification to prevent foam loss.  This manufacturing process is taking four to five weeks longer than originally planned.  No information is available in media reports explaining why the manufacturing schedule is longer than expected.

The mission to the Hubble telescope is also the only shuttle mission planned that will not go to the international space station.  This fact is relevant because it means that two shuttles have to be prepared for launch, not just one.  Two shuttles means double the work needed to get the new fuel tanks ready for launch.  A second shuttle will be prepared in the event a rescue mission is needed. Trips to the space station are less risky because the astronauts could seek shelter in the space station if the orbiter was damaged, providing a much longer window for potential rescue.

The attached PDF file contains an intermediate level root cause analysis of the delay of the Hubble shuttle mission.  It was built using the facts that were available in media reports.  As more details are known, the Cause Map can be expanded.

Sinkhole - Daisetta, TX

On May 7, 2008, a sinkhole formed in Daisetta, Texas, near the Deloach Vacuum Compnay.  The sinkhole quickly grew to approximately 900′ x 600′ x 260′.  Fortunately, no one was injured.  But it did have a severe impact, both on the Deloach company and on the town. 

We can analyze this incident in a Cause Map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page. 

See attached PDF document of the detailed level of the Cause Map.  First, on the left we begin with the impact to the goals.  There were many goals impacted in this case: safety (possibility of injury); environment (a crude oil pipeline leaked and tankers and storage tanks fell into the sinkhole., which is also an impact to the material goal, because goods were lost); customer service (in this case, residents, who were affected by the main power line severed, the main street blocked off, and the potential of anevacuation; and production (Deloach Vacuum company shut down).

It is believed that the sinkhole occurred because of the collapse of a salt dome underneath the town.  The salt dome collapsed because a portion of the salt dome dissolved due to exposure to water.  It is believed that the exposure to water was from one or more of three possible sources.  The first is that it was the natural path of groundwater, due to geological features.  The second is that the water leaked through holes drilled through the surface, either wells or drill holes for salt water disposal or oil & gas production.  The third possibility is that salt water is injected underground dissolved part of the salt dome.  Salt water is injected underground because salt water is disposed in the dome (see above) and salt water waste exists because it is taken from crude oil.

Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.

The investigation is continuing, as Texas officials try to figure out how to prevent further damage to the sinkhole.  For now, the expansion appears to be slowing down, and hopefully soon life can get back to normal in this Texas town.

UPDATE: Heparin Contaminant Identified

Earlier this year, contamination of the U.S. supply of heparin was brought to light.  A significant portion of the U.S. supply of heparin was recalled, and the death toll potentially associated with the contamination has now climbed to 81, with hundreds of adverse events also reported.  Additionally, prior to the recall there was concern for deaths and injuries associated with the contaminated drug not fulfilling its expected purpose - preventing blood clots during surgeries and kidney dialysis - because the contaminant has no blood thinning properties.  So far, the contaminated drug has been found in 10 countries thus far, increasing concern about the drug supply chain.

Researchers have verified that the contaminant in the recalled heparin is oversulfated chondroitin sulfate (OSCS) and that they have discovered a mechanism by which the contaminant can cause the adverse effects (falling blood pressure and severe allergic reactions).  Additionally, the researchers have provided a test for regulators to screen heparin for this contaminant.  

They have determined that the OSCS was present at the active ingredient supplier plant in China.  Because OSCS does not occur in nature and mimics the chemical structure of heparin so closely, it is believed that the (mostly unregulated) crude heparin suppliers in China added OSCS to increase their profit, as OSCS is many times less expensive than heparin.  The OSCS was not detected by standard impurity tests, due to its similarity with heparin.  In Congressional hearings since the event, the Food and Drug Adminstration (FDA) has said that the inspections of the Chinese plant (as well as those of most foreign plants) were inadequate due to lack of funding for the FDA mission.

The attached pdf Cause Map shows that the heparin got into the drug supply after being placed in the raw ingredients.  It was not discovered by regulators, due to the lack of a commonly used, effective test.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  As more information is released about the failings of the supply chain in this instance, we can add more details to the cause map.

Gas Pump Glitch

An Associated Press article, published on April 25, highlighted a common, often ignored problem of customers getting a different amount of gas then what they paid for.  Gas pumps contain a check valve that allows gas to start flowing at the same time the price meter starts.  As the check valves age, they can begin to hesitate and wait a period of time before gas flow begins.  This results in the consumer being overcharged because the price meter is turning before gas is flowing.    Worn check valves usually only cost consumers pennies per fill-up, but there have been instances of overcharges of 30 to 40 cents a gallon.  This issue doesn’t cost the consumer large amounts of money, but it adds frustration to a public already aggravated by record high gas prices.

To be fair, it should be mentioned that worn check valves sometimes help the consumer as well.  When a check valve hesitates at the end of a fill up, the price meter is stopped and a small amount of gas will continue to flow.  Also, to clarify, this isn’t a case of gas stations purposely gorging consumers.  It’s a situation where a common piece of machinery is wearing out and not functionally properly. 

To help prevent these types of errors, gas pumps are regularly inspected to ensure that consumers are charged for the correct amount of gas.  Regulations allow gas pumps to pass inspection if they overcharge by no more than 6 cents for every five gallons delivered.  Most states require gas pumps to be inspected every year to ensure accurate measurement of gas delivered.  Many counties try to inspect more frequently, but have difficultly because of staffing shortages and financial pressure.  

The attached PDF file contains an intermediate level root cause analysis of the worn check valves in gas pumps.  It was built using the facts that were available in media reports.  As more details are known, the Cause Map can be expanded.

Hindenburg Crash - Competing Theories

On May 6th, 1937, the Hindenburg burst into flames over the Lakehurst, NJ Naval Base, after completing a successful trip across the Atlantic.  35 of the 97 passengers (and one of the ground crew) were killed.  The Hindenburg itself was a total loss, and the popularity of airships never recovered after the accident. 

The loss of 36 lives and the loss of the Hindenburg were both caused by the fire aboard. The loss of popularity of airships was caused by both the loss of the Hindenburg, and by the loss of lives.  The next question to ask is “Why did the fire occur?”

For the Hindenburg, this is where things start to get interesting.  There are three separate theories about why the fire started.  There are people who believe very strongly in each.   Luckily for us, the beauty of the Cause Map form of a root cause analysis is that we can use it even if we haven’t determined which theory is correct.

The first theory is that the fire started from sabotage.  Because the Hindenburg was frequently used as a Nazi propaganda tool, some thought it was almost too easy of a target for sabotage from anti-Nazi activists (who included in their number the designer of Hindenburg, Dr. Hugo Eckener.)  There was even a “suspicious” character who survived the crash, a German acrobat living in America.  However, eventually the FBI dismissed the idea of sabotage as a “red herring.” 

Another theory is that the fire began when static electricity ignited the flammable cover of the airship.  The major proponent of this theory, Dr. Addison Bain, has run tests on pieces of the Hindenburg cover preserved from the wreck site.  (This was not until 1994.)  He has also found supporting evidence from historic records of the Zeppelin company.

The other theory is that static electricity ignited a flammable hydrogen-oxygen mixture.  This was the original cause attributed to the disaster by the U.S. Department of Commerce’s root cause analysis investigation after the crash.  There are also people who claim that Dr. Bain’s theory is physically impossible, and do not specifically champion a cause, but treat this one as the most likely.

Note that we’re not espousing a theory - we are just recording all of the possibilities.  Once we have done that, the cause map allows us to find solutions for any potential causes.  Once we have all the theories mapped out, we can use the cause map as a resource to determine the solutions that are most helpful, or continue our root cause analysis investigation to determine which causes are most likely.

The attached pdf document gives an intermediate level Cause Map of the incident.