Root Cause Analysis

The Danger in Hazardous Chemicals: Arkansas Meat Packing Plant Explosion

Incident Date: March 23, 2008

On Sunday morning, March 23rd, 2008, there was an explosion at the Cargill Meat Solutions plant in Booneville, Arkansas.  Thankfully no injuries have been reported, but 180 people were evacuated due to the ensuing ammonia leak.  Although not much is known about the root causes of the explosion, we can do a very simple analysis.

Read More …

18 Sailors Trapped in Capsized Tugboat

Incident Date: March 23, 2008

On Sunday March 23, a Ukrainian tug boat collided with a Chinese registered cargo ship.  The tug boat capsized and sunk in 115 feet of water, trapping 18 sailors inside the hull.  All 25 passengers on the cargo ship and seven passengers on the tugboat were rescued.  Experts believe the trapped sailors could still be alive if they were able to find air pockets inside the boat.  Unfortunately, no signal or sound coming from within the capsized ship has been detected during the 9 rescue attempts that have occurred so far.  Rescue efforts continue, but are hindered by low visibility and strong currents.

There is very little information currently available on how the collision happened. Even through the details are vague, it can be very useful to apply the root cause analysis method during this stage of an investigation.  Knowing some of the basic causes that have to be present for each type of incident can help direct the investigation efforts.  For example, if a fire occurs you already know that there was a spark, oxygen and fuel present and you can start the investigation by considering each of these causes. 

In the case of the tugboat collision, there are number of causes that had to be present for the collision to occur and they could be used as starting places for the investigation.  Beyond the really basic, like there had to be two ships present, there are a few facts that can be assumed from the beginning.  First, there are strict rules of the road that govern the path of ships, especially near land, similar to the laws that govern vehicle traffic.  Somebody didn’t follow the rules and if you can figure out who didn’t and why that will go a long way to explaining why the ships collided.  Second, every ship should have situational awareness and avoid other ships (even if that other ship is doing something strange) and both ships failed to keep their distance from the other ship.  Either this was a failure to properly monitor position or the methods used were inadequate.   In this specific case, from the damaged that both ships sustained,  I’d also be willing to bet that somebody was going to fast too close to shore. 

Each type of accident has fundamental causes that had to be present for it occur.  While many investigations lead far beyond the causes that can initially be assumed, they can be helpful place to start.  Performing a root cause analysis can help guide an investigation and ensure all the pertinent questions are asked and answered.

Tacoma Narrows Part 2: Failure of a Design

The mechanics behind the failure of the Tacoma Narrows Bridge were discussed in a previous blog entry.  There were many design issues with the bridge and the civil engineering community has done an excellent job of studying and incorporating lessons learned from the failure. But a question that may be more pertinent across all engineering disciplines is, “Why did the design process fail?” 

How did a bridge get built that would fail in a little over four months?  A root causes analysis of the bridge shows that factors that shaped the doomed bridge design are present in almost every engineering project.  There is as much to learn from the failed process that led to the design as there is from the failed design.

The primary factor that led to the bridge design was cost reduction.  The first design proposed for the Tacoma Narrows Bridge was a conventional suspension bridge that was estimated to cost $11 million.  Funding was an issue for the bridge from the beginning, and the design that was finally approved for the bridge was an elegant bridge with a narrow roadbed and short girders.  In additional to being more aesthetically pleasing, the estimated price tag of $8 million dollars was nicer to look at as well. Another contributing factor is the engineer behind the second design, a very well-known civil engineer Leon Moisseiff.  His credentials were impeccable, and he had previously consulted on the famed Golden Gate Bridge, the Bronx-Whitestone Bridge and others.  Additionally, he helped developed some of the methods used throughout the world to calculate forces in suspension bridges.

In a tale that is probably repeating somewhere right now, a cheaper, flashier design was recommended by a well respected engineer.  Nobody wanted to listen to the voices of dissension among the less well-know engineers (and there were engineers who spoke out against the new bridge design saying it was unsafe).  The project then dramatically fails.

As engineers, there is a lot we can learn from studying how past projects have balanced cost and safety.   There are stories where remarkable profits and success have been achieved by finding a cheaper way to do something.  But sometimes, as in the case of the Tacoma Narrows Bridge, the cheap way costs more in the end.
 

Deadly NYC Crane Accident

Incident Date: March 15, 2008

Unfortunately, an investigation into a deadly construction accident is currently underway in New York City.  On Saturday March 15, a 19 story crane collapsed.  Four construction workers were killed and 18 others were injured.  Emergency workers are still sorting through the rubble in an attempt to find any remaining survivors.   The crane was being used at a high-rise construction site and was attached to the side of a skyscraper.  Details as to why the crane fell are still vague, but eye witnesses report that a piece of steel fell and severed at least one tie that held the crane onto the building. Once the connection between the crane and the building was weakened, the crane toppled and split into two pieces.  As it fell, the crane smashed a 4 story townhouse and damaged parts of 3 other buildings.

What made the crane fall?  Part of doing a root cause analysis is sorting the pertinent facts from all the information that is available.  Is it relevant that neighbors had complained that the construction crews were working illegal hours and it seemed like the building was going up too quickly?  City officials had issued 13 violations to the construction project, which at first glance seems like a red flag indicating a lack of attention to safety.  But Mayor Bloomberg has said that this is a normal number of violations for a project this size.  Additionally, the crane had been inspected on the day before the accident and no violations were issued.  Did something change in 24 hours or was the inspection inadequate?  At the time the crane fell, it was being raised to enable work to begin on the next floor of the building.  Did this contribute to the accident?  Where did the piece of steel come from that supposedly fell?  At this point in the investigation there are more questions than answers.

There are many facts and theories that surface in the wake of any accident, and part of doing a root cause analysis is determining which are actually relevant.  This is a process that is much easier said than done.  The push to provide answers quickly can add to the pressure to produce a “cause” for the accident.  But as anyone familiar with the concept of root cause analysis knows, there isn’t a single “cause”, there are many causes that contributed to the accident.  The best approach is to record all possible causes and continue to gather evidence until you can eliminate all the noise and are left with the true causes.  Then the work of creating solutions that address the causes can begin.

A very high level cause map of the crane accident is below:

Tacoma Narrows: Failure of a Bridge

The power of performing a root cause analysis of a problem can be demonstrated by working through well-known engineering disasters.  For example, creating a cause map for the failure of the Tacoma Narrows Bridge helps explain why the bridge collapsed and illustrates some of the lessons that can be learned. 

The original Tacoma Narrows Bridge was opened for traffic on July 1, 1940.  A little more than four months later, the bridge violently failed and a 600 foot span of roadbed fell into the river below.  Why did the bridge tear itself apart?  What made the bridge collapse on November 7th and not some previous day?  One of the first questions asked when performing a root cause analysis is, “What is different about this issue?”   The first difference to consider was that November 7th was a windy fall day.  Construction of the bridge ended in the summer so this was the first fall the new bridge had experienced.  On the day the bridge failed, the wind was blowing across the roadbed at 42 mph.  This was the strongest the wind had blown since the bridge was constructed.  The second difference was the design of the bridge itself.  The Tacoma Narrows Bridge was particularly narrow relative to its length, making the roadbed more flexible than other suspension bridges.  Additionally, the bridge had shallow girders and was relatively weak in torsion compared to other suspension bridges built around the same time.  The combination of fall winds and the slender bridge design resulted in the collapse of the bridge.

As the wind impacted the bridge, the force twisted the roadbed until it hit a point where it was constrained by the suspender cables, and then it twisted back in the other direction.  Other suspension bridges of the time experienced similar twisting motions, but what made this bridge different was that the amplitude of the motion increased with each cycle, rather than dying out.  The bridge was unable to dissipate the wind energy, and the motion of the bridge continued to grow until the twisting motion increased to the point where the suspender cables snapped and the roadbed was dropped into the river below.  The mathematical explanation of why the bridge collapsed is fairly complex, but simply put: the bridge was underdamped causing the twisting oscillations to increase rather than decrease with each twisting cycle.

A high level cause map of the failure of the Tacoma Narrows Bridge is below:

In honor of Joseph Juran

Sadly, Joseph Juran died in New York on February 28, 2008 from natural causes.  He was an astounding 103 years old.  Dr. Juran coined the phrase “Pareto principle” after Vilifredo Pareto, an economist who noted that 20% of Italians held 80% of Italy’s wealth.  Dr. Juran applied this principle to quality, noting that 20% of causes are responsible for 80% of a problem. 
As such, it seems that this would make root cause analysis easier - find 20% of the causes, and we’re 80% done with the problem!  In practice, many root causes analyses just stop there.  However, Dr. Juran himself recognized this problem, and began referring to the causes as “the vital few and the useful many.”  He understood that there is still great value, and perhaps necessity, in determining 100% of causes, not just the “vital few” that are responsible for a disproportionate share of problems. 
Doing a visual root cause analysis, or cause map, can assist us in finding the “useful many”.  The map allows us to find 100% of the causes, not just those that are obvious or most responsible for the problem.  Because the technique effectively draws out these solutions, it ensures that we do not spend 80% of our time finding the most elusive 20% of causes!
Once we’ve found all the causes, we can then assign a solution to each cause.  At this point, your organization will prioritize the solutions using the Pareto principle.  Obviously in a world of limited resources, the solutions that should be applied first are those that can solve 80% of problems.  But it’s important that we ensure that all possible causes and all possible solutions are present in our analysis.   To successfully achieve a goal of “zero defects” or “zero injuries”, we’ll have to apply solutions to all the causes. 

Asleep at the wheel: Accidents caused by exhaustion

What happens when our root-cause-analysis of a problem leads to “operator tired” or “operator fell asleep”?  If we stop there, and blame the operator, we are missing an important opportunity to improve the safety of our organization, and potentially prevent another problem from occurring.

One of the causes of the EXXON VALDEZ oil spill is that the Third Mate who was actually manning the bridge was exhausted due to long work hours and too little sleep.  The collision of two metro trains in Washington D.C. in 2003 was caused by an operator who had worked a double shift, from 8 a.m. to 11 p.m., then returned the next day to do it again.  A few hours into his first shift, his train rolled backwards more than 2,000 feet into another train and caused millions of dollars worth of damage.  The investigation team determined that the brake had never been pressed. There have been some recent studies that show that people suffering from excessive sleep deprivation perform some tasks about as well as someone who is legally drunk.   Naturally, this is a concern for anyone operating heavy machinery or performing surgeries.  Yet rotating shift work, excessive work hours and too little time between shifts continue to occur . . . and sometimes have tragic consequences.

Based on these concerns, some organizations are trying to alleviate fatigue problems caused by their work standards.  For example, the Accreditation Council for Graduate Medical Education’s common duty hour standards took effect on July 1, 2003.  These standards reduce the number of hours medical residents can work in a week, and require adequate rest between duty periods.

When we end up with a cause on our root-cause-analysis of “operator tired” or “operator fell asleep”, it is essential that we continue to ask “Why?” to determine the factors that led to exhaustion.  Many times, regulations to ensure adequate rest before duty do not exist.  In some cases, company policy encourages or requires workload that does not allow for adequate sleep.  If we do not continue our root-cause-analysis to determine the reason that the operator is tired, we run the risk of having the same problem - or worse - happen again.

Another look at the Exxon Valdez oil spill

The Supreme Court has begun hearing Exxon Mobil Corp.’s appeal of punitive damages stemming from the 1989 oil spill caused by the grounding of the EXXON VALDEZ.   The punitive damages currently stand at $2.5 billion, reduced by appeals from the original punitive damages of $5 billion.  People across the country are eagerly awaiting the Supreme Court’s decision, which is expected in the summer, for many reasons.  While this case has several interesting legal ramifications, the one I’d like to focus on is Exxon Mobil Corp.’s liability, and how a root cause analysis can help us foresee the different areas where liability might be at issue.  When examining the causes of the EXXON VALDEZ spill, a very basic root cause analysis follows: 

What a lot of people imagine when they hear about the EXXON VALDEZ, and what the cause map above implies, is a drunken Captain haphazardly steering a gigantic oil tanker into a reef.  As is usually the case, the real issue is far more complicated.  Many people don’t realize that the Captain was not even present on the bridge when the ship struck the reef.  So his drinking did not directly cause the accident.  Yet the fact that he was drunk, which was against company regulations, is one of the main reasons that Exxon Mobil Corp. is being found liable for punitive damages.  The argument is that they knew the Captain had an alcohol problem and still allowed him to pilot the ship.  We’d have to expand upon the cause map above significantly before there was any mention of Exxon Mobil Corp. and their contribution (direct and indirect) to the spill.  A visual root cause analysis expanding upon the last two boxes is shown below.

This is why a visual form of the root cause analysis is so helpful.  Indirect causes are the easiest ones to miss, and they are frequently the biggest liability issues.  A detailed cause map allows us to see the indirect causes that can lead to liability issues - such Exxon Mobil Corp.’s inaction towards the Captain’s violation of company rules, which is part of the basis of Exxon Mobil Corp.’s liability in the ongoing litigation.  It’s only when we get down to the nitty-gritty of the root cause analysis that we can see the contributions from all the major players.  It will be interesting to see what kind of a price tag the Supreme Court will place on those contributions.

Problem Solvers are Specific

Have you ever heard anyone say “the procedure is a piece of junk?”  If you ask the person if every step of the 40-step procedure is wrong they will usually say “No, not every step.”  You can ask them to show you which step is wrong.  When they point out that step 14 is wrong, you can ask, “Is every word in step 14 wrong?”  They will usually say “Well, no, not every word, but that 5 is supposed to be a 7.  You can then say “I understand.  That is an issue.  Thanks for catching that.  I’ll get it updated.  These things have got to be clear and accurate.”

The original statement “the procedure is a piece of junk” is too general.  It refers to the procedure as one thing, not 40 things.  People that blame and complain speak in very general terms.  They group things together and generalize.  People that are very good at troubleshooting and solving problems naturally think and speak in very specific terms.  Analyzing a problem is about breaking a problem down into parts.  Analyzing problems is always about getting more specific so that very specific actions (the solutions) can be taken.

Terms like “human error”, “procedure not followed” and “training less than adequate” are used regularly by companies to explain why a particular problem occurred.  These terms are too general.  They inadvertently give the impression that the cause has been found during their analysis (root cause analysis).  Knowing that someone didn’t follow a procedure is important, but is not the end of an investigation.  We’re just getting to the good stuff.  We’re just getting the specific information that created the incident in the first place.

Our interest is not limited to fixing that person that didn’t follow that procedure.  We want to address how we developed, approved, utilized and updated this particular procedure so that the procedure process can be improved.  It’s about improving how we capture and communicate the best work practices in our organization as a whole.  This is the leverage within the organization.  To solve problems effectively be specific.  Ask those who blame and complain to help us understand the issue by being more specific.

For more information about improving the problem solving skills within your organization, visit ThinkReliability - specializing in Cause Mapping - Effective Root Cause Analysis training.

Dust Explosions: How do we prevent them from happening?

The Chemical Safety Board Investigations Manager, Stephen Selk, P.E. gave a briefing on February 17, 2008 to update to the public on the Imperial Sugar Company explosion and to provide a root cause analysis on dust explosions.  The speech was very enlightening.  One of the things he said was “The Board identified 281 [dust] fires and explosions over a 25-year period that took 119 lives and caused 718 injuries.”  So, obviously this is a concern.  But what to do about it?

When he presented the root cause analysis for dust explosions, he stated that five things were necessary for an explosion: presence of a combustible dust, presence of oxygen, dispersion of the dust into the air, confinement of the particles, and ignition energy.  For each of these requirements in the root cause analysis, there is a possible solution - but that possible solution may or may not be effective.

First, a dust explosion requires the presence of a combustible dust.  Unfortunately, the combustible dust is usually a by-product (or the actual product) of the process being performed.  The Imperial Sugar Company explosion was caused by sugar dust.  The Imperial Sugar Company refines sugar.  Sugar dust will always be present at a sugar refinery.  So, attempting to remove the combustible gas is probably not worthwhile.

What about the presence of oxygen?  Obviously, there has to be oxygen in the refinery itself for the workers to be able to breathe, but it may be possible to remove the oxygen within some of the equipment, possibly by the use of inerting equipment.  Inerting equipment using nitrogen to reduce the percentage of oxygen to below combustible levels has been used with some success in various industries and was recommended as a solution to the fuel tank explosion on TWA Flight 800.  The use of inerting equipment within processing equipment would help reduce explosions that are begun within the equipment.

I’d like to examine dispersion of the dust and confinement together.  These two requirements almost seem to be mutually exclusive.  After all, if the particles are dispersed, they aren’t being confined.  Likewise, if the particles are confined, how are they dispersed?  In reality, the particles are always confined, even if it is only by the building surrounding the processing plant.  It’s not clear how much confinement, or how much dispersion, is required for an explosion to occur.  It’s also likely dependent on the particular organic material that has become combustible dust.  So, specific solutions here would need to be tailored for each individual plant - and there may not be any that truly work.  However, there is one solution that may prevent, or lessen the effects of, follow-on explosions.  That is a blast-proof building.  In the type of blast-proof building I’m thinking of, there is a weaker section of the building that acts almost like a pressure-relief valve.  It blows out before the rest of the building and directs the explosion through a particular path.  (If the processing equipment involves hazardous materials, it could even direct it to another confined area to prevent environmental contamination.)  This solution, too, would need to be designed specifically for the task at hand, and may be prohibitively expensive.

Last, let’s focus on ignition energy.  Eliminating all sources of ignition energy in a plant seems like it would be possible, albeit complicated and possibly prohibitively expensive.  However, with greater thought, eliminating all ignition sources, including static electricity, in a plant filled with electronic equipment and wiring seems like a monumental task indeed.  However, this is where the focus on preventing explosions frequently lies.  This requires constant and careful inspection of all wiring and potential power sources.  Another consideration is that fire is a potential ignition source.  Obviously a goal of any organization is always to avoid fires in the processing equipment, but if preventing ignition is to be the primary way to avoid dust explosions, improved fire extinguishing systems may be required.  If dust explosion is a risk, automatic fire extinguishing systems should be considered.

Preventing dust explosions is a daunting task.  But we see as we examine the statistics - 281 fires and explosions over 25 years, many of them destroying lives and buildings - it is something that must be done.  Once we have completed the root cause analysis, we can look for solutions and then set about implementing them.