Root Cause Analysis - Incident Investigation

Salmonella Recall

1 Comment

By Kim Smiley

A number of food products have been recalled recently because of potential salmonella contamination.  The recall list is still growing and has the potential to affect thousands of items in nearly every aisle at the grocery store.

The contamination originated in hydrolyzed vegetable protein (HPV) which is a common, inexpensive salty and savory flavor enhancer used in a variety of products.  All HPV from Basic Food Flavors of Las Vegas made since September 17, 2009 has been recalled.   For a list of all recalled items and more information, please visit the Food and Drug Administration webpage.

The salmonella contamination occurred in the processing equipment at a one location, but HPV from that supplier was sold to food manufacturers nationwide.  HPV is a specialized product and there are only a few suppliers for it so issues at a single supplier have the potential to affect a significant percentage of the processed food supply.

The contamination was identified when a consumer of the Basic Food Flavors identified salmonella in a batch of HPV they had purchased and reported it to the FDA, utilizing the new FDA Reportable Food Registry.  The FDA then inspected Basic Food Flavors and found salmonella in the plant’s processing equipment.

The overall risk to the public is considered low.  No cases of illness from this contamination have been reported.  As long as products are heated to a sufficient temperature, either during the manufacturing process or cooked after purchase, the salmonella risk will be eliminated.  The highest risk products are ready to eat products such as chips, dips, and dip powder.

The investigation of this incident is still ongoing, but a basic root cause analysis can be started.  The safety goal is obviously impacted since salmonella can potentially cause illness and even death in the case of weakened immune systems.  In this case, the customer service goal would be impacted as well because the recall may affect customer confidence and sales of the recalled items.

Click on the “Download PDF” button to view an initial Cause Map of the salmonella contamination.  The Cause Map can be expanded as more details are available.

Root Cause Analysis - Incident Investigation

Water Pollution from Sewer

By ThinkReliability Staff

Thanks in part to the Clean Water Act, passed in 1972 and revised in 2000, most residents of the United States have continual access to clean, safe water.  However, extenuating circumstances may result in pathogens remaining in drinking water – or contaminating swimming water – resulting in potential illnesses.  In fact, researchers estimate that up to 20 million people per year become ill due to ingesting pathogens in water.  In addition to the environmental impact of untreated sewage reaching waterways, up to 400,000 basements and thousands of roads have been flooded with untreated sewage.

These floods generally occur when the sewer systems are overwhelmed or clogged.  A clogged sewage system can result from buildup of leaves, or other debris, including that from illegal dumping.

An overwhelmed sewer system is generally the result of a high volume of water passing through the system.  As the population increases, the strain on the system increases as well.  Since most municipalities do not have the funds available to upgrade or replace their systems, an aging, inappropriately sized system is all that remains to provide needed water.  However, systems are generally able to keep up with demand, except during times of high rainfall.  Many sewer systems handle both waste and rainwater through the same system.  When a heavy rainfall occurs, the system is overwhelmed, resulting in overflow of sewage.  This overflow is often directed into the waterways.  Dumping untreated or partially treated sewage into waterways is illegal, but fines are hardly ever levied.  The Federal Government may be unwilling to levy fines against municipalities for illegal dumping, especially because Federal funding to maintain sewer systems has decreased significantly.  With municipal budgets stretched already, dealing with aging sewer systems just isn’t happening.

However, there are some things that municipalities can do.  Green spaces (as opposed to paved areas) absorb rainfall, decreasing the amount directed in to the sewer system.  By planning more green space, or better drainage, the amount of rainfall that actually enters the system can be reduced.  Additionally, municipalities could redirect rainfall to keep it out of the waste portion of the sewer system. The cost of doing this may make it infeasible; however, calls for Federal stimulus money for repairs to sewer systems may result in municipalities’ ability to finally upgrade their systems.

Root Cause Analysis - Incident Investigation

Death of Luger at 2010 Winter Olympics

1 Comment

By Kim Smiley

On February 12, 2010, Nodar Kumaritashvili, an Olympic luger from the country of Georgia, was killed during a practice run.  He lost control of his sled, flew off the track and hit a steel pole.

The investigation into the accident is still ongoing, but a root cause analysis can be started with the information that is available.  This accident obviously impacts the safety goal because an athletic was killed and it also had potential to impact the schedule goal because the track was closed during the initial investigation.

There are a number of causes that can be added to the Cause Map.  One of the more obvious causes for the accident is that the athletic was traveling at high speeds.  This occurred because the crash happened near the bottom of the track so the sled was near its top speed.  Additionally, the Vancouver Olympic track is also a particularly fast track.  Top speeds on the track were predicted to be 96 mph, nearly 6 miles faster than the standing 2000 world speed record.

How did the track get designed to be so much faster than typical tracks?  There are a number of causes that contributed to fast design.  The designers choose Whistler as the site of the track because Whistler has a colder climate than the alternatives, resulting in firm, fast ice and because there is high tourist traffic there that would help make the track a commercial success after the Olympics.  Whistler was also the site of the Olympic alpine events.

The land that was available at Whistler was long and narrow.  The site was a valley approximately 100 yards by 800 yards.  By comparison, the Calgary track was about 300 yards wide and Salt Lake City’s track was 500 yards.  Designing a track to fit in the available region meant the track couldn’t include any long curves that slow down speed as is typical.

The result was the fastest track in the history of the sport.

As the investigation continues, more details become available and they can be added to the Cause Map.

In order to ensure safety during the Olympic Games, several solutions were implemented following the accident. A wooden wall was added to the curve where the accident occurred to keep athletics on the track, the steel poles were padded and events were started lower on the track to limit the maximum speed.  The lower start was predicted to slow top speeds in the men’s events by about 5 mph.

There have been several crashes on the course since the accident, but thankfully no farther significant injuries have occurred.

Root Cause Analysis - Incident Investigation

Metro Train Derailment Washington D.C.

By ThinkReliability Staff

On February 12, 2010 at approximately 10:13 A.M., a six-car Red Line Metro train taking passengers to Shady Grove derailed near the Farragut North station in Washington, D.C.  If you’ve been reading our blog, you’ve seen our reports on three previous Metro incidents in the past year (two Metro workers were killed in January, two trains collided last November, and two trains also collided last June).

Thankfully, this derailment caused only minor injuries.  However, it did result in an extremely messy commute for a lot of people, due to a severe delay in train service.  Additionally, there was likely damage to the train and/or track, which will require labor to repair.  More labor will be required for the investigation.

All the basic information, as well as the impacts to the goals (the injuries, delay in service, property damage and labor required as a result of the incident), relating to this event are captured in a problem outline.  We can also capture anything that was different at the time.  Here we note that there were major storms in the area and that the commute was especially heavy.

Once we have completed the outline, we can begin the Cause Map with the goals that were impacted.  The impacts to the goals resulted from the train derailing.  The train derailed when the front wheels slipped and the lead car came off the track.  Metro and National Transportation Safety Board (NTSB) investigators are determining the causes of the derailment, but some of the things that will be looked at as causes include: the train was moving onto a pocket track.  Other trains previously have slipped off the track while moving onto a pocket track (a side track that allows trains to pass other trains or move around construction).  It’s unclear whether the train was moving onto the pocket track to move around other trains or track work.

As previously mentioned, the snow and icy conditions (which have been extreme as of late in D.C.) may have caused the tracks to be slippery, which potentially contributed to the derailment.  It’s possible there was damage to the tracks or switch, as the area where the derailment took place is the oldest portion of the Red Line, and is due for maintenance.  Because of an extreme budget shortfall on the Metro line, repairs to tracks and cars have been delayed.  Last but not least, there’s a possibility that the weight of the rail car may have been a factor in the derailment.  The cars were extremely crowded because of an insufficient number for the commute.  Metro was not running the normal number of cars because it had not completely recovered from the storm, but there were the normal number of commuters because the Federal Government was open.  (The Federal Government usually remains closed when the Metro system is unable to run at full capacity.)

Even though we are not yet certain which factors may have contributed to the derailment, we can include them all on the Cause Map until we are able to rule some of them out.  Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.  View the beginning stage of the root cause analysis investigation by clicking on “Download PDF” above.

Root Cause Analysis - Incident Investigation

Possible Toyota Prius Recall

By Kim Smiley

A new potential safety issue has developed and Toyota may recall the newest model of the gas electric hybrid Prius that has been sold since last May.  The National Highway Traffic Safety Administration has received 124 reports from consumers claiming that the brakes don’t engage immediately at times.  Toyota has stated that the company has received 180 reports of braking problems in Japan and the United States. The reports include 4 incidents that resulted in accidents with 2 people receiving minor injuries.

Even a slight delay in the response of car braking systems can be very dangerous because cars can travel nearly 100 feet in one second at highway speeds.

No official details are known yet on what is causing the delay in brake engagement.  In one article, a power train expert speculated that it was a software glitch caused when the hybrid switched between using the electric motor and the internal combustion engine.  In the Prius design, the same motor that is powering the car, powers the brakes.  When the hybrid is switching between motors, there might be a momentarily loss of power to the brakes during the transition.

A preliminary root cause analysis can be started using the available information.  The Cause Map can be expanded and revised as necessary as new information becomes available.  Click on the “Download PDF” button above to view the initial Cause Map.

Toyota has not stated whether a formal recall will be made.  A potential recall would affect 300,000 vehicles worldwide.

This new issue comes on the heels of a major announcement on January 21 where 2.3 million cars were recalled because of sticky gas pedals that can cause sudden acceleration. Additionally, there was a recall issued in September 2009 because there was a potential for floor mats to move out of place and cause the accelerator to stick. (A previous blog addressed this issue.)

Toyota shares dropped 21 percent following the January announcement and any farther safety issues will likely negatively impact consumer confident and stock prices.

Root Cause Analysis - Incident Investigation

Traffic Monitoring Plane Makes Emergency Landing

By ThinkReliability Staff

Just before rush hour began on Monday, February 1, 2010, traffic was stopped for a different reason – a plane landed in the median and then skidded off the road.  Thanks to quick thinking and the exemplary control of the pilot, nobody was hurt, though the plane did suffer considerable damage.  As with any incident, we can look at what happened and the effects in a Cause Map, or a visual root cause analysis.

First we record the specifics of the incident, such as date, time, place, equipment and process involved.  There’s also space to write if anything was different, though in this case it’s not clear what any differences were, so we can just enter a “?” to show we’re not sure.

Next we define the incident with respect to the organization’s goals.  Although nobody was hurt, an emergency landing (especially when the plane is damaged) has the potential to cause injuries.  These potential injuries are an impact to the safety goal.  There was significant traffic back-up after the incident, which is an impact to both the customer service and the production/schedule goal.  Last but not least, the damage to the plane is an impact to the property goal.  It’s unclear whether there was an impact to the environmental or labor/time goal, so we’ll put a “?” here, too.

Once we’ve defined the impact with respect to the goals, we can begin with those impacted goals to make our Cause Map.  The impact to the safety and property goals occurred when the plane hit trees on the side of the road.  This happened because the rear wheel of the aircraft caught in the muddy median, where the pilot landed to avoid traffic, AND because the plane made an emergency landing on the New Jersey Turnpike.  (The emergency landing caused rubbernecking, which impacted the customer service and production goals.)  The plane required an emergency landing because it was losing altitude after the loss of an engine.  (The plane was in the air giving traffic reports.)  The engine was lost because it was losing oil from a leak in the right wing fuel tank.  It’s unclear what caused the leak at this time.  The pilot chose to land on the highway because it was well lit, unlike the surrounding areas and because the traffic was light since rush hour had not yet begun.

As you can see on the downloadable PDF, a thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  We can build a significant portion of the Cause Map even with the little information that is currently available.  Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.  (Click on “Download PDF” to view the beginning of the root cause analysis investigation.)

Root Cause Analysis - Incident Investigation

Two DC Metro Workers Killed

By Kim Smiley

On January 26, 2010 just before 2 am, two Metro workers were killed near the Rockville metro station.  They were crushed by a metro utility vehicle while working on the track to install safety equipment.

The utility vehicle is a gas powered truck that is designed to operate on the track when electricity is shut off.  They are called high-rail vehicles and are typically used to carry equipment.  At the time of the accident, the vehicle was placing devices that tell approaching trains that there is a work crew in the area.

Many details of this accident are not available yet, but a preliminary root cause analysis can be started.  The basic information can be documented in an Outline and an initial cause map can be started.  Click on the “Download PDF” button above to see what this would look like.

The men killed and the workers in the vehicle were not part of the same crew and it’s not clear why the driver of the truck wasn’t aware that workers were in the area.  At the time of the accident the vehicle was traveling in reverse, which is a routine mode of operation.

Safety regulations require all vehicle operators to be informed about work crew locations, but it isn’t clear if that is being done effectively.

The National Transportation Safety Board (NTSB) has begun to investigate this incident and more details should be available as their investigation progresses.   The NTSB is currently reviewing employee work history and training and gathering all relevant data such as radio recordings and work procedures.

The DC Metro system has the worst safety record of any metro system in the country.  Five workers have now been killed while on the tracks in the last seven months.  There was also a metro train accident that killed 9 people on June 22, 2009.  To see a cause map of the June accident, click here.

Root Cause Analysis - Incident Investigation

Tragedy in Bhopal

By ThinkReliability Staff

While researching the tragedy in Bhopal, India, I discovered that there are two theories about what occurred on December 3, 1984 that resulted in a tremendous loss of life. One theory is from a report done by an Engineering Consulting firm hired by Union Carbide (the company that owned the plant in question) that determines that the release was caused by sabotage. Theory #2 is that a combination of inexperienced, ineffective workers and a badly maintained plant with inadequate safety standards that was being ready for dismantling experienced a horribly catastrophic chain of events that ensured that anything that could go wrong, did. For completeness, I have included both in my final Cause Map (which you can see by clicking “Download PDF” above). But for now, I’d just like to focus on the second.

In the wee morning hours of December 3, 1984, over 40 tons (this amount is also debated, but 40 tons appears to be the most popular, purely based on number of references that mention it) of methyl isocyanate (MIC) were released over the community of Bhopal, India, with a population of 900,000. Partially because of the transient nature of the population, and partially due to the general obfuscation of data from all sources involved, the number killed ranges from 2,000 to 15,000. The 2003 annual report of the Madhya Pradesh Gas Relief and Rehabilitation Department stated that a total of 15,248 people had died as a result of the gas leak. Based on claims accepted by the Indian government, there were at least 500,000 injured. This led to what has been called “The World’s Largest Lawsuit”, which I assume refers to the number of people represented, and certainly not the monetary amount of the settlement, which is a paltry $470 million. After the accident, the plant, after a series of legal maneuvers, was abandoned. Extensive cleanup was required, and still has not been completed. The impact to the goals are shown in the outline on the downloadable PDF.

The deaths and environmental impact were caused by the release of over 40 tons of methyl isocyanate (from here on out, we’ll refer to it as MIC). The release occurred when a large volume of MIC was put through an ineffective protection system. The release lasted several hours, because workers were unable to stop it, and because of an ineffective warning system. The release occurred when a disk and valve that led to the protection system burst due to an increase in pressure. The increase in pressure was caused by an increase in temperature resulting from a reaction between MIC and water when the refrigeration system was shut down. There were 41 metric tons of MIC in the tank, stored for use in the plant. How the water was introduced is the debate in the two theories I mentioned above. But regardless, water got in to the tank, either by sabotage or by leaking through a vent line. We will probably never know exactly what happened. But we do know that ineffective safety systems can result in a massive loss of life, as happened here.

Root Cause Analysis - Incident Investigation

Today in History: Fire on the USS Enterprise

By ThinkReliability Staff

On January 13, 1969, 31 years ago, fires and explosions broke out on the USS Enterprise (CVN-65). The crewmembers spent three hours fighting the fire. When the smoke cleared, 27 crewmembers were killed and 314 were injured. Additionally, 15 aircraft were destroyed and the carrier was severely damaged.

We can address the impacts to the U.S. Navy’s goals in a problem outline as the first step of the Cause Mapping process. There was an impact to the safety goal because crewmembers were killed and injured. There was an impact to the property goal because of the 15 planes that were damaged, and the repairs that were required to the ship. (This is also an impact to the labor goal, because of the labor required for the repairs.) Additionally, the ship’s deployment was delayed, which is an impact to both the customer service and production/schedule goals.

After we’ve completed the outline, we build our Cause Map beginning with the goals that were impacted. The goals were impacted by a series of explosions and fires across the ship. These explosions and fires were fueled by jet fuel and bombs that were found on the planes on the flight deck of the carrier. The initiating event was the explosion of a Mk-32 Zuni rocket, which exploded when it overheated due to being put in the exhaust path of an aircraft starting unit.

After the incident, the Navy performed an investigation to review the causes of the incident, and made changes to improve safety. Repairs to the Enterprise were completed, and the ship is now the oldest active serving ship in the U.S. Navy.

A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page. To view the downloadable PDF, click “Download PDF” above.

Root Cause Analysis - Incident Investigation

More on the Disappearance of Flight 188

By ThinkReliability Staff

In our previous blog about Flight 188 of Northwest Airlines, we discussed the first step of a root cause analysis investigation – defining the problem – and mentioned that a detailed Cause Map could be developed when more information regarding the incident was released.

The National Transportation Safety Board (NTSB) has recently released a report on what exactly happened to the flight. We can build off of the outline we already developed to put together the Cause Map, or visual root cause analysis.

First we begin with the impacts to the goals. Most importantly, the safety and property goals were impacted due to the potential danger to the flight. This was caused by the plane overshooting the destination. The pilots flew over the destination because they were distracted, warnings were not effectively delivered to them, and they couldn’t see their destination (Minneapolis-St. Paul), since it was after dark and cloudy.

The pilots were distracted by a non-operation activity. The two pilots were utilizing the scheduling software on their laptops, both of which were open in the cockpit (possibly blocking some of the flight display). Both using personal laptops and participating in non-operational activities is prohibited by the airline.

Some may ask how it’s possible that two pilots who were flying a plane – with over a hundred passengers – could be spending all their energy on another activity. Well, the pilots did not actually have any active tasks to fly the plane. The plane was on auto-pilot, and the one task that pilots ordinarily did on a regular basis (which would have certainly alerted the pilots to their position) was sending a position report. However, a dispatcher for the airliner had asked the pilots NOT to send a report, as the reports were burdensome and unneccessary.

Warnings did not effectively get through to the pilots by sight – either the flight display was physically blocked by the laptop or the pilots weren’t looking at it because they were distracted – or sound – the plane was not equipped to send audible message (such as chimes or buzzers) to the pilots, text messages sent to them were not acknowledged, and the pilots did not hear calls for them on the radio. The air traffic controllers (who were different from the air traffic controllers who had first had contact with the plane) did not know which frequency the plane was on, so only some messages got through. Because the pilots were using the speaker instead of headsets and were, again, distracted, they missed the messages.

Both of the pilots involved had their licenses revoked. Several procedures were not followed in this instance, and the FAA and individual airlines are working on highlighting the importance of these procedures. Reading about this incident (and seeing that the pilots’ license were revoked) will probably do much to highlight the importance of the procedures. Luckily, nobody was hurt for this lesson to be learned.

View the root cause analysis investigation by clicking “Download PDF” above.