Tag Archives: fire

Root Cause Analysis - Incident Investigation

Small fire leads to thousands of canceled flights

By Kim Smiley

Starting August 8, 2016, thousands of travelers were stranded worldwide after widespread cancelations and delays of Delta Air Lines flights. The disruptions continued over several days and the impacts lingered even longer.  The flight issues made headlines around the globe and the financial impact to the company was significant.

So what happened? What caused this massive headache to so many travelers? The short answer is a small fire in an airline data center, but a much longer answer is needed to understand what caused this incident. A Cause Map, a visual format for performing a root cause analysis, can be used to analyze this issue. All of the causes that contributed to an issue are visually laid out to intuitively show cause-and-effect relationships in a Cause Map.  The Cause Map is built by asking “why” questions and adding the answers.  For an effect with more than one cause, all of the causes that contributed to the effect are listed vertically and separated by an “and”.  (Click on “Download PDF” to see an intermediate level Cause Map of this incident.)

So why were so many flights canceled and delayed? There was a system-wide computer outage and the airline depends on computer systems for everything from processing check-ins to assigning crews and gates.  Bottom line, no flights leave on time without working computer systems.  The issues originated at a single data center, but the design of the system led to cascading computer issues that impacted systems worldwide.  The airline has not released any specific details about why exactly the issue spread, but this is certainly an area investigators would want to understand in order to create a solution to prevent a similar cascading failure in the future.

In a statement, the company indicated that an electrical component failed, causing a small fire at the data center. (Again, the specifics about what type of component and what caused the failure haven’t been released.) The fire caused a transformer to shut down which resulted in a loss of primary power to the data center.  A secondary power system did kick on, but not all servers were connected to backup power.  No details have been released about why some servers were not powered by the secondary power supply.

Compounding the frustration for the impacted travelers is the fact that they were unable to get updated flight information. Flight status systems, including airport monitors, continued to show that all flights were on time during the period of the cancelations and delays.

Once a large number of flights are disrupted, it is difficult to return to a normal flight schedule.  The rotation schedule for airlines and pilots has to be redone, which can be time-consuming.  Many commercial flights operate near capacity so it can be difficult to find seats for all the passengers impacted by canceled and delayed flights.  Delta has tried to compensate travelers impacted by this incident by offering refunds and $200 in travel vouchers to people whose flights were canceled or delayed at least three hours, but an incident of this magnitude will naturally impact customer confidence in the company.

This incident is a good reminder of the importance of building robust systems with functional backups; otherwise a small problem can spread and quickly become a big problem.

Root Cause Analysis - Incident Investigation

Train Derails on Track Just Inspected

By ThinkReliability Staff

A train derailment in the Columbia River Gorge near Mosier, Oregon resulted in a fire that burned for 14 hours. The Federal Railroad Administration (FRA) preliminary investigation says the June 3rd derailment was caused by a broken lag bolt which allowed the track to spread, resulting in the 16-car derailment. Although there is only one other known instance of a broken lag bolt causing a train derailment, the FRA determined that the bolt had been damaged for some time, and had been inspected within days of the incident, raising questions about the effectiveness of these inspections.

Determining all the causes of a complex issue such as a train derailment can be difficult, but doing so will provide the widest selection of possible solutions. A Cause Map, or visual root cause analysis, addresses all aspects of the issue by developing cause-and-effect relationships for all the causes based on the impacts to an organization’s goals. We can create a Cause Map based on the preliminary investigation. Additional causes and evidence can be added to the map as more detail is known.

The first step in the Cause Mapping process is to determine the impacts to the organization’s goals. While there were no injuries in this case, the massive fire resulting from the derailment posed a significant risk to responders and nearby citizens, an impact to the safety goal. The release of 42,000 gallons of oil (although much of it was burned off in the fire) is an impact to the environmental goal. The customer service goal is impacted by the evacuation of at least 50 homes and the regulatory goal is impacted by the potential for penalties, although the National Transportation Safety Board (NTSB) has said it will not investigate the incident. The state of Oregon has requested a halt on oil traffic, which would be an impact to the schedule goal. The property goal is impacted by the damage to the train cars, and the labor/ time goal is impacted by the response and investigation.

The analysis, which is the second step in the Cause Mapping process, begins with one of the impacted goals and develops cause-and-effect relationships by asking ‘Why’ questions. In this case, the safety goal is impacted by the high potential for injuries. This is caused by the massive fire, which burned for 14 hours. There may be more than one cause resulting in an effect, such as a fire, which is caused by heat, fuel, and oxygen. The oxygen in this case is from the atmosphere. The heat source is unknown but could have been a spark caused by the train derailment. The fire was fueled by the 42,000 gallons of crude released due to damage to train cars, which were transporting crude from the Bakken oil fields, caused by the derailment.

The derailment of 16 cars of the train was caused by the broken lag bolt. Any mechanical failure, such as a break, results from the stress on that object exceeding the strength of the object. In this case, the stress was caused by the weight of the 94-car train. The length of a train carrying crude oil is not limited by federal regulations. The strength of the bolts was reduced due to previous damage, which was not identified prior to the failure. While the track strength is evaluated every 18 months by the Gauge Restraint Measurement System (GRMS), it did not identify the damage. It’s unclear the last time it was performed.

Additionally, although the track is visually inspected twice a week by the railroad, it is done by vehicle, which would have made the damage harder to spot. The FRA does not require walking inspections. Nor does the FRA inspect or review the railroad’s inspections very often – there are less than 100 inspectors for the 140,000 miles of track across the country. There are only 3 in Oregon.

As a result of the derailment, the railroad has committed to replacing the existing bolts with heavy-duty ones, performing GRMS four times a year, enhanced hyrail inspections and visual track inspections three times a week, and performing walking inspections on lag curves monthly.

The FRA is still evaluating actions against the railroad and is again calling for the installation of advanced electronic brakes, or positive train control (PTC). It has also recommended PTC after other incidents, such as the deaths of two railroad workers on April 3 (see our previous blog) and the derailment in Philadelphia last year that killed 8 (see our previous blog).

To view a one-page PDF of the Cause Mapping investigation, click on “Download PDF” above. Or, click here to read the FRA’s preliminary investigation.

Root Cause Analysis - Incident Investigation

Airplane Emergency Instructions: How do you make a work process clear?

By ThinkReliability Staff

What’s wrong with the process above?

This process provides instructions on how to remove the over-wing exit door on an airplane during an emergency.  However, imagine performing this process in an actual emergency.  During the time you spend opening the door, there will probably be people crowded behind you, frantic to get off the plane.  Step 4 indicates that after the door is detached from the plane wall, you should turn around and set the door (which is about 4’ by 2’ and can weigh more than 50 pounds) on the seats behind you.  In most cases, this will be impossible.  This is why emergency exit doors open towards the outside; in an emergency, a crush against the door will make opening the door IN impossible.

Even if it would be possible to place the door on the seat in the emergency exit row, it would likely reduce the safety of passengers attempting to exit.  As discussed, the exit door is fairly large and heavy.  It is likely to be displaced while passengers are exiting the airplane and may end up falling on a passenger, or blocking the exit path.

However, when this process was tested in training, it probably worked fine.  Why? Because it wasn’t an actual emergency, and there probably weren’t a plane full of passengers that really wanted to get out.  This is just another reason that procedures need to be tested in as close to actual situations as possible.  At the very least, any scenario under which the process is to be performed should be replicated as nearly as possible.

Now take a look at this procedure:

It’s slightly better, not telling us to put the removed door on the seat behind us, but instead it doesn’t tell us what to do with the door. Keep in mind that the person performing this procedure’s “training” likely consisted of a 30-second conversation with a flight attendant and that in all probability, the first time he or she will perform the task is during an emergency situation. When testing a procedure, it’s also helpful to have someone perform the procedure who is not familiar with it, with instructions to do only what the procedure says. In this case, that person would end up removing the door . . . and then potentially attempting to climb out of the exit with the door in their hands. This is also not a safe or efficient method of emergency escape.
This procedure provides a much better description of what should be done with the door. The picture clearly indicates that the door should be thrown out of the plane, where it is far less likely to block the exit or cause passenger injury.

The first two procedures were presumably clear to the person who created them.  But had they been tested by people with a variety of experience levels (particularly important in this case, because people of various experience levels may be required to open the doors in an emergency), the steps that really weren’t so clear may have been brought to light.

Reviewing procedures with a fresh eye (or asking someone to perform the procedure under safe conditions based only upon the written procedure) may help to identify steps that aren’t clear to everyone, even if they were to the writer.  This can improve both the safety, and the effectiveness, of any procedure used in your organization.

Root Cause Analysis - Incident Investigation

DC Metro shut down for entire day after fire for inspections

By Kim Smiley 

A fire in a DC Metro tunnel early on March 14, 2016 caused delays on three subway lines and significant disruption to both the morning and evening commutes.  There were no injuries, but the similarities between this incident and the deadly smoke incident on January 12, 2015 (see our previous blog on this incident) led officials to order a 24-hour shutdown of the entire Metro system for inspections and repairs.

The investigation into the Metro fire is still ongoing, but the information that is known can be used to build an initial Cause Map.  A Cause Map is built by asking “why” questions and visually laying out all the causes that contributed to an incident.  Cause Mapping an issue can identify areas where it may be useful to dig into more detail to fully understand a problem and can help develop effective solutions.

So why was there a fire in the Metro tunnel?  Investigators have not released details about the exact cause, but have stated that the fire was caused by issues with a jumper cable.  Jumper cables are used in the Metro system to bridge gaps in the third rail, essentially functioning as extension cords.  The Metro system uses gaps in the third rail to create safer entry and exit spaces for both workers and passengers because of the potential danger of contact with the electrified third rail.  The third rail carries 750 volts of electricity used to power Metro trains and could cause serious injury or even death if accidently touched.

The jumper cables also carry high voltage and fires and/or smoke can occur if one malfunctions.  Investigators have not confirmed the exact issue that lead to this fire, but insulation failures have been identified in other locations and is a possible cause of the fire. (Possible causes can be added to the Cause Map with a “?” to indicate that more evidence is needed.)

One of the things that is always important to consider when investigating an incident is the frequency of occurrence of similar issues.  The scope of the investigation and possible solutions considered will likely be different if it was the 20th time an incident has occurred rather than the first. In this case, the fire was similar to another incident in January 2015 that caused a passenger death.  Having a second incident occur so soon after the first naturally raised questions about whether there were more unidentified issues with jumper cables.  The Metro system uses approximately 600 jumper cables and all were inspected during the day-long shutdown. Twenty-six issues were identified and repaired. Three locations had damage severe enough that Metro would have immediately stopped running trains through them if the extent of the damage had been known.

The General Manger of the DC Metro system, Paul J. Wiedefeld, is relatively new to his position and has been both praised and criticized for the shutdown.  Trying to implement solutions and reduce risk is always a balancing act between costs and benefits.  Was the cost of a full-day shutdown and inspections of all jumper cables worth the benefit of knowing that the cable jumpers have all been inspected and repaired?  At the end of the day, it’s a judgement call, but I personally would be more comfortable riding the Metro with my children now.

Root Cause Analysis - Incident Investigation

High School Open Flame Chemistry Demonstration Ends in Injuries

By Kim Smiley

Six were injured, two seriously, in an accident involving an open flame chemistry demonstration at a high school in Fairfax County, Virginia on October 31, 2015.  At the time of the incident, the teacher was performing a well-known experiment to show the students how different chemical elements can change the color of a flame. According to students present in the classroom, the teacher was in the process of adding more flammable liquid to the experiment when a splash of fire hit students and the teacher.

A Cause Map, or visual root cause analysis, can be used to analyze this incident.  The first step in the Cause Mapping process is to fill in an outline to document all the basic background information for an incident such as time, date, and location.  Additionally, how the incident impacts the organization’s goals is listed on the bottom of the outline.  For this example, the safety goal is clearly impacted by the injuries, but there are several other impacts that need to be considered as well such as the damage to the classroom, evacuation of the school and required emergency response.  Fairfax County has also banned all open flame experiments pending a thorough investigation of this issue which can be considered an impact to the regulatory goal.

Once the Outline is complete, the Cause Map itself is built by asking “why” questions beginning with one of the impacted goals. Starting at the safety goal in this example, the first step would be to ask “why” were 6 people injured?  These injuries occurred because people were burned because there was an uncontrolled fire in a classroom, people were near the fire and no protective gear was worn.  (When there is more than one cause that contributes to an effect, the cause boxes are listed vertically and separated by “and” to show that all causes were required.)  No information has been released to the public about why the students were sitting so near the open flame experiment without any type of safety barrier or why protective gear wasn’t worn, but these are both branches of the Cause Map that should be expanded during a complete investigation.  If the same fire had occurred, injuries may have been prevented or at least been less severe if the students were farther away from the flames or if they had protective gear on to protect them from burns.  It’s important to understand why the experiment was performed as it was in order to develop solutions that could prevent injuries in the future.

There has been a little information released about why the fire was uncontrolled during the experiment.  Eyewitnesses have stated that the teacher was adding more fuel to the fire because it was starting to burn out.  As liquid fuel was added, the fire spread unexpectedly and burning fuel splashed out of the experiment location onto students and the teacher performing the experiment.  The specific details of what occurred during this specific fire have not been released and should be looked at during the detailed investigation.  Once more information is known, the Cause Map could be easily expanded to incorporate it.

The Chemical Safety Board (CSB) is not investigating this incident, but has stated that it is gathering information on it.  The recent accident appears to be similar to three accidents involving open flame experiments that injured children during an 8 week period in 2014.  These three accidents all involved experiments using flammable liquid, a flashback to the bulk containers of fuel and fire engulfing members of the audience.  Following the 2014 accidents, the CSB issued a safety bulletin titled “Key Lessons for Preventing Incidents from Flammable Chemicals in Educational Demonstrations”.   Key lessons listed from the CSB safety bulletin that should be considered when planning open flame experiments are as follows:

– Do not use bulk containers of flammable chemicals in educational demonstrations when small quantities are sufficient.

– Implement strict safety controls when demonstrations necessitate handling hazardous chemicals – including written procedures, effective training, and the required use of appropriate personal protective equipment for all participants.

– Conduct a comprehensive hazard review prior to performing any educational demonstration.

– Provide a safety barrier between the demonstration and audience.

Uncategorized

Runway Fire Forces Evacuation of Airplane

By ThinkReliability Staff

On September 8, 2015, an airplane caught fire during take-off from an airport in Las Vegas, Nevada. The pilot was able to stop the plane, reportedly in just 9 seconds after becoming aware of the fire. The crew then evacuated the 157 passengers, 27 of whom received minor injuries as a result of the evacuation by slide. Although the National Transportation Safety Board (NTSB) investigation is ongoing, information that is known, as well as potential causes that are under consideration, can be diagrammed in a Cause Map, or visual root cause analysis.

The first step of Cause Mapping is to define the problem by completing a problem outline. The problem outline captures the background information (what, when and where) of the problem, as well as the impact to the goals. In this case, the safety goal is impacted due to the passenger injuries. The evacuation of the airplane impacts the customer service goal. The NTSB investigation impacts the regulatory goal. The schedule goal is impacted by a temporary delay of flights in the area, and the property goal is impacted by the significant damage to the plane. The rescue, response and investigation is an impact to the labor goal.

The Cause Map is built by beginning with one of the impacted goals and asking “Why” questions to develop the cause-and-effect relationships that led to an issue.   In this case, the injuries were due to evacuation by slide (primarily abrasions, though some sources also said there were some injuries from smoke inhalation). These injuries were caused by the evacuation of the airplane. The airplane was evacuated due to an extensive fire. Another cause leading to the evacuation was that take-off was aborted.

The fact that take-off was able to be aborted, for which the pilot has been hailed as a hero, is actually a positive cause. Had the take-off been unable to be aborted, the result would likely have been far worse. In the case of the Concorde accident, a piece of debris on the runway ruptured a tire, which caused damage to the fuel tank, leading to a fire after the point where take-off could be aborted. Instead, the aircraft stalled and crashed into a hotel, killing all onboard the craft and 4 in the hotel. The pilot’s ability to quickly save the plane almost certainly saved many lives.

The fire is thought to have been initiated by an explosion in the left engine due a catastrophic uncontained explosion of the high-pressure compressor. This assessment is based on the compressor fragments that were found on the runway. This likely resulted from either a bird strike (as happened in the case of US Airways flight 1549), or a strike from other debris on the runway (as occurred with the Concorde), or fatigue failure of the engine components due to age. This is the first uncontained failure of this type of engine, so some consider fatigue failure to be less likely. (Reports of an airworthiness directive after cracks were detected in weld joints of compressors were in engines with different parts and a different compressor configuration.)

In this incident, the fire was unable to be put out without assistance from responding firefighters. This is potentially due to an ongoing leak of fuel if fuel lines were ruptured and the failure of the airplane’s fire suppression system, which reportedly deployed but did not extinguish the fire. Both the fuel lines and fire suppression system were likely damaged when the engine exploded. The engine’s outer casing is not strong enough to contain an engine explosion by design, based on the weight and cost of providing that strength.

The NTSB investigation is examining airplane parts and the flight data and cockpit voice recorders in order to provide a full accounting of what happened in the incident. Once these results are known, it will be determined whether this is considered an anomaly or whether changes to all planes using a similar design and configuration need to take action to prevent against a similar event recurring.

To view the initial investigation information on a one-page downloadable PDF, please click “Download PDF” above.

 

Uncategorized

Indian Point Fire and Oil Leak

By Sarah Wrenn

At 5:50 PM on May 9, 2015, a fire ignited in one of two main transformers for the Unit 3 Reactor at Indian Point Energy Center. These transformers carry electricity from the main generator to the electrical grid. While the transformer is part of an electrical system external to the nuclear system, the reactor is designed to automatically shut down following a transformer failure. This system functioned as designed and the reactor remains shut down with the ongoing investigation. Concurrently, oil (dielectric fluid) spilled from the damaged transformer into the plant’s discharge canal and some amount was also released into the Hudson River. On May 19, Fred Dacimo, vice president for license renewal at Indian Point and Bill Mohl, president of Entergy Wholesale Commodities, stated the transformer holds more than 24,000 gallons of dielectric fluid. Inspections after the fire revealed 8,300 gallons have been collected or were combusted during the fire. As a result, investigators are working to identify the remaining 16,000 gallons of oil. Based on estimates from the Coast Guard supported by NOAA, up to approximately 3,000 gallons may have gone into the Hudson River.

The graphic located here provides details regarding the event, facility layout and response.

Step 1. Define the Problem

There are a few problems in this event. Certainly, the transformer failure and fire are major problems. The transformer is an integral component to transfer electricity from the power plant to the grid. Without the transformer, production has been halted. In addition, there is an inherent risk of injury with the fire response. The site’s fire brigade was dispatched to respond to the fire and while there were no injuries, there was a potential for injury. In addition, the release of dielectric fluid and fire-retardant foam into the Hudson River is a problem. A moat around the transformer is designed to contain these fluids if released, but evidence shows that some amounts reached the Hudson River.

As shown in the timeline and noted on our problem outline, the transformer failure and fire occurred at 5:50 PM and was officially declared out 2.25 hours later.

As far as anything out of the ordinary or unusual when this event occurred, Unit 3 had just returned to operations after a shutdown on May 7 to repair a leak of clean steam from a pipe on the non-nuclear side of the plant. Also, it was noted that this is the 3rd transformer failure in the past 8 years. This frequency of transformer failures is considered unusual. The Wall Street Journal reported that the transformer that failed earlier this month replaced another transformer that malfunctioned and caught fire in 2007. Another transformer failed in 2010, which had been in operation for four years.

Multiple organizational goals were negatively impacted by this event. As mentioned above, there was a risk of injury related to the fire response. There was also a negative impact to the environment due to the release of dielectric fluid and fire-retardant foam. The negative publicity from the event impacts the organization’s customer service goal. A notification to the NRC of an Unusual Event (the lowest of 4 NRC emergency classifications) is a regulatory impact. For production/schedule, Unit 3 was shutdown May 9 and remains shutdown during the investigation. There was a loss of the transformer which needs to be replaced. Finally, there is labor/time required to address and contain the release, repair the transformer, and investigate the incident.

Step 2. Identify the Causes (Analysis)

Now that we’ve defined the problem in relation to how the organization’s goals were negatively impacted, we want to understand why.

The Safety Goal was impacted due to the potential for injury. The risk of injury exists because of the transformer fire.

 

 

The Regulatory Goal was impacted due to the notification to the NRC. This was because of the Unit 3 shutdown, which also impacts the Production/Schedule Goal. Unit 3 shutdown as this is the designed response to the emergency. This is the designed response because of the loss of the electrical transformer, which also impacts the Property/Equipment Goal. Why was the electrical transformer lost? Because of the transformer fire.

For the other goals impacted, Customer Service was because of the negative publicity which was caused by the containment, repair, investigation time and effort. This time and effort impacts the organization’s Labor/Time Goal. This time and effort was required because of the dielectric fluid and fire-retardant foam release. Why was there a release? Because the fluid and foam were able to access the river.

Why did the fluid and foam access the river?

The fire-retardant foam was introduced because the sprinkler system was ineffective. The transformer is located outside in the transformer yard which is equipped with a sprinkler system. Reports indicate that the fire was originally extinguished by the sprinklers, but then relit. Fire responders introduced fire-retardant foam and water to more aggressively address the fire. Some questions we would ask here include why was the sprinkler system ineffective at completely controlling the fire? Alternatively, is the sprinkler system designed to begin controlling the fire as an immediate response such that the fire brigade has time to respond? If this is the case, then did the sprinkler perform as expected and designed?

The transformer moat is designed to catch fluids and was unable to contain the fluid and the foam. When a containment is unable to hold the amount of fluid that is introduced, this means that either there is a leak in the containment or the amount of fluid introduced is greater than the capacity of the containment. We want to investigate the integrity of the containment and if there are any leak paths that would have allowed fluids to escape the moat. We also want to understand the volume of fluid that was introduced. The moat is capable of holding up to 89,000 gallons of fluid. A transformer contains approximately 24,000 gallons of dielectric fluid. What we don’t know is how much fire-retardant foam was introduced. If this value plus the amount of transformer fluid is greater than the capacity of the moat, then the fluid will overflow and can access the river. If this is the case, we also would want to understand if the moat capacity is sufficient, should it be larger? Also, is the moat designed such that an overflow will result in accessing the discharge canal and is this desired?

Finally, dielectric fluid accessed the river because the fluid was released from the transformer. Questions we would ask here are: Why was the fluid released and why does a transformer contain dielectric fluid? Dielectric fluid is used to cool the transformers. Other cooling methods, such as fans are also in place. The causes of the fluid release and transformer failure is still being investigated, but in addition to determining these causes, we would also ask how are the transformers monitored and maintained? The Wall Street Journal provided a statement from Jerry Nappi, a spokesman for Entergy. Nappi said both of unit 3’s transformers passed extensive electrical inspections in March. Transformers at Indian Point get these intensive inspections every two years. Aspects of the devices also are inspected daily.

Finally, we want to understand why was there a transformer fire. The transformer fire occurred because there was some heat source (ignition source), fuel, and oxygen. We want to investigate what was the heat source – was there a spark, a short in the wiring, a static electricity build up? Also, where did the fuel come from and is it expected to be there? The dielectric fluid is flammable, but are there other fuel sources that exist?

Step 3. Select the Best Solutions (Reduce the Risk)

What can be done? With the investigation ongoing, a lot of facts still need to be gathered to complete the analysis. Once that information is gathered, we want to consider what is possible to reduce the risk of having this type of event occur in the future. We would want to evaluate what can be done to address the transformer, implementing solutions to better maintain, monitor, and/or operate it. Focusing on solutions that will minimize the risk of failure and fire. However, if a failure does occur, we want to consider solutions so that the failure and fire does not result in a release. Further, we can consider the immediate response; do these steps adequately contain the release? Identifying specific solutions to the causes identified will provide reductions to the risk of future similar events.

Resources:

This Cause Map was built using publicly available information from the following resources.

De Avila, Joseph “New York State Calls for Tougher Inspections at Indian Point” http://www.wsj.com/articles/nuclear-regulatory-commission-opens-probe-at-indian-point-1432054561 Published 5/20/2015. Accessed 5/20/2015

“Entergy’s Response to the Transformer Failure at Indian Point Energy Center” http://www.safesecurevital.com/transformer_update/ Accessed 5/19/2015

“Entergy Plans Maintenance Shutdown of Indian Point Unit 3” http://www.safesecurevital.com/entergy-plans-maintenance-shutdown-of-indian-point-unit-3/ Published 5/7/2015. Accessed 5/19/2015

“Indian Point Unit 3 Safely Shutdown Following Failure of Transformer” http://www.safesecurevital.com/indian-point-unit-3-safely-shutdown-following-failure-of-transformer/ Published 5/9/2015. Accessed 5/19/2015

“Entergy Leading Response to Monitor and Mitigate Potential Impacts to Hudson River Following Transformer Failure at Indian Point Energy Center” http://www.safesecurevital.com/entergy-leading-response-to-monitor-and-mitigate-potential-impacts-to-hudson-river-following-transformer-failure-at-indian-point-energy-center/ Published 5/13/2015. Accessed 5/19/2015

“Entergy Continues Investigation of Failed Transformer, Spilled Dielectric Fluid at Indian Point Energy Center” http://www.safesecurevital.com/entergy-continues-investigation-of-failed-transformer-spilled-dielectric-fluid-at-indian-point-energy-center/ Published 5/15/2015. Accessed 5/19/2015

McGeehan, Patrick “Fire Prompts Renewed Calls to Close the Indian Point Nuclear Plant” http://www.nytimes.com/2015/05/13/nyregion/fire-prompts-renewed-calls-to-close-the-indian-point-nuclear-plant.html?_r=0 Published 5/12/2015. Accessed 5/19/2015

Screnci, Diane. “Indian Point Transformer Fire” http://public-blog.nrc-gateway.gov/2015/05/12/indian-point-transformer-fire/comment-page-2/#comment-1568543 Accessed 5/19/2015

Root Cause Analysis - Incident Investigation

New Regulations Aim to Reduce Railroad Crude Oil Spills

By ThinkReliability Staff

The tragic train derailment in Lac-Mégantic, Quebec on July 6, 2013 (see our previous blog on this topic) ushered in new concerns about the transport of crude oil by rail in the US and Canada. Unfortunately, the increased attention has highlighted a growing problem: spills of crude oil transported via rail, which can result in fires, explosions, evacuations, and potentially deaths. (Luckily there have been no fatalities since the Lac-Mégantic derailment.) According to Steve Curwood of Living on Earth, “With pipelines at capacity the boom has lead a 4,000 percent increase in the volume of crude oil that travels by rail, and that brought more accidents and more oil spills in 2014 than over the previous 38 years.”

This follows a period of increases in railroad safety – according to the US Congressional Research Service, “From 1980 to 2012, railroads reduced the number of accidents releasing hazmat product per 100,000 hazmat carloads from 14 to 1.” From October 19, 2013 to May 6, 2015, there were at least 12 railcar derailments that resulted in crude oil spills. (To see the list of events, click on “Download PDF” and go to the second page.)

Says Sarah Feinberg, acting administrator of the Federal Railroad Administration (FRA), “There will not be a silver bullet for solving this problem. This situation calls for an all-of-the-above approach – one that addresses the product itself, the tank car it is being carried in, and the way the train is being operated.” All of these potential risk-reducing solutions are addressed by the final rule released by the FRA on May 1, 2015. (On the same day, the Canadian Ministry of Transport released similar rules.) In order to view how the various requirements covered by the rule impact the risk to the public as a result of crude oil spills from railcars, we can diagram the cause-and-effect relationships that lead to the risk, and include the solutions directly over the cause they control. (To view the Cause Map, or visual root cause analysis, of crude oil train car derailments, click on “Download PDF”.)

The product: Bakken crude oil (as well as bitumen) can be more volatile than other types of crude oil and has been implicated in many of the recent oil fires and explosions. In addition to being more volatile, the composition (and thus volatility) can vary. If a material is not properly sampled and characterized, proper precautions may not be taken. The May 1 rule incorporates a more comprehensive sampling and testing program to ensure the properties of unrefined petroleum-based products are known and provided to the DOT upon request.   (Note that in the May 6, 2015 derailment and fire in Heimdahl, North Dakota, the oil had been treated to reduce its volatility, so this clearly isn’t an end-all answer.)

The tank car: Older tank cars (known as DOT-111s) were involved in the Lac-Mégantic and other 2013 crude oil fires. An upgrade to these cars, known as CPC-1232, hoped to reduce these accidents. However, CPC-1232 cars have been involved in all of the issues since 2013. According to Cynthia Quarterman, former director of the Pipeline and Hazardous Materials Safety Administration, says that the recent accidents involving the newer tank cars “confirm that the CPC-1232 just doesn’t cut it.”

The new FRA rule establishes requirements for any “high-hazard flammable train” (HHFT) transported over the US rail network. A HHFT is a train comprised of 20 or more loaded tank cars of a Class 3 flammable liquid (which includes crude oil and ethanol) in a continuous block or 35 or more loaded tank cars of a Class 3 flammable liquid across the entire train. Tank cars used in HHFTs constructed after October 1, 2015 are required to meet DOT-117 design criteria, and existing cars must be retrofitted based on a risk-based schedule.

The way the train is being operated: The way the train is being operated includes not only the mechanics of operating the train, but also the route the train takes and the notifications required along the way. Because the risk for injuries and fatalities increases as the population density increases, the rule includes requirements to perform an analysis to determine the best route for a train. Notification of affected jurisdictions is also required.

Trains carrying crude oil tend to be very large (sometimes exceeding one mile in length). This can impact stopping distance as well as increase the risk of derailment if sudden stopping is required. To reduce these risks, HHFTs are restricted to 50 mph in all areas, and 40 mph in certain circumstances based on risk (one of the criteria is urban vs. rural areas). HHFTs are also required to have in place a functioning two-way end of train or distributed power braking system. Advanced braking systems are required for trains including 70 or more loaded tank cars containing Class 3 flammable liquids and traveling at speeds greater than 30 mph, though this requirement will be phased in over decades.

It is important to note that this new rule does not address inspections of rails and tank cars. According to a study of derailments from 2001 to 2010, track problems were the most important causes of derailments (with broken rails or track welds accounting for 23% of total cars derailed). A final rule issued January 24, 2014 required railroads to achieve a specified track failure rate and to prioritize remedial action.

To view the May 1 rule regarding updates to crude-by-rail requirements, click here. To view the timeline of incidents and the Cause Map showing the cause-and-effect relationships leading to these incidents, click “Download PDF”.

Root Cause Analysis - Incident Investigation

Train Derails in West Virginia

By Kim Smiley

On February 16, 2015, a train hauling 109 tank cars of crude oil derailed in Mount Carbon, West Virginia.  It has been reported that 27 tank cars in the train derailed.  Some of the tank cars were damaged and released an unknown amount of crude oil, resulting in a large fire.  Hundreds of families in the surrounding area were evacuated, but only one injury was reported.

The accident investigation is still ongoing, but what information is known can be used to build an initial Cause Map, a visual format for performing a root cause analysis.  The Cause Map can be easily expanded as needed to document additional information as it becomes available.

The first step in the Cause Mapping process is to fill in an Outline with the basic background information for the issue as well as how the overall goals were impacted. In this example, there were many impacted goals.  The safety goal is impacted because there was an injury, the property goal is impacted because of the damage to the train, the environmental goal is impacted because of the release of oil, etc.  Once the Outline is complete, the Cause Map itself is built by starting with an impact to a goal, asking “why” questions, and laying out all the causes that contributed to an issue.

The significant aftermath of this derailment is known, but little has been released about what specifically caused the train to derail.  It was snowing heavily at the time of the accident, which may have played a role, but since more evidence is needed, a “?” is included on the Cause Map.  Data from the digital data recorder has shown that the train was not speeding at the time of the accident, which has been a factor in previous derailments.  Another fact worth noting is that the damaged train cars were a newer design that incorporated modern safety upgrades.

With so many unknowns, the Federal Railroad Administration is conducting a full-scale investigation to determine exactly what happened.  The damaged tank cars, track, and other components along with relevant maintenance and inspection records will be all be analyzed to better understand this derailment.

Unfortunately, crude oil train accidents are predicted to become increasingly common as the volume of flammable liquids being transported by rail continues to rise.  According to the Association of American Railroad, 40 times more oil was transported by rail in 2012 than in 2008. Hopefully, the lessons learned from this derailment can be used to help reduce the risk of future rail accidents.

To view the Outline and initial Cause Map for this accident, click on “Download PDF” above.

Root Cause Analysis - Incident Investigation

Deadly Train-Car Collision

By Kim Smiley

On February 3, 2015, an SUV was struck by a commuter train near Valhalla, New York.  The driver of the vehicle and 5 train passengers were killed in the accident.  The National Transportation Safety Board (NTSB) is investigating the accident to determine what went wrong.

An initial Cause Map, a visual root cause analysis, can be built to analyze and document what is known about this train-car collision.  A Cause Map visually lays out the cause-and-effect relationships that contributed to an issue and focuses on understanding all the causes, not THE root cause.  Generally, identifying more causes results in a greater number of potential solutions being considered.

So why did the train hit a vehicle?  Eyewitnesses have stated that the SUV was hit by a crossing gate as it descended.    It is not clear why the SUV didn’t stop prior to entering the railroad crossing area. The driver pulled the SUV forward onto the tracks rather than backing up and the train struck the vehicle shortly after.  Investigators don’t know why the driver stopped on the tracks, but initial reports are that all safety features, such as the crossing gate, signs and train horn, were functioning properly at the time of the accident.

Unfortunately, it’s not unusual for passengers in a vehicle struck by a train to be injured or killed, but it is less common for fatalities among the train passengers.  Investigators are working to determine what made this accident particularly dangerous for train passengers.  The NTSB plans to use information about the passengers’ injuries and a diagram of where people were sitting on the train to try to understand what happened during the collision.  Post-accident photos of the train show that significant fire damage occurred, likely fueled by the gas in the SUV.

One of the open questions is whether the electrified third rail contributed to the accident and subsequent injuries. Metro-North uses an unusual “under-running” third rail design where power is taken from the bottom of the rail.  During the collision, 400 feet of the third rail broke apart and 12 pieces pierced both the SUV and the train. This rail design uses a metal shoe that slips underneath the third rail and some think that the force of the collision may have essentially pried up the rail and threw it into the train and vehicle.

Open questions can be documented on the initial Cause Map with a question mark.  As more information becomes available, the Cause Map can quickly be updated.  Typically, Cause Maps are built in Excel and different versions can be saved as different sheets to document the investigation process.

Click on “Download PDF” above to view an initial Cause Map of this accident, built from the information in the media articles on the accident.