Monthly Archives: May 2015

Root Cause Analysis - Incident Investigation

Deadly Train Derailment Near Philadelphia

By Kim Smiley

On the evening of May 12, 2015, an Amtrak train derailed near Philadelphia, killing 8 and injuring more than 200.  The investigation is still ongoing with significant information about the accident still unknown, but changes are already being implemented to help reduce the risk of future rail accidents and improve investigations.

Data collected from the train’s onboard event recorder shows that the train sped up in the moments before the accident until it was traveling 106 mph in a 50 mph zone where the train track curved.  The excessive speed clearly played a role in the accident, but there has been little information released about why the train was traveling so fast going into a curve.  The engineer controlling the train suffered a head injury during the accident and has stated that he has no recollection of the accident. The engineer was familiar with the route and appears to have had all required training and qualifications.

As a result of this accident and the difficulty determining exactly what happened, Amtrak has announced that cameras will be installed inside locomotives to record the actions of engineers.  While the cameras may not directly reduce the risk of future accidents, the recorded data will help future investigations be more accurate and timely.

The excessive speed at the time of the accident is also fueling the ongoing debate about how trains should be controlled and the implementation of positive train control (PTC) systems that can automatically reduce speed.  There was no PTC system in place at the curve in the northbound direction where the derailment occurred and experts have speculated that one would have prevented the accident. In 2008, Congress mandated nationwide installation and operation of positive train control systems by 2015.  Prior to the recent accident, the Association of America Railroads stated that more than 80 percent of the track covered by the mandate will not have functional PTC systems by the deadline. The installation of PTC systems requires a large commitment of funds and resources as well as communication bandwidth that has been difficult to secure in some area and some think the end of year deadline is unrealistic. Congress is currently considering two different bills that would address some of the issues.  The recent deadly crash is sure to be front and center in their debates.

In response to the recent accident, the Federal Railroad Administration ordered Amtrak to submit plans for PTC systems at all curves where the speed limit is 20 mph less than the track leading to the curve for the main Northeast Corridor (running between Washington, D.C. and Boston).  Only time will tell how quickly positive train control systems will be implemented on the Northeast Corridor as well as the rest of the nation, and the debate on the best course of action will not be a simple one.

An initial Cause Map, a visual root cause analysis, can be created to capture the information that is known at this time.  Additional information can easily be incorporated into the Cause Map as it becomes available.  To view a high level initial Cause Map of this accident, click on “Download PDF”.

Uncategorized

Indian Point Fire and Oil Leak

By Sarah Wrenn

At 5:50 PM on May 9, 2015, a fire ignited in one of two main transformers for the Unit 3 Reactor at Indian Point Energy Center. These transformers carry electricity from the main generator to the electrical grid. While the transformer is part of an electrical system external to the nuclear system, the reactor is designed to automatically shut down following a transformer failure. This system functioned as designed and the reactor remains shut down with the ongoing investigation. Concurrently, oil (dielectric fluid) spilled from the damaged transformer into the plant’s discharge canal and some amount was also released into the Hudson River. On May 19, Fred Dacimo, vice president for license renewal at Indian Point and Bill Mohl, president of Entergy Wholesale Commodities, stated the transformer holds more than 24,000 gallons of dielectric fluid. Inspections after the fire revealed 8,300 gallons have been collected or were combusted during the fire. As a result, investigators are working to identify the remaining 16,000 gallons of oil. Based on estimates from the Coast Guard supported by NOAA, up to approximately 3,000 gallons may have gone into the Hudson River.

The graphic located here provides details regarding the event, facility layout and response.

Step 1. Define the Problem

There are a few problems in this event. Certainly, the transformer failure and fire are major problems. The transformer is an integral component to transfer electricity from the power plant to the grid. Without the transformer, production has been halted. In addition, there is an inherent risk of injury with the fire response. The site’s fire brigade was dispatched to respond to the fire and while there were no injuries, there was a potential for injury. In addition, the release of dielectric fluid and fire-retardant foam into the Hudson River is a problem. A moat around the transformer is designed to contain these fluids if released, but evidence shows that some amounts reached the Hudson River.

As shown in the timeline and noted on our problem outline, the transformer failure and fire occurred at 5:50 PM and was officially declared out 2.25 hours later.

As far as anything out of the ordinary or unusual when this event occurred, Unit 3 had just returned to operations after a shutdown on May 7 to repair a leak of clean steam from a pipe on the non-nuclear side of the plant. Also, it was noted that this is the 3rd transformer failure in the past 8 years. This frequency of transformer failures is considered unusual. The Wall Street Journal reported that the transformer that failed earlier this month replaced another transformer that malfunctioned and caught fire in 2007. Another transformer failed in 2010, which had been in operation for four years.

Multiple organizational goals were negatively impacted by this event. As mentioned above, there was a risk of injury related to the fire response. There was also a negative impact to the environment due to the release of dielectric fluid and fire-retardant foam. The negative publicity from the event impacts the organization’s customer service goal. A notification to the NRC of an Unusual Event (the lowest of 4 NRC emergency classifications) is a regulatory impact. For production/schedule, Unit 3 was shutdown May 9 and remains shutdown during the investigation. There was a loss of the transformer which needs to be replaced. Finally, there is labor/time required to address and contain the release, repair the transformer, and investigate the incident.

Step 2. Identify the Causes (Analysis)

Now that we’ve defined the problem in relation to how the organization’s goals were negatively impacted, we want to understand why.

The Safety Goal was impacted due to the potential for injury. The risk of injury exists because of the transformer fire.

 

 

The Regulatory Goal was impacted due to the notification to the NRC. This was because of the Unit 3 shutdown, which also impacts the Production/Schedule Goal. Unit 3 shutdown as this is the designed response to the emergency. This is the designed response because of the loss of the electrical transformer, which also impacts the Property/Equipment Goal. Why was the electrical transformer lost? Because of the transformer fire.

For the other goals impacted, Customer Service was because of the negative publicity which was caused by the containment, repair, investigation time and effort. This time and effort impacts the organization’s Labor/Time Goal. This time and effort was required because of the dielectric fluid and fire-retardant foam release. Why was there a release? Because the fluid and foam were able to access the river.

Why did the fluid and foam access the river?

The fire-retardant foam was introduced because the sprinkler system was ineffective. The transformer is located outside in the transformer yard which is equipped with a sprinkler system. Reports indicate that the fire was originally extinguished by the sprinklers, but then relit. Fire responders introduced fire-retardant foam and water to more aggressively address the fire. Some questions we would ask here include why was the sprinkler system ineffective at completely controlling the fire? Alternatively, is the sprinkler system designed to begin controlling the fire as an immediate response such that the fire brigade has time to respond? If this is the case, then did the sprinkler perform as expected and designed?

The transformer moat is designed to catch fluids and was unable to contain the fluid and the foam. When a containment is unable to hold the amount of fluid that is introduced, this means that either there is a leak in the containment or the amount of fluid introduced is greater than the capacity of the containment. We want to investigate the integrity of the containment and if there are any leak paths that would have allowed fluids to escape the moat. We also want to understand the volume of fluid that was introduced. The moat is capable of holding up to 89,000 gallons of fluid. A transformer contains approximately 24,000 gallons of dielectric fluid. What we don’t know is how much fire-retardant foam was introduced. If this value plus the amount of transformer fluid is greater than the capacity of the moat, then the fluid will overflow and can access the river. If this is the case, we also would want to understand if the moat capacity is sufficient, should it be larger? Also, is the moat designed such that an overflow will result in accessing the discharge canal and is this desired?

Finally, dielectric fluid accessed the river because the fluid was released from the transformer. Questions we would ask here are: Why was the fluid released and why does a transformer contain dielectric fluid? Dielectric fluid is used to cool the transformers. Other cooling methods, such as fans are also in place. The causes of the fluid release and transformer failure is still being investigated, but in addition to determining these causes, we would also ask how are the transformers monitored and maintained? The Wall Street Journal provided a statement from Jerry Nappi, a spokesman for Entergy. Nappi said both of unit 3’s transformers passed extensive electrical inspections in March. Transformers at Indian Point get these intensive inspections every two years. Aspects of the devices also are inspected daily.

Finally, we want to understand why was there a transformer fire. The transformer fire occurred because there was some heat source (ignition source), fuel, and oxygen. We want to investigate what was the heat source – was there a spark, a short in the wiring, a static electricity build up? Also, where did the fuel come from and is it expected to be there? The dielectric fluid is flammable, but are there other fuel sources that exist?

Step 3. Select the Best Solutions (Reduce the Risk)

What can be done? With the investigation ongoing, a lot of facts still need to be gathered to complete the analysis. Once that information is gathered, we want to consider what is possible to reduce the risk of having this type of event occur in the future. We would want to evaluate what can be done to address the transformer, implementing solutions to better maintain, monitor, and/or operate it. Focusing on solutions that will minimize the risk of failure and fire. However, if a failure does occur, we want to consider solutions so that the failure and fire does not result in a release. Further, we can consider the immediate response; do these steps adequately contain the release? Identifying specific solutions to the causes identified will provide reductions to the risk of future similar events.

Resources:

This Cause Map was built using publicly available information from the following resources.

De Avila, Joseph “New York State Calls for Tougher Inspections at Indian Point” http://www.wsj.com/articles/nuclear-regulatory-commission-opens-probe-at-indian-point-1432054561 Published 5/20/2015. Accessed 5/20/2015

“Entergy’s Response to the Transformer Failure at Indian Point Energy Center” http://www.safesecurevital.com/transformer_update/ Accessed 5/19/2015

“Entergy Plans Maintenance Shutdown of Indian Point Unit 3” http://www.safesecurevital.com/entergy-plans-maintenance-shutdown-of-indian-point-unit-3/ Published 5/7/2015. Accessed 5/19/2015

“Indian Point Unit 3 Safely Shutdown Following Failure of Transformer” http://www.safesecurevital.com/indian-point-unit-3-safely-shutdown-following-failure-of-transformer/ Published 5/9/2015. Accessed 5/19/2015

“Entergy Leading Response to Monitor and Mitigate Potential Impacts to Hudson River Following Transformer Failure at Indian Point Energy Center” http://www.safesecurevital.com/entergy-leading-response-to-monitor-and-mitigate-potential-impacts-to-hudson-river-following-transformer-failure-at-indian-point-energy-center/ Published 5/13/2015. Accessed 5/19/2015

“Entergy Continues Investigation of Failed Transformer, Spilled Dielectric Fluid at Indian Point Energy Center” http://www.safesecurevital.com/entergy-continues-investigation-of-failed-transformer-spilled-dielectric-fluid-at-indian-point-energy-center/ Published 5/15/2015. Accessed 5/19/2015

McGeehan, Patrick “Fire Prompts Renewed Calls to Close the Indian Point Nuclear Plant” http://www.nytimes.com/2015/05/13/nyregion/fire-prompts-renewed-calls-to-close-the-indian-point-nuclear-plant.html?_r=0 Published 5/12/2015. Accessed 5/19/2015

Screnci, Diane. “Indian Point Transformer Fire” http://public-blog.nrc-gateway.gov/2015/05/12/indian-point-transformer-fire/comment-page-2/#comment-1568543 Accessed 5/19/2015

Root Cause Analysis - Incident Investigation

New Regulations Aim to Reduce Railroad Crude Oil Spills

By ThinkReliability Staff

The tragic train derailment in Lac-Mégantic, Quebec on July 6, 2013 (see our previous blog on this topic) ushered in new concerns about the transport of crude oil by rail in the US and Canada. Unfortunately, the increased attention has highlighted a growing problem: spills of crude oil transported via rail, which can result in fires, explosions, evacuations, and potentially deaths. (Luckily there have been no fatalities since the Lac-Mégantic derailment.) According to Steve Curwood of Living on Earth, “With pipelines at capacity the boom has lead a 4,000 percent increase in the volume of crude oil that travels by rail, and that brought more accidents and more oil spills in 2014 than over the previous 38 years.”

This follows a period of increases in railroad safety – according to the US Congressional Research Service, “From 1980 to 2012, railroads reduced the number of accidents releasing hazmat product per 100,000 hazmat carloads from 14 to 1.” From October 19, 2013 to May 6, 2015, there were at least 12 railcar derailments that resulted in crude oil spills. (To see the list of events, click on “Download PDF” and go to the second page.)

Says Sarah Feinberg, acting administrator of the Federal Railroad Administration (FRA), “There will not be a silver bullet for solving this problem. This situation calls for an all-of-the-above approach – one that addresses the product itself, the tank car it is being carried in, and the way the train is being operated.” All of these potential risk-reducing solutions are addressed by the final rule released by the FRA on May 1, 2015. (On the same day, the Canadian Ministry of Transport released similar rules.) In order to view how the various requirements covered by the rule impact the risk to the public as a result of crude oil spills from railcars, we can diagram the cause-and-effect relationships that lead to the risk, and include the solutions directly over the cause they control. (To view the Cause Map, or visual root cause analysis, of crude oil train car derailments, click on “Download PDF”.)

The product: Bakken crude oil (as well as bitumen) can be more volatile than other types of crude oil and has been implicated in many of the recent oil fires and explosions. In addition to being more volatile, the composition (and thus volatility) can vary. If a material is not properly sampled and characterized, proper precautions may not be taken. The May 1 rule incorporates a more comprehensive sampling and testing program to ensure the properties of unrefined petroleum-based products are known and provided to the DOT upon request.   (Note that in the May 6, 2015 derailment and fire in Heimdahl, North Dakota, the oil had been treated to reduce its volatility, so this clearly isn’t an end-all answer.)

The tank car: Older tank cars (known as DOT-111s) were involved in the Lac-Mégantic and other 2013 crude oil fires. An upgrade to these cars, known as CPC-1232, hoped to reduce these accidents. However, CPC-1232 cars have been involved in all of the issues since 2013. According to Cynthia Quarterman, former director of the Pipeline and Hazardous Materials Safety Administration, says that the recent accidents involving the newer tank cars “confirm that the CPC-1232 just doesn’t cut it.”

The new FRA rule establishes requirements for any “high-hazard flammable train” (HHFT) transported over the US rail network. A HHFT is a train comprised of 20 or more loaded tank cars of a Class 3 flammable liquid (which includes crude oil and ethanol) in a continuous block or 35 or more loaded tank cars of a Class 3 flammable liquid across the entire train. Tank cars used in HHFTs constructed after October 1, 2015 are required to meet DOT-117 design criteria, and existing cars must be retrofitted based on a risk-based schedule.

The way the train is being operated: The way the train is being operated includes not only the mechanics of operating the train, but also the route the train takes and the notifications required along the way. Because the risk for injuries and fatalities increases as the population density increases, the rule includes requirements to perform an analysis to determine the best route for a train. Notification of affected jurisdictions is also required.

Trains carrying crude oil tend to be very large (sometimes exceeding one mile in length). This can impact stopping distance as well as increase the risk of derailment if sudden stopping is required. To reduce these risks, HHFTs are restricted to 50 mph in all areas, and 40 mph in certain circumstances based on risk (one of the criteria is urban vs. rural areas). HHFTs are also required to have in place a functioning two-way end of train or distributed power braking system. Advanced braking systems are required for trains including 70 or more loaded tank cars containing Class 3 flammable liquids and traveling at speeds greater than 30 mph, though this requirement will be phased in over decades.

It is important to note that this new rule does not address inspections of rails and tank cars. According to a study of derailments from 2001 to 2010, track problems were the most important causes of derailments (with broken rails or track welds accounting for 23% of total cars derailed). A final rule issued January 24, 2014 required railroads to achieve a specified track failure rate and to prioritize remedial action.

To view the May 1 rule regarding updates to crude-by-rail requirements, click here. To view the timeline of incidents and the Cause Map showing the cause-and-effect relationships leading to these incidents, click “Download PDF”.

Root Cause Analysis - Incident Investigation

ISS Supply Mission Fails

By Kim Smiley

An unmanned Progress supply capsule failed to reach the International Space Station (ISS) and is expected to burn up during reentry in the atmosphere along with 3 tons of cargo.  Extra supplies are stored on the ISS and the astronauts onboard are in no immediate danger, but the failure of this supply mission is another in a string of high-profile issues with space technology.

This issue can be analyzed by building a Cause Map, a visual format of root cause analysis.  A Cause Map intuitively lays out the causes that contributed to an issue to show the cause-and-effect relationships.  To build a Cause Map, “why” questions are asked and the answers are documented on the Cause Map along with any relevant evidence to support the cause.

So why did the supply mission fail? The mission failed because the supply capsule was unable to dock with the ISS because mission control was unable to communicate with the spacecraft.  The Progress is an unmanned Russian expendable cargo  capsule that cannot safely dock with a space station without communication with mission control.  Mission control needs to be able to verify that all systems are functional after launch and needs a communication link to navigate the unmanned capsule through docking.

Images of the capsule showed that two of the five antennas failed to unfold leading to the communication issues.  Debris spotted around the capsule while it was in orbit indicates a possible explosion.  No further information has been released about what might have caused the explosion and it may be difficult to decisively determine the cause since the capsule will be destroyed in orbit.

The ISS recycles oxygen and water to an impressive degree and food is the first supply that would run out on the ISS, but NASA has stated that there are at least four months of food onboard at this time.  The failure of this mission may mean that the cargo for future missions will need to be altered to include more basic necessities and less scientific equipment, but astronaut safety is not a concern at this time. The failure of this mission does put additional pressure on the next resupply mission scheduled to be done by SpaceX in June in addition to creating more bad press for space programs that are already struggling during a turbulent time.

To view a intermediate Cause Map of this issue, click on “Download PDF” above.