Tag Archives: industrial

Marauding Monkeys Lead to Electrical Outage in Kenya

By ThinkReliability Staff

One monkey managed to cause an electrical outage for all of Kenya – 4.7 million households and businesses – for 15 minutes to more than 3 hours. In order to determine solutions to prevent this from happening again, a thorough analysis of the problem is necessary. We will look at this issue within a Cause Map, a visual form of root cause analysis.

The first step of any problem-solving method is to define the problem. In the Cause Mapping method, the problem is defined with respect to the organization’s goals. In this case, there were several goals that were impacted. If the organization has a goal of ensuring safety of animals, that goal is impacted due to the risk of a fatality or severe injury to the monkey. (In this case, the monkey was unharmed and was turned over to the wildlife service.) The loss of power to 4.7 million businesses and households is an impact to the customer service goal. The nationwide power outage, which lasted from 15 minutes to over 3 hours, is an impact to the production/ schedule goal. Damage to the transformer is an impact to the property goal, and the time required for response and repair is an impact to the labor/ time goal.

The second step of problem-solving is the analysis. Using the Cause Mapping method, cause-and-effect relationships are developed. One of the impacted goals is used as the first effect. Asking “Why” questions is one way to determine cause-and-effect relationships. However, there may be more than one cause required to produce an effect. In this example, the power outage resulted from a cascading effect on the country’s generators. This cascading effect was caused by the loss of a hydroelectric facility, which provides 20% of the country’s electricity, and the unreliability of the power grid, due to aging infrastructure. All of these causes were required for this scenario: had the country had a more reliable power grid or more facilities so that the country was not so dependent on one, the loss of the hydroelectric site would not have resulted in nationwide outage.

Continuing the analysis, the loss of the hydroelectric facility was caused by an overload when a key transformer at the site was tripped. According to the power company, the trip was caused by a monkey falling onto the transformer. (There is also photographic evidence showing a monkey in the area of the transformer.) In order for the monkey to fall onto the transformer, it had to be able to access the transformer. The monkey in this case is believed to have fallen off the roof. How this occurred is still unclear, because the facility is secured by an electric fence designed specifically for protection against “marauding wild animals”.

The last step of problem-solving is to determine solutions, based on the analysis of this problem. The utility says it is “looking at ways of further enhancing security” at all their power plants. Unfortunately, total protection against outages caused by animals is impossible. In the United States, animal-caused outages are believed to cause at least $18 billion in lost economy every year. Just this May, raccoons caused outages to 40,000 in Seattle and 5,600 in Colorado Springs. This year also saw outages caused by squirrels, snakes, starlings and geese. Other unusual outages include work on a transformer causing an outage with economic loss of $118 million in Arizona (see our blog on this subject) and a woman with a shovel who cut internet service to nearly all of Armenia (see our blog on this subject).

Because power outages due to animals and other issues can’t be completely eliminated, ensuring a robust power grid is important to minimize the impact from and duration of outages. Calls for improvements to the aging infrastructure in Kenya have resulted from this incident, but these kinds of solutions require not only the cooperation of the utilities, but the country as a whole.

To view the problem outline and Cause Map for this incident, please click on “Download PDF” above

Worker dies while manually measuring tank

By Kim Smiley

The potential danger of confined spaces is well documented, but nine fatalities have shown that people working near open hydrocarbon storage hatches can also be exposed to dangerous levels of hydrocarbon gases and oxygen-deficient atmospheres.  NPR recently highlighted this issue in an article entitled “Mysterious Death Reveals Risk In Federal Oil Field Rules” that discussed the death of Dustin Bergsing.  His job duties included opening the hatch on a crude oil storage tank to measure the level of the oil and was found dead next to an open hatch.  He was healthy and only 21 years old.

A Cause Map, a visual format for performing a root cause analysis, can be used to help explain what happened to cause his death.  A Cause Map intuitively lays out the cause-and-effect relationships that contributed to an issue and is built by asking “why” questions.  Click on “Download PDF” to view a high level Cause Map of this accident.

So why did his death occur?  An autopsy showed that his death occurred because he had hydrocarbons in his blood.  This occurred because he was exposed to hydrocarbon vapor and he remained in the dangerous environment. (When two causes both contribute to an effect, they are listed vertically on the Cause Map and separated by an “and”.)

When a person is exposed to hydrocarbon vapor, they get disoriented before passing out so it is very difficult for them to get to safety on their own.  Bergsing was working alone at the time of his death and no one was aware that he was in trouble before it was too late.

He was exposed to hydrocarbon gases because he opened a hatch on a crude oil storage tank and the gas had collected at the top of the tank.  He opened the hatch because he planned to manually measure the tank level by dropping a rope inside. Manual tank measurement is a common method to determine level in crude oil storage tanks. Crude oil contains volatile hydrocarbons that can bubble out of the crude oil and collect at the top; the gas will rush out of the tank if a hatch is opened.

Additionally, he wasn’t wearing adequate PPE equipment because it wasn’t required by any regulations and there was limited awareness of this danger.

After his and the other deaths, the industry is starting to become more aware of this issue.  The National Institute for Occupational Safety and Health (NIOSH) and the Occupational Safety and Health Administration (OSHA) issued a hazard alert bulletin that identified health and safety risks to workers who manually gauge or sample fluids on production and flowback tanks from exposure to hydrocarbon gases and vapors and exposure to oxygen-deficient atmospheres. In addition to working to raise awareness of the issue, OSHA and NIOSH made recommendations to improve working safety that include the following:

– Implementing alternate procedures that allow workers to monitor tank levels and sample without opening hatches

– Installing hatch pressure indicators

– Conducting worker exposure assessments

– Providing training on the hazard and posting hazard signage

– Not permitting employees to work alone

Please read the OSHA and NIOSH hazard alert bulletin for more information and a full list of the recommendations. Many of the recommendations would be expensive and time-consuming to implement, but some may be relatively simple ways to reduce risk. Continuing to provide information to workers about the potential hazards might be a good first step to improve their safety.

Landslide of construction debris buries town, kills dozens

By ThinkReliability Staff

Shenzhen, China has been growing fast. After a dump site closed in 2013, construction debris from the rapid expansion was being dumped everywhere. In an effort to contain the waste, a former rock quarry was converted to a dump site. Waste at the site reached 100 meters high, despite environmental assessments warning about the potential for erosion. On December 20, 2015, the worries of residents, construction workers and truckers came true when the debris slipped from the quarry, covering 380,000 square meters (or about 60 football fields) with thick soil as much as 4 stories high.

A Cause Map can be built to analyze this issue. One of the steps in the Cause Mapping process is to determine how the issue impacted the overall goals. In this case, the landslide severely impacted multiple goals. Primarily, the safety goal was impacted due to a significant number of deaths. 58 have been confirmed dead, and at least 25 are missing. The environmental goal and customer service goal were impacted due to the significant area covered by construction waste. The regulatory goal is impacted because 11 have been detained as part of an ongoing criminal investigation. The property goal is impacted by the 33 buildings that were destroyed. The labor goal is also impacted, as are more than 10,600 people participating in the rescue effort.

The Cause Map is built by visually laying out the cause-and-effect relationships that contributed to the landslide. Beginning with the impacted goals and asking “Why” questions develops the cause-and-effect relationships. The deaths and missing persons resulted from being buried in construction waste. Additionally, the confusion over the number of missing results from the many unregistered migrants in the rapidly growing area. The area was buried in construction waste when waste spread over a significant area, due to the landslide.

The landslide resulted from soil and debris that was piled 100 meters high, and unstable ground in a quarry. The quarry was repurposed as a waste dump in order to corral waste, which had previously been dumped anywhere after the closure of another dump. Waste and debris was piled so high because of the significant construction debris in the area. There was heavy construction in the area because of the rapid growth, resulting in a lot of debris. Incentives (dumpsite operators make money on each load dumped) encourage a high amount of waste dumping. Illegal dumping also adds to the total.

While an environmental impact report warned of potential erosion, and the workers and truck drivers at the dump registered concerns about the volume of waste, these warnings weren’t heeded. Experts point to multiple recent industrial accidents in China (such as the warehouse fire/ explosion in Tianjin in August, the subject of a previous blog) as evidence of the generally lax enforcement of regulations. Heavy rains contributed to ground instability, as did the height of the debris, and the use of the site as a quarry prior to being a waste dump.

Actions taken in other cities in similar circumstances include charging more for dumping debris in an effort to encourage the reuse of materials and monitoring dump trucks with GPS to minimize illegal dumping. These actions weren’t implemented in Shenzhen prior to the landslide, but this accident may prompt their implementation in the future. Before any of that can happen, Shenzhen has a long way to go cleaning up the construction debris covering the city.

Waste Released from Gold King Mine

By Renata Martinez

On August 5, 2015 over 3 million gallons of waste was released from Gold King Mine into Cement Creek which then flowed into the Animas River. The orangish colored plume moved over 100 miles downstream from Silverton, Colorado through Durango reaching the San Juan River in New Mexico and eventually making its way to Lake Powell in Utah (although the EPA stated that the leading edge of the plume was no longer visible by the time it reached Lake Powell a week after the release occurred).

Some of the impacts were immediate.  No workers at the mine site were hurt in the incident but the collapse of the mine opening and release of water can be considered a near miss because there was potential for injuries. After the release, there were also potential health risks associated with the waste itself since it contained heavy metals.

Water sources along the river were impacted and there’s potential that local wells could be contaminated with the waste.   To mitigate the impacts, irrigation ditches that fed crops and livestock were shut down.  Additionally, the short-term impacts include closure of the Animas River for recreation (impacting tourism in Southwest Colorado) from August 5-14.

The long-term environmental impacts will be evaluated over time, but it appears that the waste may damage ecosystems in and along the plume’s path. There are ongoing investigations to assess the impact to wildlife and aquatic organisms, but so far the health effects from skin contact or incidental ingestion of contaminated river water are not considered significant.

“Based on the data we have seen so far, EPA and the Agency for Toxic Substances and Disease Registry (ATSDR) do not anticipate adverse health effects from exposure to the metals detected in the river water samples from skin contact or incidental (unintentional) ingestion. Similarly, the risk of adverse effects to livestock that may have been exposed to metals detected in river water samples from ingestion or skin contact is low. We continue to evaluate water quality at locations impacted by the release.”

The release occurred when the EPA was working to stabilize the existing adit (a horizontal shaft into a mine which is used for access or drainage). The force of the weight of a pool of waste in the mine overcame the strength of the adit, releasing the water into the environment.  The  EPA’s scope of work at Gold King Mine also included assessing the ongoing leaks from the mine to determine if the discharge could be diverted to retention ponds at the Red and Bonita sites.

The wastewater had been building up since the adit collapsed in 1995.  There are networks and tunnels that allow water to easily flow between the estimated 22,000 mine sites in Colorado.  As water flows through the sites it reacts with pyrite and oxygen to form sulfuric acid.  When the water is not treated and it contacts (naturally occurring) minerals such as zinc, lead, cadmium, copper and aluminum and breaks down the heavy metals, leaving tailings.  The mines involved in this incident were known to have been leaking waste for years.  In the 90s, the EPA agreed to postpone adding the site to the Superfund National Priorities List (NPL), so long as progress was made to improve the water quality of the Animas River.  Water quality improved until about 2005 at which point it was re-assessed.  Again in 2008, the EPA postponed efforts to include this area on the NPL.  From the available information, it’s unclear if this area and the waste pool would have been treated if the site was on the NPL.

In response, the “EPA is working closely with first responders and local and state officials to ensure the safety of citizens to water contaminated by the spill. ” Additionally, retention ponds have been built below the mine site to treat the water and continued sampling is taking place to monitor the water.

So how do we prevent this from happening again?  Mitigation efforts to prevent the release were unsuccessful.  This may have been because the amount of water contained in the mine was underestimated.  Alternatively, if the amount of water in the mine was anticipated (and the risk more obvious) perhaps the excavation work could have been planned differently to mitigate the collapse of the tunnel.  As a local resident, I’m especially curious to learn more facts about the specific incident (how and why it occurred) and how we are going to prevent this from recurring.

The EPA has additional information available (photos, sampling data, historic mine information) for reference: http://www2.epa.gov/goldkingmine

Explosions raise concern over hazardous material storage

By ThinkReliability Staff

On August 12, a fire began at a storage warehouse in Tianjin, China. More than a thousand firefighters were sent in to fight the fire. About an hour after the firefighters went in, two huge explosions registered on the earthquake measurement scale (2.3 and 2.9, respectively). Follow-on explosions continued and at least 114 firefighters, workers and area residents have been reported dead so far, with 57 still missing (at this point, most are presumed dead).

Little is known for sure about what caused the initial fire and continuing explosions. What is known is that the fire, explosions and release of hazardous chemicals that were stored on site have caused significant impacts to the surrounding population and rescuers. These impacts can be used to develop cause-and-effect relationships to determine the causes that contributed to an event. It’s particularly important in an issue like this – where so many were adversely affected – to find effective solutions to reduce the risk of a similar incident recurring in the future.

Even with so much information unavailable, an initial root cause analysis can identify many issues that led to an adverse event. In this case, the cause of the initial fire is still unknown, but the site was licensed to handle calcium carbide, which releases flammable gases when exposed to water. If the chemical was present on site, the fire would have continued to spread when firefighters attempted to fight it using water. Contract firefighters, who are described as being young and inexperienced, have said that they weren’t adequately trained for the hazards they faced. Once the fire started, it likely ignited explosive chemicals, including the 800 tons of ammonium nitrate and 500 tons of potassium nitrate stored on site.

Damage to the site released those and other hazardous chemicals. More than 700 tons of sodium cyanide were reported to be stored at the site, though it was only permitted 10 tons at a time. Sodium cyanide is a particular problem for human safety. Says David Leggett, a chemical risk consultant, “Sodium cyanide is a very toxic chemical. It would take about a quarter of teaspoon to kill you. Another problem with sodium cyanide is that it can change into prussic acid, which is even more deadly.”

But cleaning up the mess is necessary, especially because there are residents living within 2,000 ft. of the site, despite regulations that hazardous sites are a minimum of 3,200 ft. away from residential areas. Developers who built an apartment building within the exclusion zone say they were told the site stored only common goods. Rain could make the situation worse, both by spreading the chemicals and because of the potential that the released chemicals will react with water.

The military has taken over the response and cleanup. Major General Shi Luze, chief of the general staff of the military region, said, “After on-site inspection, we have found several hundred tons of cyanide material at two locations. If the blasts have ripped the barrels open, we neutralize it with hydrogen peroxide or other even better methods. If a large quantity is already mixed with other debris, which may be dangerous, we have built 1-meter-high walls around it to contain the material — in case of chemical reactions if it rains. If we find barrels that remain intact, we collect them and have police transport them to the owners.”

In addition to sending in a team of hazardous materials experts to neutralize and/or contain the chemicals and limiting the public from the area in hopes to limit further impact to public safety, the state media had said they were trying to prevent rain from falling, presumably using the same strategies developed to ensure clear skies for the 2008 Summer Olympics. Whether it worked or not hasn’t been said, but it did rain on August 18, nearly a week after the blast, leaving white foam that residents have said creates a burning or itchy sensation with contact.

View an initial Cause Map of the incident by clicking on “Download PDF” above.

Fatal Bridge Collapse Near Cincinnati

By Kim Smiley

On the evening of January 19, 2015, an overpass on Interstate 75 near Cincinnati collapsed, killing one and injuring another.  The overpass was undergoing construction when it unexpectedly collapsed onto the road below it, which was still open to traffic.

This incident can be analyzed by building a Cause Map, a visual root cause analysis, to intuitively lay out the many causes that contributed to an accident by showing the cause-and-effect relationships.  Understanding all the causes that played a role, as opposed to focusing on a single root cause, expands the potential solutions that can be considered and can lead to better problem prevention.  A Cause Map is built by asking “why” questions and documenting the answers.

In this example, a construction worker was operating an excavator on the overpass when it collapsed.  When the bridge collapsed the worker was crushed by the steel beams he was moving.   The additional weight of evacuator and steel beams on the overpass likely contributed to the collapse.   The overpass was being demolished as part of a project to remake this section of the Interstate and a portion of the overpass had already been removed.  The work that had been done appears to have made the structure of the bridge unstable, but the construction company was not aware of the potential danger so the worker was operating on top of the overpass and the road beneath it was still open to traffic.

A truck driver traveling under the overpass at the time of collapse suffered only minor injuries, but came within inches of being crushed by the bridge. It really was simple luck that no other vehicles were involved.  Had the collapse happened earlier in the day when there was more traffic, the number of fatalities may very well have been higher.  As investigators review this accident, one of the things they will need to review is the fact that the road below the bridge was open to traffic at the time of the collapse.  An additional relevant piece of information is that the construction company had financial incentives to keep the road open as much as possible because they would be fined for any amount of time that traffic was disrupted.

In addition to the safety impacts of this accident, the overpass collapse dramatically impacted traffic on a busy road with an estimated 200,000 vehicles traveling on it daily.  It took nearly a day to get all lanes of the interstate cleaned up and reopened to traffic.  No one wants to close roads unnecessarily and the goal of minimizing traffic is an excellent one, but it has to be balanced with safety.  The collapse of the overpass wasn’t an unforeseeable random accident and the demolition needs to be done in a safe manner.

Failure of the Teton Dam in 1976

by ThinkReliability Staff

On June 5, 1976, workers were called to the Teton Dam on the Teton River in Idaho to attempt to repair a leak.  Workers in bulldozers narrowly avoided being sucked into the dam with their equipment, and watched helplessly as the dam was breached.  It would kill 14 people and cause nearly hundreds of millions of dollars in property and environmental damage.  To examine what went wrong, we can perform a visual root cause analysis, or Cause Map.

The Cause Mapping process begins by determining the impacts to an organization’s goals.  From the perspective of the government, specifically the Bureau of Reclamation, the safety goal is impacted because of the 14 deaths.  The environmental goal is impacted due to the severe impact the dam failure and subsequent flooding had on the ecosystem of the area.  The customer service goal was impacted due to the evacuation of three towns.  The production goal is impacted due to the abandonment of the dam – at a cost of approximately $50 million.  Additionally, property damage of at least $400 million (some estimates are much higher) is an impact to the property goal.  (There were also substantial claims related to the loss of property and livelihood from impacts to industries, particularly fishing.)

Once we have determined the impacts to the goals, we begin with an impacted goal, such as the safety goal, and ask “Why” questions to determine the cause-and-effect relationships that led to the impacted goals (also known as “problems”.)  In the case of the Teton Dam failure, people were killed due to a massive wave of water released from the dam (which was filled to capacity) when it failed.  The dam failure was also the cause of severe damage to the dam, which was never rebuilt, leading to the impacted production goal.

The failure of the dam was found to be caused by erosion and inadequate strength.  Due to the less than ideal geological conditions of the site (which was picked because there were no “good” sites available), unequal stress distribution and inadequate fill material (which was used from the site) led to reduced strength.  Susceptible materials and seepage from leaks in the embankment, caused by joints that were not resistant to water pressure due to inadequate testing, and inadequate protection from water due to an over-reliance on an ineffective curtain intended to stop flow, led to the erosion.

Many geologists had predicted problems with the dam before it was built.  Specifically, in his book “Normal Accidents”, Charles Perrow states “The Bureau ignored its own data that rocks in the area were full of fissures, and in addition they filled the dam too fast . . . All it takes to bring a dam down is one crack, if that crack wets the soil within the interior portions of the dam, turning it into a quagmire.”

Although tragic, and expensive, the failure of the Teton Dam did lead to many reforms in the Bureau of Reclamation, who is responsible for dam safety.  Detailed geological studies performed in order to determine the causes of the dam failure also provided additional insight to the strength provided by various types of earth, erosion and seepage.

Contaminated Water Issues Remain at Fukushima

by ThinkReliability Staff

High levels of contaminated water leaving the highly damaged reactors at the Fukushima Daiichi nuclear power plant in Japan are creating issues for the personnel on site, who are working frantically to keep the reactor safe and working towards decommissioning and closing down the site.  Additionally, there is continued concern for the ongoing safety of the site, as the high volume of water could potentially threaten the safety of the reactors.

We can look at these issues in a Cause Map, or visual root cause analysis.  With a Cause Map, the first step is to determine how the issue impacts the organization’s goals.  In this case, we can consider the goals from the perspective of the utility company that owns the power plant.  There is an impact to the safety goal because of the potential risk for another accident, according to the Chairman of the Nuclear Regulatory Authority.  The leakage of contaminated water is an impact to the environmental goal.  There is concern about the lack of a comprehensive plan by the utility, which can be considered an impact to the customer service goal.  The massive construction efforts required to install tanks to store the water are an impact to the property goal and the efforts by the workers to control the flow are an impact to the labor goal.

Once the impacts to the goals have been determined, the next step is asking “Why” questions to determine the cause-and-effect relationships that led to the incident.  In this case, the issues resulting in the high rate of contaminated water needing to be stored are that high rates of water are entering the reactor, becoming contaminated due to the damage inside the buildings from the earthquake and tsunami on March 11, 2011, and the water has to be removed from the building.

The water is entering the buildings because the plant is in the groundwater flow path from the mountains to the ocean and there is insufficient protection to prevent the water accessing the plant.   Severe cracking in the reactor buildings from the earthquake/tsunami are unable to be repaired due to the high residual levels of radioactivity.  The utility rejected plans to build a wall to protect the reactor.  It is believed this is because the utility had planned to dump the water into the ocean.   Additionally, according to the Japan Atomic Energy Commission, the issues from the water weren’t something that were thought of, as the focus was on the nuclear issues.  All involved in the cleanup, including the utility, have had their hands full, so it’s likely something as benign-seeming as water just wasn’t on the list of immediate concerns.

The contaminated water must be pumped out of the building to avoid swamping the cooling systems, which are still needed to remove decay heat that continues to be produced even after the reactors are shut down.  It appears that the original plan was to filter the water and dump it into the ocean, but even after filtering, a high level (about one hundred times the level released from a healthy plant) of tritium would remain.  Public outcry has ended the possibility of being able to dispose of the water that way.  Wastewater pits originally built for this purpose were found last month to be leaking, necessitating the installation of hundreds of tanks for water storage.

For now, the utility workers continue to install tanks to hold the radioactive water.  The task is so overwhelming, it’s not clear if there are any other plans to try and slow the tide of contaminated water.  However, outside experts are attempting to provide assistance.  The International Atomic Energy Agency completed its initial review of the decommissioning plans last month.  The final team report is expected later this month.

To view the Outline and Cause Map, please click “Download PDF” above.  Or click here to read more.

Deadly Explosion at Texas Fertilizer Plant

By ThinkReliability Staff

An explosion at a fertilizer plant in West, Texas, destroyed much of the town and killed between 5-15 people.   (Search and rescue is still ongoing.)  At least 160 were injured but that number may increase.  The material involved in the explosion was ammonium nitrate, a popular fertilizer.

Capturing the impacts to the goals as a result of an issue is essential to understanding the true effect.  In this case, the fatalities and injuries were severe.  The property damage, which included the plant, as well as the homes of more than 100 families, was also extensive.  An environmental impact resulted from the release of ammonia, which is a respiratory irritant. There was some level of evacuation, which can be considered an impact to the customer service goal, though the high number of injuries has led some to believe the evacuation was not widespread enough.  Additionally, ongoing search and rescue, and firefighting operations are an impact to the labor goal.

These goals were all impacted due to the explosion at the fertilizer plant.  Ammonium nitrate can explode when ignited at very high temperatures.  In this case, a fire provided the high heat.  We can capture these causes in a Cause Map, or a visual form of root cause analysis.  The cause of the fire itself is as yet unknown, though if that is determined we can add it to the Cause Map as well.

What is known is that efforts to prevent explosion were ineffective.  The plant did not believe that an explosion was possible.  Its internal safety review had a worst-case scenario of a ten-minute ammonia release, causing no injuries.  It is fairly rare that ammonia nitrate explodes; only 17 known cases of unintended ammonia nitrate explosions resulting in fatalities have occurred since 1921.  Firefighters were on scene fighting the fire when the explosion occurred, leading to many responder fatalities and injuries.  Oversight at the facility was limited; OSHA has not inspected the facility for at least the last five years.

It is worth exploring why large amounts of ammonium nitrate were present.  Ammonium nitrate is an inexpensive, effective fertilizer.  It is particularly good at delivering nitrogen to food-bearing plants, like fruit trees.  The use of nitrogen greatly increases the yield of food from these plants.  (It is said to increase the carrying capacity, or number of people who can be supported by a hectare of land – from 1.9 to 4.3.)  Given the shortage of food-growing land, this is certainly important.   However, the benefits must be considered alongside the risk and certainly in the future more oversight of these types of facilities may be needed to protect the public from the process as they benefit from the results.

To view the Outline and Cause Map, please click “Download PDF” above.

The Super Bowl Blacks Out in New Orleans

By Kim Smiley

The Super Bowl is always one of the most talked about television events of the year and this year the game was even more interesting than usual.  An impressive comeback attempt following a game delaying blackout made this one to remember.

The question of what caused the highly publicized blackout can be analyzed by building a Cause Map, an intuitive, visual format for performing a root cause analysis.  The first step in building a Cause Map is to fill in an Outline with the background information for the issue.  The goals that are impacted by an issue are listed on the bottom of the Outline.  In this example, the schedule goal is impacted because the Super Bowl was delayed; the material goal is impacted because a component called an electrical relay device needs to be replaced; and the customer service goal was impacted because the delay changed the momentum of the game significantly.    Individual fans may disagree, but the companies who have profits impacted by the Super Bowl probably consider the momentum shift a pleasant side effect of the blackout since the last 17 minutes of this game were the most watched.  Once the Outline is complete, the Cause Map is built by asking “why” questions.

Starting with the schedule goal, the next step would be to ask “why” the Super Bowl was delayed.    This happened because the game wasn’t able to be played because of a partial loss of power.  The electrical company has announced that a component called an electrical relay device failed, but the exact reason it failed hasn’t been determined.   Another cause that can be added to the Cause Map is that the backup power was insufficient to power the whole Stadium.  This cause is worth considering because a possible solution to this problem could be to add a more robust back up system to mitigate any future power issues.

The relay had been installed during major system upgrades that were performed during the previous two years to ensure that the stadium was ready for the demands of hosting the Super Bowl.  The relay was added to protect the Superdome electrical equipment if there was a cable failure between the incoming power lines (operated by the electric company) and the lines that run through the stadium.

This power problem is still being reviewed and it is still being determined if an independent review of the issue is necessary.  Once more facts are known, they can be easily incorporated into the Cause Map.  The final step in the Cause Mapping process would be to develop solutions that would help mitigate the issue and prevent future power failures.

See more power outage cause maps:

The Costa Allegra Loses Power

Power Outage Stretches from Arizona to California

Chile Power Outage 

Want us to cause map a specific power outage for you? Contact us at  info@thinkreliability.com and we’ll give you a “lights out” root cause analysis.