Tag Archives: power outage

Root Cause Analysis - Incident Investigation

Ice Bucket Challenge Ends in Serious Injuries

By Kim Smiley

In a terrible reminder that awful things can happen at any time, two firefighters were seriously injured helping the Campbellsville University’s marching band raise money for amyotrophic lateral sclerosis (ALS) research by participating in the trendy ice bucket challenge.  If you ever log onto Facebook, you are probably already familiar with the concept behind the ice bucket challenge, but in case you are not a social media fan, the idea behind the ice bucket challenge is that friends tag each other to either donate $100 to an ALS-related charity  or dump a bucket of ice water over their head.  If you choose the ice bucket, you are supposed to take a video or photo as evidence and post it online.

Trying to create an entertaining video of the ice bucket dumping is part of the fun for many of the participants.  In order to make a memorable video to post on social media, the firefighters that were injured used a fire truck ladder to dump ice water on the band from above.  While on the ladder, the firefighters were near high voltage power lines (although they never actually touched the lines) and electricity arced out, injuring four firefighters.  Two firefighters were treated and released, but two were still hospitalized days later.  One was listed as stable, but the other was in critical condition.

This accident clearly illustrates that high voltage can be extremely dangerous even if you don’t touch the equipment. An arc flash can occur when a flashover of electric current leaves its intended path and travels through the air from one conductor to another or to the ground.  The closer a person is when an arc happens, the more dangerous it is.  Arcs are exceptionally hot and can cause very serious injuries and even death from several feet away when high voltage is in use.

The Public Service Commission stated that they will investigate the location to ensure that the power line had the correct clearance from the ground, trees and structures, but initial reports do not indicate any problems with the power poles.  Possible solutions that could be used to reduce the risk of a similar problem in the future are increased education on the risks of high voltage and ensuring that adequate warning signs are in place.

These have been the most dramatic injuries associated with the ice bucket challenge, but there are a slew of videos featuring buckets dropped on heads, slips and a variety of other unintended outcomes that look painful.  If you are considering doing the ice bucket challenge, please remember that a gallon of water weighs over 8 pounds.  A five gallon bucket filled with water is pretty heavy.  Think the plan through carefully before you ask somebody to dump water on you off a balcony because it may end badly.

Root Cause Analysis - Incident Investigation

Poisoned Dead Mice Parachuted onto Guam to Kill Snakes

By Kim Smiley

On December 1, 2013, 2,000 dead, poisoned neonatal mice were parachuted onto Guam on a unique mission to fight an invasive species, the brown tree snake. The parachutes are designed to catch in the trees and tempt the snakes, who live in the trees, into eating the mice. The mice are pumped full of acetaminophen, a chemical that the snakes are particularly sensitive to because it affects their blood’s ability to carry oxygen.

There are an estimated 2 million brown tree snakes on Guam so the 2,000 poisoned mice will only impact a very small percentage of the population, but scientists hope that the information they learn from this drop will help them plan larger mice drops in the future.  This is the fourth and largest dead mice drop so far and cost 8 million dollars.  Some of the mice were embedded with data-transmitting radios for this drop which will allow scientists to better gauge the effectiveness of this technique.

While the 8 million dollar price tag sounds high, it’s important to realize that the damage done by the brown tree snakes each year is significant.  Since their accidental introduction to the island, brown tree snakes have destroyed the native ecosystem, decimating the native bird population.  Brown tree snakes are also fantastic climbers and they routinely get into electrical equipment.  They cause an average of 80 power outages a year, resulting in costs as high as $4 million for repairs and lost productivity annually. (See our previous blog for more information.)

Even through the problem of the brown tree snakes is fairly well understood, an effective solution has been difficult to find.  There have been a number of different things tried over the years: snake traps, snake-sniffing dogs and snake-hunting inspectors have all been used, but the snakes have completely over un the island.  As farfetched as it sounds, parachuting dead mice seems to be the most promising solution at present.  It works because the snakes are very sensitive to acetaminophen; they only need to ingest about one-sixth of a standard pill for it to be effective.  This means that non-target animals are unlikely to be heavily impacted by the mice drops.  A pig or dog would need to eat around 500 of the baited mice for the dose to be lethal. One of the concerns is that snakes tend to avoid prey that is already dead, but information from the radio transmitters used in the recent drop should confirm if the mice are an effective bait.

One thing I know for sure, I would have loved to be in the brainstorm meeting the first time someone suggested parachuting dead mice.  This example is a good reminder to all of us to keep an open mind.  Every now and then, the most bizarre solution suggested turns out to be the best.

Root Cause Analysis - Incident Investigation

The Super Bowl Blacks Out in New Orleans

By Kim Smiley

The Super Bowl is always one of the most talked about television events of the year and this year the game was even more interesting than usual.  An impressive comeback attempt following a game delaying blackout made this one to remember.

The question of what caused the highly publicized blackout can be analyzed by building a Cause Map, an intuitive, visual format for performing a root cause analysis.  The first step in building a Cause Map is to fill in an Outline with the background information for the issue.  The goals that are impacted by an issue are listed on the bottom of the Outline.  In this example, the schedule goal is impacted because the Super Bowl was delayed; the material goal is impacted because a component called an electrical relay device needs to be replaced; and the customer service goal was impacted because the delay changed the momentum of the game significantly.    Individual fans may disagree, but the companies who have profits impacted by the Super Bowl probably consider the momentum shift a pleasant side effect of the blackout since the last 17 minutes of this game were the most watched.  Once the Outline is complete, the Cause Map is built by asking “why” questions.

Starting with the schedule goal, the next step would be to ask “why” the Super Bowl was delayed.    This happened because the game wasn’t able to be played because of a partial loss of power.  The electrical company has announced that a component called an electrical relay device failed, but the exact reason it failed hasn’t been determined.   Another cause that can be added to the Cause Map is that the backup power was insufficient to power the whole Stadium.  This cause is worth considering because a possible solution to this problem could be to add a more robust back up system to mitigate any future power issues.

The relay had been installed during major system upgrades that were performed during the previous two years to ensure that the stadium was ready for the demands of hosting the Super Bowl.  The relay was added to protect the Superdome electrical equipment if there was a cable failure between the incoming power lines (operated by the electric company) and the lines that run through the stadium.

This power problem is still being reviewed and it is still being determined if an independent review of the issue is necessary.  Once more facts are known, they can be easily incorporated into the Cause Map.  The final step in the Cause Mapping process would be to develop solutions that would help mitigate the issue and prevent future power failures.

See more power outage cause maps:

The Costa Allegra Loses Power

Power Outage Stretches from Arizona to California

Chile Power Outage 

Want us to cause map a specific power outage for you? Contact us at  info@thinkreliability.com and we’ll give you a “lights out” root cause analysis.

Root Cause Analysis - Incident Investigation

Power Outage Stretches from Arizona to California

By ThinkReliability Staff

On September 8, 2011, work on a fault capacitor in Arizona began a series of events that resulted in the worst power outage in the Southwest for 15 years.  Although there were no injuries reported as a result of the power outage, there was a high potential for injuries and/or deaths, as hospitals shut down and at least one airport lost runway lighting.  Raw sewage leaked onto beaches and millions found themselves without power.  The economic losses from this incident are reported to be as high as $118 million.  The Federal Energy Regulatory Commission (FERC) will be conducting an investigation to determine how simple capacitor work resulted in an incident with such extreme effects.

The issues related to this power outage are complicated, and can be more clearly understood in a visual format, such as a Cause Map.  We can examine the cause-and-effect relationships that resulted in the impacted goals discussed above.  The potential for injury was caused by a loss of electrical power to hospitals and airports.   The loss of power was caused by a grid crash, resulting from insufficient power and high demand (at least partially due to a heat wave).  Power stations that normally provide electricity were automatically shut down when a current reverse (normally the current runs from Arizona to California) resulted from the loss of a transmission line resulting from the capacitor work.  Although “operator error” has been mentioned as a potential cause, it’s undesirable that one operator’s error could cause such an extreme power outage.  The system should be designed to prevent this, and the investigation will hopefully address issues in the system that contributed to the extent of the outage.

In addition to losing power stations, insufficient base-load capacity in the area (long a source of concern) meant that standby plants could not be brought up fast enough to prevent the crashing of the grid.  Also, renewable wind and solar energy sources weren’t much help due to less than ideal weather conditions for production (cloudy with low wind).

The FERC’s investigation will determine causes that contributed to this power outage and will provide recommendations to limit these types of incidents in the future.  Specifically, they will determine what allowed a simple capacitor issue to result in an extensive power outage and will also consider the grid stability in the area.  However, in the meantime, some individual businesses discovered a boon in having their own generators.  Additionally, U.S. Navy ships in port in San Diego used their generators to supply power to the grid.  While these actions certainly helped lessen the effect of the outage (and brought in a lot of business to locations that did have generators), broader improvements are needed to prevent these types of issues in the future.

To view the Outline and Cause Map, please click “Download PDF” above.

Root Cause Analysis - Incident Investigation

Potential Power of Solar Flares

By Kim Smiley

The largest solar flare in recorded history occurred on September 1, 1859.  As the energy released from the sun hit the earth’s atmosphere, the skies erupted in a rainbow of colored auroras that were visible as far south as Jamaica and Hawaii.  The most alarming consequences of this “Carrington Event” (named for solar astronomer Richard Carrington who witnessed it) were its effect on the telegraph system. Operators were shocked and telegraph paper caught fire.

No solar flares approaching the magnitude of the Carrington Event have occurred since, but the question must be asked – What if a similarly sized solar flare happened today?

There is some debate on how severe the consequences would be, but the bottom line is that modern technology would be significantly impacted by a large solar flare.  When large numbers of charged particles bombard the earth’s atmosphere (as occurs during a large solar flare), the earth’s magnetic field is deformed.  A changing magnetic field will induce current in wires that are inside it resulting in large currents in electrical components within the earth’s atmosphere during a solar fare.

Satellites would likely malfunction, taking with them wireless communication, GPS capabilities and other technologies.  This would severely impact the modern world, but the largest impact would likely be to the power grid.  There is debate on how long power would be out and how severe the damage is, but it is clear that solar flares have the ability to significantly damage the power grid.  Solar flares much smaller than the Carrington Event have caused blackouts, but power was returned relatively quickly.  One of the more impressive of these examples occurred in 1989 when the entire province of Quebec lost power for about 12 hours. (Click here to read more.)

NASA works to predict and monitor sun activity so that preventive actions can be taken to help minimize damage if a large solar flare occurs.  For example, portions of the power grid could be shut down to help protect against overheating.  Scientists continue to study the issue, working to improve predictions for sun flare activity and learn how to better protect technology from them.  Click the “Download PDF” button above to view a high level Cause Map, a visual root cause analysis, built for this issue.

More information can be found in a report by the National Academy of Sciences, Severe Space Weather Events–Understanding Societal and Economic Impacts and the NASA website.

Root Cause Analysis - Incident Investigation

Power Outage Chile

By ThinkReliability Staff

A power outage struck Chile less than a month after an earthquake struck.  The power outage affected an area of nearly 2,000 kilometers and roughly 80% of Chile’s population.  Power in most areas was restored within several areas.  However, it was estimated that power to some in the Bio Bio region – which received more severe infrastructure damage – might be out for the better part of a week.

A power outage is an impact to the customer service and production/schedule goal.  The power outage was caused by the collapse of the Central Interconnected System (Sistema Interconectado Central).  The grid collapse was due to a lack of backup power capabilities, which was caused by a fragile power grid as a result of the earthquake, and interruption to the main power grid.  This interruption was caused by a disruption at the biggest substation due to a damaged transformer.  It’s unclear what caused the damage to the transformer, but it is believed to be related to the earthquake that hit in February.  We show this by adding a cause box with a question mark between “damaged transformer” and “earthquake on Feb. 27th”.

Repairs to the damaged transformer were required, which is an impact to the property and labor goals.

The Chilean government pledged to repair the transformer within 48 hours and stabilize the transmission lines within a week.  Interim solutions to get the electricity flowing were to isolate the damaged unit and install a reserve.  Additionally, Chileans have been asked to conserve electricity to minimize the amount of power transmitted through the lines.

By clicking ‘Download PDF” above, you can see the thorough root cause analysis built as a Cause Map that captures all of the currently known information in a simple, intuitive format that fits on one page.

Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.