All posts by Kim Smiley

Mechanical engineer, consultant and blogger for ThinkReliability, obsessive reader and big believer in lifelong learning
Root Cause Analysis - Incident Investigation

Nearly 2.6 million GM Vehicles Recalled, Costs Soar to 1.3 Billion

By Kim Smiley

During the first quarter of 2014, General Motors (GM) recalled 2.6 million vehicles due to ignition switch issues tied to at least 13 deaths.  Costs associated with the issue are estimated to be around $1.3 billion and, possibility even more damaging to the long term health of the company, is the beating the company’s reputation has taken.

The ignition switch issues are caused by a small, inexpensive part called a switch indent plunger.  An ignition switch has four main positions (off, accessories, on, and start) and the switch indent plunger holds the ignition switch into position.  In the accidents associated with the recent recall, the ignition switch slipped out of the on position and into the accessories position because the ignition switch plunger didn’t have enough torque to hold it in place.  When the ignition is put into the accessories mode, the car loses both power steering and power braking, and the air bags won’t inflate.  It’s easy to see how a situation that makes a car less safe and more difficult to control can quickly create a dangerous, or even deadly, situation.  Additionally, it’s important to know the problem is most likely to occur when driving on a bumpy surface or if a heavy key ring is pulling on the key.

The other key element of this issue is how the problem has been handled by GM.  There are a lot of hard questions being asked about what was known about the problem and when it was known.  It is known that the faulty part was redesigned in 2006 to address the problem, but the new design of the part wasn’t given a new part number as would normally be done.  Multiple federal inquiries are working to determine when it was known that the faulty parts posed a danger to drivers and why there was such a long delay before a recall was done.  The fact that the redesigned part wasn’t assigned a new part number has also lead to questions about whether there was an attempt to cover up the issue. GM is not civilly liable for deaths and injuries associated with the faulty ignition switches because of its 2009 bankruptcy, but the company could potentially be found criminally liable.

No company ever wants to recall a product, but it’s important to remember that how the recall is handled is just as important as getting the technical details right.  Consumers need to believe that a company will do the right thing and that any safety concerns will quickly and openly be addressed.  Once consumers lose faith in a company’s integrity the cost will be far greater than the price of a recall.

If you drive a GM car, you can get more information about the recall here.  The recalled models are Chevrolet Cobalts and Pontiac G5s from the 2005 through 2007 model years; Saturn Ion compacts from 2003 through 2007; and Chevrolet HHR SUVs, and Pontiac Solstice and Saturn Sky sports cars from 2006 and 2007.

To view the Outline and Cause Map showing the root cause analysis of this issue, please click “Download PDF” above.  Or click here to read more.

Root Cause Analysis - Incident Investigation

Dangerous Combination: Propane Shortages and a Bitterly Cold Winter

By Kim Smiley

Propane shortages and skyrocketing prices in parts of the United States have made it difficult for some homeowners to affordably and consistently heat their homes this winter.   The brutally cold winter many regions are experiencing is also worsening both the causes and effects of the shortages.

A Cause Map can be built to help understand this issue.  Cause Maps are a visual format for performing a root cause analysis that intuitively lay out the causes that contributed to an issue to show the cause-and-effect relationships.  To view a high level Cause Map of this issue, click on “Download PDF” above.

Why have there been recent propane shortages in regions of the United States?  This question is particularly interesting given the fact that propane production in the United States has increased 15 percent in the past year.   One of the reasons that propane prices have dramatically increased is because of a spike in demand.  There was a larger than normal grain crop this fall, which was also wetter than usual.  Wet grains must be dried prior to storing to prevent spoiling and propane is used in the process.  Local propane supplies were depleted in some areas because five times more propane was used to dry crops this year than last.   About 5 percent of homes in the United States depend on propane for heat and the unusually frigid temperatures this winter have resulted in additional increases in propane demand.

In addition to the increase in demand, there have been issues replenishing local supplies of propane quickly enough to support the increased demand.  There have been some logistical problems transporting propane this winter.  The Cochin pipeline was out of service for repairs, limiting how quickly propane could be transported to areas experiencing shortages.  There were rail rerouting issues that impacted shipments from Canada.

Additionally, many are asking questions about what role propane exports have played into the domestic shortages.   Propane exports have quadrupled in the last 3 years.  New mining techniques and improved infrastructure have made exporting propane to foreign markets more lucrative and companies have begun to ship more propane overseas. As more propane is shipped to foreign markets, there is less available for use in the United States.

The propane shortages are an excellent example of supply and demand in action.  Increasing demand combined with decreasing supply will result in higher prices.  Unfortunately addressing the problem isn’t simple. There are very complex logistic and economic issues that need to be addressed, but if people don’t have access to affordable heating, the situation can quickly become dangerous, or even deadly.  In the short term, lawmakers are taking a number of steps to get propane shipped to the impacted areas, but how the US chooses to deal with this issue in the long term is still being debated.

Root Cause Analysis - Incident Investigation

Olympic Track Worker Hit By Bobsled

By Kim Smiley

A worker at the bobsled track for the Sochi Winter Olympics was hit by a bobsled on February 13, 2014.  The worker suffered two broken legs and a possible concussion, but is reported to be stable after undergoing surgery.  There was also minor damage done to the track.  Part of a lighting system suspended from the ceiling was replaced and time was needed to clean small plastic shards off the ice.

Investigation into this accident is still underway, but the information that is available in the media can be used to build an initial Cause Map. One of the advantages of using Excel to build Cause Maps is that they can be easily modified to incorporate additional information once the investigation is complete.

When beginning the Cause Mapping process, the first step is to fill in an Outline with the basic background information for an issue.  How an incident impacted the overall organizational goals is also documented on the bottom half of the Outline.  Once the Outline is completed, the Cause Map is built by asking “why” questions. (Click on “Download PDF” above to view a high level Cause Map and Outline for this accident.)

So why was the worker hit by a bobsled?  This occurred because a forerunner sled was sent down the track while the worker was on the track.  The forerunner sled was on the track because they are used to test the track prior to training runs and competitions, and training was scheduled later that day.  Forerunner sleds ensure that ice conditions are good and that all systems, like the timing system, are functional.  People at the top of the track can’t see the entire track so there wasn’t an easy way for them to identify the position of the worker prior to running the sled.  Initial reports are that the normal announcements were made to the workers prior to running the forerunner sled so it doesn’t appear that the people on the top of the track had any reason to suspect a problem.

The worker was on the track doing work to prepare it for the training runs and competition scheduled that day.  We can safely assume that he was unaware that the forerunner sled was running the track at the same time.  Investigators have determined that the worker was using a loud motorized air blower and believe he was unable to hear both the announcement and the approaching bobsled.  Two other workers were also working on the track, but they were able to scramble out of danger as the bobsled approached.  Until the investigation is complete, it won’t be clear if other factors were involved, but it seems the use of loud equipment played a role in the accident.

The final step in the Cause Mapping process is to find solutions to reduce the risk of a problem reccurring.  It appears that the current method of letting workers know to clear the track isn’t adequate in all situations.  Officials will need to modify the process, especially when loud equipment is in use, to ensure the safety of all workers.  Workers need to be on the track at times in order to do their jobs and there needs to be a way to ensure they have moved to a safe location prior to any sled running the track.

It’s worth noting this is not the first time someone has been hit by a bobsled. In 2005, recent silver medalist skeleton racer Noelle Pikus-Pace was hit by a bobsled.  She shattered a leg and ended up missing the 2006 Turin Olympics as a result.  This accident occurred on a different track, but it highlights the dangers of bobsled tracks and the important of ensuring safety.

Root Cause Analysis - Incident Investigation

Concerns Raised About Safety of Olympic Slopestyle Course

By Kim Smiley 

One of the stories making headlines leading up to the start of the 2014 Winter Olympics was concern about the safety of the slopestyle course.  There were early rumblings about the slopestyle course, especially after a few falls during training runs, but the media interest intensified after well-known snowboarder Shaun White withdrew from the event.   There is also a heighten sensitivity to safety concerns after the death of a luger during the last Winter Olympics , which was the first  death in Olympic training or competition since 1964.

Safety of the athletes involved in the Olympics is obviously paramount, but media coverage of slopestyle course safety concerns is also an issue because it created negative press for both the Olympics and the host country.  A Cause Map can be built to help analyze this issue and illustrate all the factors involved with the controversy surrounding the Olympic slopestyle course. (To see a high level Cause Map of this issue, click on “Download PDF”.)

Several athletes fell during training runs on the slopestyle course, which led to questions about course safety.   There were some injuries on the course, the most notable being Torstein Horgmo of Norway who broke his collarbone during a practice run.  Horgmo was a favorite to medal in the event and was unable to compete after his injury, which has to be heartbreaking.

The course is different from the typical slopestyle course, partly because this is the Olympics and the designer wanted an exciting course.   Athletes are getting more air time from the jumps on the course because they are large step-down jumps where the landing zones are below the ramps.  Designing the first Olympic slopestyle course was a unique challenge and there was no precedent.

The weather has been an added challenge for the course designer.  The jumps were created intentionally oversized with plans to modify them as needed to help accommodate melting concerns in the above freezing weather.  It’s much easier to make a jump smaller, as opposed to larger, so designers would rather err on the size of too big.  Rain and warm weather also played havoc with plans to test the course.  A test event scheduled for last February was canceled because of weather.  Tests were scheduled to allow for more time to groom the course prior to the Olympics, but six days of massive rains pushed course completion past schedule.

It’s also worth noting that there is inherent danger in slopestyle.  Slopestyle is an extreme sport with snowboarders performing high intensity tricks in the air.  Factor in the pressure to bring the goods in an Olympic event and snowboarders are going to be pushing their limits.  The falls don’t all happen on the jumps, despite media focus on the large jumps on this course.  Torstein Horgmo’s Olympic-ending crash occurred on the stair set on top of the course.   While a course can be made too dangerous, there will never be a completely safe slopestyle course because of the nature of the sport.

Snowboarder Shaun White made headlines when he pulled out of slopestyle because of injury concerns, but it’s also important to remember that slopestyle isn’t White’s main event.  Although White failed to reach the podium this Olympics, he was the defending gold medalist on the halfpipe and wasn’t willing to risk his chance to compete in that event.  White suffered minor injuries from a crash on the slopestyle course and he didn’t want to impact his halfpipe chances by getting hurt worse.  Halfpipe came after slopestyle so the consequences of a potential injury were high for White.  I’m willing to bet he would have been much more likely to compete in slopestyle if it occurred after the halfpipe event.

The slopestyle course was modified after training runs, which is typical for an untested slopestyle course.  Forty to fifty centimeters were removed from the top deck of the jumps and snow was added to the knuckles of each landing.  The course crew has been credited for listening to athletes’ concerns and being responsive to issues. Lessons learned from the experience with the first Olympics slopestyle course will hopefully help things go smoother next time.  I hope the focus during the next Olympics is on the amazing athletes and not so much on the course.

Root Cause Analysis - Incident Investigation

Millions Impacted by Data Breach At Target

By Kim Smiley

Are you one of the millions of customers affected by the recent data breach at Target?  Because I am.  I for one am curious about how data for approximately 40 million credit and debit cards was compromised at one of the United States’ largest retailers.

The investigation is ongoing and many details about the data breach haven’t been released, but an initial Cause Map can be built to begin analyzing this incident.  The latest information released is that the Justice  Department is  performing an investigation into this incident.  An initial Cause Map can capture the information that is available now and can easily be expanded to include more detail in the future.  A box with a question mark can be used to indicate that more information is needed on the Cause Map. (Click on “Download PDF” to view an Outline and high level Cause Map.)

One of the causes that I think is worth discussing is that retailers in the United States are being specifically targeted for this type of attack in recent years.  The vast majority of credit and debit cards in use in the United States are magnetic strip cards, while Europe has been transitioning to newer credit card technology that uses chips.   Magnetic strip credit cards are a more desirable target for criminals because the technology to create fake magnetic strip cards is readily available.  The data on magnetic strip cards also stays the same while chips use unique codes for each transaction.  Cards with chips also require a pin when used, adding an additional layer of protection.

So why does the United States still use magnetic strip cards?  One of the main complicating factors is money.  Transitioning to cards that use chips requires a significant investment of money by both banks and retailers.  It is estimated that the cost to transition to the higher tech cards will be $8 billion so the money required is considerable. Both parties are nervous about being the first to commit to the process.

Rising credit card fraud rates in the United States have been increasing the pressure to move to newer credit card technology.  Credit card fraud rates in the U.S. have doubled in the 10 years since Europe began using chip cards.  As long as the United States remains the softest target, the rates are likely to increase.

On a positive note, the transition to the newer chip cards should be gaining traction in the next few years.  Credit card companies have typically footed the bill for credit card fraud, but many card companies have stated that merchants or banks that have not transitioned to chip cards will be held accountable for fraudulent purchases that the higher tech cards would have prevented by the end of 2015.

The frustrating thing is that there are limited ways individual consumers can protect themselves short of switching to cash.  You can be smart about where you swipe your cards, for example avoiding unmanned ATM kiosks, but a major retailer like Target didn’t seem suspicious.  As somebody who has had multiple instances of credit card fraud in the last few years, I look forward to a safer credit card in the future.

Root Cause Analysis - Incident Investigation

300,000 Unable to Use Water after Chemical Spill in West Virginia

By Kim Smiley

Hundreds of thousands of West Virginians were unable to use their water for days after it was contaminated by a chemical spill on January 9, 2014. About 7,500 gallons of 4-methyl-cyclohexane-methanol, known as MCHM, leaked out of a storage tank and into the Elk River.   At the time of the spill, little information was known about MCHM, but officials ordered residents not to use the use the water because the chemical can cause vomiting, nausea, and skin, eye and throat irritation.  The ban on water usage obviously meant that residents should not drink the water, but they were also told not to cook, bathe, wash clothes or brush their teeth with it.

The investigation into this incident is still ongoing, but some information is available.  An initial Cause Map, or visual root cause analysis, can be built now and it can easily be expanded in the future.  A Cause Map is used to illustrate the cause-and-effect relationships between the many causes that contribute to any incident.  In this example, it is known that the MCHM leaked into the river because it was being stored in a tank near the river and the tank failed.  MCHM was being stored in a tank because it is used in coal processing and it was profitable for the company to sell it.

The cause of the tank failure hasn’t been officially determined, but the company who owned the facility has stated that an object punctured the tank after the ground under the tank froze.  (Suspected causes can be included on the Cause Map with a question mark to indicate that more evidence is needed to confirm their validity.)

The tank in question was older, built about 70 years ago.  There were no regulations that required the tank to be inspected while it was being used to store MCHM because the chemical is not currently legally considered a hazardous material.  The tank is also an atmospheric tank so it is exempt from current federal safety inspections because it is not under pressure, cooled or heated.

Many are asking questions about why a tank full of a chemical that can make people sick that was so close to the water supply had so little regulation and no required inspections.  The debate that has been sparked by this accident will force a close review of current regulations governing these types of facilities.

It’s also alarming how little was known about this chemical prior to this accident.  It’s still not well understood exactly how dangerous MCHM is.  Experts have stated that the long term impacts should be minimal, but it would be awfully reassuring to the people living in the area if there was more information about the chemical available.

Companies need to have a clear understanding of the risks involved in their operations if they hope to reduce the risk to the lowest reasonable level and develop effective emergency response plans to deal with any issues that do arise.  As the old saying goes – failure to plan is planning to fail.  Just ask the company involved.  Freedom Industries filed bankruptcy papers on January 17, 2013 as a direct result of this accident.

Root Cause Analysis - Incident Investigation

Freight Train Carrying Crude Oil Explodes After Colliding With Another

By Kim Smiley

On Monday, December 30, 2013, a 106-car freight train carrying crude oil derailed in North Dakota and violently exploded after colliding with another derailed train that was on the tracks.  No injuries were reported, but the accident did cause an impressive plume of hazardous smoke and major damage to two freight trains.

The investigation into the accident is ongoing and it’s still unknown what caused the first train to derail. Investigators have stated that it appears that there was nothing wrong with the railroad track or with the signals.  It is known that a westbound freight train carrying grain derailed about 2:20 pm.  A portion of this train jumped onto the track in front of the eastbound train.  There wasn’t enough time for the mile long train loaded with crude oil to stop and it smashed into the grain train, causing the eastbound oil train to derail.  (To see a Cause Map of this accident, click on “Download PDF” above.)

Train cars carrying crude oil were damaged and oil leaked out during the accident.  The train accident created near ideal conditions for an explosion: sparks and a large quantity of flammable fluid.   The fire burned for more than 24 hours, resulting in a voluntary evacuation of nearby Casselton, North Dakota due to concerns over air quality.  The track was closed for several days while the initial investigation was performed and the track was cleaned up.

The accident has raised several important issues.  The safety of the train cars used to transport oil has been questioned.  Starting in 2009, tank train cars have been built to tougher safety standards, but most tank cars in use are older designs that haven’t been retrofitted to meet the more stringent standards.  This accident, and others that have involved the older design tank cars in recent year, have experts asking hard questions about their safety and whether they should still be in use.

The age of the train cars is particularly concerning since the amount of oil being transported by rail has significantly expanded in result years.  Around 9,500 carloads of oil were reportedly transported in 2008 and nearly 300,000 carloads were moved during the first three quarters of 2013.  The oil industry in North Dakota has rapidly expanded in recent years as new technology makes oil extraction in the area profitable.   North Dakota is now second only to Texas in oil production since the development of the Bakken shale formation.  Pretty much the only way to transport the crude oil extracted in North Dakota is via rail.  There isn’t a pipeline infrastructure or other alternative available.

And most of the time, transporting oil via freight train is a safe evolution.  The Association of American Railroads has reported that 99.99 percent of all hazardous materials shipped by rail reach the destination safely.  But it’s that 0.01 percent that can get you in trouble.  As a nation, we have to decide if where we are at is good enough or if it’s worth the money to require all tank cars used to transport oil to be retrofitted to meet the newest safety standards, a proposition that isn’t cheap.

Root Cause Analysis - Incident Investigation

Ceiling Collapse in London’s 112-year-old Apollo Theatre Injures Dozens

By Kim Smiley

On December 19, 2013, 76 people were injured when a large section of plaster fell from the ceiling of London’s historic Apollo Theatre.  Luckily there were no fatalities as a result, but six people were seriously injured in the accident.

The investigation is still underway, but an initial Cause Map can be built to begin analyzing the incident.  The first step in the Cause Mapping process is to fill in an Outline with the basic background information as well as formally list how the incident impacts the goals so that no part of a multifaceted problem is neglected.  It’s important to understand how an issue impacts all goals, such has safety issues, financial considerations, schedule delays, etc. There are times when different solutions can help mitigate risks to separate goals so it is useful to list all impacted goals for clarity.   Listing the impacted goals will also help focus the investigation on the most important elements.

Another very important part of the Outline is a space where any relevant differences are listed.  Anything that was different at the time an incident occurred is usually a good place to start digging during an investigation.  For this example, there was heavy rain during the hour preceding the ceiling collapse.  It’s also worth noting that the Apollo Theatre is 112 years old.

Investigators have not announced what led to the ceiling collapse, but early speculation is that rain water leaked through the roof and settled onto the plaster.  The theory is that the additional weight from the water was more than the ceiling could handle and it fell, taking a lighting rig and part of a balcony with it.   If this was the case, there will need to be hard questions asked about the adequacy of current building codes and inspection requirements.  Currently, the roof on the Apollo Theatre was only required to be inspected every 3 years.  It appears that the Theatre was up to date on and had passed all required inspections so the required periodicity may need to be re-evaluated in light of the recent failure.

Any suspected causes that haven’t been proven yet can be included on the Cause Map, but are marked with a “?” to indicated that they need additional evidence.  This helps document what has been considered during an investigation and questions that still need to be answered.

To view an Outline and the initial Cause Map of the Apollo Theatre ceiling collapse, click on “Download PDF” above.

 

Root Cause Analysis - Incident Investigation

Poisoned Dead Mice Parachuted onto Guam to Kill Snakes

By Kim Smiley

On December 1, 2013, 2,000 dead, poisoned neonatal mice were parachuted onto Guam on a unique mission to fight an invasive species, the brown tree snake. The parachutes are designed to catch in the trees and tempt the snakes, who live in the trees, into eating the mice. The mice are pumped full of acetaminophen, a chemical that the snakes are particularly sensitive to because it affects their blood’s ability to carry oxygen.

There are an estimated 2 million brown tree snakes on Guam so the 2,000 poisoned mice will only impact a very small percentage of the population, but scientists hope that the information they learn from this drop will help them plan larger mice drops in the future.  This is the fourth and largest dead mice drop so far and cost 8 million dollars.  Some of the mice were embedded with data-transmitting radios for this drop which will allow scientists to better gauge the effectiveness of this technique.

While the 8 million dollar price tag sounds high, it’s important to realize that the damage done by the brown tree snakes each year is significant.  Since their accidental introduction to the island, brown tree snakes have destroyed the native ecosystem, decimating the native bird population.  Brown tree snakes are also fantastic climbers and they routinely get into electrical equipment.  They cause an average of 80 power outages a year, resulting in costs as high as $4 million for repairs and lost productivity annually. (See our previous blog for more information.)

Even through the problem of the brown tree snakes is fairly well understood, an effective solution has been difficult to find.  There have been a number of different things tried over the years: snake traps, snake-sniffing dogs and snake-hunting inspectors have all been used, but the snakes have completely over un the island.  As farfetched as it sounds, parachuting dead mice seems to be the most promising solution at present.  It works because the snakes are very sensitive to acetaminophen; they only need to ingest about one-sixth of a standard pill for it to be effective.  This means that non-target animals are unlikely to be heavily impacted by the mice drops.  A pig or dog would need to eat around 500 of the baited mice for the dose to be lethal. One of the concerns is that snakes tend to avoid prey that is already dead, but information from the radio transmitters used in the recent drop should confirm if the mice are an effective bait.

One thing I know for sure, I would have loved to be in the brainstorm meeting the first time someone suggested parachuting dead mice.  This example is a good reminder to all of us to keep an open mind.  Every now and then, the most bizarre solution suggested turns out to be the best.

Root Cause Analysis - Incident Investigation

Metro Train Derails in the Bronx, Killing 4 and Injuring More Than 60

By Kim Smiley

Four passengers were killed and dozens more sent to the hospital after a metro train derailed in the Bronx early Sunday, December 1, 2013.  At the time of the accident, the train was carrying about 150 passengers and was traveling to Grand Central Terminal in New York City. The aftermath of the accident was horrific with all seven cars of the commuter train derailing. Metro-North has been operating for more than 30 years and this was the first accident that resulted in passenger deaths.

A Cause Map, or visual root cause analysis, can be built to help analyze this accident.  There is still a lot of investigative work that needs to be done to understand what caused the derailment, but the information that is available can be used to create an initial Cause Map.  The Cause Map can easily be expanded later to incorporate more information as it becomes available.  The first step when building a Cause Map is to fill in an Outline with the basic background information.  The impacts to the goals are also documented on the bottom of the Outline.  The impacted goals are then used to begin building the Cause Map.

In this example, the safety goal is clearly impacted because there were four fatalities and over 60 people injured.  The schedule goal is also significantly impacted because this portion of rail will be closed during most of the investigation.  The National Transportation Safety Board has estimated that the investigation will take 7 to 10 days.  The track closure is particularly impacting because this is a major artery into New York City with a ridership of 15.9 million in 2012.  Once the impacted goals are documented, the Cause Map itself is built by asking “why” questions.

So why did the train derail?  The details aren’t known yet, but there is still some information that should be documented on the Cause Map.  A question mark is included after a cause that may have contributed to an issue, but requires more evidence or investigation.  It’s useful to document these open questions during an investigation to ensure that all the pertinent questions are asked and nothing is overlooked.  (If it is determined that a cause didn’t play a role, it can be crossed out on the Cause Map to show that the cause was considered, but ruled out.)  Two factors that likely  played a role in the derailment are the speed of the train and the track design where the accident occurred.  There is a sharp curve in the track where the derailment happened.  Trains are required to reduce their speed before traveling it.  The latest reports from the investigation are that the train was traveling 82 mph in a 30 mph zone. The train operator has stated that the brakes malfunctioned and didn’t respond when he tried to reduce speed and that the train was traveling too fast over the curved track.

Investigators have recovered the data recorder from the train which will provide  more information and if there was a problem with the brakes.  Investigators will also interview all the relevant personnel and determine what happened to cause this deadly crash.  Once the investigation is completed, any necessary solutions can be implemented to reduce the risk that a similar accident occurs in the future.

To view a completed Outline and initial Cause Map of this incident, click on “Download PDF” above.