Hubble Focusing Issues [ August 4th, 2008 ] Posted in » Root Cause Analysis - Incident Investigation

Hubble TelescopeThe Hubble Space Telescope was launched on April 24, 1990.  Once in orbit, it was quickly discovered that the images from Hubble were blurred.  An investigation into the issue revealed that Hubble’s primary mirror was not built to specification and couldn’t properly focus the light.  Specifically, the mirror was flattened too much away from the center and caused the light reflected from the edge of the mirror to focus on a slightly different location than the light reflected from the center.   The primary mirror in Hubble was only off specification by 2.3 micrometers, but the result to the $1.5 billion dollar project was disastrous. 

Solving Hubble’s focus issues was no small feat.  How do you repair a mirror that can’t be replaced on orbit when it is cost prohibitive to bring it back to earth for repair?  The answer was to modify the lens (which met specifications) to work with the off specification mirror.  COSTAR (Corrective Optics Space Telescope Axial Replacement) was added to Hubble during the first servicing mission in December 1993.  COSTAR is essentially eyeglasses for Hubble, additional lens built with the same error as the mirror, but in the opposite direction so that the effects of the off specification mirror shape are canceled out.  With the addition of COSTAR, Hubble met original design goals.

The primary mirror was constructed with a flaw because the tool, called a null corrector, used to create the template to guide the shaping of the mirror was itself flawed.  Null correctors use precisely located mirrors and lens to determine the shape of a mirror.  In order to assemble null correctors, reflected light is used to measure the distance between the mirror and the lens inside the tool.  When the null corrector used to shape the Hubble’s primary mirror was assembled a measurement error was made.  A small amount of reflective coating had fallen off an internal piece of the instrument and the laser used to perform the measurement reflected off the wrong location, resulting in a lens being 1.3 mm to far from the mirror.  Null correctors are extremely precise and do not change once assembled so the Hubble team used a single instrument to guide the mirror shape.  A single flawed tool and inadequate quality controls resulted in a flawed mirror.

Root Cause Analysis :: Hubble Focus Issue A visual representation of root cause analysis has been created as a Cause Map that can be downloaded.

General Slocum Steamship Fire

On June 15, 1904, a church group headed out for an excursion through New York City’s East River on the Steamship General Slocum.  Approximately half an hour after the ship left the pier, it caught fire.  Despite being only hundreds of yards from shore, the Captain continued to go full speed ahead in hopes of beaching at North Brother Island, where a hospital was located.  This served to fan the flames quickly over the entire highly flammable ship, killing many in the inferno.  Most of those who were not killed by the fire drowned, even though the Captain did successfully beach the ship at North Brother Island, due to the depth of the water and lack of safety equipment.

Download Root Cause Analysis DocumentTo perform a root cause analysis of the General Slocum tragedy, we can use a cause map.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.  First we look at the impact to the goals.  On the General Slocum there were at least 1,021 fatalities of the passengers and crew that were aboard.  (However, only two of the crew were killed.)  There were other goals that were affected but the magnitude of the loss of life makes any other goals less significant.  The deaths and injuries were the impact to the safety goals.

Passengers drowned because they were in water over their heads with inadequate help or safety equipment.  Passengers were either in the water because they fell when the deck collapsed, or because they jumped into the water trying to avoid the fire.  The water was too deep to stand because only the bow was in shallow water and the passengers could not reach the bow.  This was due to a poor decision on the Master’s part (namely his decision to beach the ship at a severe angle, with the bow in towards the island, instead of parallel to the island, where passengers would have been able to wade to shore.)  Note that the Master himself (and most of the crew) were on the bow side of the ship and were able to (and did) jump off and wade to shore.  The safety equipment, including life preservers, life boats, and life rafts, was mostly unusable due to inadequate upkeep and inadequate inspections.

Passengers (and two crewmembers) were also killed by fire.  Once the fire was started, it spread rapidly and was not put out.    The fire spread rapidly because the ship was highly flammable.  When this ship was constructed, there was no consideration of flammability.  Additionally, the current of air created by the vessel speeding ahead drove the fire across the ship.  The fact that an experienced Master would have allowed this situation was considered misconduct, negligence and inattention to duty - the charges for which the Master was later convicted.   The fire was not put out because of inadequate crew effort and insufficient fire-fighting equipment.  The crew effort was inadequate of a lack of training.  The fire-fighting equipment was insufficient because of inadequate upkeep and inadequate inspections.  (Possibly you are noticing a theme here?)

Although many people have not heard of the General Slocum tragedy, many of its lessons learned have been implemented to make ship travel safer today, although many of the solutions were not implemented widely enough or in time to prevent the Titanic disaster from occurring eight years later.  (Although there were actually more people killed on the General Slocum, it is believed that the Titanic disaster is more well known because the passengers on Titanic were wealthy, as opposed to the working class passengers on General Slocum.  It is also surmised that sympathy for the highly German population aboard General Slocum was diminished as World War I began.)

In a macabre ending to a gruesome story, ships began replacing their outdated, decrepit life preservers after the investigation on General Slocum.  It was later found that the company selling these new life preservers had hidden iron bars within the buoyant material, in a dastardly attempt to raise their apparent weight.  Unfortunately there were no adequate laws (then) against selling defective life-saving equipment.

June 28th, 2008 | Leave a Comment

Blood Substitute Risk

A study recently published by the Journal of the American Medical Association presented a review of clinical trials of hemoglobin-based blood substitutes.  This study showed that the clinical trials resulted in increased risk of heart attack and death for the patients being studied with no clinical benefit. 

Download Root Cause Analysis DocumentWe will examine this issue using the Cause Mapping process.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.

In clinical trials one of the overall goals is to have zero injuries. The blood substitute trials led to a 30% increased risk of death, and a 2.7-fold increase in heart attack, causing increased risk with no clinical benefit.  The two goals that are impacted in the blood substitute example are the safety goal and the customer service goal.

In this example all of our impacts to the goals are caused by the increased risk of heart attack (myocardial infarction).  Additionally, there was no clinical benefit shown because the use of blood substitutes did not limit blood transfusions. 

Why was there an increased risk of heart attack?  The increased risk of heart attack is caused by decreased blood flow, which is caused by blood vessel contraction (vasoconstriction).  This occurs because nitric oxide is responsible for blood dilation, hemoglobin molecules scavenge nitric oxide, and a patient receives an infusion of hemoglobin.

The patient receives an infusion of hemoglobin because the patients are unaware of the risk, and because of ongoing clinical trials of hemoglobin-based blood substitutes.  These trials are ongoing because hemoglobin-based blood substitutes have been developed and because clinical trials are being performed.

The hemoglobin-based blood substitutes have been developed because blood substitutes are being developed and most of the blood substitutes are hemoglobin-based, because hemoglobin is seen as the most promising substitute.  The blood substitutes are being developed because they would be better in remote areas or for portability, to help deal with the shortage of blood, and to reduce problems from blood transfusions.

The clinical trials were performed because they were approved by the FDA;  there was no checking by scientists, review boards, or the public; and the companies continued clinical trials.  There was no checking, and the companies continued the trials, because there was a lack of information available.

The FDA and the blood companies are still trying to figure out how to go forward based on these new results.  Because of the potential usefulness of blood substitutes, especially in military applications, it’s likely we’ll continue to see progress on this issue.

June 12th, 2008 | Leave a Comment

Reactor Vessel Head Degradation - Davis-Besse Nuclear Power Station

On March 7, 2002, during refueling, a cavity measuring approximately 4 x 6 inches was discovered that had completely eaten through the more than 6″ thick reactor pressure vessel head of Unit #1 reactor at Davis-Besse Nuclear Power Station.  Fortunately, the thin stainless steel cladding layer had held the reactor pressure, although it was not designed to do so.  The loss of the vessel head was also a loss of a principal fission product barrier (one of the three responsible for ensuring radiaoctive fission products remain within the pressure boundary).  This was an impact to the safety goal.  The loss of a principal fission product barrier is also considered a “significant precursor to core damage” by the NRC, which is another impact to the safety goals.  All told, the cavity resulted in nearly $300 million worth in repairs and upgrades, and a two-year closure of the plant, during electricty production at Davis-Besse was severely reduced.  These were impacts to the material, production, and customer service goals. 

Download PDF documentLet’s examine a high level root cause analysis and review some of the causes of the cavity.  A thorough root cause analysis built as a Cause Map can capture all of the causes in a simple, intuitive format that fits on one page.

The cavity was caused by continued boric acid corrosion.  The corrosion occurred when leaking coolant evaporated into boric acid.  This occurred because of a through-wall crack in a nozzle caused by primary water stress corrosion cracking that was undetected.  The corrosion also occured because the leakage was undetected, due to delayed inspections and an ineffective leakage detection methods. 

The boric acid was not removed because it was not viewed as a safety concern.

The corrosion occurred due to inadequate corrosion control, where the corrosion was not detected because of a lack of a full inspection of the head,  and because early signs of corrosion were ignored, or missed.  The oil corrosion products were not completely removed, because they were difficult to remove and their removal was on a “best-effort” basis.  Additionally, the control was inadequate because the rate was higher than expected.

Even more detail can be added to this Cause Map as the analysis continues. As with any investigation the level of detail in the analysis is based on the impact of the incident on the organization’s overall goals.

June 2nd, 2008 | Leave a Comment

Site Map   Root Cause Analysis