By Kim Smiley
The Consular Consolidated Database (CCD) is the global database used by the U.S. State Department to process visas and other travel documents. On July 20, 2014, the CCD experienced software issues and had to be taken offline. The outage lasted several days with the CCD being returned to service with limited capacity on July 23. The CCD is huge, one of the largest Oracle-based warehouses in the world, and is used to process a hefty number of visas each year and the effects of the software glitch have been felt worldwide. The State Department processed over 9 million immigrant and non-immigrant visas overseas in 2013 so a delay of even a few days means a significant backlog.
This issue can be analyzed by building a Cause Map, a visual root cause analysis. A Cause Map visually lays out the different causes that contribute to an issue so that the problem is better understood and a wider range of solutions can be considered. The first step in the Cause Mapping process is to define the problem, which includes documenting the overall impacts to the goal. Most problems impact more than one goal and this example is no exception.
The customer service goal is clearly impacted because thousands – and potentially even millions – have had their travel document processing delayed. The negative publicity can also be considered an impact to the customer service goal because this software glitch isn’t doing the international image of the U.S. any favors. The delay in travel document services is an impact to the production/schedule goal and the recovery effort and investigation into the problems impact the labor/time goal. Additionally, there are potential economic impacts to both individuals who may have had to change travel plans and to the U.S. economy because these issues may discourage international tourism.
The next step in the Cause Mapping method is to build the Cause Map. This is done by asking “why” questions and using the answer to visually lay out the cause-and-effect relationships. The delay in processing travel documents occurred because the CCD is needed to process them and the CCD had to be taken offline as a result of software issues. Why were there issues with the database? Maintenance was done on the CCD on July 20 and the performance issues began shortly thereafter. The maintenance was done to improve system performance and to fix previous intermittent performance issues. The State Department has stated that this was not a terrorist act or anything more malicious than a software glitch. An investigation is currently underway to determine exactly what caused the software glitch, but the details have not been released at this time. It can be assumed that the test program for the software was inadequate since the glitch wasn’t identified prior to implementation.
The final step in the Cause Mapping process is to identify solutions that can be implemented to reduce the risk of a problem recurring. Details of exactly what was done to deal with the issue in the short term and bring the CCD back online aren’t available, but the State Department has stated that additional servers were added to increase capacity and improve response time. There is also a plan to improve the CCD in the longer term by upgrading to a newer version of the Oracle database software by the end of the year which will hopefully prove more stable.
To view an Outline and high level Cause Map of this issue, click on “Download PDF” above.