In an era of budgetary downsizing, large-scale Department of Defense (DoD) programs such as Major Defense Acquisition Programs (MDAPs) and Major Automated Information Systems (MAIS) cost billions of dollars and attract Congressional attention as a source of savings. These programs are fraught with uncertainty, and the expected costs often change over the many years during which they exist. The DoD then faces hard choices between capability and affordability.
To produce cost estimates of these large-scale programs, domain experts, cost analysts, and others must make judgments about often unprecedented and very complex software-reliant systems. The challenge is even greater earlier in a program lifecycle when experts must make judgments based on analogy with existing systems that do not share all of the same characteristics as the proposed new MDAP or MAIS.
Accurate estimates for the cost of a program are crucial because estimates of the development and system lifecycle costs set the measures against which program budgets are funded. Programs are then held to those estimates. Because these estimates are provided early in a lifecycle that often spans decades and involves complex new technologies, it is hardly surprising that estimates are often inaccurate. Another way to state this problem is that programs are unable to quantify uncertainty.
We developed a novel method called Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE). The QUELCE method helps quantify and make visible the uncertainty and, thus, the confidence in a program's cost estimate. The QUELCE method enables a client to convene a set of domain experts to formulate an early lifecycle cost estimate expressed as a cost distribution rather than a single point.
The method's five-step process begins with identifying potential future changes to the expected program execution that will influence program cost. This is followed by probabilistic modeling of the interrelationships of the program change drivers and Monte Carlo simulation of cost model inputs to create program cost estimate distributions. Because many of the inputs are based on subject-matter expert judgment, this workshop also involves training to calibrate expert judgment through a series of exercises.
QUELCE results in a cost estimate represented as a distribution from which a decision maker can understand the level of risk associated with a particular cost value. It also produces an executable model that can be used to run alternative scenarios and that can be updated in the future for reestimation purposes.
Sheard, Sarah. The QUELCE Method: Using Change Drivers to Estimate Program Costs. CMU/SEI-2016-TN-006. Software Engineering Institute, Carnegie Mellon University, 2016.
Sheard, Sarah. The QUELCE Method: Using Change Drivers to Estimate Program Costs. Software Engineering Institute, Carnegie Mellon University, 2016.
Goldenson, Dennis; & Stoddard, Robert. Quantifying Uncertainty in Expert Judgment: Initial Results (CMU/SEI-2013-TR-001). Software Engineering Institute, Carnegie Mellon University, 2013.
Ferguson, Robert; Goldenson, Dennis; McCurley, James; Stoddard, Robert; Zubrow, David; & Anderson, Debra. Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE) (CMU/SEI-2011-TR-026). Software Engineering Institute, Carnegie Mellon University, 2011.