Software Engineering Institute | Carnegie Mellon University
Software Engineering Institute | Carnegie Mellon University

Quantifying Uncertainty in Early Lifecycle Cost Estimation

The Problem

In an era of budgetary downsizing, large-scale Department of Defense (DoD) programs such as Major Defense Acquisition Programs (MDAPs) and Major Automated Information Systems (MAIS) cost billions of dollars and attract Congressional attention as a source of savings. These programs are fraught with uncertainty, and the expected costs often change over the many years during which they exist. The DoD then faces hard choices between capability and affordability.

To produce cost estimates of these large-scale programs, domain experts, cost analysts, and others must make judgments about often unprecedented and very complex software-reliant systems. The challenge is even greater earlier in a program lifecycle when experts must make judgments based on analogy with existing systems that do not share all of the same characteristics as the proposed new MDAP or MAIS.

Accurate estimates for the cost of a program are crucial because estimates of the development and system lifecycle costs set the measures against which program budgets are funded. Programs are then held to those estimates. Because these estimates are provided early in a lifecycle that often spans decades and involves complex new technologies, it is hardly surprising that estimates are often inaccurate. Another way to state this problem is that programs are unable to quantify uncertainty.

Our Approach

We developed a novel method called Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE). The QUELCE method helps quantify and make visible the uncertainty and, thus, the confidence in a program's cost estimate. The QUELCE method enables a client to convene a set of domain experts to formulate an early lifecycle cost estimate expressed as a cost distribution rather than a single point.

The method's five-step process begins with identifying potential future changes to the expected program execution that will influence program cost. This is followed by probabilistic modeling of the interrelationships of the program change drivers and Monte Carlo simulation of cost model inputs to create program cost estimate distributions. Because many of the inputs are based on subject-matter expert judgment, this workshop also involves training to calibrate expert judgment through a series of exercises.


QUELCE Process
QUELCE: Reducing Complexity

QUELCE: Modeling Uncertainty

QUELCE results in a cost estimate represented as a distribution from which a decision maker can understand the level of risk associated with a particular cost value. It also produces an executable model that can be used to run alternative scenarios and that can be updated in the future for reestimation purposes.

Learn More

Blog Posts

Improving the Reliability of Expert Opinion within Early Lifecycle Cost Estimation

Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE): An Update

Reports

Sheard, Sarah. The QUELCE Method: Using Change Drivers to Estimate Program Costs. CMU/SEI-2016-TN-006. Software Engineering Institute, Carnegie Mellon University, 2016.

Sheard, Sarah. The QUELCE Method: Using Change Drivers to Estimate Program Costs. Software Engineering Institute, Carnegie Mellon University, 2016.

Goldenson, Dennis; & Stoddard, Robert. Quantifying Uncertainty in Expert Judgment: Initial Results (CMU/SEI-2013-TR-001). Software Engineering Institute, Carnegie Mellon University, 2013.

Ferguson, Robert; Goldenson, Dennis; McCurley, James; Stoddard, Robert; Zubrow, David; & Anderson, Debra. Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE) (CMU/SEI-2011-TR-026). Software Engineering Institute, Carnegie Mellon University, 2011.

Webinar

Quantifying Uncertainty in Early Lifecycle Cost Estimation

Podcast

Quantifying Uncertainty in Early Lifecycle Cost Estimation

Pilot the Method in Your Organization

Contact the SEI if your organization would like to use the QUELCE Workshop to help quantify uncertainty in early lifecycle cost estimation.