QUELCE: Quantifying Uncertainty in Early Lifecycle Cost Estimation
Created September 2017
Costs for new systems are hard to predict, and the larger the program, the more difficult it is to estimate. We developed a method to quantify uncertainty and increase confidence in a program's cost estimate.
How Do You Measure Uncertainty?
The DoD must ensure that it creates maximum-impact systems with minimum taxpayer money. This goal requires allocated funds to be just the right amount. But costs for new systems are hard to predict. In addition, Major Defense Acquisition Programs and Major Automated Information Systems are so large that their size magnifies the typical problems that new systems experience. Our solution helps programs understand what problems are likely to occur and how much more cost would result if they do occur. Called QUELCE, or Quantifying Uncertainty in Early Lifecycle Cost Estimation, it works for both new and modified programs.
To produce cost estimates of new programs, subject-matter experts (SMEs) and cost analysts must use their experience to make “best guess” estimates of likely costs for unprecedented and complex software-reliant systems. The challenge is greater early in a program lifecycle, when experts make judgments based on analogies with existing systems that do not share many characteristics with the proposed new system.
Accurate estimates for the cost of a program are crucial for several reasons. Cost estimates for development and sustainment set the measures against which programs are funded. Projects are then managed according to those estimates, which may cause them to take shortcuts during development, deal with troubling contract modifications, or face termination if problems arise that drive the costs up too much. Because a project team must provide these estimates early in a lifecycle that spans years and involves complex new technologies, they are often inaccurate. This unquantified uncertainty prevents accurate estimates and managed costs.
We developed a method called Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE). The QUELCE method helps quantify uncertainty, capture risks, and increase confidence in a program's cost estimate. QUELCE enables a client to gather a set of SMEs to anticipate and document risks that may impact future cost. The result is a calculation of cost estimate as a distribution rather than a single number.
Because many inputs come from SMEs' judgment, the workshop begins with a series of training exercises to improve judgment skills. The experts then identify potential future changes to the program that could influence cost. We call these change drivers. They include changes such as funding cuts, new stakeholders, and supply-chain interruptions. The experts next assess the probability that each change driver will occur and the degree to which it could affect other potential changes.
The QUELCE method then uses Bayesian Belief Network modeling to quantify uncertainties among change drivers, which become inputs to cost models. This step integrates with inputs from typical cost-estimating tools such as COCOMO, CoSYSMo, and SEER. Finally, QUELCE applies Monte Carlo simulation to calculate thousands of what-if possibilities. The result is a cost estimate in the form of a distribution for each change scenario, as opposed to a single “guesstimate.”
Benefits of QUELCE
- uses a growing repository of DoD experience in program execution. Data from this repository provides SMEs with a starter set of potential changes to consider.
- provides a detailed model of all potential change drivers. Traditional methods cover only a fraction of the possible alternatives.
- provides cost estimates with defined upsides and downsides. A decision maker can use this distribution to assess the level of risk associated with a cost value.
- produces a model that a program can use to run more scenarios later. A program can assess different what-if possibilities and re-estimate cost as changes occur.
- helps ensure that acquisition programs are funded at levels consistent with the level of risk to achieving success. A program will also have fewer and less severe cost overruns due to poor estimates.
"The QUELCE methodology, training, supporting tools, and expert guidance could not have been more relevant, timely, and insightful given our currently highly cost-competitive environment." —Neil Mackertich, Raytheon Company
Who Should Use This Method
- Project Management Offices (PMOs) that need cost estimates for large complex systems to meet government contracting and acquisition requirements
- DoD systems that are pre-Milestone A, pre-Milestone B, or rebaselining
- Service cost organizations that want to assess the quality of cost estimates
August 17, 2016 Technical Note
This technical note introduces Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE), a method for estimating program costs early in development.read
November 01, 2012 Podcast
In this podcast episode, Jim McCurley and Robert Stoddard discuss a new method developed by the SEI's Software Engineering Measurement and Analysis (SEMA) team, Quantifying Uncertainty in Early Lifecycle Cost Estimation (QUELCE).learn more
August 06, 2012 Blog Post
As part of our research related to early acquisition lifecycle cost estimation for the Department of Defense (DoD), my colleagues in the SEI's Software Engineering Measurement & Analysis initiative and I began envisioning a potential solution that would rely heavily...read