NEWS AT SEI
This article was originally published in News at SEI on: May 1, 2006
Return on investment (ROI) is the heart of business—it’s why we form companies, develop them, merge them, and even sell or close them. But, as applied to information security, determining ROI has always been a challenge. How do you quantify the value of something that is less likely to happen if you spend money to prevent it?
Increasingly, however, ROI is a necessary metric for evaluating security investments. If you’re a chief information officer (CIO) or chief information security officer (CISO), you may be asked to provide hard numbers rather than relying on fear, uncertainty, and doubt to sell your proposals. If you’re a CEO, you may ask for hard numbers rather than simply being willing to approve a security investment “because it must be done, or else.” You also may recognize the wisdom of using hard numbers to compare several security-investment alternatives and choose the one that best contributes to the organization’s bottom line.
This column explores the importance of ROI metrics to an organization’s security efforts and why it’s important to formalize these metrics.
First, let’s touch briefly on what ROI means. Return on investment measures the expected improvement over the status quo (for example, a 50 percent reduction in the number of firewall breaches) against the cost of the action required to achieve the improvement (for example, the purchase of a new firewall).
Note that in the context of security, improvement is generally characterized not as a concrete gain, but rather as a reduction in risk.
So, how do you determine how much you’ll save if you make a specific investment? There are several methods you can use, but first you need to decide what to measure. Here are six types of loss you might consider for your ROI calculations [Allen 03]:
This is only a partial list. Think about other areas you could measure. Do you take into account the indirect costs of a breach to your customers, suppliers, and stakeholders? How can you really calculate that?
As you might suspect, there’s a degree of uncertainty and subjectivity involved, but once you decide what to measure and estimate, the question of how to measure it should be somewhat easier. The most effective measures are likely to be those you already are using, because these will enable you to compare security projects with all other projects. Security is only one aspect of business, albeit an important one; as such it’s best to analyze it in concert with all other investments and expenses.
One typical method for measuring security ROI is payback, a simple calculation that compares annualized loss expectancy against the expected savings as a result of an investment.
Here’s an example: You determine that your chance of suffering a breach due to password compromise is approximately 90 percent in any given year (based on historical incidents and surveys). You determine that if you institute a security-awareness program, you can increase the use of strong passwords throughout the organization and reduce your chance of a password compromise incident to approximately 30 percent in any given year.
If password compromise costs your organization, on average, $150,000 per incident, and you’ve lowered the chance of such an incident taking place from 90 percent to 30 percent, you will have reduced your total annualized loss expectancy from 90 percent of $150,000 to 30 percent of $150,000 [Allen 03]. Your expected loss will have dropped from $135,000 per year to $45,000 per year.
Have you saved $90,000 per year? No. If a password breach still occurs, you’ve saved nothing in that year. But breaches will happen less often. The annualized loss expectancy helps you calculate the expected magnitude of savings from reduced risk, so on average you will have saved $90,000 per year by instituting the training program.
In this example, your decision about whether or not to invest could change based on the cost of the training program. For example, if the program cost only $10,000 per year to implement and maintain, the clear decision would be to make the investment. Conversely, if the program cost $100,000 per year, it clearly would not be worthwhile. But what if the program cost $80,000 per year? Then the decision would be more difficult, as we’ll see later in this column.
A second method, NPV or net present value, adds another dimension to payback by considering the time value of money–the fact that money spent today is worth more than savings realized tomorrow. The best thing about NPV as it relates to security is that many other units of the organization–such as Finance–are probably using it.
For the calculation we just did for the security-awareness program and the reduced password compromises, using NPV would change the exact numbers in the result significantly, though it would still be obvious that the awareness program shows a strongly positive ROI.
For example, if you were using NPV and looking at money you’d be saving today, nothing would change. However, with security-awareness training, the savings are in the future. Each year, you would expect to save $90,000. However, $90,000 tomorrow is worth less than $90,000 today.
NPV does not just apply to future savings. Future costs also are cheaper than costs incurred in the present. For example, a firewall that requires a $100,000 payment up front actually costs more than a firewall that requires a $100,000 payment two years from now.
So, if you’re calculating projected savings or costs using NPV, you need to know the discount rate, which determines how much less your money will be worth in the future. This isn’t something your organization can determine; it’s a rate that affects the economy as a whole at any given time.
Let’s say the discount rate is 10 percent. If you expect to save $90,000 a year from now, NPV will tell you the value of that savings in today’s dollars. To calculate NPV, divide the expected yearly savings ($90,000) by 1.1 (that’s 1 plus the discount rate) to the power of 1 (because we’re calculating one year out). The result is approximately $82,000.
Carrying our example further, the $90,000 we save two years from now will be worth $90,000 divided by 1.1 to the power of 2 (because now we’re calculating two years out). The result is approximately $74,000. For year three, it’s $90,000 divided by 1.1 to the power of 3, for a result of about $68,000.
If you’re trying to weigh costs and benefits, and some of the costs are immediate but the benefits are long term, NPV can provide a more accurate measure of whether a project is truly worthwhile.
It is important to note that, as in any risk assessment scenario, the numbers derived from such calculations are not precise and should not be viewed as such.
For example, you have to determine the cost of a breach to your organization. The cost of a password compromise to a financial-services firm won’t be the same as the cost to a clothing retailer. You have to look at your business, assess your risk and historical losses from categories of security breaches, and determine a number that represents an accurate estimate of these costs per incident. This, of course, implies that you’re collecting and tracking these costs.
Likewise, historical data can help you determine your risk for each kind of incident. For example, what is the probability that your organization will experience an insider attack? How about an earthquake that destroys hardware and severs network connectivity?
These numbers will be unique to your organization. If you’re a high-profile target, you may discover that your risk of a certain kind of break-in is 100 percent in any given year, and you’ll probably be willing to spend more to mitigate it than an organization that faces just a 20 percent risk.
Lastly, you need to determine how much you think a security investment is going to lower your risk.
As a result, there’s always going to be some subjectivity in the numbers you use to calculate the value or benefit of security investments. Even so, these numbers have value. The value is in their consistent use across multiple projects. The same ROI method applied consistently across all projects will produce a meaningful measure of value and a means for comparing among alternatives.
For example, if you evaluate a new firewall purchase in the same way that you evaluate a security-awareness training program, you can make a rational judgment about which investment will produce a greater ROI.
Unfortunately, ROI has rarely been measured consistently for security projects in the past, if it was measured at all. Many security proposals are approved based on fears of public disclosure of customer information, regulatory compliance, or pressure to keep pace with competitors. That doesn’t mean they aren’t necessary, but it does make it difficult to gauge their worth in terms of ROI and to compare proposals against one another.
In summary, when determining the expected costs and benefits of various security investments, consistency is key, as is the quality of your risk assessment.
The timing of your ROI assessment is also important. It’s best to calculate returns before any capital is spent, although an ROI evaluation of ongoing projects could help you determine how they are progressing in comparison with other projects. In fact, it’s probably advisable to conduct follow-up assessments to make sure you’re getting the desired return on investment. But these follow-up reviews shouldn’t take the place of an initial, thorough comparison of alternative investments.
You may have given equal weight to a new firewall purchase and a training program, for example. Subsequently, you may have found that the training program produced significant positive returns while the firewall purchase did not. Some sort of prioritization should have occurred before these investments were made, so that the training program took precedence over the firewall. Did it?
Without a consistent metric, the answer could be “maybe or maybe not.” And that’s the antithesis of ROI. ROI gives us criteria for making better-informed decisions and choices.
This discussion may have made it sound as if security ROI is just like ROI for other projects. In some ways, that’s true, but in other ways, it isn’t. Economists who’ve studied security ROI have discovered that straight numbers don’t tell the whole story. According to economists Lawrence Gordon and Martin Loeb of the University of Maryland, you should never spend more than 37 percent—and often less—of your expected savings on a security investment [Gordon 02]. Above that level, it usually doesn’t pay off in the real world. There are several possible reasons for this, including the high level of uncertainty inherent in the calculations.
So, for example, if the NPV of a security-awareness training program is $200,000, you probably shouldn’t spend more than $74,000 in today’s dollars on the program.
Remember, though, that in the context of a whole business with many security projects competing for limited funds, the most important thing is not the hard numbers. The most important thing is that you use the same defined set of criteria to evaluate each potential security project. For example, it is misleading to compare an ROI calculation for a firewall purchase that does not take into account the potential for loss of reputation with an ROI calculation for an intrusion-detection system purchase that does take into account this potential loss. Even if you use NPV for both calculations, you are not measuring the same thing.
Likewise, it is misleading to compare a payback calculation for a firewall with an NPV calculation for an intrusion-detection system, even if the exact same criteria are taken into account. In this case, the inconsistency in methods results in a meaningless comparison.
This is not about hard numbers. That’s not where the value is. The value is in the ability to compare projects and choose wisely among them. Consistency is what is most important, both in what you choose to measure and in how you choose to measure it. Make your decision, and then stick with it.
Once you decide what to measure and which method to use, write it into your security policy and ensure that everyone reports out accordingly. It’s vital to maintain consistency over time. If you’ve been using NPV to evaluate security projects and you start using payback—especially if other divisions in your organization are still using NPV—you’re going to find it hard to compare new projects with ongoing ones.
Policy establishes stable guidelines. Once something is in policy, it’s harder to change it. This should hold true for ROI as much as for other aspects of security. So first, decide what you’re going to measure; second, decide how you’re going to measure it; and third, write it into your security policy.
Allen, Julia. “Making the Business Case for Information Security: Selling to Senior Management.” Carnegie Mellon University at InfoSec World 2003, March 10, 2003.
Gordon, Lawrence A., and Martin P. Loeb. “The Economics of Information Security Investment.” ACM Transactions on Information and System Security, Vol. 5, No. 4, November 2002, p. 440.
Rasch, Mark. “Cleaning Up Disclosure.” SecurityFocus, April 11, 2005.
Stephanie Losi is a graduate student in the Information Security Policy and Management Program at Carnegie Mellon University. She is working with CERT to develop security awareness content for executives and information security personnel. In addition to her work at CERT, Losi has authored online courses dealing with business ethics and served as managing editor of the E-Commerce Times. She earned her bachelor’s degree in journalism from Northwestern University.
The views expressed in this article are the author's only and do not represent directly or imply any official position or view of the Software Engineering Institute or Carnegie Mellon University. This article is intended to stimulate further discussion about this topic.
For more information
Please tell us what you
think with this short
(< 5 minute) survey.