Home > All, Security Issues > Return on Prevention Study: Measuring the value of security technologies, controls and governance practices

Return on Prevention Study: Measuring the value of security technologies, controls and governance practices

This is a study by the Ponemon Institute. Ponemon describes itself thus:

Ponemon Institute conducts independent research on privacy, data protection and information security policy. Our goal is to enable organizations in both the private and public sectors to have a clearer understanding of the trends in practices, perceptions and potential threats that will affect the collection, management and safeguarding of personal and confidential information about individuals and organizations. Ponemon Institute research informs organizations on how to improve upon their data protection initiatives and enhance their brand and reputation as a trusted enterprise.
Ponemon Institute

Well, this study certainly does that. But it is not without its problems. It claims to be a study on the return on prevention (RoP), which is a metric used by Ponemon to counter the problems that information security has with the more established concept of return on investment (RoI). RoI is valuable in demonstrating the value of a product in terms of increased revenue above the cost of the product. But security does not increase revenue; it prevents loss. So return on prevention is clearly more pertinent, and easier to understand.

My problem is this: the amount of loss prevented is an unknown since it has been prevented. So ‘return on prevention’ is a value judgement, ‘What do we think we would have lost if we hadn’t prevented it?’. But the report doesn’t explain this; indeed it defines RoP very specifically and in scientific terms thus:

RoP = ∑i = 1  [{Intrinsic value of prevention (i)} / {Investment(0)}] – {residual value of investment(n)}
For i = 1 to n periods.

Last year, Professor Dietmar Bräunig of the University of Giessen conducted a study as part of the project entitled “Quality in Prevention” in occupational health and safety. A commentary in DGUV (Deutsche Gesetzliche Unfallversicherung) states:

For the enterprises considered by the study, the value for the return on prevention is 1.6. This means that each euro invested in occupational safety and health generates a potential for increased commercial success of 1.60 euro.
Prevention pays off – all the more so during an economic crisis

The return on prevention is clearly stated: 1.6. But our Ponemon study does not give a return on prevention figure for any of the security products or methodologies it mentions. And that is because, despite its title, this is not a study on Return of Prevention; it is a study on security practitioners’ perception of the value of different security products, controls and procedures.

And in that it is enormously valuable. RoP, regardless of its scientific mathematical formula, is not an actual measurement – it is a way of looking at things. It is a sales approach. The salesman (whether that is the security vendor or in-house security staff) goes to the purse-holding Board and says, ‘don’t think in terms of how much revenue this will generate; think in terms of how much loss it will prevent’. The value of this report is that it clarifies users’ perception of what products provide value for money. But what it doesn’t do is what the title says: it does not measure the value of security technologies because the measurement of hypothetical loss is impossible.

click for full size

Four mobile security technologies that earn a very high RoP

Let me give one specific example. Figure six in the report shows ‘Four mobile security technologies that earn a very high RoP’, and the text explains

Interviews with a representative sub-sample of 44 IT practitioners revealed that investments in certain technologies (such as anti-virus & anti-malware) as a relatively low cost as measured on a per user, node or endpoint basis rather then as an enterprise-wide solution.  This view may explain, at least in part, why AV, device encryption and other point solutions that are easy to implement yield a high RoP. In contrast, enterprise security technologies that require special expertise to operate and implement yield a low RoP.3

I don’t understand the first sentence, but the second sentence states that these products ‘yield a high RoP’. They may do, but this report does not show any such thing. It shows what the participants believe offers the best value in preventing loss. It certainly does not provide a return on prevention figure. However, the information it does provide is still valuable. It demonstrates that in users’ perception, AV provides the best value for money in mobile security; followed by encryption. It doesn’t tell us that this is a fact; only that users believe it to be true.

Now, this study was commissioned by F-Secure and Vodafone. F-Secure will be cock-a-hoop: AV products are already perceived by the market to be the best value for mobile security. Encryption providers know they will have to try harder. Vodafone knows that it could concentrate on AV and encryption as a means of selling its services. Providers of biometric security products, not even visible in this report, know they have a lot to do: they have to make their products easier to use and understand, and cheaper to buy and implement.

But, I repeat, this is not a definitive study into RoP in security products. Consider again these four mobile security technologies. Are they protecting the mobile device route into the corporate server; or are they protecting the personal and corporate data held on the mobile device? The report doesn’t specify; but the security requirement for each function is very different. AV would help protect the device from being ‘owned’ and used as an entry to the corporate server; but will do nothing to safeguard the data on the device, both personal and corporate, if it is lost or stolen. Encryption would. But encryption would be less useful in defending the device as a route into the servers (other than as part of a VPN of course). Note that our hypothetical but not included biometric device could help secure both aspects by preventing anything but authorised access to the device.

Brian Burton

Brian Burton, head of IT security, Vodafone

To a degree I suspect I am nit-picking. Certainly Brian Burton of Vodafone, does not consider the report to be anything other than a survey of perceptions.

“That’s what it is,” he says. “The study is on perception; just perceptions and perspectives on security solutions – what is interesting is that right across industry these perceptions appear the same. But how our sales teams and how our professional services teams will actually build a business case with the customer for an individual product or an individual service is down to that individual customer’s perception – not an overall market perception. For example, as a big organization, we know what problems we have. If someone comes to me and says I can solve your problems, it’s down to me to work out a business case if I want to buy his products; but I can use the concept of return on prevention with my management to prepare that business case far more easily than using return on investment.”

That, I think, is good, realistic and honest; and I believe and hope that Vodafone (and F-Secure) will get benefit and value from the report they commissioned. A genuine study into return on prevention for security products would benefit the security industry enormously; but I fear this isn’t it, and it may never happen. Who would pay for it? An industry whose products might be killed off if shown to provide low or negative RoP? Or an independent university that has to watch every penny it spends? Either way, regardless of the title bestowed by Ponemon, this is not a Return on Prevention study.

The study concludes:

In conclusion, we believe the RoP can help lower these hurdles and make it easier for IT and IT security practitioners to make the business case for enabling security technologies and related control activities.

Absolutely true. But this report doesn’t make that case; and pretending it is something other than it is, helps no-one.

Categories: All, Security Issues
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s