AMTSO: a serious attempt to clean up anti-malware testing; or just a great big con?

Thesis: that anti-malware testing is ultimately meaningless
Back in February I posted a short article: Product testing: valuable or meaningless? It included the comment:

That’s not the worst of it. If the product in question is in any way anti-malware, the vendor can simply claim that the product kills 99% of all known germs. The validation process will inevitably prove it to be true and the company has a marketing bonus that is actually meaningless. Why? Because the product will inevitably be tested against the Wild List.
Exhibit One

You may take it from this that I am, or at least have been, somewhat sceptical about anti-malware product testing. I offer that article as Exhibit One.

Back in April of 2009, ESET published a list of the top ten mistakes in anti-malware testing (see panel below). You may take it from this that product testing is not as easy as it may first appear. I offer this list as Exhibit Two.

More recently, on 3 June of this year, Luis Corrons (technical director at PandaLabs) blogged about Panda’s new cloud-based anti-malware product. In this blog he includes a graphic comparing different products and different test results. His own product did very well (or he wouldn’t have posted the graphic); but that’s not the point here. Look at the test results for AVG, and imagine you are seeing them alone. Under AV-Comparatives, AVG did not do too well;  but in the PCSecurityLabs test, the same product did very well. So what conclusion should we draw? That AVG is no good or that AVG is damn good? That the AV-Comparatives test tested for samples that are not real malware and were rightly ignored by AVG (to its detriment); or that the PCSecurityLabs test was short on really testing zero-day malware and consequently gives a false impression of AVG? The point is, we don’t know and we can’t tell. I offer this graphic as Exhibit Three.

Exhibit Three: graphic from Luis Corrons' blog

And I suggest that these three Exhibits amply support my sceptical view that security product testing is ultimately pretty meaningless.

10 common testing mistakes 1. Using samples received via email or on a honeypot machine, without checking that they really are malicious software. 2. Using one of the programs to be tested to validate the samples. 3. Assuming any sample detected by two or more scanners as malicious to be valid. This may bias the test in favour of products that flag everything that meets very broad criteria as suspicious, and against products that are more discriminating and fastidious about false positives. 4. Using VirusTotal or a similar service to check the samples and assume that any product that doesn’t report them as malicious can’t detect them. This will once again give the advantage to scanners that flag everything as “suspicious”, and will also disadvantage scanners that use some form of dynamic or behavioural analysis. 5. Using the default settings for detection testing, without trying to configure each product to the same level of paranoia. 6. Using default settings for scanning speed. This may bias products that get their speed advantage by cutting corners on detection. 7. Asking vendors to supply samples. This may allow the vendor to bias the results in their own favour by including samples that other companies are unlikely to have access to, and to the disadvantage of companies who consider it unethical to share samples outside their web of trust. 8. Categorising samples incorrectly, leading to possible errors in configuration. For instance, not all products flag certain kinds of “greyware” (described by some vendors as “possibly unwanted applications” or similar) as malware by default. 9. Too much self belief. If, when testing two products that use the same version of the same engine, they score completely differently, it is unsafe to assume that there must be something wrong with the lower-scoring product. It is just as likely to be a problem with the setup or methodology. 10. Not including a contact point or allowing any right to reply. Be open in the methodology used and the objective of the evaluation, to allow others to verify. Exhibit Two

Step up the Anti-Malware Testing Standards Organization
The anti-malware industry, however, takes a slightly different view. Yes, it seems to say, there are problems with anti-malware testing. But let’s not throw the baby out with the bath water; let’s solve those problems and move forward confidently. It was from this desire that AMTSO (the Anti-Malware Testing Standards Organization) was born. Yesterday AMTSO published two new documents: Performance Testing Guidelines and Whole Product Protection Testing Guidelines. Now, depending on your standpoint, these documents are either very good, or just further evidence of the industry conspiracy designed to fool the public (you and me) into spending more money on them. AMTSO does itself no favours: membership entirely comprises of the anti-malware vendors, almost all of the leading test organizations, and a couple of well-known independent virus researchers. I would have been happier if the independent researchers at least included Joe Wells, the original founder of the Wild List and a critic of what has become of the Wild List (memo to AMTSO – go get Joe Wells to raise street cred); and even happier if there was some serious user representation. As it is, AMTSO is utterly incestuous – and the issue from in-breeding is usually sickly.

So. Am I being unfair? Negative is easy; you don’t need proof, just doubt. Positive is harder: it needs to be proven. The only solution is to go to the industry and say “I’m doubtful – prove me wrong.” I did this with three simple questions:

• Is this the anti-malware industry looking after itself?
• Is it even representative of the anti-malware industry?
• Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)

Is this the anti-malware industry looking after itself?
Definitely not, says Stuart Taylor of Sophos, and Chairman of the Board of Directors of AMTSO. “AMTSO was formed almost 3 years ago by mainly technical people to address the problem that tests being published were not giving potential customers useful information on which to base their purchasing decisions… There is plenty of oversight and contention to keep us honest.”

“No,” agrees Eric Sites, CTO at Sunbelt, “because The AMTSO also includes members such as testing organizations and independent research labs like ICSA, West Coast Labs, AV-Test, PC Security Labs, and AV-Comparatives, companies like Cascadia Labs, Dennis Technology Labs, and Hispasec (VirusTotal). AMTSO also has individual members such as Mario Vuksan, Vesselin Bontchev… AMTSO is very active now trying to sign up new reports that test anti-malware products.” I have to say that was absolutely not the case with ICSA and the Wild List a couple of years ago – but that’s a different story and things may have changed.

Eric Sites

But, yes (I can almost hear ‘of course’) “it’s the anti-malware industry looking after itself,” says David Harley, one of the world’s leading virus experts and currently with ESET. But not necessarily for negative reasons. “Testing hurts products that get bad reviews. But it’s not only about marketing and sales. Poor testing is at best irrelevant. But sometimes poor testing hurts good products and promotes not-so-good products. That’s not good for the misevaluated products, but it’s far worse for the customer.”

OK. Stupid and naive question on my part. Round One to the industry.

Is AMTSO even representative of the anti-malware industry?
“Yes,” says Sunbelt’s Eric Sites. “AMTSO has almost all independent anti-malware testing companies as members, as well as all major anti-malware vendors. Additionally, there is a big push this year to sign up new members.”

David Harley digs deeper. “There’s an important point to be made here. The relationship between anti-malware companies and anti-malware testers is more complex than you’re suggesting. Testers are not the antivirus industry, though we have a symbiotic and in some senses essential relationship. Getting representatives from both these industries onto the same team may be AMTSO’s biggest achievement to date. (Well, that and getting consensus on several major documents from a large constituency of tough-minded, hard-nosed, rugged individualists who have moulded scepticism into an art form.)

David Harley

“Don’t get me wrong: co-operation with the vendor community is one of the things that gives the mainstream testing community its edge over the noisy (semi-)amateur testers, but that community jealously guards its independence, as it should. Testers are obligated to be as accurate in their testing as humanly (or digitally) possible, but their responsibility is to their audience, not to the vendors. The stuff you may have heard about AMTSO being the vendors keeping testers on a leash is aggressive marketing and sour grapes.

“That said, the alignment of two complementary skill sets (good testers know a lot about malware/anti-malware, and good vendors do a lot of internal testing) puts AMTSO in a better position than most to inform, educate and generally try to raise standards (rather than introduce standardization, which isn’t in our game plan). And the Advisory Board, which is meant to “keep us honest” includes academics et al who are decidedly not within the AV industry.”

I’m going to class this one as a draw. I find David’s arguments compelling, but (and I know this is cheating because I used the term ‘anti-malware industry’) I would be more convinced if there was some representation of the needs of the PBI (that’s you and me again) in this process. It’s still, ultimately, AMTSO looking after its own members with its own rules. The PBI will simply have to accept what its given without any input to it.

Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)
“I would have to say yes (anti-malware works)”, says Stuart Taylor. “From within SophosLabs I can report that just the other week we were receiving 100,000 brand new files every day. Of those new files, we were detecting around 80% already, despite not having seen these particular files before. If you combine that with the other proactive technologies such as HIPs then you get very good detection and a typical user just cannot do without.”

Stuart Taylor

“Of course it works,” says David Harley, “better for some people than for others. What it doesn’t do is meet the expectations of people who think that it’s valueless if it doesn’t catch 100% of malware (not to mention a lot of other stuff that you may or may not expect it to catch, legitimately or otherwise) with 0.00% false positives, no resource or administration overhead, and at zero cost.

“It’s not the 100% solution. There isn’t one, which is why any honest researcher will tell you that you need multi-layered defences (at home or in the enterprise), not simple AV scanning. Whether you need multiple packages or a suite is not a debate I’m going to get into now.”

But is testing a con?

“Yes and No,” says Eric Sites. “Yes, if the tests are done poorly. For example, just taking a bunch of malware samples from web page and scanning them with AV command line scanners using VirusTotal gives very misguided results,  even if all of the samples are detected by every AV engine.  Testing like this does not show that these products will stop a single piece of malware if tries to infect end users’ computers.

“No, because there are some tests that are done using best practices that really do show the quality and effectiveness of anti-malware products.”

David Harley is as forthright as he’s been throughout. “It’s not a con. It’s a partially successful attempt to solve the insoluble.

“Anti-malware testing isn’t controlled by the vendor industry. If it were, the world would be a very different place. Actually, I think you have to distinguish between (at least) two distinct types of testing.

“Comparative testing is based on a number of assumptions that don’t always hold up. For one thing, that testers are better at gathering (and, more importantly, validating and classifying) samples than vendors. Or, in terms of other types of testing, that they’re better at assessing ergonomic feasibility, performance, resource usage and so on. (In the second case, at least, that might sometimes be so, but I see no reason to assume that it’s always the case, and every reason not to.)

“For another, that a given detection test uses a sample set that is an accurate reflection of the totality of malicious programs currently in the world. That’s unprovable, so we have to accept best endeavours and reasonable competence instead of quantification: the AMTSO review analysis process could be described as a way of assessing whether a given test or test report meets those standards by comparing it to the AMTSO “Fundamental Principles”. And yes, that’s turning the testing process on its head: I see it as essential that the testing industry sees itself as accountable to its audience. If at the moment the most feasible way for the industry to acknowledge that is to have its temperature taken by an organization with quite a few vendor members, so be it. I’m open to (and have suggested) other approaches, but this is the best we have right now (in my not so humble opinion).

“Certification testing isn’t based on finding the “best” product, but on establishing a baseline value for consistently acceptable performance. Personally, I think that’s a healthier, or at any rate more attainable approach. It’s less about magnifying small variations in performance, or isolating layers of protection artificially, in order to establish clear winners and losers. If there was less margin for error in testing methodology, that would be less of a problem, but in many cases the error margin is quite large enough to invalidate any pseudo-statistical analysis.”

So, for my final question, I think the industry wins again. AV software works most of the time. AMTSO is here, at least in part, to remove the possibility of ‘con’ from the testing.

Takeaway
A final word from David. “It isn’t in AMTSO’s charter, but I think the take-home message I’d want people to carry away from this interview is this:

“Testing (any software testing) is harder than you probably think it is. Testing security software has its own special problems: it’s a highly dynamic technology countering another highly dynamic technology, and the criminal nature of the threat means that the software is even more “black box” than software that’s simply trying to obviate Intellectual Property theft. No product and no tester has a monopoly on truth, and AMTSO doesn’t have all the answers. What it does have is the attention of a group of people who would like to see the right questions asked when a product, or a tester, comes under scrutiny.”

But as the author of this piece, I reserve unto myself the absolutely final word. My opinion has been modified, but not totally converted. I can see that there are some altruistic people within AMTSO, and I can see that the aim is laudable: to develop testing procedures that will allow the buyer to have confidence in the tests and the products he buys. But the fact remains that AMTSO is almost entirely composed of AV software vendors (who need to sell their products), and AV testers (who almost entirely get their funding from the vendors who successfully sell their products (either directly or indirectly in the form of magazine advertising). This is not good for the outsiders’ perception of AMTSO. It may be that this is the next big thing for AMTSO to address: better presentation with the inclusion of a voice from the PBI (that’s you and me, remember).

UPDATE

Pedro Bustamente, Senior Research Advisor at Panda Security, also comments:

Pedro Bustamente

Is this the anti-malware industry looking after itself?
Actually it’s quite the contrary. The objective of AMTSO is to advance the current state of test methodologies so that they become a better representation of real-life scenarios. Currently most AV tests either only test a portion of anti-malware products or do so with outdated methodologies. The objective of AMTSO is to develop tools so that independent testers and consumers alike can be informed and distinguish between ìa good test vs. a bad test.

Is it even representative of the anti-malware industry?
There’s quite a lot of non-vendor members in AMTSO, from academia, testing labs, magazine publishers, external board of advisors, etc. I’m not sure what the balance is but there is quite a lot of non-AV-vendor members.

Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)
You’d be surprised how much the AV industry does not like AV testing. But the reason is not because it is a massive con, but rather because most tests are not really representative of real-life situations.

UPDATE

Alice Decker, Senior Threat Researcher at Trend Micro Deutschland, also comments:

Is this the anti-malware industry looking after itself?

Alice Decker

This is what people like to believe. But, surely every organization looks after itself – for example, aren’t the IPI (International Press Institute), APWG (Anti-Phishing Working Group), ISO (International Organization for Standardization), WToO (World Tourism Organization) all looking after themselves in a wider sense?

AMTSO is an organization which brings together software vendors and software testers, establishing bridges of communication and knowledge exchange. The inter-vendor competition did not disappear through the establishment of AMTSO, but the industry’s passion for protecting their customer’s provides a strong ethical basis. Participating security vendors truly support independent testing methodologies and AMTSO presents computer security/protection in a fair and understandable way for users. However, to really successfully carry the message to users, one key element is currently missing: the fair involvement of the Magazines.

Is it even representative of the anti-malware industry?
Yes, it is representative, and yes, it is currently a small percentage of the whole industry. However, AMTSO is open to new members from the anti-malware Test, Vendor or Media branches.

Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)
Testing is not a con.  In most cases, the testers (specialist individuals or companies) are the results provider for the Magazines.  Meanwhile, the vendors compete against each other – so by working with testers and improving their protection, our customers are also better protected.

Also, the testers control the test environment, and work to deliver an impartial test.  Trend Micro believes that these tests need to evolve to deliver real-world style testing, to help demonstrate those products that protect against real-world threats.

Does it work? Does Police-concept work in the real world? Sure, it does.