“Our little gnomes in the backroom,” says the excellent Shadowserver in an announcement headed ‘New AV Test Suite’, “have been working feverishly for the last several months to put the finishing touches on our new Anti-Virus backend test systems.”
Malware testing, as we know, is a tricky business. AMTSO, the Anti-malware Testing Standards Organization, has expended much energy and expertise in developing detailed methodologies designed to ensure fair, unbiased and accurate anti-virus tests. But do we get this from Shadowserver? Do we get a new AV comparison source that we can realistically access for accurate unbiased information on the different AV products available to us? Let’s see.
Shadowserver starts off with a fair comment.
No single vendor detects 100%, nor can they ever. To expect complete protection will always be science-fiction.
That being said, it goes on…
…you can see the different statistics of the different vendors in our charts.
Here’s a couple of examples.
The one thing that really leaps out here is that Panda apparently misses (shown in green) far more of the test samples than Avira. This is counterintuitive. Panda is a commercial product backed by one of the world’s leading security companies. Avira, which I personally trust sufficiently to use on my XP netbook, is a free product. Shadowserver provides a partial answer:
The longest running issue has been our inability to use Windows based AV applications. We can now handle that, however it is still not what you might buy for home or commercial use. We are utilizing a special command-line-interface version from each of the vendors that we are using. This is not something you can purchase or utilize normally. These are all special version but most of them do use the same engines and signatures that the commercial products use.
This is important. Luis Corrons, technical director at PandaLabs elaborated:
What ShadowServer does is not an antivirus test. As they say, they do not even use commercial products, but special versions. Furthermore, it is static analysis of files they capture. It is a statistic. But the data cannot be used to say “product x detects more than product y” or “product x detects this percentage” as they are not using any of the other security layers used in real products (behavioural analysis/blocking, firewall, URL filtering, etc). The most you can say with this system is product x was able to detect y percent of files using their signatures and heuristics (the oldest antivirus technologies).
This is important. The AV companies have long recognised that the original signature database solution to malware cannot match the speed with which new signatures are required for polymorphic virus families. So they have supplemented their signature detection with more advanced and sophisticated methodologies.
In our case (Panda) ShadowServer is using an engine which is a few years old (at least 5) and of course is not using the cloud, so I can guarantee that our results are going to be awful. We have been asking SS for years to use a new version, but they were not supporting Windows. Now that they are supporting it, they forgot to mention it, but it’s not a problem as we’ll be sending them a new version with cloud connection. Anyway, even though in that way the results will be way better, or even if we are the number 1 vendor, that doesn’t mean anything, as it is only a static analysis of some files.
One solution would be for Shadowserver to work more closely with AMTSO. Shadowserver is not currently a member of AMTSO. I urge it to join. And I urge AMTSO to waive all membership fees so that this non-profit free service organization can do so. Both parties would benefit enormously. In the meantime, I asked David Harley, a director of AMTSO and research fellow at ESET, for his personal thoughts.
Shadowserver has never been discussed within AMTSO, that I remember… In the past they’ve shied away from suggesting that their statistics are suitable for direct comparison of vendor performance. One of the reasons they cited for that is that their testing has been focused on Linux/gateway versions, and you can’t assume that desktop versions will perform in the same way across a range of products. Including some Windows products will make a difference in that respect, but I can’t say how much, because I don’t know which versions they’re using. Where gateway products are used, it’s unlikely that the whole range of detection techniques are used that an end-point product uses. Detection is often dependent on execution context, certainly where detection depends on some form of dynamic analysis. A gateway product on an OS where the binary can’t execute may not detect what its desktop equivalent does, because the context is inappropriate. On the other hand, the gateway product’s heuristics may be more paranoid. Either way, there’s a possibility for statistical bias…
This isn’t a criticism of Shadowserver, which does some really useful work. I just don’t think I could recommend this as a realistic guide to comparative performance assessment…
Neither Luis nor David are known to shy away from the truth, whether of themselves or their products. But both seem fairly clear: Shadowserver is good; but this service is not yet ready. Shadowserver’s AV test suite will not give a realistic view of different AV products’ actual capabilities. Not yet. It needs more work. I’m certain that will happen. But for the time being at least, don’t use Shadowserver’s statistics to form an opinion on the relative merits of different AV products.
UPDATE from Shadowserver
It is difficult to not compare one vendor to the next due to how we have the data
structured on the pages. It would be impossible not to try and derive conclusions
from those results. While that is the case, our goal is not to create a real
comparison site for everyone to try and compete to see which AV vendor is better
than the next…
That is not our purpose…
That being said, our purposes in doing AV testing is simple. We wanted to know what
each malware was supposed to be for categorization purposes, and of course just to
see what happened. We collect a lot of malware daily and trying to find ways of
tying our data together is important.
Because we are volunteers and a non-profit we really enjoy sharing what we find
no matter how odd. We even enjoy talking about when we screw something up or
when we encounter something exciting. Everything here is for you our public to
enjoy, discuss, and even criticize…
Shadowserver, 8 September.
Last year I voiced two main concerns about AMTSO, the anti-malware testing standards organisation. One was collusion in the false marketing impression given by claims of 100% test success against malware in the Wild(list). I won’t repeat my concerns here (see instead the original articles AMTSO: a serious attempt to clean up anti-malware testing; or just a great big con? and Anti Malware Testing Standards Organization: a dissenting view). Sadly, there has never really been any acknowledgement that this is a valid concern; nevermind any action on it.
The second concern is that AMTSO is effectively a closed shop: it is largely by the industry for the industry; and for that reason alone it cannot be trusted. This caused no inconsiderable heat, with some members of AMTSO feeling that I was saying that they personally could not be trusted. Others, however, accepted that it was a valid issue.
Well, I am now delighted that AMTSO has made serious attempts to address the problem. Last October it announced a new low-cost subscription fee in an attempt to get more people involved:
While AMTSO recognizes that strict requirements for full membership are necessary to ensure it achieves its objectives, it also understands that the fees put it out of reach for many interested individuals that may have a valuable contribution to improving the objectivity, quality and relevance of testing methodologies. Hopefully, the new low cost subscription model will widen the reach of the organisation and enable more people to have a say in the future of anti-malware testing.
Philipp Wolf, of AMTSO member Avira
This new subscription currently stands at €25 per annum. I don’t know how many subscribers it has attracted – but I doubt that it is many. “They will also have the right to attend meetings, though not as voting members.” Why should I pay money to have no ultimate say in things?
Today, however, AMTSO has launched an open (and free!) “forum where anyone may post and join in testing-related discussions.” Users are still unable to vote on AMTSO issues, but that’s fair enough. Discussions, like justice, should be seen to be done. Provided that AMTSO moderators do not censor this discussion forum (other than the usual legal requirements), it will “provide a discussion point where anyone with a question or an opinion on the testing of anti-malware software can make their voice heard.”
For that, AMTSO deserves to be commended.
I must say that I have known and respected David Harley for many years. I still respect him enormously. Indeed, I respect the anti-virus industry in general. So, having said that, a quick response to David’s article. In it, he says:
That brings us to Kevin Townsend, who could never be described as AMTSO’s best friend…
…But then I realized that he might have been misled by this statement on the AMTSO home page…
I would just like to say that I don’t believe that I have been misled. I know that AMTSO is open to the people I would like to see within it. But the fact is they are not in it. So my point is simply that AMTSO needs to go and get them. Just saying that “AMTSO membership is open to any corporation, institution or unaffiliated individual interested in participating in this organization” is not in itself sufficient if they don’t join. However, as soon as AMTSO has sufficient representation from within the AV user community, it will gain the credibility it deserves.
The first I heard of it was a Tweet from Luis Corrons towards the end of last week:
A bit cryptic, but the reference to me is almost certainly in relation to (one of) my two previous criticisms of AMTSO: firstly that its membership is almost entirely incestuous and that without involvement from outside of the industry its recommendations cannot be trusted; and secondly that use of any testing that is allowed to suggest 100% efficacy against viruses in the wild is disingenuous (see AMTSO: a serious attempt to clean up anti-malware testing; or just a great big con? and Anti Malware Testing Standards Organization: a dissenting view).
But now AMTSO itself has released details (this is on the former criticism, not the latter):
The Anti-Malware Testing Standards Organization (AMTSO, www.amtso.org), an international organization that encourages improved methodologies for testing security programs, announced today the imminent availability of a new subscription model that will open up membership to a wider audience.
Neil Rubenking, Lead Analyst at PC Magazine, commented, “As a member of AMTSO’s Advisory Board I’ve been privileged to interact and work with the group’s members and committees. AMTSO membership is open to individuals, but the 2,000 €/year price puts full membership out of reach for all but the most dedicated. The new subscription model will now allow all interested parties to make a marked contribution to the development of better testing methodologies.”
The new membership model will apparently cost just €20 (presumably per year), and is clearly a move in the right direction. But from the released information you don’t seem to get much for this. You get access to the
…educational resources that are already freely available on the AMTSO website [and] the development of documentation and participation on AMTSO’s email discussion boards, where some of the world’s foremost experts in the anti-malware industry and the testing industry leave vendor bias aside, in order to pursue lively conversations on the intricacies of malware testing, its fallacies and real-world ways in which to improve it.
In short, you get access to a specialist mailing list, and “the right to attend meetings, though not as voting members.”
I don’t want to sound churlish, because this is a major movement for AMTSO, but you get to speak your mind with no guarantee that anyone will listen, and certainly no say in what AMTSO actually does. It is nowhere near what I personally would like to see: the recruitment of senior technicians from some of the major corporate AV users; with full voting rights. If this simply isn’t possible, perhaps AMTSO could tell us why?
So, all in all, a tiny step in the right direction.
Well, I see Dan Raywood of Secure Computing magazine has entered the discussions on AMTSO (see here); and has included a link to the article where I allow AMTSO members to speak freely, but not one to my critical article. I would have been happier if he had acknowledged the connection (past or present) between SC and West Coast Labs (a member of AMTSO). Not doing so does nothing to minimize concerns.
Anyway, the article gives David Harley (of ESET and AMTSO) a pretty free hand in describing and answering some of the recent criticisms of AMTSO.
He claimed that he would not have made an investment of time and energy if he did not believe that there is a need for major improvements in testing and the public understanding of testing.
And of course he’s right. My concern is not with the intention of AMTSO but with the structure of AMTSO. Where is the voice of the user?
He highlighted three problems – firstly while AMTSO is not a profit-making organisation, the subscription fee is fairly hefty.
Then reduce it. ‘Costs’ is not an adequate reason. If you look at the membership list of AMTSO, it is the companies not the individuals that are listed. And the response to negative criticism put out in a co-ordinated reply was done on company not private blogs. So the AV companies are involved. Remember the ash cloud flying problems? Sophos had about 60 staff caught up in Eastern Europe at the time. Their response? To hire a rather large private jet to get people home. The AV companies have the money to solve this – but they’re trying to appear at arm’s length. Frankly, it doesn’t wash. So where is the voice of the user?
That gives rise to another issue. Since we all have full-time jobs, we can’t give AMTSO the time and attention some of us would like to…
Another non-wash. If AMTSO members haven’t got the time to do it right, don’t do it at all. And like I said, it’s the companies that are listed as members, not the individuals.
The second problem is that the group includes security vendors, as well as testers and product certification agencies. Harley admitted that while mainstream vendors and testers do not necessarily see that as a problem, most people do not see it that way, rather they see it as the foxes guarding the hen house.
Precisely so. But the answer is simple. Get some users into AMTSO.
So how could standards be raised in a more general sense? Harley said that this would be by improving the quality and availability of information about tests and testing, and by making testers more accountable for the accuracy and quality of their testing.
I have no problem with this. In fact I have no problem with AMTSO, and have great respect for David Harley personally. What AMTSO says it is doing is a very good thing, and I think it is making a fair job of it. But the fact remains that without input from the users of AV products it cannot be taken seriously. If this means that the AV companies have to come off the fence, admit their involvement and put some serious money into the organisation, then so be it. But it must recruit from the users of AV products.
I had hoped that I need say no more about AMTSO – at least for a while. But I have to say something about its latest comments signed by several members and posted simultaneously to multiple blogs. First some background. I wrote my first article and allowed AMTSO members to express their views freely. It subsequently seemed, and I was so warned, that some areas of AMTSO were taking this article as my approval of the organization, even though it expressed some of my reservations.
I subsequently, and consequently, wrote a second article to outline my opinions. The second article was more forceful than the first: it is sad but true that when talking to industry, you need to shout to be heard. But I would have been content with this: to let readers see the views of AMTSO in the first article and my own in the second; and then come to their own conclusions. AMTSO clearly has a right to respond, and has done so in comments to the second article and individual blog postings elsewhere by Andrew Lee and David Harley. Kurt Wismer, not a member of AMTSO, has also responded.
Now a group of AMTSO members has published a new coordinated blog across many sites, and I feel that I need to respond to that. This piece has, in its first paragraph:
Given some recent negative publicity aimed at AMTSO (example), we want to collectively clarify the following points on behalf the anti-malware industry, where we come from, and indirectly on behalf of AMTSO.
Testing and Accountability
The ‘example’ link points to my ‘dissenting’ article. It is the only critical article referenced. It is reasonable to assume, therefore, that this posting is meant as a rebuttal to my article – and AMTSO is perfectly entitled to do so. The problem is that AMTSO defends areas that I have not, and would not, criticise. The effect of this is to suggest that I am unreasonable and vindictive. I would therefore ask that readers of this AMTSO post look again at what I actually wrote.
You will see that my only criticism of the anti-malware industry is that it sometimes misleads the market by allowing the suggestion that 100% detection of viruses ‘in the Wild(List)’ is the same as 100% detection of viruses in the wild. In the same article I point out how valuable and necessary the anti-malware industry is. My criticism of AMTSO is that it does not censure this practice.
Apart from this, my comments point to just one criticism with a simple solution: AMTSO lacks credibility because it is the industry laying down rules for itself. (In comments to my article, Mark Kennedy, one of the signatories to this AMTSO piece disagrees. His view is that AMTSO is credible because its work is credible. My view is that its work lacks credibility because AMTSO lacks credibility.) But the solution is very, very simple. AMTSO should include members taken from the customers of the anti-malware industry. This would give AMTSO credibility; and that credibility would allow its work to be credible.
So this is my problem with this AMTSO posting. It states that given some recent negative publicity aimed at AMTSO by me, it wants to collectively put the matter straight. It then goes on to list a series of points that are either irrelevant to me, or to which I am in whole-hearted agreement. The implication, and these people are more than clever enough to know this, is that my criticisms are trivial. It is a clever way of dismissing me and praising themselves as being super-reasonable folks.
I would ask five things of readers:
- read my original articles and see what I actually did say before believing what I am accused of saying
- search this blog for ‘PandaLabs’ (one of the signatories of the AMTSO post) to see how unreasonably anti the AV industry I really am
- search this blog for ‘David Harley’ (one of the signatories of the AMTSO post) to see how prejudiced I am against AMTSO members
- ask yourself why, when they say they are responding to negative criticism from me, do they not even mention the only two criticisms I actually make. Why is the WildList advertising sacrosanct? Why are there no users within AMTSO? Solve these two issues and I, for one, have no other criticism
- and finally, make up your own mind: be manipulated neither by me nor anyone else.
There have been a few responses to my latest article on AMTSO; for example
- AMTSO revisited, Kurt Wismer
- I AMTSO confused…, David Harley
It’s worth reading them for an alternative view to mine; and if anyone comes across others, please add them as a comment below.
I would like to add only one point. Would you be happy with a government that said to you, you don’t have the intelligence or knowledge to understand things; so we, the government, the police and the judiciary, are going to tell you how things are and how things are to be done and you don’t have any say in it?
The principle, I repeat, the principle, is exactly the same. For AMTSO to be trusted in the good things it claims to do, it must be ultimately subject to the voice of the user.
I feel I need to make a comment about the article Anti Malware Testing Standards Organization: a dissenting view. There is a war of words in the comments; which is a good thing. But I’m afraid that my own message might get lost in all this. It is:
- the anti-malware industry is a good and necessary thing to keep us as safe as possible on the internet
- I have the highest regard for the technical people in that industry: no-one can doubt that the internet is a safer place for the work of people like Mikko Hypponen, Chet Wisniewski, David Harley, Luis Corrons, Graham Cluley, Rik Ferguson and all the others
- my concern is the way in which that industry markets itself
- use of the WildList in testing allows the industry to claim 100% success against viruses; and this is dangerously misleading and should be stopped
- the anti-malware testing industry is a parasite (I mean this biologically, not insultingly) on the anti-malware industry
- I do not believe that the anti-malware testing industry can tell us very much about the anti-malware products
- the declared intention of AMTSO, to put trust and confidence and accuracy into anti-malware testing, is good
- the structure of AMTSO, an incestuous relationship between most vendors and most testers with no inclusion of users, is bad: it is too open to abuse and misuse
- the argument between Sophos and NSS is irrelevant; if not them now, it will be other protagonists in the future: this is inevitable
- AMTSO should be dissolved. A new organization with user companies at the heart, funded by the anti-malware industry and with the same intent, should be deployed. The anti-malware industry and the anti-malware testing industry should have representation on that new organization, but no controlling influence.
On June 15 I posted the article AMTSO: a serious attempt to clean up anti-malware testing; or just a great big con? The purpose of the article was to look at AMTSO, the Anti Malware Testing Standards Organisation; and I invited AMTSO members to justify themselves. Now I want to give a dissenting view, largely my own, and to look at AMTSO from outside of the tent. I shall be asking two principal questions:
- is AMTSO serious about improving the value of anti-malware testing?
- who does AMTSO serve?
Is AMTSO serious about improving the value of anti-malware testing?
I recently blogged about two new threats discovered in the wild by M86 Security: Asprox returns: fast-flux SQL injection attack; and Skype: old vulnerability, new exploit – in the wild. In both cases, M86 ran the malware they had discovered against VirusTotal (a respected site you can use to see what anti-malware products make of any submitted file). For the former, VirusTotal showed that only 7 out 42 anti-malware products detected the Asprox malware; while for the latter, only one AV product out of the 42 detected the Skype malware.
This would seem at odds with all of those marketing claims we see from the anti-malware industry, which state that their particular product detects between 97 and 100 per cent of all malware in the wild. An example is the VB100 award issued by VirusBulletin, one of the leading anti-malware test organisations. In VB’s own words:
The VB100 award was first introduced in 1998. In order to display the VB100 logo, an anti-virus product must have demonstrated in our tests that:
- It detects all In the Wild viruses during both on-demand and on-access scanning.
- It generates no false positives when scanning a set of clean files.
The product must fulfil these criteria in its default state.
I cannot think of a single anti-malware product that doesn’t boast similarly high scores, if not from VB, then from ICSA or West Coast Labs. But Virus Bulletin and VirusTotal cannot both be right. Well, the explanation is in the Virus Bulletin statement ‘in the wild’. It contains a link to this:
The WildList Organization collects monthly virus reports from anti-virus experts around the world. The data from the reports are compiled to produce The WildList – a list of those viruses currently spreading throughout a diverse user population. A virus that is reported by two or more of the WildList reporters will appear in the top-half of the list and is deemed to be ‘In the Wild’.
In recent times, the list has been used by Virus Bulletin and other anti-virus product testers [such as ICSA and West Coast Labs] as the definitive guide to the viruses found in the real world.
So, ‘in the wild’ is actually a sub-set of the viruses that are actually ‘in the wild’: it means only those viruses that are included in the WildList’s list of those viruses it has found in the wild. It gets worse.
- the WildList requires submission of a virus sample from at least two separate researchers
- many of the researchers are the anti-virus companies themselves
- in-built latency within the process can mean that it can take 3 months from the detection of a new virus to its inclusion within the WildList being used in a test
- this latency means that, almost by definition, the Wild List includes little, if any, of the biggest threat to end-users: zero-day malware
- members of the WildList Organization get to see the WildList when it is published; and yes, that includes the majority of AV companies
So what does this all mean? It means that the WildList is not a list of viruses in the wild, but a list of the majority of viruses that were in the wild several months ago. It means that the anti-virus test is against a set of viruses that the anti-virus companies already know about. It means that anything less than 100% success against the WildList is probably down to incompetence in the anti-virus company. It means that the average anti-virus buyer is being conned about the true situation.
So the answer to my first question, is AMTSO serious about improving the quality of anti-malware testing, is ‘no’. It would not allow the use of a test process, by its own members, that so clearly misleads the public if it were.
Who does AMTSO serve?
Let’s not prevaricate: the question is ‘does AMTSO serve the anti-malware user, or itself, the anti-malware industry?’ To answer this question I’m going to look at two things: the AMTSO Fundamental Principles of Testing, and the application of those principles by its Review Board.
The very first principle, headlined Testing must not endanger the public, includes the categoric statement: “In addition, new malware must not be created for testing purposes.” Why not? How can you test the true heuristic behavioral capabilities of an AV product without testing it against a brand new sample that you absolutely know it has never experienced before? To include this restriction under the banner of not endangering the public is also misleading: there is nothing essentially incompatible between developing a new virus and keeping the public safe.
I am not alone in being puzzled by this. Ed Moyles from SecurityCurve is similarly surprised:
Yes, yes… it’s terrible to create new malware – completely unethical. Yup, under any circumstances. Even if it doesn’t leave the lab, even if it doesn’t replicate, and even if it doesn’t have a hostile payload. Yep – still terrible. We know this because shady, fly-by-night organizations like Consumer Reports, University of Calgary, or Sanoma State are always springing up like mushrooms. Their clear intent is to bring down the Internet, wreak havoc, and otherwise mock everything that is just and holy… Sigh. I just can’t get my head around the argument.
SecurityCurve, June 16th, 2010
The problem for AMTSO is that there is one very obvious reason that comes to mind. Could it be that inclusion of new samples would increase the number of ‘fails’ in the test, and thereby lower the success rate so beloved by the industry for marketing purposes? AMTSO could respond that it isn’t a real ‘fail’ since the malware doesn’t actually exist; but as a user I would reply that it is more important to get an idea on how the product might respond to zero-day threats. So is this an example of AMTSO looking after itself?
Let’s move on to the Review Board. There are at the time of writing just two reviews: one on a Dennis Technology Labs test report, and one on an NSS Labs report. One AMTSO review is favourable and the other is not. I do not know enough about either of the testing companies or their test methodologies to comment on the reports themselves, but I think it is illuminating to compare the framework of the AMTSO reviews.
Dennis Technology Labs is a member of AMTSO. The testing was paid for by Symantec, a member of AMTSO. Symantec performed very well in these tests. The review of the report by AMTSO was requested by Dennis Technology Labs. The review was favourable. The test report is effectively endorsed.
NSS Labs is not a member of AMTSO (although it used to be). The testing was paid for by NSS independent of any anti-malware vendor (in the hope of recouping the cost via sales of the report). Sophos performed very badly in the report. The review of the report by AMTSO was requested by Sophos. The review was not favourable. The test report is effectively dismissed.
What does this look like? To me it looks like a duck; and If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck. AMTSO has its say about the NSS test report in its published review. I asked Rick Moy, President of NSS Labs, for his view of the AMTSO review. On AMTSO itself, he commented, “I have had drinks and long discussions with 90% of the folks in AMTSO. There is some very old-school thinking afoot, and a fair amount of protectionism. While they have good intentions, there is probably just too much business interest being represented.”
But what about their review of his test report?
Every vendor reviewed the methodology before. In fact I had sent it to them in 2008 and solicited comments before running the test. Every vendor but Sophos cooperated and gave us software and reviewed settings of the products. None complained about the methodology… But when the results came out, folks from AVG, ESET, Symantec and especially Sophos went crazy.
I cooperated for months of craziness. They all essentially demanded we give them free consulting and tear through samples to find what was wrong with our test. Well, it was a real-world test of fresh malware that had not been shared around amongst the vendors, that simple. Sophos even made brazen false claims that we had not contacted them. After much harangue, we produced email correspondence with the chairman of AMTSO and Lab Director at Sophos showing that we had, multiple times, and even reversed samples with them to help them troubleshoot. No sanctions or reprimand was made. Instead they redoubled their efforts to discredit the test.
Rick Moy, President, NSS Labs
So the answer to my second question, who does AMTSO serve, is that it serves the anti-malware industry: it is self-serving. In fairness, it rarely claims to be in the best interests of the user (except when it is trying to justify its guidelines). There are no user members, and it is not open to users: “AMTSO membership is open to academics, reviewers, publications, testers and vendors, subject to guidelines determined by AMTSO.” But in that case, it should keep itself to itself, and not send out press releases nor make its website nor its judgments available to users.
There are three main conclusions I draw from this look at AMTSO.
Firstly, the biggest problem I have with AMTSO is that it declares itself to be the sole arbiter of what is good in anti-malware testing: it is the prosecutor, judge and jury. I find this intensely arrogant. The sole judge of a test should be the user. The tester has to prove to the user that the tests are valid. If the vendor objects, he has to prove to the user that the tests are invalid. The idea that the vendor has only to prove his case to other vendors with identical vested interests is patently absurd and would be dismissed in any other industry.
Secondly, if AMTSO was serious about setting and maintaining testing standards for anti-malware products in accordance with its own charter, it would ban the WildList in its current form. WildList testing is dangerous. Users who buy security on the basis of ‘detects ALL viruses in the wild’ are likely to believe that they are completely safe from viruses when they most certainly are not, and might consequently behave less carefully on the internet.
And thirdly, AMTSO should immediately recuse itself from the purpose of setting anti-malware testing standards until, and unless, an open, independent, user-centric body can be established. To this body, the vendors should have every right to make representation; and to this body, the testing industry (separately) should have every right to make representation. Only then are we likely to have anti malware testing standards that are independent, valid and trustworthy.
I have no beef with any of the anti-malware companies. They are essential to our security; and we all, every one of us, must have at least one of their anti-malware products installed on our computers for our security. I have no beef with any of the individuals within AMTSO. They all have far greater knowledge of threats and solutions on the internet than do I. My beef is with AMTSO itself. It is, in its present form, a stain on an otherwise excellent industry.
Thesis: that anti-malware testing is ultimately meaningless
Back in February I posted a short article: Product testing: valuable or meaningless? It included the comment:
That’s not the worst of it. If the product in question is in any way anti-malware, the vendor can simply claim that the product kills 99% of all known germs. The validation process will inevitably prove it to be true and the company has a marketing bonus that is actually meaningless. Why? Because the product will inevitably be tested against the Wild List.
You may take it from this that I am, or at least have been, somewhat sceptical about anti-malware product testing. I offer that article as Exhibit One.
Back in April of 2009, ESET published a list of the top ten mistakes in anti-malware testing (see panel below). You may take it from this that product testing is not as easy as it may first appear. I offer this list as Exhibit Two.
More recently, on 3 June of this year, Luis Corrons (technical director at PandaLabs) blogged about Panda’s new cloud-based anti-malware product. In this blog he includes a graphic comparing different products and different test results. His own product did very well (or he wouldn’t have posted the graphic); but that’s not the point here. Look at the test results for AVG, and imagine you are seeing them alone. Under AV-Comparatives, AVG did not do too well; but in the PCSecurityLabs test, the same product did very well. So what conclusion should we draw? That AVG is no good or that AVG is damn good? That the AV-Comparatives test tested for samples that are not real malware and were rightly ignored by AVG (to its detriment); or that the PCSecurityLabs test was short on really testing zero-day malware and consequently gives a false impression of AVG? The point is, we don’t know and we can’t tell. I offer this graphic as Exhibit Three.
And I suggest that these three Exhibits amply support my sceptical view that security product testing is ultimately pretty meaningless.
10 common testing mistakes
1. Using samples received via email or on a honeypot machine, without checking that they really are malicious software.
2. Using one of the programs to be tested to validate the samples.
3. Assuming any sample detected by two or more scanners as malicious to be valid. This may bias the test in favour of products that flag everything that meets very broad criteria as suspicious, and against products that are more discriminating and fastidious about false positives.
4. Using VirusTotal or a similar service to check the samples and assume that any product that doesn’t report them as malicious can’t detect them. This will once again give the advantage to scanners that flag everything as “suspicious”, and will also disadvantage scanners that use some form of dynamic or behavioural analysis.
5. Using the default settings for detection testing, without trying to configure each product to the same level of paranoia.
6. Using default settings for scanning speed. This may bias products that get their speed advantage by cutting corners on detection.
7. Asking vendors to supply samples. This may allow the vendor to bias the results in their own favour by including samples that other companies are unlikely to have access to, and to the disadvantage of companies who consider it unethical to share samples outside their web of trust.
8. Categorising samples incorrectly, leading to possible errors in configuration. For instance, not all products flag certain kinds of “greyware” (described by some vendors as “possibly unwanted applications” or similar) as malware by default.
9. Too much self belief. If, when testing two products that use the same version of the same engine, they score completely differently, it is unsafe to assume that there must be something wrong with the lower-scoring product. It is just as likely to be a problem with the setup or methodology.
10. Not including a contact point or allowing any right to reply. Be open in the methodology used and the objective of the evaluation, to allow others to verify.
Step up the Anti-Malware Testing Standards Organization
The anti-malware industry, however, takes a slightly different view. Yes, it seems to say, there are problems with anti-malware testing. But let’s not throw the baby out with the bath water; let’s solve those problems and move forward confidently. It was from this desire that AMTSO (the Anti-Malware Testing Standards Organization) was born. Yesterday AMTSO published two new documents: Performance Testing Guidelines and Whole Product Protection Testing Guidelines. Now, depending on your standpoint, these documents are either very good, or just further evidence of the industry conspiracy designed to fool the public (you and me) into spending more money on them. AMTSO does itself no favours: membership entirely comprises of the anti-malware vendors, almost all of the leading test organizations, and a couple of well-known independent virus researchers. I would have been happier if the independent researchers at least included Joe Wells, the original founder of the Wild List and a critic of what has become of the Wild List (memo to AMTSO – go get Joe Wells to raise street cred); and even happier if there was some serious user representation. As it is, AMTSO is utterly incestuous – and the issue from in-breeding is usually sickly.
So. Am I being unfair? Negative is easy; you don’t need proof, just doubt. Positive is harder: it needs to be proven. The only solution is to go to the industry and say “I’m doubtful – prove me wrong.” I did this with three simple questions:
- Is this the anti-malware industry looking after itself?
- Is it even representative of the anti-malware industry?
- Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)
Is this the anti-malware industry looking after itself?
Definitely not, says Stuart Taylor of Sophos, and Chairman of the Board of Directors of AMTSO. “AMTSO was formed almost 3 years ago by mainly technical people to address the problem that tests being published were not giving potential customers useful information on which to base their purchasing decisions… There is plenty of oversight and contention to keep us honest.”
“No,” agrees Eric Sites, CTO at Sunbelt, “because The AMTSO also includes members such as testing organizations and independent research labs like ICSA, West Coast Labs, AV-Test, PC Security Labs, and AV-Comparatives, companies like Cascadia Labs, Dennis Technology Labs, and Hispasec (VirusTotal). AMTSO also has individual members such as Mario Vuksan, Vesselin Bontchev… AMTSO is very active now trying to sign up new reports that test anti-malware products.” I have to say that was absolutely not the case with ICSA and the Wild List a couple of years ago – but that’s a different story and things may have changed.
But, yes (I can almost hear ‘of course’) “it’s the anti-malware industry looking after itself,” says David Harley, one of the world’s leading virus experts and currently with ESET. But not necessarily for negative reasons. “Testing hurts products that get bad reviews. But it’s not only about marketing and sales. Poor testing is at best irrelevant. But sometimes poor testing hurts good products and promotes not-so-good products. That’s not good for the misevaluated products, but it’s far worse for the customer.”
OK. Stupid and naive question on my part. Round One to the industry.
Is AMTSO even representative of the anti-malware industry?
“Yes,” says Sunbelt’s Eric Sites. “AMTSO has almost all independent anti-malware testing companies as members, as well as all major anti-malware vendors. Additionally, there is a big push this year to sign up new members.”
David Harley digs deeper. “There’s an important point to be made here. The relationship between anti-malware companies and anti-malware testers is more complex than you’re suggesting. Testers are not the antivirus industry, though we have a symbiotic and in some senses essential relationship. Getting representatives from both these industries onto the same team may be AMTSO’s biggest achievement to date. (Well, that and getting consensus on several major documents from a large constituency of tough-minded, hard-nosed, rugged individualists who have moulded scepticism into an art form.)
“Don’t get me wrong: co-operation with the vendor community is one of the things that gives the mainstream testing community its edge over the noisy (semi-)amateur testers, but that community jealously guards its independence, as it should. Testers are obligated to be as accurate in their testing as humanly (or digitally) possible, but their responsibility is to their audience, not to the vendors. The stuff you may have heard about AMTSO being the vendors keeping testers on a leash is aggressive marketing and sour grapes.
“That said, the alignment of two complementary skill sets (good testers know a lot about malware/anti-malware, and good vendors do a lot of internal testing) puts AMTSO in a better position than most to inform, educate and generally try to raise standards (rather than introduce standardization, which isn’t in our game plan). And the Advisory Board, which is meant to “keep us honest” includes academics et al who are decidedly not within the AV industry.”
I’m going to class this one as a draw. I find David’s arguments compelling, but (and I know this is cheating because I used the term ‘anti-malware industry’) I would be more convinced if there was some representation of the needs of the PBI (that’s you and me again) in this process. It’s still, ultimately, AMTSO looking after its own members with its own rules. The PBI will simply have to accept what its given without any input to it.
Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)
“I would have to say yes (anti-malware works)”, says Stuart Taylor. “From within SophosLabs I can report that just the other week we were receiving 100,000 brand new files every day. Of those new files, we were detecting around 80% already, despite not having seen these particular files before. If you combine that with the other proactive technologies such as HIPs then you get very good detection and a typical user just cannot do without.”
“Of course it works,” says David Harley, “better for some people than for others. What it doesn’t do is meet the expectations of people who think that it’s valueless if it doesn’t catch 100% of malware (not to mention a lot of other stuff that you may or may not expect it to catch, legitimately or otherwise) with 0.00% false positives, no resource or administration overhead, and at zero cost.
“It’s not the 100% solution. There isn’t one, which is why any honest researcher will tell you that you need multi-layered defences (at home or in the enterprise), not simple AV scanning. Whether you need multiple packages or a suite is not a debate I’m going to get into now.”
But is testing a con?
“Yes and No,” says Eric Sites. “Yes, if the tests are done poorly. For example, just taking a bunch of malware samples from web page and scanning them with AV command line scanners using VirusTotal gives very misguided results, even if all of the samples are detected by every AV engine. Testing like this does not show that these products will stop a single piece of malware if tries to infect end users’ computers.
“No, because there are some tests that are done using best practices that really do show the quality and effectiveness of anti-malware products.”
David Harley is as forthright as he’s been throughout. “It’s not a con. It’s a partially successful attempt to solve the insoluble.
“Anti-malware testing isn’t controlled by the vendor industry. If it were, the world would be a very different place. Actually, I think you have to distinguish between (at least) two distinct types of testing.
“Comparative testing is based on a number of assumptions that don’t always hold up. For one thing, that testers are better at gathering (and, more importantly, validating and classifying) samples than vendors. Or, in terms of other types of testing, that they’re better at assessing ergonomic feasibility, performance, resource usage and so on. (In the second case, at least, that might sometimes be so, but I see no reason to assume that it’s always the case, and every reason not to.)
“For another, that a given detection test uses a sample set that is an accurate reflection of the totality of malicious programs currently in the world. That’s unprovable, so we have to accept best endeavours and reasonable competence instead of quantification: the AMTSO review analysis process could be described as a way of assessing whether a given test or test report meets those standards by comparing it to the AMTSO “Fundamental Principles”. And yes, that’s turning the testing process on its head: I see it as essential that the testing industry sees itself as accountable to its audience. If at the moment the most feasible way for the industry to acknowledge that is to have its temperature taken by an organization with quite a few vendor members, so be it. I’m open to (and have suggested) other approaches, but this is the best we have right now (in my not so humble opinion).
“Certification testing isn’t based on finding the “best” product, but on establishing a baseline value for consistently acceptable performance. Personally, I think that’s a healthier, or at any rate more attainable approach. It’s less about magnifying small variations in performance, or isolating layers of protection artificially, in order to establish clear winners and losers. If there was less margin for error in testing methodology, that would be less of a problem, but in many cases the error margin is quite large enough to invalidate any pseudo-statistical analysis.”
So, for my final question, I think the industry wins again. AV software works most of the time. AMTSO is here, at least in part, to remove the possibility of ‘con’ from the testing.
A final word from David. “It isn’t in AMTSO’s charter, but I think the take-home message I’d want people to carry away from this interview is this:
“Testing (any software testing) is harder than you probably think it is. Testing security software has its own special problems: it’s a highly dynamic technology countering another highly dynamic technology, and the criminal nature of the threat means that the software is even more “black box” than software that’s simply trying to obviate Intellectual Property theft. No product and no tester has a monopoly on truth, and AMTSO doesn’t have all the answers. What it does have is the attention of a group of people who would like to see the right questions asked when a product, or a tester, comes under scrutiny.”
But as the author of this piece, I reserve unto myself the absolutely final word. My opinion has been modified, but not totally converted. I can see that there are some altruistic people within AMTSO, and I can see that the aim is laudable: to develop testing procedures that will allow the buyer to have confidence in the tests and the products he buys. But the fact remains that AMTSO is almost entirely composed of AV software vendors (who need to sell their products), and AV testers (who almost entirely get their funding from the vendors who successfully sell their products (either directly or indirectly in the form of magazine advertising). This is not good for the outsiders’ perception of AMTSO. It may be that this is the next big thing for AMTSO to address: better presentation with the inclusion of a voice from the PBI (that’s you and me, remember).
Pedro Bustamente, Senior Research Advisor at Panda Security, also comments:
Is this the anti-malware industry looking after itself?
Actually it’s quite the contrary. The objective of AMTSO is to advance the current state of test methodologies so that they become a better representation of real-life scenarios. Currently most AV tests either only test a portion of anti-malware products or do so with outdated methodologies. The objective of AMTSO is to develop tools so that independent testers and consumers alike can be informed and distinguish between ìa good test vs. a bad test.
Is it even representative of the anti-malware industry?
There’s quite a lot of non-vendor members in AMTSO, from academia, testing labs, magazine publishers, external board of advisors, etc. I’m not sure what the balance is but there is quite a lot of non-AV-vendor members.
Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)
You’d be surprised how much the AV industry does not like AV testing. But the reason is not because it is a massive con, but rather because most tests are not really representative of real-life situations.
Alice Decker, Senior Threat Researcher at Trend Micro Deutschland, also comments:
Is this the anti-malware industry looking after itself?
This is what people like to believe. But, surely every organization looks after itself – for example, aren’t the IPI (International Press Institute), APWG (Anti-Phishing Working Group), ISO (International Organization for Standardization), WToO (World Tourism Organization) all looking after themselves in a wider sense?
AMTSO is an organization which brings together software vendors and software testers, establishing bridges of communication and knowledge exchange. The inter-vendor competition did not disappear through the establishment of AMTSO, but the industry’s passion for protecting their customer’s provides a strong ethical basis. Participating security vendors truly support independent testing methodologies and AMTSO presents computer security/protection in a fair and understandable way for users. However, to really successfully carry the message to users, one key element is currently missing: the fair involvement of the Magazines.
Is it even representative of the anti-malware industry?
Yes, it is representative, and yes, it is currently a small percentage of the whole industry. However, AMTSO is open to new members from the anti-malware Test, Vendor or Media branches.
Is anti-malware testing a massive con to fool the buyer into buying the software? (That is, does anti-malware software even work in the real world?)
Testing is not a con. In most cases, the testers (specialist individuals or companies) are the results provider for the Magazines. Meanwhile, the vendors compete against each other – so by working with testers and improving their protection, our customers are also better protected.
Also, the testers control the test environment, and work to deliver an impartial test. Trend Micro believes that these tests need to evolve to deliver real-world style testing, to help demonstrate those products that protect against real-world threats.
Does it work? Does Police-concept work in the real world? Sure, it does.