“Our little gnomes in the backroom,” says the excellent Shadowserver in an announcement headed ‘New AV Test Suite’, “have been working feverishly for the last several months to put the finishing touches on our new Anti-Virus backend test systems.”
Malware testing, as we know, is a tricky business. AMTSO, the Anti-malware Testing Standards Organization, has expended much energy and expertise in developing detailed methodologies designed to ensure fair, unbiased and accurate anti-virus tests. But do we get this from Shadowserver? Do we get a new AV comparison source that we can realistically access for accurate unbiased information on the different AV products available to us? Let’s see.
Shadowserver starts off with a fair comment.
No single vendor detects 100%, nor can they ever. To expect complete protection will always be science-fiction.
That being said, it goes on…
…you can see the different statistics of the different vendors in our charts.
Here’s a couple of examples.
The one thing that really leaps out here is that Panda apparently misses (shown in green) far more of the test samples than Avira. This is counterintuitive. Panda is a commercial product backed by one of the world’s leading security companies. Avira, which I personally trust sufficiently to use on my XP netbook, is a free product. Shadowserver provides a partial answer:
The longest running issue has been our inability to use Windows based AV applications. We can now handle that, however it is still not what you might buy for home or commercial use. We are utilizing a special command-line-interface version from each of the vendors that we are using. This is not something you can purchase or utilize normally. These are all special version but most of them do use the same engines and signatures that the commercial products use.
This is important. Luis Corrons, technical director at PandaLabs elaborated:
What ShadowServer does is not an antivirus test. As they say, they do not even use commercial products, but special versions. Furthermore, it is static analysis of files they capture. It is a statistic. But the data cannot be used to say “product x detects more than product y” or “product x detects this percentage” as they are not using any of the other security layers used in real products (behavioural analysis/blocking, firewall, URL filtering, etc). The most you can say with this system is product x was able to detect y percent of files using their signatures and heuristics (the oldest antivirus technologies).
This is important. The AV companies have long recognised that the original signature database solution to malware cannot match the speed with which new signatures are required for polymorphic virus families. So they have supplemented their signature detection with more advanced and sophisticated methodologies.
In our case (Panda) ShadowServer is using an engine which is a few years old (at least 5) and of course is not using the cloud, so I can guarantee that our results are going to be awful. We have been asking SS for years to use a new version, but they were not supporting Windows. Now that they are supporting it, they forgot to mention it, but it’s not a problem as we’ll be sending them a new version with cloud connection. Anyway, even though in that way the results will be way better, or even if we are the number 1 vendor, that doesn’t mean anything, as it is only a static analysis of some files.
One solution would be for Shadowserver to work more closely with AMTSO. Shadowserver is not currently a member of AMTSO. I urge it to join. And I urge AMTSO to waive all membership fees so that this non-profit free service organization can do so. Both parties would benefit enormously. In the meantime, I asked David Harley, a director of AMTSO and research fellow at ESET, for his personal thoughts.
Shadowserver has never been discussed within AMTSO, that I remember… In the past they’ve shied away from suggesting that their statistics are suitable for direct comparison of vendor performance. One of the reasons they cited for that is that their testing has been focused on Linux/gateway versions, and you can’t assume that desktop versions will perform in the same way across a range of products. Including some Windows products will make a difference in that respect, but I can’t say how much, because I don’t know which versions they’re using. Where gateway products are used, it’s unlikely that the whole range of detection techniques are used that an end-point product uses. Detection is often dependent on execution context, certainly where detection depends on some form of dynamic analysis. A gateway product on an OS where the binary can’t execute may not detect what its desktop equivalent does, because the context is inappropriate. On the other hand, the gateway product’s heuristics may be more paranoid. Either way, there’s a possibility for statistical bias…
This isn’t a criticism of Shadowserver, which does some really useful work. I just don’t think I could recommend this as a realistic guide to comparative performance assessment…
Neither Luis nor David are known to shy away from the truth, whether of themselves or their products. But both seem fairly clear: Shadowserver is good; but this service is not yet ready. Shadowserver’s AV test suite will not give a realistic view of different AV products’ actual capabilities. Not yet. It needs more work. I’m certain that will happen. But for the time being at least, don’t use Shadowserver’s statistics to form an opinion on the relative merits of different AV products.
UPDATE from Shadowserver
It is difficult to not compare one vendor to the next due to how we have the data
structured on the pages. It would be impossible not to try and derive conclusions
from those results. While that is the case, our goal is not to create a real
comparison site for everyone to try and compete to see which AV vendor is better
than the next…
That is not our purpose…
That being said, our purposes in doing AV testing is simple. We wanted to know what
each malware was supposed to be for categorization purposes, and of course just to
see what happened. We collect a lot of malware daily and trying to find ways of
tying our data together is important.
Because we are volunteers and a non-profit we really enjoy sharing what we find
no matter how odd. We even enjoy talking about when we screw something up or
when we encounter something exciting. Everything here is for you our public to
enjoy, discuss, and even criticize…
Shadowserver, 8 September.
Last year I voiced two main concerns about AMTSO, the anti-malware testing standards organisation. One was collusion in the false marketing impression given by claims of 100% test success against malware in the Wild(list). I won’t repeat my concerns here (see instead the original articles AMTSO: a serious attempt to clean up anti-malware testing; or just a great big con? and Anti Malware Testing Standards Organization: a dissenting view). Sadly, there has never really been any acknowledgement that this is a valid concern; nevermind any action on it.
The second concern is that AMTSO is effectively a closed shop: it is largely by the industry for the industry; and for that reason alone it cannot be trusted. This caused no inconsiderable heat, with some members of AMTSO feeling that I was saying that they personally could not be trusted. Others, however, accepted that it was a valid issue.
Well, I am now delighted that AMTSO has made serious attempts to address the problem. Last October it announced a new low-cost subscription fee in an attempt to get more people involved:
While AMTSO recognizes that strict requirements for full membership are necessary to ensure it achieves its objectives, it also understands that the fees put it out of reach for many interested individuals that may have a valuable contribution to improving the objectivity, quality and relevance of testing methodologies. Hopefully, the new low cost subscription model will widen the reach of the organisation and enable more people to have a say in the future of anti-malware testing.
Philipp Wolf, of AMTSO member Avira
This new subscription currently stands at €25 per annum. I don’t know how many subscribers it has attracted – but I doubt that it is many. “They will also have the right to attend meetings, though not as voting members.” Why should I pay money to have no ultimate say in things?
Today, however, AMTSO has launched an open (and free!) “forum where anyone may post and join in testing-related discussions.” Users are still unable to vote on AMTSO issues, but that’s fair enough. Discussions, like justice, should be seen to be done. Provided that AMTSO moderators do not censor this discussion forum (other than the usual legal requirements), it will “provide a discussion point where anyone with a question or an opinion on the testing of anti-malware software can make their voice heard.”
For that, AMTSO deserves to be commended.
I must say that I have known and respected David Harley for many years. I still respect him enormously. Indeed, I respect the anti-virus industry in general. So, having said that, a quick response to David’s article. In it, he says:
That brings us to Kevin Townsend, who could never be described as AMTSO’s best friend…
…But then I realized that he might have been misled by this statement on the AMTSO home page…
I would just like to say that I don’t believe that I have been misled. I know that AMTSO is open to the people I would like to see within it. But the fact is they are not in it. So my point is simply that AMTSO needs to go and get them. Just saying that “AMTSO membership is open to any corporation, institution or unaffiliated individual interested in participating in this organization” is not in itself sufficient if they don’t join. However, as soon as AMTSO has sufficient representation from within the AV user community, it will gain the credibility it deserves.
The first I heard of it was a Tweet from Luis Corrons towards the end of last week:
A bit cryptic, but the reference to me is almost certainly in relation to (one of) my two previous criticisms of AMTSO: firstly that its membership is almost entirely incestuous and that without involvement from outside of the industry its recommendations cannot be trusted; and secondly that use of any testing that is allowed to suggest 100% efficacy against viruses in the wild is disingenuous (see AMTSO: a serious attempt to clean up anti-malware testing; or just a great big con? and Anti Malware Testing Standards Organization: a dissenting view).
But now AMTSO itself has released details (this is on the former criticism, not the latter):
The Anti-Malware Testing Standards Organization (AMTSO, www.amtso.org), an international organization that encourages improved methodologies for testing security programs, announced today the imminent availability of a new subscription model that will open up membership to a wider audience.
Neil Rubenking, Lead Analyst at PC Magazine, commented, “As a member of AMTSO’s Advisory Board I’ve been privileged to interact and work with the group’s members and committees. AMTSO membership is open to individuals, but the 2,000 €/year price puts full membership out of reach for all but the most dedicated. The new subscription model will now allow all interested parties to make a marked contribution to the development of better testing methodologies.”
The new membership model will apparently cost just €20 (presumably per year), and is clearly a move in the right direction. But from the released information you don’t seem to get much for this. You get access to the
…educational resources that are already freely available on the AMTSO website [and] the development of documentation and participation on AMTSO’s email discussion boards, where some of the world’s foremost experts in the anti-malware industry and the testing industry leave vendor bias aside, in order to pursue lively conversations on the intricacies of malware testing, its fallacies and real-world ways in which to improve it.
In short, you get access to a specialist mailing list, and “the right to attend meetings, though not as voting members.”
I don’t want to sound churlish, because this is a major movement for AMTSO, but you get to speak your mind with no guarantee that anyone will listen, and certainly no say in what AMTSO actually does. It is nowhere near what I personally would like to see: the recruitment of senior technicians from some of the major corporate AV users; with full voting rights. If this simply isn’t possible, perhaps AMTSO could tell us why?
So, all in all, a tiny step in the right direction.
Well, I see Dan Raywood of Secure Computing magazine has entered the discussions on AMTSO (see here); and has included a link to the article where I allow AMTSO members to speak freely, but not one to my critical article. I would have been happier if he had acknowledged the connection (past or present) between SC and West Coast Labs (a member of AMTSO). Not doing so does nothing to minimize concerns.
Anyway, the article gives David Harley (of ESET and AMTSO) a pretty free hand in describing and answering some of the recent criticisms of AMTSO.
He claimed that he would not have made an investment of time and energy if he did not believe that there is a need for major improvements in testing and the public understanding of testing.
And of course he’s right. My concern is not with the intention of AMTSO but with the structure of AMTSO. Where is the voice of the user?
He highlighted three problems – firstly while AMTSO is not a profit-making organisation, the subscription fee is fairly hefty.
Then reduce it. ‘Costs’ is not an adequate reason. If you look at the membership list of AMTSO, it is the companies not the individuals that are listed. And the response to negative criticism put out in a co-ordinated reply was done on company not private blogs. So the AV companies are involved. Remember the ash cloud flying problems? Sophos had about 60 staff caught up in Eastern Europe at the time. Their response? To hire a rather large private jet to get people home. The AV companies have the money to solve this – but they’re trying to appear at arm’s length. Frankly, it doesn’t wash. So where is the voice of the user?
That gives rise to another issue. Since we all have full-time jobs, we can’t give AMTSO the time and attention some of us would like to…
Another non-wash. If AMTSO members haven’t got the time to do it right, don’t do it at all. And like I said, it’s the companies that are listed as members, not the individuals.
The second problem is that the group includes security vendors, as well as testers and product certification agencies. Harley admitted that while mainstream vendors and testers do not necessarily see that as a problem, most people do not see it that way, rather they see it as the foxes guarding the hen house.
Precisely so. But the answer is simple. Get some users into AMTSO.
So how could standards be raised in a more general sense? Harley said that this would be by improving the quality and availability of information about tests and testing, and by making testers more accountable for the accuracy and quality of their testing.
I have no problem with this. In fact I have no problem with AMTSO, and have great respect for David Harley personally. What AMTSO says it is doing is a very good thing, and I think it is making a fair job of it. But the fact remains that without input from the users of AV products it cannot be taken seriously. If this means that the AV companies have to come off the fence, admit their involvement and put some serious money into the organisation, then so be it. But it must recruit from the users of AV products.
I had hoped that I need say no more about AMTSO – at least for a while. But I have to say something about its latest comments signed by several members and posted simultaneously to multiple blogs. First some background. I wrote my first article and allowed AMTSO members to express their views freely. It subsequently seemed, and I was so warned, that some areas of AMTSO were taking this article as my approval of the organization, even though it expressed some of my reservations.
I subsequently, and consequently, wrote a second article to outline my opinions. The second article was more forceful than the first: it is sad but true that when talking to industry, you need to shout to be heard. But I would have been content with this: to let readers see the views of AMTSO in the first article and my own in the second; and then come to their own conclusions. AMTSO clearly has a right to respond, and has done so in comments to the second article and individual blog postings elsewhere by Andrew Lee and David Harley. Kurt Wismer, not a member of AMTSO, has also responded.
Now a group of AMTSO members has published a new coordinated blog across many sites, and I feel that I need to respond to that. This piece has, in its first paragraph:
Given some recent negative publicity aimed at AMTSO (example), we want to collectively clarify the following points on behalf the anti-malware industry, where we come from, and indirectly on behalf of AMTSO.
Testing and Accountability
The ‘example’ link points to my ‘dissenting’ article. It is the only critical article referenced. It is reasonable to assume, therefore, that this posting is meant as a rebuttal to my article – and AMTSO is perfectly entitled to do so. The problem is that AMTSO defends areas that I have not, and would not, criticise. The effect of this is to suggest that I am unreasonable and vindictive. I would therefore ask that readers of this AMTSO post look again at what I actually wrote.
You will see that my only criticism of the anti-malware industry is that it sometimes misleads the market by allowing the suggestion that 100% detection of viruses ‘in the Wild(List)’ is the same as 100% detection of viruses in the wild. In the same article I point out how valuable and necessary the anti-malware industry is. My criticism of AMTSO is that it does not censure this practice.
Apart from this, my comments point to just one criticism with a simple solution: AMTSO lacks credibility because it is the industry laying down rules for itself. (In comments to my article, Mark Kennedy, one of the signatories to this AMTSO piece disagrees. His view is that AMTSO is credible because its work is credible. My view is that its work lacks credibility because AMTSO lacks credibility.) But the solution is very, very simple. AMTSO should include members taken from the customers of the anti-malware industry. This would give AMTSO credibility; and that credibility would allow its work to be credible.
So this is my problem with this AMTSO posting. It states that given some recent negative publicity aimed at AMTSO by me, it wants to collectively put the matter straight. It then goes on to list a series of points that are either irrelevant to me, or to which I am in whole-hearted agreement. The implication, and these people are more than clever enough to know this, is that my criticisms are trivial. It is a clever way of dismissing me and praising themselves as being super-reasonable folks.
I would ask five things of readers:
- read my original articles and see what I actually did say before believing what I am accused of saying
- search this blog for ‘PandaLabs’ (one of the signatories of the AMTSO post) to see how unreasonably anti the AV industry I really am
- search this blog for ‘David Harley’ (one of the signatories of the AMTSO post) to see how prejudiced I am against AMTSO members
- ask yourself why, when they say they are responding to negative criticism from me, do they not even mention the only two criticisms I actually make. Why is the WildList advertising sacrosanct? Why are there no users within AMTSO? Solve these two issues and I, for one, have no other criticism
- and finally, make up your own mind: be manipulated neither by me nor anyone else.
There have been a few responses to my latest article on AMTSO; for example
- AMTSO revisited, Kurt Wismer
- I AMTSO confused…, David Harley
It’s worth reading them for an alternative view to mine; and if anyone comes across others, please add them as a comment below.
I would like to add only one point. Would you be happy with a government that said to you, you don’t have the intelligence or knowledge to understand things; so we, the government, the police and the judiciary, are going to tell you how things are and how things are to be done and you don’t have any say in it?
The principle, I repeat, the principle, is exactly the same. For AMTSO to be trusted in the good things it claims to do, it must be ultimately subject to the voice of the user.