Anti Malware Testing Standards Organization: a dissenting view
On June 15 I posted the article AMTSO: a serious attempt to clean up anti-malware testing; or just a great big con? The purpose of the article was to look at AMTSO, the Anti Malware Testing Standards Organisation; and I invited AMTSO members to justify themselves. Now I want to give a dissenting view, largely my own, and to look at AMTSO from outside of the tent. I shall be asking two principal questions:
- is AMTSO serious about improving the value of anti-malware testing?
- who does AMTSO serve?
Is AMTSO serious about improving the value of anti-malware testing?
I recently blogged about two new threats discovered in the wild by M86 Security: Asprox returns: fast-flux SQL injection attack; and Skype: old vulnerability, new exploit – in the wild. In both cases, M86 ran the malware they had discovered against VirusTotal (a respected site you can use to see what anti-malware products make of any submitted file). For the former, VirusTotal showed that only 7 out 42 anti-malware products detected the Asprox malware; while for the latter, only one AV product out of the 42 detected the Skype malware.
Caution: one of David Harley’s ‘common mistakes’ in How to Screw Up Testing is “Using VirusTotal or a similar service to check the samples and assume that any product that doesn’t report them as malicious can’t detect them. This will once again give the advantage to scanners that flag everything as “suspicious”, and will also disadvantage scanners that use some form of dynamic or behavioural analysis. It’s certainly not a real test, and it’s a form of pseudo-testing that VirusTotal itself discourages.”
M86 Security agrees with David Harley in a test environment, but comments: “The value in using VirusTotal is that it reflects what a lot of organizations will be using in live environments rather than a test lab.”
VirusTotal adds: “Currently, there is not any solution that offers a 100% effectiveness rate for detecting viruses and malware. You may become a victim of misleading advertising, if you buy such a product under those premises.”
This would seem at odds with all of those marketing claims we see from the anti-malware industry, which state that their particular product detects between 97 and 100 per cent of all malware in the wild. An example is the VB100 award issued by VirusBulletin, one of the leading anti-malware test organisations. In VB’s own words:
The VB100 award was first introduced in 1998. In order to display the VB100 logo, an anti-virus product must have demonstrated in our tests that:
- It detects all In the Wild viruses during both on-demand and on-access scanning.
- It generates no false positives when scanning a set of clean files.
The product must fulfil these criteria in its default state.
http://www.virusbtn.com/vb100/index
I cannot think of a single anti-malware product that doesn’t boast similarly high scores, if not from VB, then from ICSA or West Coast Labs. But Virus Bulletin and VirusTotal cannot both be right. Well, the explanation is in the Virus Bulletin statement ‘in the wild’. It contains a link to this:
The WildList Organization collects monthly virus reports from anti-virus experts around the world. The data from the reports are compiled to produce The WildList – a list of those viruses currently spreading throughout a diverse user population. A virus that is reported by two or more of the WildList reporters will appear in the top-half of the list and is deemed to be ‘In the Wild’.
In recent times, the list has been used by Virus Bulletin and other anti-virus product testers [such as ICSA and West Coast Labs] as the definitive guide to the viruses found in the real world.
http://www.virusbtn.com/resources/wildlists/index.xml
So, ‘in the wild’ is actually a sub-set of the viruses that are actually ‘in the wild’: it means only those viruses that are included in the WildList’s list of those viruses it has found in the wild. It gets worse.
- the WildList requires submission of a virus sample from at least two separate researchers
- many of the researchers are the anti-virus companies themselves
- in-built latency within the process can mean that it can take 3 months from the detection of a new virus to its inclusion within the WildList being used in a test
- this latency means that, almost by definition, the Wild List includes little, if any, of the biggest threat to end-users: zero-day malware
- members of the WildList Organization get to see the WildList when it is published; and yes, that includes the majority of AV companies
So what does this all mean? It means that the WildList is not a list of viruses in the wild, but a list of the majority of viruses that were in the wild several months ago. It means that the anti-virus test is against a set of viruses that the anti-virus companies already know about. It means that anything less than 100% success against the WildList is probably down to incompetence in the anti-virus company. It means that the average anti-virus buyer is being conned about the true situation.
So the answer to my first question, is AMTSO serious about improving the quality of anti-malware testing, is ‘no’. It would not allow the use of a test process, by its own members, that so clearly misleads the public if it were.
Who does AMTSO serve?
Let’s not prevaricate: the question is ‘does AMTSO serve the anti-malware user, or itself, the anti-malware industry?’ To answer this question I’m going to look at two things: the AMTSO Fundamental Principles of Testing, and the application of those principles by its Review Board.
The very first principle, headlined Testing must not endanger the public, includes the categoric statement: “In addition, new malware must not be created for testing purposes.” Why not? How can you test the true heuristic behavioral capabilities of an AV product without testing it against a brand new sample that you absolutely know it has never experienced before? To include this restriction under the banner of not endangering the public is also misleading: there is nothing essentially incompatible between developing a new virus and keeping the public safe.
I am not alone in being puzzled by this. Ed Moyles from SecurityCurve is similarly surprised:
Yes, yes… it’s terrible to create new malware – completely unethical. Yup, under any circumstances. Even if it doesn’t leave the lab, even if it doesn’t replicate, and even if it doesn’t have a hostile payload. Yep – still terrible. We know this because shady, fly-by-night organizations like Consumer Reports, University of Calgary, or Sanoma State are always springing up like mushrooms. Their clear intent is to bring down the Internet, wreak havoc, and otherwise mock everything that is just and holy… Sigh. I just can’t get my head around the argument.
SecurityCurve, June 16th, 2010
The problem for AMTSO is that there is one very obvious reason that comes to mind. Could it be that inclusion of new samples would increase the number of ‘fails’ in the test, and thereby lower the success rate so beloved by the industry for marketing purposes? AMTSO could respond that it isn’t a real ‘fail’ since the malware doesn’t actually exist; but as a user I would reply that it is more important to get an idea on how the product might respond to zero-day threats. So is this an example of AMTSO looking after itself?
Let’s move on to the Review Board. There are at the time of writing just two reviews: one on a Dennis Technology Labs test report, and one on an NSS Labs report. One AMTSO review is favourable and the other is not. I do not know enough about either of the testing companies or their test methodologies to comment on the reports themselves, but I think it is illuminating to compare the framework of the AMTSO reviews.
Dennis Technology Labs is a member of AMTSO. The testing was paid for by Symantec, a member of AMTSO. Symantec performed very well in these tests. The review of the report by AMTSO was requested by Dennis Technology Labs. The review was favourable. The test report is effectively endorsed.
Summary Table from the DTL Report
NSS Labs is not a member of AMTSO (although it used to be). The testing was paid for by NSS independent of any anti-malware vendor (in the hope of recouping the cost via sales of the report). Sophos performed very badly in the report. The review of the report by AMTSO was requested by Sophos. The review was not favourable. The test report is effectively dismissed.
Summary Table from the NSS Report
What does this look like? To me it looks like a duck; and If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck. AMTSO has its say about the NSS test report in its published review. I asked Rick Moy, President of NSS Labs, for his view of the AMTSO review. On AMTSO itself, he commented, “I have had drinks and long discussions with 90% of the folks in AMTSO. There is some very old-school thinking afoot, and a fair amount of protectionism. While they have good intentions, there is probably just too much business interest being represented.”
But what about their review of his test report?
Every vendor reviewed the methodology before. In fact I had sent it to them in 2008 and solicited comments before running the test. Every vendor but Sophos cooperated and gave us software and reviewed settings of the products. None complained about the methodology… But when the results came out, folks from AVG, ESET, Symantec and especially Sophos went crazy.
I cooperated for months of craziness. They all essentially demanded we give them free consulting and tear through samples to find what was wrong with our test. Well, it was a real-world test of fresh malware that had not been shared around amongst the vendors, that simple. Sophos even made brazen false claims that we had not contacted them. After much harangue, we produced email correspondence with the chairman of AMTSO and Lab Director at Sophos showing that we had, multiple times, and even reversed samples with them to help them troubleshoot. No sanctions or reprimand was made. Instead they redoubled their efforts to discredit the test.
Rick Moy, President, NSS Labs
So the answer to my second question, who does AMTSO serve, is that it serves the anti-malware industry: it is self-serving. In fairness, it rarely claims to be in the best interests of the user (except when it is trying to justify its guidelines). There are no user members, and it is not open to users: “AMTSO membership is open to academics, reviewers, publications, testers and vendors, subject to guidelines determined by AMTSO.” But in that case, it should keep itself to itself, and not send out press releases nor make its website nor its judgments available to users.
Conclusions
There are three main conclusions I draw from this look at AMTSO.
Firstly, the biggest problem I have with AMTSO is that it declares itself to be the sole arbiter of what is good in anti-malware testing: it is the prosecutor, judge and jury. I find this intensely arrogant. The sole judge of a test should be the user. The tester has to prove to the user that the tests are valid. If the vendor objects, he has to prove to the user that the tests are invalid. The idea that the vendor has only to prove his case to other vendors with identical vested interests is patently absurd and would be dismissed in any other industry.
Secondly, if AMTSO was serious about setting and maintaining testing standards for anti-malware products in accordance with its own charter, it would ban the WildList in its current form. WildList testing is dangerous. Users who buy security on the basis of ‘detects ALL viruses in the wild’ are likely to believe that they are completely safe from viruses when they most certainly are not, and might consequently behave less carefully on the internet.
And thirdly, AMTSO should immediately recuse itself from the purpose of setting anti-malware testing standards until, and unless, an open, independent, user-centric body can be established. To this body, the vendors should have every right to make representation; and to this body, the testing industry (separately) should have every right to make representation. Only then are we likely to have anti malware testing standards that are independent, valid and trustworthy.
RIDER
I have no beef with any of the anti-malware companies. They are essential to our security; and we all, every one of us, must have at least one of their anti-malware products installed on our computers for our security. I have no beef with any of the individuals within AMTSO. They all have far greater knowledge of threats and solutions on the internet than do I. My beef is with AMTSO itself. It is, in its present form, a stain on an otherwise excellent industry.
Kevin Townsend
Kevin Townsend 
Question… What is AMTSO? Did they replace the Anti-Spyware Coalition? And here it why it is important… BECAUSE THEY GOT IT WRONG!
Correct me if necessary, BUT wasn’t there an organization of AVAS companies which went by the name Anti-Spyware Coalition?
And didn’t this organization include ALL of the major AVAS companies?
And didn’t they decide which programs would be permitted under their radar when they chose to not detect and prevent programs like WebWatcher by Awareness Technologies (ATI) and other programs which are supposedly being used by parent to monitor children. I assure you no parent needs the obfuscation, redirection and keyword notifications provided by WebWatcher.
It goes even further, when programs like WebWatcher get a pass, because the first thing that it does is disables the AVAS software from functioning. THE AVAS companies are ingoring their program, BUT THEIR PROGRAM IS DISABLING THE AVAS COMPLETELY IN RETURN. What better way to further conceal their program than on a machine which now can have MULTIPLE ‘infections’.
An extra slash in the registry disables all the virus definitions (McAfee). A junction point can assure that any scan of the system never completes to the point where a user can act on the intrusion (Kaspersky). And if the ATI advertising is correct NORTON SYMANTEC is/was securing the data being hijacked from your computer EVEN IF YOU ARE RUNNING THEIR PROGRAM YOURSELF. THAT”S RIGHT SYMANTEC WAS NOT PROTECTING YOU, AND WAS PROTECTING THE PERSON WHO IS STEALING YOUR DATA, and corrupting your life.
There is no exit from a targeted intrusion until there is detection, prevention and prosecution for the misuse and illegal use of these programs.
I have watched WebWatcher go from a detected piece of spyware in 2006, to a branded package which avoided detection/prevention SIMPLY BECAUSE IT WAS PACKAGED AND SOLD, to a bootkit and then to a rootkit which installs itself on a virtual drive and runs from virtual memory with minimal traces on the hard disc or BIOS records.
The damage this program can do it documented here… http://www.work2bdone.com/live or GOOGLE: TERRORISTIC DIVORCE.
LikeLike
OK. We seem pretty close to agreeing to disagree. No point scoring – just a comment. I don’t think being ‘open’ to user representation is enough. You need to have them within AMTSO. To this end, I think you have to actively go out and recruit them.
LikeLike
Agreed. And we are. Of course, active recruitment can bring on its own charge of “self selecting”, but we shall try. And of course, any user or (better still) user group is free to contact us.
Our first thoughts are IT departments of large companies, and User Groups. Some (most) of what we do is fairly well down in the weeds, so it helps to have people who have spent some time thinking about testing and testing issues.
Thanx, this has been fun. We really would welcome your comments on our documents. Another document of particular note based on your original post would be “The Issues Involved in the Creation of Samples for Testing” (http://www.amtso.org/amtso—download—issues-involved-in-the-creation-of-samples-for-testing.html) You will see in that that there is no explicit ban on the creation of samples. We point out the issues that are created when artificial samples are constructed. It was clearly the most controversial document AMTSO has produced, as there are some within AMTSO who feel the outright ban was correct (I am not of that opinion). One of the things I am most proud of with AMTSO is the willingness to tackle the tough subjects (a particularly nasty one we are now working on involves evaluating the impact of False Positives).
LikeLike
Please pay attention. I have not dismissed the work; I have said it lacks credibility. It lacks credibility not because of its content, but because of the structure of AMTSO. Can you not see the difference?
Now, if I have a computer that is undefended, it will be compromised. Since this is inevitable it is reasonable to say that the undefended computer lacks credibility from the moment it is connected to the internet. It is also reasonable to assume that the computer will be rooted; perhaps more accurately, it is unreasonable to assume that it will not be rooted.
What happens next simply shows that we live in different worlds. In my world, that computer can never be trusted until the disk is reformatted and the software re-installed. In your world, a security expert can come along, look at it, and declare it to be safe; and on his say-so it is immediately credible. Remind me to not let you near my computer.
So, taking this analogy back to AMTSO, restructuring it to include independent users is akin to re-installing the software. The software itself probably won’t change; just as AMTSO’s existing opus probably won’t change. The only difference is that it will have gained credibility.
LikeLike
You switched the rules in you computer example. My point is if this image is examined and found to be clean that “at that time” the machine was credible. I made no claims to how long it would remain so.
Your statement “It lacks credibility not because of its content, but because of the structure of AMTSO” underscores that your definition of credibility a distinction which lacks meaning. Content is all that matters. Source is irrelevant. You seem to feel that the users you want us to include are incapable of reading our work and finding it to have value. I have more faith in them than that.
To prove it I offer you this challenge: Look at our paper on Whole Product testing (http://www.amtso.org/amtso—download—issues-involved-in-the-creation-of-samples-for-testing.html). Then give specifics where a test following this best practices would be biased towards the AV companies. Then show how *any* test being conducted today is superior to our Whole Product test.
If you cannot then your claims against AMTSO are simply not credible. Not because of who made them, but from the fact that they cannot be backed up.
You raised the charge. I ask you now to prove it.
LikeLike
No. Read what I said.
I simply do not accept that. I don’t. You do. Get over it.
Now you’re just being silly. I have never said that AMTSO’s rules are biased. The whole point is that without the independent presence of users, they could be biased. It is that possibility that removes credibility from AMTSO.
Back to being silly. You’re trying to play semantics; but trying to build cleverness based on a false premise. It doesn’t stand up.
But you’re trying to make me prove a charge I never made. I repeat, without user participation on AMTSO, AMTSO lacks credibility. That is my opinion. I stand by it. Your opinion is different. I don’t care what you do with your opinion; that’s your prerogative.
But please tell me this: why is AMTSO so reluctant to admit user representation? This is the whole point of my concerns. Bring in user representation and all of my criticisms (apart from allowing misleading claims based on WildList detection rates) simply go away.
LikeLike
Thanx. Your clarifications will make it very clear to anyone reading this thread your meaning of credible and mine. I am very comfortable with the distinction.
Where do you get that AMTSO is reluctant to allow “user representation” (another of your opinions unsupported by fact)? You seem to be operating under the misconception that AMTSO is some sort of closed club. We would be more than happy to have such representation. There is no secret to join AMTSO. One need simply send a request to membership.queries@amtso.org (we cleverly hid that on the membership page of our website).
LikeLike
It shouldn’t surprise you, why would someone want to join an organization that has no credibility? I still believe our credibility is defined by the quality and openness of our work. It should not matter who has an idea, it is the quality of the idea that matters. To deem an idea “not credible” simply because of who came up with it is prejudice (i.e. to pre-judge). If you are going to define our group as not credible (as opposed to incredible), you should identify the faults in our work that demonstrate it. While not claiming that our work is perfect, I do believe it represents the best collection of information on the issues involved in testing ever presented. I think your criticism would be better served by pointing out where we are wrong rather than who we are. If you believe that tests conducted using the best practices we describe would be biased, then please identify the bias. You will find we have never advocated the types of tests you have decried. We tend to focus on the “whole product” tests.
It is also our contention that an ideal test would not have to be modified to test a completely new technological approach to AV. The test should concentrate on the real world infection, and not the technologies used to block it. This is one of the concerns we had regarding NSS. The URL’s that NSS used were the end result of an infection scenario. No user ever opened up their browser and went to those URL’s. A SPAM mail might have tricked them to browsing to it (which is why we told them SPAM should be been included in their test), or some other redirection might have gotten them there. There test bypassed all that, and thus created an artificial scenario. We did not say their test was wrong, it was limited. Had they adjusted their claims there would have been no problem.
There were other serious flaws in the NSS test, most egregious being not re-imaging the machines after infection (really testing 101). However, at this time AMTSO only evaluates regarding our 9 principles, and does not evaluate the methodology (only requiring that the methodology be disclosed, and the claims be limited to what the methodology covers).
LikeLike
OK, I’ll try one more time. Imagine a computer with no security defences. It may be fine. It may be a good, clean computer. It may, in fact, be the best computer in the world. But you cannot trust it because it lacks the thing that gives it credibility. Therefore it lacks credibility.
AMTSO is like that. It may be the best thing since sliced bread. But without the involvement of the people it is selling to, it lacks the thing that would give it credibility. Therefore it lacks credibility.
This has nothing to do with the quality of AMTSO’s work. It has nothing to do with NSS. It is an industry writing its own rules, and that is not acceptable.
The anti-malware industry does not exist because of malware, it exists because there is a user market willing to pay you for your products. And yet you have excluded those people from the process of defining excellence within your industry; and you have given yourself the unfettered ability to censure anything with which you disagree. It is the exclusion of the independent user voice that takes away your credibility.
The sad thing is that it would be so easy to remedy.
LikeLike
Kevin,
Nonsense and more nonsense. Our work is available for anyone to judge. You have dismissed it (or said it should be dismissed) simply because of where it came from.
We’ll take your analogy. You have this computer system with no security software. You dismiss it because of that. But if the contents of that computer are available for independent analysis and is then deemed clean it has credibility. You are making my point for me.
Thanx!
LikeLike
I think perhaps my meaning of ‘credibility’ is not clear. The work of AMTSO may be brilliant; but because you are the industry (not just a part of it), there is no basis for trust in that work. It is the fact that your work is open to abuse, not that it has been abused or that there is anything wrong with it, that makes it lack credibility.
I would recommend two things to give yourself credibility. The first is to disallow the use of advertising that could in any way lead users to believe that 100% detection of viruses in the WildList is the same as 100% detection of viruses in the wild.
The second would be to recruit a statistically valid number of users (I don’t know how many that would be). I would suggest that they come from the IT Departments of the larger corporations. I would suggest that at least one of these users should sit on any future Review Board; and I would suggest that future reviews allow for dissenting views from individual members.
LikeLike
I don’t buy your credibility argument. Using that same argument I would say that any tester’s results should be discounted because they themselves came up with the methodology. Credibility comes from the openness and transparency of the work, which any external source can view and draw conclusions from. By this same argument, a tester who does not reveal their methodology sufficiently should be distrusted.
To your second point, we are looking for just such organizations to become members and to assist. Our search is not helped by those who claim the organization has no credibility.
Lastly, we did allow for dissenting views. It was just the dissenters chose a different method to express themselves. I leave it to you to decide which would have been the more professional way to proceed.
LikeLike
>By this same argument, a tester who does not reveal their methodology sufficiently should be distrusted.
Yes
>Our search is not helped by those who claim the organization has no credibility.
That surprises me
>It was just the dissenters chose a different method to express themselves.
That is too subtle for me
LikeLike
Kevin,
This may sound strange to you, but Symantec’s reasoning for helping to establish AMTSO was *precisely* to help users. From Symantec’s perspective, we have limited development resources. Engineers prefer to utilize those to develop real technologies to solve real problems. To that end we had exploit detection for many drive-by downloads and, more recently, reputation based system. However, there is another portion of the business that wants to make money (that’s a revelation, isn’t it?), and for them it is important to win reviews. If the reviews in question are not representative of what is happening in the real world (and we both agree they were not), then optimizing for them would divert resources from *real protection* to *test detection*, and that does not serve the end user. If we were to ignore the tests and just focus on the technology, then flawed tests (and I do lump NSS into this category) would continue to put out a false representation as to the protection capabilities of the products.
However, if testing were more like the real user experience, then optimizing for the user would also optimize for the reviews, and we would have a win/win situation. Then we would be free to compete on the value of our actual protection, and not some thin slice that tester chooses to explore. You can see this all the way back in 2007 when I presented this at the first CARO workshop on testing.
The real challenge we have found within AMTSO is not convincing the testers to provide better tests (though better testing is much more expensive than what was done in the past), but rather to convince those who pay for and publish the tests that they need to promote this better testing (i.e. magazines). Ironically, these are the very same people who are supposed to be the voices of the users (interesting circle, no?).
One last point I will make about artificial testing scenarios (i.e. creating samples): we devoted an entire paper to the issues involved with this. The pros, the cons, the concerns are all presented there (or here: http://www.amtso.org/amtso—download—issues-involved-in-the-creation-of-samples-for-testing.html).
Mark Kennedy, Distinguished Engineer, Symantec, Secretary and Member of BoD, AMTSO
LikeLike
I am happy to concede that every single person within AMTSO is there for altruistic reasons. I am happy to concede that it is possible for AMTSO to benefit the user.
The problem is that in its current form, AMTSO ultimately has no, and can have no, credibility. It requires an idependent viewpoint at its heart; and that should come from the user. Without it, AMTSO will always be subject to suggestions that it is governance of the industry, by the industry, for the industry.
Kurt is, as far as I can see, the only person to address this issue. I may have got it wrong, but he seems to suggest that user participation is not necessary because the user does not understand the problems involved; and therefor should leave it to the experts (apologies to Kurt if this is too simplistic a description of his view). In this we are diametrically opposed.
LikeLike
We do have an Advisory Board consisting (primarily) of people unrelated to the AntiMalware industry. Who, precisely, would you suggest to represent the user? We would likely be more than happy to have them on board.
For the record, I do reject the notion that simply because we are part of the industry that we have no credibility. I would welcome any external criticism of the body of work so far produced where it can show that we are acting in anyone’s interest other than the users. Likewise, any comments you would like to make on our collection of documents would be more than welcome.
LikeLike
As quoted in the article, VirusBulletin states that in order to display the VB100 logo, a product must detect “all In the Wild viruses during both on-demand and on-access scanning.” In recent months I have received at least 3 press releases boasting about VB100 certification: Agnitum, Vexira, and CA (I’m sure I will find more if I look hard). The clear inference that users are meant to draw is that these products stop all viruses. This is misleading.
It doesn’t matter if Panda doesn’t do this. It doesn’t matter if Agnitum, Central Command and CA are not members of AMTSO. The point is that you and VirusBulletin are members of AMTSO. By not censuring the use of something like the VB100 award, you are implying that it is acceptable. AMTSO is therefore being misleading.
(I haven’t checked the detailed wording for the West Coast Labs and ICSA awards; so VirusBulletin may or may not be unique. My point is that if it happens at all, AMTSO loses credibility.)
LikeLike
Hi Kevin,
Maybe it is because English is not my mother language, the thing is that I don’t understand how you obtain the conclusion “AMTSO is not serious about improving the value of anti-malware testing”. I’ve read your text several times, sorry if I don’t get it. I do agree that the WildList is not reflecting the whole reality, but I’d say that AMTSO thinks exactly the same.
It seems that you think that AMTSO is encouraging the use of the WildList, which is not the case. Could you please help me to understand your point?
Thanks in advance,
Luis Corrons
LikeLike
It’s a personal viewpoint. My view is that the WildList in its current format allows the AV industry to put forward misleading claims. Many users will assume that ‘detects all viruses in the Wild’ actually means ‘detects all viruses in the wild on the internet’. This is not true; therefore it is misleading. If AMTSO is trying to prevent misleading testing it would start here and prohibit the use of WildList testing in this form. That it doesn’t leads me to believe that it cannot genuinely claim to be seeking to improve testing standards. Not encouraging it is not enough. It must specifically prevent the use of the WildList as an acceptable form of product testing.
LikeLike
Thank you very much, that was really fast! 🙂
This is a document Pedro Bustamante wrote some years ago about the Wildlist:
Click to access TheDisconnectBetweentheWildListandReality.pdf
Many things have happened since them, and in fact some improvements have been made within the Wildlist and many more will have to come, anyway your point is still valid, any test based only in the Wildlist is not acceptable. The weak point I see here is that I am not aware of any test that is based just in the WildList.
I really don’t think that AMTSO has to react to something that is not happening in the real world, I think it’s better to focus on real issues. A good demonstration is the document published by AMTSO called “Best Practices for Validation of Samples”, and some documents AMTSO is still worlking on (sample selection, etc.) All these documents make no sense if AMTSO would want, as you say, to have tests made just with the Wildlist (something that as far as I know doesn’t exist in the real world.)
LikeLike
AMTSO is not censoring press releases from companies, nor suggesting marketing strategies, nor saying how logos have to be used.
As I’m sure you have read the Virus Bulletin comparative review, you have seen that in the test they use tens of thousands of recent malware samples, and you will have seen how each product has scored.
And I agree that you or anyone else don’t like this test, or any other test. One thing that anyone can do is asking to AMTSO for an “Analysis Request”, as you know, which is much more constructive that saying “let’s close AMTSO”.
LikeLike
My preference would be to see AMTSO continue but under the control of users rather than the anti-malware industry.
LikeLike
Good article. The question is what is “better” testing according to the vendors? As you rightly pointed out, they still push misleading tests on the the public. If vendors thought “better” testing = “more accurate & realistic” testing they would not need AMTSO. They could commission those tests separately. But to try and silence critics… that takes joint effort and the cloak of legitimacy afforded by an “independent” organization. Enter AMTSO. In the United States we call that racketeering and collusion.
The situation is reminiscent of big tobacco saying their internal tests showed no harmful side effects. Or big banks saying their industry was too complicated to be subjected to outside regulation and that only they had the expertise required to determine what was too risky. Or big oil…
—
As you see from Mark’s comment, there is no denial that the Sophos claim against NSS Labs (re:contact) was untrue. The fact that AMTSO failed to sanction Sophos for bringing forth false claims speaks volumes of what their agenda is. The demands that NSS Labs do work for free were an attempt to make us chose between a (false) claim of being uncooperative vs. wasting our time and money refuting bogus claims. Also, Mark should check his records before throwing out untrue statements under the guise of “if I recall correctly”. The fact is that Sophos miscategorized a sample as “corrupt” and NSS Labs proved otherwise.
Further, AMTSO “findings” were far from balanced. They claimed our test was flawed because NSS Labs had not tested anti-spam – yes, spam – during our web-based socially engineered malware test. The argument was that we had not exhausted all of the possible protection methods. Keep in mind that this type of malware is not delivered by email; It is posted on websites such as Facebook, MySpace, etc. And also keep in mind that according to multiple AV vendors, web-based socially engineered malware accounts for over 50% of malware. Email is only 10%-14% nowadays.
AMTSO’s second complaint about our report was that we cautioned users about some products, but hadn’t tested the entire product. — Their argument is silly. Here is why: Would you not fail a car whose brakes didn’t work 30%-40% of the time? If the brakes fail, do you care how fast it can go from 0-100? The same logic applies to our test. If an AV vendor is unable to reliably perform a primary function (like block web-based malware), we are perfectly comfortable with cautioning people accordingly. Especially since web-based socially engineered malware represents over 50% of the malware threat.
The bottom line is that a vendor cartel can hardly be trusted to police itself. And AMTSO has shown its true colors by not applying its own standards evenly; criticizing NSS Labs while giving a free pass to various “100%” tests.
I am glad to see their ruse isn’t fooling you.
LikeLike
Sorry Vikram, I have to disagree, if you review the submission to AMTSO, there was no claim of lack of contact. It simply wasn’t in there so I don’t understand why NSS keeps saying it was.
SophosLabs researchers are some of the most experienced and highly quailified analysts in the industry and we have over 20 years experience in the industry. With all due respect, to suggest that we “needed help” to reverse engineer a single sample is laughable.
On the last point, I would simply say there are other tests where similar threat vectors have been tested, with different results (including your latest test as I understand it).
So my car based analogy would if I was looking for an insurance quote, I would look at a number of different quotes and compare cover, cost etc. rather than buy the first one I saw.
LikeLike
Good point, although I’m not sure anything said in your previous article was contentious. I actually thought the review of the NSS test was reasonably balanced. It found in favour of NSS in all but two of it’s principles. The only real criticism was in the conclusions drawn were not based on the test.
LikeLike
Did you ask anyone from Sophos or AMTSO to comment on the points made by NSS?
I don’t want to rake up past discussions, but at no point did we ask NSS for “Free consulting” or “help reversing” we simply asked for a sample from the test so that we could verify the results which seemed out of step with what our customers are experiencing. If I recall correctly the sample they sent was corrupt and did not execute.
One final point, I applaud the attempts by NSS to focus on the full capabilities of security products rather than simply submitting samples to Virustotal or scanning large numbers of files. It is a better way to evaluate effectiveness and is one of the things AMTSO has been focussing on. However, recommending that customers don’t purchase a product, and even recommending they change product on the basis of one test, for one particular type of malware seems to be over stating the claims of that test.
For example, another test using a very similar methodology, run by a well respected organisation AV-Test.org and sponsored by Trend Micro (who performed well in the NSS test) showed Sophos coming a close second which at the very least shows inconsistency.
Mark Harris – VP of SophosLabs, Sophos
LikeLike
>Did you ask anyone from Sophos or AMTSO to comment on the points made by NSS?
No. Nor did I ask anyone to comment on AMTSO’s points in the previous article: http://bit.ly/dsvYAm
You do, of course, have full right of reply here.
LikeLike