AMTSO responds to negative criticism
I had hoped that I need say no more about AMTSO – at least for a while. But I have to say something about its latest comments signed by several members and posted simultaneously to multiple blogs. First some background. I wrote my first article and allowed AMTSO members to express their views freely. It subsequently seemed, and I was so warned, that some areas of AMTSO were taking this article as my approval of the organization, even though it expressed some of my reservations.
I subsequently, and consequently, wrote a second article to outline my opinions. The second article was more forceful than the first: it is sad but true that when talking to industry, you need to shout to be heard. But I would have been content with this: to let readers see the views of AMTSO in the first article and my own in the second; and then come to their own conclusions. AMTSO clearly has a right to respond, and has done so in comments to the second article and individual blog postings elsewhere by Andrew Lee and David Harley. Kurt Wismer, not a member of AMTSO, has also responded.
Now a group of AMTSO members has published a new coordinated blog across many sites, and I feel that I need to respond to that. This piece has, in its first paragraph:
Given some recent negative publicity aimed at AMTSO (example), we want to collectively clarify the following points on behalf the anti-malware industry, where we come from, and indirectly on behalf of AMTSO.
Testing and Accountability
The ‘example’ link points to my ‘dissenting’ article. It is the only critical article referenced. It is reasonable to assume, therefore, that this posting is meant as a rebuttal to my article – and AMTSO is perfectly entitled to do so. The problem is that AMTSO defends areas that I have not, and would not, criticise. The effect of this is to suggest that I am unreasonable and vindictive. I would therefore ask that readers of this AMTSO post look again at what I actually wrote.
You will see that my only criticism of the anti-malware industry is that it sometimes misleads the market by allowing the suggestion that 100% detection of viruses ‘in the Wild(List)’ is the same as 100% detection of viruses in the wild. In the same article I point out how valuable and necessary the anti-malware industry is. My criticism of AMTSO is that it does not censure this practice.
Apart from this, my comments point to just one criticism with a simple solution: AMTSO lacks credibility because it is the industry laying down rules for itself. (In comments to my article, Mark Kennedy, one of the signatories to this AMTSO piece disagrees. His view is that AMTSO is credible because its work is credible. My view is that its work lacks credibility because AMTSO lacks credibility.) But the solution is very, very simple. AMTSO should include members taken from the customers of the anti-malware industry. This would give AMTSO credibility; and that credibility would allow its work to be credible.
So this is my problem with this AMTSO posting. It states that given some recent negative publicity aimed at AMTSO by me, it wants to collectively put the matter straight. It then goes on to list a series of points that are either irrelevant to me, or to which I am in whole-hearted agreement. The implication, and these people are more than clever enough to know this, is that my criticisms are trivial. It is a clever way of dismissing me and praising themselves as being super-reasonable folks.
I would ask five things of readers:
- read my original articles and see what I actually did say before believing what I am accused of saying
- search this blog for ‘PandaLabs’ (one of the signatories of the AMTSO post) to see how unreasonably anti the AV industry I really am
- search this blog for ‘David Harley’ (one of the signatories of the AMTSO post) to see how prejudiced I am against AMTSO members
- ask yourself why, when they say they are responding to negative criticism from me, do they not even mention the only two criticisms I actually make. Why is the WildList advertising sacrosanct? Why are there no users within AMTSO? Solve these two issues and I, for one, have no other criticism
- and finally, make up your own mind: be manipulated neither by me nor anyone else.
Kevin Townsend 
none whatsoever. that observation was meant to inform, not to persuade.
while you actively deny it, your words in numerous instances betray an obvious bias against AMTSO. david provided an excellent example in pointing out your remarks about AMTSO being a “stain”.
i can define ordinary people as the complement of the set of people with specialized knowledge in the malware field. now, please define “technical people”. i’m interested to see if you can define a set of people with a relevant body of knowledge that aren’t already part of the anti-malware community.
simplification? gross oversimplification you mean – to the point of being flat out wrong. sellers (ab)use testers to attract buyers. buyers most certainly do not use testers in the vast majority of cases.
the argument will always be there regardless of what anyone does. there will always be room for the uninformed to spout nonsense – which is precisely why the uninformed do not belong inside AMTSO.
you quoted him at length in a post you’ve claimed on this very page reflected your own opinions. the obvious implication is that you’ve accepted his story.
don’t be disingenuous. you and i both know there are many levels of deception and not all of them can accurately be described as lying. i’ve already called it what it is – it was spin.
the fact that they fail to name them in that coordinated post points to a desire to repair the relationship. naming and shaming would have been far more acrimonious and would have made such repair much less feasible.
you’ve made it clear that one of those posts represents your personal opinion and one does not – that’s the difference.
nothing will remove that possibility. as stated previously, there will always be room for the uninformed to spout nonsense.
so are computer manufacturers and power generating companies. the independent testinging industry is unconcerned with which product you buy or even if you buy a product at all. their concern is that you base whatever decision you happen to make on meaningful information so that that decision can be ‘informed’.
as a matter of fact you haven’t explained it even once. not concretely, not with any degree of specificity. the above is the closest you’ve come so far and now it sounds like you’re suggesting they be token members (which i believe david has mentioned AMTSO already has).
once again – AMTSO is about testing, not the way tests are used by other parties after the fact. that is outside of AMTSO’s scope. AMTSO is concerned with the quality of the tests themselves, not the infinite ways that they can be misused by marketing departments and others.
dictating how businesses can provide value to their shareholders (no deceptive marketing practices for you!) is a much hairier problem than developing guidelines and best practices to improve the quality of testing. AMTSO wisely chose to limit it’s scope to what it was feasibly capable of doing. they’ve left tilting at windmills to someone else (like you or me, for example) to tackle.
LikeLike
OK, you and I are never going to agree. We could carry on slagging each other off – but it won’t change anything: we each think the other is fundamentally wrong.
LikeLike
I’m sorry if you feel targeted. I don’t think it was anyone’s intention to target or misrepresent you, and certainly not mine. But suggestions that AMTSO is aiming to “gently become the de facto provenance for all things AV” and that “AMTSO should be dissolved” and is “a stain on otherwise excellent industry”, or value-loaded comments about “the good things AMTSO claims to do” are not, in my opinion, positive criticism.
LikeLike
I don’t know how many times I have to say this. I believe that what AMTSO is trying to do is good. I believe that without the involvement of users/buyers you are and will remain open to accusations of bias in favour of the AV sellers. So far only Kurt Wismer has addressed this issue (and he doesn’t belong to AMTSO). He believes that buyers/users cannot bring anything to AMTSO’s table. He believes that AMTSO cannot interfere with how the AV industry uses WildList tests in its advertising. We can at least get to a position where we disagree.
The rest of AMTSO refuses to state its position and tries to turn everything into a personal issue.
LikeLike
with all due respect, you’re starting to sound like a raging egomaniac.
you yourself observed that the post in question addressed several points that had nothing to do with your post, but you still insist that somehow it did have something to do with your post?
they didn’t reference you, they referenced your post. as it happens there was one other blog post referenced – did it occur to you that there might be something that your post and the other referenced post had in common? did it occur to you that that commonality is what the joint post was really all about?
what is clear is that your biases are colouring your judgment. what isn’t clear is whether you’re aware those biases exist.
that almost certainly had to do with your assertion that ordinary people, rather than the ones with technical expertise that AMTSO currently has, should be running the show at AMTSO – that testers should have to prove to ordinary people rather than experts that their tests are methodologically valid (thereby setting a much, much lower bar).
uncritically accepting the words of a tester (one with an ax to grind, no less) is precisely what you did in the post they referenced.
exactly what impression did you think you were giving people when you compared the 2 AMTSO reviews and underscored that the less favourable one pertained to a tester that wasn’t part of AMTSO?
they did better than that – they linked to a post where the culprit was not only named but also quoted more extensively than anywhere else i’ve seen to date.
no it isn’t. go read the membership list again. or are you seriously suggesting that the testing industry and the av industry are one and the same?
Relevance to the argument?
What are these biases?
If I said or implied that I think ordinary people (could you define ‘ordinary people’?) should be running AMTSO I apologise. I believe that the AV market should be included in AMTSO. I most certainly believe that it is absurdly arrogant to suggest that technical people working for other industries are incapable of understanding malware and testing. The AV market comprises sellers, testers and buyers. The buyers use the testers to chose their seller (simplification). To exclude the buyers from the process of ‘controlling’ (simplification) testing leaves the impression that the sellers (and to a lesser extent the testers who are largely dependent upon the goodwill of the sellers) are trying to stitch up testing to their own advantage. I am not arguing that this is the case, only that this argument will always be there until the buyers are included in AMTSO.
You’re being terribly one-sided. One: I quoted Rick Moy. Two: if he is lying, please say so; if you are not willing to say that he is lying, stop hiding behind insinuation. Three: are you saying that the AV industry has no axe to grind with NSS? Four: where is the difference between me quoting Rick Moy in this article, and verbatim quoting Stuart Taylor (AMTSO), Eric Sites (AMTSO), David Harley (AMTSO), Pedro Bustamente (AMTSO), Alice Decker (AMTSO) in the first article?
That the current structure of AMTSO leaves it open to such accusations; and that the inclusion of buyers/users within AMTSO will remove that possibility
Of course they are. They are both part of the process of selling AV product to the rest of the world.
I have explained this many, many times. It is their presence that is required. But unlike you I do not discount the possibility that they could add to the debates.
If AMTSO does not at least censure the use of WildList testing to imply that products detect 100% of viruses in the wild, then it leaves itself open to the view that it is manipulating testing and the use of testing for its own, and only its own, purposes; and devil take the buyer.
LikeLike
I’m afraid this isn’t about you. While your blog was cited as an extreme example of negative commentary – I don’t think anyone else has suggested that AMTSO has achieved nothing whatsoever and should let someone else do the work that wasn’t being done until AMTSO was formed – it was never intended as a point by point rebuttal of any of your articles. And while none of the signatories have been secretive about their association with AMTSO, I’m not aware that AMTSO has ever commented officially on any of your articles.
Nor am I aware that AMTSO ever mistook your first article for approval of AMTSO. Some people felt that you’d raised some fair points in that article, notably about user engagement, on which I subsequently commented in a Security Week article and elsewhere (e.g. http://amtso.wordpress.com/2010/07/06/amtso-not-iso-standards-and-accountability/), and that further input from you might benefit the organization. If you’ve heard anything more sinister than that, I can’t shed any further light on it.
I don’t believe that you’re dishonest or vindictive, but I don’t think all your comments have been reasonable or balanced. While I appreciate your high opinion of me, that doesn’t mean I have to accept your dismissal of all the work that I and many other volunteers have put into raising testing standards through AMTSO.
I seem to recall you making many more than two points of criticism, but I’ll certainly address the two you mention above.
1) There is nothing sacrosanct about WildList testing. It’s become much less useful in recent years, and that’s why mainstream testing organizations have either stopped doing it or supplemented it with other technologies. The frequent suggestion that it’s what all labs but one do is marketing, not fact. I quite agree that 100% in a WildCore-based test doesn’t imply 100% detection of all realworld threats or anything near it. It has some residual value in certification testing, less in comparative testing. So why does it still have some value? Because it’s verifiable. The kind of dynamic (in a broad sense) testing that AMTSO members have been advocating is much more difficult to execute and to verify, and that’s the sort of problem that AMTSO members are trying to solve. In the meantime, WL testing still tells you something useful. It doesn’t tell you which is the “best” product at overall detection, but nor does a comparative test that offers no way to validate its methodology or test set.
2) Why aren’t there users in AMTSO? I seem to be answering that question time and time again, notably in my response to your initial questions, and in a blog at http://amtso.wordpress.com/2010/07/06/amtso-not-iso-standards-and-accountability/. But since you didn’t use those sections of my response (or comment on that blog), I’ll dig them out and post them somewhere. As I’ve also pointed out elsewhere, on-demand responses to criticism of AMTSO is not in my job description, so it may be few days before I do.
LikeLike
This is very much about me. And I have not the slightest doubt that, if not initially intended, the authors are very aware of that result. You cannot say that you are writing in response to “some recent negative publicity aimed at AMTSO” referencing only me and immediately in the same sentence continue “we want to collectively clarify the following points on behalf of the anti-malware industry, in which we work, and indirectly on behalf of AMTSO.” That very clearly associates me with the points you wish to clarify.
Then, using the smoke and mirrors arguments that I am beginning to expect, you clarify points that no-one to my knowledge has actually queried. These include:
Please show me where anybody, never-mind me, has suggested that.
Evidence, please.
Evidence, please.
I absolutely agree with you. But unless you name names and call the ‘culprit’ out, this is meaningless. Unless you are willing to prove openly what you claim, then you are merely smearing anyone and everyone who disagrees with you. More smoke and mirrors.
This is exactly what I have been calling for. But AMTSO is not transparent and cannot be transparent until it includes representation from the market the AV industry sells to.
So, while you point to me as providing negative publicity, you then ‘clarify’ points in which we have total agreement. The clear implication is that I do not agree with these admirable statements; when it is perfectly clear that only a bounder would disagree. This is typical smoke and mirrors argument. It is about me. And that’s my response to the first sentence in your current comment.
Your next comment is clearly directed at me:
This is miles away from what I am suggesting. I have criticised nothing that you have achieved. I have criticised not one of your publications. My criticism is of the structure of AMTSO, which is wholly composed of those within the AV industry. I am not suggesting that someone else should do the work; I am suggesting that you include within AMTSO the independent voice of your users. Without that voice, AMTSO in its current form lacks credibility. Mark Kennedy disagrees with me, but my view is that your work cannot have credibility when the organization lacks it. There is a structural fault in the foundations; and because of that you cannot trust the structure built on top. But I say very very clearly that all you need do is recruit the users. I have said elsewhere and I will repeat here that you probably won’t have to change anything in the documents and procedures you have already developed – but without the inclusion of independent users those procedures lack credibility. Your argument is, once again, smoke and mirrors claiming that I have said something I haven’t. Do you want me to say it more clearly? You are claiming I have said things I haven’t said in order to discredit what I have said.
There you go again. I am not dismissing your work, I am merely saying that it needs the involvement of people outside of the AV industry in order to be credible; and I am suggesting that this independent involvement could/should come from your customers.
Yes, I made more than two criticisms; but they are all built around these two critical points.
But my question was “Why is the WildList advertising sacrosanct?” You do nothing to answer this question. AMTSO claims to be working towards the elimination of misleading testing, yet it allows (or at least does not censure) misleading advertising that can lead users to believe that particular AV products detect 100% of viruses in the wild.
I assume you are referring to this:
I’m sorry, this is not an answer. AMTSO needs user involvement in order to gain credibility. You should, therefore, be actively approaching the big user organizations and asking them to become involved. And I repeat, I can think of no criticism I have that will not go away once you do this. Rather than negative criticism, my views are positive criticism: I am explaining how you can improve, not dismissing you or your work. And I simply do not understand why you waltz around trying to suggest I’m saying things other than I am and completely avoiding the two issues I actually raise.
LikeLike