Home > All, Security Issues > Anti-virus is essential – it’s just not as good as they tell us

Anti-virus is essential – it’s just not as good as they tell us

Cyveillance is a company that knows what’s happening on the internet. That’s what it does; intelligence is its business. “Cyveillance,” says Eric Olson, Vice President, Solutions Assurance, “is built on a foundation of home-grown technology for monitoring and sifting through the open source or publicly available internet – in short, we hoover up anything we can from the public internet; and that produces a very large, noisy, unstructured and mostly pornographic haystack out of which we have built a system that not only collects the haystack, but also finds the needle. That needle can be just about whatever we or our clients care to define as a relevant needle. So our capability, if you like. is to do what you could do sitting in any cybercafé just poking around on the internet – it’s just we do it on a grander scale.”

Eric Olson, Vice President, Solutions Assurance, Cyveillance, Inc

And how grand is quite mind-blowing: information is collected from more than 200 million unique domain names, 190 million unique websites, 80 million blogs, 90,000 message boards, thousands of IRC/chat channels, billions of spam emails, tiny URLs and more. The intent is to provide the intelligence that allows proactive security – the ability to recognize a threat before it becomes a problem, and to solve the problem before it becomes a disaster.

Eric gives two examples. A bank might hear about credit card numbers being found or traded on the internet. It might go to Cyveillance who would then look for credit cards in its haystack, and would be able to tell the bank, ‘hey, these cards of yours have been compromised – you should cancel them.’ That, says Olson, “is a data-level example of what we do. An intelligence-level example might be the tweet or blog posting that says ‘I just lost my job, I told my wife and she left me, my kids hate me, I have nothing left to live for, I’m going back there with my Kalashnikov to kill everyone I used to work with’ – you would be surprised how many of those we find in a day. So if that company or executive is a protected customer of ours we will call their Head of Security and say ‘hey, somebody is threatening to come back to the office with a machine gun, and you might want to be on the lookout for that’.”

In short, Cyveillance is all about proactive security. “We provide realtime intelligence,” says Olson, “about cyber threats, malware, exploits, viruses, social engineering, scams, spear phishing etcetera ad nauseam early enough for a response to be proactive rather than reactive. We move the defensive perimeter out into the wild and allow an intelligence-led approach to security — more predictive and proactive than reactive.

But there is a problem. The whole security industry has become mired in the concept of reactive security: it’s what most of us believe security is all about. But reactive is never good enough: closing the stable doors after the horse has bolted has never yet stopped the horse from bolting.

So periodically Cyveillance does something to demonstrate the weakness of the reactive approach: it tests the leading anti-virus products against a current selection of malware samples it has found in its haystack. And the AV products are usually found wanting. The methodology is very simple: the test solely uses malware samples discovered by Cyveillance ‘in the wild’, and only those samples that are confirmed as malware by at least three of the tested AV products. For the latest test, this process provided a data set of approximately 1700 malware samples collected over a 3 day period in April.

Cyveillance wanted to make things reflect real life as far as possible; so it installed the AV products on PCs and then tested them against the viruses. Nothing special – just how things happen in real life. But those of us accustomed to seeing AV test results in the high 90% success bracket – if not VirusBulletin’s VB100 certification – should prepare for a shock. ESET’s Nod32 product was by far the most successful – but still detected just 38% of the viruses. The average detection rate for the 15 products tested was less than 19%.

AV product detection rates on day one of the test

I asked Olson why the anti-virus products were so poor at detecting viruses. “It’s really quite simple,” he replied. “If I’m a criminal and I make my money out of writing malicious code, I’m not going to waste my time releasing my malware into the wild if I’m not going to get a return on my effort – so I spend the $30 dollars or whatever to buy a copy of each these AV products and I test my code against the AV products. If they stop my malware when I’m getting ready to release it, then I rewrite the code until they don’t. So the answer to why these detection rates are so bad is not because the people at ESET or Sophos or Symantec or McAfee are stupid – they’re not – they’re clever, hard-working dedicated people – the problem is that all the protection they offer in desktop or network AV programs is also available to the criminal. That criminal will buy a copy of Norton, run it against his stuff, and if Norton fields it, he fiddles with his code until it doesn’t.” In reality, the criminal doesn’t even need to pay out for the anti-virus programs. “There are also various foreign equivalents to VirusTotal,” continued Olson. “If you upload a binary to VirusTotal that isn’t detected by the AV products, VirusTotal shares the sample with the industry – but there are dozens of equivalent foreign sites that don’t do this.”

This ability to write new malware, or adapt existing malware, to beat the AV products is well-known. So Cyveillance felt that the real test for the anti-virus products was to see how quickly they could recognise and catch the failures; so it retested all the AV products regularly over a 30 day period to see how quickly they caught up with the malware. The results can be seen in the following graphic (click for a full-size version): but the bottom line is that the AV products are still not that good.

click for full size

Malware Detection Rates over 30 Days by AV Vendor

Cyveillance tested thirteen popular AV solutions to determine their detection rate over a 30 day period and found that popular solutions only detect an average of 18.9% of new malware attacks. By day eight, AV solutions average a 45.7% detection rate. This rises to 56.6% on day 15, 60.3% by day 22, and 61.7% after 30 days. Top AV solutions take an average of 11.6 days to catch up to new malware. Since this does not include malware signatures undetected even after 30 days, users should not rely on the AV industry as their only line of defense.

I asked Olson if he got much negative feedback from the AV vendors. He said they had some interest when they first started doing the tests, but that they simply explained what they were doing, and how they were doing it, and nobody could complain. “As a company, our principle interest is not viruses – we’re not binary crackers – our principle interest is in studying what is happening out on the internet right now; and one of the things that happens is that we download all of this malware. The key point about our methodology and studies about malware being distributed in the wild is that we are doing what users do, not what virus crackers in control of their own standards and environments do – we’re clicking on links and visiting pages and getting infected like normal users – the only difference between what I’m doing and what my mom does is that I’m doing it on purpose; I’m trying to make it happen and she’s trying to avoid it; and my mom relies on one or more of those products we’ve just pilloried to keep her safe from all of those things, and they’re doing a not very good job of it.”

So, I asked Cyveillance, Why did you do this thing? “We’re not suggesting that you don’t use the tools that exist out there,” said Brian Hedquist, Cyveillance Director, Marketing and Communications. “But from an enterprise point of view we’re suggesting that you have to take other methods of protecting yourself – the AV companies have bandied their solutions around for so long that enterprises have got complacent with their firewalls and anti-virus – but the bad guys are getting better and better all the time and companies can’t just sit around thinking they’re protected when they’re not.”

“Anti-virus is absolutely necessary,” agrees Olson. “It is necessary but not sufficient. Companies have to get away from the reactive security mindset. Proactive security is possible when you have the intelligence, and Cyveillance has that intelligence. The technology arms race always favours the bad guy; so you have to train your employees; and train your executives who may be targeted on a personal basis one on one; and thoroughly research social engineering attacks. So, why did we do this? Partly because we believe that AV is necessary but insufficient; and partly because some of those other things that can help with proactive security, we can offer.”

Cyveillance’s research findings

Categories: All, Security Issues
  1. Daniel Schrader
    September 29, 2010 at 10:36 pm

    You quote the Cyveillance study as though it was gospel – when it looks like it is a poor attempt to do detection testing.

    I am with Symantec, so my bias is clear, but that study:
    1. Doesn’t even list what versions of what products were tested.
    2. Says nothing about the environment
    3. Appears to have simply tested signature scanners – no attempt was made to see if reputation scanners such as our own Norton Insight, NIPS, HIPS, Safe Search or other security features would block the threat. Quoting the study, “The 1,708 confirmed malware files were run through the latest release of the top desktop antivirus solutions” – so this was just testing scanning of static files, no real-time behavioral detection would have been engaged, no download scanning, no source reputation, no IPS – in other words, 75% of the securty features were bypassed by testing files on the disk.

    4. Says that they tested, “three-day period of April 20, 2010 through April 22, 2010, resulting in an overall total data set of approximately 1,708 confirmed malware files” – confirmed by who? We encountered 240 million unique malware files last year (mostly changed encryptions of a much smaller number of threats) – so were these 1,700 unique viruses, trojans or worms or 5 viruses mutated a few hundred times? Who is to say they were even real threats?

    5. You misrepresent the testing when you state the testing was real life like, “Nothing special – just how things happen in real life”. That is completely wrong, the testing was of static files on a hard disk – this was not a test of the whole security product or of the user experience.

    I suggest you stick to tests from groups that know what they are doing – and who don’t have their own ax to grind such as av-test.org or av-comparatives.org.


  2. Randy Abrams
    August 10, 2010 at 6:18 am

    Wow, you start of by saying that “Cyveillance is a company that knows what’s happening on the internet. That’s what it does; intelligence is its business.” and then back it up by quoting it’s marketing agenda. True to what their marketing wants, you then infer that they know about all aspects of cybersecurity despite overwhelming evidence that they know little or nothing about testing antivirus software or being able to identify malicious software with in house expertise.

    I your rants against AMTSO you say there need to be users in AMTSO. What users do you have in mind? People like you? How are such users to contribute? Look at what you missed. According to Cyberveillance’s methodology, they eliminate ALL zero days from their testing. If 3 AV products don’t detect a sample then it is discarded. By definition a zero day isn’t detected by any. And you missed that? Why do you think they discarded these samples? It is because they know what stolen credit cards look like, but not what malware looks like. For malware they rely upon 3 antivirus products to tell them… it isn’t their area of expertise or knowledge, so they have to eliminate all zero days. You really think that is good methodology and reliable testing? Then they eliminate untold numbers of “day 1 malware because only one or two products detected it. They eliminate this malware because they obviously don’t know how to identify malware.

    Next, if 3 AV products false positive on the same file they claim it is malware, at least that is what their methodology explains.

    How good are they at collecting samples? They claim they collected samples for 3 days (April 20, 21, and 22) and ended up with 1708 samples. A ESET alone, on average we find 200,000 samples of unique new threats each day. 1708 divided by 3 is about 570 samples a day and we find about 200,000 each day. What do you think that does to the percentages?

    You go on to make the absurd claim that “Cyveillance wanted to make things reflect real life as far as possible; so it installed the AV products on PCs and then tested them against the viruses. Nothing special – just how things happen in real life.” Whose real life are you talking about? Are you actually unaware that browsers, such as IE, Firefox, Chrome, and Safari block access to many infected sites, so users would never see the malware? Are you actually unaware that some AV products, including ESET block access to malicious sties as well? Are you actually unaware that some AV products include anti-spam which often filters out the emails that contain the URLS where the malicious files are stored? Are you actually unaware that products, such as ESET’s also use more aggressive setting in realtime than on demand, which means that PROACTIVE approaches are used to block threats that may not be detected in the unreal scenario’s that exemplify the shoddy testing that Cyveillance performed?

    You missed the easy stuff… the really easy stuff. I am not talking about the browser’s blocking threats. I am not talking about the differences in realtime vs on demand settings, I am talking about the methodology eliminating ALL zero days and any “Day 1” threats that are not detected by at least 3 products. If you can’t see the amateur and easy stuff, what are you going to add to scientific approaches to testing methodology?

    I do realize that because I work for an antivirus company (which for most of my career I did not) you will say it is marketing hype, but consider for a moment… long before you explain about “ITW testing” and the VB 100 award, as well as ICSA Labs and Checkmark certification use of “the wild list”, as an employee of an antivirus company I posted the following blogs about the Wildlist. http://blog.eset.com/2008/06/07/50-vb100-awards, http://blog.eset.com/2008/05/30/the-av-industry-from-the-outside-in-and-the-inside-out.

    Also you might watch the following and get a better idea of the motivation of people – http://www.youtube.com/watch?v=u6XAPnuFjJc

    Most, if not all of the researchers who are members of AMTSO are putting in a lot of their own time… any idea why?

    So, you took the marketing bait of Cyveillance and were set up like a bowling pin. You couldn’t seem to even detect the blatant flaws in their methodology.

    Would you go to a Gynecologist for advice about facial reconstructive surgery? Would you go to an oncologist for advice about carpal tunnel syndrome?

    Just because someone has expertise in one area of security doesn’t make them an expert in all areas, but, as long as it isn’t the marketing department of an antivirus company, you appear to be willing to fall for marketing hook, line, and sinker.

    Randy Abrams
    Director of Technical Education


  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s