People in the IT security community soon become familiar with the names of various types of malware. Even the general public may recognise some of the more notorious forms: Stuxnet, Flame and Zeus, for example.

But it’s easy to forget that all malware starts out with no name, which is the way perpetrators of attacks would like it to stay. Names are given by the defenders in the fight against computer crime.

New malware is perhaps the biggest challenge facing the IT security industry – unknown-unknowns as a former US Defense Secretary Donald Rumsfeld put it. The term used in the industry is zero-day attack.

Security firms have become pretty good at spotting the known stuff over the years. At its most basic, that function is a big part of what most desktop antivirus and content-filtering products do. They say, “We’ve seen this before and it’s bad news.” This type of signature-based defence still works and is necessary to prevent many mass-market random attacks. However, on its own it is no longer enough.

To stop zero-day attacks takes something more. The issue is pressing because the most dangerous attacks are targeted rather than random. Malware is often crafted for a specific attack – they are after you or your organisation.

So, for example, the Flame malware may have been seen before, but a specific version of it may be unfamiliar. Traditional signature recognition does not work. It is this targeting that is used in many attacks that have come to be termed advanced persistent threats, or APTs. The attack keeps going until the target is penetrated, whatever it takes.

There are a number of approaches for spotting and mitigating zero-day attacks:

1. File-reputation services

The providers of such services, which include Symantec, McAfee, Trend Micro and Blue Coat Systems, know what is bad or black, as well as good or white. They assess new stuff that lies in between based on various factors and create greylists.

Their customers can decide on the acceptable level of risk by selecting at what point they start blocking content on the grey-scale. Bit9 is another security vendor providing such a service. When Flame was named, it checked its records and found it had already been blocking a single instance of it for one customer eight months before it was named. Being unnamed does not mean unseen.

2. Check everything

Security vendor FireEye claims to offer 100 per cent protection. It has been growing fast and hired former McAfee boss Dave DeWalt to chair its board. FireEye treats everything as suspicious rather than white-, grey- or blacklisting.

To do this all executable files are detonated in a safe environment on one of its network-based appliances before being passed through to their destination. FireEye also checks picture, PDF and other file types to ensure they are not being used to disguise malware.

3. Better privilege management

The instance of Flame that Bit9 detected needed Windows admin rights to run. The user of the PC where the firm had detected the attack did not have such rights. Access was probably gained via a Windows vulnerability that allowed it to run at the admin and not the user level.

The granting, management and ongoing use of admin rights on Windows devices are often poorly managed. They need not be. With the right tools in place, admin activity can be limited and audited. If these measures had been taken, Flame might not have been able to run at all or would have been soon spotted. Such tools are provided by vendors such as BeyondTrust, Avecto and Viewfinity.

4. Advanced security intelligence

All the mechanisms listed above are point-security products. They work by looking at network traffic at a single point on the network or by better securing of a particular end-user device. Advanced security intelligence, or ASI, is a different approach and supplements point-security products by taking a more holistic view of IT systems.

They are souped-up versions of existing security information and event management (SIEM) tools, which look at wide range of information in real time to detect threats and some are terming next-generation SIEM (NG-SIEM).

Even if the malware had not been blocked based on reputation and admin rights were not controlled, the communication with a suspicious IP address, and regular running of an unusual file at a strange time of day would soon raise a red flag. Vendors include LogRhythm, IBM via its Q1 Labs acquisition, and McAfee via its NitroSecurity acquisition.

LogRhythm recently reported a case where one of its customers observed, simultaneously, multiple machines attempting connections to unauthorised IP Addresses outside its own network. LogRhythm’s NG-SIEM product had been collecting log data from the customer’s firewalls. An out-of-the-box rule detected the suspicious activity. The destination ports of the connection attempts were identified as associated with Trojan traffic and the servers were cleaned up.

In this case, the actual Trojan malware had been detected some time before and cleaned, but its job had been to deliver and install its payload – a rootkit – which had already done its work before being detected.

5. Advanced security practices

File reputation, file detonation, Windows admin rights, NG-SIEM – these are all advanced security practices that businesses should be considering as they heed reports such as that issued by the UK’s MI5 recently, which records an “astonishing level of cyber-attacks”.

They are not alternative measures to existing security products but form part of a multi-layered approach, which is the only way to stand a chance in an increasingly threatening security climate.

Quocirca’s report Advanced Cyber Security Intelligence is available free to TechRepublic readers.