Log Management: a necessary part of information security
Log Management (the use of software to automatically manage computer system logs) started as a good idea, became an important part of computer security, and now stands on the verge of being mandatory. This parallels the history of computer security itself: from ‘good idea’ through ‘essential’ to ‘legal requirement’. The reason for the similarity is simple: log management provides one of the fundamental pillars of information security itself: accountability (the principle that the cause of any event should be both identified and responsible)1. So, as the concept of security has become more important, so has log management; for it is primarily through the analysis of a computer system’s logs (the electronic audit trail) that the perpetrator of a security event can be identified, and that associated vulnerabilities can be found and fixed.
It sounds very simple, the management of logs, but it is actually very hard. So we are going to look at why it is essential and what we need to look for before we invest in Log Management (LM).
The problems in LM
The primary task of log management is managing the collection, protection, organisation and storage of logs in a manner that will create a reliable forensic audit trail. That’s not just some of the logs, but all of the logs, all of the time, and under all conditions. So a log management solution must automatically manage the whole process of monitoring the status of disparate log files; rotating them at predetermined times or thresholds; collecting essential meta data to make them useful; securing them and forwarding them to central collection points. It is not easy, and could not be achieved manually.
This whole process is then complicated firstly by the sheer size of the logs involved, and secondly by the complexity of their content. Every computer in the enterprise will generate its own logs. Within each server there will be operating system logs. Major applications, such as databases, will produce their own logs. Each different security device or system, including firewalls, anti-malware software, intrusion detection/prevention systems, the corporate VPN and so on will also produce their own logs. This very soon amounts to a huge amount of data, and when multiplied by every computer in the network, and indexed in order to be managed, it is clear that storage rapidly becomes a major and ever increasing issue.
The second difficulty is the complexity of content. We don’t mean that the log content is difficult to understand, but simply that each different system log is likely to be in a different format. Some are in text, some in binary; some in comma separated databases, some in tab delimited databases; some are SNMP and some are XML. The problem here is that the LM will need to recognise and relate events across multiple system formats from multiple system logs over multiple computers in order to create a single audit trail for any particular security incident — and that means understanding and correlating all of the different formats.
These two difficulties inevitably generate a third: the security of the logs themselves. This applies both to the data in transit from the source device to the amalgamated database, and the data when at rest in the database. This is particularly important given that rootkits, a major and growing security problem, specifically seek to alter log data in order to hide their presence.
If LM is so difficult, why bother? Why not simply gather the logs into a central source and examine them manually when necessary? Well, there are three main reasons. The first is that it is simply impractical (it would take too long, cost too much and be too subject to inevitable human error that would invalidate any forensic reliability) to attempt to monitor logs without LM. The second reason is that LM provides a much improved corporate security stance. And the third is that we don’t really have a choice any more: compliance with legal, commercial or bureaucratic regulations means that we can no longer realistically conduct business without LM.
Improved security stance.
Automated analysis of aggregated logs provides a number of features, including:
- the identification of security incidents as they happen. This makes it possible for a breach or vulnerability to be closed before any serious damage occurs; fraud to be stopped before huge losses transpire; and new policies to be developed to minimise future risk.
- the monitoring of high risk activity in near-realtime. This allows us to take action to prevent the activity developing into a fully fledged security incident. Examples of such activity could include access to specific critical files and folders; escalation of enhanced privileges to normal users; suspicious configuration changes such as switching off auditing; alteration of network configuration files and many more.
- the identification of policy violations. It’s no good having the perfect security policy if we don’t enforce it; and we can’t enforce it if we don’t know when it is violated and by whom. LM provides the perfect way to maintain, enforce and continually improve our security policy; and thereby improve our actual security.
- the identification of operational problems. This, surprisingly, is still mainstream security, since another of its fundamental pillars is ‘availability’; and availability can be damaged as much be internal network overloads and staff malfeasance as it can by external denial of service attacks.
Compliance is adherence to the standards specified in legal, commercial and sometimes simply bureaucratic regulations. The one thing they have in common is that we cannot ignore them if we wish to continue in business.
- legal requirements. Governments are obsessive in passing legislation that has a direct effect on information security. But it is rarely consistent. As usual, there is a fundamental difference between the United States and Europe. The US tends to produce specific laws for a specific purpose with a specific penalty. Europe tries to produce generalised laws that will catch everything, and tend to be nonspecific. Thus we have the US Sarbanes-Oxley Act (SOX) that is quite precise in its requirements (including the requirement for an audit trail to be kept for seven years) and has significant sanctions against individual directors. (The UK’s equivalent, the Code of Conduct, merely says “comply or explain why you don’t”.)
- In Europe, however, we have national implementations of the very wide-ranging Data Protection Directive with its insistence that personal data be kept secure – without ever explaining what that means or how it should be achieved. (This is relevant to LM since logs could contain personal data which must therefore be held securely; that is, the confidentiality of the audit trail must be ensured.) The whole thing is then further complicated by the international nature of modern business. Any non-European company that wishes to do business with Europe needs to be acceptable to Europe’s data protection requirements. Similarly, any company anywhere in the world that is a subsidiary of a US company or is itself listed in the US must comply with SOX. And we need to add to this many different laws with different requirements over different lengths of time; and the ones in the pipeline; and the ones that we yet know nothing about.
The seventh data protection principle of the UK’s Data Protection Act
Appropriate technical and organisational measures shall be taken against unauthorised or unlawful processing of personal data and against accidental loss or destruction of, or damage to, personal data.
- commercial requirements. The most pressing commercial requirement is to comply with the Payment Card Industry’s Data Security Standard (PCI-DSS). It’s not required by law; but if we don’t comply we will not be able to process credit card payments: which makes it rather compelling. One of the requirements is that we “Retain audit trail history for at least one year, with a minimum of three months immediately available for analysis…” There are also detailed instructions on how the audit trail should be managed.
- bureaucratic requirements. Bureaucracy has always laid down its own rules. In the UK, the government’s Code of Connection (CoCo) mandates that organisations must create an audit trail of system activity in order to facilitate future investigations, and that they should do this by collecting and storing logs from relevant systems and devices. It’s not the law; but it is required by any public body (including the Law) or other approved organisation that wishes or needs to connect to the Government Secure Internet; which means that we cannot do business with government without CoCo even if we are government.
Compliance, as we can see, is a complex soup. Large companies with their own corporate lawyers could, in theory, employ a lawyer to specify exactly what laws and regulations they need to follow in order to conduct their business. But it wouldn’t be very satisfactory and few companies could afford to do so. So how can the average business manage its compliance requirements without breaking the bank? One solution could be to conform to the single most widely recognised and highly regarded set of security specifications currently available: ISO/IEC 27000. This has no validity in law; but it is widely believed that if we conform to ISO/IEC 27000, we will most likely comply with just about all legal, commercial and bureaucratic security requirements.
Summary: what to look for in LM
We have seen that LM will improve our security and is necessary for compliance with multiple targets that are often inconsistent and unspecified. But we have also concluded that to maximise our security and simultaneously obtain the widest possible compliance at the minimum effort we can follow the LM requirements specified in ISO/IEC 27000. Section 13 of 27002 concentrates on just this. It is entitled Information Security Incident Management, and specifies the necessary level of log management in considerable detail. What is particularly clear, however, is that great care should be taken in not just collecting and using event information, care should also be taken in preserving the integrity of that data. The driver is legally admissible forensic evidence. If the audit trail has been or could be corrupted or altered between its source and its analysis it cannot definitively prove the cause or perpetrator of the event. It is effectively and legally irrelevant.
But we are finally in a position to summarise what we need to look for in a Log Management product:
- the original logs should be maintained in their original format to ensure forensic readiness, to allow re-analysis under different analysis rules, and to allow use by other applications
- scalability: logs grow and continue to grow very rapidly; and if we grow the company as well, then we need a system able to handle a huge and increasing load – the logs of anything from ten to thousands of servers
- minimising spurious event data: storage requirements can be reduced with a system that offers control over the level of auditing undertaken; but note that this requires considerable care over the audit and retention policies implemented
- central store: logs should be collected from throughout the enterprise into a central database
- the transfer of the logs from source to database must be done securely in a manner that retains forensic capability
- the database must be secure, so that its own integrity, confidentiality, availability and accountability – and completeness – is maintained
- features should be available to allow logs to be indexed to provide rapid and unstructured queries as required
- extensible rules-based analysis should be available to allow alerts on new types of threat
- it should be managed centrally in a virtualised environment to ensure ease and flexibility of use.
Choosing the right log management system will inevitably improve security, almost certainly help with compliance, and probably save us money.
1 The other pillars are availability (information should be available to authorised people when they need it), confidentiality (sensitive information should only be available to the people authorised to see it) and integrity (information should not be subject to any unauthorised alteration): and all four are relevant to any discussion of log management. (back)