From the compliant log monitoring to advanced threat detection

Not very long time ago networks were small and computer hackers are rare. Now there are a lot of transnational enterprise networks and hackers were transformed to cyber criminals that use found vulnerabilities to steal money on routine basis. This article will describe importance of log monitoring and which solution may be used to simplify it and make more efficient.

Why to keep logs?

For many organization the question “why to keep logs” has a very simply answer – “because we have to do”. In fact there are many regulations and laws that require gathering and keeping logs. There are:

  1. PCI DSS (Data Security Standards)
  2. SOX
  3. Gramm-Leach-Bliley Act (GLBA)
  4. Many others! (VISA CISP, FFIEC, Basel II, etc)


The PCI DSS requirement #10 requires “track and monitor all access to network resources and cardholder data”. The requirement also requires keep audit trail history for at least one year and at least three months of history must be immediately available for analysis.


The “Security Standards for the Protection of Electronic Protected Health Information” requires keeping records about “action, activity and assessment”. Though it is not directly related to any network or firewall logs, it is worth to keep for example file server logs that indicate access to patient information.


The Gramm Leach Bliley Act requires keeping records about financial activity. The same thing like for HIPPA – electronic records about access to particular financial documents or controls have to be recorder and kept for further audit.

An organization may be subject of one or more regulations are mentioned above. It is hard and difficult job to meet all requirements and put required controls in place. Also it may be very expensive taking into account how much data an organization has to keep for the given period of time (from one month up to 6 years). The data may occupy many terabytes of disk or tapes.

But what is actual purpose of keeping all these data? Regulations usually state that records keeping are required to be able to audit usage of critical and confidential information and identify unauthorized and/or unintended access that may be malicious. It will not help you to identify threats in real-time and effectively block unauthorized access apply required countermeasures.

There are number of well known security breaches that highlight significance of ability to quickly identify and react on a threat.

TJ Maxx

March 2009. TJ Maxx reported more than 45 million credit and debit card numbers were stolen. Actual breach happened in 2006 or may be even earlier in 2005. Overall cost of fixing the computer systems, lawsuits, investigations, claims, etc is about $256 million.

EMS Corporation, RSA division

March 2011. RSA, the security division of EMC Corporation announced about discovered security breach. An intruder was able to silently penetrate to the RSA protected network and supposedly stole security material for RSA ID tokens (seeds). Total cost of fixing computer systems, replacing compromised tokens is estimated about $66 million.

HBGary Federal

February 2011. The “Anonymous” hacked the security company HBGary Federal. Company’s confidential information was published and company’s reputation was literally destroyed. The breach had significant resonance in the information security community.


In 2011 the Sony Game Network was hacked and more than 100 million customer accounts were compromised. Sony estimates that about 12 million credit card numbers were stolen. Estimate breach cost is about $170 million.

Pretty impressive, isn’t it? All above examples are just small part of security breaches, the most notable. As you can see, the last 2011 year was very productive for hackers.

But how it is related to the log monitoring? In each case log analysis and audit would help to identify source of attack and what an attacker did. The hack of HBGary made even log auditing is impossible: an attacker deleted all relevant logs and eliminate possibility of a “post-mortem” analysis. Making you compliant does not mean that you are able to successfully detect a threat and properly react on it.

Planning log management/collection

In general requirements to have a log management solution is driven by necessity to have visibility into ongoing activity in an organization network. In nowadays it is challenging for any non-tiny network – too much happens at any moment of time. And it is why you have to monitor the network.

A log management solution should be able to collect logs from constantly growing network during reasonable period of time. Usually minimum 1 year. There are few factors to consider during the planning phase:

  1. What regulations dictate to collect?
  2. What are important activities happened in your network outside of regulations? It may be business application logon/logoff, internal web servers access, email exchange, network communication with critical internal servers, VPN (virtual private network) access, etc.
  3. Scalability of a chosen log management solution.

Scalability is very important. It is common thing when a log monitoring solutions is deployed and more a more servers start feeding own logs to the system and soon it was discovered that amount of logs exceeded the limit. As the result the log management solution should be expanded or completely redesigned because wrong assumptions about scalability were made. It may cause unplanned expenses to deploy a renewed adequate log management solution that meets your needs.

Types of Log Management solutions

There are two major types of log management solutions: host based and distributed.

Host based solutions

A host based solution usually is based on a single server that accumulates all logs from all connected devices (servers, workstations, network devices, etc).

Figure 1 Host based log management system


In addition the central storage for logs (database) may be hosted on a separate server and the primary server hosts the application.

Figure 2 Application server and database servers


Advantage: easy to manage

Disadvantage: scalability issues

Distributed solutions

A host based solution may be not suitable for a network with huge volume of events. For example if estimated event flow is 100,000 EPS (events per second) you definitely need a distributed solution.

Figure 3 Host based log management system


Transition to SIEM

The picture above depicts a set of log management appliances (servers) that collects logs from many servers. Such architecture allows expanding the log management infrastructure pretty easily. Some solutions allow peer-to-peer communication among nodes to run queries and reports across all log management servers from one of them.

Advantage: scalability

Disadvantage: expensive, management is complicated (multiples appliances/servers)


Let’s assume that a log management solution has been successfully deployed in your network. It gathers logs from multiple devices such as servers (OS-levels logs), business applications, web servers (Apache and IIS), network devices (CISCO, Juniper, etc), workstations (Windows, Linux, Mac OS X), etc, etc. You can run report to see successful user logon, failed user logon, attempts to reach some of servers, and so on and so on. Will it help you to quickly identify unauthorized access attempts, malicious propagation to your protected network? Probably not. The reason is simple: usually log management solutions have lack of automated threat detection and alerting. They allow to efficiently collecting logs, run reports against them but there are some issues:

  1. Heterogeneous logs;
  2. Inability to detects and alert about threats in real-time.

A system that allows addressing these gaps is called Security Information and Event Management (SIEM).

Heterogeneous logs

Each vendor has own format for produced logs. See below few examples (not real logs though):

Juniper Netscreen
2012-02-21 13:49:37 – GHH-Server - [] user01
Aug 13 19:37:59 basta sshd[22308]: PW-ATTEMPT: joer
Augl 13 19:37:59 basta sshd[22308]: Failed password for root from


If it is required to identify an intruder that used stolen credentials to successfully login via SSH, it will not easy task taking into account that logs from different sources (network routers, SSH logs, etc) have to be analyzed. Furthermore, usually vendors use many “sub-formats” to produce log records in a single product  and it also make analysis of logs more difficult.

Some SIEM vendors offer a solution. For example ESM ArcSight unifies incoming logs and converts them to a single Common Event Format (CEF).

Triggers and rules

A SIEM system has ability to put special triggers or rules to warn a security analyst that some conditions a met that may indicate a potential security issue.

A good example is IDS (Intrusion Detection System). Typical IDS is very “noisy”. It produces a lot of events (signatures) that may indicate a potential issue. Another issue is that many legitimate systems may triggers IDS signature and also produce events. Also inability to visually distinguish between events from internal and external networks make situation even worse. Monitoring IDS in a big network is literally nightmare. It is easy to miss something really important. A SIEM solves the challenge: instead of continuous 24/7 monitoring, a couple of rules may be put in place to monitor specific cases:

  1. Successful brute force attack: many failed logons – one successful logon from the same host;
  2. Attempts to brute force: more than 10 failed logons from the same host;
  3. Successful logons within short period of time from distant geographic location using the same user id.
  4. Detection of vulnerability on a scanned host and further detection of attempt to exploit the vulnerability may reveal what is called APT – advanced persistent threat.

The examples #3 and #4 require creating a correlation rule that compares fields from events are belonging to different sources (IDS logs, vulnerability scanner logs). The list of use cases is expandable.


One of evident advantages of a SIEM is ability to visualize historical events or real-time events. The picture below is an example how ESM ArcSight shows interaction among host basing on network and IDS events. This is a good example of the proverb: “a picture is worth a thousand words”. Quick look on a dashboard may quickly reveal a propagating attack.

Figure 4 ESM ArcSight visualizes hosts interaction

Scalability issue

A SIEM system is powerful, it compares all incoming events against deployed rules and this fact instantly highlights the key disadvantage of a SIEM system: it is resource consuming. All existing SIEM are host based (a single application server) and cannot handle event flow more than 5,000 EPS (events per second). Using other words – SIEMs solution are not truly scalable. There are some workarounds though: deploying hierarchy of SIEM servers. Event flow is distributed among the-same-level SIEMs and low-level SIEMs forwards just correlated events and alert to the next level (one or more SIEMS servers). Here is a typical hierarchy:

Figure 5 SIEM servers hierarchy

SIEM and Log management solutions

According to the Gartner’s report was published in 2011, there are few leaders on the SIEM market:

  • HP/ArcSight – ESM
  • IBM – Q1
  • RSA (EMC)-  enVision
  • LogLogic
  • NitroSecurity
  • Symantec – Security Manager

All products offer similar features, each of them has own advantages and disadvantages. Choosing right SIEM solution is not easy task.


Log management systems and SIEM systems has a common feature: all of them collect and aggregate logs. It raises an evident question – is it possible to deploy just a SIEM and don’t use a separate log management system because the SIEM already collects the logs? The answer completely depends on your needs. If estimated volume of gathered logs can be accommodated by the SIEM system then it may be used as a log management system and satisfy regulatory requirements. In case of a big enterprise network expected volume of logs will be big enough to think about deploying both solutions: A log management system to satisfy regulatory requirements and a SIEM to meet security needs. It worth to think about interoperability of the chosen log management system and the SIEM. The log management system may be used to feed selected logs to the SIEM for security analysis and decrease overall cost.

Here is a short summary about advantages and disadvantages of SIEM and Log Management systems.

  Pro Contra
SIEM Real-time threat detection. Alerting and visualization Poor scalability
Log Management system Good scalability Lack of real-time threat detection and alerting

Leave a comment