Organizations currently monitor and manage the security of their infrastructures by setting up Security Operation Centres (SOC) to make security-related decisions (e.g., which system is under attack, what has been compromised, where has an access breach occurred, how many attacks have happened in the last 12 hours). A SOC obtains an integrated view of the monitored infrastructure by employing a Security Information and Event Management (SIEM) system. These systems that incorporate the functionality to collect logs and events from multiple sources, correlate these events together and then produce summarised measurements, data trends and different types of visualisations to help system administrators and other security professionals. SIEMs are traditionally are complex and costly to deploy and maintain.

The SIEM market is a growing one. According to a recent Gartner report (2016 Magic Quadrant for Security Information and Event Management), the SIEM market grew from $1.5 billion to approximately $1.69 billion, achieving a growth rate of about 14%. There are many high quality products from large IT vendors. Examples are IBM QRadar, HP ArcSight, Splunk, LogRhythm and AlienVault OSSIM. The spectrum of new attacks (with hundreds of novel kinds of malware each month, including the ones related with advanced persistent threats) and the complexity of the IT infrastructures require a well-structured and integrated monitoring of security events.

Despite their widespread use and the impressive market growth, current SIEMs still have many limitations:

1. The threat intelligence capacity of SIEMs is still in its infancy. Consequently, the systems are unable to automatically recognize novel threats that may affect (whole, or parts of) the monitored infrastructure, requiring considerable human intervention to adapt and react to changes in the threat landscape. This happens despite the availability of rich and up-to-date security-related information sources on the Internet (e.g., social media, blogs, security newsfeeds), which current SIEMs are unable to use.

2. Current systems can show any “low-level” data related with the received events, but they have little “intelligence” to process this data and extract high-level information. These low-level data (e.g., number of failed logins in a server) are only accessible and meaningful to a limited subgroup of system admins, and are difficult to translate to high-level metrics for senior, C-level managers (such as executives and decision-makers who may need to make decisions on security expenditure, but may not necessarily be well versed in the technical details). This impacts, for instance, the capacity of SOC coordinators to justify the return on investment in security for an organization.

3. Most data visualisation techniques in current SIEMs are rudimentary. Advanced data visualisation in current SIEMs is still limited. This can seriously impact the ability of the SOCs to deal with incidents as and when they happen, in a timely manner.

4. The event correlation capabilities of SIEMs are as good as the quality of the events fed to it. Imprecise events and alarms generated by imperfect monitoring devices will be taken as correct by the SIEM and the uncertainties associated with these events are never communicated.

5. Due to storage and event processing constraints, SIEMs are incapable of retaining the collected events for a long duration. This limits their use in conducting forensic investigations in the long run, for example on advanced persistent threats, or other historical incidents.

The Diversity-enhanced SIEM (DiSIEM) project aims to address these limitations by complementing existing SIEMs with a set of components for accessing diverse data sources, feeding enhanced events to the SIEM and generating enhanced reports and metrics to better support the security operation centres.