IBM QRadar - Security Information Event Management - it's information management

Tags: #<Tag:0x00007f8a11430128> #<Tag:0x00007f8a11430060> #<Tag:0x00007f8a187bfeb8> #<Tag:0x00007f8a187bfdf0> #<Tag:0x00007f8a187bfd00> #<Tag:0x00007f8a187bfc10> #<Tag:0x00007f8a187bfb48> #<Tag:0x00007f8a187bfa80> #<Tag:0x00007f8a187bf9b8> #<Tag:0x00007f8a187bf8f0> #<Tag:0x00007f8a187bf828> #<Tag:0x00007f8a187bf738>



IBM QRadar is one of the most popular SIEMs (2017), and it’s a very powerful security (analytics) product, especially for SOC[1] related tasks.

It allows deep customisation and integrations with many (existing) components of your security infrastructure.

There is a Community Edition available: Check this here. I have installed it on CentOS 7, and it works[tm]. If you are interested in integrating QRadar CE with some OpenSource tools like pfSense, Nginx, Apache / mod_security etc. you have come to the right place. Work in progress[tm]. Note: it was :slight_smile:

QRadar? Isn't that like...

If you are interested in a competitive product analysis, you can check the Gartner Magic Quadrant or Forrester market research articles…

These generic prescriptive comparisons have very little meaning. Orgs have individual and key business security requirements. – Just because there is a dot in the leader quadrant that doesn’t mean that it will lead you somewhere…
– This isn’t well understood in InfoSec management these days. You cannot buy security. You cannot outsource responsibility. But InfoSec problems usually do not withstand targeted problem-solving. QRadar can get you there, but it’s a long road.

On the downside QRadar is one of the SIEMs with a rather complex / costly licensing model. It can become very expensive if you don’t have your goals, a clear security strategy and a lot of confidence in it.

I have had eloquent (FUD-master) consultants from IBM in my office, who told me (and my boss) that IBM Watson, Machine Learning and Artificial Intelligence, can replace /me (as security manager). “Today”.

One of these IBM consultants was in his 50ies, and he was a strong believer in both IBM and rapid progress. While I personally admire innovations, and I have a good retirement plan, daily reality is that many InfoSec tasks cannot be automated (yet). Neither with a SIEM, nor with ML / AI. I haven’t seen a SIEM doing the security awareness training, or Data Privacy Management. But if you have one, that does all of this, please…

Little QRadar can automate a lot of the pesky tech tasks. log analysis and correlation. It’s awesome for SOC analysis. It can run scripts once it detects events. Send alerts and notifications. You can team up on resolving issues. It’s feature-rich and mature.

And that is why this little Wiki article is worth a read or two. This is not the QRadar manual. This is an independant perspective.

Do SIEMs catch attackers - QRadar versus data-thief!

A SIEM is only as good as your security strategy.

Now… this is not what you wanted to hear. I know. Most security managers have a… ah “security strategy”. One, that does not allow tactical inter-operation between offence and defence. What does that mean?

– These strategies do not account for the fact, that attackers move strategically. Attackers are intelligent, prepared, committed and well funded. Ransomware groups are organised.

A SIEM is a strategic information resource, for intelligence gathering and analysis. Its purpose is to break down information into actionable items. Into indicators of success; or failure. Of potential compromise. Likelihood and magnitude. Metrics for the security program, and how to measure its individual performance points.

SIEM FAQs that aren't frequently asked

Do SIEMs catch attackers?

– I hope one day the police will do that… You may reveal attacker activity, if you know how what to look for. SIEMs are made to reveal suspicious activity to guide Threat Hunting (proactive) and Incident Response (reactive). In other words: they are a system to setup informative and monitoring controls, which are often required as compliance related control objectives.

Do SIEM boxes detect attackers without configuration? Can I just tick the box because I bought the PCI SIEM?

– No, you cannot do that. Sales says yes in pre-sales and no afterwards.

Do SIEMs work without security staff?

– A SIEM project usually is relatively complex and challenging. Not necessarily on the tech aspects. There are other factors of involved, besides licensing.

Can SIEMs be attacked? Do they have product security vulnerabilities on their own?

– Yes. You need to patch SIEMs.

So SIEMs are no silver bullets, and just like any other IT project, limited in their return of investment and core functionality.

Consequences of SIEM dreaming - ignorance is bliss

SIEMs don’t work and don’t catch attackers for most companies out there: SIEMs do not dummy down security.

If you are looking for something like that, you need to get a Managed Security Service Provider (MSSP), write Policies and Procedures, outline a project plan and perform a stakeholder and business requirement analysis for information security… you know the drill. Hey, and Patch Management. Yes, that as well. And what about Whitelisting… and what about …

Why mention all of this here: because SIEM deployments often fail due to a lack of consideration on exactly these aspects. Seriously: please read this twice. Then get a chimney and throw all the brochures from IBM, LogRythm, Splunk, HP, … into it for some divination magic to find out what the best SIEM is. Because evaluations aren’t common for these kinds of products. QRadar is not the best SIEM. It’s just the one, which gets written about here.

QRadar - high level overview

In 2 points, hands on:

  • QRadar’s focus is defined by network zones. Assets exist in their zones, which have different security requirements. In these zones certain events have a different impact on the overall security posture.
  • QRadar will not keep raw logs, only normalised data. if you need raw logs for certain IR forensic tasks, you need to check for an additional solution
  • The underlying tech is a RHEL / CentOS, PostgreSQL and Java. There is an RDS overlay in QRadar called Aria, which is optimised for Log Ingest and searching.

QRadar CE - lab development

There is a Community Edition available: Check this here. It will allow you to use 50 log events per second and 5000 flows (Netflow) per minute.

md5sum QRadarCE7_3_0_20171013140512.GA.iso
936277622e6f382f0f20056489a72981  QRadarCE7_3_0_20171013140512.GA.iso

In my test lab I feed QRadar from ipt_netflow and from pfSense’s softflowd. Works for me[tm]. Testing is key.

QRadar and the Network Hierarchy

This should be the first step during a setup.

Check this button in the Admin tab:


And put in the CIDRs from the docs:


Also make sure you add the external IPs somewhere, and tag the gateway points for various kinds of ingress and egress for your hosting platforms.

This adds context to the assets. Next question: how do the assets get into QRadar?

QRadar and the Asset Discovery

Asset Discovery in QRadar can work passively with logs. Or actively with a Vulnerability Scanner. Personally I really don’t get why we use Vuln Scanners for Asset Discovery. There are many better ways to do this actually.

Tenable Nessus and IBM QRadar for Asset Discovery

It’s important to do these things in the right order:

1.) If you use Nessus >= 6 (with the JSON API) the following suggestions may apply to you. Be patient.
2.) Setup a Nessus user for QRadar, login to Nessus as that user, and create a profile named “Basic Scan” with the default vuln scan settings (this is just for active asset discovery)
3.) Check out the screenshot:


4.) What to check for: can the QRadar appliance connect to the Nessus instance? Does the “Basic” profile / scan Policy Name exist? Are the CIDR ranges of the scan job scope really your’s? Do you really use the JSON API?

Double check, click save and schedule the scans from the Admin tab. After the first scan job has finished lots of assets should get populated into QRadar and you can start with the Asset and Vulnerability Management procedures.

In a similar fashion, with different vuln scan profiles, you can get different vulnerability discovery results. But that’s a different topic.

I don’t recommend to use surface scans for internal asset discovery. There are Agents for that, which can be rolled out.

QRadar and the Logs - there is a DSM?

Generally speaking: QRadar 7 auto-configures and guesses the Log ingest type right, if there is a DSM. There are many available integrations.

Creating an LSX to get support for Darktrace

For Darktrace (DT) (an Anomaly Detection System (ADS)) there is no pre-defined integration.

In one of my earlier blog posts I just threw out some XML / RegEx code for an LSX for Suricata (which is an OpenSource IDS - like Snort on steroids). We can just use something like that again.

Darktrace -> Syslog LEEF -> QRadar -> Event Management and Correlation

The workflow here starts with Syslog LEEF, because it’s the most compatible start point for this specific Log Source.

Step 1: Logging and a simple format analysis

So first of all we need to setup Darktrace to send logs to our QRadar app.


Then… let’s take a look at the LEEF Syslog input from the Darktrace Source in our QRadar.

<165>Nov 14 10:24:30 
Compliance/File Storage/Dropbox|externalId=12345
src=    dst=    

The line breaks are added for readability only.

Step 2: Regex prototyping and why we cannot use GNU grep for this
grep -Po "dst=[\d{1,3}\.]+" sample.log

QRadar will use Java style RegEx. grep's -P is for PCRE mode. grep's default is POSIX mode, if you don’t specify anything. – But we can prototype the RegEx in QRadar with the extractor dialogue.

Step 2.5: RegEx prototyping

You can open up an event, and simply extract a property in the UI (click Extract Property)

Take this simple RegEx for DT’s event names: Darktrace\|\w+\|\d+\.\d+\.\d+\|((\w+|\s|/)+)


This matches strings like:

Darktrace|DCIP|3.0.3|Compliance/File Storage/Dropbox
Darktrace|DCIP|3.0.3|Anomalous Connection/Multiple Connections to New External TCP Port

It extracts the last section. Now here is the catch:


Do you see Event Name somewhere? Anything like it? Nope… so how do we name the events? Oh my… LSX. This is a major drawback. You need to do some coding.

Strep 3: the first parsing step for an LSX for Darktrace

The LSX XML needs to map the fields of the log line. The template is based on this walk-through document from SANS.

    <?xml version="1.0" encoding="UTF-8"?>

    Author:				Your Name <[email protected]>
    Device Type:		Example SampleTron 5000 (FakeOS)
    Device Version:		Fakeware 2.7.1
    Protocol:			Syslog

    Custom Property regular expressions for Event Viewer:
    Sample-ID:			\sPolicy\sID\:\s(.*?)\;
    Sample-Group:		\sGroup\sName\:\s(.*?)\;

    Common Regular Expressions:
    IP Address:		\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}
    Port Number:	\d{1,5}
    MAC Address:	(?:[0-9a-fA-F]{2}\:){5}[0-9a-fA-F]{2}
    Protocol:		(tcp|udp|icmp|gre)
    Device Time:	\w{3}\s\d{2}\s\d{2}:\d{2}:\d{2}
    White Space:	\s
    Tab:			\t
    Match Anything:	.*?

    <device-extension xmlns="event_parsing/device_extension">
    	<!-- Do not remove the "allEventNames" value -->
    	<pattern id="allEventNames" xmlns=""><![CDATA[(.*)]]></pattern>
    	<!-- Everything below this line can be modified -->
    	<pattern id="EventName" xmlns=""><![CDATA[Darktrace\|\w+\|\d+\.\d+\.\d+\|((\w+|\s|/)+)]]></pattern>
    	<pattern id="EventCategory" xmlns=""><![CDATA[Darktrace\|\w+\|\d+\.\d+\.\d+\|(\w+)(\w+|\s|/)+]]></pattern>
    	<match-group order="1" description="Log Source Extension" xmlns="">
    		<matcher field="EventName" order="1" pattern-id="EventName" capture-group="1"/>
    		<matcher field="EventCategory" order="1" pattern-id="EventCategory" capture-group="1"/>
    		<event-match-multiple pattern-id="allEventNames" capture-group-index="1" device-event-category="unknown" send-identity="OverrideAndAlwaysSend" />

Now we upload it into QRadar, using the click-through dialogues:


Strep 4: Apply the LSX to a Log Source - LEEF


If you see this, you have an error in the XML. Maybe a missing ] at these nice CDATA segments:


Ok. Works. For me… I don’t have this error any more :wink:

If we extend the LEEF Log Source with an LSX, we don’t need the dst and src, because these are universal enough to be auto-assigned. Same for severity.

But yes: RegEx matchers in XML with CDATA elements. Great. Thanks IBM.

But, isn't this over the top?

Quick question: wouldn’t it be easier to dump all the logs into Splunk and to search through them directly?

Short answer: no, because you cannot evaluate dumps in an automated fashion. You need structured data for that.

Longer answer: every SIEM I know needs RegEx. Whether it’s ArcSight or Prelude Pro. Some people, when it comes to log tasks, try to make a point, that parsing isn’t necessary. - “Just throw the logs into ELK. SIEM done”. These people don’t work with log information, automated evaluation and correlation. These people don’t do threat hunting or incident response. And they have no idea about information models and security reporting.

Go, parse all the things. Because a log line in a SIEM without a parsing rule is a waste of memory.

Step 5: parse the fields and map them

This way we get the EventName (and other meta-data elements). We can teach the QRadar rule-engine to report on certain kinds of events in the Offenses tab later. I’d like to have a concrete numeric breach quantifier, but for now the sev field will do.
We can also get the EventCategory if we slightly modify the RegEx. Just add a matcher for this.

Another drawback with QRadar is, that these LSX definitions are not applied retroactively. That means any parsing and enrichment happens only after we add it. Although QRadar only works on normalized data. We have to wait, and debug. This costs a lot of time.

Summary: LSX = pain = gain

This is a good example of a task, that is often underestimated in deployments. At some point some new security system is bought. The vendor claims to be compatible, but of course… if you want to use the logs effectively within a 3rd party product you are on your own. The SIEM game at its best.

SIEMs can fail. Using them requires a set of skills, like RegEx, XML… lightweight coding. Still, it’s not everyone’s favourite task.

  1. Security Operations Center, often part of a Computer Emergency Response Team (CERT) ↩︎