Splunk - Boss Of The SOC!? - Wiki

Tags: #<Tag:0x00007f9ff788c770> #<Tag:0x00007f9ff788c630> #<Tag:0x00007f9ff788c4f0> #<Tag:0x00007f9ff788c3b0> #<Tag:0x00007f9ff788c270>


This Digital Forensics and Incident Response (DFIR) Wiki article uses an artificial research data-set to exemplify analysis steps for the purpose of “Threat Hunting”. Threat Hunting is a term, that is replacing “Log Analysis” here, because many security professionals believe that it needs to evolve. Beyond Logs for example…

The user names and corporations are generic, and do not identify individuals, trademarks or corporate entities.

The following examples use the freely available version of Splunk, which is a widely distributed Log Management and Analysis product from Splunk Inc..

Splunk by example with BOTS

This article uses the Splunk BOTS data-set in order to exemplify some basic SOC skills with Splunk.

Prepare a Splunk server based on Ubuntu Server

The start-point is a basic Ubuntu Linux VM on a VMware ESXi in the lab:

[email protected]:/data/import$ cat /etc/lsb-release 

I use Splunk 7.1, which is located in /opt/splunk

[email protected]:/# /opt/splunk/bin/splunk start

Splunk> Map. Reduce. Recycle.

Checking prerequisites...
	Checking http port [8000]: open
	Checking mgmt port [8089]: open
	Checking appserver port []: open
	Checking kvstore port [8191]: open
	Checking configuration...  Done.
	Checking critical directories...	Done
	Checking indexes...
		Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
	Checking filesystem compatibility...  Done
	Checking conf files for problems...
	Checking default conf files for edits...
	Validating installed files against hashes from '/opt/splunk/splunk-7.1.0-2e75b3406c5b-linux-2.6-x86_64-manifest'
	All installed files intact.
All preliminary checks passed.

I mounted a disk in /data for the Splunk Indexes:

[email protected]:/# grep data /etc/fstab 
/dev/sdb	/data	ext4	defaults,noatime	0 	1

And softlinked the var folder so that the indexes are not on the /:

lrwxrwxrwx  1 splunk splunk       5 May  6 18:46 var -> /data
[email protected]:/opt/splunk# 

There are more elegant ways to do this! You can set the $SPLUNK_DB variable.

Import an index

Let’s investigate the file listing:

[email protected]:/data/import$ tar -tzf botsv1_data_set.tgz 

Ok, looks good. Let’s stop splunk.

Let’s just add the index botsv1 to the indexes in /data. First I moved the data:

[email protected]:/data# ls -al | grep bot
drwx------  6 splunk splunk  4096 Sep  8  2016 botsv1

I edited the respective confg manually, and added this at the end:

From /etc/system/default/indexes.conf/indexes.conf:

homePath   = $SPLUNK_DB/botsv1/db
coldPath   = $SPLUNK_DB/botsv1/colddb
thawedPath = $SPLUNK_DB/botsv1/thaweddb

Now check…

/opt/splunk/bin/splunk start
Checking indexes...
	Validated: _audit _internal _introspection _telemetry _thefishbucket botsv1 history main summary

It’s in there, locked and loaded:


Now let’s just make sure the role you see fit for the user that conducts this SOC exercise is configured to search the included botsv1 index by default.


This way the Apps are going to be able to find the data, and you do not have to edit all the search queries.

Install the Splunk Apps

After installing some Apps you can go to the Apps Dropdown menu, and just check that you have some of the things:

Next goto Splunk Security Essentials and navigate to the Overview:

Welcome to the SOC?! Well no… this is an example. Isn’t it?

From pretty pictures to tactical SIE analysis

First of all we need to become familiar with the source types in the index. Because I am a frequent user of the Splunk Python SDK I will add a search keyword in the first line of the Splunk Search Processing Language (SPL) {2}.

index=botsv1 earliest=0 | top limit=20 sourcetype


What do we see:

  • Suricata - long story short: EVE JSON output
  • types of WinEventLog, that should get parsed correctly so that you get available fields such as LogName, SourceName {3}
    • (?) AD Logs?
  • logs from a Fortigate UTM, which essentially is a FortiOS with services like a VPN, a Web-Proxy, AV, Firewall and other elements of an endpoint security architecture
  • Splunk Streams, which represent packet dissection for DHCP, LDAP, HTTP, SMB, DNS…

What do we learn here:

(a) we need to cross-examine the logs from multiple systems in order to derive the magnitude and credibility of our Threat Hunting analysis.
(b) Splunk adds parsers via the Apps


{1} Splunk Python SDK examples with the search keyword

{2} Splunk SPL

{3} SANS paper Detecting Security Incidents Using Windows Workstation Event Logs by Anthony Russ

Introduction for Sysmon EDR with Splunk

Splunk exemplifies that you can use Microsoft SysMon as a Enterprise Detect and Response (EDR) tool for Digital Forensics and Incident Response (DFIR) on Microsoft Windows systems {1} {2}.
You can archive similar results with an ELK stack {2.1}.

In order to stream Sysmon event logs to Splunk you need to use a specific configuration so that the fitting Technology AddOn (TA) {3}.

The Sysmon TA comes with parsers that will extract the fields so that you get a statistical overview from the Sysmon logs in botsv1 (given that you configured it as an additional default index for the active user role, that opens the app to investigate the logs).

What we see:

  • we have 830 000 log events (this is not equal to log lines)
  • we have 2 spikes: (1) for Mon, Aug 8 and (2) Mon, Aug 22
    • the timeframe does not indicate that this is in the mornings, but if it could be the scheduled reboot after Patch Tuesday. Keep that in mind.
  • the overall event breakdown indicates around 80% of Sysmon’s logs being network events
  • from the users in the logs we can see that we are dealing with WAYNECORPINC (Windows Domain)
    • there is an IIS APPPOOL domain {4.1}, which indicates that Microsoft IIS is being used. The user in that domain is called joomla, which is a popular Content Management System (CMS) that has got many Common Vulnerabilities and Exposures (CVE) entries {4.2}
  • we can see Windows’ system users, such as local NT Authority users {5.1}, Administrator {5.2}, DWM-# {5.3}, which are related to the Windows NT architecture
    • at this point in time we should ask why we can see local user activity in the WAYNECORPINC domain

{1} Sysmon Splunk App GitHub repository

{2} Sysmon DFIR config for EDR purposes
{2.1} Sysmon DFIR Readme

{3} Microsoft Windows Sysinternals Sysmon

{4} Windows Domains and Workgroups
{4.1} Microsoft IIS Application Pool Identities
{4.2} Joomla CVE statistics

{5} Architecture of Windows NT
{5.1} Windows LocalSystem and NetworkService users
{5.2} Windows Administrator and SYSTEM accounts
{5.3} Windows DWM user

{6} Effectively Enhancing our SOC with Sysmon & PowerShell Logging to detect and respond to today’s real-world threats

Introduction to Windows EventLog analysis with Splunk

Microsoft Windows EventLogs can be filtered down within Splunk as well. There are various strategies on how to pre-filter and forward the relevant security logs. Current advice may include to use an agent, that can filter EventLogs based on queries {2}. If you use Splunk’s Universal Forwarder you can set filters.

If you lean towards the ELK stack, the WinLogBeats agent {3} has options to pre-process and filter events before they get forwarded to Elasticsearch or Logstash.

Either way you are centralising the logs, and parse them for analysis. In regards to Threat Hunting via Windows Eventlogs the question is what to look for.

to be continued

Audit Failure - example search

index=botsv1 earliest=0  sourcetype="WinEventLog:Security" Keywords="Audit Failure" | top limit=20 Account_Name20 Account_Name

{1} Splunk and Windows Event Log: Best Practices, Reduction and Enhancement

{2} NxLog EventLog query

{3} WinLogBeats Drop events feature