Automate Incident Response with Carbon Black / Bit9 API

soc
python
data_science
incident_response
Tags: #<Tag:0x00007f0cb0542ea0> #<Tag:0x00007f0cb0542d60> #<Tag:0x00007f0cb0542c20> #<Tag:0x00007f0cb0542ae0>

#1

Keep it simple, keep it functional

… but don’t keep it stupid!

It’s hard to find a script-able Endpoint Protection. Most vendors are interested in keeping their solutions tied together, and locked down. They try to isolate their competitors and want to force customers to have to buy every solution from one vendor. Exclusively.

Carbon Black / Bit9 is a little different. In the following I show why this is extremely useful, and what you can do with the APIs.This is about Incident Response with automation. Potentially.

  • Carbon Black is an Endpoint Security monitoring component. - Like Sysinternals with central logging, and threat feeds. These feeds range from VirusTotal to various IoCs.
  • Bit9 is an Endpoint Security policy enforcement tool. It can block Malware binaries and apply many different other rules.

REST safe and sound: let's use the the Bit9 API

The Bit9 API is well documented. I use Python to get some data from it:

import requests, json

# --- Prepare our request header and url ---
authJson={
  'X-Auth-Token': '12345',  
  'content-type': 'application/json'
}
bit9_apiUrl = 'https://bit9.sample/api/bit9platform' 

This API token is important. It’s as powerful as the login to Bit9.

Asset lists from Bit9

You can retrieve asset information from Bit9, just like in the Web interface.

import pandas

def get_systems_in_some_net():
    comps = requests.get(bit9_apiUrl + '/v1/computer?q=ipAddress:192.168.1.*', headers=authJson, verify=b9StrongCert).json()
    return comps

my_computers = pandas.DataFrame( get_systems_in_admin_net() )
print admin_computers.columns

I use Pandas here, because it’s extremely simple to use.

The reply is:

Index([u'CLIPassword', u'SCEPStatus', u'activeDebugFlags', u'activeDebugLevel',
       u'activeKernelDebugLevel', u'agentCacheSize', u'agentMemoryDumps',
       u'agentQueueSize', u'agentVersion', u'automaticPolicy',
       u'cbSensorFlags', u'cbSensorId', u'cbSensorVersion', u'ccFlags',
       u'ccLevel', u'clVersion', u'computerTag', u'connected', u'dateCreated',
       u'daysOffline', u'debugDuration', u'debugFlags', u'debugLevel',
       u'deleted', u'description', u'disconnectedEnforcementLevel',
       u'enforcementLevel', u'forceUpgrade', u'hasDuplicates',
       u'hasHealthCheckErrors', u'id', u'initPercent', u'initializing',
       u'ipAddress', u'isActive', u'kernelDebugLevel', u'lastPollDate',
       u'lastRegisterDate', u'localApproval', u'macAddress', u'machineModel',
       u'memorySize', u'name', u'osName', u'osShortName', u'platformId',
       u'policyId', u'policyName', u'policyStatus', u'policyStatusDetails',
       u'previousPolicyId', u'prioritized', u'processorCount',
       u'processorModel', u'processorSpeed', u'refreshFlags',
       u'supportedKernel', u'syncFlags', u'syncPercent', u'systemMemoryDumps',
       u'tamperProtectionActive', u'tdCount', u'template',
       u'templateCloneCleanupMode', u'templateCloneCleanupTime',
       u'templateCloneCleanupTimeScale', u'templateComputerId',
       u'templateDate', u'templateTrackModsOnly', u'uninstalled',
       u'upgradeError', u'upgradeErrorCount', u'upgradeErrorTime',
       u'upgradeStatus', u'users', u'virtualPlatform', u'virtualized'],
      dtype='object')

Any of these columns can be used problematically now. Let’s see who uses which IP in this network:

`print computers[[u'users', u'ipAddress']]`

                                            users   ipAddress
0                                                None  192.168.1.1
1                                 Domain\Jack$  192.168.1.2
2                                 Domain\Jim$  192.168.1.3
3               Domain\jim_admin,Domain\jack_admin   192.168.1.100

So there is one shared workstation. Easy.

Alright.. 3 months of Malware alerts please and make it pretty

import unicodedata

# based upon prior cells we re-use the bit9 root URL and API key variable
def get_mw_mfile_events():
    events = requests.get(
        bit9_apiUrl + '/v1/event?q=timestamp>-12w&q=subtypeName:Malicious file detected', 
        headers=authJson, verify=b9StrongCert).json()
    return events
mw_alerts = pandas.DataFrame( get_24h_mw_mfile_events() )

This gives us a very nice DataFrame object. A lot of things can be done with these infos. Let’s get the SHA1 hashes.

mw_alerts[u'description'] = mw_alerts[u'description'].map(lambda x: x.encode('ascii', 'ignore').decode('utf-8'))
mw_alerts[u'description'] = mw_alerts[u'description'].map(lambda x: str(x))
mw_alerts[u'sha1_hash'] = mw_alerts[u'description'].map(lambda st: st[st.find("[")+1:st.find("]")])

This adds a new column, and selects the string of the Malware alert description, which always is between square brackets. A short-coming here is, that Bit9 uses SHA1 but Carbon Black uses MD5.

We normalize the strings in the DataFrame to be able to work with the columns. Possible tasks include sending the hashes to a threat feed or blocking them with some other tool.

For many visualizations, and especially for tend analysis, the time domain is very important. This way we assign frequency information to the alerts. Pandas can help to normalize the timestamp. It’s a one-liner:

mw_alerts[u'timestamp'] = pandas.to_datetime(mw_alerts[u'timestamp'])

Now let’s make it pretty. We will iterate over the alerts, and check if the reported file still exists on the client. Some Malware just lingers…

for i, alert in mw_alerts.iterrows():
    computerName = unicodedata.normalize('NFKD', alert[u'computerName']).encode('ascii','ignore')
    computerName_wo_Domain = computerName.split("\\")
    
    file_hash = alert[u'sha1_hash']
       
    desc = "-----------------------------------------------------------------------\n"
    desc+= "*Affected computer:* " + computerName + "\n"
    desc+= "\n"
    desc+= "*Malware-File:* " + alert[u'pathName'] + "\\" + alert[u'fileName'] + "\n"
    
    # note that the filename changes per event
    # we could make a dataframe with a sub-index for the fileCatalog reply
    fileSearchURL = bit9_apiUrl + '/v1/fileCatalog?q=fileName:' + str(alert[u'fileName'])
    # print fileSearchURL
    fileInfo = requests.get(fileSearchURL, headers=authJson, verify=b9StrongCert).json()
    hosts_with_file = fileInfo[0][u"prevalence"]
    
    if hosts_with_file != 0:
        desc+= "The file is present at " + str(hosts_with_file) + " systems at CCP\n"
        desc+= "If this is larger than 1 this might be a Malware compaign.\n"
    else:
        desc+= "BANG!"

And for every BANG there shall be block. You can easily kill and block the files now. I recommend using the cbapi package for that. It creates objects, to interact with Bit9 and/or CarbonBlack.

You could also script a Live Response shell command. But that is a little risky, if you know what I mean. I leave that to you and your staging environment. Read the cbapi docs on how to manage the API key. Don’t hard-code it.

import cbapi

from cbapi.response import CbEnterpriseResponseAPI, Process, Binary, Sensor
c=CbEnterpriseResponseAPI()

Now that is all copy paste from the manual… Add this to the for-loop above:

for proc in c.select(Process).where('process_name:' + alert[u'fileName'] + ' AND hostname:' + computerName_wo_Domain[1]):
    desc += "Active Network connections of the Malware: " + str(proc.netconn_count) + "\n"
    desc += "Operating system is " + str(proc.os_type) + "\n"
    sensors.add(proc.sensor)

This should give you the following infos:

  • a Malware is present on the endpoint, and there has been a Malware alert
  • it exists as a process, and has got active network connections
  • You will figure out which OS it is
  • (And with Pandas it’s easy to make some user specific stats.)

Based on these infos you can isolate the host, or remote detonate the Malware. - Depending on how your SoC processes are defined. Essentially the coding consists of string comparisons and API requests. There isn’t any magic here. Building up CB queries, using an official API… even a security engineer can code with it.

Production servers, SoC and tickets with meaningful infos

Obviously, in a production environment, you don’t remote detonate processes with such a Python script. And auto-delete the file with the Live Response shell. Just in case you don’t know why: False Positives.

What you can do without much risk is generating a JIRA ticket using the jira package (and the JIRA API). Most organizations use JIRA today, and it’s not much code anyways. You can run such a Python script via cron every 10 minutes with an adequate time frame in the search queries. I also recommend using a MongoDB to commit the retrieved information locally, and to query it before submitting a ticket via the API. This way you can track the state and avoid double-alerting.

Make a file named jira.auth and insert a line like jack:coolpassword

with open('jira.auth') as f:
    credentials = [ x.strip().split(':') for x in f.readlines()]

To illustrate why that is a useful way to avoid hard coded creds:

for user, passwd in credentials:
    print user
    print passwd

Now let’s push tickets to the SoC. Add this to the for-loop:

authend_jira = JIRA('https://jira.ccpgames.com', basic_auth=(user, passwd))
    
new_issue = authend_jira.create_issue(project='1337, 
                                      summary='[Malware on ' + computerName + ']', 
                                      description=desc, 
                                      issuetype={'name':'SoC Incident'}
                                     )

Now, why should you do this. You could also just let Bit9 or Carbon Black mail to the ticket system. The answer is that tickets, especially if you auto-generate them, need meaningful information. It’s not a good idea to write a ticket named: “Malware Alert” and to have a sparse description in there like “dd.exe has been found on Jack-WS”.

It’s a bad idea because it’s idiotic. :wink: Such alerts are meaningless waste of time and effort. Instead an alert, which contains relevant information, will be taken care of in a professional and serious manor. And I don’t say that this example here is complete. Obviously you can

  • query threat feeds to add story
  • grab a PCAP from the IDS ring buffer and upload it to a analysis system
  • check for IDS alerts, which might indicate TOR connectivity or a Malware campaign
  • grab the mail server logs, to check for the recent sender / subjects and attachments
  • run the attachments through a Sandbox and add the logs to the ticket
  • grab the Security Gateway logs from the central log management
  • grab the DLP logs for this user and workstation

In short you can attach a lot of context. It’s important that you have script-able components to do that.

Results

Making use of the CB and Bit9 APIs to manage Malware incidents is a win. Security tools without good and powerful APIs belong into a different decade. and it might come as a surprise to you, but IBM QRadar 7 has got a decent API. But that’s another blog post… soonish[tm].


Netflow data analysis with SiLK and Pandas
Malware Analysis: foundation level workflows