Keep it simple, keep it functional
… but don’t keep it stupid!
It’s hard to find a script-able Endpoint Protection. Most vendors are interested in keeping their solutions tied together, and locked down. They try to isolate their competitors and want to force customers to have to buy every solution from one vendor. Exclusively.
Carbon Black / Bit9 is a little different. In the following I show why this is extremely useful, and what you can do with the APIs.This is about Incident Response with automation. Potentially.
- Carbon Black is an Endpoint Security monitoring component. - Like Sysinternals with central logging, and threat feeds. These feeds range from VirusTotal to various IoCs.
- Bit9 is an Endpoint Security policy enforcement tool. It can block Malware binaries and apply many different other rules.
REST safe and sound: let's use the the Bit9 API
The Bit9 API is well documented. I use Python to get some data from it:
import requests, json
# --- Prepare our request header and url ---
bit9_apiUrl = 'https://bit9.sample/api/bit9platform'
This API token is important. It’s as powerful as the login to Bit9.
Asset lists from Bit9
You can retrieve asset information from Bit9, just like in the Web interface.
comps = requests.get(bit9_apiUrl + '/v1/computer?q=ipAddress:192.168.1.*', headers=authJson, verify=b9StrongCert).json()
my_computers = pandas.DataFrame( get_systems_in_admin_net() )
I use Pandas here, because it’s extremely simple to use.
The reply is:
Index([u'CLIPassword', u'SCEPStatus', u'activeDebugFlags', u'activeDebugLevel',
u'activeKernelDebugLevel', u'agentCacheSize', u'agentMemoryDumps',
u'agentQueueSize', u'agentVersion', u'automaticPolicy',
u'cbSensorFlags', u'cbSensorId', u'cbSensorVersion', u'ccFlags',
u'ccLevel', u'clVersion', u'computerTag', u'connected', u'dateCreated',
u'daysOffline', u'debugDuration', u'debugFlags', u'debugLevel',
u'deleted', u'description', u'disconnectedEnforcementLevel',
u'enforcementLevel', u'forceUpgrade', u'hasDuplicates',
u'hasHealthCheckErrors', u'id', u'initPercent', u'initializing',
u'ipAddress', u'isActive', u'kernelDebugLevel', u'lastPollDate',
u'lastRegisterDate', u'localApproval', u'macAddress', u'machineModel',
u'memorySize', u'name', u'osName', u'osShortName', u'platformId',
u'policyId', u'policyName', u'policyStatus', u'policyStatusDetails',
u'previousPolicyId', u'prioritized', u'processorCount',
u'processorModel', u'processorSpeed', u'refreshFlags',
u'supportedKernel', u'syncFlags', u'syncPercent', u'systemMemoryDumps',
u'tamperProtectionActive', u'tdCount', u'template',
u'templateDate', u'templateTrackModsOnly', u'uninstalled',
u'upgradeError', u'upgradeErrorCount', u'upgradeErrorTime',
u'upgradeStatus', u'users', u'virtualPlatform', u'virtualized'],
Any of these columns can be used problematically now. Let’s see who uses which IP in this network:
`print computers[[u'users', u'ipAddress']]`
0 None 192.168.1.1
1 Domain\Jack$ 192.168.1.2
2 Domain\Jim$ 192.168.1.3
3 Domain\jim_admin,Domain\jack_admin 192.168.1.100
So there is one shared workstation. Easy.
Alright.. 3 months of Malware alerts please and make it pretty
# based upon prior cells we re-use the bit9 root URL and API key variable
events = requests.get(
bit9_apiUrl + '/v1/event?q=timestamp>-12w&q=subtypeName:Malicious file detected',
mw_alerts = pandas.DataFrame( get_24h_mw_mfile_events() )
This gives us a very nice DataFrame object. A lot of things can be done with these infos. Let’s get the SHA1 hashes.
mw_alerts[u'description'] = mw_alerts[u'description'].map(lambda x: x.encode('ascii', 'ignore').decode('utf-8'))
mw_alerts[u'description'] = mw_alerts[u'description'].map(lambda x: str(x))
mw_alerts[u'sha1_hash'] = mw_alerts[u'description'].map(lambda st: st[st.find("[")+1:st.find("]")])
This adds a new column, and selects the string of the Malware alert description, which always is between square brackets. A short-coming here is, that Bit9 uses SHA1 but Carbon Black uses MD5.
We normalize the strings in the
DataFrame to be able to work with the columns. Possible tasks include sending the hashes to a threat feed or blocking them with some other tool.
For many visualizations, and especially for tend analysis, the time domain is very important. This way we assign frequency information to the alerts. Pandas can help to normalize the timestamp. It’s a one-liner:
mw_alerts[u'timestamp'] = pandas.to_datetime(mw_alerts[u'timestamp'])
Now let’s make it pretty. We will iterate over the alerts, and check if the reported file still exists on the client. Some Malware just lingers…
for i, alert in mw_alerts.iterrows():
computerName = unicodedata.normalize('NFKD', alert[u'computerName']).encode('ascii','ignore')
computerName_wo_Domain = computerName.split("\\")
file_hash = alert[u'sha1_hash']
desc = "-----------------------------------------------------------------------\n"
desc+= "*Affected computer:* " + computerName + "\n"
desc+= "*Malware-File:* " + alert[u'pathName'] + "\\" + alert[u'fileName'] + "\n"
# note that the filename changes per event
# we could make a dataframe with a sub-index for the fileCatalog reply
fileSearchURL = bit9_apiUrl + '/v1/fileCatalog?q=fileName:' + str(alert[u'fileName'])
# print fileSearchURL
fileInfo = requests.get(fileSearchURL, headers=authJson, verify=b9StrongCert).json()
hosts_with_file = fileInfo[u"prevalence"]
if hosts_with_file != 0:
desc+= "The file is present at " + str(hosts_with_file) + " systems at CCP\n"
desc+= "If this is larger than 1 this might be a Malware compaign.\n"
And for every
BANG there shall be block. You can easily kill and block the files now. I recommend using the cbapi package for that. It creates objects, to interact with Bit9 and/or CarbonBlack.
You could also script a Live Response shell command. But that is a little risky, if you know what I mean. I leave that to you and your staging environment. Read the
cbapi docs on how to manage the API key. Don’t hard-code it.
from cbapi.response import CbEnterpriseResponseAPI, Process, Binary, Sensor
Now that is all copy paste from the manual… Add this to the
for proc in c.select(Process).where('process_name:' + alert[u'fileName'] + ' AND hostname:' + computerName_wo_Domain):
desc += "Active Network connections of the Malware: " + str(proc.netconn_count) + "\n"
desc += "Operating system is " + str(proc.os_type) + "\n"
This should give you the following infos:
- a Malware is present on the endpoint, and there has been a Malware alert
- it exists as a process, and has got active network connections
- You will figure out which OS it is
- (And with Pandas it’s easy to make some user specific stats.)
Based on these infos you can isolate the host, or remote detonate the Malware. - Depending on how your SoC processes are defined. Essentially the coding consists of string comparisons and API requests. There isn’t any magic here. Building up CB queries, using an official API… even a security engineer can code with it.
Production servers, SoC and tickets with meaningful infos
Obviously, in a production environment, you don’t remote detonate processes with such a Python script. And auto-delete the file with the Live Response shell. Just in case you don’t know why: False Positives.
What you can do without much risk is generating a JIRA ticket using the
jira package (and the JIRA API). Most organizations use JIRA today, and it’s not much code anyways. You can run such a Python script via cron every 10 minutes with an adequate time frame in the search queries. I also recommend using a MongoDB to commit the retrieved information locally, and to query it before submitting a ticket via the API. This way you can track the state and avoid double-alerting.
Make a file named
jira.auth and insert a line like
with open('jira.auth') as f:
credentials = [ x.strip().split(':') for x in f.readlines()]
To illustrate why that is a useful way to avoid hard coded creds:
for user, passwd in credentials:
Now let’s push tickets to the SoC. Add this to the
authend_jira = JIRA('https://jira.ccpgames.com', basic_auth=(user, passwd))
new_issue = authend_jira.create_issue(project='1337,
summary='[Malware on ' + computerName + ']',
Now, why should you do this. You could also just let Bit9 or Carbon Black mail to the ticket system. The answer is that tickets, especially if you auto-generate them, need meaningful information. It’s not a good idea to write a ticket named: “Malware Alert” and to have a sparse description in there like “dd.exe has been found on Jack-WS”.
It’s a bad idea because it’s idiotic. Such alerts are meaningless waste of time and effort. Instead an alert, which contains relevant information, will be taken care of in a professional and serious manor. And I don’t say that this example here is complete. Obviously you can
- query threat feeds to add story
- grab a PCAP from the IDS ring buffer and upload it to a analysis system
- check for IDS alerts, which might indicate TOR connectivity or a Malware campaign
- grab the mail server logs, to check for the recent sender / subjects and attachments
- run the attachments through a Sandbox and add the logs to the ticket
- grab the Security Gateway logs from the central log management
- grab the DLP logs for this user and workstation
In short you can attach a lot of context. It’s important that you have script-able components to do that.
Making use of the CB and Bit9 APIs to manage Malware incidents is a win. Security tools without good and powerful APIs belong into a different decade. and it might come as a surprise to you, but IBM QRadar 7 has got a decent API. But that’s another blog post… soonish[tm].