Inspec with Jenkins, Ansible and Splunk for compliance dashboards - self-auditing CI platforms

automation
jenkins
ansible
compliance
security_metrics
rsyslog
Tags: #<Tag:0x00007fbc55b77b68> #<Tag:0x00007fbc55b77a00> #<Tag:0x00007fbc55b778c0> #<Tag:0x00007fbc55b77780> #<Tag:0x00007fbc55b77618> #<Tag:0x00007fbc55b774d8>

#1

Compliance as code - automation is key

The following blog post documents an exercise with Inspec, Jenkins, Ansible and Splunk.

The core idea of the protoype is:

  1. Script reproduce-able audit tasks with Inspec and summarize a Baseline
  2. Schedule the system audits to happen automatically
  3. Report the results across all hosts within the environment
  4. Keep a track record of the compliance for external audits

This can be archived in simple steps.

What is...

  • Inspec {1} – it essentially allows to specify “Compliance as Code” and to automate the respective checks, e.g. based on the popular CIS benchmarks.
    Alerernatives to using Inspec can be CIS-CAT or even OpenSCAP.
    If you ever worked as a security consultant, chances are that you and your colleges shared a couple of assessment scripts to check the client’s systems. Inspec is a more comprehensive approach.

  • Jenkins {2} – Jenkins is used like a remote cronjob scheduler here. It regularly runs jobs, and registers errors. For many DevOps environments Jenkins is the heart of the Pipeline.
    Alternatives include TeamCity from IntelliJ, but I have never really used it.

  • Ansible {3} – is a simple IT automation framework, that is often used as a configuration management tool.
    Alerternatives include Puppet, Chef or… God forbid… cfengine.

  • Splunk {4} – is a commercial log-management and analytics platform for technical data. You can port the approach to ELK or Graylog with ease {4}

References

{1} Inspec - OpenSource Audit and Automated Testing Framework
{2} Jenkins - OpenSource Automation Server
{3} Ansible - OpenSource IT automation
{4} Graylog - OpenSource Log Management

Jenkins and Ansible?!

You can also use Ansible Tower, but using Jenkins allows this prototype to be more agnostic to the config-management / automation tool in question.

Regardless of whether it’s Puppet, Chef or Ansible… the workflow is the same. Generally the automation workflows are also Git centric. This part is skipped here, to keep it to the point.
Adding a prior git co / git pull isn’t the point. The point is to manage Pull Requests and branches.

The idea of using Ansible is, that it has got the inventory for the environment in scope. If you use the same config-management tool like Dev and/or Ops, you can skip the enumeration step for the most part in your assessments.

Generally neither Ansible nor Jenkins are typical Compliance tools. Due to this this blog post goes through a local example setup, with references on how to run the tests on remote hosts. Inspec can run commands directly via SSH on remote hosts. But you still have to save the output somewhere, and to report the results.

Setup Jenkins on Ubuntu Server 16.04 LTS

TThis is straight forward if you go for the package manager driven approach:

[email protected]:/etc/apt$ grep jenkins /etc/apt/sources.list
deb https://pkg.jenkins.io/debian-stable binary/

Then the Jenkins install may lead to something like this.

[email protected]:/etc/apt$ grep "Installed" <(apt-cache policy jenkins)
  Installed: 2.107.3

You cannot use Java 9 for Jenkins for now:

[email protected]:/etc/apt$ java -version
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

Start the jenkins service (it’s on TCP:8080) and install the Ansible Plugin.

10

So far: we have a standard Jenkins server installation with a common Plugin on a common platform, but we cannot use it with Ansible yet. Time to add this.

Setup Ansible on Ubuntu Server 16.04 LTS

True, Jenkins does have an SSH plugin, that can run remote commands. Chances are, that you will set it up, and realize that it doesn’t work as well as Ansible’s ad-hoc commands and Playbooks.
– And that you will need Ansible later anyways, if you want to deploy Inspec.

Setup Ansible with a user for Jenkins

On Ubuntu 16.04 LTS Server:

[email protected]:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.4 LTS"
[email protected]:~$ apt-cache policy ansible
ansible:
  Installed: 2.0.0.2-2ubuntu1
  Candidate: 2.0.0.2-2ubuntu1
...
[email protected]:~$ which ansible
/usr/bin/ansible

For now we don’t need a Playbook. Let’s add the local system first. Usually you would not do this, but this is a prototype environment.

Add to the end of /etc/ansible/hosts:

[local]
127.0.0.1

Test it with the ping module, and accept the local fingerprint if necessary (which is not printed):

[email protected]:~$ ansible local -m ping
127.0.0.1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

On the hosts, which get config managed, you can setup a password-less sudo option. This is required, because we want to automate admin stuff.

From the respective sudoers config:

# Allow members of group sudo to execute any command
%sudo   ALL=(ALL:ALL) ALL
jenkins  ALL = (ALL) NOPASSWD: ALL

Test it:

[email protected]:~$ sudo -i
[email protected]:/# cd /home
[email protected]:/home/# ln -s /var/lib/jenkins .
[email protected]:~# su jenkins
[email protected]:/root$ sudo ls
[email protected]:/root$ ls
ls: cannot open directory '.': Permission denied
[email protected]:/root$ ssh-keygen
[email protected]:/root$ ssh-copy-id [email protected]

Password-less sudo for local privileges?

In theory you could code the password into the Jenkins / Ansible build jobs, but this isn’t necessary here / insecure and not the standard way how you setup an Ansible Pipeline. You don’t want to spread credentials, if you don’t have to.

A better approach (for another blog post) is using Vault, which has a Plugin for Jenkins that can come in handy. This is fancy and the API keys are temporary. For a prototype environment this is not necessary.

Bottom line is: no hard-coded credentials. Yes, password-less sudo, but you need to exchange RSA keys first to get an SSH session.

Jenkins' user as Ansible user?

You could try run Jenkins’ jobs as a separate user, but that isn’t possible as far as I know. You can of course prevent the jenkins user to log in locally via SSH, and use a different user for Ansible on remote systems.

For this local test… we don’t care.

[email protected]:/root$  ansible all -m ping --sudo
127.0.0.1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

So far: we have added our local Jenkins host into the local Ansible config-management repo and setup a dedciated Jenkins SSH user for the auto-auditing pipeline that can run privileged commands.

Run an ad-hoc Bash command with the Jenkins SSH user

[email protected]:~$ ansible local -a "whoami"
127.0.0.1 | SUCCESS | rc=0 >>
jenkins

And with sudo:

[email protected]:~$ ansible local -a "whoami" --sudo
127.0.0.1 | SUCCESS | rc=0 >>
root

Fitting Jenkins and Ansible together

42

Here is the console output from the Job:

Started by user Marius
Building in workspace /var/lib/jenkins/workspace/Ansible Job - Test
[Ansible Job - Test] $ ansible local -a whoami -f 5
127.0.0.1 | SUCCESS | rc=0 >>
jenkins
Finished: SUCCESS

Generate the crontab for this

Let’s run this on weekdays, 15:00:

H 15 * * 1-5

Sounds like a sane time to run the compliance-scripts to audit the state on a daily basis.

The Jenkins job with an Ansible Playbook for Inspec CIS compliance checks

Finally we can write the job for Inspec, and forward the Output.

For this example the CIS Benchmark for Ubuntu seems to be fitting

[email protected]:~$ mkdir Source
[email protected]:~$ cd Source/
[email protected]:~/Source$ git clone https://github.com/wunzeco/cis-ubuntu-14.04-benchmark
Cloning into 'cis-ubuntu-14.04-benchmark'...
remote: Counting objects: 127, done.
remote: Total 127 (delta 0), reused 0 (delta 0), pack-reused 127
Receiving objects: 100% (127/127), 50.20 KiB | 0 bytes/s, done.
Resolving deltas: 100% (61/61), done.
Checking connectivity... done.

Cool.

[email protected]:~/Source$ realpath cis-ubuntu-14.04-benchmark/
/var/lib/jenkins/Source/cis-ubuntu-14.04-benchmark

Define expectations:

grep "%Compliance" <(inspec exec /var/lib/jenkins/Source/cis-ubuntu-14.04-benchmark --reporter=progress   | grep -E 'examples.*failures.*pending'   | awk '{ s = 100 * ($1 - $3) / $1; print "Summary: " $0 "\n%Compliance: " s }')
%Compliance: 73.1572 

With | logger -i -t inspec the oneliner will log the following line into /var/log/syslog:

May 25 15:07:03 shell inspec[60627]: %Compliance: 73.1572

We get the date, the host, the application[PID] and the message.

We obviously plan to edit the Inspec profile, so that it outputs the message as a CSV or as a JSON key-value scheme. This helps with the graphs in Splunk later. This is just a prototype.

A simple Inspec Bash script for Jenkins

Our Bash oneliner becomes a script:

#!/bin/bash

TESTDIR="/var/lib/jenkins/Source/cis-ubuntu-14.04-benchmark"

OUTPUT=$(inspec exec $TESTDIR --reporter=progress \
  | grep -E 'examples.*failures.*pending' \
  | awk '{ s = 100 * ($1 - $3) / $1; print "Summary: " $0 "\n%Compliance: " s }')

LOG=$(grep '%Compliance' <<< $OUTPUT)

echo $LOG | logger -i -t inspec

Let’s collect the path to define the job:

[email protected]:~/Source$ realpath inspec_run.sh
/var/lib/jenkins/Source/inspec_run.sh
[email protected]:~/Source$ chmod +x inspec_run.sh

36

[email protected]:~/Source$ sudo tail /var/log/syslog | grep inspec
May 25 15:22:24 shell ansible-command: Invoked with warn=True executable=None chdir=None _raw_params=/var/lib/jenkins/Source/inspec_run.sh removes=None creates=None _uses_shell=False
May 25 15:22:40 shell inspec[68748]: Summary: 719 examples, 193 failures, 35 pending %Compliance: 73.1572

If you want to report which tests have failed, you should change the output format first. But you can also just echo the variable, of course.

The Ansible Inspec Playbook for remote hosts

For remote hosts you’d put this into a Playbook.

---
- name: Transfer and execute a Inspec script.
  hosts: webenv
  remote_user: audit_user
  sudo: no
  tasks:
     - name: Transfer the script
       copy: src=inspec.sh dest=/home/audit_user mode=0777

     - name: Execute the script
       command: bash /home/audit_user/inspec.sh

The output will still be in Jenkins:

Started by user Marius
Building in workspace /var/lib/jenkins/workspace/InSpec Ansible
[InSpec Ansible] $ ansible local -m command -a /var/lib/jenkins/Source/inspec_run.sh -f 5
127.0.0.1 | SUCCESS | rc=0 >>

Finished: SUCCESS

You can see, that on remote hosts sudo isn’t required as well. This depends on the tests you want to run.

Relay to multiple destinations using the Rsyslog client

This is just a side note, but you may find this interesting if you have a local Rsyslog client, and you want to relay Syslog messages to multiple destinations:

$ActionQueueType LinkedList
$ActionQueueFileName Forward1
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
*.* @1.2.3.4:5514

$ActionQueueType LinkedList
$ActionQueueFileName Forward2
$ActionResumeRetryCount -1
$ActionQueueSaveOnShutdown on
*.* @5.6.7.8:514

One @ in Rsyslog’s config syntax is for UDP, @@ is for TCP.

If you use TCP, you have to use a queue. Some services (like Lighttpd) may make blocking calls into the local logging, and stop working if you don’t queue up the messages.

The Splunk Compliance Dashboard

The motto of this final step of the exercise is:

What is being measured, improves

Setup a listener in Splunk

The Syslog ingest here is UDP:514:

52

And the result should be:

53

Check the inspec Syslog

search
index="linux_hosts" sourcetype="syslog" process="inspec"

26

I forgot to format the message properly, so let’s do a field extractor for this prototype. Show off with some RegEx magic at the final step!

--
Compliance:\s(?<inspec_com>\d+.\d+)

43

Damn, that works:

00

Fire up the Jenkins job a couple of times, and make a nice graph…

Now you can group hosts by source (if you have multiple), CIDR (if you like a challenge) or simply create a dashboard… and see which hosts do not fit the Compliance baseline and aren’t hardened / configured correctly / updated and so on.

All you need to know is one commit away. If you know the Inspec DSL, which is another topic.

Summary

The protoype delivers initial results, and based on the chosen technology there do not seem to be problems with maturing it and scaling it up to work for entire hosting platforms.

Changelog

25.05.2018 - published
26.05.2018 - added explanations why Jenkins + Ansible are a goodchoice to run Inspec. added option to report the exact test results to Jenkins.


The Zero-Trust proxy hype - can we reverse-proxify everything and ditch VPNs?