MicroService architecture can haz HIDS - dig into sysdig

Tags: #<Tag:0x00007f332d13d190> #<Tag:0x00007f332d13cfb0> #<Tag:0x00007f332d13cd80> #<Tag:0x00007f332d13cab0> #<Tag:0x00007f332d13c7e0> #<Tag:0x00007f332d13c628> #<Tag:0x00007f332d13c448>


One container to monitor them all...


Containers are a modern word for wrappers for (legacy) apps. In Docker “Containers” you can host old environments, in their own little (or not so little) compartment. Like waste separation. Make sure it does not mix. Otherwise the garbagemen are going to ring

Docker containers are like a Zone or a Jail, but unlike a LXC or even RKT. And don’t think of the whole Containerization like a New-Age User Mode Linux. It’s something entirely different. You need DevOps for this. Not just Devs. - Not just any garbagemen.

Container Management

And because we are so different, I mean efficient, consolidated [and so on, business blah bla] we have a Container Management system. An orchestration system. A cluster orchestration engine. A virtualised Micro-Service environment.

At the beginning of my career, virtualisation wasn’t very central to IT operations. The concept became approachable with Xen, 2003ish. It became popular around 2007 with hardware-assistance (what’s IOMMU / VT-x today).

Actually Xen introduced Unikernels in 2015, and you can think of Docker as a system as well, which shares the host’s kernel. Due to this the “Micro”-systems have a lighter footprint than Virtual Machines which each have their own kernel.
The lightweight way of compartmentalisation is an advantage, especially for hosting platforms that have to deal with performance requirements.

And Microsoft is 10 years behind

We need small operating systems for our containers today. That’s the reason why Microsoft introduced NanoServer and Windows Server Core, in order to fight for the last single digits of market share for their failing server products.

Their typical Windows Server [2016] base install is too large to compete against Linux here. - Because architecturally within hosting environments based on Docker, you strive to create many small Micro-Services. And you orchestrate them like a Swarm. Many small systems with a small footprints.
Windows is the big blue whale: not fast enough, not flexible enough, and no one understands its biorhythm. It sometimes falls asleep, and no one knows why.

Micro-Services are many - we have k8s, are we Goog yet

We split the swarm of Micro-Services and group them. Like a large house, with many tenants, which will have floors for certain kinds of people.


These groups have many different names. Needs. And life-times.

Let’s call them Pods, and focus on Kubernetes for the time being. Pods have these Containers. These can be on one or more Kube-Nodes, which are physical or virtual servers.

What a mouth full… Kubernetes sounds like a recipe for cucumber salad actually. There are a lot of SMBs with small-medium hosting environments, which think they are Google because they run some Docker containers. In reality you only need these additional layers if you are big and important. But Cloud providers saw that problem of misinformed small players on large scale hosting platforms and offered the Cloud-model: sharing is caring.

In the Cloud they already had IaaS and PaaS models. What the misinformed customers needed was something fancier and newer. Backend as a Service (BaaS) sounded too boring. Server-less architecture! A server architecture without servers. That sounded like a bestseller. Like fat-free fat, and sugar-free sugar. Or like an expensive cucumber salat, with an extra portion of nothing. Let’s share!

Now everyone can have cucumber-free cucumber salat… I mean groups of compartments on real or virtual boxes, on-premise or in the shared Cloud. And something manages these compartments across all the boxes. Something… secure? Something transparent? Something compliant? Nope.

DevOps like security?

If it doesn’t slow them down, they do.


Who? - Me? No. I am a DevOp! I don’t care about that. My focus is to be fast. Go way! You must make me a SecDevOp before I do that!

The challenges with continuous security-monitoring in a Micro-Service environment may include:

  • no fixed asset inventory
    • services are short-lived and get deployed often / continuously
    • Pods can migrate between Kube-Nodes to balance the load
      • stateless service design
  • additional layers of networking
    • medium to large Kubernetes hosting platforms may use Software Defined Networking (SDN) and a custom overlay network (something like Flannel)
      • that means visibility of internal service communication between the Pods might be difficult to obtain. For example a /24 from the distributed switches (vSwitch or NSX) and a /16 Flannel overlay across the Kube-Nodes is not easily SPAN-able.
        • typical SDN topics here are “micro-segmentation” (communication between hosts in the same network), dynamic SDN groups (less hardcoded variables in the ACLs), configs of the multi-functional appliances (an NSX appliance can be a border router with eBGP, a Core LBs, or even distributed and stateful network firewall gateway with specific ACLs…)
  • Docker containers may not have a libc or a Bash. Traditional Endpoint protection clients might not work due to this :slight_smile:
  • deploying Docker containers, where each instance has a client (OSSEC, Tripwire etc.) is serious overhead now.
  • Inside a Docker container services may run as root, because the environment does not provision specific users. Given that the compartment will only have one user anyway, it’s not very relevant if that is root or something like www-data
    • this can get changed within the Dockerfiles, but whether you want to do this isn’t a straight-forward evaluation.

Advantages may include:

  • Dockerfiles can easily be audited at the repository
  • short-lived Container instances may mean, that you won’t have persistent threats inside the platform unless your build chain has been infected and the threat gets re-deployed alongside the production services
  • Release builds can get tested for security issues
  • Post-exploitation within a Micro-Service environment might be complex, because you have to escape the Docker container to access the (virtual) host
  • Docker images are portable, so that it’s possible to replicate an environment
  • Docker images can get diffed

Some traditional security approaches may collide with Kubernetes:

  • Kubenertes has services, like kubeadm, kube-proxy etc. which run as root. There are SELinux policies files for Kubernetes.
    • But in the same sense you should take a look at VMware ESXi or VSphere… The principle of least privilege is a good orientation, but when it comes to infrastructure services it’s important to understand the requirements as well. Kubernetes needs to set iptables for the load-balancing and availability functions.
    • Kube-Nodes are not multi-user systems
      • with Linux based virtualization technologies like KVM there is separate user (usually qemu-kvm).

In short: there are major differences to consider with Kubernetes. It’s another layer on top of virtualisation for the most part.

Many companies have Kube-Nodes in VSphere, and connect this Micro-Service environment to Storage, Database and other systems, which are stateful / persistent.

And usually the Load Balancers / network edge devices remain, like an F5 or a distributed NSX Load-Balancer. Simply because of failover scenarios, BGP peering, distributed firewalls and so on. Kubernetes will not change everything. It’s just marketed that way. Which brings us back to the original issue: in case you don’t have load on a massive scale, you do not need the additional layer, which is k8s.

The SecDevOps MCORPG


In our SecDevOps Massive Container Role Play Game we have skilled speed, but forgot to check for resistance / resilience. Typical for DevOps, we must also skill awareness. I mean the good security guys.

Because they[tm] wrapped the legacy Apps in Dockerfiles and thought they’d be fine. Then they started to play the game like back in the days, when they thought Java would solve all security problems. It just never did. And we find out, that Java is a mess every time we take a look. Some story, different tech.

But the good news is, that we know what services we have in the Micro-Service environment, if we enumerate the Pods. Finally we have a (ITIL) Service Catalogue, and we can search for services if we enumerate the labels. We can also easily get an asset list, because there is one central overview of all the containers in the cluster. CMDB! All the things!

Are we there yet? Max level!!!

The problem is, when your Kube cluster hits max-level, you need some very special potions to skill resilience. And we aren’t going to get that much Acid from Gandalf, to be able to feed every single Container. So what we do is: we apply it to a Kube-Node, and not to all the Containers. One Kube-Node at a time.

Sysdig (Falco) is a Container, that gets deployed on the Kube-Nodes. And it works for all of the other Container instances on that node. It peaks into them via the Docker sockets and some proc / sys read-only mounts. Then it collects the stuff into a trace, and feeds this into a central server (which is only available in the Enterprise version).

A Container to secure containers

So why mention all the stories about virtualization, DevOps etc… Simple answer: most security people have little or no understanding about these things. They start their engagement, fire up a couple of audits, and tick a couple of boxes. In this particular scenario this leads to many issues.

Sysdig has limits. It’s not like a Next-Gen Endpoint Protection / Incident Response tool, and not like CarbonBlack Enterprise or CrowdStrike Falcon.

Sysdig probably doesn’t hit the security project’s agenda without a Compliance requirement. Something like ISO 27001, and DevOps having Dev and Ops access to the live env. In terms of ISO 27001 this is a problem with the ISMS requirements (including A 6.1.2 - Segregation of Duties, A 14.2.2 - System change control procedures).

You know the drill: all DevOps need all access from everywhere to everything at any given time. And Jump hosts are so 90ies. OpenSSH’s ssh -t or ProxyCommands are too sophisticated. DevOps have no time for this. Because no one has time for security technologies, regardless of how simple or useful they are. Whether it’s JumpHosts or SysDig Secure.
Unless… we have SecDevOps, we won’t have time for this.

Sysdig, ISO27k and DevOps SDLC

Ok, let’s stop feeding the DevOps (meme), and get into the juicy parts of being SecDevOps.


  • SysDig (Secure) can be used for monitoring controls to log if a DevOp attaches directly to the Containers in the live / production env
  • SysDig can log for file modifications, which can be issued directly. Usually Devs would use their deployment pipelines. But that’s usually not enforced in DevOps shops.
  • SysDig has forensic triage features for Incident Response. IR readiness is a GDPR, ISO and PCI requirement. No one is going to perform a forensic live response on a Kube-Cluster without an IR tool. That means quantifying data-breaches in Micro-Service environments is hard without proper traceability controls.
    • SysDig does have remediation features to kill or pause infected containers. This works via the Docker daemon. This way you can acquire evidence.
  • you won’t get an NIDS within the Kube-cluster, that delivers results. Try, but prepare to fail.

In simpler terms: Kubernetes is a meta-layer and you need meta- / App-level information to assess potential security incidents on that layer. Like the DNS resolver history, process-based traces, and file-access timelines. SysDig aims to deliver that:

A PCAP will get you nothing. Just some IPs, which pop in, and some data that goes from somewhere to anywhere. Like in an any-any ACL network on top of a /24, where DHCP leases expire every hour.

kube-dns monitoring is much more promising:

Because Malware / post-exploitation makes DNS noise, unless your attackers are skilled. It’s about Application transparency here. Because a Container essentially is an App.


You should evaluate ways to integrate security controls into your Micro-Service architecture on an App level. There aren’t too many ways, which is a symptom of a bigger problem: security gets consolidated away, if it doesn’t keep up. Businesses need speed more than they need InfoSec.

SysDig is one possible way to approach security in a Micro-Service environment (and it’s not limited to k8s), but it’s quite complex to deal with tools like that: you need to know a lot. Dev, Ops, Sec and App. Otherwise you cannot use it. SecDevAppOps!

Due to this you need to do your background research. And this blog post does exactly that: it helps you to dig into SysDig. Even if you don’t realise that.


09.03.2018 - published as blog
13.03.2018 - added some clarifications on the overlay nets and the DevOps vs. SecDevOps irony
15.03.2018 - simplified the SDN and Kubeernetes network overlay part a little.