Small security improvements - layer by layer since the 80ies
Is information security a luxury? Like time? InfoSec professionals say security has become a commodity. Must be an expensive one, if so few organizations are making the effort… 
In the following I give technical examples of the possibilities to improve the security without direct product costs. – No, security isn’t a luxury. Let’s figure out what it is by example.
Small security improvements - practical things layer by layer
- added pf Alias based block rule to force traffic over the CDN
- enabled livepatch - Canonical Livepatch Service
- enabled VMware ESXi Side-Channel-Aware Scheduler v1
- added HSTS, kept TLS to >= 1.1
- added DNSSEC
- SPF and DKIM - I am a DMARC postmaster now
- Summary - layers of fail?
☑ added pf Alias based block rule to force traffic over the CDN
Hosting “web systems” isn’t fun. Patching is not fun… Without further ado I’d like to jump into a security issue with CDN networks; like Cloudflare. – The CDN which markets itself as a “free” solution to secure personal websites.
I decided to enforce the usage of Cloudflare (non-paid subscription) via pfSense (community version). The external interface of a server at
144.76.Y.X should only be reachable via “Cloudflare”. Otherwise it’s too easy to bypass to have a real security effect.
– How do we put that into ACLs without hardcoding the CDN IP ranges? These ranges can change…
Check this out:
% pfctl -sr | grep cflare pass in quick on em0 reply-to (em0 144.76.Y.X) inet proto tcp from <cflarev4> to 192.168.1.XXX port = http flags S/SA keep state label "USER_RULE: NAT http"
With pfSense you can define an URL alias (
<cflarev4>) and it will use
curl to pull in the CIDRs from a list in order to automatically define the IP ranges. It will update this hourly, depending on your configuration.
You can enforce the CDN both ways (ingress and egress) and apply that to port
80 (http) and port
443 (https) only. pfSense is a stateful firewall gateway here.
Results: while this isn’t state of the art and just based on free-tier and community services (Cloudflare and Netgate), it still adds an improvement. Bypassing any CDN or WAF is more difficult with IP range enforcement rules: more noisy, less direct.
You cannot just send HTTP requests to the external interface at
144.76.Y.X, because pfSense will be in your way. Even if you override the local DNS resolver with a hosts file entry, you will not be able to get through.
I adopted Cloudflare Access as my personal-use Zero Trust network to bridge my security development, logging and monitoring systems into the internet; via an identity-aware approach. Might blog about that soon[tm].
The overall point is, that one needs to mind network ACLs (on the pfSense level e.g.) to ensure that the higher layer security controls (CDN, WAF, TLS, security headers to make use of browser-security improvements) apply in a gapless fashion.
It’s very easy to make mistakes here nowadays. Security isn’t simple, it’s layers over layers. And an increasing amount of these layers isn’t going to be on-premise. With this setup I am already in a hybrid scenario. With all the advantages and disadvantages.
☑ enabled livepatch - Canonical Livepatch Service
livepatch as a service for supported systems under their Long Term Support (LTS) release line.
livepatch's architecture is documented at the Linux kernel project..
Today Ksplice and the patches would be freely available for Desktops (and not for headless servers). Canonical offers a free-tier subscription for up to 3 personal systems, regardless of the purpose.
Experiences with KLP and Canonical livepatch
livepatch on a Docker host. For example this shiny Discourse website runs within a Docker container. This way I get
chroot-like compartmentalization with the upstream DevOps fluff. Fluff like a Git driven deployment workflow and revertability to prior images. Security fluff incorporated.
Here’s a quick output of
livepatch in action:
[email protected]:~# canonical-livepatch status client-version: 9.3.0 architecture: x86_64 cpu-model: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz last-check: 2019-05-18T20:54:29+02:00 boot-time: 2019-05-02T18:39:33+02:00 uptime: 386h16m32s status: - kernel: 4.4.0-98.121-generic running: true livepatch: checkState: checked patchState: kernel-upgrade-required version: "34.1" fixes: |- * CVE-2015-7837 LP: #1509563 ... * CVE-2016-8655 LP: #1646318 * CVE-2017-9242
The S in LTS does not stand for Secure. There are open CVEs. Even for this recent kernel, speaking in relative terms.
livepatch to a Docker host system makes sense, because dockerized processes share the kernel. Less CVEs mean less attack surface.
Again the point is, that one needs to mind the lower layers (kernel) to get higher-level security functions (Docker compartmentalization, deployments from upstream Git to get patches).
But how low do we need to go?
☑ enabled VMware ESXi Side-Channel-Aware Scheduler v1
It seems the CPU side-channel mitigations are being developed by CPU vendors: slower by the day. At least that’s how this game is played currently.
Security is that unwanted stepchild that is now supposed to drive CPU sales because it continues to acclerate asset deprecation and lowers the lease-term length for lessees.
Costs costs costs - performance?
I moved my “road-warrior” ESXi to the Side-Channel-Aware scheduler version 1 (SCAv1; on a 6.7 release line now).
– Mostly because I do have the resources to do that here. In the context of actual post-exploitation the respective side-channel attacks matter for shared-hosting setups (“in the cloud”). The point being, that these are setups, where an arbitrary co-lessee may allow a side-channel attack to take place against the shared CPU.
In summary: what this VMware KB article says is that Hyperthreading gets deactivated or restricted, and that you lose up to 50% of virtual CPU-core resources depending on the concurrency and parallelization patterns of your load.
Trade-off: security versus performance
I don’t see the need for this specific performance / security trade-off for dedicated servers and for on-premise hosting strategies, where there’s no need to factor in a co-lessee into the threat-model.
Post-exploitation scenarios exist for dedicated setups as well, which qualifies this as a medium-high risk (magnitude and credibility). But during my tests the side-channels had a low bandwidth and the behavior appears to be detectable. Maybe we can trade one mitigation for another?!
If you have the resources, you should go for SCAv1. Not everyone has. Not everyone has got sensitive information or critical processes within VMware setups. Patching the web-apps and virtual-machines (kernel and userland) is much more important in most scenarios. The exploitation chain requires a vulnerable virtual machine.
The most important “layer” is not the CPU / barebones hardware but the the security architecture. It’s quite possible to avoid becoming a lessee, and to stay an owner. Own your boxes, and you won’t get 0wned
☑ added HSTS, kept TLS to >= 1.1
Moving into application layers again: HSTS stands for “HTTP Strict Transport Security” and is a rule-mechanism where the browser (client) and server (web server sending headers) agree that encryption going to be used from now on.
There is a
max-age field in the headers, which during the exchange specifies the period during which this agreement will remain to be valid. By design browsers cache this agreement and enforce it.
➜ ~ curl -s -D- https://because-security.com | grep Transport Strict-Transport-Security: max-age=2592000; includeSubDomains; preload
HSTS is specified in RFC 6797
2592000equals 30 days (converted from seconds). You have to set this to
0if you want to turn TLS off.
includeSubDomains“signals the UA that the HSTS Policy applies to this HSTS Host as well as any subdomains of the host’s domain name”
HSTS preloading is a little known strategy to pre-populate those (important) sites, which should enforce HSTS in advance, even for first-time visitors. For this in order to work one needs to specify a
max-age with at least 1 year. I did not do this, and therefore this setting is ineffective.
Quick SSL / TLS assessment
An A is fine. I don’t serve personal or sensitive data here anyway. It’s not horrible.
TLS >= 1.1 isn’t going to cause me major headaches. No cardholder-data here.
HSTS is a good thing to add if we understand the implications of that agreement between clients and servers. Especially for mobile networks it can help to establish encryption as a default, because browsers tend to cache redirects differently.
And in general it’s a more reliable way to do so than permanent HTTP (
301) redirects, because it can apply for the entire domain.
☑ added DNSSEC
Some people say that DNSSEC doesn’t add any security value. Some…
➜ ~ dig because-security.com +dnssec ; <<>> DiG 9.10.6 <<>> because-security.com +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2130 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
ad stands for “Authenticated Data” afaik, but I’d have to look this up.
DNSSEC isn’t the most prominent security standard there is for a variety of reasons that are beyond the scope of this little blog.
Results: it’s safe to say it doesn’t hurt /me here. Or anyone. But then again, I am not a browser developer who makes millions with web advertisements.
DNSSEC is nice to have, but not worth the effort. Let me just set a
SSHFP DNS record and leave this as a reference:
debug1: Server host key: ssh-ed25519 SHA256:... debug1: found 2 insecure fingerprints in DNS debug1: mismatching host key fingerprint found in DNS @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. The fingerprint for the ED25519 key sent by the remote host is ... Please contact your system administrator. Update the SSHFP RR in DNS with the new host key to get rid of this message. debug1: Host '...' is known and matches the ED25519 host key.
☑ SPF and DKIM - I am a DMARC postmaster now
So far we tangent a lot of the daily InfoSec pain-points.
- Firewalls ACLs and web-fronts (like CDNs and WAFs),
- SSL / TLS, browser-headers and server policies (HSTS),
- Container-runtimes (like Docker) and kernel patches (Linux KLP)
- Side-channel attacks (CPU hyper-threading issues) and hypervisor security (ESXi)
Time for eMail… Yes! Email isn’t a boring topic. Because in Email we trust. Or not… that’s why we put it into the clouds. Like CO2. And other poisons. What do environmental protection and information security have in common? Let’s find out… Out of sight, out of mind?
Even savvy mail server admins do not know half of these cryptic shortcuts. Due to this there are a lot of “problematic” (Microsoft Office 365 and Goog Suite) Email setups.
Few orgs host their own mail servers these days (2019), and since it’s not on their premises they don’t pay attention to it.
- unless it’s for marketing, and all the newsletters get classified as Spam.
- or if VIPs within the organization get phishing mails.
- or the domains get blacklisted…
SPF & DKIM = DMARC
Often “cloud mail” / business Email setups have neither
- Sender Policy Framework (RFC 7208 alias SPF) nor
- Domain Keys Identified Mail signatures (RFC 6376 alias DKIM).
Consequently they are without Domain-based Message Authentication Reporting, and Conformance (RFC 7489 alias DMARC).
In short: No SPF and DKIM - no DMARC. No nothing. Naked Emails. Not beautiful… given how old that standard is.
In less cryptic terms: most setups are without security and indirectly allow Spam and Phishing attacks to lead to Business Email Compromise (BEC) because they don’t mind security as part of their Email delivery setup. We live in 2019: you cannot just set
MX records and call it a day. Because security.
Fixing this means setting TXT records here
DKIM, SPF and DMARC - check
G Suite provides a lot of self-assessment tools, which you will find to be helpful during this journey through modern mail-architectures and RFC drafts.
dig for the SPF TXT record:
➜ ~ dig -t txt because-security.com | grep -A1 "ANSWER SECTION" ;; ANSWER SECTION: because-security.com. 300 IN TXT "v=spf1 include:_spf.google.com ~all"
dig for the DKIM TXT record:
➜ ~ dig TXT because-security._domainkey.because-security.com | grep -A 2 "ANSWER SECTION" ;; ANSWER SECTION: because-security._domainkey.because-security.com. 260 IN TXT "v=DKIM1; k=rsa; p=MII..."
dig for the DMARC TXT record:
➜ ~ dig TXT _dmarc.because-security.com | grep -A2 "ANSWER SECTION" ;; ANSWER SECTION: _dmarc.because-security.com. 274 IN TXT "v=DMARC1; p=quarantine; pct=5; rua=mailto:[email protected]"
Let’s check this from a mail client. We can do this by inspecting the mail headers. G Suite follows the ARC draft:
ARC-Authentication-Results: i=1; mx.google.com; dkim=pass [email protected] header.s=because-security ... spf=pass ... dmarc=pass
Good. Now let’s add a S/MIME certificate and even more RFCs become relevant…
Results: Email isn’t like it was in 1982 any more. During the last 5 to 10 years new additions have been added to improve security.
For now it’s opt-in, or live with the consequences. – Better opt-in now. Especially if you want to lead and influence this. One interesting approach is Brand Indicators for Message Identification (BIMI), which gives companies marketing incentives to setup DMARC in order to get their logo displayed more exclusively. That kind of approach worked well with EV-SSL certs to phase out MD5 in SSL certificates.
Summary - layers of fail?
Or is information security layered to fail? It’s amazing what we are dealing with today: from CPU architecture issues on bare silicon to issues with evaluating textual information from standards that date back until 1982[11:1]. Security is an all-in-one journey through the IT landscapes. With everything that is good and bad.
Security isn’t a luxury. It’s not layered to fail. And it’s not a commodity. It’s not a process or a state. It’s not a management problem or a business risk influence. It’s all of it. Small improvements, day by day. Since 1982 and earlier.
Process finished with exit code 0