Letsencrypt with strong SSL and Lighttpd

Tags: #<Tag:0x00007f76fba5f2e8> #<Tag:0x00007f76fba5f130> #<Tag:0x00007f76fba5ef50> #<Tag:0x00007f76fba5edc0> #<Tag:0x00007f76fba5ec30>

A web server is not an SSL terminator

Looking at the past months of SSL bugs, I think it’s crazy that we insist on handling SSL in the same web servers, which serve our applications.

The web server processes hold credentials and other important infos, and they usually also have links to the DBMS where all the customer data resides.

Here is a simple way to separate SSL offloading from the application. If we are honest about it, the complexity of crypto handling is an application itself.
And while we are at it, let’s also automate certificate renewal processes and make it free. At least until the Letsencrypt initiative is terminated, which will happen sooner of later.

Letsencrypt 1on1

Here is how to get a cert for foobar.de :

/etc/init.d/apache2 stop
/root/.local/share/letsencrypt/bin/letsencrypt certonly --standalone -d foobar.de -d www.foobar.de

Cool, done. You will get a cert for foobar.de with a validity period of roughly 2 months. Now let’s automate the renewal process:

Open up crontab-generator (yes, really because cron syntax sucks)

sudo crontab -e

Paste:

* * */5 * * /bin/sh /root/Source/letsencrypt/letsencrypt-auto renew > /var/log/letsencrypt/cronjob.log

Cert renewal is now automated. Easy.

Logrotation for Letsencrypt - because we can

Add a file to /ect/logrotate.d/

/var/log/letsencrypt/*.log
{
    rotate 3
    missingok
    nocompress
    create
}

Now, letsencrypt does not concatinate the cert’s public and private key. Here is a script for that:

#!/bin/bash

for dir in $(ls -d /etc/letsencrypt/live/*); do
    cat $dir/privkey.pem $dir/fullchain.pem > $dir/fullkeychain.pem
done

This is kind of bad - from an automation standpoint. Especially if there is an issue… So consider this script a bad practice. If someone has a better solution how to automatically handle certs with letsencrypt and tools, which expect combined files, let me know :wink:

* * */5 * * /bin/bash /opt/concat_cert_and_key.sh

Result

This will check every 5 days if you need to renew a cert, and it will do it for you. It will log it and you can inspect the logs (which I would forward to ELK, where I’d setup a SSL Dashboard in Kibana).

Lighty SSL 1on1

I have been thinking of using Pound. It’s an SSL offloader with some basic HAproxy like software LB functions.
I decided against it, because I know Lighttpd pretty well and Pound is also just using libssl / OpenSSL. Unless they start using a hardened SSL stack, which is developed with security in mind, the benefits are too few. It claims to be an SSL sanitizer, but I’d argue that to be blunt at this point.

What offloading (and LB’ing) with Lighttpd does:

  • better application control: as long as it’s a separate process, which is handling the SSL
  • the ciphers can be configured, and the order can be enforced
  • headers can be added (and there are LB features)
  • there is SNI support
  • you can log stuff

Prepare Apache

The backend in this case is just an Apache2, with a couple of domain vHosts. You don’t need mod_ssl in the Apache2, but mod_rpaf comes in handy to deal with the HTTP headers and to ensure the logs make sense.

a2dismod ssl
a2dismod gnutls
a2enmod rpaf

Run netstat -tulpn and make sure there is no listener on 443. Lighttpd needs TCP:443.

Isolation? Really?

The OS level isolation in this particular example case is a simple KVM guest, which is NATed with iptables and dnsmasq. That is as simple as it gets for Linux virtualization…

… and you don’t need Docker NAT hacks to get the traffic out. If we are honest for a second: if you restart a Docker container with Port Mapping you have an issue or two with iptables afterwards. :wink: And virtualization or containerization have never been security concerns anyways.

Lighttpd 1on2

You can easily get strong SSL encryption with Lighttpd as a Reverse Proxy / HTTP LB. Lighty exposes a lot of config parameters to do this.

Diffie Hellman parameters

DHE strength can be useful in some scenarios to make is harder for a potential attacker to abuse computational / mathematical benefits with Elliptic Curves on ECDHE. Another - maybe even more valid - concern is, that there is an increasing of Wireless hotspot men-in-the-middle attacks, where companies see 1000s of accounts being compromised due to issues with their SSL / TLS. For now it’s not DHE related, but OpenSSL has many critical security issues within the protocols.

Personally I just do it because “I can”, and I don’t think that it is more relevant than the hashing algorithm of the CA / private key. The cert hash should preferably not be MD5 or SHA1. It’s kind of hard to ask for a free cert from Letsencrypt though, and to be picky at the same time. However I don’t understand why CAs remain using MD5 especially, since almost any HSM supports better algorithms at practically no additional computation cost.

DH params for the prime numbers literally take 10 minutes or more to generate. For now it’s a single-core operation. It will max out 1 CPU core during the generation. Practically there is no need to do it offline, unless you host your front end web servers in single core micro instances. You can also generate them on different systems and just deploy them.

time openssl dhparam -out dhparam_crazy.pem 4096

real	26m17.710s
user	26m8.879s
sys	0m5.857s

It’s 4 x Intel Xeon E312xx (Sandy Bridge) 3,4 GhZ.

Lighty SSL ReverseProxy

Put this info conf-enabled:

$HTTP["host"]== "foobar.de"  {   

    ssl.engine = "enable"
    ssl.use-sslv2 = "disable"
    ssl.use-sslv3 = "disable"
    # ssl.verifyclient.exportcert = "enable"
    # ssl.verifyclient.username = "enable"

    # url.redirect = (".*" => "https://%0$0")

    proxy.server =  ("" =>
                                (
                                        ( "host" => "192.168.100.X", "port" => 8080, "fix-redirects" => 1 )
                                
                ))
    
    ssl.pemfile = "/etc/letsencrypt/live/foobar.de/fullkeychain.pem"
    ssl.ca-file = "/etc/letsencrypt/live/foobar.de/cert.pem"
    ssl.use-compression = "disable"
    # ssl.cipher-list = "AES256+EECDH:AES256+EDH:!aNULL:!eNULL"
    ssl.cipher-list = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"  

    ssl.dh-file = "/etc/letsencrypt/live/foobar.de/dhparam_crazy.pem"
    ssl.ec-curve = "secp384r1"

}

Done.

You can do this per domain / web service VM. It’s pretty straight forward at this point to copy the foobar.de config and to generate the foobar.com.