Using Docker and LetsEncrypt to enable multiple TLDs on a single IP

Setting up TLS1 (commonly referred to as SSL) has always been a confusing journey through convention, implementation, and clients (browsers, http infrastructures) alignment.

Do you buy a certificate from a Certificate Authority, or run with your own self signed certificates? Self signed certificates aren't really an option if you're planning on having external users of your websites/api's.
Even some name registrars are giving out free or extremely inexpensive certificates to go along with your new domain registration.

Unfortunately most of these certificates are only good for a single FQDN2

Skip to the setup or continue reading for some more detail and backstory if you're into that sorta thing ;)

SSL/TLS why multiple IPs are usually needed

The primary reason in most cases that multiple IP's are need for hosting multiple secure websites is due to how SSL/TLS security handshaking happens.

SSL/TLS exchange can't support multiple certificates per IP:PORT due to protocol definition and implementations. This ain't changing, don't fight it.

The Key Exchange happens at the IP level. A website being secure and trusted is actually a client side validation process implementation. The industry has provided a standard for how this all works, and everyone operates inside these constraints. Now you know how bugs happen :)

Example of accessing a TLS enabled website

Until the HTTP GET the client isn't using the HTTP protocol yet, which is where this gets confusing for some. With the ubiquitous usage of host header mappings on port 80, this has proved to be a sticking point for people trying enable secure communications to their servers when they are hosting multiple FQDNs.

Because of the nature of this exchange, it becomes clear very quickly that you will need a certificate that has all of your dns names that will resolve to your IP and port 443.

Types of Certificates

DV Certificates

Domain validated or DV certificates are the most common type of SSL certificate. They are verified using only the domain name. Typically, the CA exchanges confirmation email with an address listed in the domain’s WHOIS record. Alternatively, the CA provides a verification file which the owner places on the website to be protected. Either method confirms that the domain is controlled by the party requesting the certificate.

OV Certificates

Organization validated or OV certificates require more validation than DV certificates, but provide more trust. For this type, the CA will verify the actual business that is attempting to get the certificate (the information required for OV certificates). The organization’s name is also listed in the certificate, giving added trust that both the website and the company are reputable. OVs are usually used by corporations, governments and other entities that want to provide an extra layer of confidence to their visitors.

EV Certificates

Extended validation or EV certificates provide the maximum amount of trust to visitors, and also require the most effort by the CA to validate. Per guidelines set by the CA/Browser Forum, extra documentation must be provided to issue an EV certificate.As in the OV, the EV lists the company name in the certificate itself, However, a fully validated EV certificate will also show the name of the company or organization in the address bar itself, and the address bar is displayed in green. This is an immediate, visual way that viewers can know that extra steps were taken to confirm the site they’re visiting – which is why most large companies and organizations choose EV certificates.

Which cert type to use?

Since LetsEncrypt only issues DV certificates this makes the decision pretty easy. As such, if you are in a position where you need to give your users more peace of mind, then you should consider one of these deeper validation levels. But it will cost you. Usually these certificates come with insurance if it's compromised at the CA level (if you give up your private key that's on you)

How much does it cost?

Here are some example price breakdowns as of 10/09/2016 from SSL2Buy (a site I've used in the past to get ssl certs)

SAN

Wildcard SAN

What's the difference between these?

SAN

Explicit FQDN's are in the certificate, any requests for DNS names that resolve to the same IP address will result in failed/untrusted exchanges

Limited to www.sitename.tld, sitename.tld, othersite.tld

Wildcard SAN
  • Single domain wildcarding, Example: *.website1.com
  • Nested Subdomain support not included Example: *.subsite.website1.com not covered

I have 6 domains currently, plus all the various CNAMES I may be messing with such as mail, www, ftp, smtp, pop3 etc... etc... Based on the pricing above, it looks like the Wildcard SAN is the certificate that covers the most ground.

So 2 domains are included @ $144, that leaves me 4 additional ones @ $60 each, we're talking $388/yr to mess around. Ummmmm really?!


Enter LetsEncrypt, It's free!

(like a puppy)

From https://letsencrypt.org/about

Let’s Encrypt is a free, automated, and open certificate authority (CA), run for the public’s benefit. It is a service provided by the Internet Security Research Group (ISRG).

They support single domain and SAN certificates and have their own command line tooling to make a lot of this process easier.

Things to know before switching over

LetsEncrypt FAQ

  • Certificates obtained expire in 90 days
  • Domain Validation (DV) certificates only, so you can use them for any server that uses a domain name, like web servers, mail servers, FTP servers, and many more.
  • Certificate Compatibility?
  • Plan to automate the renewal
  • Cert issuance is in two phases
    • Requesting the cert
    • Modifying DNS or adding a file to a website share to prove ownership
  • Financial reimbursement is minimal with free certs as it should be. If you need more insurance on your certificate, then pony up the $$$. Although I haven't seen or read about anybody ever needing to make a claim against this.

If all these things are ok with you, continue!


Server Setup

My Digital Ocean droplet is running CoreOS. If you have Docker installed on any other distribution these instructions will still be fine for you. My setup doesn't require any specifics about CoreOS, it's just a purpose built image that has very few dependencies and is focused on running Docker.
a

nginx:alpine @ DockerHub

Provides reverse proxy, and subsequently SSL/TLS termination support for all other containers.

ghost:x.x @ DockerHub

Ghost is a blogging platform based on NodeJS. It fairly light weight and the editing format is Markdown. Highly recommended if you just want to type and spend a lot less time futzing with formatting, and website building and deployment.

Docker Networking

The nginx container gets attached to the Dockerhost networking layer --net=host.
CAUTION: Be aware that this doesn't always produce the results that you may think. Be sure to read up on docker in regards to IPTables being brought in when this kind of networking is done. http://www.dasblinkenlichten.com/docker-networking-101-host-mode/

We want all traffic to go through NGINX so that it can serve up our certificate that we will be using for all the things ;)

Setup NGINX to do SSL Offloading

For my setup I will have NGINX perform SSL Offloading for ALL requests. This is because I will be using one SAN certificate for all of my hosted domains.
Additionally in my case anyone that comes into the server with HTTP will be automatically sent to HTTPS.
NGINX has about a trillion ways to skin this cat, this is what I've come up with

FILE: nginx.conf

http {  
    ssl_dhparam /etc/ssl/certs/dhparam.pem;
    ssl_certificate     /etc/ssl/certs/fullchain.pem;
    ssl_certificate_key /etc/ssl/private/privkey.pem;
    #Without this, you will see OSP errors in nginx
    ssl_trusted_certificate /etc/ssl/certs/chain.pem;
#
# All your other things
#

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

FILE: reverseproxy.for.allsites

#One upstream per docker container
#The Ghost containers listen on port 80 internally 
#but each container is given a unique Port externally for 
#the purposes of routing
upstream codedad {  
    server 0.0.0.0:6001;
    keepalive 8;
}

upstream alicornscribbles {  
    server 0.0.0.0:6002;
    keepalive 8;
}

upstream batcavecattery {  
    server 0.0.0.0:6003;
    keepalive 8;
}

server {  
    listen 80;
    listen [::]:80;

    server_name  .codedad.net .batcavecattery.com .alicornscribbles.com;

    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {  
    listen 443 ssl;

    server_name   ~(?<domain>(\w*|\d*))\.\w*$;
    ssl_session_timeout 5m;

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
        ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;
        add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        ssl_stapling on; # Requires nginx >= 1.3.7
        ssl_stapling_verify on; # Requires nginx => 1.3.7
        resolver 8.8.8.8 8.8.4.4 valid=300s;
        resolver_timeout 5s;

    location /.well-known/acme-challenge/ {
        root /var/www/letsencrypt;
    }

    location / {
        proxy_pass_request_headers on;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://$domain/$request_uri;
        proxy_redirect off;
    }
}

Integrating LetsEncrypt Flow

Distilled down here is the flow

  1. Certbot packages up your request from the command line parameters passed, or if none are passed it will prompt you for input
  2. Certbot then writes a file to the -w aka webroot location specified
  3. Request is sent to LetsEncrypt to validate
  4. LetsEncrypt attempt to call http://{registered san name}/.well-known/acme-challenge/{somefilehash}
  5. If it gets the file signature back that it was expecting, then your certificate will be signed.
  6. Files are written to

    • /etc/letsencrypt/live/codedad.net
    • /etc/letsencrypt/live/codedad.net/cert.pem
    • /etc/letsencrypt/live/codedad.net/chain.pem
    • /etc/letsencrypt/live/codedad.net/fullchain.pem
    • /etc/letsencrypt/live/codedad.net/privkey.pem
Certbot Docker tooling

There is a docker container already that contains the certbot tooling quay.io/letsencrypt/letsencrypt:latest

I created a shell script to run certbot in a container with it's outputs/inputs written to the local docker host disk. Probably should use a data container here but that's another post

Certbot also supports a config file if you don't want to specify everything on the command line. This makes it easier to tweak your setup (or create a process to rebuild it) without getting into the middle of all your volume mapping

https://certbot.eff.org/docs/using.html#configuration-file

Certbot uses cli.ini from /etc/letsencrypt by default so create it and put it in a folder

Remember that we are mapping folders into the container using our current working directory.

touch `pwd`/etc/letsencrypt/cli.ini  
vim `pwd`/etc/letsencrypt/cli.ini  

Contents of cli.ini

# All flags used by the client can be configured here. Run Certbot with
# "--help" to learn more about the available options.
rsa-key-size = 4096

email = me@dontbotheremailingme.wackadoos

domains = www.codedad.net, codedad.net, www.batcavecattery.com, batcavecattery.com 

# use a text interface instead of ncurses
text = True

authenticator = webroot  
webroot-path = /var/www/letsencrypt

certbot.sh

#!/bin/sh
mkdir -p `pwd`/var/lib/letsencrypt  
mkdir -p `pwd`/etc/letsencrypt  
mkdir -p `pwd`/var/www/letsencrypt

docker run -it --rm --name certbot \  
            -v "`pwd`/etc/letsencrypt:/etc/letsencrypt" \
        -v "`pwd`/var/www/letsencrypt:/var/www/letsencrypt" \
            -v "`pwd`/var/lib/letsencrypt:/var/lib/letsencrypt" \
            quay.io/letsencrypt/letsencrypt:latest "$@"

This script will create a var/lib/letsencrypt and a etc/letsencrypt folders in your current working directory if they don't exist (if they do it will be silent)

Then runs the letsencrypt container and the certbot command with the commandline options you passed to the script.
Once the script executes the container instance that was running the tooling is removed automatically so you don't have to worry about docker containers hanging about

Certbot usage

To create a SAN cert we will want to put all our eggs in one basket as it were.

./certbot.sh certonly --webroot -w /var/www/letsencrypt -d  www.codedad.net -d codedad.net -d batcavecattery.com -d www.batcavecattery.com -d alicornscribbles.net

-w is the webroot where LetsEncrypt will write the validation files it requests from your webserver. This will need to be the same root location where you will setup NGINX to serve up static files from {host.tld}/.well-known/acme-challenge/

Where are my files? After the exchange is successful there will be series of folders written to the containers /etc/letsencrypt/ which we've mapped to ~/etc/letsencrypt. This is how your save the outputs and inputs between container runs.

My NGINX Container startup command. This handles mapping all of items NGINX wants via volume mappings from my host.

docker run -d \  
  --name nginx \
  -v `pwd`/etc/ssl:/etc/ssl \
  -v `pwd`/nginx.conf:/etc/nginx/nginx.conf \
  -v `pwd`/sites-available/reverseproxy.for.allsites:/etc/nginx/sites-enabled/reverseproxy.for.allsites \
  -v `pwd`/etc/letsencrypt:/etc/letsencrypt \
  -v `pwd`/var/www/letsencrypt:/var/www/letsencrypt \
  --net=host \
  nginx:alpine

What does all this mean?
It will start the nginx container with volume mappings to my specific site configurations. Typically NGINX convention is to have a sites-availble folder and a sites-enabled folder and you create a symbolic link from available -> enabled. The volume mappings shortcut the symbolic linking process as the files don't live inside the container.
I'm using other Docker containers to host NODEJS applications.

After the container starts we can see how the directory structure is mapped

docker exec nginx ls /etc/nginx/sites-enabled  
drwxr-xr-x    2 root     root          4096 Oct 10 21:45 .  
drwxr-xr-x    1 root     root          4096 Oct 10 21:45 ..  
-rw-r--r--    0 500      500           2234 Oct 10 21:45 reverseproxy.for.allsites

Setup your NGINX config for your website to serve up static files ${webroot-path}/.well-known/acme-challenge, this will allow for automated Certificate authorization, as part of the process is to put a specific file in the website root so that LetsEncrypt can validate you are who you say you are

In my specific case everything is on the same IP address. Even though I have multiple websites, I can't reasonable get multiple certificates because only one certificate can be exposed per IP:PORT combination.

Example of URL in browser:
https://www.codedad.net
https://www.batcavecattery.com

Lets say both dns names resolve to 192.168.1.53
What happens?

  1. TLS connects to port 443 for 192.168.1.53 that resolved via DNS
  2. NGINX receives the connection on 443 and returns the certificate configured.
    Caveat If you have defined your certificate to be used in your site specific configuration file, the first site that is listening on port 443 will have it's certificate served first. This is why we want one cert with all of our SAN names

Here is an SSL exchange using a valid Common Name
curl -v https://www.codedad.net

* Rebuilt URL to: https://www.codedad.net/
*   Trying 192.168.1.53...
* Connected to www.codedad.net (192.168.1.53) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; CN=*.codedad.net
*  start date: Mar  4 16:31:44 2016 GMT
*  expire date: Apr 12 00:29:02 2017 GMT
*  subjectAltName: host "www.codedad.net" matched cert's "*.codedad.net"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=AlphaSSL CA - SHA256 - G2
*  SSL certificate verify ok.

Up till here the host header hasn't even gotten involved with the request. It just connected to the IP, understood we wanted it to be secure because of HTTPS. Only after it gets the cert does it validate the chain. Then passed the connection along if the chain is ok, and then performs the actual GET below

> GET / HTTP/1.1
> Host: www.codedad.net
> User-Agent: curl/7.50.1
> Accept: */*

Here is an SSL exchange to the same server, but different common name. My current cert doesn't have this domain name in it's SAN settings

What you would see in Chrome:

Verbose Curl output:
curl -v https://www.batcavecattery.com

* Rebuilt URL to: https://www.batcavecattery.com/
*   Trying 192.168.1.53...
* Connected to www.batcavecattery.com (192.168.1.53) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: OU=Domain Control Validated; CN=*.codedad.net
*  start date: Mar  4 16:31:44 2016 GMT
*  expire date: Apr 12 00:29:02 2017 GMT
*  subjectAltName does not match www.batcavecattery.com
* SSL: no alternative certificate subject name matches target host name 'www.batcavecattery.com'
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name 'www.batcavecattery.com'  

The best way to solve this in my current setup is to create a new website configuration that is only responsible for one thing, that's to serve up a static route at ${whateverhost}/.well-known/acme-challenge that maps to a location that certbot will write the acme-challenge file.

Wrap-Up

If you're looking for a vm infrastructure to play around with these sorts of things, I would highly recommend Digital Ocean (This is a referral link that will get you $10 credit if you sign up). They are uncomplicated and going from nothing to having a machine online is merely a matter of 30 seconds. Support floating IP's, Block storage, backend networking, and Multi-Region support.


  1. Wikipedia Definition of TLS/SSL

  2. FQDN: Fully Qualified Domain Name