Let’s Encrypt + Salt: cert+key distribution

Let’s Encrypt is a valuable service that has enabled individuals and businesses of all sizes to secure all of their websites and other services with SSL certificates entirely free of charge. Salt is a powerful configuration management tool for setting up and maintaining the state of your servers. Here I look at how a server team can use Salt to distribute their certbot-generated SSL certificates and keys securely to the appropriate servers on their network.

Starting point

We have a single virtual machine that is used for requesting and renewing SSL certificates through certbot (using a DNS plugin). Given that Let’s Encrypt certificates renew quite often, what we need is an automated way of distributing the certificates and keys to the relevant servers/VMs hosting our different websites and services. We already have a Salt configuration management setup managing our servers, so it only makes sense to use that.

Salt has a mechanism for storing encrypted secrets using GnuPG encryption, which is an ideal way of storing and transferring the certificate files (and more importantly the key files) in a secure manner.

Given all this, the only trick is how to integrate these components in a reliable and easy-to-use manner.

Pre-requisites

What you will need:

A salt-master, set up with:
- A salt-pillar in a git repository.
- GPG encryption configured.
A machine that is used for requesting/auto-renewing certs via certbot, with:
- A clone of the salt-pillar git repository and the ability for the root user to push directly to origin/master.
- The public GPG key of the salt-pillar in the root user’s GPG keyring.

Encrypting and storing certs + keys

The heart of this operation will be a script on the server on which you have certbot running. This script will handle detecting which certs need adding to the repository, encrypting them, writing them in a YAML-friendly format, and finally adding, committing and pushing them to the salt-pillar git repository. The script will need to run as root in order to have access to the certificate and key files provided by certbot.

This is the script that we use, though you may find there is plenty of room for improvement (I wrote this script and I am very much a novice when it comes to bash scripting):

#!/bin/bash
set -euo pipefail
REPOBASE="/path/to/salt-pillar/"
REPOPATH="${REPOBASE}le-certs/"
GPGKEY="name of GPG key"
/usr/bin/git -C "$REPOBASE" fetch -q
/usr/bin/git -C "$REPOBASE" reset -q --hard origin/master
LOG=""
for i in /etc/letsencrypt/live/*; do
  if [[ -d "$i" ]]; then
    DOMAIN=`basename $i`
    SAFEDOMAIN="${DOMAIN//./_}"
    mkdir -p "$REPOPATH$SAFEDOMAIN"
    FILENAME="$REPOPATH$SAFEDOMAIN/init.sls"
    if [[ ! -e "$FILENAME" ]] || [[ "$i/fullchain.pem" -nt "$FILENAME" ]]; then
      echo "#!yamlex|gpg" > "$FILENAME"
      echo "le-certs: !aggregate" >> "$FILENAME"
      echo "  $DOMAIN:" >> "$FILENAME"
      echo "    chain: |" >> "$FILENAME"
      cat "$i/chain.pem" | gpg --armor --batch --trust-model always \
        --encrypt -r "$GPGKEY" \
        | sed 's/\(.*\)/      \1/' >> "$FILENAME"
      echo "    fullchain: |" >> "$FILENAME"
      cat "$i/fullchain.pem" | gpg --armor --batch --trust-model always \
        --encrypt -r "$GPGKEY" \
        | sed 's/\(.*\)/      \1/' >> "$FILENAME"
      echo "    privkey: |" >> "$FILENAME"
      cat "$i/privkey.pem" | gpg --armor --batch --trust-model always \
        --encrypt -r "$GPGKEY" \
        | sed 's/\(.*\)/      \1/' >> "$FILENAME"
      echo "    combined: |" >> "$FILENAME"
      cat "$i/privkey.pem" "$i/fullchain.pem" | gpg --armor --batch \
        --trust-model always --encrypt -r "$GPGKEY" \
        | sed 's/\(.*\)/      \1/' >> "$FILENAME"
      LOG+=$(printf "\n  * $DOMAIN")
    fi
  fi
done
/usr/bin/git -C "$REPOBASE" add -A
/usr/bin/git -C "$REPOBASE" commit -q \
-m "Automatic letsencrypt certificate update for domains:$LOG" \
>/dev/null || true
/usr/bin/git -C "$REPOBASE" push -q

Breakdown

That’s quite a lot, so let’s go through some of the pieces.

First we set up some variables that will need tweaking based on your setup:

REPOBASE="/path/to/salt-pillar/"
REPOPATH="${REPOBASE}le-certs/"
GPGKEY="name of GPG key"

REPOBASE points to the location of the (non-bare) clone of the salt-pillar git repository on this server.
REPOPATH points to where within the git repository you want the SSL certificates to be stored. In our case we chose it to be in a directory called le-certs/.
GPGKEY is the name/identifier of the GPG key used to store secrets in the salt-pillar.

Next we ensure that the local git repository is in a clean and current state:

/usr/bin/git -C "$REPOBASE" fetch -q
/usr/bin/git -C "$REPOBASE" reset -q --hard origin/master

Rather than doing a git pull, we’re doing a fetch followed by a hard reset to make sure there are no local uncommitted or unpushed changes (this clone of the repository should never be manually edited).

Next we come to the main loop that goes through all the cert directories found in /etc/letsencrypt/live/:

for i in /etc/letsencrypt/live/*; do
  if [[ -d "$i" ]]; then
    ...
  fi
done

It simply iterates everything in the /etc/letsencrypt/live directory and ignores anything that is not a directory (primarily so we don’t try to do anything with the README file).

    DOMAIN=`basename $i`
    SAFEDOMAIN="${DOMAIN//./_}"
    mkdir -p "$REPOPATH$SAFEDOMAIN"
    FILENAME="$REPOPATH$SAFEDOMAIN/init.sls"

Here we work out the domain name of the certificate based on its filename, and then because . has a special significance within Salt state filenames, we are replacing it with an underscore. We then use mkdir -p to create the subdirectory for this certificate within the git repo if it doesn’t yet exist. Finally we build the name of the file in which we store everything related to this certificate.

Next a check to see if we need to do anything with this certificate today:

    if [[ ! -e "$FILENAME" ]] || [[ "$i/fullchain.pem" -nt "$FILENAME" ]]; then
      ...
    fi

Here we are checking if the file already exists in the repository, and if so, if the current certificate is newer than the one stored in git. If the file already exists and is already the current version, we can simply ignore it and move on, otherwise we proceed.

Next we proceed to actually generating the YAML content that Salt will understand, starting with:

      echo "#!yamlex|gpg" > "$FILENAME"
      echo "le-certs: !aggregate" >> "$FILENAME"
      echo "  $DOMAIN:" >> "$FILENAME"

The first line indicates that we are using a couple of extensions to the standard Salt YAML format. The yamlex allows us to use the salt.renderers.yamlex !aggregate feature, which is important if multiple certificates need to be sent to the same server. The gpg is of course what allows the data to be GPG-encrypted.
The second line sets up a pillar item called le-certs, using !aggregate to allow Salt to merge it with other le-certs items declared elsewhere.
The third line sets up a pillar item under le-certs, with the name of the certificate’s domain.

Within this le-certs/<DOMAIN> pillar item, we then set up 4 items with the actual data in. We’ll look at just one of them:

      echo "    chain: |" >> "$FILENAME"
      cat "$i/chain.pem" | gpg --armor --batch --trust-model always \
        --encrypt -r "$GPGKEY" \
        | sed 's/\(.*\)/      \1/' >> "$FILENAME"

This takes the chain.pem file within the letsencrypt certificate directory, GPG-encrypts it and appends it to the YAML file, with the appropriate variable name and required indentation.

Note that we are storing pillar items for the chain.pem, fullchain.pem and privkey.pem files. In addition we are also creating a new “combined” item which contains both privkey.pem and fullchain.pem in a single file. This is done specifically to make it easier to work with certificate files for haproxy, which requires the files to be in this format.

The resulting files made will look something like this:

#!yamlex|gpg
le-certs: !aggregate
  example.com:
    chain: |
      -----BEGIN PGP MESSAGE-----
      ...
      -----END PGP MESSAGE----
    fullchain: |
      -----BEGIN PGP MESSAGE-----
      ...
      -----END PGP MESSAGE----
    privkey: |
      -----BEGIN PGP MESSAGE-----
      ...
      -----END PGP MESSAGE----
    combined: |
      -----BEGIN PGP MESSAGE-----
      ...
      -----END PGP MESSAGE----

Finally we commit our changes (with a sensible commit message) and push them to the origin repository:

/usr/bin/git -C "$REPOBASE" add -A
/usr/bin/git -C "$REPOBASE" commit -q \
-m "Automatic letsencrypt certificate update for domains:$LOG" \
>/dev/null || true
/usr/bin/git -C "$REPOBASE" push -q

Triggering script runs

You could simply have this script run in a regular cron job, but perhaps a better option is to use certbot’s renewal hooks by placing this script in the /etc/letsencrypt/renewal-hooks/post/ directory (remember that the script will need executable permissions, ie. chmod +x update-salt.sh).

Certificate distribution

It’s all very well having the certificates and keys in salt-pillar, but we’re far from done. We still need to assign the certificates to the right servers, and we need to have the cert files installed on the servers.

Assigning the certificates to the right servers is very simple. A single line needs adding to the top.sls file in the salt-pillar repository for each certificate assignment. Here’s an example:

base:
  'mailserver':
    - le-certs.mail_example_com
  'webserver[1-2]':
    - match: pcre
    - le-certs.example_com
    - le-certs.example_org

Here we are saying that the mail.example.com certificate & key should be sent to the server called mailserver, and that the example.com and example.org certificates & keys should be sent to the servers called webserver1 and webserver2.

At this point you should be able to test that the pillar items are being distributed correctly. On the Salt master, you can run eg. sudo salt 'webserver1' pillar.items to see the (decrypted) data that the minion will receive. Likewise on the recipient server, you can run sudo salt-call pillar.items to get the exact same information. You would see something like this:

local:
     ----------
     le-certs:
         ----------
         example.com:
             ----------
             chain:
                 -----BEGIN CERTIFICATE-----
                 ...
                 -----END CERTIFICATE-----
...

Now that we have the pillar data accessible where we need it, we need to write a Salt state to handle it. This is actually quite simple:

{% for domain, cert in pillar.get('le-certs', {}).items() %}
/etc/ssl/private/{{ domain }}-chain.pem:
  file.managed:
    - contents_pillar: le-certs:{{ domain }}:chain
    - mode: '0644'

/etc/ssl/private/{{ domain }}-fullchain.pem:
  file.managed:
    - contents_pillar: le-certs:{{ domain }}:fullchain
    - mode: '0644'

/etc/ssl/private/{{ domain }}-privkey.pem:
  file.managed:
    - contents_pillar: le-certs:{{ domain }}:privkey
    - mode: '0600'

/etc/ssl/private/{{ domain }}-combined.pem:
  file.managed:
    - contents_pillar: le-certs:{{ domain }}:combined
    - mode: '0600'
{% endfor %}

This iterates through all le-certs pillar items available to the server and writes them to files in /etc/ssl/private/

Then all we need is the following in Salt’s top.sls file:

base:
  '*':
    - le-certs

In order to have this applied to every server that we manage.

Final steps: triggering service reloads

When a new certificate is distributed to a server (particularly when this is a renewal) you need to make sure that the services that use the certificate are reloaded. If you are already managing such services through Salt, it’s simply a case of adding the cert file to the watch list on the service. If not, then a simple minimal configuration example would be:

nginx:
  service.running:
    - watch:
      - file: /etc/ssl/private/example.com-fullchain.pem

in a Salt state file used by the server (replacing nginx with whatever service should be restarted: haproxy, dovecot, postfix, …).

Congratulations, you’re done! You’ll want to make sure that your Salt state files are being applied on a regular basis to all of your servers so that renewals are handled in a timely manner.

Main photo by Markus Spiske