LetsEncrypt SSL certificates for vhosts within the mail stack – via Postfix (SMTP) and Dovecot (IMAP)

For a long time, I didn’t care about using self-signed SSL certificates for the mail stack because 1) they still secured the connection to the server, and 2) those certificates weren’t seen or utilised by anyone other than me. However, for my datacentre infrastructure which houses clients’ websites and email, using self-signed or generic certificates (even for the mail stack) wasn’t a very good solution as mail clients (e.g Thunderbird or Outlook) notify users of the problematic certs. Clients with dedicated mail servers could use valid certificates (freely from LetsEncrypt) without problem, but those on shared infrastructure posed a different issue—how can mail for different domains all sharing the same IPv4 address use individualised certificates for their SMTP and IMAP connections? This article will explain the method that I used to assign domain-specific certificates for the full mail stack using LetsEncrypt’s certbot for the certs themselves, the Postfix MTA (mail transfer agent [for SMTP]), and the Dovecot IMAP server. This article is tailored to Gentoo Linux, but should be easily applied to nearly any distribution. Two general caveats are:

• The locations of files may be different for your distribution. Consult your distribution’s documentation for the appropriate file locations
• I use OpenRC as my init system in Gentoo. If your distribution uses systemd, you will need to replace any reference to code snippets containing /etc/init.d/$SERVICE $ACTION with systemctl $ACTION $SERVICE
  • As an example, I mention restarting Dovecot with /etc/init.d/dovecot restart
  • A systemd user would instead issue systemctl restart dovecot

I will provide a significant amount of ancillary information pertaining to each step of this process. If you are comfortable with the mail stack, some of it may be rudimentary, so feel free to skip ahead. Conversely, if any of the concepts are new or foreign to you, please reference Gentoo’s documentation for setting up a Complete Virtual Mail Server as a prerequisite. For the remainder of the article, I am going to use two hypothetical domains of domain1.com and domain2.com. Any time that they are referenced, you will want to replace them with your actual domains.

BIND (DNS) configurations

In order to keep the web stack and mail stack separate in terms of DNS, I like to have mail.domain1.com & mail.domain2.com subdomains for the MX records, but just have them point to the same A record used for the website. Some may consider this to be unnecessary, but I have found the separation helpful for troubleshooting if any problems should arise. Here are the relevant portions of the zone files for each domain:

# grep -e 'A\|MX ' /etc/bind/pri/domain1.com.external.zone 
domain1.com.		300	IN	A	$IP_ADDRESS
mail.domain1.com.	300	IN	A	$IP_ADDRESS

# grep -e 'A\|MX ' /etc/bind/pri/domain2.com.external.zone 
domain2.com.		300	IN	A	$IP_ADDRESS
mail.domain2.com.	300	IN	A	$IP_ADDRESS

In the above snippets, $IP_ADDRESS should be the actual IPv4 address of the webserver. It should be noted that, in this setup, the web stack and the mail stack reside on the same physical host, so the IP is the same for both stacks.

Apache (webserver) configurations

As mentioned above, I keep the web stack and mail stack separate in terms of DNS. For the LetsEncrypt certificates (covered in the next section), though, I use the same certificate for both stacks. I do so by generating the cert for both the main domain and the ‘mail’ subdomain. In order for this to work, I make the ‘mail’ subdomain a ServerAlias in the Apache vhost configurations:

# grep -e 'ServerName \|ServerAlias ' www.domain1.com.conf 
	ServerName domain1.com
	ServerAlias www.domain1.com mail.domain1.com

# grep -e 'ServerName \|ServerAlias ' www.domain2.com.conf 
	ServerName domain2.com
	ServerAlias www.domain2.com mail.domain2.com

This allows the verification of the ‘mail’ subdomain to be done via the main URL instead of requiring a separate public-facing site directory for it.

LetsEncrypt SSL certificates

LetsEncrypt is a non-profit certificate authority (CA) that provides X.509 (TLS) certificates free-of-charge. The issued certificates are only valid for 90 days, which encourages automated processes to handle renewals. The recommended method is to use the certbot tool for renewals, and there are many plugins available that provide integration with various webservers. Though I run a combination of Apache and NGINX, I prefer to not have certbot directly interact with them. Rather, I choose to rely on certbot solely for the certificate generation & renewal, and to handle the installation thereof via other means. For this tutorial, I will use the ‘certonly’ option with the webroot plugin:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/$DOMAIN/$HOST/htdocs/ --domains $DOMAIN,$DOMAIN,$DOMAIN

In the code snippet above, you will replace $DOMAIN with the actual domain, and $HOST with the subdomain. So, for our two hypothetical domains, the commands translate as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,mail.domain1.com

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain2.com/www/htdocs/ --domains domain2.com,www.domain2.com,mail.domain2.com

The webroot plugin will create a temporary file under ${webroot-path}/.well-known/acme-challenge/ and then check that file via HTTP in order to validate the server. Make sure that the directory is publicly accessible or else the validation will fail. Once certbot validates the listed domains—in this setup, the ‘www’ and ‘mail’ subdomains are just aliases to the primary domain (see the BIND and Apache configurations sections above)—it will generate the SSL certificates and place them under /etc/letsencrypt/live/domain1.com/ and /etc/letsencrypt/live/domain2.com/, respectively. There will be two files for each certificate:

• fullchain.pem –> the public certificate
• privkey.pem –> the private key for the certificate

Though these certificates are often used in conjunction with the web stack, we are going to use them for securing the mail stack as well.

Dovecot (IMAP) configurations

Now that we have the certificates for each domain, we’ll start by securing the IMAP server (i.e. Dovecot) so that the users’ Mail User Agent (MUA, or more colloquially, “email client” [like Thunderbird or Outlook]) will no longer require a security exception due to a domain mismatch. Adding the domain-specific SSL certificate to Dovecot is a straightforward process that only requires two directives per domain. For domain1.com and domain2.com, add the following lines to /etc/dovecot/conf.d/10-ssl.conf:

local_name mail.domain1.com {
  ssl_cert = </etc/letsencrypt/live/domain1.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain1.com/privkey.pem
}

local_name mail.domain2.com {
  ssl_cert = </etc/letsencrypt/live/domain2.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain2.com/privkey.pem
}

Those code blocks can be copied and pasted for any additional virtual hosts or domains that are needed. As with any configuration change, make sure to restart the application in order to make the changes active:

/etc/init.d/dovecot restart

Postfix (SMTP) configurations

Configuring Dovecot to use the SSL certificate for securing the IMAP connection from the user’s email client to the server is only one part of the process—namely the connection when the user is retrieving mails from the server. This next part will use the same certificate to secure the SMTP connection (via the Postfix SMTP server) for sending mails.

The first step is to create a file that will be used for mapping each certificate to its respective domain. Postfix can handle this correlation via Server Name Indication (SNI), which is an extension of the TLS protocol that indicates the hostname of the server at the beginning of the handshake process. Though there is no naming requirement for this map file, I chose to create it as /etc/postfix/vmail_ssl. The format of the file is:

$DOMAIN   $PRIVATE_KEY   $PUBLIC_KEY_CHAIN

So for our example of domain1.com and domain2.com, the file would consist of the following entries:

mail.domain1.com /etc/letsencrypt/live/domain1.com/privkey.pem /etc/letsencrypt/live/domain1.com/fullchain.pem

mail.domain2.com /etc/letsencrypt/live/domain2.com/privkey.pem /etc/letsencrypt/live/domain2.com/fullchain.pem

Though this file is in plaintext, Postfix doesn’t understand the mapping in this format. Instead, a base64-encoded Berkeley DB file needs to be used. Thankfully, Postfix makes creating such a file very easy via the postmap utility. Once you have created and populated your /etc/postfix/vmail_ssl file with the entries for each domain, issue the following command:

# postmap -F hash:/etc/postfix/vmail_ssl

which will create the Berkeley DB file (named vmail_ssl.db) in the same directory:

# find /etc/postfix/ -type f -iname '*vmail_ssl*'
/etc/postfix/vmail_ssl.db
/etc/postfix/vmail_ssl

# file /etc/postfix/vmail_ssl
/etc/postfix/vmail_ssl: ASCII text

# file /etc/postfix/vmail_ssl.db 
/etc/postfix/vmail_ssl.db: Berkeley DB (Hash, version 10, native byte-order)

Now that we have created the mapping table, we have to configure Postfix to use it for SMTP connections. It’s acceptable to have both the SNI-mapped certificates AND a generic SSL certificate as the default (for when a domain isn’t listed in the mapping table). Postfix can have both directives specified simultaneously. To do so, the following directives need to be added to /etc/postfix/main.cf (the comments explain both sets of directives):

## Default SSL cert for SMTP if SNI is not enabled
smtpd_tls_cert_file = /etc/ssl/mail/server.pem
smtpd_tls_key_file = /etc/ssl/mail/server.key

## Mappings for SMTP SSL certs when SNI is enabled
tls_server_sni_maps = hash:/etc/postfix/vmail_ssl

After making those modifications to Postfix’s main configuration file, it is required to restart it:

/etc/init.d/postfix restart

That’s it! Now the full mail stack is secured using domain-specific SSL certificates for both IMAP and SMTP connections. The remaining sections below will explain some maintenance-related procedures such as handling the LetsEncrypt certificate renewals & updating the mappings in Postfix automatically, as well as verifying it’s all working as intended (and some troubleshooting tips in case it’s not). 🙂

Automatic renewal (cron) configurations

As mentioned in the LetsEncrypt section above, the certificates that they issue are only valid for a period of 90 days. One of the reasons for the relatively short validity period is to encourage automation when it comes to renewing them. I choose to handle the renewals automatically via cron:

# tail -n 4 /var/spool/cron/crontabs/root 

## LetsEncrypt certificate renewals on first of each month
## See /etc/letsencrypt/renewal-hooks/post/ for Postfix & Apache hooks 
0  2  1  *  *   /usr/bin/certbot renew --quiet

This cron entry instructs LetsEncrypt’s certbot to check the validity of ALL certificates at 02:00 (server time) on the first of every month (if that format is unfamiliar to you, see Wikipedia’s article on cron). The renew subcommand will automatically generate a new certificate for any found to expire within the next 30 days, and the quiet option will silence any output except for errors, which is appropriate for use with a cron job.

That’s the procedure for renewing the certificate automatically, but what about then automatically updating the appropriate stack configurations—in particular, Postfix’s vmail_ssl mappings table (and Apache, but that’s outside the scope of this tutorial)? If the certificate is renewed, but it is not updated in Postfix’s hash table, there will be a mismatch error. As mentioned in the comment on the cron entry, I chose to handle those configuration updates automatically via certbot’s ‘renewal hooks’, which can be found under /etc/letsencrypt/renewal-hooks/. In this case, the configuration updates need to happen after certificate renewal, so they are put under the post/ subdirectory.

I have two scripts that run after a certificate renewal, but only the 01_postfix_smtp_ssl.sh one is applicable for the mail stack:

# ls /etc/letsencrypt/renewal-hooks/post/
01_postfix_smtp_ssl.sh  02_apache.sh

# cat /etc/letsencrypt/renewal-hooks/post/01_postfix_smtp_ssl.sh 
#!/bin/bash
/usr/sbin/postmap -F hash:/etc/postfix/vmail_ssl
/etc/init.d/postfix restart
exit 0

The simple script issues the same postmap command from the ‘Postfix (SMTP) configurations‘ section above, and then restarts Postfix. If everything goes smoothly, it will exit cleanly (‘exit 0’). The script ensures that the new certificate is immediately applied to the Postfix configuration so that there aren’t validation errors after the automated renewal process.

Verification of the certificates

If everything went according to plan, valid SSL certificates should be in place for both mail.domain1.com and mail.domain2.com. Like any good engineer, though, we don’t want to just assume that it’s working as intended. So… we should test it! You could just open an email client of your choice and view the certificates for IMAP and SMTP connections. Personally, though, I prefer using terminal-based utilities as I find them to be more efficient. In this case, we can use the openssl command for connecting to each domain as a test, and the basic syntax is:

For SMTP:

openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp

For IMAP:

openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com

These commands will output a lot of information including the full public certificate, the issuing authority (LetsEncrypt), handshake details, SSL session details, and so on. If you’re interested in all of those details, feel free to issue the commands as they are above (obviously swapping out the actual domains and the ports that you use for SMTP and IMAP). If, however, you simply want to confirm that the certificates are valid, you can pipe the commands to grep in order to limit the output:

For SMTP:

$ openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
250 CHUNKING
subject=CN = domain1.com
Verify return code: 0 (ok)

For IMAP:

$ openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
subject=CN = domain1.com
Verify return code: 0 (ok)

If you see output similar to what’s above, then everything is working as it should be. In particular, you want to make sure that references to the ‘CN’ match the domain, and that you see a ‘Verify return code:’ of 0 (ok) Pat yourself on the back and grab your beverage of choice to celebrate a job well done. 🙂

Additional information

If you have already been using a domain for a website or other service, chances are that you have already generated a LetsEncrypt SSL certificate for it. Thankfully LetsEncrypt makes it easy to append a new subdomain to an existing certificate instead of having to generate a completely separate one for the ‘mail’ subdomain used in this guide (e.g. mail.domain1.com).

The first step is to find the certificate that you want to modify (in this case, domain1.com) and see which subdomains are covered under it. This can be accomplished using the certbot certificates command. The output will look something like this:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com
   Expiry Date: 2021-05-02 06:03:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

The important part is the list of subdomains on the Domains: line because you need to reference ALL of them when using the --expand flag that follows. Using the output from above, the command would be constructed as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,staging.domain1.com,mail.domain1.com --expand

If certbot indicates that the new certificate has been generated without any errors, you can check it again using the certbot certificates command from above and validate that now the ‘mail’ subdomain is listed as well:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com mail.domain1.com
   Expiry Date: 2021-05-14 06:05:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

Troubleshooting

I can’t anticipate the full gamut of problems that could potentially arise when going through this guide, but I will try to cover some common pitfalls here. If you run into a problem, feel free to comment and I will try to help you through it.

>>> Postfix error about the table hash:

If Postfix won’t start after the modifications from the sections above, and you see a line like this in the mail logs:

[postfix/smtpd] warning: table hash:/etc/postfix/vmail_ssl.db: key mail.domain1.com: malformed BASE64 value: /etc/letsencrypt/live/domain1

then the problem stems from running postmap without the -F flag. Try it again with that flag: postmap -F hash:/etc/postfix/vmail_ssl which should create a syntactically correct hash table, allowing Postfix to properly start up.