Fix for delayed text messages (SMS and MMS) on Android

Recently, my mobile device manufacturer (OnePlus) decided to roll out updates to a new version of their Android-based operating system (OxygenOS) to use Android 11. After this particular update, I noticed a big delay in sending and receiving text messages (both SMS and MMS). At first I thought it was simply a temporary problem, but it didn’t go away. Doing some troubleshooting, I figured that it had to be related to one of three things:

  • A problem with my carrier (Deutsche Telekom)
  • A problem with the version of the Android 11 ‘Messages’ app
  • A problem with a setting that was silently changed during the update to Android 11

I tried installing a different messaging app called Textra (which, by the way, I liked so much that I have continued using it), but that didn’t fix the problem. I confirmed that it wasn’t a carrier problem by asking some friends who also use the same provider, and their messaging was immediate as it should be. That left option #3 as the likely culprit.

After modifying several settings without finding a fix, I started delving a little more deeply into potential problems. That’s when I remembered that there are “secret codes” that you can enter using your dialler that can help with troubleshooting (see this article for a listing of some of them). The one that I wanted was for testing the carrier settings, and it can be accessed by going to your dialler and entering:

*#*#4636#*#*

On the resulting screen, tap “Phone information” and scroll to the bottom to tap the “Trigger Operator Provisioning” button (note that your screen may look a little different):

Android carrier settings screen
Click to enlarge

After doing so, my text messaging (both SMS and MMS) started working again without any delay.

Cheers,

Zach

LetsEncrypt SSL certificates for vhosts within the mail stack – via Postfix (SMTP) and Dovecot (IMAP)

For a long time, I didn’t care about using self-signed SSL certificates for the mail stack because 1) they still secured the connection to the server, and 2) those certificates weren’t seen or utilised by anyone other than me. However, for my datacentre infrastructure which houses clients’ websites and email, using self-signed or generic certificates (even for the mail stack) wasn’t a very good solution as mail clients (e.g Thunderbird or Outlook) notify users of the problematic certs. Clients with dedicated mail servers could use valid certificates (freely from LetsEncrypt) without problem, but those on shared infrastructure posed a different issue—how can mail for different domains all sharing the same IPv4 address use individualised certificates for their SMTP and IMAP connections? This article will explain the method that I used to assign domain-specific certificates for the full mail stack using LetsEncrypt’s certbot for the certs themselves, the Postfix MTA (mail transfer agent [for SMTP]), and the Dovecot IMAP server. This article is tailored to Gentoo Linux, but should be easily applied to nearly any distribution. Two general caveats are:

  • The locations of files may be different for your distribution. Consult your distribution’s documentation for the appropriate file locations
  • I use OpenRC as my init system in Gentoo. If your distribution uses systemd, you will need to replace any reference to code snippets containing /etc/init.d/$SERVICE $ACTION with systemctl $ACTION $SERVICE
    • As an example, I mention restarting Dovecot with /etc/init.d/dovecot restart
    • A systemd user would instead issue systemctl restart dovecot

I will provide a significant amount of ancillary information pertaining to each step of this process. If you are comfortable with the mail stack, some of it may be rudimentary, so feel free to skip ahead. Conversely, if any of the concepts are new or foreign to you, please reference Gentoo’s documentation for setting up a Complete Virtual Mail Server as a prerequisite. For the remainder of the article, I am going to use two hypothetical domains of domain1.com and domain2.com. Any time that they are referenced, you will want to replace them with your actual domains.

BIND (DNS) configurations

In order to keep the web stack and mail stack separate in terms of DNS, I like to have mail.domain1.com & mail.domain2.com subdomains for the MX records, but just have them point to the same A record used for the website. Some may consider this to be unnecessary, but I have found the separation helpful for troubleshooting if any problems should arise. Here are the relevant portions of the zone files for each domain:

# grep -e 'A\|MX ' /etc/bind/pri/domain1.com.external.zone 
domain1.com.		300	IN	A	$IP_ADDRESS
mail.domain1.com.	300	IN	A	$IP_ADDRESS

# grep -e 'A\|MX ' /etc/bind/pri/domain2.com.external.zone 
domain2.com.		300	IN	A	$IP_ADDRESS
mail.domain2.com.	300	IN	A	$IP_ADDRESS

In the above snippets, $IP_ADDRESS should be the actual IPv4 address of the webserver. It should be noted that, in this setup, the web stack and the mail stack reside on the same physical host, so the IP is the same for both stacks.

Apache (webserver) configurations

As mentioned above, I keep the web stack and mail stack separate in terms of DNS. For the LetsEncrypt certificates (covered in the next section), though, I use the same certificate for both stacks. I do so by generating the cert for both the main domain and the ‘mail’ subdomain. In order for this to work, I make the ‘mail’ subdomain a ServerAlias in the Apache vhost configurations:

# grep -e 'ServerName \|ServerAlias ' www.domain1.com.conf 
	ServerName domain1.com
	ServerAlias www.domain1.com mail.domain1.com

# grep -e 'ServerName \|ServerAlias ' www.domain2.com.conf 
	ServerName domain2.com
	ServerAlias www.domain2.com mail.domain2.com

This allows the verification of the ‘mail’ subdomain to be done via the main URL instead of requiring a separate public-facing site directory for it.

LetsEncrypt SSL certificates

LetsEncrypt is a non-profit certificate authority (CA) that provides X.509 (TLS) certificates free-of-charge. The issued certificates are only valid for 90 days, which encourages automated processes to handle renewals. The recommended method is to use the certbot tool for renewals, and there are many plugins available that provide integration with various webservers. Though I run a combination of Apache and NGINX, I prefer to not have certbot directly interact with them. Rather, I choose to rely on certbot solely for the certificate generation & renewal, and to handle the installation thereof via other means. For this tutorial, I will use the ‘certonly’ option with the webroot plugin:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/$DOMAIN/$HOST/htdocs/ --domains $DOMAIN,$DOMAIN,$DOMAIN

In the code snippet above, you will replace $DOMAIN with the actual domain, and $HOST with the subdomain. So, for our two hypothetical domains, the commands translate as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,mail.domain1.com

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain2.com/www/htdocs/ --domains domain2.com,www.domain2.com,mail.domain2.com

The webroot plugin will create a temporary file under ${webroot-path}/.well-known/acme-challenge/ and then check that file via HTTP in order to validate the server. Make sure that the directory is publicly accessible or else the validation will fail. Once certbot validates the listed domains—in this setup, the ‘www’ and ‘mail’ subdomains are just aliases to the primary domain (see the BIND and Apache configurations sections above)—it will generate the SSL certificates and place them under /etc/letsencrypt/live/domain1.com/ and /etc/letsencrypt/live/domain2.com/, respectively. There will be two files for each certificate:

  • fullchain.pem –> the public certificate
  • privkey.pem –> the private key for the certificate

Though these certificates are often used in conjunction with the web stack, we are going to use them for securing the mail stack as well.

Dovecot (IMAP) configurations

Now that we have the certificates for each domain, we’ll start by securing the IMAP server (i.e. Dovecot) so that the users’ Mail User Agent (MUA, or more colloquially, “email client” [like Thunderbird or Outlook]) will no longer require a security exception due to a domain mismatch. Adding the domain-specific SSL certificate to Dovecot is a straightforward process that only requires two directives per domain. For domain1.com and domain2.com, add the following lines to /etc/dovecot/conf.d/10-ssl.conf:

local_name mail.domain1.com {
  ssl_cert = </etc/letsencrypt/live/domain1.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain1.com/privkey.pem
}

local_name mail.domain2.com {
  ssl_cert = </etc/letsencrypt/live/domain2.com/fullchain.pem
  ssl_key = </etc/letsencrypt/live/domain2.com/privkey.pem
}

Those code blocks can be copied and pasted for any additional virtual hosts or domains that are needed. As with any configuration change, make sure to restart the application in order to make the changes active:

/etc/init.d/dovecot restart

Postfix (SMTP) configurations

Configuring Dovecot to use the SSL certificate for securing the IMAP connection from the user’s email client to the server is only one part of the process—namely the connection when the user is retrieving mails from the server. This next part will use the same certificate to secure the SMTP connection (via the Postfix SMTP server) for sending mails.

The first step is to create a file that will be used for mapping each certificate to its respective domain. Postfix can handle this correlation via Server Name Indication (SNI), which is an extension of the TLS protocol that indicates the hostname of the server at the beginning of the handshake process. Though there is no naming requirement for this map file, I chose to create it as /etc/postfix/vmail_ssl. The format of the file is:

$DOMAIN   $PRIVATE_KEY   $PUBLIC_KEY_CHAIN

So for our example of domain1.com and domain2.com, the file would consist of the following entries:

mail.domain1.com /etc/letsencrypt/live/domain1.com/privkey.pem /etc/letsencrypt/live/domain1.com/fullchain.pem

mail.domain2.com /etc/letsencrypt/live/domain2.com/privkey.pem /etc/letsencrypt/live/domain2.com/fullchain.pem

Though this file is in plaintext, Postfix doesn’t understand the mapping in this format. Instead, a base64-encoded Berkeley DB file needs to be used. Thankfully, Postfix makes creating such a file very easy via the postmap utility. Once you have created and populated your /etc/postfix/vmail_ssl file with the entries for each domain, issue the following command:

# postmap -F hash:/etc/postfix/vmail_ssl

which will create the Berkeley DB file (named vmail_ssl.db) in the same directory:

# find /etc/postfix/ -type f -iname '*vmail_ssl*'
/etc/postfix/vmail_ssl.db
/etc/postfix/vmail_ssl

# file /etc/postfix/vmail_ssl
/etc/postfix/vmail_ssl: ASCII text

# file /etc/postfix/vmail_ssl.db 
/etc/postfix/vmail_ssl.db: Berkeley DB (Hash, version 10, native byte-order)

Now that we have created the mapping table, we have to configure Postfix to use it for SMTP connections. It’s acceptable to have both the SNI-mapped certificates AND a generic SSL certificate as the default (for when a domain isn’t listed in the mapping table). Postfix can have both directives specified simultaneously. To do so, the following directives need to be added to /etc/postfix/main.cf (the comments explain both sets of directives):

## Default SSL cert for SMTP if SNI is not enabled
smtpd_tls_cert_file = /etc/ssl/mail/server.pem
smtpd_tls_key_file = /etc/ssl/mail/server.key

## Mappings for SMTP SSL certs when SNI is enabled
tls_server_sni_maps = hash:/etc/postfix/vmail_ssl

After making those modifications to Postfix’s main configuration file, it is required to restart it:

/etc/init.d/postfix restart

That’s it! Now the full mail stack is secured using domain-specific SSL certificates for both IMAP and SMTP connections. The remaining sections below will explain some maintenance-related procedures such as handling the LetsEncrypt certificate renewals & updating the mappings in Postfix automatically, as well as verifying it’s all working as intended (and some troubleshooting tips in case it’s not). 🙂

Automatic renewal (cron) configurations

As mentioned in the LetsEncrypt section above, the certificates that they issue are only valid for a period of 90 days. One of the reasons for the relatively short validity period is to encourage automation when it comes to renewing them. I choose to handle the renewals automatically via cron:

# tail -n 4 /var/spool/cron/crontabs/root 

## LetsEncrypt certificate renewals on first of each month
## See /etc/letsencrypt/renewal-hooks/post/ for Postfix & Apache hooks 
0  2  1  *  *   /usr/bin/certbot renew --quiet

This cron entry instructs LetsEncrypt’s certbot to check the validity of ALL certificates at 02:00 (server time) on the first of every month (if that format is unfamiliar to you, see Wikipedia’s article on cron). The renew subcommand will automatically generate a new certificate for any found to expire within the next 30 days, and the quiet option will silence any output except for errors, which is appropriate for use with a cron job.

That’s the procedure for renewing the certificate automatically, but what about then automatically updating the appropriate stack configurations—in particular, Postfix’s vmail_ssl mappings table (and Apache, but that’s outside the scope of this tutorial)? If the certificate is renewed, but it is not updated in Postfix’s hash table, there will be a mismatch error. As mentioned in the comment on the cron entry, I chose to handle those configuration updates automatically via certbot’s ‘renewal hooks’, which can be found under /etc/letsencrypt/renewal-hooks/. In this case, the configuration updates need to happen after certificate renewal, so they are put under the post/ subdirectory.

I have two scripts that run after a certificate renewal, but only the 01_postfix_smtp_ssl.sh one is applicable for the mail stack:

# ls /etc/letsencrypt/renewal-hooks/post/
01_postfix_smtp_ssl.sh  02_apache.sh

# cat /etc/letsencrypt/renewal-hooks/post/01_postfix_smtp_ssl.sh 
#!/bin/bash
/usr/sbin/postmap -F hash:/etc/postfix/vmail_ssl
/etc/init.d/postfix restart
exit 0

The simple script issues the same postmap command from the ‘Postfix (SMTP) configurations‘ section above, and then restarts Postfix. If everything goes smoothly, it will exit cleanly (‘exit 0’). The script ensures that the new certificate is immediately applied to the Postfix configuration so that there aren’t validation errors after the automated renewal process.

Verification of the certificates

If everything went according to plan, valid SSL certificates should be in place for both mail.domain1.com and mail.domain2.com. Like any good engineer, though, we don’t want to just assume that it’s working as intended. So… we should test it! You could just open an email client of your choice and view the certificates for IMAP and SMTP connections. Personally, though, I prefer using terminal-based utilities as I find them to be more efficient. In this case, we can use the openssl command for connecting to each domain as a test, and the basic syntax is:

For SMTP:

openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp

For IMAP:

openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com

These commands will output a lot of information including the full public certificate, the issuing authority (LetsEncrypt), handshake details, SSL session details, and so on. If you’re interested in all of those details, feel free to issue the commands as they are above (obviously swapping out the actual domains and the ports that you use for SMTP and IMAP). If, however, you simply want to confirm that the certificates are valid, you can pipe the commands to grep in order to limit the output:

For SMTP:

$ openssl s_client -connect mail.domain1.com:25 -servername mail.domain1.com -starttls smtp | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
250 CHUNKING
subject=CN = domain1.com
Verify return code: 0 (ok)

For IMAP:

$ openssl s_client -connect mail.domain1.com:993 -servername mail.domain1.com | grep -e 'subject=CN \|Verify return code:'
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = domain1.com
verify return:1
subject=CN = domain1.com
Verify return code: 0 (ok)

If you see output similar to what’s above, then everything is working as it should be. In particular, you want to make sure that references to the ‘CN’ match the domain, and that you see a ‘Verify return code:’ of 0 (ok) Pat yourself on the back and grab your beverage of choice to celebrate a job well done. 🙂

Additional information

If you have already been using a domain for a website or other service, chances are that you have already generated a LetsEncrypt SSL certificate for it. Thankfully LetsEncrypt makes it easy to append a new subdomain to an existing certificate instead of having to generate a completely separate one for the ‘mail’ subdomain used in this guide (e.g. mail.domain1.com).

The first step is to find the certificate that you want to modify (in this case, domain1.com) and see which subdomains are covered under it. This can be accomplished using the certbot certificates command. The output will look something like this:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com
   Expiry Date: 2021-05-02 06:03:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

The important part is the list of subdomains on the Domains: line because you need to reference ALL of them when using the --expand flag that follows. Using the output from above, the command would be constructed as:

# /usr/bin/certbot certonly --agree-tos --non-interactive --webroot --webroot-path /var/www/domains/domain1.com/www/htdocs/ --domains domain1.com,www.domain1.com,staging.domain1.com,mail.domain1.com --expand

If certbot indicates that the new certificate has been generated without any errors, you can check it again using the certbot certificates command from above and validate that now the ‘mail’ subdomain is listed as well:

Certificate Name: domain1.com
   Serial Number: $some_alphanumeric_string
   Key Type: RSA
   Domains: domain1.com www.domain1.com staging.domain1.com mail.domain1.com
   Expiry Date: 2021-05-14 06:05:19+00:00 (VALID: 60 days)
   Certificate Path: /etc/letsencrypt/live/domain1.com/fullchain.pem
   Private Key Path: /etc/letsencrypt/live/domain1.com/privkey.pem

Troubleshooting

I can’t anticipate the full gamut of problems that could potentially arise when going through this guide, but I will try to cover some common pitfalls here. If you run into a problem, feel free to comment and I will try to help you through it.

>>> Postfix error about the table hash:

If Postfix won’t start after the modifications from the sections above, and you see a line like this in the mail logs:

[postfix/smtpd] warning: table hash:/etc/postfix/vmail_ssl.db: key mail.domain1.com: malformed BASE64 value: /etc/letsencrypt/live/domain1

then the problem stems from running postmap without the -F flag. Try it again with that flag: postmap -F hash:/etc/postfix/vmail_ssl which should create a syntactically correct hash table, allowing Postfix to properly start up.

Baïkal (CalDAV) 0.7.0 in Gentoo

Just this past week, the new version of of Baïkal (0.7.0)—a PHP CalDAV and CardDAV server based on Sabre—was released, and one of the key changes was that support was added for more modern versions of PHP (like 7.4).

Since my personal Gentoo server is running the ~amd64 branch, I had to wait for this release in order to get my CalDAV server up and running. For the most part, installing Baïkal 0.7.0 was a straightforward process, but there were a couple of “gotchas” along the way.

The first (and most confusing) problem came after the installation/initial configuration when I tried to access my newly-created user’s calendar via the URL:

https://dav.MYDOMAIN.com/html/dav.php/calendar/MYUSERNAME/default

I knew that something was wrong when it wouldn’t even prompt me for credentials. Instead, the logs indicated the following error message:

[Tue Jun 02 14:13:05.529805 2020] [proxy_fcgi:error] [pid 32165:tid 139743908050688] [client 71.81.87.208:38910] AH01071: Got error 'PHP message: LogicException: Requested uri (/html/dav.php) is out of base uri (/s/html/dav.php/) in /var/www/domains/MYDOMAIN/dav/htdocs/vendor/sabre/http/lib/Request.php:184

I couldn’t figure out where the “/s/” was coming in before the “/html” portion, but that was certainly the cause of the error message. I filed an issue for it, and though I still don’t know the source of the problem, I was able to work around it by adding a trailing slash to the DocumentRoot for that particular vhost:

# pwd && diff -Nut dav.MYDOMAIN.conf.PRE-20200602_docroot dav.MYDOMAIN.conf
/etc/apache2/vhosts.d/includes
--- dav.MYDOMAIN.conf.PRE-20200602_docroot 2020-06-02 17:23:20.246281195 -0400 +++ dav.MYDOMAIN.conf 2020-06-02 17:20:59.892270352 -0400
@@ -1,7 +1,7 @@
- DocumentRoot "/var/www/domains/MYDOMAIN/dav/htdocs"
+ DocumentRoot "/var/www/domains/MYDOMAIN/dav/htdocs/"

After solving that strange problem, I was at least prompted for credentials when I accessed the calendar URL from above. After logging in, I ran into one more problem, though:

Class 'XMLWriter' not found

This problem was much easier to fix. I simply needed to add the ‘xmlwriter‘ USE flag to dev-lang/php (I also added ‘xmlreader‘ for good measure), emerge it again, and restart PHP-FPM. Other distributions (like CentOS) will likely need to install the ‘php-xml’ package (or something similar).

After that fix, I am happy to report that Baïkal 0.7.0 is working beautifully, and I have my calendars synced across all my devices. I personally use Thunderbird with Lightning on my computers, and a combination of DAVx5 with Simple Calendar Pro on my Android devices.