Apr 06 2023

Fixing Intel Wi-Fi 6 AX200 latency and ping spikes in Linux

Recently, I purchased a mini-PC with an embedded Intel Wi-Fi 6 AX200 wireless chipset and installed Gentoo Linux on it. The reason I purchased the mini-PC was to replace my ageing music server (running MPD), but I also wanted to use it for playing retro video games through my entertainment centre. Everything went well with the OS installation, but during that time, I had it plugged into my router via a wired connection. Once I moved the mini-PC into place in my sitting room and started using the wireless connection, I noticed that there was substantial lag even when doing something simple like typing in a terminal via SSH.

A quick ping test showed that there was clearly a problem with the wireless. For my home network, I consider any ping response time of >=5ms to be unacceptable and indicative of an underlying network problem:

$ ping -c 20 192.168.1.120
PING 192.168.1.120 (192.168.1.120) 56(84) bytes of data.
64 bytes from 192.168.1.120: icmp_seq=1 ttl=64 time=214 ms
64 bytes from 192.168.1.120: icmp_seq=2 ttl=64 time=30.7 ms
64 bytes from 192.168.1.120: icmp_seq=3 ttl=64 time=54.4 ms
64 bytes from 192.168.1.120: icmp_seq=4 ttl=64 time=75.1 ms
64 bytes from 192.168.1.120: icmp_seq=5 ttl=64 time=97.8 ms
64 bytes from 192.168.1.120: icmp_seq=6 ttl=64 time=122 ms
64 bytes from 192.168.1.120: icmp_seq=7 ttl=64 time=142 ms
64 bytes from 192.168.1.120: icmp_seq=8 ttl=64 time=2.46 ms
64 bytes from 192.168.1.120: icmp_seq=9 ttl=64 time=2.30 ms
64 bytes from 192.168.1.120: icmp_seq=10 ttl=64 time=4.72 ms
64 bytes from 192.168.1.120: icmp_seq=11 ttl=64 time=26.3 ms
64 bytes from 192.168.1.120: icmp_seq=12 ttl=64 time=2.30 ms
64 bytes from 192.168.1.120: icmp_seq=13 ttl=64 time=71.7 ms
64 bytes from 192.168.1.120: icmp_seq=14 ttl=64 time=94.6 ms
64 bytes from 192.168.1.120: icmp_seq=15 ttl=64 time=116 ms
64 bytes from 192.168.1.120: icmp_seq=16 ttl=64 time=139 ms
64 bytes from 192.168.1.120: icmp_seq=17 ttl=64 time=161 ms
64 bytes from 192.168.1.120: icmp_seq=18 ttl=64 time=184 ms
64 bytes from 192.168.1.120: icmp_seq=19 ttl=64 time=205 ms
64 bytes from 192.168.1.120: icmp_seq=20 ttl=64 time=23.5 ms

Though 4 of the 20 ping response times were under my 5-millisecond threshold, the other 16 were not only above it, but many of them were nonsensically high for a small home network (e.g. 214ms).

In this type of scenario, the first thing that I consider is the driver and/or firmware for the wireless adapter (namely, the Intel Wi-Fi 6 AX200). For nearly all modern Intel wireless chips the Linux driver is the in-kernel iwlwifi driver, so I didn’t pay too much attention there. That driver has two possible modules to use in conjunction:

  • DVM (iwldvm)
    • The module that supports the firmware for a specific group of (primarily) AGN chips
  • MVM (iwlmvm)
    • The module that supports the firmware for a much broader scope of Intel wireless chips

I had chosen the iwlmvm module and built it into my kernel for convenience. I also then chose to load the corresponding firmware directly into the kernel as well. The gigantic linux-firmware package contains all the various options for the iwlwifi supporting firmware, and from that table, I saw that the original firmware for the Intel Wi-Fi 6 AX200 was named:

iwlwifi-cc-46.3cfab8da.0.ucode

Looking at the current linux-firmware git tree (at the time of this writing), the relevant firmware packages were:

  • iwlwifi-cc-a0-50.ucode
  • iwlwifi-cc-a0-59.ucode
  • iwlwifi-cc-a0-66.ucode
  • iwlwifi-cc-a0-72.ucode
  • iwlwifi-cc-a0-73.ucode
  • iwlwifi-cc-a0-74.ucode
  • iwlwifi-cc-a0-77.ucode

Using my trial-and-error approach of rebooting and looking at the output of dmesg | grep iwlwifi to find the first version the kernel attempted (but failed) to load, I found that the correct version was iwl-cc-a0-72.ucode, and passed via the kernel’s firmware loader.

After trying various options (such as switching from wpa_supplicant to iwd, and using an older firmware blob), I finally found the fix for the latency and ping spikes: power saving. By issuing iw wlan0 set power_save off, and then starting a new ping test, I could immediately see that the problem was fixed:

$ ping -c 20 192.168.1.120
PING 192.168.1.120 (192.168.1.120) 56(84) bytes of data.
64 bytes from 192.168.1.120: icmp_seq=1 ttl=64 time=2.90 ms
64 bytes from 192.168.1.120: icmp_seq=2 ttl=64 time=1.68 ms
64 bytes from 192.168.1.120: icmp_seq=3 ttl=64 time=2.43 ms
64 bytes from 192.168.1.120: icmp_seq=4 ttl=64 time=2.68 ms
64 bytes from 192.168.1.120: icmp_seq=5 ttl=64 time=3.08 ms
64 bytes from 192.168.1.120: icmp_seq=6 ttl=64 time=2.80 ms
64 bytes from 192.168.1.120: icmp_seq=7 ttl=64 time=3.25 ms
64 bytes from 192.168.1.120: icmp_seq=8 ttl=64 time=3.17 ms
64 bytes from 192.168.1.120: icmp_seq=9 ttl=64 time=2.83 ms
64 bytes from 192.168.1.120: icmp_seq=10 ttl=64 time=3.01 ms
64 bytes from 192.168.1.120: icmp_seq=11 ttl=64 time=2.77 ms
64 bytes from 192.168.1.120: icmp_seq=12 ttl=64 time=2.80 ms
64 bytes from 192.168.1.120: icmp_seq=13 ttl=64 time=3.37 ms
64 bytes from 192.168.1.120: icmp_seq=14 ttl=64 time=2.52 ms
64 bytes from 192.168.1.120: icmp_seq=15 ttl=64 time=2.71 ms
64 bytes from 192.168.1.120: icmp_seq=16 ttl=64 time=2.83 ms
64 bytes from 192.168.1.120: icmp_seq=17 ttl=64 time=3.25 ms
64 bytes from 192.168.1.120: icmp_seq=18 ttl=64 time=2.78 ms
64 bytes from 192.168.1.120: icmp_seq=19 ttl=64 time=2.48 ms
64 bytes from 192.168.1.120: icmp_seq=20 ttl=64 time=3.37 ms

--- 192.168.1.120 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 19036ms
rtt min/avg/max/mdev = 1.679/2.834/3.371/0.379 ms

Now that I had found the solution to the problem, the next task was to make the changes persistent across reboots. Of course, I could throw that iw command command into an rc.local script or something like that, but that seemed hackish to me. Instead, I decided to change my approach for loading the iwlwifi driver and firmware. Rather than having both the driver and the firmware built-in to the kernel, I chose to load them as kernel modules. Doing so allowed me to pass configuration options to the modules when they load. I made a configuration file at /etc/modprobe.d/iwlwifi.conf with the following contents: 

$ cat /etc/modprobe.d/iwlwifi.conf 

## Has the same effect has running `iw wlan0 set power_save off`
## Both options sets are needed as iwlmvm will override iwlwifi :(
options iwlwifi power_save=0
## iwlmvm 1=always on, 2=balanced, 3=low-power
options iwlmvm power_scheme=1

As I mentioned in the comments there, BOTH options need to be set: one option passed to the iwlwifi module and the other option passed to the iwlmvm module. Also note the available parameters for the iwlmvm power_scheme:

  • 1 = always on
  • 2 = balanced
  • 3 = low-power

I have validated that these settings work across reboots, and that I no longer see the latency or ping spikes when connecting to this mini-PC over the Intel Wi-Fi 6 AX200. Some of these instructions may be specific to Gentoo and/or the OpenRC init system that I choose to use, but they should be readily adaptable to other distributions and init systems.

Cheers,
Nathan Zachary

Mar 22 2023

Photos from my October 2022 trip to Germany and Switzerland

These days it seems like there simply isn’t enough time in the day. Though my trip to Germany and Switzerland was in October of 2022, I haven’t had time to post some of the photos here on my blog. I’m not going to go through the whole trip, or post all the photos that I took throughout it, but I do want to share some of my favourites.

Feel free to click on any of the thumbnails to see larger versions (though, even the larger versions are still only 25% of the original size).  If you would like to use the full-resolution image, please leave a comment and I will contact you to discuss.

Switzerland:

The mountains in the Swiss Alps are truly remarkable, and I think that some of these shots capture the beauty.  The first few are from the town of Lauterbrunnen, and the latter set is from around Lake Oeschinen (Oeschinensee in German).

Sun caressing the cliff side in Lauterbrunnen
Sun caressing the cliff side in Lauterbrunnen

Last glimpses of light looking south in the valley of Lauterbrunnen
Last glimpses of light looking south in the valley of Lauterbrunnen

Beautiful forests work their way up the mountainside in Lauterbrunnen
Beautiful forests work their way up the mountainside in Lauterbrunnen

The summit footpath down to Lake Oeschinen
The summit footpath down to Lake Oeschinen

Another view to the south from the summit of Lake Oeschinen
View to the south from the summit of Lake Oeschinen

Starkly different view to the north from the summit of Lake Oeschinen
Starkly different view to the north from the summit of Lake Oeschinen

Germany:

The following two photos were from Wiesbaden and the Rheingau region of Germany, respectively. The first is of St. Elizabeth Russian Orthodox Church in Wiesbaden, and the second is at Weingut Robert Weil (a great producer of Rheingau Riesling) with the storms rolling in.

St. Elizabeth Russian Orthodox Church in Wiesbaden, Germany in the sunlight
St. Elizabeth Russian Orthodox Church in Wiesbaden, Germany in the sunlight

Weingut Robert Weil with the ominous clouds of an oncoming storm
Weingut Robert Weil with the ominous clouds of an oncoming storm

Aug 19 2021

Fix for delayed text messages (SMS and MMS) on Android

Recently, my mobile device manufacturer (OnePlus) decided to roll out updates to a new version of their Android-based operating system (OxygenOS) to use Android 11. After this particular update, I noticed a big delay in sending and receiving text messages (both SMS and MMS). At first I thought it was simply a temporary problem, but it didn’t go away. Doing some troubleshooting, I figured that it had to be related to one of three things:

  • A problem with my carrier (Deutsche Telekom)
  • A problem with the version of the Android 11 ‘Messages’ app
  • A problem with a setting that was silently changed during the update to Android 11

I tried installing a different messaging app called Textra (which, by the way, I liked so much that I have continued using it), but that didn’t fix the problem. I confirmed that it wasn’t a carrier problem by asking some friends who also use the same provider, and their messaging was immediate as it should be. That left option #3 as the likely culprit.

After modifying several settings without finding a fix, I started delving a little more deeply into potential problems. That’s when I remembered that there are “secret codes” that you can enter using your dialler that can help with troubleshooting (see this article for a listing of some of them). The one that I wanted was for testing the carrier settings, and it can be accessed by going to your dialler and entering:

*#*#4636#*#*

On the resulting screen, tap “Phone information” and scroll to the bottom to tap the “Trigger Operator Provisioning” button (note that your screen may look a little different):

Android carrier settings screen
Click to enlarge

After doing so, my text messaging (both SMS and MMS) started working again without any delay.

Cheers,

Zach