Docker in WSL2 (the right way ++)

This is an extension of very helpful article Running Docker on WSL2 without Docker Desktop (the right way) by Felipe Santos. My post attempts to go further and solve a very specific issue within WSL - the absence of nftables support. This is something you might not encounter unless you start running docker in docker or attempt to run docker images that rely on nftables support in some way.

I tend to use Molecule quite heavily and I often end up using docker to emulate actual full blown linux systems or kubernetes cluster in a container, which is where the absence of some kernel features shows up.

Note
The rest of this article assumes you already followed Felipe’s article linked above and that you can successfully run simple docker images in WSL2. It also assumes that you’re Using Ubuntu 20.04 WSL distribution on Windows 11 version 21H2, build 22000. Other distributions and other versions of Windows might work the same way, but your mileage might vary.

First of all there are workarounds for this situation like enabling legacy iptables functionality in the container that needs to run docker inside:

# Inside debian bullseye based container dockerd fails to run due to nf_tables support missing:
$ dockerd
# ...snip...
failed to start daemon: Error initializing network controller: error obtaining controller instance: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain:  (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.7 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1 (exit status 4))

# switch to legacy iptables:
$ update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode
$ dockerd
# ..and docker daemon now runs fine..

If you can control the nested container environment, this might be the way to go. However as we go deeper and start running containers in these containers this becomes quite cumbersome.

This is where we have to..

Build a custom kernel

Ah the good old days. However, let’s put a slightly modern spin on it. The Kind WSL2 documentation shows how to compile your own kernel inside a docker container. We can already run containers and this avoids polluting your WSL deployment with build dependencies, so let’s do just that. (I’m going to use similar approach, but will diverge a bit to keep things simple)

# Run Ubuntu image to get build environment.
docker run --name wsl-kernel-builder --rm -it ubuntu:20.04 bash
Note
The rest of the commands will be executed inside the container we just started
# Get kernel version
KERNEL_VERSION="$(uname -r)"

# Install dependencies
apt update
apt install -y git build-essential flex bison libssl-dev libelf-dev bc


# Checkout WSL2 Kernel repo and cd into it
git clone --branch "linux-msft-wsl-${KERNEL_VERSION%%-*}" --depth 1 https://github.com/microsoft/WSL2-Linux-Kernel.git kernel
cd kernel

# Copy configuration file
cp Microsoft/config-wsl ./config

# Enable all the *NFT* features
awk '/NFT/{print $2 "=y"}' Microsoft/config-wsl >> config

# Also enable CONFIG_NETFILTER_XT_MATCH_RECENT while we're at it
echo "CONFIG_NETFILTER_XT_MATCH_RECENT=y" >> config

# feel free to add your stuff here, if you need something specific enabled

Now that we’re happy with the configuration, let’s compile the kernel:

make -j$(nproc) KCONFIG_CONFIG=./config

You might get asked about extra features that can be enabled, it’s usually good idea to enable those with y. Then compilation will start. Grab a tea and wait. If the compilation was successful we’ll have bzImage ready:

Kernel: arch/x86/boot/bzImage is ready

We’ll need to copy this image out of the container somewhere WSL can access it:

Note
This step will be executed outside of the build container. So open a new WSL shell to run it but keep the build container running in the previous shell.
# Modify your destination path to something suitable, I'm just using my windows home directory
docker cp wsl-kernel-builder:/kernel/arch/x86/boot/bzImage /mnt/c/Users/mprasil/bzImage

You can now exit out of your build docker container. Because we used --rm to run it, docker will remove container automatically to clean up all the mess.

Use the custom kernel image

This can be set in the .wslconfig. You need to create a configuration in your windows home directory. (In my case that’s C:\Users\mprasil\.wslconfig):

[wsl2]
kernel=c:\\Users\\mprasil\\bzImage
Note
Change the path to wherever you copied the bzImage file. Don’t forget to escape the backslash characters as I do above.

To actually apply the changes, we need to restart the WSL distribution. To do that open command prompt (your windows command prompt, not WSL shell) and shutdown WSL:

Note
This will shutdown all WSL distributions you might have running and all of the GUI apps from WSL you might be using, so make sure to save your work before you proceed.
wsl --shutdown

Verify that custom image is used

Now we can start WSL shell as we usually do. WSL should pick up our custom configuration and use our kernel image. As a quick test, I’ll run the same debian bullseye image and try to run dockerd:

# Inside debian bullseye based container
$ dockerd
# ...snip...
INFO[2022-02-18T13:20:38.848409000Z] Daemon has completed initialization
INFO[2022-02-18T13:20:38.889224500Z] API listen on /var/run/docker.sock

Success 🎉

Undo the custom kernel changes

If anything failed, or you no longer need to use custom image, all you have to do is remove the .wslconfig override. If you have no other configuration there, just removing the file and restarting WSL distribution should revert to the default kernel provided by the WSL distribution you’re using.

2022-02-18 00:00 Miroslav Prasil