Docker in WSL2 (the right way ++)

Published: 2022-02-18 Tags: docker Category: Linux

This is an extension of very helpful article Running Docker on WSL2 without Docker Desktop (the right way) by Felipe Santos. My post attempts to go further and solve a very specific issue within WSL - the absence of nftables support. This is something you might not encounter unless you start running docker in docker or attempt to run docker images that rely on nftables support in some way.

I tend to use Molecule quite heavily and I often end up using docker to emulate actual full blown linux systems or kubernetes cluster in a container, which is where the absence of some kernel features shows up.

First of all there are workarounds for this situation like enabling legacy iptables functionality in the container that needs to run docker inside:

# Inside debian bullseye based container dockerd fails to run due to nf_tables support missing:
$ dockerd
# ...snip...
failed to start daemon: Error initializing network controller: error obtaining controller instance: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain:  (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.7 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1 (exit status 4))

# switch to legacy iptables:
$ update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives: using /usr/sbin/iptables-legacy to provide /usr/sbin/iptables (iptables) in manual mode
$ dockerd
# ..and docker daemon now runs fine..

If you can control the nested container environment, this might be the way to go. However as we go deeper and start running containers in these containers this becomes quite cumbersome.

This is where we have to..

Build a custom kernel🔗

Ah the good old days. However, let’s put a slightly modern spin on it. The Kind WSL2 documentation shows how to compile your own kernel inside a docker container. We can already run containers and this avoids polluting your WSL deployment with build dependencies, so let’s do just that. (I’m going to use similar approach, but will diverge a bit to keep things simple)

# Run Ubuntu image to get build environment.
docker run --name wsl-kernel-builder --rm -it ubuntu:20.04 bash

# Get kernel version
KERNEL_VERSION="$(uname -r)"

# Install dependencies
apt update
apt install -y git build-essential flex bison libssl-dev libelf-dev bc python3 pahole


# Checkout WSL2 Kernel repo and cd into it
git clone --branch "linux-msft-wsl-${KERNEL_VERSION%%-*}" --depth 1 https://github.com/microsoft/WSL2-Linux-Kernel.git kernel
cd kernel

# Copy configuration file
cp Microsoft/config-wsl ./config

# Enable all the *NFT* features
awk '/NFT/{print $2 "=y"}' Microsoft/config-wsl >> config

# Also enable CONFIG_NETFILTER_XT_MATCH_RECENT while we're at it
echo "CONFIG_NETFILTER_XT_MATCH_RECENT=y" >> config

# feel free to add your stuff here, if you need something specific enabled

Now that we’re happy with the configuration, let’s compile the kernel:

make -j$(nproc) KCONFIG_CONFIG=./config

You might get asked about extra features that can be enabled, it’s usually good idea to enable those with y. Then compilation will start. Grab a tea and wait. If the compilation was successful we’ll have bzImage ready:

Kernel: arch/x86/boot/bzImage is ready

We’ll need to copy this image out of the container somewhere WSL can access it:

# Modify your destination path to something suitable, I'm just using my windows home directory
docker cp wsl-kernel-builder:/kernel/arch/x86/boot/bzImage /mnt/c/Users/mprasil/bzImage

You can now exit out of your build docker container. Because we used --rm to run it, docker will remove container automatically to clean up all the mess.

Use the custom kernel image🔗

This can be set in the .wslconfig. You need to create a configuration in your windows home directory. (In my case that’s C:\Users\mprasil\.wslconfig):

[wsl2]
kernel=c:\\Users\\mprasil\\bzImage

To actually apply the changes, we need to restart the WSL distribution. To do that open command prompt (your windows command prompt, not WSL shell) and shutdown WSL:

wsl --shutdown

Verify that custom image is used🔗

Now we can start WSL shell as we usually do. WSL should pick up our custom configuration and use our kernel image. As a quick test, I’ll run the same debian bullseye image and try to run dockerd:

# Inside debian bullseye based container
$ dockerd
# ...snip...
INFO[2022-02-18T13:20:38.848409000Z] Daemon has completed initialization
INFO[2022-02-18T13:20:38.889224500Z] API listen on /var/run/docker.sock

Success 🎉

Undo the custom kernel changes🔗

If anything failed, or you no longer need to use custom image, all you have to do is remove the .wslconfig override. If you have no other configuration there, just removing the file and restarting WSL distribution should revert to the default kernel provided by the WSL distribution you’re using.