Keyboard meet head

Where my head meets keyboard..

On dishwashers and Nix

I have many unfinished blogposts about Nix. It is a very broad topic and it's hard not to get lost half-way through writing it. So I decided to try something different. Get a bit more personal and try to write about one of my weird obsessions and about a piece of technology that I learned to appreciate.

One is dishwasher and the other is Nix. And you'll be the judge which one is which. There surely is some kitchen sink joke in there, but let's just dive right in.

Dishwashers actually save water and energyπŸ”—

Did you know that to wash up by hand you use around 20L of water? In comparison, an efficient dishwasher uses just 9L to wash up to 15 place settings. -- ENER

I struggle to explain this to my parents or in-laws visiting us from abroad. Their "I'll just wash this one cup by hand" isn't actually helping even though they mean to be helpful. Sure, public (mains) water supply in Ireland is free. But that doesn't mean it's not valuable and hot water is also relatively expensive due to energy prices.

Washing by hand is just more wasteful. And it's not easy to get that point across. After couple cups washed like that, we end up using more water and energy than fully loaded dishwasher.

It's similar with Nix. A lot of people see it as major waste of effort. Spending hours on automating setup that takes minutes to do manually. I initially learned Nix out of curiosity and I didn't really expect any big ROI. Almost two years later I'm convinced I was wrong.

To give you an example: I had a low power PC on my LAN serving as AdGuard Home DNS server. This device died one day and I just had to wait a week to get replacement delivered. Being bombarded by ads didn't sound like good time to me, so I took one of the other mini PCs I had - this one is sometimes used as media player and reinstalled complete AdGuard system on it. When the replacement arrived I have re-installed all devices to serve their dedicated purpose.

This required reinstalling AdGuard system twice (temporary and permanent HW) and media player once. (back to its original purpose) And here's the point: It only took few minutes - and most of it hands off, waiting for Nix to finish after I ran:

sudo nixos-install --no-root-passwd --root /mnt --flake ".#carmille"

This deploys complete AdGuard system (that is including things like monitoring, log collection, VPN, SSH keys auth, customized shell, etc..) from scratch. All I had to do is tell nixos-install which of the systems in my NixOS configuration (in this case carmille*) should it deploy and point it to the disk partition it should use mounted at /mnt.

*In case you're curious, the system name is a nod to one of the earliest ethical hackers and true hero - RenΓ© Carmille

But what about the media player? On first boot it started with fully configured desktop environment, key mappings for my remote controller and welcoming me with Jellyfin login interface in full screen. The login prompt is the only giveaway that the system was completely reinstalled.

Previously, I had extensive Ansible configuration in place before. So extensive, it took longer to run the playbook than to reinstall complete Nix system from scratch. And Nix manages bigger part of the system. By far. All of it in fact.

Setting whole server from scratch makes the benefit more obvious, just like fully loaded dishwasher using less than 10L of water. But there are these little configuration changes. These metaphorical "cups". How do you disable annoying notification popups on media player system? It requires some effort to initially set that in NixOS.

# In case you're curious, for GNOME it's:
dconf.settings."org/gnome/desktop/notifications".show-banners = false;

But once you do that, it's done for good. When you replace the hardware or upgrade the systems, the configuration is going to be there as it should. I no longer end up with this tail of little things to configure after fresh reinstall weeks later. (Lock screen, screen blanking, startup apps, automatic suspend, etc..) All is ready on first boot and I can rely on that.

You don't need to pre-wash the dishesπŸ”—

A lot of people waste literal tons of water pre-washing dishes before putting them into the dishwasher. Most likely this is not necessary - the dishwasher usually already has pre-wash cycle. Just scrape off the big bits, your appliance will do the rest. This is also why you should put the washing powder into the dispenser if there's one. Some people just throw it in with the dishes and then complain about dishwasher not doing the job properly - chances are most of the active chemicals got washed away with initial pre-wash water.

Learn how to use your appliance properly and you can save yourself some effort and some money. And you even get better end results.

A common thing I see online is people "trying Nix" by using the graphical installer, booting into GNOME and perhaps installing a package. They look for some "store app" or use nix-env as one would apt-get and usually end up confused what is the fuss about or why even bother using this weird distro when it's just a bit more janky Arch.

The truth is, Nix requires some effort before you see the benefits. As someone said:

Many simple things are surprisingly hard in Nix, but many hard things are surprisingly simple.

And if you "try Nix" like any other regular distribution, you mostly see simple things that are harder. It takes some learning and effort before you realize you're doing some things with ease that would be extremely hard or pretty much impossible in more traditional system.

I've switched from iptables to nftables on a system with about dozen services (some with custom firewall rules) by changing one line in my config. One (not really necessary, but I played it safe) reboot later I was done.

What's more, right in the boot menu there was an entry I could choose to boot into system as it was before the change - as if it never happened - should anything go horribly wrong. Some distributions achieve rollback functionality by using filesystem snapshots. But it's like comparing git to an occasionally created backup on thumb drive. Sure, you can go back to some earlier state, but often enough you want something in between - keep most of the changes except one or two. It's just not the same.

In my opinion, low effort YouTube videos often do outsized disservice to NixOS, focusing on the superficial differences between distributions. More reasonable way to try Nix is probably to just use Nix on top of your current distribution. (again, normally hard thing that is easy with Nix)

Run nix-shell -p mc to be dropped into a shell where you have midnight commander available, type exit to go back to shell where mc does not exist. You can do this with most of the 120k packages available.

I have no Python binary in my $PATH, yet I'm using many Python-based applications. It's actually more involved to have Python available system wide. This is especially useful when I need to run very old software that requires Python version 2 or to run winbox - windows executable that requires Wine. Nix lets me do that without actually having Wine "fully" deployed or Python 2 getting in my way. Hard things easy.

With a bit of knowledge about your system, you can save yourself a lot of effort and often get much better results at the same time.

It does not take much effort to load dishwasher properly but it's well worth doingπŸ”—

Arguably, I'm somewhat of a dishwasher perfectionist. I take my time loading the plates in, go from smallest one, pick the order so that they sit neatly inside, making sure each still has some space around it. Approach the whole load as some sort of mini puzzle where the goal isn't to place each individual piece of tableware neatly but to end up with the whole dishwasher neatly organized. Sometimes it's just the Marie Kondo kind of joy of having things nicely arranged. But often times just rearranging the plates and cups actually allows me to put much more stuff inside and make the dishwasher even more efficient.

Like dishwasher load, our systems are frequently just thrown together in a hurry. We've embraced cattle over pet approach (without talking to actual farmers to see how they care about cattle) and pretty much accepted that duct taping stuff together with some bash and loose abstractions is the way to go.

Take Ansible. A pretty good configuration management system. Writing more involved logic in YAML can be tedious at times, but it's a good tool for many things. There is one thing that is hard to do in Ansible - idempotence. Far too many playbooks out there generate system change on each run even though system configuration isn't really changing. There is one even harder thing that most playbooks don't even bother implementing - to be able to undo all the changes made.

Consider this very basic ansible task:

- name: Install curl
  ansible.builtin.package:
    name: curl
    state: present

It sort of succeeds being idempotent, running it multiple times should not generate a change. What happens if we try to reverse the logic and make sure curl is removed, because our role no longer needs it?

- name: Remove curl
  ansible.builtin.package:
    name: curl
    state: absent

This seems to work in isolation. But if there's any other role that still needs (and installs) curl, we can end up in a situation where we re-install package repeatedly every time the playbook is executed.

When I used Ansible to manage my servers, I solved this problem by deploying each service in a separate LXC container. That way it was easy to remove services that I decided to deprecate - just nuke the virtual system. I could do something similar by operating small Kubernetes cluster and deploying workload on top of that. That would be different kind of overhead many people choose to accept.

The core of the issue isn't with Ansible or Kubernetes. It's the fact that you're trying to manage a system that is inherently not fully manageable. Kubernetes solves this problem by pretty much ignoring the base OS and deploying workload on top of itself. If - for example - your workload needs specific set of modules or boot-time configuration, this is your responsibility. Ansible tries its best to help you at task level, but there is no good way to manage the entire system (the set of roles) as one coherent unit.

How does this work in Nix on NixOS?

environment.systemPackages = [
  pkgs.curl
];

This tells Nix to deploy curl as "system" package. (that is to be available as a command in my system) It makes sure it is installed if not already present. Applying the above configuration twice won't do any extra work. This configuration is idempotent out of the box.

Also, I can add some other configuration like:

  systemd.services.fetch-url = {
    script = "${pkgs.curl}/bin/curl http://example.com";
  };

This creates Systemd service that will fetch some example URL with curl. These two configurations do not affect each other. If I remove curl from systemPackages, the service will still have curl available. But If I remove both, Nix knows that curl can be safely removed. In other words, NixOS is managed by Nix as a complete system. It solves the problem of what to add or remove by knowing what needs to be present. And it knows that before it even starts making changes - you can even compare configurations of two systems (or versions of the same system) that aren't installed anywhere.

And thanks to this control over entire system, I can neatly pack many services on the same system without any overhead, because I can effortlessly completely remove service just by removing its Nix configuration. Hard things easy.

Dishwasher tablets suckπŸ”—

I've mentioned the pre-wash cycle already. Some dishwashers even have separate (smaller) dispenser for this initial rinse phase. You can use it to add some washing powder and help your appliance to do most of the hard work early on. That way a lot of the grime leaves dishwasher with the initial pre-wash waste and the rest of the process can be done with much cleaner water. If you don't have any separate dispenser available, you can usually add a bit of washing powder anywhere inside the dishwasher compartment.

These days, there's an obvious problem with that - the dishwasher tablets or gel capsules. The war on washing powder happened without us noticing and it's long over. I am way too young to remember any of it, so I'm not sure on what promise were the tablets sold to us, but it certainly wasn't convenience or being more economic, because these are neither. We are forced to use way too much of the detergent for half-load cycle and there's no way to use these with pre-wash. What's worse, actual washing powder is pretty much impossible to buy. (in regular shops around here)

Maybe you can't quite see how this powder thing bothers me. Let me try some more relatable analogy: The Snaps

On Linux, my favorite log aggregation system, Loki can be installed as single binary that's about 60MB. Why is Snap version of Loki quite a bit bigger? Well it's obvious once we look at the contents:

# du -sh loki.snap/bin/* | sort -r
71M     loki/bin/promtail
54M     loki/bin/loki
48M     loki/bin/logcli
4.0K    loki/bin/loki.wrapper
4.0K    loki/bin/config-set
4.0K    loki/bin/config-get
4.0K    loki/bin/common

There's promtail and logcli binary packaged in as well. This makes perfect sense as these tools are frequently used together. Current Nix package of Loki has these bundled in as well:

# nix build nixpkgs#grafana-loki && ls result/bin
logcli  loki  loki-canary  lokitool  promtail

However Nix also leaves us the flexibility to decide what will be deployed:

minimal_loki = pkgs.grafana-loki.overrideAttrs (final: prev: {
  subPackages = [
    "cmd/loki"
  ];
});

And with that we get just the thing we asked for:

# nix build .#minimal_loki && ls result/bin
loki

This is of course a bit heavy-handed example that I used because it's easy to visualize. More typical use cases would be compiling the software with non-standard flags, using source code from forked repository, installing only subset of font variants or seamlessly using software that wasn't even packaged for NixOS.

Some things should be washed by hand and that is okayπŸ”—

Another thing people often do is put things that are not dishwasher-safe into dishwasher. This can damage both dishwasher and the thing being washed. I try to avoid this situation early on by buying stuff that is built to be washed in dishwasher, but sometimes that's simply not possible. And that's fine. Making conscious decision in specific cases and washing few bits manually is okay, there's no point being religious about it.

It's similar with NixOS. It makes no sense to force Nix configuration on everything. I still use docker to run many services to benefit from the container isolation, my Nix code is often in this perpetual MVP state. I adore other people's extensive Nix configurations as much as I adore other much more minimal setups. There is certain beauty in "just enough to move on" code out there.

I've made the mistake myself early on. I tried to be smart, prematurely making my setup very generic and abstract. Once I had good grasp on how Nix works, most of my code improvements were about removing unnecessary complexity.

Don't get discouraged by cathedrals some folks built, maybe you need just a simple hut and that is okay.

Dishwasher is actually simple device doing simple workπŸ”—

You add dirty plates and cleaning chemicals, dishwasher then does couple cycles of replacing and heating water and spraying it inside the compartment. That's pretty much it.

The actual chemicals used are more complicated, there are enzymes, bleaches, surfactants and so on, but for day-to-day use you don't need to know any of that. You just need to make sure to add enough of the chemicals and to put stuff inside neatly, exposing dirty surfaces to the water spray.

My experience with NixOS is similar. In many ways NixOS is some of the purest Linux-based distributions out there. There are no Snaps, Flatpaks and Containers with their overlays, portals and complicated cgroups - unless you actually want that. The file system hierarchy is bit unusual and there is so many symlinks everywhere, but there really is no major magic going on, no secret technology behind the scenes.

Just Nix.

There's no comment system on this page, but I do accept feedback. If you are interested in commenting, send me a message and I may publish your comments, in edited form.