xesite/talks/nixos-pain-2021-11-10.markdown

---
title: How Nix and NixOS Get So Close to Perfect
date: 2021-11-10
slides_link: /static/talks/nixos-pain.pdf
tags:
 - nix
 - nixos
 - docker
 - packagingcon
---

# How Nix and NixOS Get So Close to Perfect

## Author's Note

Since my [last
talk](https://christine.website/talks/systemd-the-good-parts-2021-05-16) was so
well-recieved, I thought I'd do this talk on NixOS much in the same way as I did
in the systemd one. I have published this talk as a slide deck, a transcript
(thanks to massaged YouTube auto-captions) and finally as a YouTube recording of
the talk itself. I submitted this as a prerecorded talk to
[PackagingCon](https://packaging-con.org).

This format of talk takes so long to put together, but I feel the result is
worth it. I get to use skills that I rarely get to pull out of my hat. Enjoy!

## YouTube Embed

<center>
<iframe width="800" height="450" src="https://www.youtube.com/embed/qjq2wVEpSsA"
title="YouTube video player" frameborder="0" allow="accelerometer; autoplay;
clipboard-write; encrypted-media; gyroscope; picture-in-picture"
allowfullscreen></iframe>
</center>

YouTube link: [https://youtu.be/qjq2wVEpSsA](https://youtu.be/qjq2wVEpSsA)

## Transcript

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/001.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/001.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/001.d.png" alt="">
  </picture>
</center>

Hi, my name is Xe. Today I'm going to talk about Nix and NixOS. This is my
favorite Linux distribution and it's one of my favorite tools for building
software. However it has a lot of rough edges that make it hard to learn and
make it just not as perfect as it could be. In this talk I'm going to go over a
lot of what makes it great and what I'd love to see make it even better.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/002.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/002.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/002.d.png" alt="">
  </picture>
</center>

As I said my name is Xe. I write a lot about Nix and NixOS and use it a lot
personally and soon professionally. As a disclaimer, this presentation may
contain opinions. These opinions are my own and not necessarily the opinion of
my employer. I do not intend ill will to any people or their work in this
presentation, and I want this to be better because I am passionate about these
tools. I believe that this is the best and obvious choice to do things.

The qr code on the slide links to my website christine.website. I'll have the
talk on the website within the same day the presentation you're watching right
now.

### Why NixOS is Great

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/003.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/003.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/003.d.png" alt="">
  </picture>
</center>

Let's start with why NixOS is great. NixOS is great because it lets you pick
from cookie cutter templates to make a server do exactly what you want. It
builds on the shoulders of giants to make it easy and effective to make your
servers built purpose. As an example here's a little NixOS module that enables
nginx and postgres on a server.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/004.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/004.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/004.d.png" alt="">
  </picture>
</center>

That's it! This modularity also extends to your own custom modules. I've done
some write-ups on how to write these custom modules but here's an example for my
gemini server.

Finally, one of the best parts of NixOS is that it makes it hard to do something
the wrong way. You can't just hack up a systemd unit on the fly. You need to do
it the right way in configuration management. This makes it easier for you to
ensure servers aren't being tampered with without going through a review process
and so you can go back to a project in six months and still have some idea on
how it's supposed to run on a computer.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/006.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/006.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/006.d.png" alt="">
  </picture>
</center>

However one of the biggest things in my book is the fact that NixOS lets you
undo configuration changes. Worst case you might need to reboot into an older
config; but in general if you mess something up you can go back. This is a
lifesaver, especially when you mess up network configuration.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/007.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/007.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/007.d.png" alt="">
  </picture>
</center>

So this sounds great and all but you might be thinking "there's a catch, right?"
There is a catch, it is hard to learn. The tooling and documentation are not the
best and they are the most important parts of the stack that you deal with when
you're learning it.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/008.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/008.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/008.d.png" alt="">
  </picture>
</center>

This is a little comic showing a rocket-powered wagon which is kind of what
NixOS can feel like at times. One of the bigger tooling issues is somewhat
technical somewhat social right now the Nix universe is in the middle of
switching to a new hermetic view of the world they call "flakes". Flakes has a
lot of differences between classic Nix and it makes a lot of techniques and
configuration non-transferable between the two. It has effectively soft split
the community between people that use flakes and people that don't use flakes. I
personally don't use flakes because I haven't seen good arguments as for why I
should.

Nix the language can look a bit like a combination of haskell and bash in ways
that are kind of deceiving to people that don't have solid experience with
haskell or other functional programming languages. This is a little bit of code
that breaks a host colon porn thing into just the port number so that you can
add it to a firewall rule. Additionally it also checks if you have tls enabled
with the http certification for Let's Encrypt and adds port 80 for that. If you
aren't really familiar with Haskell or other functional languages (and without
the 14 plus lines of comments explaining what's going on from the place i pulled
this from), it's going to be difficult to understand what's going on.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/010.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/010.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/010.d.png" alt="">
  </picture>
</center>

Another annoying part is that Nix the package manager, Nix the language and
NixOS the operating system all have very similar names and kind of semantically
override. I have created a handy diagram that maps out the relationships between
them and here it is:

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/012.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/012.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/012.d.png" alt="">
  </picture>
</center>

Nix the language is not nix the package manager, even though Nix the package
manager uses Nix the language in order to do things. NixOS the os is not Nix the
package manager even though it uses Nix the package manager to manage packages.
NixOS the os is not Nix the language even though NixOS uses Nix to configure
itself. The overall relationship is similar to the holy trinity or javascript
equality rules and can be a bit deceiving to learn at first.

Another paper cut is that Nix does have a REPL so that you can hack up things
quickly and so you can get to learn the language a bit better. However, the REPL
can take different syntax than you can put in files. If I want to declare a
variable like foo = "bar" and then use it somewhere in the REPL, I have to do
foo = "bar" without a semicolon and without a let. That can be very annoying at
first because you can hack up something in a REPL and then your instinct is to
go and paste it into a file; but you need to edit it a little bit and it's not
entirely obvious at first.

NixOS has modules to configure itself, however these modules are only as
flexible as they are written to be. If someone doesn't allow you to do a certain
type of configuration to a module to a program that's behind an NixOS module,
you just can't do it without fixing the module. These sort of make them more
like templates instead of functions for reaching a desired system state.
However, this will get you most of the way there (but when you get into very
complicated setups it can get challenging).

Adding on to this, most of the modules in the standard set that are shipped with
NixOS are not documented in the NixOS manual, including nginx. There's a search
site that lets you query the list of options in the standard set, however if
you're not entirely sure what you're doing there's not always a good template to
start from when your needs change beyond what the NixOS module is doing. You can
actually monkey patch it, however you have to monkey patch the side effects of
the modules rather than monkey patching the module itself. As an example of an
obscure module let's look at WeeChat. WeeChat is an irc client (it's the one I
use) and NixOS has the ability to manage WeeChat for you.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/016.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/016.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/016.d.png" alt="">
  </picture>
</center>

This is all the documentation for the WeeChat module. All of these things expand
out and have more detail, but these are all the settings that you have directly.
Another thing about this module is that it's great for running one instance of
Weechat, but if you want to run multiple copies of it or like a community shell
box you either need to write your own NixOS module that allows you to do that or
use NixOS containers to do that. At that point why not use Docker?

Another huge paper cut is with disclosure and vulnerability detection for
security reasons. Something important for both standard production and
certification is the ability to answer the question "how do I know a server is
patched against a certain vulnerability?" There are a lot of inherent advantages
to NixOS that would make this a lot easier however there is not really a good
way to do it. There are not regular communications about security
vulnerabilities. This can make certification difficult.

Another annoying thing is backports. There are no hard rules on what is to be
backported and what is not to be backported which means that packages in stable
branches likely will bitrot from what the upstream intended. Most of the time
you can run NixOS unstable to work around this, but it may not be the best idea
to run the unstable branch in production.

Also if you want to configure PAM to do something special such as send a slack
message to some channel whenever someone logs into a machine or runs a sudo
command and that option is not already in the pam options in nixpkgs, you're
basically doomed because pam is not very configurable on NixOS. The PAM modules
offer you the ability to enable things like TOTP two factor auth, but that's
about it.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/022.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/022.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/022.d.png" alt="">
  </picture>
</center>

Another annoyance comes when you're trying to deploy software on a production
cluster using Nix itself. There's not really a good tool in the standard Nix
tool set for this but there are tools like NixOps and Morph. These allow you to
describe the state of your entire fleet of computers with the same NixOS module
syntax that you use for local machines. However the documentation for these is
lacking. NixOps does have a fairly decent manual and the documentation for Morph
is a bunch of example configuration files. I have a little screenshot of part of
my morph configuration for my home lab. This manages the NixOS machine under my
desk. There are other options than using NixOps and Morph however these are the
more Nix native approaches to do it.

Because NixOps and Morph aren't very documented it makes it an annoying catch-22
situation where learning NixOps and Morph requires you to already know how
NixOps and Morph work, which can make it hard to get started from scratch. This
is part of the reason why I make all of my NixOS configs public on GitHub. It
lets people have something to go off of when they're trying to figure out how to
do more complicated things. Without publishing my configs on GitHub I would be
afraid that people would get incredibly lost (like I was when i was trying to
figure it all out).

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/024.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/024.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/024.d.png" alt="">
  </picture>
</center>

As an example of things that just got me totally lost let's talk about keys.
NixOps and Morph use the term "key" where other ecosystems would use the term
"secret". They basically allow you to have values that aren't managed in your
git repo for things like database credentials, AWS credentials or other api
keys. Normally you want to have things so that this service (for example, this
service that I wrote for myself called mi) depends on the keys for it being
present. In NixOps and Morph what you're supposed to do is you're supposed to
make a secret for that. It will create a systemd service and then you sequence
the systemd job to start after the key. However, the documentation doesn't
really make it clear on how to do this; and I had to figure this out by
searching GitHub for NixOS code, praying someone already figured it out and
made it open source.

Another annoyance is that you can't pull values from other machines in your
cluster. If you have a VPN with a dynamic ip address and you want to pull that
ip address to use in various bits of configuration, You're going to have to hard
code it somewhere; which means that it's a bit more difficult to do things
dynamically. In comparison when using something like Ansible it is trivial to
pull this information off of a machine with its fact system.

### A Vision of A Better Place

Finally let's talk about what I think could make all of this better easier to
learn (and a much more obvious choice for production). These are my ideas. They
may not entirely work in the real world but in an ideal world this is what I'd
love to see.

First of all I would love to see the documentation being the strongest part of
the NixOS ecosystem. Documentation is the difference between understanding
something and not understanding something. You generally have to understand
something to be able to use it productively. In general, the standard
documentation should cover how to get started, what you can do, and detailed
documentation on every single thing that ships with NixOS by default. There
should be no module in the library of modules without documentation on how to
use it and an example or two of where you'd use some of the weirder options.


<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/028.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/028.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/028.d.png" alt="">
  </picture>
</center>

Error messages are also critical for understanding what's going on. Here is an
example of an error message that i have encountered in Nix and NixOS a lot of
times that just has utterly baffled me every time. No adding the --show-trace
flag does not show more detailed location information. In this case I got it by
sequencing a package import in the wrong place in a way that didn't seem obvious
to me, but without better error messages you you just have no idea what's going
on.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/029.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/029.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/029.d.png" alt="">
  </picture>
</center>

Let's consider some examples of better error messages I've seen around in other
projects. Here's Rest and Elm. In this case Rust is saying that "I don't know
what you're doing with this value, you're trying to return something from an if
statement but then you're just totally ignoring it" and in Elm List.nap is used
in place of List.map. In both of them the compiler tries to work with you to
help you figure out what went wrong. Another great thing about what Rust does is
that it has error codes that you can google for when you have something going
wrong so that you can more easily understand what you're doing wrong and how to
learn how to not do it in the future.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/030.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/030.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/030.d.png" alt="">
  </picture>
</center>

This is being worked on, here's an example of what the better error messages
look like. I just wish this was here sooner.

[NOTE: this was true at the time of recording (late October 2021), but it has
since landed with Nix 2.4. Good work, Nix team!](conversation://Mara/hacker)

Another way that things could get a lot better is by taking advantage of
language specific package managers to automatically figure out what to do
instead of having to fight them against what they are doing. There are tools
that automate this but ideally I'd like to see this just automatically happen
using import from derivation or something like that. As it is right now
packaging things like Go, Node and other things are really iterative and
annoyingly non-trivial.

Maybe it would be better to have modules act kind of more like functions than
templates. It would be nice to have modules return something that you can splice
into your configuration instead of the modules being enabling something into
your configuration so that you can manually monkey patch things and run multiple
instances of a service if you wanted to. You can work around this (like I
mentioned) but it would be better if this was a native solution.

<center>
  <picture>
    <source srcset="/static/talks/nixos-pain/033.d.avif" type="image/avif">
    <source srcset="/static/talks/nixos-pain/033.d.webp" type="image/webp">
    <img src="/static/talks/nixos-pain/033.d.png" alt="">
  </picture>
</center>

And that about wraps it up. NixOS is actually pretty great it's just really
frustrating to get started with. I really wish it was easier but right now it
just isn't. If you have any questions about this talk please feel free to ping
me on twitter or some contact method on my website. I love answering these
questions. I'll stick around in the chat for a bit and answer questions if you
want to ask them there. Thank you, stay safe and be well.