site/blog/thoughts-on-nix-2020-01-28.markdown

title	date	tags
Thoughts on Nix	2020-01-28	nix packaging dependencies

I don't really know how I feel about Nix. It's a functional package manager that's designed to help with dependency hell. It also lets you define packages using Nix, which is an identically named yet separate thing. Nix has untyped expressions that help you build packages like this:

{ stdenv, fetchurl, perl }:

stdenv.mkDerivation {
  name = "hello-2.1.1";
  builder = ./builder.sh;
  src = fetchurl {
    url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz;
    sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465";
  };
  inherit perl;
}

In theory, this is great. It's obvious what needs to be done to the system in order for the "hello, world" package and what it depends on (in this case it depends on only the standard environment because there's no additional dependencies specified), to the point that this approach lets you avoid all major forms of DLL hell, while at the same time creating its own form of hell: nixpkgs, or the main package source of Nix.

Now, you may ask, how do you get that hash? Try and build the package with an obviously false hash and use the correct one from the output of the build command! That seems safe!

Let's say you have a modern app that has dependencies with npm, Go and Elm. Let's focus on the Go side for now. How would we do that when using Go modules?

{ pkgs ? import <nixpkgs> { } }:
let
x = buildGoModule rec {
  name = "Xe-x-${version}";
  version = "1.2.3";

  src = fetchFromGitHub {
    owner = "Xe";
    repo = "x";
    rev = "v${version}";
    sha256 = "0m2fzpqxk7hrbxsgqplkg7h2p7gv6s1miymv3gvw0cz039skag0s";
  };

  modSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j"; 

  subPackages = [ "." ]; 
}

in {
  x = x;
}

And this will fetch and build the entirety of my x repo into a single massive package that includes everything. Let's say I want to break it up into multiple packages so that I can install only one or two parts of it, such as my license command:

Let's make a function called gomod.nix that includes everything to build the go modules:

# gomod.nix
pkgs: repo: modSha256: attrs:
  with pkgs;
  let defaultAttrs = {
    src = repo;
    modSha256 = modSha256;
  };

  in buildGoModule (defaultAttrs // attrs)

And then let's invoke this with a few of the commands in there:

{ pkgs ? import <nixpkgs> { } }:
let
  stdenv = pkgs.stdenv;
  version = "1.2.3";
  repo = pkgs.fetchFromGitHub {
    owner = "Xe";
    repo = "x";
    rev = "v${version}";
    sha256 = "0m2fzpqxk7hrbxsgqplkg7h2p7gv6s1miymv3gvw0cz039skag0s";
  };

  modSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j";
  mk = import ./gomod.nix pkgs repo modSha256;

  appsluggr = mk {
    name = "appsluggr";
    version = version;
    subPackages = [ "cmd/appsluggr" ];
  };

  johaus = mk {
    name = "johaus";
    version = version;
    subPackages = [ "cmd/johaus" ];
  };

  license = mk {
    name = "license";
    version = version;
    subPackages = [ "cmd/license" ];
  };

  prefix = mk {
    name = "prefix";
    version = version;
    subPackages = [ "cmd/prefix" ];
  };

in {
  appsluggr = appsluggr;
  johaus = johaus;
  license = license;
  prefix = prefix;
}

And when we build this, we notice that ALL of the dependencies for my x repo (at least a hundred because it's got a lot of stuff in there) are downloaded FOUR TIMES, even though they don't change between them. I could avoid this by making each dependency its own Nix package, but that's not a productive use of my time.

Add on having to do this for the Node dependencies, and the Elm dependencies and this is at least 200 if not more packages needed for my relatively simple CRUD app that has creative choices in technology.

Oh, even better, the build directory isn't writable. So when your third-tier dependency has a generation step that assumes the build directory is writable, you suddenly need to become an expert in how that tool works so you can shunt it writing its files to another place. And then you need to make sure those files don't end up places they shouldn't be, lest you fill your disk with unneeded duplicate node_modules folders that really shouldn't be there in the first place (but are there because you gave up).

Then you need to make sure that works on another machine, because even though Nix itself is "functionally pure" (save the heat generated by the CPU executing your cloud-native, multitenant parallel adding service) this is a PACKAGE MANAGER. You know, the things that handle STATE, like FILES on the DISK. That's STATE. GLOBALLY MUTABLE STATE.

One of the main advantages of this approach is that the library dependencies of every project are easy to reproduce on other machines. Consider the ldd(1) (which shows the dynamic libraries associated with a program) output of ls on my Ubuntu system vs a package I installed from Nix:

$ ldd $(which ls)
        linux-vdso.so.1 (0x00007ffd2a79f000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f00f0e16000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f00f0a25000)
        libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f00f07b3000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f00f05af000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f00f1260000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f00f0390000)

All of these dependencies are managed by apt(8) and are supposedly reproducible on other Ubuntu systems. Compare this to the ldd(1) output of a Nix program:

$ ldd $(which dhall)
        linux-vdso.so.1 (0x00007fff0516a000)
        libm.so.6 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libm.so.6 (0x00007fc20ed8d000)
        libz.so.1 => /nix/store/a3q9zl42d0hmgwmgzwkxi5qd88055fh8-zlib-1.2.11/lib/libz.so.1 (0x00007fc20ed6e000)
        libncursesw.so.6 => /nix/store/24xdpjcg2bkn2virdabnpncx6f98kgfw-ncurses-6.1-20190112/lib/libncursesw.so.6 (0x00007fc20ec8c000)
        libpthread.so.0 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libpthread.so.0 (0x00007fc20ed4d000)
        librt.so.1 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/librt.so.1 (0x00007fc20ed43000)
        libutil.so.1 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libutil.so.1 (0x00007fc20ed3c000)
        libdl.so.2 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libdl.so.2 (0x00007fc20ed37000)
        libgmp.so.10 => /nix/store/4gmyxj5blhfbn6c7y3agxczrmsm2bhzv-gmp-6.1.2/lib/libgmp.so.10 (0x00007fc20ebf7000)
        libffi.so.7 => /nix/store/qa8wyi9pckq1d3853sgmcc61gs53g0d3-libffi-3.3/lib/libffi.so.7 (0x00007fc20ed2a000)
        libc.so.6 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libc.so.6 (0x00007fc20ea41000)
        /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc20ecfe000)

Each dynamic library dependency has its package hash in the folder path. This also means that the hash of its parent packages are present in there, which root all the way back to where/when its ultimate parent package was built. This makes Nix packages a kind of blockchain.

Nix also allows users to install their own packages into the global nix store at /nix. No, you can't change this, but you can symlink it to another place if you (like me) have a partition setup with / having less disk space than /home. You also need to set a special environment variable so Nix shuts up about you doing this. This is really fun on macOS Catalina where the root filesystem is read only. There is a workaround (that I had to trawl into the depths of Google page cache to get, because of course I did), but the Nix team themselves seem unaware of it.

So, to recap: Nix is an attempt at a radically different approach to package management. It assumes too much about the state of everything and puts odd demands on people as a result. Language-specific package managers can and will fight Nix unless they are explicitly designed to handle Nix's weirdness. As a side effect of making its package management system usable by normal users, it exposes the package manager database to corruption by any user mistake, curl2bash or malicious program on the system. All that functional purity uwu and statelessness can vanish into a puff of logic without warning.

But everything's immutable so that means it's okay right?

---

Based on this twitter thread but a LOT less sarcastic.