diff --git a/blog/thoughts-on-nix-2020-01-28.markdown b/blog/thoughts-on-nix-2020-01-28.markdown new file mode 100644 index 0000000..5e5b9c0 --- /dev/null +++ b/blog/thoughts-on-nix-2020-01-28.markdown @@ -0,0 +1,244 @@ +--- +title: Thoughts on Nix +date: 2020-01-28 +tags: + - nix + - packaging + - dependencies +--- + +I don't really know how I feel about [Nix][nix]. It's a functional package +manager that's designed to help with dependency hell. It also lets you define +packages using [Nix][nixlang], which is an identically named yet separate thing. +Nix has _untyped_ expressions that help you build packages like this: + +[nix]: https://nixos.org/nix/ +[nixlang]: https://nixos.org/nix/manual/#chap-writing-nix-expressions + +```nix +{ stdenv, fetchurl, perl }: + +stdenv.mkDerivation { + name = "hello-2.1.1"; + builder = ./builder.sh; + src = fetchurl { + url = ftp://ftp.nluug.nl/pub/gnu/hello/hello-2.1.1.tar.gz; + sha256 = "1md7jsfd8pa45z73bz1kszpp01yw6x5ljkjk2hx7wl800any6465"; + }; + inherit perl; +} +``` + +In theory, this is great. It's obvious what needs to be done to the system in +order for the "hello, world" package and what it depends on (in this case it +depends on only the standard environment because there's no additional +dependencies specified), to the point that this approach lets you avoid all +major forms of [DLL hell][dllhell], while at the same time creating its own form +of hell: [nixpkgs][nixpkgs], or the main package source of Nix. + +[dllhell]: https://en.wikipedia.org/wiki/DLL_Hell +[nixpkgs]: https://nixos.org/nixpkgs/manual/ + +Now, you may ask, how do you get that hash? Try and build the package with an +obviously false hash and use the correct one from the output of the build +command! That seems safe! + +Let's say you have a modern app that has dependencies with npm, Go and Elm. +Let's focus on the Go side for now. How would we do that when using Go modules? + +```nix +{ pkgs ? import { } }: +let +x = buildGoModule rec { + name = "Xe-x-${version}"; + version = "1.2.3"; + + src = fetchFromGitHub { + owner = "Xe"; + repo = "x"; + rev = "v${version}"; + sha256 = "0m2fzpqxk7hrbxsgqplkg7h2p7gv6s1miymv3gvw0cz039skag0s"; + }; + + modSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j"; + + subPackages = [ "." ]; +} + +in { + x = x; +} +``` + +And this will fetch and build [the entirety of my `x` repo][Xex] into a single +massive package that includes _everything_. Let's say I want to break it up into +multiple packages so that I can install only one or two parts of it, such as my +[`license`][Xelicense] command: + +[Xex]: https://github.com/Xe/x +[Xelicense]: https://github.com/Xe/x/blob/master/cmd/license/main.go + +Let's make a function called `gomod.nix` that includes everything to build the +go modules: + +```nix +# gomod.nix +pkgs: repo: modSha256: attrs: + with pkgs; + let defaultAttrs = { + src = repo; + modSha256 = modSha256; + }; + + in buildGoModule (defaultAttrs // attrs) +``` + +And then let's invoke this with a few of the commands in there: + +```nix +{ pkgs ? import { } }: +let + stdenv = pkgs.stdenv; + version = "1.2.3"; + repo = pkgs.fetchFromGitHub { + owner = "Xe"; + repo = "x"; + rev = "v${version}"; + sha256 = "0m2fzpqxk7hrbxsgqplkg7h2p7gv6s1miymv3gvw0cz039skag0s"; + }; + + modSha256 = "1879j77k96684wi554rkjxydrj8g3hpp0kvxz03sd8dmwr3lh83j"; + mk = import ./gomod.nix pkgs repo modSha256; + + appsluggr = mk { + name = "appsluggr"; + version = version; + subPackages = [ "cmd/appsluggr" ]; + }; + + johaus = mk { + name = "johaus"; + version = version; + subPackages = [ "cmd/johaus" ]; + }; + + license = mk { + name = "license"; + version = version; + subPackages = [ "cmd/license" ]; + }; + + prefix = mk { + name = "prefix"; + version = version; + subPackages = [ "cmd/prefix" ]; + }; + +in { + appsluggr = appsluggr; + johaus = johaus; + license = license; + prefix = prefix; +} +``` + +And when we build this, we notice that ALL of the dependencies for my `x` repo +(at least a hundred because it's got a lot of stuff in there) are downloaded +_FOUR TIMES_, even though they don't change between them. I could avoid this by +making each dependency its own Nix package, but that's not a productive use of +my time. + +Add on having to do this for the Node dependencies, and the Elm dependencies and +this is at least 200 if not more packages needed for my relatively simple CRUD +app that has creative choices in technology. + +Oh, even better, the build directory isn't writable. So when your third-tier +dependency has a generation step that assumes the build directory is writable, +you suddenly need to become an expert in how that tool works so you can shunt it +writing its files to another place. And then you need to make sure those files +don't end up places they shouldn't be, lest you fill your disk with unneeded +duplicate node\_modules folders that really shouldn't be there in the first +place (but are there because you gave up). + +Then you need to make sure that works on another machine, because even though +Nix itself is "functionally pure" (save the heat generated by the CPU executing +your cloud-native, multitenant parallel adding service) this is a PACKAGE +MANAGER. You know, the things that handle STATE, like FILES on the DISK. That's +STATE. GLOBALLY MUTABLE STATE. + +One of the main advantages of this approach is that the library dependencies of +every project are easy to reproduce on other machines. Consider the +[`ldd(1)`][ldd1] (which shows the dynamic libraries associated with a program) +output of `ls` on my Ubuntu system vs a package I installed from Nix: + +[ldd1]: http://man7.org/linux/man-pages/man1/ldd.1.html + +```console +$ ldd $(which ls) + linux-vdso.so.1 (0x00007ffd2a79f000) + libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f00f0e16000) + libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f00f0a25000) + libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f00f07b3000) + libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f00f05af000) + /lib64/ld-linux-x86-64.so.2 (0x00007f00f1260000) + libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f00f0390000) +``` + +All of these dependencies are managed by [`apt(8)`][apt8] and are supposedly +reproducible on other Ubuntu systems. Compare this to the `ldd(1)` output of a +Nix program: + +[apt8]: http://manpages.ubuntu.com/manpages/bionic/man8/apt.8.html + +``` +$ ldd $(which dhall) + linux-vdso.so.1 (0x00007fff0516a000) + libm.so.6 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libm.so.6 (0x00007fc20ed8d000) + libz.so.1 => /nix/store/a3q9zl42d0hmgwmgzwkxi5qd88055fh8-zlib-1.2.11/lib/libz.so.1 (0x00007fc20ed6e000) + libncursesw.so.6 => /nix/store/24xdpjcg2bkn2virdabnpncx6f98kgfw-ncurses-6.1-20190112/lib/libncursesw.so.6 (0x00007fc20ec8c000) + libpthread.so.0 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libpthread.so.0 (0x00007fc20ed4d000) + librt.so.1 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/librt.so.1 (0x00007fc20ed43000) + libutil.so.1 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libutil.so.1 (0x00007fc20ed3c000) + libdl.so.2 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libdl.so.2 (0x00007fc20ed37000) + libgmp.so.10 => /nix/store/4gmyxj5blhfbn6c7y3agxczrmsm2bhzv-gmp-6.1.2/lib/libgmp.so.10 (0x00007fc20ebf7000) + libffi.so.7 => /nix/store/qa8wyi9pckq1d3853sgmcc61gs53g0d3-libffi-3.3/lib/libffi.so.7 (0x00007fc20ed2a000) + libc.so.6 => /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/libc.so.6 (0x00007fc20ea41000) + /nix/store/aag9d1y4wcddzzrpfmfp9lcmc7skd7jk-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007fc20ecfe000) +``` + +Each dynamic library dependency has its package hash in the folder path. This +also means that the hash of its parent packages are present in there, which root +all the way back to where/when its ultimate parent package was built. This makes +Nix packages a kind of blockchain. + +Nix also allows users to install their own packages into the _global_ nix store +at `/nix`. No, you can't change this, but you can symlink it to another place if +you (like me) have a partition setup with `/` having less disk space than +`/home`. You also need to set a special environment variable so Nix shuts up +about you doing this. This is _really fun_ on macOS Catalina where [the root +filesystem is read only][catalinareadonly]. There is a +[workaround][nixcatalinahack] (that I had to trawl into the depths of Google +page cache to get, because of course I did), but the [Nix team themselves seem +unaware of it][nixcatalinabug]. + +[catalinareadonly]: https://support.apple.com/en-ca/HT210650 +[nixcatalinahack]: https://webcache.googleusercontent.com/search?q=cache:lbaImO5JBJ4J:https://tutorials.technology/tutorials/using-nix-with-catalina.html+&cd=3&hl=en&ct=clnk&gl=ca +[nixcatalinabug]: https://github.com/NixOS/nix/issues/2925 + +So, to recap: Nix is an attempt at a radically different approach to package +management. It assumes too much about the state of everything and puts odd +demands on people as a result. Language-specific package managers can and will +fight Nix unless they are explicitly designed to handle Nix's weirdness. As a +side effect of making its package management system usable by normal users, it +exposes the package manager database to corruption by any user mistake, +curl2bash or malicious program on the system. All that functional purity uwu and +statelessness can vanish into a puff of logic without warning. + +[But everything's immutable so that means it's okay +right?](https://utcc.utoronto.ca/~cks/space/blog/tech/RealWorldIsMutable) + +--- + +[Based on this twitter +thread](https://twitter.com/theprincessxena/status/1221949146787209216?s=21) but +a LOT less sarcastic.