--- title: "Nix Flakes: Packages and How to Use Them" date: 2022-02-27 tags: - nix - nixos - docker - systemd series: nix-flakes vod: twitch: https://www.twitch.tv/videos/1409855764 youtube: https://youtu.be/eUFBD-6yAWQ ---
[Nix flakes are still marked as experimental. This documentation has a small chance of bitrotting. I will make every attempt to update it if things change, however flakes have been fairly consistent for a few years now.](conversation://Cadey/coffee)
[What is a package? I've seen this term thrown around with phrases like "Nix is a package manager" or "language-specific package manager" or even "download the debian package and install it", but it's not really clear to me what a package is. What is a package?](conversation://Mara/hmm) A package is a bundle of files. These files could be program executables, resources such as stylesheets or images, or even a container image. Most of the time you don't deal with packages directly and instead you use a _package manager_ (a program whose sole goal in life is to deal with packages) to do actions for you. This post is going to cover how to define packages in Nix and how Nix flakes let you manage multiple packages per project more easily. ## What is a Package? In Nix, you build packages by creating _derivations_ that define the build steps and associated inputs (such as the compiler) to end up with the resulting outputs (derivation being the product of deriving something). Consider a package like this: ```nix # hello-shell.nix with import { }; stdenv.mkDerivation { name = "hello-HEAD"; src = ./.; installPhase = '' echo "Hello" > $out ''; } ``` Then we can build this package with `nix-build hello-shell.nix` and a `result` symlink will show up in your current working directory. Then you can view what it says with `cat`: ```console $ cat ./result Hello ``` This is all it takes to make a Nix package. You need to name the package, give it input source code somehow, and potentially give it build instructions. Everything else we'll cover today will build on top of this. Let's look back at the Go [example package](https://github.com/Xe/gohello/blob/caf54cdff7d8dd9bd9df4b3b783a72fe75c9a11e/flake.nix#L31-L54) I walked us through in [the last post](https://christine.website/blog/nix-flakes-1-2022-02-21): ```nix # ... packages = forAllSystems (system: let pkgs = nixpkgsFor.${system}; in { go-hello = pkgs.buildGoModule { pname = "go-hello"; inherit version; # In 'nix develop', we don't need a copy of the source tree # in the Nix store. src = ./.; # This hash locks the dependencies of this package. It is # necessary because of how Go requires network access to resolve # VCS. See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for # details. Normally one can build with a fake sha256 and rely on native Go # mechanisms to tell you what the hash should be or determine what # it should be "out-of-band" with other tooling (eg. gomod2nix). # To begin with it is recommended to set this, but one must # remeber to bump this hash when your dependencies change. vendorSha256 = "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo="; }; }); # ... ``` This uses a different builder, one called [`pkgs.buildGoModule`](https://nixos.org/manual/nixpkgs/stable/#ssec-language-go). This is like the `stdenv.mkDerivation` builder, except it is explicitly made to handle Go projects. There are some other flags that you can set in `buildGoModule` that can be useful. You can see examples in the NixOS manual page [here](https://nixos.org/manual/nixpkgs/stable/#ssec-language-go). Another useful builder is [Naersk](https://github.com/nix-community/naersk). Naersk will automatically derive build instructions for Rust projects using the `Cargo.toml` and `Cargo.lock` files. This means that your build step can look as small as this: ```nix naersk-lib.buildPackage ./. ``` [You can think of these builders as templates for doing larger builds. This is kinda like the ONBUILD Dockerfile instruction, but it isn't limited to Docker. The main difference is that Nix builds are more like functions (inputs and outputs) and Docker builds focus on the individual commands you run to get the result you want. Both eventually compile down to shell commands anyways!](conversation://Mara/hacker) ## A More Useful Package This "hello world" program isn't very useful on its own, however we can use it as the basis for making something a bit more useful. I have made a template for a "Hello world" HTTP server [here](https://github.com/Xe/templates/tree/main/go-web-server). Let's make a new folder for it and then initialize it: [If you want to make your own templates, see how to do that here.](conversation://Mara/hacker) ```shell mkdir -p ~/tmp/gohello-http cd ~/tmp/gohello-http git init nix flake init -t github:Xe/templates#go-web-server ``` [You may see a message from direnv about needing to approve its content. This will use Nix flake's cached interpreter to give you all the advantages of something like Lorri without having to install Lorri.](conversation://Mara/hacker) Then make an initial commit and run it: ```shell git add . git commit -sm "initial commit" nix build ./result/bin/web-server ``` [Why are you using `git add .` everywhere? Shouldn't the files be picked up implicitly?](conversation://Mara/hmm) [Not always. Nix flakes only deals with files that are tracked by git when you use it in a git repository. This means that if you want the changes to be observed by Nix, you need to add them to git somehow. `git add` is good enough for this.](conversation://Cadey/enby) Or you can run it directly with `nix run`: ```shell nix run ``` ## Docker Images Most of the time you will build software with Nix, however that doesn't stop you from building things like Docker images with Nix. Remember that you can have the output of any shell commands be run in a Nix build (the only catch is that they can't access the internet directly), so you can build a Docker image out of that web server template by defining another package: ```nix # flake.nix # after defaultPackage packages = { docker = let web = self.defaultPackage.${system}; in pkgs.dockerTools.buildLayeredImage { name = web.pname; tag = web.version; contents = [ web ]; config = { Cmd = [ "/bin/web-server" ]; WorkingDir = "/"; }; }; }; ``` This will build a Docker image with the web-server binary in it. To build it, run these commands: ```shell git add . nix build .#docker ``` [What's with that last argument to `nix build`, won't that be read as a shell comment?](conversation://Mara/hmm) [It's a reference to the package in the flake. Shell only parses comments when the `#` is the first character after whitespace, so this is more of a URL fragment than a comment. It's telling `nix build` to build the flake package named `docker`.](conversation://Cadey/enby) It will put the resulting docker image in `./result`. To load it into docker use the following command: ```console $ docker load < result Loaded image: web-server:20220227 ``` [Your image tag may differ depending on when you build this image. This is deterministic because that date is derived from the date that the current git commit was made.](conversation://Mara/happy) Then you can run it with `docker run`: ```shell docker run -itp 3031:3031 web-server:20220227 ``` Then poke it with curl: ```console $ curl http://[::]:3031 hello from nix! ``` You can push this image to the Docker hub like any other image. Another cool thing about this is that when you update the program, it'll only actually load the images that changed. Let's edit the hello world message: ```go http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "hello from nix building a docker image!") }) ``` And then re-build and load it into Docker: ```shell git add . nix build .#docker docker load < result ``` [Woah, when I did that it only updated 2 layers. The first time that I loaded it there were something like 7 layers. What's up with that?](conversation://Mara/hmm) [When you use `buildLayeredImage`, each Nix package that contributes to the image gets put in its own Docker layer. This means that only the things that have changed actually need to be considered, so when you push an updated image to another machine, the only things that will actually be pushed are the application binary and the symlink farm pointing to the `contents` of the Docker image.](conversation://Cadey/enby) ## systemd Portable Services
[EDIT(20220227 17:24): It seems that these are even more experimental than I thought. systemd Portable Services don't seem to work properly with the `StateDirectory` and `CacheDirectory` directives unless you are running a git HEAD version of systemd. Maybe you should wait a few years before trying to use them for anything serious. By then most of the kinks should be worked out.](conversation://Cadey/coffee)
[systemd Portable Services](https://systemd.io/PORTABLE_SERVICES/) function like Docker, but they work at the systemd level and allow you to integrate into systemd instead of running on the side of it. This gives you access to systemd's readiness signaling, logging pipeline and dependency graph so that you can integrate like a native service. They are like containers, but without a lot of the headaches around networking, stateful storage and logging. They are just systemd services at their core. [These are kinda like Ubuntu's Snaps or Flatpaks, but they operate purely at the system level and are focused at providing things for system services instead of user-facing applications. Ubuntu's Snaps do let you create system services, but they are basically exclusively used on Ubuntu. systemd Portable Services let you target more than just Ubuntu. In the next few years with more releases of systemd, Portable Services should be easier to use and will be more integrated with the system than Docker is.](conversation://Mara/hacker) There is currently an [open pull request](https://systemd.io/PORTABLE_SERVICES/) for adding Portable Service building support to nixpkgs, however we can mess around with it today thanks to [my portable-svc overlay](https://tulpa.dev/cadey/portable-svc) that copies in the contents of that pull request. [In Nix, an overlay is a set of additional packages or functions that is put on top of nixpkgs. This overlay defines the `portableService` function that is needed to build portable services.](conversation://Mara/hacker) To make this into a portable service, first we need to add my overlay to the flake inputs: ```nix # flake.nix inputs = { nixpkgs.url = "nixpkgs/nixos-unstable"; utils.url = "github:numtide/flake-utils"; portable-svc.url = "git+https://tulpa.dev/cadey/portable-svc.git?ref=main"; }; ``` Then add it as an argument to the `outputs` function: ```nix outputs = { self, nixpkgs, utils, portable-svc }: ``` And then change how we are importing the `pkgs` variable. The `pkgs` variable we're currently using is imported like this: ```nix let pkgs = nixpkgs.legacyPackages.${system}; ``` This works, however there isn't a way to specify an overlay into this. We need to change this into a manual import of nixpkgs with the overlay specified, like this: ```nix let pkgs = import nixpkgs { overlays = [ portable-svc.overlay ]; inherit system; }; ``` This will let us use the `portableService` function in Nix package definitions. Next we need to make a systemd service unit for the web server. The exact path to the program binary can and will change with every build, so it would be good to have this templated. Make a folder called `systemd`: ```shell mkdir systemd ``` And put the following contents in `systemd/web-server.service.in`: ```systemd [Unit] Description=A web service [Service] DynamicUser=yes ExecStart=@web@/bin/web-server [Install] WantedBy=multi-user.target ``` Then under the docker package definition, add the package that will template out the systemd unit: ```nix web-service = pkgs.substituteAll { name = "web-server.service"; src = ./systemd/web-server.service.in; web = self.defaultPackage.${system}; }; ``` You can build it with `nix build .#web-service`, the output will look something like this: ```systemd [Unit] Description=A web service [Service] DynamicUser=yes ExecStart=/nix/store/yl863jm907wfr7gq9j0c4bd3d4bdc4vp-web-server-20220227/bin/web-server [Install] WantedBy=multi-user.target ``` [The `@web@` in the template was replaced with the nix store path for the web server!](conversation://Mara/happy) Then you can add the bit that builds the portable service: ```nix portable = let web = self.defaultPackage.${system}; in pkgs.portableService { inherit (web) version; name = web.pname; description = "A web server"; units = [ self.packages.${system}.web-service ]; }; ``` Then you can build it with `nix build`: ```shell nix build .#portable ``` And then take a look at `./result`: ```console $ file $(readlink ./result) /nix/store/1da6b90i75n03kqlzzfdwxii0j0bzxaf-web-server_20220227.raw: Squashfs filesystem, little endian, version 4.0, xz compressed, 9555806 bytes, 2010 inodes, blocksize: 1048576 bytes, created: Tue Jan 1 00:00:00 1980 ```
At the time of writing this article, the most reliable way to test portable services is to use Arch Linux. So you could use something like [waifud](https://github.com/Xe/waifud) to spin up an Arch Linux VM: ```console $ waifuctl create -d arch -h logos -s 20 created instance jangmo-o on logos jangmo-o: running jangmo-o: init: IP address: 10.77.129.208 ``` Then copy it over with `scp`: ```console $ scp (readlink ./result) xe@10.77.129.208:web-server_20220227.raw ```
Then you can use `portablectl` to attach it to the system: ```console $ sudo portablectl attach ./web-server_20220227.raw [...] Created symlink /etc/portables/web-server_20220227.raw → /home/xe/web-server_20220227.raw. ``` And then start it like any systemd service: ```console $ sudo systemctl start web-server ``` [If you want the service to start automatically, add `--enable --now` to the `portablectl attach` command. That will enable the service in systemd and then start it, like when you run `systemctl enable --now something.service`.](conversation://Mara/hacker) And then inspect the service's status with `systemctl`: ```console $ sudo systemctl status web-server ● web-server.service - A web service Loaded: loaded (/etc/systemd/system.attached/web-server.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system.attached/web-server.service.d └─10-profile.conf, 20-portable.conf Active: active (running) since Sun 2022-02-27 18:21:01 UTC; 20s ago Main PID: 960 (web-server) Tasks: 5 (limit: 513) Memory: 8.1M CPU: 189ms CGroup: /system.slice/web-server.service └─960 /nix/store/yl863jm907wfr7gq9j0c4bd3d4bdc4vp-web-server-20220227/bin/web-server Feb 27 18:21:01 jangmo-o systemd[1]: Started A web service. Feb 27 18:21:01 jangmo-o web-server[960]: 2022/02/27 18:21:01 listening for HTTP on :3031 ``` And finally poke it with curl: ```console $ curl http://[::]:3031 hello from nix building a docker image! ``` [That ain't Docker, chief!](conversation://Numa/delet) [I know, I know, I didn't adjust the message from when I wrote the Docker example.](conversation://Cadey/facepalm) And then you can change the handler to something like: ```go http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "PORTABLE=%s\n", os.Getenv("PORTABLE")) }) ``` Rebuild the image with `nix build`: ```console $ git add . $ nix build .#portable ``` Copy it to the arch VM with `scp`: ```console $ scp (readlink ./result) xe@10.77.129.208:web-server_20220227.raw ``` And finally run `portablectl reattach` to upgrade it: ```console $ sudo portablectl reattach --now ./web-server_20220227.raw Queued /org/freedesktop/systemd1/job/858 to call RestartUnit on portable service web-server.service. ``` Then you can see that it restarted the unit with `systemctl status`: ```console $ sudo systemctl status web-server ● web-server.service - A web service Loaded: loaded (/etc/systemd/system.attached/web-server.service; disabled; vendor preset: disabled) Drop-In: /etc/systemd/system.attached/web-server.service.d └─10-profile.conf, 20-portable.conf Active: active (running) since Sun 2022-02-27 18:30:04 UTC; 37s ago Main PID: 1074 (web-server) Tasks: 6 (limit: 513) Memory: 8.1M CPU: 182ms CGroup: /system.slice/web-server.service └─1074 /nix/store/j1mfz3ydn13qmvcgrql33zi0dwb3x7dk-web-server-20220227/bin/web-server Feb 27 18:30:04 jangmo-o systemd[1]: Started A web service. Feb 27 18:30:04 jangmo-o web-server[1074]: 2022/02/27 18:30:04 listening for HTTP on :3031 ``` And finally poke it with curl: ```console $ curl http://[::]:3031 PORTABLE=web-server_20220227.raw ``` And there you go! Nix created a portable system service, we spawned it on a newly created Arch Linux VM and then were able to update it so that we could replace the message. --- Nix builds can do more than just turn code into software. They can create Docker images, Portable Services, virtual machine images and more. The only real limit is what you can imagine. Flakes make it easier to pull in and munge about packages. Before flakes you'd need to have a few `.nix` files like `docker.nix` for the docker image and `portable.nix` for the portable service. You'd also have to pull in something like [Niv](https://github.com/nmattia/niv) to make sure everything uses the same version of nixpkgs, and even then it's opt-in, not opt-out, so it's easy to mess things up and not use the pinned versions of things. Flakes make that explicit behavior implicit, so you can't bring in dependencies you aren't aware of. If you want to see the code repo I developed while writing this post, see [cadey/gohello-http](https://tulpa.dev/cadey/gohello-http) on my git server. Thanks for reading!