From 8fc69d4978ad5686d7149cc2032d44738633eafa Mon Sep 17 00:00:00 2001 From: Xe Iaso Date: Sun, 27 Feb 2022 19:12:53 +0000 Subject: [PATCH] nix flakes 2: fun with packages Signed-off-by: Xe Iaso --- blog/nix-flakes-2-2022-02-27.markdown | 570 ++++++++++++++++++++++++++ src/post/frontmatter.rs | 7 + templates/blogpost.rs.html | 4 + 3 files changed, 581 insertions(+) create mode 100644 blog/nix-flakes-2-2022-02-27.markdown diff --git a/blog/nix-flakes-2-2022-02-27.markdown b/blog/nix-flakes-2-2022-02-27.markdown new file mode 100644 index 0000000..f75ba98 --- /dev/null +++ b/blog/nix-flakes-2-2022-02-27.markdown @@ -0,0 +1,570 @@ +--- +title: "Nix Flakes: Packages and How to Use Them" +date: 2022-02-27 +tags: + - nix + - nixos + - docker + - systemd +series: nix-flakes +vod: + twitch: https://www.twitch.tv/videos/1409855764 + youtube: https://youtu.be/eUFBD-6yAWQ +--- + +
+ +[Nix flakes are still marked as experimental. This documentation has a small +chance of bitrotting. I will make every attempt to update it if things change, +however flakes have been fairly consistent for a few years +now.](conversation://Cadey/coffee) + +
+ +[What is a package? I've seen this term thrown around with phrases like "Nix is a +package manager" or "language-specific package manager" or even "download the +debian package and install it", but it's not really clear to me what a package +is. What is a package?](conversation://Mara/hmm) + +A package is a bundle of files. These files could be program executables, +resources such as stylesheets or images, or even a container image. Most of the +time you don't deal with packages directly and instead you use a _package +manager_ (a program whose sole goal in life is to deal with packages) to do +actions for you. This post is going to cover how to define packages in Nix and +how Nix flakes let you manage multiple packages per project more easily. + +## What is a Package? + +In Nix, you build packages by creating _derivations_ that define the build steps +and associated inputs (such as the compiler) to end up with the resulting +outputs (derivation being the product of deriving something). Consider a package +like this: + +```nix +# hello-shell.nix +with import { }; +stdenv.mkDerivation { + name = "hello-HEAD"; + src = ./.; + installPhase = '' + echo "Hello" > $out + ''; +} +``` + +Then we can build this package with `nix-build hello-shell.nix` and a `result` +symlink will show up in your current working directory. Then you can view what +it says with `cat`: + +```console +$ cat ./result +Hello +``` + +This is all it takes to make a Nix package. You need to name the package, give +it input source code somehow, and potentially give it build instructions. +Everything else we'll cover today will build on top of this. + +Let's look back at the Go [example +package](https://github.com/Xe/gohello/blob/caf54cdff7d8dd9bd9df4b3b783a72fe75c9a11e/flake.nix#L31-L54) +I walked us through in [the last +post](https://christine.website/blog/nix-flakes-1-2022-02-21): + +```nix +# ... +packages = forAllSystems (system: + let pkgs = nixpkgsFor.${system}; + in { + go-hello = pkgs.buildGoModule { + pname = "go-hello"; + inherit version; + # In 'nix develop', we don't need a copy of the source tree + # in the Nix store. + src = ./.; + + # This hash locks the dependencies of this package. It is + # necessary because of how Go requires network access to resolve + # VCS. See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for + # details. Normally one can build with a fake sha256 and rely on native Go + # mechanisms to tell you what the hash should be or determine what + # it should be "out-of-band" with other tooling (eg. gomod2nix). + # To begin with it is recommended to set this, but one must + # remeber to bump this hash when your dependencies change. + vendorSha256 = + "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo="; + }; + }); +# ... +``` + +This uses a different builder, one called +[`pkgs.buildGoModule`](https://nixos.org/manual/nixpkgs/stable/#ssec-language-go). +This is like the `stdenv.mkDerivation` builder, except it is explicitly made to +handle Go projects. There are some other flags that you can set in +`buildGoModule` that can be useful. You can see examples in the NixOS manual +page [here](https://nixos.org/manual/nixpkgs/stable/#ssec-language-go). + +Another useful builder is [Naersk](https://github.com/nix-community/naersk). +Naersk will automatically derive build instructions for Rust projects using the +`Cargo.toml` and `Cargo.lock` files. This means that your build step can look as +small as this: + +```nix +naersk-lib.buildPackage ./. +``` + +[You can think of these builders as templates for doing larger builds. This is +kinda like the ONBUILD +Dockerfile instruction, but it isn't limited to +Docker. The main difference is that Nix builds are more like functions (inputs +and outputs) and Docker builds focus on the individual commands you run to get +the result you want. Both eventually compile down to shell commands +anyways!](conversation://Mara/hacker) + +## A More Useful Package + +This "hello world" program isn't very useful on its own, however we can use it +as the basis for making something a bit more useful. I have made a template for +a "Hello world" HTTP server +[here](https://github.com/Xe/templates/tree/main/go-web-server). Let's make a +new folder for it and then initialize it: + +[If you want to make your own templates, see how to do that here.](conversation://Mara/hacker) + +```shell +mkdir -p ~/tmp/gohello-http +cd ~/tmp/gohello-http +git init +nix flake init -t github:Xe/templates#go-web-server +``` + +[You may see a message from direnv about +needing to approve its content. This will use Nix flake's cached interpreter to +give you all the advantages of something like Lorri without having to +install Lorri.](conversation://Mara/hacker) + +Then make an initial commit and run it: + +```shell +git add . +git commit -sm "initial commit" +nix build +./result/bin/web-server +``` + +[Why are you using `git add .` everywhere? Shouldn't the files be picked up +implicitly?](conversation://Mara/hmm) + +[Not always. Nix flakes only deals with files that are tracked by git when you +use it in a git repository. This means that if you want the changes to be +observed by Nix, you need to add them to git somehow. `git add` is good enough +for this.](conversation://Cadey/enby) + +Or you can run it directly with `nix run`: + +```shell +nix run +``` + +## Docker Images + +Most of the time you will build software with Nix, however that doesn't stop you +from building things like Docker images with Nix. Remember that you can have the +output of any shell commands be run in a Nix build (the only catch is that they +can't access the internet directly), so you can build a Docker image out of that +web server template by defining another package: + +```nix +# flake.nix + +# after defaultPackage +packages = { + docker = let + web = self.defaultPackage.${system}; + in pkgs.dockerTools.buildLayeredImage { + name = web.pname; + tag = web.version; + contents = [ web ]; + + config = { + Cmd = [ "/bin/web-server" ]; + WorkingDir = "/"; + }; + }; +}; +``` + +This will build a Docker image with the web-server binary in it. To build it, +run these commands: + +```shell +git add . +nix build .#docker +``` + +[What's with that last argument to `nix build`, won't that be read as a shell +comment?](conversation://Mara/hmm) + +[It's a reference to the package in the flake. Shell only parses comments when +the `#` is the first character after whitespace, so this is more of a URL +fragment than a comment. It's telling `nix build` to build the flake package +named `docker`.](conversation://Cadey/enby) + +It will put the resulting docker image in `./result`. To load it into docker use +the following command: + +```console +$ docker load < result +Loaded image: web-server:20220227 +``` + +[Your image tag may differ depending on when you build this +image. This is deterministic because that date is derived from the date that the +current git commit was made.](conversation://Mara/happy) + +Then you can run it with `docker run`: + +```shell +docker run -itp 3031:3031 web-server:20220227 +``` + +Then poke it with curl: + +```console +$ curl http://[::]:3031 +hello from nix! +``` + +You can push this image to the Docker hub like any other image. Another cool +thing about this is that when you update the program, it'll only actually load +the images that changed. Let's edit the hello world message: + +```go +http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + fmt.Fprintln(w, "hello from nix building a docker image!") +}) +``` + +And then re-build and load it into Docker: + +```shell +git add . +nix build .#docker +docker load < result +``` + +[Woah, when I did that it only updated 2 layers. The first time that I loaded it +there were something like 7 layers. What's up with +that?](conversation://Mara/hmm) + +[When you use `buildLayeredImage`, each Nix package that contributes to the +image gets put in its own Docker layer. This means that only the things that +have changed actually need to be considered, so when you push an updated image +to another machine, the only things that will actually be pushed are the +application binary and the symlink farm pointing to the `contents` of the Docker +image.](conversation://Cadey/enby) + +## systemd Portable Services + +[systemd Portable Services](https://systemd.io/PORTABLE_SERVICES/) function like +Docker, but they work at the systemd level and allow you to integrate into +systemd instead of running on the side of it. This gives you access to systemd's +readiness signaling, logging pipeline and dependency graph so that you can +integrate like a native service. They are like containers, but without a lot of +the headaches around networking, stateful storage and logging. They are just +systemd services at their core. + +[These are kinda like Ubuntu's Snaps or Flatpaks, but they operate purely at the +system level and are focused at providing things for system services instead of +user-facing applications. Ubuntu's Snaps do let you create system services, but +they are basically exclusively used on Ubuntu. systemd Portable Services let you +target more than just Ubuntu. In the next few years with more releases of +systemd, Portable Services should be easier to use and will be more integrated +with the system than Docker is.](conversation://Mara/hacker) + +There is currently an [open pull request](https://systemd.io/PORTABLE_SERVICES/) +for adding Portable Service building support to nixpkgs, however we can mess +around with it today thanks to [my portable-svc +overlay](https://tulpa.dev/cadey/portable-svc) that copies in the contents of +that pull request. + +[In Nix, an overlay is a set of additional packages or functions that is put on +top of nixpkgs. This overlay defines the `portableService` function that is +needed to build portable services.](conversation://Mara/hacker) + +To make this into a portable service, first we need to add my overlay to the +flake inputs: + +```nix +# flake.nix +inputs = { + nixpkgs.url = "nixpkgs/nixos-unstable"; + utils.url = "github:numtide/flake-utils"; + portable-svc.url = "git+https://tulpa.dev/cadey/portable-svc.git?ref=main"; +}; +``` + +Then add it as an argument to the `outputs` function: + +```nix +outputs = { self, nixpkgs, utils, portable-svc }: +``` + +And then change how we are importing the `pkgs` variable. The `pkgs` variable +we're currently using is imported like this: + +```nix +let pkgs = nixpkgs.legacyPackages.${system}; +``` + +This works, however there isn't a way to specify an overlay into this. We need +to change this into a manual import of nixpkgs with the overlay specified, like +this: + +```nix +let pkgs = import nixpkgs { + overlays = [ portable-svc.overlay ]; + inherit system; +}; +``` + +This will let us use the `portableService` function in Nix package definitions. + +Next we need to make a systemd service unit for the web server. The exact path +to the program binary can and will change with every build, so it would be good +to have this templated. Make a folder called `systemd`: + +```shell +mkdir systemd +``` + +And put the following contents in `systemd/web-server.service.in`: + +```systemd +[Unit] +Description=A web service + +[Service] +DynamicUser=yes +ExecStart=@web@/bin/web-server + +[Install] +WantedBy=multi-user.target +``` + +Then under the docker package definition, add the package that will template out +the systemd unit: + +```nix +web-service = pkgs.substituteAll { + name = "web-server.service"; + src = ./systemd/web-server.service.in; + web = self.defaultPackage.${system}; +}; +``` + +You can build it with `nix build .#web-service`, the output will look something +like this: + +```systemd +[Unit] +Description=A web service + +[Service] +DynamicUser=yes +ExecStart=/nix/store/yl863jm907wfr7gq9j0c4bd3d4bdc4vp-web-server-20220227/bin/web-server + +[Install] +WantedBy=multi-user.target +``` + +[The `@web@` in the template was replaced with the nix store path for the web +server!](conversation://Mara/happy) + +Then you can add the bit that builds the portable service: + +```nix +portable = let + web = self.defaultPackage.${system}; +in pkgs.portableService { + inherit (web) version; + name = web.pname; + description = "A web server"; + units = [ self.packages.${system}.web-service ]; +}; +``` + +Then you can build it with `nix build`: + +```shell +nix build .#portable +``` + +And then take a look at `./result`: + +```console +$ file $(readlink ./result) +/nix/store/1da6b90i75n03kqlzzfdwxii0j0bzxaf-web-server_20220227.raw: +Squashfs filesystem, +little endian, +version 4.0, +xz compressed, +9555806 bytes, +2010 inodes, +blocksize: 1048576 bytes, +created: Tue Jan 1 00:00:00 1980 +``` + +
+ +At the time of writing this article, the most reliable way to test portable +services is to use Arch Linux. So you could use something like +[waifud](https://github.com/Xe/waifud) to spin up an Arch Linux VM: + +```console +$ waifuctl create -d arch -h logos -s 20 +created instance jangmo-o on logos +jangmo-o: running +jangmo-o: init: IP address: 10.77.129.208 +``` + +Then copy it over with `scp`: + +```console +$ scp (readlink ./result) xe@10.77.129.208:web-server_20220227.raw +``` + +
+ +Then you can use `portablectl` to attach it to the system: + +```console +$ sudo portablectl attach ./web-server_20220227.raw +[...] +Created symlink /etc/portables/web-server_20220227.raw → /home/xe/web-server_20220227.raw. +``` + +And then start it like any systemd service: + +```console +$ sudo systemctl start web-server +``` + +[If you want the service to start automatically, add `--enable --now` to the +`portablectl attach` command. That will enable the service in systemd and then +start it, like when you run `systemctl enable --now +something.service`.](conversation://Mara/hacker) + +And then inspect the service's status with `systemctl`: + +```console +$ sudo systemctl status web-server +● web-server.service - A web service + Loaded: loaded (/etc/systemd/system.attached/web-server.service; disabled; vendor preset: disabled) + Drop-In: /etc/systemd/system.attached/web-server.service.d + └─10-profile.conf, 20-portable.conf + Active: active (running) since Sun 2022-02-27 18:21:01 UTC; 20s ago + Main PID: 960 (web-server) + Tasks: 5 (limit: 513) + Memory: 8.1M + CPU: 189ms + CGroup: /system.slice/web-server.service + └─960 /nix/store/yl863jm907wfr7gq9j0c4bd3d4bdc4vp-web-server-20220227/bin/web-server + +Feb 27 18:21:01 jangmo-o systemd[1]: Started A web service. +Feb 27 18:21:01 jangmo-o web-server[960]: 2022/02/27 18:21:01 listening for HTTP on :3031 +``` + +And finally poke it with curl: + +```console +$ curl http://[::]:3031 +hello from nix building a docker image! +``` + +[That ain't Docker, chief!](conversation://Numa/delet) + +[I know, I know, I didn't adjust the message from when I wrote the Docker +example.](conversation://Cadey/facepalm) + +And then you can change the handler to something like: + +```go +http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { + fmt.Fprintf(w, "PORTABLE=%s\n", os.Getenv("PORTABLE")) +}) +``` + +Rebuild the image with `nix build`: + +```console +$ git add . +$ nix build .#portable +``` + +Copy it to the arch VM with `scp`: + +```console +$ scp (readlink ./result) xe@10.77.129.208:web-server_20220227.raw +``` + +And finally run `portablectl reattach` to upgrade it: + +```console +$ sudo portablectl reattach --now ./web-server_20220227.raw +Queued /org/freedesktop/systemd1/job/858 to call RestartUnit on portable service +web-server.service. +``` + +Then you can see that it restarted the unit with `systemctl status`: + +```console +$ sudo systemctl status web-server +● web-server.service - A web service + Loaded: loaded (/etc/systemd/system.attached/web-server.service; disabled; vendor preset: disabled) + Drop-In: /etc/systemd/system.attached/web-server.service.d + └─10-profile.conf, 20-portable.conf + Active: active (running) since Sun 2022-02-27 18:30:04 UTC; 37s ago + Main PID: 1074 (web-server) + Tasks: 6 (limit: 513) + Memory: 8.1M + CPU: 182ms + CGroup: /system.slice/web-server.service + └─1074 /nix/store/j1mfz3ydn13qmvcgrql33zi0dwb3x7dk-web-server-20220227/bin/web-server + +Feb 27 18:30:04 jangmo-o systemd[1]: Started A web service. +Feb 27 18:30:04 jangmo-o web-server[1074]: 2022/02/27 18:30:04 listening for HTTP on :3031 +``` + +And finally poke it with curl: + +```console +$ curl http://[::]:3031 +PORTABLE=web-server_20220227.raw +``` + +And there you go! Nix created a portable system service, we spawned it on a +newly created Arch Linux VM and then were able to update it so that we could +replace the message. + +--- + +Nix builds can do more than just turn code into software. They can create Docker +images, Portable Services, virtual machine images and more. The only real limit +is what you can imagine. + +Flakes make it easier to pull in and munge about packages. Before flakes you'd +need to have a few `.nix` files like `docker.nix` for the docker image and +`portable.nix` for the portable service. You'd also have to pull in something +like [Niv](https://github.com/nmattia/niv) to make sure everything uses the same +version of nixpkgs, and even then it's opt-in, not opt-out, so it's easy to mess +things up and not use the pinned versions of things. Flakes make that explicit +behavior implicit, so you can't bring in dependencies you aren't aware of. + +If you want to see the code repo I developed while writing this post, see +[cadey/gohello-http](https://tulpa.dev/cadey/gohello-http) on my git server. + +Thanks for reading! diff --git a/src/post/frontmatter.rs b/src/post/frontmatter.rs index 595079f..fa4c65c 100644 --- a/src/post/frontmatter.rs +++ b/src/post/frontmatter.rs @@ -13,6 +13,13 @@ pub struct Data { pub thumb: Option, pub show: Option, pub redirect_to: Option, + pub vod: Option, +} + +#[derive(Eq, PartialEq, Deserialize, Default, Debug, Serialize, Clone)] +pub struct Vod { + pub twitch: String, + pub youtube: String, } enum State { diff --git a/templates/blogpost.rs.html b/templates/blogpost.rs.html index 8ace300..3b60689 100644 --- a/templates/blogpost.rs.html +++ b/templates/blogpost.rs.html @@ -68,6 +68,10 @@
+@if post.front_matter.vod.is_some() { +

This post was written live on Twitch. You can check out the stream recording on Twitch here and on YouTube here.

+} +