diff --git a/blog/nix-flakes-1-2022-02-21.markdown b/blog/nix-flakes-1-2022-02-21.markdown new file mode 100644 index 0000000..4ff22cd --- /dev/null +++ b/blog/nix-flakes-1-2022-02-21.markdown @@ -0,0 +1,527 @@ +--- +title: "Nix Flakes: an Introduction" +date: 2022-02-21 +tags: + - nix + - nixos +series: nix-flakes +author: Twi +--- + +Nix is a package manager that lets you have a more deterministic view of your +software dependencies and build processes. One if its biggest weaknesses out of +the box is that there are very few conventions on how projects using Nix should +work together. It's like having a build system but also having to configure +systems to run software yourself. This could mean copying a NixOS module out of +the project's git repo, writing your own or more. In contrast to this, [Nix +flakes](https://nixos.wiki/wiki/Flakes) define a set of conventions for how +software can be build, run, integrated and deployed without having to rely on +external tools such as [Niv](https://github.com/nmattia/niv) or +[Lorri](https://github.com/nix-community/lorri) to help you do basic tasks in a +timely manner. + +This is going to be a series of posts that will build on eachother. This post +will be an introduction to Nix flakes and serve as a "why should I care?" style +overview of what you can do with flakes without going into too much detail. Most +of these will get separate posts (some more than one post). + +In my opinion, here are some of the big reasons you should care about Nix +flakes: + +- Flakes adds project templates to Nix +- Flakes define a standard way to say "this is a program you can run" +- Flakes consolidate development environments into project configuration +- Flakes can pull in dependencies from outside git repos trivially +- Flakes can work with people that don't use flakes too +- Flakes supports using private git repos +- Flakes let you define system configuration alongside your application code +- Flakes let you embed the git hash of your configurations repository into + machines you deploy + +## Project Templates + +One of the big annoying parts about getting into Nix is that setting up projects +isn't totally a defined science. Nix configurations just tend to grow +organically and can easily become weird or difficult to understand for people +that didn't start the project. Nix flakes helps fix this by doing a few things: + +1. Defining a `flake.nix` as the central "hub" for your project's dependencies, + exposed packages, NixOS configuration modules [and + more](https://nixos.wiki/wiki/Flakes#Output_schema). +2. Shipping a [set of templates](https://github.com/NixOS/templates) so that you + can get projects started easily. Think something like + [Yeoman](https://yeoman.io) but built directly into Nix. You can also define + your own templates in your `flake.nix`. + +As an example that we will use for the rest of this post to help explain it, +let's make a Go project with their Go template. First you will need to enable +Nix flakes on your machine. If you are using NixOS, add this to your +`configuration.nix` file: + +```nix +nix = { + package = pkgs.nixFlakes; + extraOptions = '' + experimental-features = nix-command flakes + ''; +}; +``` + +Then rebuild your system and you can continue along with the article. + +If you are not on NixOS, you will need to either edit `~/.config/nix/nix.conf` +or `/etc/nix/nix.conf` and add the following line to it: + +``` +experimental-features = nix-command flakes +``` + +[You may need to restart the Nix daemon here, but if you are unsure how Nix was +set up on that non-NixOS machine feel free to totally restart your +computer.](conversation://Mara/hacker) + +Now go to a temporary folder and run these commands to make a folder and create +a new flake from a template: + +```console +mkdir ~/tmp/go-demo +cd ~/tmp/go-demo +nix flake new -t templates#go-hello . +git init && git add . +``` + +This will create a few files in the folder: + +```console +$ ls +flake.lock flake.nix go.mod main.go +``` + +Then you can look at `flake.nix` to see what's up: + +```nix +{ + description = "A simple Go package"; + + # Nixpkgs / NixOS version to use. + inputs.nixpkgs.url = "nixpkgs/nixos-21.11"; + + outputs = { self, nixpkgs }: + let + + # Generate a user-friendly version number. + version = builtins.substring 0 8 self.lastModifiedDate; + + # System types to support. + supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ]; + + # Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'. + forAllSystems = nixpkgs.lib.genAttrs supportedSystems; + + # Nixpkgs instantiated for supported system types. + nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; }); + + in + { + + # Provide some binary packages for selected system types. + packages = forAllSystems (system: + let + pkgs = nixpkgsFor.${system}; + in + { + go-hello = pkgs.buildGoModule { + pname = "go-hello"; + inherit version; + # In 'nix develop', we don't need a copy of the source tree + # in the Nix store. + src = ./.; + + # This hash locks the dependencies of this package. It is + # necessary because of how Go requires network access to resolve + # VCS. See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for + # details. Normally one can build with a fake sha256 and rely on native Go + # mechanisms to tell you what the hash should be or determine what + # it should be "out-of-band" with other tooling (eg. gomod2nix). + # To begin with it is recommended to set this, but one must + # remeber to bump this hash when your dependencies change. + #vendorSha256 = pkgs.lib.fakeSha256; + + vendorSha256 = "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo="; + }; + }); + + # The default package for 'nix build'. This makes sense if the + # flake provides only one package or there is a clear "main" + # package. + defaultPackage = forAllSystems (system: self.packages.${system}.go-hello); + }; +} +``` + +This defines a single Go package that is supported on macOS and Linux for 64 bit +x86 processors and 64 bit ARM processors. + +[In practice this spread should cover all of the main targets you'll need to +care about for local development and cloud +deployment.](conversation://Mara/hacker) + +You can then build the flake with `nix build`: + +```console +$ nix build +``` + +And then run it: + +```console +$ ./result/bin/go-hello +Hello Nix! +``` + +## Standard Default Package + +Let's take a closer look at the higher level things in the flake: + +```nix +{ + description = "A simple Go package"; + + inputs.nixpkgs.url = "nixpkgs/nixos-21.11"; + + outputs = { self, nixpkgs }: { + packages = { ... }; + defaultPackage = { ... }; + }; +} +``` + +[A note: in the rest of this article (and series of articles), when I refer to a +"flake output", I am referring to an attribute in the `outputs` attribute of +your `flake.nix`. Ditto with "flake input" referring to the `inputs` attribute +of your `flake.nix`.](conversation://Cadey/enby) + +When you ran `nix build` earlier, it defaulted to building the package in +`defaultPackage`. You can also build the `go-hello` package by running this +command: + +```console +$ nix build .#go-hello +``` + +And if you want to build the copy I made for this post: + +```console +$ nix build github:Xe/gohello +$ ./result/bin/go-hello +Hello reader! +``` + +A standard default package means that you can more easily build software without +having to read documentation on what file to build. `nix build` will Just Work™️. + +## Exposing Packages as Applications + +Additionally, you can expose a package as an application. This allows you to +simplify that above `nix build` and `./result/bin/go-hello` cycle into a single +`nix run` command. Open `flake.nix` in your favorite editor and let's configure +`go-hello` to be the default app: + +```nix +# below defaultPackage + +defaultApp = forAllSystems (system: { + type = "app"; + program = "${self.packages.${system}.go-hello}/bin/go-hello"; +}); +``` + +Then you can run it with `nix run`: + +```console +$ nix run +Hello Nix! +``` + +Or you can run my copy: + +```console +$ nix run github:Xe/gohello/main +Hello reader! +``` + +[What is that extra part of the URL path for? Is that a git +branch?](conversation://Mara/hmm) + +[Yes, you can use that syntax to set the git branch that Nix should build from. +By default it will use the default branch (typically `main`), but sometimes you +need to specify a branch or commit directly.](conversation://Cadey/enby) + +## Development Environment Configuration + +One of Nix's superpowers is the ability to declaratively manage the development +environment for a project so that you can be sure that everyone working on the +project is using the same tools. + +[I use this with all of my projects to the point that when I am outside of a +project folder I do not have any development tools +available.](conversation://Cadey/enby) + +Flakes has the ability to specify this using the `devShell` flake output. You +can add it to your `flake.nix` using this: + +```nix +# after defaultApp + +devShell = forAllSystems (system: + let pkgs = nixpkgsFor.${system}; + in pkgs.mkShell { + buildInputs = with pkgs; [ go gopls goimports go-tools ]; + }); +``` + +[I consider this to be a basic Go development environment. It includes standard +tools such as the language server, `goimports` for better formatting and tools +like staticcheck. If you use staticcheck +regularly at work, please consider throwing Dominik a couple bucks +a month if you find it useful. It helps the project be more +self-sustaining.](conversation://Mara/happy) + +Then you can enter the development shell with `nix develop`: + +``` +$ nix develop + +[cadey@pneuma:~/tmp/gohello]$ go version +go version go1.16.9 linux/amd64 +``` + +And then hack at your project all you want. You can send this git repo to a +friend and they will have the same setup. + +## External Dependencies + +Now let's talk about inputs. Flake inputs let you add external dependencies to a +project. As an example, let's look at the `nixpkgs` input: + +```nix +# Nixpkgs / NixOS version to use. +inputs.nixpkgs.url = "nixpkgs/nixos-21.11"; +``` + +This defines the release of nixpkgs that should be used for the project. This +template defaults to NixOS 21.11's version of nixpkgs, however we can upgrade it +to nixos-unstable by changing it to this: + +```nix +# Nixpkgs / NixOS version to use. +inputs.nixpkgs.url = "nixpkgs/nixos-unstable"; +``` + +Then we can run `nix flake update` and then `nix develop` and see that we are +running a newer version of Go: + +```console +$ nix flake update +warning: updating lock file '/home/cadey/tmp/gohello/flake.lock': +• Updated input 'nixpkgs': + 'github:NixOS/nixpkgs/77aa71f66fd05d9e7b7d1c084865d703a8008ab7' (2022-01-19) + → 'github:NixOS/nixpkgs/2128d0aa28edef51fd8fef38b132ffc0155595df' (2022-02-16) + +$ nix develop + +[cadey@pneuma:~/tmp/gohello]$ go version +go version go1.17.7 linux/amd64 +``` + +This also lets you pull in other Nix flakes projects, such as my CSS framework +[Xess](https://github.com/Xe/Xess): + +```nix +inputs.xess.url = "github:Xe/Xess"; +inputs.xess.inputs.nixpkgs.follows = "nixpkgs"; +``` + +[Why is that second line needed?](conversation://Mara/hmm) + +[By default when you pull in another project with Nix flakes, it treats that +project as an entirely separate universe and only interacts with the outputs of +that flake. This means it pulls in its own version of nixpkgs, each dependency +it has can pull in that own version of nixpkgs and vice versa ad infinitum. By +making Xess' nixpkgs input follows our own one, we are saying "I understand this +may be incompatible, but please use this version of nixpkgs instead". This can +help larger projects with many inputs (such as a nixos configs repo made by +someone with too many throwaway side projects) evaluate and build faster. Nix +flakes does have a cached evaluator, but still it helps to avoid the problem in +the first place.](conversation://Cadey/enby) + +Or anything you want! A useful library to pull in is +[flake-utils](https://github.com/numtide/flake-utils), that can help you +simplify your `flake.nix` and get rid of those ugly `forAllSystems` and +`nixpkgsFor` functions in the `flake.nix` that this post used by default. For an +example of a flake that uses this library, see [this +`flake.nix`](https://tulpa.dev/Xe/mara/src/branch/main/flake.nix) from the IRC +bot that lives in [`#xeserv`](https://web.libera.chat/#xeserv). + +[Adapting this trivial example to use `flake-utils` is an excellent exercise for +the reader!](conversation://Mara/happy) + +## Backwards Compatibility + +Normally you need to enable Nix flakes in your Nix daemon to take advantage of +them. This is great for when you can do that, but sometimes you'll need to make +things work for people without flakes enabled. This could happen when needing to +graft in a Nix flakes project to one without flakes enabled. There is a library +called [flake-compat](https://github.com/edolstra/flake-compat) that makes this +easy. + +Add the following to your flake inputs: + +```nix +inputs.flake-compat = { + url = "github:edolstra/flake-compat"; + flake = false; +}; +``` + +And then create `default.nix` with the following contents: + +```nix +(import ( + let + lock = builtins.fromJSON (builtins.readFile ./flake.lock); + in fetchTarball { + url = "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz"; + sha256 = lock.nodes.flake-compat.locked.narHash; } +) { + src = ./.; +}).defaultNix +``` + +And `shell.nix` with the following contents: + +```nix +(import ( + let + lock = builtins.fromJSON (builtins.readFile ./flake.lock); + in fetchTarball { + url = "https://github.com/edolstra/flake-compat/archive/${lock.nodes.flake-compat.locked.rev}.tar.gz"; + sha256 = lock.nodes.flake-compat.locked.narHash; } +) { + src = ./.; +}).shellNix +``` + +Then you can use `nix-build` and `nix-shell` like you have in other Nix +projects. + +## Private Git Repos + +Nix flakes has native support for private git repositories as inputs. This can +be useful when trying to build software you don't want to release as open to the +world. To use a private repo, your flake input URL should look something like +this: + +``` +ssh+git://git@github.com:user/repo +``` + +[I'm pretty sure you could use private git repos outside of flakes, however it +was never really clear to me _how_ you end up doing +it.](conversation://Cadey/coffee) + +## Embed NixOS Modules in Flakes + +The biggest ticket item for me is that it lets you embed NixOS modules in flakes +themselves. This lets you define the system configuration for software right +next to where the software is defined, thus shipping it as a unit. Using this +you can make installing software a matter of adding it to your system's flake, +adding the module and then enabling the settings you want to enable. + +As an example, here is the NixOS module for that IRC bot I mentioned: + +```nix +nixosModules.bot = { config, lib, ... }: { + options.within.services.mara-bot.enable = + lib.mkEnableOption "enable Mara bot"; + + config = lib.mkIf config.within.services.mara-bot.enable { + users.groups.mara-bot = { }; + + users.users.mara-bot = { + createHome = true; + isSystemUser = true; + home = "/var/lib/mara-bot"; + group = "mara-bot"; + }; + + systemd.services.mara-bot = { + wantedBy = [ "multi-user.target" ]; + environment.RUST_LOG = "tower_http=debug,info"; + unitConfig.ConditionPathExists = "/var/lib/mara-bot/config.yaml"; + serviceConfig = { + User = "mara-bot"; + Group = "mara-bot"; + Restart = "always"; + WorkingDirectory = "/var/lib/mara-bot"; + ExecStart = "${self.defaultPackage."${system}"}/bin/mara"; + }; + }; + }; +}; +``` + +The key important part here is the `ExecStart` line. It points back to the +flake's default package (which is hopefully where the bot's code is defined), +and then has systemd manage that. + +I plan to use this to radically simplify my nixos-configs repo. Right now it has +a lot of code that is very project-specific and if I can move that into the +projects in question, I can eliminate a lot of code out of the core of my +configs repo. + +## Embedding Configuration Git Hash into Systems + +Finally, Nix flakes lets you see the configuration version of a system by +embedding it at the build step. Normally NixOS lets you see the following +information with `nixos-version --json`: + +```json +{ + "nixosVersion": "22.05pre348581.c07b471b52b", + "nixpkgsRevision": "c07b471b52be8fbc49a7dc194e9b37a6e19ee04d" +} +``` + +You have the NixOS version and the nixpkgs hash. That doesn't tell you what +configuration you are running or anything about it though. However with flakes +you can embed the git hash of your configuration into the system config: + +```json +{ + "configurationRevision": "f53891121ce4204f57409cbe9e6fcee3b030a350", + "nixosVersion": "22.05.20220210.48d63e9", + "nixpkgsRevision": "48d63e924a2666baf37f4f14a18f19347fbd54a2" +} +``` + +This can let you make a URL pointing to the commit in that output: + +```console +$ echo "https://tulpa.dev/cadey/nixos-configs/src/commit/$(ssh logos nixos-version --json | jq -r .configurationRevision)" +``` + +Which will spit out a link to +[cadey/nixos-configs@f53891121](https://tulpa.dev/cadey/nixos-configs/src/commit/f53891121ce4204f57409cbe9e6fcee3b030a350). + +I'll cover more on how to do this in the NixOS deployment post. + +--- + +There is a lot more to get into with each of these topics. I'm only really +giving a very high level overview on them while I learn more and migrate over my +NixOS configurations to flakes +[piecemeal](https://tulpa.dev/cadey/nixos-configs). This has also given me the +opportunity to clean things up and chew out a lot of the fat from my NixOS +configurations. More to come when it is ready.