--- title: "Nix Flakes: an Introduction" date: 2022-02-21 tags: - nix - nixos series: nix-flakes author: Twi --- Nix is a package manager that lets you have a more deterministic view of your software dependencies and build processes. One if its biggest weaknesses out of the box is that there are very few conventions on how projects using Nix should work together. It's like having a build system but also having to configure systems to run software yourself. This could mean copying a NixOS module out of the project's git repo, writing your own or more. In contrast to this, [Nix flakes](https://nixos.wiki/wiki/Flakes) define a set of conventions for how software can be build, run, integrated and deployed without having to rely on external tools such as [Niv](https://github.com/nmattia/niv) or [Lorri](https://github.com/nix-community/lorri) to help you do basic tasks in a timely manner. This is going to be a series of posts that will build on eachother. This post will be an introduction to Nix flakes and serve as a "why should I care?" style overview of what you can do with flakes without going into too much detail. Most of these will get separate posts (some more than one post). In my opinion, here are some of the big reasons you should care about Nix flakes: - Flakes adds project templates to Nix - Flakes define a standard way to say "this is a program you can run" - Flakes consolidate development environments into project configuration - Flakes can pull in dependencies from outside git repos trivially - Flakes can work with people that don't use flakes too - Flakes supports using private git repos - Flakes let you define system configuration alongside your application code - Flakes let you embed the git hash of your configurations repository into machines you deploy [Something that may also help you understand why flakes matter is that Nix by itself is more akin to Dockerfiles. Dockerfiles help you build the software, but they don't really help you run or operate the software. Nix flakes is more akin to docker-compose, they help you compose packages written in Nix to run across machines.](conversation://Mara/happy) ## Project Templates One of the big annoying parts about getting into Nix is that setting up projects isn't totally a defined science. Nix configurations just tend to grow organically and can easily become weird or difficult to understand for people that didn't start the project. Nix flakes helps fix this by doing a few things: 1. Defining a `flake.nix` as the central "hub" for your project's dependencies, exposed packages, NixOS configuration modules [and more](https://nixos.wiki/wiki/Flakes#Output_schema). 2. Shipping a [set of templates](https://github.com/NixOS/templates) so that you can get projects started easily. Think something like [Yeoman](https://yeoman.io) but built directly into Nix. You can also define your own templates in your `flake.nix`. As an example that we will use for the rest of this post to help explain it, let's make a Go project with their Go template. First you will need to enable Nix flakes on your machine. If you are using NixOS, add this to your `configuration.nix` file: ```nix nix = { package = pkgs.nixFlakes; extraOptions = '' experimental-features = nix-command flakes ''; }; ``` Then rebuild your system and you can continue along with the article. If you are not on NixOS, you will need to either edit `~/.config/nix/nix.conf` or `/etc/nix/nix.conf` and add the following line to it: ``` experimental-features = nix-command flakes ``` [You may need to restart the Nix daemon here with `sudo systemctl restart nix-daemon.service`, but if you are unsure how Nix was set up on that non-NixOS machine feel free to totally restart your computer.](conversation://Mara/hacker) Now go to a temporary folder and run these commands to make a folder and create a new flake from a template: ```console mkdir ~/tmp/go-demo cd ~/tmp/go-demo nix flake new -t templates#go-hello . git init && git add . ``` This will create a few files in the folder: ```console $ ls flake.lock flake.nix go.mod main.go ``` Then you can look at `flake.nix` to see what's up: ```nix { description = "A simple Go package"; # Nixpkgs / NixOS version to use. inputs.nixpkgs.url = "nixpkgs/nixos-21.11"; outputs = { self, nixpkgs }: let # Generate a user-friendly version number. version = builtins.substring 0 8 self.lastModifiedDate; # System types to support. supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ]; # Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'. forAllSystems = nixpkgs.lib.genAttrs supportedSystems; # Nixpkgs instantiated for supported system types. nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; }); in { # Provide some binary packages for selected system types. packages = forAllSystems (system: let pkgs = nixpkgsFor.${system}; in { go-hello = pkgs.buildGoModule { pname = "go-hello"; inherit version; # In 'nix develop', we don't need a copy of the source tree # in the Nix store. src = ./.; # This hash locks the dependencies of this package. It is # necessary because of how Go requires network access to resolve # VCS. See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for # details. Normally one can build with a fake sha256 and rely on native Go # mechanisms to tell you what the hash should be or determine what # it should be "out-of-band" with other tooling (eg. gomod2nix). # To begin with it is recommended to set this, but one must # remeber to bump this hash when your dependencies change. #vendorSha256 = pkgs.lib.fakeSha256; vendorSha256 = "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo="; }; }); # The default package for 'nix build'. This makes sense if the # flake provides only one package or there is a clear "main" # package. defaultPackage = forAllSystems (system: self.packages.${system}.go-hello); }; } ``` This defines a single Go package that is supported on macOS and Linux for 64 bit x86 processors and 64 bit ARM processors. [In practice this spread should cover all of the main targets you'll need to care about for local development and cloud deployment.](conversation://Mara/hacker) You can then build the flake with `nix build`: ```console $ nix build ``` And then run it: ```console $ ./result/bin/go-hello Hello Nix! ``` ## Standard Default Package Let's take a closer look at the higher level things in the flake: ```nix { description = "A simple Go package"; inputs.nixpkgs.url = "nixpkgs/nixos-21.11"; outputs = { self, nixpkgs }: { packages = { ... }; defaultPackage = { ... }; }; } ``` [A note: in the rest of this article (and series of articles), when I refer to a "flake output", I am referring to an attribute in the `outputs` attribute of your `flake.nix`. Ditto with "flake input" referring to the `inputs` attribute of your `flake.nix`.](conversation://Cadey/enby) When you ran `nix build` earlier, it defaulted to building the package in `defaultPackage`. You can also build the `go-hello` package by running this command: ```console $ nix build .#go-hello ``` And if you want to build the copy I made for this post: ```console $ nix build github:Xe/gohello $ ./result/bin/go-hello Hello reader! ``` A standard default package means that you can more easily build software without having to read documentation on what file to build. `nix build` will Just Work™️. ## Exposing Packages as Applications Additionally, you can expose a package as an application. This allows you to simplify that above `nix build` and `./result/bin/go-hello` cycle into a single `nix run` command. Open `flake.nix` in your favorite editor and let's configure `go-hello` to be the default app: ```nix # below defaultPackage defaultApp = forAllSystems (system: { type = "app"; program = "${self.packages.${system}.go-hello}/bin/go-hello"; }); ``` Then you can run it with `nix run`: ```console $ nix run Hello Nix! ``` Or you can run my copy: ```console $ nix run github:Xe/gohello/main Hello reader! ``` [What is that extra part of the URL path for? Is that a git branch?](conversation://Mara/hmm) [Yes, you can use that syntax to set the git branch that Nix should build from. By default it will use the default branch (typically `main`), but sometimes you need to specify a branch or commit directly.](conversation://Cadey/enby) ## Development Environment Configuration One of Nix's superpowers is the ability to declaratively manage the development environment for a project so that you can be sure that everyone working on the project is using the same tools. [I use this with all of my projects to the point that when I am outside of a project folder I do not have any development tools available.](conversation://Cadey/enby) Flakes has the ability to specify this using the `devShell` flake output. You can add it to your `flake.nix` using this: ```nix # after defaultApp devShell = forAllSystems (system: let pkgs = nixpkgsFor.${system}; in pkgs.mkShell { buildInputs = with pkgs; [ go gopls goimports go-tools ]; }); ``` [I consider this to be a basic Go development environment. It includes standard tools such as the language server, `goimports` for better formatting and tools like staticcheck. If you use staticcheck regularly at work, please consider throwing Dominik a couple bucks a month if you find it useful. It helps the project be more self-sustaining.](conversation://Mara/happy) Then you can enter the development shell with `nix develop`: ``` $ nix develop [cadey@pneuma:~/tmp/gohello]$ go version go version go1.16.9 linux/amd64 ``` And then hack at your project all you want. You can send this git repo to a friend and they will have the same setup. ## External Dependencies Now let's talk about inputs. Flake inputs let you add external dependencies to a project. As an example, let's look at the `nixpkgs` input: ```nix # Nixpkgs / NixOS version to use. inputs.nixpkgs.url = "nixpkgs/nixos-21.11"; ``` This defines the release of nixpkgs that should be used for the project. This template defaults to NixOS 21.11's version of nixpkgs, however we can upgrade it to nixos-unstable by changing it to this: ```nix # Nixpkgs / NixOS version to use. inputs.nixpkgs.url = "nixpkgs/nixos-unstable"; ``` Then we can run `nix flake update` and then `nix develop` and see that we are running a newer version of Go: ```console $ nix flake update warning: updating lock file '/home/cadey/tmp/gohello/flake.lock': • Updated input 'nixpkgs': 'github:NixOS/nixpkgs/77aa71f66fd05d9e7b7d1c084865d703a8008ab7' (2022-01-19) → 'github:NixOS/nixpkgs/2128d0aa28edef51fd8fef38b132ffc0155595df' (2022-02-16) $ nix develop [cadey@pneuma:~/tmp/gohello]$ go version go version go1.17.7 linux/amd64 ``` This also lets you pull in other Nix flakes projects, such as my CSS framework [Xess](https://github.com/Xe/Xess): ```nix inputs.xess.url = "github:Xe/Xess"; inputs.xess.inputs.nixpkgs.follows = "nixpkgs"; ``` [Why is that second line needed?](conversation://Mara/hmm) [By default when you pull in another project with Nix flakes, it treats that project as an entirely separate universe and only interacts with the outputs of that flake. This means it pulls in its own version of nixpkgs, each dependency it has can pull in that own version of nixpkgs and vice versa ad infinitum. By making Xess' nixpkgs input follows our own one, we are saying "I understand this may be incompatible, but please use this version of nixpkgs instead". This can help larger projects with many inputs (such as a nixos configs repo made by someone with too many throwaway side projects) evaluate and build faster. Nix flakes does have a cached evaluator, but still it helps to avoid the problem in the first place.](conversation://Cadey/enby) Or anything you want! A useful library to pull in is [flake-utils](https://github.com/numtide/flake-utils), that can help you simplify your `flake.nix` and get rid of those ugly `forAllSystems` and `nixpkgsFor` functions in the `flake.nix` that this post used by default. For an example of a flake that uses this library, see [this `flake.nix`](https://tulpa.dev/Xe/mara/src/branch/main/flake.nix) from the IRC bot that lives in [`#xeserv`](https://web.libera.chat/#xeserv). [Adapting this trivial example to use `flake-utils` is an excellent exercise for the reader! This library should really be shipped with flakes by default.](conversation://Mara/happy) ## Backwards Compatibility Normally you need to enable Nix flakes in your Nix daemon to take advantage of them. This is great for when you can do that, but sometimes you'll need to make things work for people without flakes enabled. This could happen when needing to graft in a Nix flakes project to one without flakes enabled. There is a library called [flake-compat](https://github.com/edolstra/flake-compat) that makes this easy. Create `default.nix` with the following contents: ```nix (import ( fetchTarball { url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz"; sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; } ) { src = ./.; }).defaultNix ``` And `shell.nix` with the following contents: ```nix (import ( fetchTarball { url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz"; sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; } ) { src = ./.; }).shellNix ``` Then you can use `nix-build` and `nix-shell` like you have in other Nix projects. ## Private Git Repos Nix flakes has native support for private git repositories as inputs. This can be useful when trying to build software you don't want to release as open to the world. To use a private repo, your flake input URL should look something like this: ``` git+ssh://git@github.com:user/repo ``` [I'm pretty sure you could use private git repos outside of flakes, however it was never really clear to me _how_ you end up doing it.](conversation://Cadey/coffee) [I'm told you can bash Niv into shape enough to do this, but yeah it's never really been clear how you do this.](conversation://Mara/hmm) ## Embed NixOS Modules in Flakes The biggest ticket item for me is that it lets you embed NixOS modules in flakes themselves. This lets you define the system configuration for software right next to where the software is defined, thus shipping it as a unit. Using this you can make installing software a matter of adding it to your system's flake, adding the module and then enabling the settings you want to enable. As an example, here is the NixOS module for that IRC bot I mentioned: ```nix nixosModules.bot = { config, lib, ... }: { options.within.services.mara-bot.enable = lib.mkEnableOption "enable Mara bot"; config = lib.mkIf config.within.services.mara-bot.enable { users.groups.mara-bot = { }; users.users.mara-bot = { createHome = true; isSystemUser = true; home = "/var/lib/mara-bot"; group = "mara-bot"; }; systemd.services.mara-bot = { wantedBy = [ "multi-user.target" ]; environment.RUST_LOG = "tower_http=debug,info"; unitConfig.ConditionPathExists = "/var/lib/mara-bot/config.yaml"; serviceConfig = { User = "mara-bot"; Group = "mara-bot"; Restart = "always"; WorkingDirectory = "/var/lib/mara-bot"; ExecStart = "${self.defaultPackage."${system}"}/bin/mara"; }; }; }; }; ``` The key important part here is the `ExecStart` line. It points back to the flake's default package (which is hopefully where the bot's code is defined), and then has systemd manage that. I plan to use this to radically simplify my nixos-configs repo. Right now it has a lot of code that is very project-specific and if I can move that into the projects in question, I can eliminate a lot of code out of the core of my configs repo. ## Embedding Configuration Git Hash into Systems Finally, Nix flakes lets you see the configuration version of a system by embedding it at the build step. Normally NixOS lets you see the following information with `nixos-version --json`: ```json { "nixosVersion": "22.05pre348581.c07b471b52b", "nixpkgsRevision": "c07b471b52be8fbc49a7dc194e9b37a6e19ee04d" } ``` You have the NixOS version and the nixpkgs hash. That doesn't tell you what configuration you are running or anything about it though. However with flakes you can embed the git hash of your configuration into the system config: ```json { "configurationRevision": "f53891121ce4204f57409cbe9e6fcee3b030a350", "nixosVersion": "22.05.20220210.48d63e9", "nixpkgsRevision": "48d63e924a2666baf37f4f14a18f19347fbd54a2" } ``` This can let you make a URL pointing to the commit in that output: ```console $ echo "https://tulpa.dev/cadey/nixos-configs/src/commit/$(ssh logos nixos-version --json | jq -r .configurationRevision)" ``` Which will spit out a link to [cadey/nixos-configs@f53891121](https://tulpa.dev/cadey/nixos-configs/src/commit/f53891121ce4204f57409cbe9e6fcee3b030a350). I'll cover more on how to do this in the NixOS deployment post. --- There is a lot more to get into with each of these topics. I'm only really giving a very high level overview on them while I learn more and migrate over my NixOS configurations to flakes [piecemeal](https://tulpa.dev/cadey/nixos-configs). This has also given me the opportunity to clean things up and chew out a lot of the fat from my NixOS configurations. More to come when it is ready.