xesite/blog/nix-flakes-1-2022-02-21.markdown

---
title: "Nix Flakes: an Introduction"
date: 2022-02-21
tags:
 - nix
 - nixos
series: nix-flakes
author: Twi
---

Nix is a package manager that lets you have a more deterministic view of your
software dependencies and build processes. One if its biggest weaknesses out of
the box is that there are very few conventions on how projects using Nix should
work together. It's like having a build system but also having to configure
systems to run software yourself. This could mean copying a NixOS module out of
the project's git repo, writing your own or more. In contrast to this, [Nix
flakes](https://nixos.wiki/wiki/Flakes) define a set of conventions for how
software can be build, run, integrated and deployed without having to rely on
external tools such as [Niv](https://github.com/nmattia/niv) or
[Lorri](https://github.com/nix-community/lorri) to help you do basic tasks in a
timely manner.

This is going to be a series of posts that will build on eachother. This post
will be an introduction to Nix flakes and serve as a "why should I care?" style
overview of what you can do with flakes without going into too much detail. Most
of these will get separate posts (some more than one post).

In my opinion, here are some of the big reasons you should care about Nix
flakes:

- Flakes adds project templates to Nix
- Flakes define a standard way to say "this is a program you can run"
- Flakes consolidate development environments into project configuration
- Flakes can pull in dependencies from outside git repos trivially
- Flakes can work with people that don't use flakes too
- Flakes supports using private git repos
- Flakes let you define system configuration alongside your application code
- Flakes let you embed the git hash of your configurations repository into
  machines you deploy
  
[Something that may also help you understand why flakes matter is that Nix by
itself is more akin to Dockerfiles. Dockerfiles help you build the software, but
they don't really help you run or operate the software. Nix flakes is more akin
to docker-compose, they help you compose packages written in Nix to run across
machines.](conversation://Mara/happy)

## Project Templates

One of the big annoying parts about getting into Nix is that setting up projects
isn't totally a defined science. Nix configurations just tend to grow
organically and can easily become weird or difficult to understand for people
that didn't start the project. Nix flakes helps fix this by doing a few things:

1. Defining a `flake.nix` as the central "hub" for your project's dependencies,
   exposed packages, NixOS configuration modules [and
   more](https://nixos.wiki/wiki/Flakes#Output_schema).
2. Shipping a [set of templates](https://github.com/NixOS/templates) so that you
   can get projects started easily. Think something like
   [Yeoman](https://yeoman.io) but built directly into Nix. You can also define
   your own templates in your `flake.nix`.
   
As an example that we will use for the rest of this post to help explain it,
let's make a Go project with their Go template. First you will need to enable
Nix flakes on your machine. If you are using NixOS, add this to your
`configuration.nix` file:

```nix
nix = {
  package = pkgs.nixFlakes;
  extraOptions = ''
    experimental-features = nix-command flakes
  '';
};
```

Then rebuild your system and you can continue along with the article.

<xeblog-conv name="Mara" mood="hacker">EDIT: You can use WSL for this. See
[here](/blog/nix-flakes-4-wsl-2022-05-01) for more information.</xeblog-conv>

If you are not on NixOS, you will need to either edit `~/.config/nix/nix.conf`
or `/etc/nix/nix.conf` and add the following line to it:

```
experimental-features = nix-command flakes
```

[You may need to restart the Nix daemon here with `sudo systemctl restart
nix-daemon.service`, but if you are unsure how Nix was set up on that non-NixOS
machine feel free to totally restart your computer.](conversation://Mara/hacker)

Now go to a temporary folder and run these commands to make a folder and create
a new flake from a template:

```console
mkdir ~/tmp/go-demo
cd ~/tmp/go-demo
nix flake new -t templates#go-hello .
git init && git add .
```

This will create a few files in the folder:

```console
$ ls
flake.lock  flake.nix  go.mod  main.go
```

Then you can look at `flake.nix` to see what's up:

```nix
{
  description = "A simple Go package";

  # Nixpkgs / NixOS version to use.
  inputs.nixpkgs.url = "nixpkgs/nixos-21.11";

  outputs = { self, nixpkgs }:
    let

      # Generate a user-friendly version number.
      version = builtins.substring 0 8 self.lastModifiedDate;

      # System types to support.
      supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ];

      # Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'.
      forAllSystems = nixpkgs.lib.genAttrs supportedSystems;

      # Nixpkgs instantiated for supported system types.
      nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; });

    in
    {

      # Provide some binary packages for selected system types.
      packages = forAllSystems (system:
        let
          pkgs = nixpkgsFor.${system};
        in
        {
          # The default package for 'nix build'. This makes sense if the
          # flake provides only one package or there is a clear "main"
          # package.
          default = pkgs.buildGoModule {
            pname = "go-hello";
            inherit version;
            # In 'nix develop', we don't need a copy of the source tree
            # in the Nix store.
            src = ./.;

            # This hash locks the dependencies of this package. It is
            # necessary because of how Go requires network access to resolve
            # VCS.  See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for
            # details. Normally one can build with a fake sha256 and rely on native Go
            # mechanisms to tell you what the hash should be or determine what
            # it should be "out-of-band" with other tooling (eg. gomod2nix).
            # To begin with it is recommended to set this, but one must
            # remeber to bump this hash when your dependencies change.
            #vendorSha256 = pkgs.lib.fakeSha256;

            vendorSha256 = "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo=";
          };
        });
    };
}
```

This defines a single Go package that is supported on macOS and Linux for 64 bit
x86 processors and 64 bit ARM processors.

[In practice this spread should cover all of the main targets you'll need to
care about for local development and cloud
deployment.](conversation://Mara/hacker)

You can then build the flake with `nix build`:

```console
$ nix build
```

And then run it:

```console
$ ./result/bin/go-hello
Hello Nix!
```

## Standard Default Package

Let's take a closer look at the higher level things in the flake:

```nix
{
  description = "A simple Go package";
  
  inputs.nixpkgs.url = "nixpkgs/nixos-21.11";
  
  outputs = { self, nixpkgs }: {
    packages = { ... };
  };
}
```

[A note: in the rest of this article (and series of articles), when I refer to a
"flake output", I am referring to an attribute in the `outputs` attribute of
your `flake.nix`. Ditto with "flake input" referring to the `inputs` attribute
of your `flake.nix`.](conversation://Cadey/enby)

When you ran `nix build` earlier, it defaulted to building the package in
`defaultPackage`. You can also build the `go-hello` package by running this
command:

```console
$ nix build .#go-hello
```

And if you want to build the copy I made for this post:

```console
$ nix build github:Xe/gohello
$ ./result/bin/go-hello
Hello reader!
```

A standard default package means that you can more easily build software without
having to read documentation on what file to build. `nix build` will Just Work™️.

## Exposing Packages as Applications

Additionally, you can expose a package as an application. This allows you to
simplify that above `nix build` and `./result/bin/go-hello` cycle into a single
`nix run` command. Open `flake.nix` in your favorite editor and let's configure
`go-hello` to be the default app:

```nix
# below defaultPackage

defaultApp = forAllSystems (system: {
  type = "app";
  program = "${self.packages.${system}.go-hello}/bin/go-hello";
});
```

Then you can run it with `nix run`:

```console
$ nix run
Hello Nix!
```

Or you can run my copy:

```console
$ nix run github:Xe/gohello/main
Hello reader!
```

[What is that extra part of the URL path for? Is that a git
branch?](conversation://Mara/hmm)

[Yes, you can use that syntax to set the git branch that Nix should build from.
By default it will use the default branch (typically `main`), but sometimes you
need to specify a branch or commit directly.](conversation://Cadey/enby)

## Development Environment Configuration

One of Nix's superpowers is the ability to declaratively manage the development
environment for a project so that you can be sure that everyone working on the
project is using the same tools. 

[I use this with all of my projects to the point that when I am outside of a
project folder I do not have any development tools
available.](conversation://Cadey/enby)

Flakes has the ability to specify this using the `devShell` flake output. You
can add it to your `flake.nix` using this:

```nix
# after defaultApp

devShell = forAllSystems (system:
  let pkgs = nixpkgsFor.${system};
  in pkgs.mkShell {
    buildInputs = with pkgs; [ go gopls goimports go-tools ];
  });
```

[I consider this to be a basic Go development environment. It includes standard
tools such as the language server, `goimports` for better formatting and tools
like <a href="https://staticcheck.io">staticcheck</a>. If you use staticcheck
regularly at work, please consider throwing <a
href="https://github.com/users/dominikh/sponsorship">Dominik</a> a couple bucks
a month if you find it useful. It helps the project be more
self-sustaining.](conversation://Mara/happy)

Then you can enter the development shell with `nix develop`:

```
$ nix develop

[cadey@pneuma:~/tmp/gohello]$ go version
go version go1.16.9 linux/amd64
```

And then hack at your project all you want. You can send this git repo to a
friend and they will have the same setup.

## External Dependencies

Now let's talk about inputs. Flake inputs let you add external dependencies to a
project. As an example, let's look at the `nixpkgs` input:

```nix
# Nixpkgs / NixOS version to use.
inputs.nixpkgs.url = "nixpkgs/nixos-21.11";
```

This defines the release of nixpkgs that should be used for the project. This
template defaults to NixOS 21.11's version of nixpkgs, however we can upgrade it
to nixos-unstable by changing it to this:

```nix
# Nixpkgs / NixOS version to use.
inputs.nixpkgs.url = "nixpkgs/nixos-unstable";
```

Then we can run `nix flake update` and then `nix develop` and see that we are
running a newer version of Go:

```console
$ nix flake update
warning: updating lock file '/home/cadey/tmp/gohello/flake.lock':
• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/77aa71f66fd05d9e7b7d1c084865d703a8008ab7' (2022-01-19)
  → 'github:NixOS/nixpkgs/2128d0aa28edef51fd8fef38b132ffc0155595df' (2022-02-16)
  
$ nix develop

[cadey@pneuma:~/tmp/gohello]$ go version
go version go1.17.7 linux/amd64
```

This also lets you pull in other Nix flakes projects, such as my CSS framework
[Xess](https://github.com/Xe/Xess):

```nix
inputs.xess.url = "github:Xe/Xess";
inputs.xess.inputs.nixpkgs.follows = "nixpkgs";
```

[Why is that second line needed?](conversation://Mara/hmm)

[By default when you pull in another project with Nix flakes, it treats that
project as an entirely separate universe and only interacts with the outputs of
that flake. This means it pulls in its own version of nixpkgs, each dependency
it has can pull in that own version of nixpkgs and vice versa ad infinitum. By
making Xess' nixpkgs input follows our own one, we are saying "I understand this
may be incompatible, but please use this version of nixpkgs instead". This can
help larger projects with many inputs (such as a nixos configs repo made by
someone with too many throwaway side projects) evaluate and build faster. Nix
flakes does have a cached evaluator, but still it helps to avoid the problem in
the first place.](conversation://Cadey/enby)

Or anything you want! A useful library to pull in is
[flake-utils](https://github.com/numtide/flake-utils), that can help you
simplify your `flake.nix` and get rid of those ugly `forAllSystems` and
`nixpkgsFor` functions in the `flake.nix` that this post used by default. For an
example of a flake that uses this library, see [this
`flake.nix`](https://tulpa.dev/Xe/mara/src/branch/main/flake.nix) from the IRC
bot that lives in [`#xeserv`](https://web.libera.chat/#xeserv).

[Adapting this trivial example to use `flake-utils` is an excellent exercise for
the reader! This library should really be shipped with flakes by
default.](conversation://Mara/happy)

## Backwards Compatibility

Normally you need to enable Nix flakes in your Nix daemon to take advantage of
them. This is great for when you can do that, but sometimes you'll need to make
things work for people without flakes enabled. This could happen when needing to
graft in a Nix flakes project to one without flakes enabled. There is a library
called [flake-compat](https://github.com/edolstra/flake-compat) that makes this
easy.

Create `default.nix` with the following contents:

```nix
(import (
  fetchTarball {
    url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz";
    sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; }
) {
  src =  ./.;
}).defaultNix
```

And `shell.nix` with the following contents:

```nix
(import (
  fetchTarball {
    url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz";
    sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; }
) {
  src =  ./.;
}).shellNix
```

Then you can use `nix-build` and `nix-shell` like you have in other Nix
projects.

## Private Git Repos

Nix flakes has native support for private git repositories as inputs. This can
be useful when trying to build software you don't want to release as open to the
world. To use a private repo, your flake input URL should look something like
this:

```
git+ssh://git@github.com:user/repo
```

[I'm pretty sure you could use private git repos outside of flakes, however it
was never really clear to me _how_ you end up doing
it.](conversation://Cadey/coffee)

[I'm told you can bash Niv into shape enough to do this, but yeah it's never
really been clear how you do this.](conversation://Mara/hmm)

## Embed NixOS Modules in Flakes

The biggest ticket item for me is that it lets you embed NixOS modules in flakes
themselves. This lets you define the system configuration for software right
next to where the software is defined, thus shipping it as a unit. Using this
you can make installing software a matter of adding it to your system's flake,
adding the module and then enabling the settings you want to enable.

As an example, here is the NixOS module for that IRC bot I mentioned:

```nix
nixosModules.bot = { config, lib, ... }: {
  options.within.services.mara-bot.enable =
    lib.mkEnableOption "enable Mara bot";

  config = lib.mkIf config.within.services.mara-bot.enable {
    users.groups.mara-bot = { };

    users.users.mara-bot = {
      createHome = true;
      isSystemUser = true;
      home = "/var/lib/mara-bot";
      group = "mara-bot";
    };

    systemd.services.mara-bot = {
      wantedBy = [ "multi-user.target" ];
      environment.RUST_LOG = "tower_http=debug,info";
      unitConfig.ConditionPathExists = "/var/lib/mara-bot/config.yaml";
      serviceConfig = {
        User = "mara-bot";
        Group = "mara-bot";
        Restart = "always";
        WorkingDirectory = "/var/lib/mara-bot";
        ExecStart = "${self.packages."${system}".default}/bin/mara";
      };
    };
  };
};
```

The key important part here is the `ExecStart` line. It points back to the
flake's default package (which is hopefully where the bot's code is defined),
and then has systemd manage that.

I plan to use this to radically simplify my nixos-configs repo. Right now it has
a lot of code that is very project-specific and if I can move that into the
projects in question, I can eliminate a lot of code out of the core of my
configs repo.

## Embedding Configuration Git Hash into Systems

Finally, Nix flakes lets you see the configuration version of a system by
embedding it at the build step. Normally NixOS lets you see the following
information with `nixos-version --json`:

```json
{
  "nixosVersion": "22.05pre348581.c07b471b52b",
  "nixpkgsRevision": "c07b471b52be8fbc49a7dc194e9b37a6e19ee04d"
}
```

You have the NixOS version and the nixpkgs hash. That doesn't tell you what
configuration you are running or anything about it though. However with flakes
you can embed the git hash of your configuration into the system config:

```json
{
  "configurationRevision": "f53891121ce4204f57409cbe9e6fcee3b030a350",
  "nixosVersion": "22.05.20220210.48d63e9",
  "nixpkgsRevision": "48d63e924a2666baf37f4f14a18f19347fbd54a2"
}
```

This can let you make a URL pointing to the commit in that output:

```console
$ echo "https://tulpa.dev/cadey/nixos-configs/src/commit/$(ssh logos nixos-version --json | jq -r .configurationRevision)"
```

Which will spit out a link to
[cadey/nixos-configs@f53891121](https://tulpa.dev/cadey/nixos-configs/src/commit/f53891121ce4204f57409cbe9e6fcee3b030a350).

I'll cover more on how to do this in the NixOS deployment post.

---

There is a lot more to get into with each of these topics. I'm only really
giving a very high level overview on them while I learn more and migrate over my
NixOS configurations to flakes
[piecemeal](https://tulpa.dev/cadey/nixos-configs). This has also given me the
opportunity to clean things up and chew out a lot of the fat from my NixOS
configurations. More to come when it is ready.