forked from cadey/xesite
529 lines
18 KiB
Markdown
529 lines
18 KiB
Markdown
---
|
||
title: "Nix Flakes: an Introduction"
|
||
date: 2022-02-21
|
||
tags:
|
||
- nix
|
||
- nixos
|
||
series: nix-flakes
|
||
author: Twi
|
||
---
|
||
|
||
Nix is a package manager that lets you have a more deterministic view of your
|
||
software dependencies and build processes. One if its biggest weaknesses out of
|
||
the box is that there are very few conventions on how projects using Nix should
|
||
work together. It's like having a build system but also having to configure
|
||
systems to run software yourself. This could mean copying a NixOS module out of
|
||
the project's git repo, writing your own or more. In contrast to this, [Nix
|
||
flakes](https://nixos.wiki/wiki/Flakes) define a set of conventions for how
|
||
software can be build, run, integrated and deployed without having to rely on
|
||
external tools such as [Niv](https://github.com/nmattia/niv) or
|
||
[Lorri](https://github.com/nix-community/lorri) to help you do basic tasks in a
|
||
timely manner.
|
||
|
||
This is going to be a series of posts that will build on eachother. This post
|
||
will be an introduction to Nix flakes and serve as a "why should I care?" style
|
||
overview of what you can do with flakes without going into too much detail. Most
|
||
of these will get separate posts (some more than one post).
|
||
|
||
In my opinion, here are some of the big reasons you should care about Nix
|
||
flakes:
|
||
|
||
- Flakes adds project templates to Nix
|
||
- Flakes define a standard way to say "this is a program you can run"
|
||
- Flakes consolidate development environments into project configuration
|
||
- Flakes can pull in dependencies from outside git repos trivially
|
||
- Flakes can work with people that don't use flakes too
|
||
- Flakes supports using private git repos
|
||
- Flakes let you define system configuration alongside your application code
|
||
- Flakes let you embed the git hash of your configurations repository into
|
||
machines you deploy
|
||
|
||
[Something that may also help you understand why flakes matter is that Nix by
|
||
itself is more akin to Dockerfiles. Dockerfiles help you build the software, but
|
||
they don't really help you run or operate the software. Nix flakes is more akin
|
||
to docker-compose, they help you compose packages written in Nix to run across
|
||
machines.](conversation://Mara/happy)
|
||
|
||
## Project Templates
|
||
|
||
One of the big annoying parts about getting into Nix is that setting up projects
|
||
isn't totally a defined science. Nix configurations just tend to grow
|
||
organically and can easily become weird or difficult to understand for people
|
||
that didn't start the project. Nix flakes helps fix this by doing a few things:
|
||
|
||
1. Defining a `flake.nix` as the central "hub" for your project's dependencies,
|
||
exposed packages, NixOS configuration modules [and
|
||
more](https://nixos.wiki/wiki/Flakes#Output_schema).
|
||
2. Shipping a [set of templates](https://github.com/NixOS/templates) so that you
|
||
can get projects started easily. Think something like
|
||
[Yeoman](https://yeoman.io) but built directly into Nix. You can also define
|
||
your own templates in your `flake.nix`.
|
||
|
||
As an example that we will use for the rest of this post to help explain it,
|
||
let's make a Go project with their Go template. First you will need to enable
|
||
Nix flakes on your machine. If you are using NixOS, add this to your
|
||
`configuration.nix` file:
|
||
|
||
```nix
|
||
nix = {
|
||
package = pkgs.nixFlakes;
|
||
extraOptions = ''
|
||
experimental-features = nix-command flakes
|
||
'';
|
||
};
|
||
```
|
||
|
||
Then rebuild your system and you can continue along with the article.
|
||
|
||
<xeblog-conv name="Mara" mood="hacker">EDIT: You can use WSL for this. See
|
||
[here](/blog/nix-flakes-4-wsl-2022-05-01) for more information.</xeblog-conv>
|
||
|
||
If you are not on NixOS, you will need to either edit `~/.config/nix/nix.conf`
|
||
or `/etc/nix/nix.conf` and add the following line to it:
|
||
|
||
```
|
||
experimental-features = nix-command flakes
|
||
```
|
||
|
||
[You may need to restart the Nix daemon here with `sudo systemctl restart
|
||
nix-daemon.service`, but if you are unsure how Nix was set up on that non-NixOS
|
||
machine feel free to totally restart your computer.](conversation://Mara/hacker)
|
||
|
||
Now go to a temporary folder and run these commands to make a folder and create
|
||
a new flake from a template:
|
||
|
||
```console
|
||
mkdir ~/tmp/go-demo
|
||
cd ~/tmp/go-demo
|
||
nix flake new -t templates#go-hello .
|
||
git init && git add .
|
||
```
|
||
|
||
This will create a few files in the folder:
|
||
|
||
```console
|
||
$ ls
|
||
flake.lock flake.nix go.mod main.go
|
||
```
|
||
|
||
Then you can look at `flake.nix` to see what's up:
|
||
|
||
```nix
|
||
{
|
||
description = "A simple Go package";
|
||
|
||
# Nixpkgs / NixOS version to use.
|
||
inputs.nixpkgs.url = "nixpkgs/nixos-21.11";
|
||
|
||
outputs = { self, nixpkgs }:
|
||
let
|
||
|
||
# Generate a user-friendly version number.
|
||
version = builtins.substring 0 8 self.lastModifiedDate;
|
||
|
||
# System types to support.
|
||
supportedSystems = [ "x86_64-linux" "x86_64-darwin" "aarch64-linux" "aarch64-darwin" ];
|
||
|
||
# Helper function to generate an attrset '{ x86_64-linux = f "x86_64-linux"; ... }'.
|
||
forAllSystems = nixpkgs.lib.genAttrs supportedSystems;
|
||
|
||
# Nixpkgs instantiated for supported system types.
|
||
nixpkgsFor = forAllSystems (system: import nixpkgs { inherit system; });
|
||
|
||
in
|
||
{
|
||
|
||
# Provide some binary packages for selected system types.
|
||
packages = forAllSystems (system:
|
||
let
|
||
pkgs = nixpkgsFor.${system};
|
||
in
|
||
{
|
||
# The default package for 'nix build'. This makes sense if the
|
||
# flake provides only one package or there is a clear "main"
|
||
# package.
|
||
default = pkgs.buildGoModule {
|
||
pname = "go-hello";
|
||
inherit version;
|
||
# In 'nix develop', we don't need a copy of the source tree
|
||
# in the Nix store.
|
||
src = ./.;
|
||
|
||
# This hash locks the dependencies of this package. It is
|
||
# necessary because of how Go requires network access to resolve
|
||
# VCS. See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for
|
||
# details. Normally one can build with a fake sha256 and rely on native Go
|
||
# mechanisms to tell you what the hash should be or determine what
|
||
# it should be "out-of-band" with other tooling (eg. gomod2nix).
|
||
# To begin with it is recommended to set this, but one must
|
||
# remeber to bump this hash when your dependencies change.
|
||
#vendorSha256 = pkgs.lib.fakeSha256;
|
||
|
||
vendorSha256 = "sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo=";
|
||
};
|
||
});
|
||
};
|
||
}
|
||
```
|
||
|
||
This defines a single Go package that is supported on macOS and Linux for 64 bit
|
||
x86 processors and 64 bit ARM processors.
|
||
|
||
[In practice this spread should cover all of the main targets you'll need to
|
||
care about for local development and cloud
|
||
deployment.](conversation://Mara/hacker)
|
||
|
||
You can then build the flake with `nix build`:
|
||
|
||
```console
|
||
$ nix build
|
||
```
|
||
|
||
And then run it:
|
||
|
||
```console
|
||
$ ./result/bin/go-hello
|
||
Hello Nix!
|
||
```
|
||
|
||
## Standard Default Package
|
||
|
||
Let's take a closer look at the higher level things in the flake:
|
||
|
||
```nix
|
||
{
|
||
description = "A simple Go package";
|
||
|
||
inputs.nixpkgs.url = "nixpkgs/nixos-21.11";
|
||
|
||
outputs = { self, nixpkgs }: {
|
||
packages = { ... };
|
||
};
|
||
}
|
||
```
|
||
|
||
[A note: in the rest of this article (and series of articles), when I refer to a
|
||
"flake output", I am referring to an attribute in the `outputs` attribute of
|
||
your `flake.nix`. Ditto with "flake input" referring to the `inputs` attribute
|
||
of your `flake.nix`.](conversation://Cadey/enby)
|
||
|
||
When you ran `nix build` earlier, it defaulted to building the `default` entry
|
||
in `packages`. You can also build the `default` package by running this
|
||
command:
|
||
|
||
```console
|
||
$ nix build .#default
|
||
```
|
||
|
||
And if you want to build the copy I made for this post:
|
||
|
||
```console
|
||
$ nix build github:Xe/gohello
|
||
$ ./result/bin/go-hello
|
||
Hello reader!
|
||
```
|
||
|
||
A standard default package means that you can more easily build software without
|
||
having to read documentation on what file to build. `nix build` will Just Work™️.
|
||
|
||
## Exposing Packages as Applications
|
||
|
||
Additionally, you can expose a package as an application. This allows you to
|
||
simplify that above `nix build` and `./result/bin/go-hello` cycle into a single
|
||
`nix run` command. Open `flake.nix` in your favorite editor and let's configure
|
||
`go-hello` to be the default app:
|
||
|
||
```nix
|
||
# below packages
|
||
|
||
apps = forAllSystems (system: {
|
||
default = {
|
||
type = "app";
|
||
program = "${self.packages.${system}.default}/bin/go-hello";
|
||
};
|
||
});
|
||
```
|
||
|
||
Then you can run it with `nix run`:
|
||
|
||
```console
|
||
$ nix run
|
||
Hello Nix!
|
||
```
|
||
|
||
Or you can run my copy:
|
||
|
||
```console
|
||
$ nix run github:Xe/gohello/main
|
||
Hello reader!
|
||
```
|
||
|
||
[What is that extra part of the URL path for? Is that a git
|
||
branch?](conversation://Mara/hmm)
|
||
|
||
[Yes, you can use that syntax to set the git branch that Nix should build from.
|
||
By default it will use the default branch (typically `main`), but sometimes you
|
||
need to specify a branch or commit directly.](conversation://Cadey/enby)
|
||
|
||
## Development Environment Configuration
|
||
|
||
One of Nix's superpowers is the ability to declaratively manage the development
|
||
environment for a project so that you can be sure that everyone working on the
|
||
project is using the same tools.
|
||
|
||
[I use this with all of my projects to the point that when I am outside of a
|
||
project folder I do not have any development tools
|
||
available.](conversation://Cadey/enby)
|
||
|
||
Flakes has the ability to specify this using the `devShell` flake output. You
|
||
can add it to your `flake.nix` using this:
|
||
|
||
```nix
|
||
# after defaultApp
|
||
|
||
devShell = forAllSystems (system:
|
||
let pkgs = nixpkgsFor.${system};
|
||
in {
|
||
default = pkgs.mkShell {
|
||
buildInputs = with pkgs; [ go gopls gotools go-tools ];
|
||
};
|
||
});
|
||
```
|
||
|
||
[I consider this to be a basic Go development environment. It includes standard
|
||
tools such as the language server, `goimports` for better formatting and tools
|
||
like <a href="https://staticcheck.io">staticcheck</a>. If you use staticcheck
|
||
regularly at work, please consider throwing <a
|
||
href="https://github.com/users/dominikh/sponsorship">Dominik</a> a couple bucks
|
||
a month if you find it useful. It helps the project be more
|
||
self-sustaining.](conversation://Mara/happy)
|
||
|
||
Then you can enter the development shell with `nix develop`:
|
||
|
||
```
|
||
$ nix develop
|
||
|
||
[cadey@pneuma:~/tmp/gohello]$ go version
|
||
go version go1.16.9 linux/amd64
|
||
```
|
||
|
||
And then hack at your project all you want. You can send this git repo to a
|
||
friend and they will have the same setup.
|
||
|
||
## External Dependencies
|
||
|
||
Now let's talk about inputs. Flake inputs let you add external dependencies to a
|
||
project. As an example, let's look at the `nixpkgs` input:
|
||
|
||
```nix
|
||
# Nixpkgs / NixOS version to use.
|
||
inputs.nixpkgs.url = "nixpkgs/nixos-21.11";
|
||
```
|
||
|
||
This defines the release of nixpkgs that should be used for the project. This
|
||
template defaults to NixOS 21.11's version of nixpkgs, however we can upgrade it
|
||
to nixos-unstable by changing it to this:
|
||
|
||
```nix
|
||
# Nixpkgs / NixOS version to use.
|
||
inputs.nixpkgs.url = "nixpkgs/nixos-unstable";
|
||
```
|
||
|
||
Then we can run `nix flake update` and then `nix develop` and see that we are
|
||
running a newer version of Go:
|
||
|
||
```console
|
||
$ nix flake update
|
||
warning: updating lock file '/home/cadey/tmp/gohello/flake.lock':
|
||
• Updated input 'nixpkgs':
|
||
'github:NixOS/nixpkgs/77aa71f66fd05d9e7b7d1c084865d703a8008ab7' (2022-01-19)
|
||
→ 'github:NixOS/nixpkgs/2128d0aa28edef51fd8fef38b132ffc0155595df' (2022-02-16)
|
||
|
||
$ nix develop
|
||
|
||
[cadey@pneuma:~/tmp/gohello]$ go version
|
||
go version go1.17.7 linux/amd64
|
||
```
|
||
|
||
This also lets you pull in other Nix flakes projects, such as my CSS framework
|
||
[Xess](https://github.com/Xe/Xess):
|
||
|
||
```nix
|
||
inputs.xess.url = "github:Xe/Xess";
|
||
inputs.xess.inputs.nixpkgs.follows = "nixpkgs";
|
||
```
|
||
|
||
[Why is that second line needed?](conversation://Mara/hmm)
|
||
|
||
[By default when you pull in another project with Nix flakes, it treats that
|
||
project as an entirely separate universe and only interacts with the outputs of
|
||
that flake. This means it pulls in its own version of nixpkgs, each dependency
|
||
it has can pull in that own version of nixpkgs and vice versa ad infinitum. By
|
||
making Xess' nixpkgs input follows our own one, we are saying "I understand this
|
||
may be incompatible, but please use this version of nixpkgs instead". This can
|
||
help larger projects with many inputs (such as a nixos configs repo made by
|
||
someone with too many throwaway side projects) evaluate and build faster. Nix
|
||
flakes does have a cached evaluator, but still it helps to avoid the problem in
|
||
the first place.](conversation://Cadey/enby)
|
||
|
||
Or anything you want! A useful library to pull in is
|
||
[flake-utils](https://github.com/numtide/flake-utils), that can help you
|
||
simplify your `flake.nix` and get rid of those ugly `forAllSystems` and
|
||
`nixpkgsFor` functions in the `flake.nix` that this post used by default. For an
|
||
example of a flake that uses this library, see [this
|
||
`flake.nix`](https://tulpa.dev/Xe/mara/src/branch/main/flake.nix) from the IRC
|
||
bot that lives in [`#xeserv`](https://web.libera.chat/#xeserv).
|
||
|
||
[Adapting this trivial example to use `flake-utils` is an excellent exercise for
|
||
the reader! This library should really be shipped with flakes by
|
||
default.](conversation://Mara/happy)
|
||
|
||
## Backwards Compatibility
|
||
|
||
Normally you need to enable Nix flakes in your Nix daemon to take advantage of
|
||
them. This is great for when you can do that, but sometimes you'll need to make
|
||
things work for people without flakes enabled. This could happen when needing to
|
||
graft in a Nix flakes project to one without flakes enabled. There is a library
|
||
called [flake-compat](https://github.com/edolstra/flake-compat) that makes this
|
||
easy.
|
||
|
||
Create `default.nix` with the following contents:
|
||
|
||
```nix
|
||
(import (
|
||
fetchTarball {
|
||
url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz";
|
||
sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; }
|
||
) {
|
||
src = ./.;
|
||
}).defaultNix
|
||
```
|
||
|
||
And `shell.nix` with the following contents:
|
||
|
||
```nix
|
||
(import (
|
||
fetchTarball {
|
||
url = "https://github.com/edolstra/flake-compat/archive/99f1c2157fba4bfe6211a321fd0ee43199025dbf.tar.gz";
|
||
sha256 = "0x2jn3vrawwv9xp15674wjz9pixwjyj3j771izayl962zziivbx2"; }
|
||
) {
|
||
src = ./.;
|
||
}).shellNix
|
||
```
|
||
|
||
Then you can use `nix-build` and `nix-shell` like you have in other Nix
|
||
projects.
|
||
|
||
## Private Git Repos
|
||
|
||
Nix flakes has native support for private git repositories as inputs. This can
|
||
be useful when trying to build software you don't want to release as open to the
|
||
world. To use a private repo, your flake input URL should look something like
|
||
this:
|
||
|
||
```
|
||
git+ssh://git@github.com/user/repo?ref=main
|
||
```
|
||
|
||
[I'm pretty sure you could use private git repos outside of flakes, however it
|
||
was never really clear to me _how_ you end up doing
|
||
it.](conversation://Cadey/coffee)
|
||
|
||
[I'm told you can bash Niv into shape enough to do this, but yeah it's never
|
||
really been clear how you do this.](conversation://Mara/hmm)
|
||
|
||
## Embed NixOS Modules in Flakes
|
||
|
||
The biggest ticket item for me is that it lets you embed NixOS modules in flakes
|
||
themselves. This lets you define the system configuration for software right
|
||
next to where the software is defined, thus shipping it as a unit. Using this
|
||
you can make installing software a matter of adding it to your system's flake,
|
||
adding the module and then enabling the settings you want to enable.
|
||
|
||
As an example, here is the NixOS module for that IRC bot I mentioned:
|
||
|
||
```nix
|
||
nixosModules.bot = { config, lib, ... }: {
|
||
options.within.services.mara-bot.enable =
|
||
lib.mkEnableOption "enable Mara bot";
|
||
|
||
config = lib.mkIf config.within.services.mara-bot.enable {
|
||
users.groups.mara-bot = { };
|
||
|
||
users.users.mara-bot = {
|
||
createHome = true;
|
||
isSystemUser = true;
|
||
home = "/var/lib/mara-bot";
|
||
group = "mara-bot";
|
||
};
|
||
|
||
systemd.services.mara-bot = {
|
||
wantedBy = [ "multi-user.target" ];
|
||
environment.RUST_LOG = "tower_http=debug,info";
|
||
unitConfig.ConditionPathExists = "/var/lib/mara-bot/config.yaml";
|
||
serviceConfig = {
|
||
User = "mara-bot";
|
||
Group = "mara-bot";
|
||
Restart = "always";
|
||
WorkingDirectory = "/var/lib/mara-bot";
|
||
ExecStart = "${self.packages."${system}".default}/bin/mara";
|
||
};
|
||
};
|
||
};
|
||
};
|
||
```
|
||
|
||
The key important part here is the `ExecStart` line. It points back to the
|
||
flake's default package (which is hopefully where the bot's code is defined),
|
||
and then has systemd manage that.
|
||
|
||
I plan to use this to radically simplify my nixos-configs repo. Right now it has
|
||
a lot of code that is very project-specific and if I can move that into the
|
||
projects in question, I can eliminate a lot of code out of the core of my
|
||
configs repo.
|
||
|
||
## Embedding Configuration Git Hash into Systems
|
||
|
||
Finally, Nix flakes lets you see the configuration version of a system by
|
||
embedding it at the build step. Normally NixOS lets you see the following
|
||
information with `nixos-version --json`:
|
||
|
||
```json
|
||
{
|
||
"nixosVersion": "22.05pre348581.c07b471b52b",
|
||
"nixpkgsRevision": "c07b471b52be8fbc49a7dc194e9b37a6e19ee04d"
|
||
}
|
||
```
|
||
|
||
You have the NixOS version and the nixpkgs hash. That doesn't tell you what
|
||
configuration you are running or anything about it though. However with flakes
|
||
you can embed the git hash of your configuration into the system config:
|
||
|
||
```json
|
||
{
|
||
"configurationRevision": "f53891121ce4204f57409cbe9e6fcee3b030a350",
|
||
"nixosVersion": "22.05.20220210.48d63e9",
|
||
"nixpkgsRevision": "48d63e924a2666baf37f4f14a18f19347fbd54a2"
|
||
}
|
||
```
|
||
|
||
This can let you make a URL pointing to the commit in that output:
|
||
|
||
```console
|
||
$ echo "https://tulpa.dev/cadey/nixos-configs/src/commit/$(ssh logos nixos-version --json | jq -r .configurationRevision)"
|
||
```
|
||
|
||
Which will spit out a link to
|
||
[cadey/nixos-configs@f53891121](https://tulpa.dev/cadey/nixos-configs/src/commit/f53891121ce4204f57409cbe9e6fcee3b030a350).
|
||
|
||
I'll cover more on how to do this in the NixOS deployment post.
|
||
|
||
---
|
||
|
||
There is a lot more to get into with each of these topics. I'm only really
|
||
giving a very high level overview on them while I learn more and migrate over my
|
||
NixOS configurations to flakes
|
||
[piecemeal](https://tulpa.dev/cadey/nixos-configs). This has also given me the
|
||
opportunity to clean things up and chew out a lot of the fat from my NixOS
|
||
configurations. More to come when it is ready.
|