18 KiB
title | date | tags | series | vod | ||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Nix Flakes: Packages and How to Use Them | 2022-02-27 |
|
nix-flakes |
|
A package is a bundle of files. These files could be program executables, resources such as stylesheets or images, or even a container image. Most of the time you don't deal with packages directly and instead you use a package manager (a program whose sole goal in life is to deal with packages) to do actions for you. This post is going to cover how to define packages in Nix and how Nix flakes let you manage multiple packages per project more easily.
What is a Package?
In Nix, you build packages by creating derivations that define the build steps and associated inputs (such as the compiler) to end up with the resulting outputs (derivation being the product of deriving something). Consider a package like this:
# hello-shell.nix
with import <nixpkgs> { };
stdenv.mkDerivation {
name = "hello-HEAD";
src = ./.;
installPhase = ''
echo "Hello" > $out
'';
}
Then we can build this package with nix-build hello-shell.nix
and a result
symlink will show up in your current working directory. Then you can view what
it says with cat
:
$ cat ./result
Hello
This is all it takes to make a Nix package. You need to name the package, give it input source code somehow, and potentially give it build instructions. Everything else we'll cover today will build on top of this.
Let's look back at the Go example package I walked us through in the last post:
# ...
packages = forAllSystems (system:
let pkgs = nixpkgsFor.${system};
in {
go-hello = pkgs.buildGoModule {
pname = "go-hello";
inherit version;
# In 'nix develop', we don't need a copy of the source tree
# in the Nix store.
src = ./.;
# This hash locks the dependencies of this package. It is
# necessary because of how Go requires network access to resolve
# VCS. See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for
# details. Normally one can build with a fake sha256 and rely on native Go
# mechanisms to tell you what the hash should be or determine what
# it should be "out-of-band" with other tooling (eg. gomod2nix).
# To begin with it is recommended to set this, but one must
# remeber to bump this hash when your dependencies change.
vendorSha256 =
"sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo=";
};
});
# ...
This uses a different builder, one called
pkgs.buildGoModule
.
This is like the stdenv.mkDerivation
builder, except it is explicitly made to
handle Go projects. There are some other flags that you can set in
buildGoModule
that can be useful. You can see examples in the NixOS manual
page here.
Another useful builder is Naersk.
Naersk will automatically derive build instructions for Rust projects using the
Cargo.toml
and Cargo.lock
files. This means that your build step can look as
small as this:
naersk-lib.buildPackage ./.
You can think of these builders as templates for doing larger builds. This is kinda like the ONBUILD Dockerfile instruction, but it isn't limited to Docker. The main difference is that Nix builds are more like functions (inputs and outputs) and Docker builds focus on the individual commands you run to get the result you want. Both eventually compile down to shell commands anyways!
A More Useful Package
This "hello world" program isn't very useful on its own, however we can use it as the basis for making something a bit more useful. I have made a template for a "Hello world" HTTP server here. Let's make a new folder for it and then initialize it:
If you want to make your own templates, see how to do that here.
mkdir -p ~/tmp/gohello-http
cd ~/tmp/gohello-http
git init
nix flake init -t github:Xe/templates#go-web-server
You may see a message from direnv about needing to approve its content. This will use Nix flake's cached interpreter to give you all the advantages of something like Lorri without having to install Lorri.
Then make an initial commit and run it:
git add .
git commit -sm "initial commit"
nix build
./result/bin/web-server
Why are you using git add .
everywhere? Shouldn't the files be picked up
implicitly?
Or you can run it directly with nix run
:
nix run
Docker Images
Most of the time you will build software with Nix, however that doesn't stop you from building things like Docker images with Nix. Remember that you can have the output of any shell commands be run in a Nix build (the only catch is that they can't access the internet directly), so you can build a Docker image out of that web server template by defining another package:
# flake.nix
packages = {
default = ...;
docker = let
web = self.packages.${system}.default;
in pkgs.dockerTools.buildLayeredImage {
name = web.pname;
tag = web.version;
contents = [ web ];
config = {
Cmd = [ "/bin/web-server" ];
WorkingDir = "/";
};
};
};
This will build a Docker image with the web-server binary in it. To build it, run these commands:
git add .
nix build .#docker
What's with that last argument to nix build
, won't that be read as a shell
comment?
It will put the resulting docker image in ./result
. To load it into docker use
the following command:
$ docker load < result
Loaded image: web-server:20220227
Then you can run it with docker run
:
docker run -itp 3031:3031 web-server:20220227
Then poke it with curl:
$ curl http://[::]:3031
hello from nix!
You can push this image to the Docker hub like any other image. Another cool thing about this is that when you update the program, it'll only actually load the images that changed. Let's edit the hello world message:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "hello from nix building a docker image!")
})
And then re-build and load it into Docker:
git add .
nix build .#docker
docker load < result
systemd Portable Services
systemd Portable Services function like Docker, but they work at the systemd level and allow you to integrate into systemd instead of running on the side of it. This gives you access to systemd's readiness signaling, logging pipeline and dependency graph so that you can integrate like a native service. They are like containers, but without a lot of the headaches around networking, stateful storage and logging. They are just systemd services at their core.
There is currently an open pull request for adding Portable Service building support to nixpkgs, however we can mess around with it today thanks to my portable-svc overlay that copies in the contents of that pull request.
To make this into a portable service, first we need to add my overlay to the flake inputs:
# flake.nix
inputs = {
nixpkgs.url = "nixpkgs/nixos-unstable";
utils.url = "github:numtide/flake-utils";
portable-svc.url = "git+https://tulpa.dev/cadey/portable-svc.git?ref=main";
};
Then add it as an argument to the outputs
function:
outputs = { self, nixpkgs, utils, portable-svc }:
And then change how we are importing the pkgs
variable. The pkgs
variable
we're currently using is imported like this:
let pkgs = nixpkgs.legacyPackages.${system};
This works, however there isn't a way to specify an overlay into this. We need to change this into a manual import of nixpkgs with the overlay specified, like this:
let pkgs = import nixpkgs {
overlays = [ portable-svc.overlay ];
inherit system;
};
This will let us use the portableService
function in Nix package definitions.
Next we need to make a systemd service unit for the web server. The exact path
to the program binary can and will change with every build, so it would be good
to have this templated. Make a folder called systemd
:
mkdir systemd
And put the following contents in systemd/web-server.service.in
:
[Unit]
Description=A web service
[Service]
DynamicUser=yes
ExecStart=@web@/bin/web-server
[Install]
WantedBy=multi-user.target
Then under the docker package definition, add the package that will template out the systemd unit:
web-service = pkgs.substituteAll {
name = "web-server.service";
src = ./systemd/web-server.service.in;
web = self.packages.${system}.default;
};
You can build it with nix build .#web-service
, the output will look something
like this:
[Unit]
Description=A web service
[Service]
DynamicUser=yes
ExecStart=/nix/store/yl863jm907wfr7gq9j0c4bd3d4bdc4vp-web-server-20220227/bin/web-server
[Install]
WantedBy=multi-user.target
The @web@
in the template was replaced with the nix store path for the web
server!
Then you can add the bit that builds the portable service:
portable = let
web = self.packages.${system}.default;
in pkgs.portableService {
inherit (web) version;
name = web.pname;
description = "A web server";
units = [ self.packages.${system}.web-service ];
};
Then you can build it with nix build
:
nix build .#portable
And then take a look at ./result
:
$ file $(readlink ./result)
/nix/store/1da6b90i75n03kqlzzfdwxii0j0bzxaf-web-server_20220227.raw:
Squashfs filesystem,
little endian,
version 4.0,
xz compressed,
9555806 bytes,
2010 inodes,
blocksize: 1048576 bytes,
created: Tue Jan 1 00:00:00 1980
At the time of writing this article, the most reliable way to test portable services is to use Arch Linux. So you could use something like waifud to spin up an Arch Linux VM:
$ waifuctl create -d arch -h logos -s 20
created instance jangmo-o on logos
jangmo-o: running
jangmo-o: init: IP address: 10.77.129.208
Then copy it over with scp
:
$ scp (readlink ./result) xe@10.77.129.208:web-server_20220227.raw
Then you can use portablectl
to attach it to the system:
$ sudo portablectl attach ./web-server_20220227.raw
[...]
Created symlink /etc/portables/web-server_20220227.raw → /home/xe/web-server_20220227.raw.
And then start it like any systemd service:
$ sudo systemctl start web-server
And then inspect the service's status with systemctl
:
$ sudo systemctl status web-server
● web-server.service - A web service
Loaded: loaded (/etc/systemd/system.attached/web-server.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system.attached/web-server.service.d
└─10-profile.conf, 20-portable.conf
Active: active (running) since Sun 2022-02-27 18:21:01 UTC; 20s ago
Main PID: 960 (web-server)
Tasks: 5 (limit: 513)
Memory: 8.1M
CPU: 189ms
CGroup: /system.slice/web-server.service
└─960 /nix/store/yl863jm907wfr7gq9j0c4bd3d4bdc4vp-web-server-20220227/bin/web-server
Feb 27 18:21:01 jangmo-o systemd[1]: Started A web service.
Feb 27 18:21:01 jangmo-o web-server[960]: 2022/02/27 18:21:01 listening for HTTP on :3031
And finally poke it with curl:
$ curl http://[::]:3031
hello from nix building a docker image!
I know, I know, I didn't adjust the message from when I wrote the Docker example.
And then you can change the handler to something like:
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "PORTABLE=%s\n", os.Getenv("PORTABLE"))
})
Rebuild the image with nix build
:
$ git add .
$ nix build .#portable
Copy it to the arch VM with scp
:
$ scp $(readlink ./result) xe@10.77.129.208:web-server_20220227.raw
And finally run portablectl reattach
to upgrade it:
$ sudo portablectl reattach --now ./web-server_20220227.raw
Queued /org/freedesktop/systemd1/job/858 to call RestartUnit on portable service
web-server.service.
Then you can see that it restarted the unit with systemctl status
:
$ sudo systemctl status web-server
● web-server.service - A web service
Loaded: loaded (/etc/systemd/system.attached/web-server.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system.attached/web-server.service.d
└─10-profile.conf, 20-portable.conf
Active: active (running) since Sun 2022-02-27 18:30:04 UTC; 37s ago
Main PID: 1074 (web-server)
Tasks: 6 (limit: 513)
Memory: 8.1M
CPU: 182ms
CGroup: /system.slice/web-server.service
└─1074 /nix/store/j1mfz3ydn13qmvcgrql33zi0dwb3x7dk-web-server-20220227/bin/web-server
Feb 27 18:30:04 jangmo-o systemd[1]: Started A web service.
Feb 27 18:30:04 jangmo-o web-server[1074]: 2022/02/27 18:30:04 listening for HTTP on :3031
And finally poke it with curl:
$ curl http://[::]:3031
PORTABLE=web-server_20220227.raw
And there you go! Nix created a portable system service, we spawned it on a newly created Arch Linux VM and then were able to update it so that we could replace the message.
Nix builds can do more than just turn code into software. They can create Docker images, Portable Services, virtual machine images and more. The only real limit is what you can imagine.
Flakes make it easier to pull in and munge about packages. Before flakes you'd
need to have a few .nix
files like docker.nix
for the docker image and
portable.nix
for the portable service. You'd also have to pull in something
like Niv to make sure everything uses the same
version of nixpkgs, and even then it's opt-in, not opt-out, so it's easy to mess
things up and not use the pinned versions of things. Flakes make that explicit
behavior implicit, so you can't bring in dependencies you aren't aware of.
If you want to see the code repo I developed while writing this post, see cadey/gohello-http on my git server.
Thanks for reading!