artemis
/
xesite
forked from cadey/xesite
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

nix flakes 2: fun with packages 

Signed-off-by: Xe Iaso <me@christine.website>
This commit is contained in:
Cadey Ratio 2022-02-27 19:12:53 +00:00
parent 9e781d2246
commit 8fc69d4978
3 changed files with 581 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

570
blog/nix-flakes-2-2022-02-27.markdown Normal file
View File

@ -0,0 +1,570 @@
---
title: "Nix Flakes: Packages and How to Use Them"
date: 2022-02-27
tags:
- nix
- nixos
- docker
- systemd
series: nix-flakes
vod:
twitch: https://www.twitch.tv/videos/1409855764
youtube: https://youtu.be/eUFBD-6yAWQ
---
<div class="warning">
[Nix flakes are still marked as experimental. This documentation has a small
chance of bitrotting. I will make every attempt to update it if things change,
however flakes have been fairly consistent for a few years
now.](conversation://Cadey/coffee)
</div>
[What is a package? I've seen this term thrown around with phrases like "Nix is a
package manager" or "language-specific package manager" or even "download the
debian package and install it", but it's not really clear to me what a package
is. What is a package?](conversation://Mara/hmm)
A package is a bundle of files. These files could be program executables,
resources such as stylesheets or images, or even a container image. Most of the
time you don't deal with packages directly and instead you use a _package
manager_ (a program whose sole goal in life is to deal with packages) to do
actions for you. This post is going to cover how to define packages in Nix and
how Nix flakes let you manage multiple packages per project more easily.
## What is a Package?
In Nix, you build packages by creating _derivations_ that define the build steps
and associated inputs (such as the compiler) to end up with the resulting
outputs (derivation being the product of deriving something). Consider a package
like this:
```nix
# hello-shell.nix
with import <nixpkgs> { };
stdenv.mkDerivation {
name = "hello-HEAD";
src = ./.;
installPhase = ''
echo "Hello" > $out
'';
}
```
Then we can build this package with `nix-build hello-shell.nix` and a `result`
symlink will show up in your current working directory. Then you can view what
it says with `cat`:
```console
$ cat ./result
Hello
```
This is all it takes to make a Nix package. You need to name the package, give
it input source code somehow, and potentially give it build instructions.
Everything else we'll cover today will build on top of this.
Let's look back at the Go [example
package](https://github.com/Xe/gohello/blob/caf54cdff7d8dd9bd9df4b3b783a72fe75c9a11e/flake.nix#L31-L54)
I walked us through in [the last
post](https://christine.website/blog/nix-flakes-1-2022-02-21):
```nix
# ...
packages = forAllSystems (system:
let pkgs = nixpkgsFor.${system};
in {
go-hello = pkgs.buildGoModule {
pname = "go-hello";
inherit version;
# In 'nix develop', we don't need a copy of the source tree
# in the Nix store.
src = ./.;
# This hash locks the dependencies of this package. It is
# necessary because of how Go requires network access to resolve
# VCS. See https://www.tweag.io/blog/2021-03-04-gomod2nix/ for
# details. Normally one can build with a fake sha256 and rely on native Go
# mechanisms to tell you what the hash should be or determine what
# it should be "out-of-band" with other tooling (eg. gomod2nix).
# To begin with it is recommended to set this, but one must
# remeber to bump this hash when your dependencies change.
vendorSha256 =
"sha256-pQpattmS9VmO3ZIQUFn66az8GSmB4IvYhTTCFn6SUmo=";
};
});
# ...
```
This uses a different builder, one called
[`pkgs.buildGoModule`](https://nixos.org/manual/nixpkgs/stable/#ssec-language-go).
This is like the `stdenv.mkDerivation` builder, except it is explicitly made to
handle Go projects. There are some other flags that you can set in
`buildGoModule` that can be useful. You can see examples in the NixOS manual
page [here](https://nixos.org/manual/nixpkgs/stable/#ssec-language-go).
Another useful builder is [Naersk](https://github.com/nix-community/naersk).
Naersk will automatically derive build instructions for Rust projects using the
`Cargo.toml` and `Cargo.lock` files. This means that your build step can look as
small as this:
```nix
naersk-lib.buildPackage ./.
```
[You can think of these builders as templates for doing larger builds. This is
kinda like <a
href="https://docs.docker.com/engine/reference/builder/#onbuild">the ONBUILD
Dockerfile instruction</a>, but it isn't limited to
Docker. The main difference is that Nix builds are more like functions (inputs
and outputs) and Docker builds focus on the individual commands you run to get
the result you want. Both eventually compile down to shell commands
anyways!](conversation://Mara/hacker)
## A More Useful Package
This "hello world" program isn't very useful on its own, however we can use it
as the basis for making something a bit more useful. I have made a template for
a "Hello world" HTTP server
[here](https://github.com/Xe/templates/tree/main/go-web-server). Let's make a
new folder for it and then initialize it:
[If you want to make your own templates, see how to do that <a
href="https://peppe.rs/posts/novice_nix:_flake_templates/">here</a>.](conversation://Mara/hacker)
```shell
mkdir -p ~/tmp/gohello-http
cd ~/tmp/gohello-http
git init
nix flake init -t github:Xe/templates#go-web-server
```
[You may see a message from <a href="https://direnv.net/">direnv</a> about
needing to approve its content. This will use Nix flake's cached interpreter to
give you all the advantages of something like <a
href="https://github.com/nix-community/lorri">Lorri</a> without having to
install Lorri.](conversation://Mara/hacker)
Then make an initial commit and run it:
```shell
git add .
git commit -sm "initial commit"
nix build
./result/bin/web-server
```
[Why are you using `git add .` everywhere? Shouldn't the files be picked up
implicitly?](conversation://Mara/hmm)
[Not always. Nix flakes only deals with files that are tracked by git when you
use it in a git repository. This means that if you want the changes to be
observed by Nix, you need to add them to git somehow. `git add` is good enough
for this.](conversation://Cadey/enby)
Or you can run it directly with `nix run`:
```shell
nix run
```
## Docker Images
Most of the time you will build software with Nix, however that doesn't stop you
from building things like Docker images with Nix. Remember that you can have the
output of any shell commands be run in a Nix build (the only catch is that they
can't access the internet directly), so you can build a Docker image out of that
web server template by defining another package:
```nix
# flake.nix
# after defaultPackage
packages = {
docker = let
web = self.defaultPackage.${system};
in pkgs.dockerTools.buildLayeredImage {
name = web.pname;
tag = web.version;
contents = [ web ];
config = {
Cmd = [ "/bin/web-server" ];
WorkingDir = "/";
};
};
};
```
This will build a Docker image with the web-server binary in it. To build it,
run these commands:
```shell
git add .
nix build .#docker
```
[What's with that last argument to `nix build`, won't that be read as a shell
comment?](conversation://Mara/hmm)
[It's a reference to the package in the flake. Shell only parses comments when
the `#` is the first character after whitespace, so this is more of a URL
fragment than a comment. It's telling `nix build` to build the flake package
named `docker`.](conversation://Cadey/enby)
It will put the resulting docker image in `./result`. To load it into docker use
the following command:
```console
$ docker load < result
Loaded image: web-server:20220227
```
[Your image tag may differ depending on when you build this
image. This is deterministic because that date is derived from the date that the
current git commit was made.](conversation://Mara/happy)
Then you can run it with `docker run`:
```shell
docker run -itp 3031:3031 web-server:20220227
```
Then poke it with curl:
```console
$ curl http://[::]:3031
hello from nix!
```
You can push this image to the Docker hub like any other image. Another cool
thing about this is that when you update the program, it'll only actually load
the images that changed. Let's edit the hello world message:
```go
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "hello from nix building a docker image!")
})
```
And then re-build and load it into Docker:
```shell
git add .
nix build .#docker
docker load < result
```
[Woah, when I did that it only updated 2 layers. The first time that I loaded it
there were something like 7 layers. What's up with
that?](conversation://Mara/hmm)
[When you use `buildLayeredImage`, each Nix package that contributes to the
image gets put in its own Docker layer. This means that only the things that
have changed actually need to be considered, so when you push an updated image
to another machine, the only things that will actually be pushed are the
application binary and the symlink farm pointing to the `contents` of the Docker
image.](conversation://Cadey/enby)
## systemd Portable Services
[systemd Portable Services](https://systemd.io/PORTABLE_SERVICES/) function like
Docker, but they work at the systemd level and allow you to integrate into
systemd instead of running on the side of it. This gives you access to systemd's
readiness signaling, logging pipeline and dependency graph so that you can
integrate like a native service. They are like containers, but without a lot of
the headaches around networking, stateful storage and logging. They are just
systemd services at their core.
[These are kinda like Ubuntu's Snaps or Flatpaks, but they operate purely at the
system level and are focused at providing things for system services instead of
user-facing applications. Ubuntu's Snaps do let you create system services, but
they are basically exclusively used on Ubuntu. systemd Portable Services let you
target more than just Ubuntu. In the next few years with more releases of
systemd, Portable Services should be easier to use and will be more integrated
with the system than Docker is.](conversation://Mara/hacker)
There is currently an [open pull request](https://systemd.io/PORTABLE_SERVICES/)
for adding Portable Service building support to nixpkgs, however we can mess
around with it today thanks to [my portable-svc
overlay](https://tulpa.dev/cadey/portable-svc) that copies in the contents of
that pull request.
[In Nix, an overlay is a set of additional packages or functions that is put on
top of nixpkgs. This overlay defines the `portableService` function that is
needed to build portable services.](conversation://Mara/hacker)
To make this into a portable service, first we need to add my overlay to the
flake inputs:
```nix
# flake.nix
inputs = {
nixpkgs.url = "nixpkgs/nixos-unstable";
utils.url = "github:numtide/flake-utils";
portable-svc.url = "git+https://tulpa.dev/cadey/portable-svc.git?ref=main";
};
```
Then add it as an argument to the `outputs` function:
```nix
outputs = { self, nixpkgs, utils, portable-svc }:
```
And then change how we are importing the `pkgs` variable. The `pkgs` variable
we're currently using is imported like this:
```nix
let pkgs = nixpkgs.legacyPackages.${system};
```
This works, however there isn't a way to specify an overlay into this. We need
to change this into a manual import of nixpkgs with the overlay specified, like
this:
```nix
let pkgs = import nixpkgs {
overlays = [ portable-svc.overlay ];
inherit system;
};
```
This will let us use the `portableService` function in Nix package definitions.
Next we need to make a systemd service unit for the web server. The exact path
to the program binary can and will change with every build, so it would be good
to have this templated. Make a folder called `systemd`:
```shell
mkdir systemd
```
And put the following contents in `systemd/web-server.service.in`:
```systemd
[Unit]
Description=A web service
[Service]
DynamicUser=yes
ExecStart=@web@/bin/web-server
[Install]
WantedBy=multi-user.target
```
Then under the docker package definition, add the package that will template out
the systemd unit:
```nix
web-service = pkgs.substituteAll {
name = "web-server.service";
src = ./systemd/web-server.service.in;
web = self.defaultPackage.${system};
};
```
You can build it with `nix build .#web-service`, the output will look something
like this:
```systemd
[Unit]
Description=A web service
[Service]
DynamicUser=yes
ExecStart=/nix/store/yl863jm907wfr7gq9j0c4bd3d4bdc4vp-web-server-20220227/bin/web-server
[Install]
WantedBy=multi-user.target
```
[The `@web@` in the template was replaced with the nix store path for the web
server!](conversation://Mara/happy)
Then you can add the bit that builds the portable service:
```nix
portable = let
web = self.defaultPackage.${system};
in pkgs.portableService {
inherit (web) version;
name = web.pname;
description = "A web server";
units = [ self.packages.${system}.web-service ];
};
```
Then you can build it with `nix build`:
```shell
nix build .#portable
```
And then take a look at `./result`:
```console
$ file $(readlink ./result)
/nix/store/1da6b90i75n03kqlzzfdwxii0j0bzxaf-web-server_20220227.raw:
Squashfs filesystem,
little endian,
version 4.0,
xz compressed,
9555806 bytes,
2010 inodes,
blocksize: 1048576 bytes,
created: Tue Jan 1 00:00:00 1980
```
<div class="warning" style="padding:1em">
At the time of writing this article, the most reliable way to test portable
services is to use Arch Linux. So you could use something like
[waifud](https://github.com/Xe/waifud) to spin up an Arch Linux VM:
```console
$ waifuctl create -d arch -h logos -s 20
created instance jangmo-o on logos
jangmo-o: running
jangmo-o: init: IP address: 10.77.129.208
```
Then copy it over with `scp`:
```console
$ scp (readlink ./result) xe@10.77.129.208:web-server_20220227.raw
```
</div>
Then you can use `portablectl` to attach it to the system:
```console
$ sudo portablectl attach ./web-server_20220227.raw
[...]
Created symlink /etc/portables/web-server_20220227.raw → /home/xe/web-server_20220227.raw.
```
And then start it like any systemd service:
```console
$ sudo systemctl start web-server
```
[If you want the service to start automatically, add `--enable --now` to the
`portablectl attach` command. That will enable the service in systemd and then
start it, like when you run `systemctl enable --now
something.service`.](conversation://Mara/hacker)
And then inspect the service's status with `systemctl`:
```console
$ sudo systemctl status web-server
● web-server.service - A web service
Loaded: loaded (/etc/systemd/system.attached/web-server.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system.attached/web-server.service.d
└─10-profile.conf, 20-portable.conf
Active: active (running) since Sun 2022-02-27 18:21:01 UTC; 20s ago
Main PID: 960 (web-server)
Tasks: 5 (limit: 513)
Memory: 8.1M
CPU: 189ms
CGroup: /system.slice/web-server.service
└─960 /nix/store/yl863jm907wfr7gq9j0c4bd3d4bdc4vp-web-server-20220227/bin/web-server
Feb 27 18:21:01 jangmo-o systemd[1]: Started A web service.
Feb 27 18:21:01 jangmo-o web-server[960]: 2022/02/27 18:21:01 listening for HTTP on :3031
```
And finally poke it with curl:
```console
$ curl http://[::]:3031
hello from nix building a docker image!
```
[That ain't Docker, chief!](conversation://Numa/delet)
[I know, I know, I didn't adjust the message from when I wrote the Docker
example.](conversation://Cadey/facepalm)
And then you can change the handler to something like:
```go
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "PORTABLE=%s\n", os.Getenv("PORTABLE"))
})
```
Rebuild the image with `nix build`:
```console
$ git add .
$ nix build .#portable
```
Copy it to the arch VM with `scp`:
```console
$ scp (readlink ./result) xe@10.77.129.208:web-server_20220227.raw
```
And finally run `portablectl reattach` to upgrade it:
```console
$ sudo portablectl reattach --now ./web-server_20220227.raw
Queued /org/freedesktop/systemd1/job/858 to call RestartUnit on portable service
web-server.service.
```
Then you can see that it restarted the unit with `systemctl status`:
```console
$ sudo systemctl status web-server
● web-server.service - A web service
Loaded: loaded (/etc/systemd/system.attached/web-server.service; disabled; vendor preset: disabled)
Drop-In: /etc/systemd/system.attached/web-server.service.d
└─10-profile.conf, 20-portable.conf
Active: active (running) since Sun 2022-02-27 18:30:04 UTC; 37s ago
Main PID: 1074 (web-server)
Tasks: 6 (limit: 513)
Memory: 8.1M
CPU: 182ms
CGroup: /system.slice/web-server.service
└─1074 /nix/store/j1mfz3ydn13qmvcgrql33zi0dwb3x7dk-web-server-20220227/bin/web-server
Feb 27 18:30:04 jangmo-o systemd[1]: Started A web service.
Feb 27 18:30:04 jangmo-o web-server[1074]: 2022/02/27 18:30:04 listening for HTTP on :3031
```
And finally poke it with curl:
```console
$ curl http://[::]:3031
PORTABLE=web-server_20220227.raw
```
And there you go! Nix created a portable system service, we spawned it on a
newly created Arch Linux VM and then were able to update it so that we could
replace the message.
---
Nix builds can do more than just turn code into software. They can create Docker
images, Portable Services, virtual machine images and more. The only real limit
is what you can imagine.
Flakes make it easier to pull in and munge about packages. Before flakes you'd
need to have a few `.nix` files like `docker.nix` for the docker image and
`portable.nix` for the portable service. You'd also have to pull in something
like [Niv](https://github.com/nmattia/niv) to make sure everything uses the same
version of nixpkgs, and even then it's opt-in, not opt-out, so it's easy to mess
things up and not use the pinned versions of things. Flakes make that explicit
behavior implicit, so you can't bring in dependencies you aren't aware of.
If you want to see the code repo I developed while writing this post, see
[cadey/gohello-http](https://tulpa.dev/cadey/gohello-http) on my git server.
Thanks for reading!

7
src/post/frontmatter.rs
View File

@ -13,6 +13,13 @@ pub struct Data {
pub thumb: Option<String>, pub thumb: Option<String>,
pub show: Option<bool>, pub show: Option<bool>,
pub redirect_to: Option<String>, pub redirect_to: Option<String>,
pub vod: Option<Vod>,
}
#[derive(Eq, PartialEq, Deserialize, Default, Debug, Serialize, Clone)]
pub struct Vod {
pub twitch: String,
pub youtube: String,
} }
enum State { enum State {

4
templates/blogpost.rs.html
View File

@ -68,6 +68,10 @@
<hr /> <hr />
@if post.front_matter.vod.is_some() {
<p>This post was written live on <a href="https://twitch.tv/princessxen">Twitch</a>. You can check out the stream recording on Twitch <a href="@post.front_matter.vod.as_ref().unwrap().twitch">here</a> and on YouTube <a href="@post.front_matter.vod.as_ref().unwrap().youtube">here</a>.</p>
}
<!-- The button that should be clicked. --> <!-- The button that should be clicked. -->
<button onclick="share_on_mastodon()">Share on Mastodon</button> <button onclick="share_on_mastodon()">Share on Mastodon</button>