14 KiB

Raw Blame History

title

date

series

How I Start: Nix

Nix is a tool that helps people create reproducible builds. This means that given a known input, you can get the same output on other machines. Let's build and deploy a small Rust service with Nix. This will not require the Rust compiler to be installed with rustup or similar.

Setting up your environment
A new project
Setting up the Rust compiler
Serving HTTP
A simple package build
Shipping it in a docker image

Setting up your environment

The first step is to install Nix. If you are using a Linux machine, run this script:

$ curl https://nixos.org/nix/install | sh

This will prompt you for more information as it goes on, so be sure to follow the instructions carefully. Once it is done, close and re-open your shell. After you have done this, nix-env should exist in your shell. Try to run it:

$ nix-env
error: no operation specified
Try 'nix-env --help' for more information.

Let's install a few other tools to help us with development. First, let's install lorri to help us manage our development shell:

$ nix-env --install --file https://github.com/target/lorri/archive/master.tar.gz

This will automatically download and build lorri for your system based on the latest possible version. Once that is done, open another shell window (the lorri docs include ways to do this more persistently, but this will work for now) and run:

$ lorri daemon

Now go back to your main shell window and install direnv:

$ nix-env --install direnv

Next, follow the shell setup needed for your shell. I personally use fish with oh my fish, so I would run this:

$ omf install direnv

Finally, let's install niv to help us handle dependencies for the project. This will allow us to make sure that our builds pin everything to a specific set of versions, including operating system packages.

$ nix-env --install niv

Now that we have all of the tools we will need installed, let's create the project.

A new project

Go to your favorite place to put code and make a new folder. I personally prefer ~/code, so I will be using that here:

$ cd ~/code
$ mkdir helloworld
$ cd helloworld

Let's set up the basic skeleton of the project. First, initialize niv:

$ niv init

This will add the latest versions of niv itself and the packages used for the system to nix/sources.json. This will allow us to pin exact versions so the environment is as predictable as possible. Sometimes the versions of software in the pinned nixpkgs are too old. If this happens, you can update to the "unstable" branch of nixpkgs with this command:

$ niv update nixpkgs -b nixpkgs-unstable

Next, set up lorri using lorri init:

$ lorri init

This will create shell.nix and .envrc. shell.nix will be where we define the development environment for this service. .envrc is used to tell direnv what it needs to do. Let's try and activate the .envrc:

$ cd .
direnv: error /home/cadey/code/helloworld/.envrc is blocked. Run `direnv allow`
to approve its content

Let's review its content:

$ cat .envrc
eval "$(lorri direnv)"

This seems reasonable, so approve it with direnv allow like the error message suggests:

$ direnv allow

Now let's customize the shell.nix file to use our pinned version of nixpkgs. Currently, it looks something like this:

# shell.nix
let
  pkgs = import <nixpkgs> {};
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
  ];
}

This currently imports nixpkgs from the system-level version of it. This means that different systems could have different versions of nixpkgs on it, and that could make the shell.nix file hard to reproduce between machines. Let's import the pinned version of nixpkgs that niv created:

# shell.nix
let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs {};
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
  ];
}

And then let's test it with lorri shell:

$ lorri shell
lorri: building environment........ done
(lorri) $

And let's see if hello is available inside the shell:

(lorri) $ hello
Hello, world!

You can set environment variables inside the shell.nix file. Do so like this:

# shell.nix
let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs {};
in
pkgs.mkShell {
  buildInputs = [
    pkgs.hello
  ];
  
  # Environment variables
  HELLO="world";
}

Wait a moment for lorri to finish rebuilding the development environment and then let's see if the environment variable shows up:

$ cd .
direnv: loading ~/code/helloworld/.envrc
<output snipped>
$ echo $HELLO
world

Now that we have the basics of the environment set up, lets install the Rust compiler.

Setting up the Rust compiler

First, add nixpkgs-mozilla to niv:

$ niv add mozilla/nixpkgs-mozilla

Then create nix/rust.nix in your repo:

# nix/rust.nix
{ sources ? import ./sources.nix }:

let
  pkgs =
    import sources.nixpkgs { overlays = [ (import sources.nixpkgs-mozilla) ]; };
  channel = "nightly";
  date = "2020-03-08";
  targets = [ ];
  chan = pkgs.rustChannelOfTargets channel date targets;
in chan

This creates a nix function that takes in the pre-imported list of sources, creates a copy of nixpkgs with Rust at the nightly version 2020-03-08 overlaid into it, and exposes the rust package out of it. Let's add this to shell.nix:

# shell.nix
let
  sources = import ./nix/sources.nix;
  rust = import ./nix/rust.nix { inherit sources; };
  pkgs = import sources.nixpkgs { };
in
pkgs.mkShell {
  buildInputs = [
    rust
  ];
}

Then ask lorri to recreate the development environment. This may take a bit to run because it's setting up everything the Rust compiler requires to run.

$ lorri shell
(lorri) $

Let's see what version of Rust is installed:

(lorri) $ rustc --version
rustc 1.43.0-nightly (823ff8cf1 2020-03-07)

This is exactly what we expect. Rust nightly versions get released with the date of the previous day in them. To be extra sure, let's see what the shell thinks rustc resolves to:

(lorri) $ which rustc
/nix/store/w6zk1zijfwrnjm6xyfmrgbxb6dvvn6di-rust-1.43.0-nightly-2020-03-07-823ff8cf1/bin/rustc

And now exit that shell and reload direnv:

(lorri) $ exit
$ cd .
direnv: loading ~/code/helloworld/.envrc
$ which rustc
/nix/store/w6zk1zijfwrnjm6xyfmrgbxb6dvvn6di-rust-1.43.0-nightly-2020-03-07-823ff8cf1/bin/rustc

And now we have Rust installed at an arbitrary nightly version for that project only. This will work on other machines too. Now that we have our development environment set up, let's serve HTTP.

Serving HTTP

Rocket is a popular web framework for Rust programs. Let's use that to create a small "hello, world" server. We will need to do the following:

Create the new Rust project
Add Rocket as a dependency
Write our "hello world" route
Test a build of the service with cargo build

Create the new Rust project

Create the new Rust project with cargo init:

$ cargo init --vcs git .
     Created binary (application) package

This will create the directory src and a file named Cargo.toml. Rust code goes in src and the Cargo.toml file configures dependencies. Adding the --vcs git flag also has cargo create a gitignore file so that the target folder isn't tracked by git.

Add Rocket as a dependency

Open Cargo.toml and add the following to it:

[dependencies]
rocket = "0.4.3"

Then download/build Rocket with cargo build:

$ cargo build

This will download all of the dependencies you need and precompile Rocket, and it will help speed up later builds.

Write our "hello world" route

Now put the following in src/main.rs:

#![feature(proc_macro_hygiene, decl_macro)] // language features needed by Rocket

// Import the rocket macros
#[macro_use]
extern crate rocket;

// Create route / that returns "Hello, world!"
#[get("/")]
fn index() -> &'static str {
    "Hello, world!"
}

fn main() {
    rocket::ignite().mount("/", routes![index]).launch();
}

Test a build

Rerun cargo build:

$ cargo build

This will create the binary at target/debug/helloworld. Let's run it locally and see if it works:

$ ./target/debug/helloworld &
$ curl http://127.0.0.1:8000
Hello, world!
$ fg
<press control-c>

The HTTP service works. We have a binary that is created with the Rust compiler Nix installed.

A simple package build

Now that we have the HTTP service working, let's put it inside a nix package. We will need to use naersk to do this. Add naersk to your project with niv:

$ niv add nmattia/naersk

Now let's create helloworld.nix:

# import niv sources and the pinned nixpkgs
{ sources ? import ./nix/sources.nix, pkgs ? import sources.nixpkgs { }}:
let
  # import rust compiler
  rust = import ./nix/rust.nix { inherit sources; };
  
  # configure naersk to use our pinned rust compiler
  naersk = pkgs.callPackage sources.naersk {
    rustc = rust;
    cargo = rust;
  };
  
  # tell nix-build to ignore the `target` directory
  src = builtins.filterSource
    (path: type: type != "directory" || builtins.baseNameOf path != "target")
    ./.;
in naersk.buildPackage {
  inherit src;
  remapPathPrefix =
    true; # remove nix store references for a smaller output package
}

And then build it with nix-build:

$ nix-build helloworld.nix

This can take a bit to run, but it will do the following things:

Download naersk
Download every Rust crate your HTTP service depends on into the Nix store
Run your program's tests
Build your dependencies into a Nix package
Build your program with those dependencies
Place a link to the result at ./result

Once it is done, let's take a look at the result:

$ du -hs ./result/bin/helloworld
2.1M    ./result/bin/helloworld

$ ldd ./result/bin/helloworld
        linux-vdso.so.1 (0x00007fffae080000)
        libdl.so.2 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libdl.so.2 (0x0
0007f3a01666000)
        librt.so.1 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/librt.so.1 (0x0
0007f3a0165c000)
        libpthread.so.0 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libpthread
.so.0 (0x00007f3a0163b000)
        libgcc_s.so.1 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libgcc_s.so.
1 (0x00007f3a013f5000)
        libc.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6 (0x000
07f3a0123f000)
        /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib6
4/ld-linux-x86-64.so.2 (0x00007f3a0160b000)
        libm.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libm.so.6 (0x000
07f3a010a9000)

This means that the Nix build created a 2.1 megabyte binary that only depends on glibc, the implementation of the C language standard library that Nix prefers.

For repo cleanliness, add the result link to the gitignore:

$ echo 'result*' >> .gitignore

Shipping it in a Docker image

Now that we have a package built, let's ship it in a docker image. nixpkgs provides dockerTools which helps us create docker images out of Nix packages. Let's create default.nix with the following contents:

{ system ? builtins.currentSystem }:

let
  sources = import ./nix/sources.nix;
  pkgs = import sources.nixpkgs { };
  helloworld = import ./helloworld.nix { inherit sources pkgs; };

  name = "xena/helloworld";
  tag = "latest";

in pkgs.dockerTools.buildLayeredImage {
  inherit name tag;
  contents = [ helloworld ];

  config = {
    Cmd = [ "/bin/helloworld" ];
    Env = [ "ROCKET_PORT=5000" ];
    WorkingDir = "/";
  };
}

And then build it with nix-build:

$ nix-build default.nix

This will create a tarball containing the docker image information as the result of the Nix build. Load it into docker using docker load:

$ docker load -i result

And then run it using docker run:

$ docker run --rm -itp 52340:5000 xena/helloworld

Now test it using curl:

$ curl http://127.0.0.1:52340
Hello, world!

And now you have a docker image you can run wherever you want. The buildLayeredImage function used in default.nix also makes Nix put each dependency of the package into its own docker layer. This makes new versions of your program very efficient to upgrade on your clusters, realistically this reduces the amount of data needed for new versions of the program down to what changed. If nothing but some resources in their own package were changed, only those packages get downloaded.

This is how I start a new project with Nix. I put all of the code described in this post in this GitHub repo in case it helps. Have fun and be well.

For some "extra credit" tasks, try and see if you can do the following:

Use the version of niv that niv pinned
Customize the environment of the container by following the Rocket configuration documentation
Add some more routes to the program
Read the Nix documentation and learn more about writing Nix expressions
Configure your editor/IDE to use the direnv path

14 KiB Raw Blame History