diff --git a/blog/how-i-start-nix-2020-03-08.markdown b/blog/how-i-start-nix-2020-03-08.markdown new file mode 100644 index 0000000..282096b --- /dev/null +++ b/blog/how-i-start-nix-2020-03-08.markdown @@ -0,0 +1,574 @@ +--- +title: "How I Start: Nix" +date: 2020-03-08 +series: howto +tags: + - nix + - rust +--- + +# How I Start: Nix + +[Nix][nix] is a tool that helps people create reproducible builds. This means that +given the a known input, you can get the same output on other machines.Let's +build and deploy a small Rust service with Nix. This will not require the Rust +compiler to be installed with [rustup][rustup] or similar. + +[nix]: https://nixos.org/nix/ +[rustup]: https://rustup.rs + +- Setting up your environment +- A new project +- Setting up the Rust compiler +- Serving HTTP +- A simple package build +- Shipping it in a docker image + +## Setting up your environment + +The first step is to install Nix. If you are using a Linux machine, run this +script: + +```console +$ curl https://nixos.org/nix/install | sh +``` + +This will prompt you for more information as it goes on, so be sure to follow +the instructions carefully. Once it is done, close and re-open your shell. After +you have done this, `nix-env` should exist in your shell. Try to run it: + +```console +$ nix-env +error: no operation specified +Try 'nix-env --help' for more information. +``` + +Let's install a few other tools to help us with development. First, let's +install [lorri][lorri] to help us manage our development shell: + +[lorri]: https://github.com/target/lorri + +``` +$ nix-env --install --file https://github.com/target/lorri/archive/master.tar.gz +``` + +This will automatically download and build lorri for your system based on the +latest possible version. Once that is done, open another shell window (the lorri +docs include ways to do this more persistently, but this will work for now) and run: + +```console +$ lorri daemon +``` + +Now go back to your main shell window and install [direnv][direnv]: + +[direnv]: https://direnv.net + +```console +$ nix-env --install direnv +``` + +Next, follow the [shell setup][direnvsetup] needed for your shell. I personally +use `fish` with [oh my fish][omf], so I would run this: + +[direnvsetup]: https://direnv.net/docs/hook.html +[omf]: https://github.com/oh-my-fish/oh-my-fish + +```console +$ omf install direnv +``` + +Finally, let's install [niv][niv] to help us handle dependencies for the +project. This will allow us to make sure that our builds pin _everything_ to a +specific set of versions, including operating system packages. + +[niv]: https://github.com/nmattia/niv + +```console +$ nix-env --install niv +``` + +Now that we have all of the tools we will need installed, let's create the +project. + +# A new project + +Go to your favorite place to put code and make a new folder. I personally prefer +`~/code`, so I will be using that here: + +```console +$ cd ~/code +$ mkdir helloworld +$ cd helloworld +``` + +Let's set up the basic skeleton of the project. First, initialize niv: + +```console +$ niv init +``` + +This will add the latest versions of `niv` itself and the packages used for the +system to `nix/sources.json`. This will allow us to pin exact versions so the +environment is as predictable as possible. Sometimes the versions of software in +the pinned nixpkgs are too old. If this happens, you can update to the +"unstable" branch of nixpkgs with this command: + +```console +$ niv update nixpkgs -b nixpkgs-unstable +``` + +Next, set up lorri using `lorri init`: + +```console +$ lorri init +``` + +This will create `shell.nix` and `.envrc`. `shell.nix` will be where we define +the development environment for this service. `.envrc` is used to tell direnv +what it needs to do. Let's try and activate the `.envrc`: + +```console +$ cd . +direnv: error /home/cadey/code/helloworld/.envrc is blocked. Run `direnv allow` +to approve its content +``` + +Let's review its content: + +```console +$ cat .envrc +eval "$(lorri direnv)" +``` + +This seems reasonable, so approve it with `direnv allow` like the error message +suggests: + +```console +$ direnv allow +``` + +Now let's customize the `shell.nix` file to use our pinned version of nixpkgs. +Currently, it looks something like this: + +```nix +# shell.nix +let + pkgs = import {}; +in +pkgs.mkShell { + buildInputs = [ + pkgs.hello + ]; +} +``` + +This currently imports nixpkgs from the system-level version of it. This means +that different systems could have different versions of nixpkgs on it, and that +could make the `shell.nix` file hard to reproduce between machines. Let's import +the pinned version of nixpkgs that niv created: + +```nix +# shell.nix +let + sources = import ./nix/sources.nix; + pkgs = import sources.nixpkgs {}; +in +pkgs.mkShell { + buildInputs = [ + pkgs.hello + ]; +} +``` + +And then let's test it with `lorri shell`: + +```console +$ lorri shell +lorri: building environment........ done +(lorri) $ +``` + +And let's see if `hello` is available inside the shell: + +```console +(lorri) $ hello +Hello, world! +``` + +You can set environment variables inside the `shell.nix` file. Do so like this: + +```nix +# shell.nix +let + sources = import ./nix/sources.nix; + pkgs = import sources.nixpkgs {}; +in +pkgs.mkShell { + buildInputs = [ + pkgs.hello + ]; + + # Environment variables + HELLO="world"; +} +``` + +Wait a moment for lorri to finish rebuilding the development environment and +then let's see if the environment variable shows up: + +```console +$ cd . +direnv: loading ~/code/helloworld/.envrc + +$ echo $HELLO +world +``` + +Now that we have the basics of the environment set up, lets install the Rust +compiler. + +# Setting up the Rust compiler + +First, add [nixpkgs-mozilla][nixpkgsmoz] to niv: + +[nixpkgsmoz]: https://github.com/mozilla/nixpkgs-mozilla + +```console +$ niv add mozilla/nixpkgs-mozilla +``` + +Then create `nix/rust.nix` in your repo: + +```nix +# nix/rust.nix +{ sources ? import ./sources.nix }: + +let + pkgs = + import sources.nixpkgs { overlays = [ (import sources.nixpkgs-mozilla) ]; }; + channel = "nightly"; + date = "2020-03-08"; + targets = [ ]; + chan = pkgs.rustChannelOfTargets channel date targets; +in chan +``` + +This creates a nix function that takes in the pre-imported list of sources, +creates a copy of nixpkgs with Rust at the nightly version `2020-03-08` overlaid +into it, and exposes the rust package out of it. Let's add this to `shell.nix`: + +```nix +# shell.nix +let + sources = import ./nix/sources.nix; + rust = import ./nix/rust.nix { inherit sources; }; + pkgs = import sources.nixpkgs { }; +in +pkgs.mkShell { + buildInputs = [ + rust + ]; +} +``` + +Then ask lorri to recreate the development environment. This may take a bit to +run because it's setting up everything the Rust compiler requires to run. + +```console +$ lorri shell +(lorri) $ +``` + +Let's see what version of Rust is installed: + +```console +(lorri) $ rustc --version +rustc 1.43.0-nightly (823ff8cf1 2020-03-07) +``` + +This is exactly what we expect. Rust nightly versions get released with the +date of the previous day in them. To be extra sure, let's see what the shell +thinks `rustc` resolves to: + +```console +(lorri) $ which rustc +/nix/store/w6zk1zijfwrnjm6xyfmrgbxb6dvvn6di-rust-1.43.0-nightly-2020-03-07-823ff8cf1/bin/rustc +``` + +And now exit that shell and reload direnv: + +```console +(lorri) $ exit +$ cd . +direnv: loading ~/code/helloworld/.envrc +$ which rustc +/nix/store/w6zk1zijfwrnjm6xyfmrgbxb6dvvn6di-rust-1.43.0-nightly-2020-03-07-823ff8cf1/bin/rustc +``` + +And now we have Rust installed at an arbitrary nightly version for _that project +only_. This will work on other machines too. Now that we have our development +environment set up, let's serve HTTP. + +## Serving HTTP + +[Rocket][rocket] is a popular web framework for Rust programs. Let's use that to +create a small "hello, world" server. We will need to do the following: + +[rocket]: https://rocket.rs + +- Create the new Rust project +- Add Rocket as a dependency +- Write our "hello world" route +- Test a build of the service with `cargo build` + +### Create the new Rust project + +Create the new Rust project with `cargo init`: + +```console +$ cargo init --vcs git . + Created binary (application) package +``` + +This will create the directory `src` and a file named `Cargo.toml`. Rust code +goes in `src` and the `Cargo.toml` file configures dependencies. Adding the +`--vcs git` flag also has cargo create a [gitignore][gitignore] file so that the +target folder isn't tracked by git. + +[gitignore]: https://git-scm.com/docs/gitignore + +### Add Rocket as a dependency + +Open `Cargo.toml` and add the following to it: + +```toml +[dependencies] +rocket = "0.4.3" +``` + +Then download/build Rocket with `cargo build`: + +```console +$ cargo build +``` + +This will download all of the dependencies you need and precompile Rocket. This +will help speed up later builds. + +### Write our "hello world" route + +Now put the following in `src/main.rs`: + +```rust +#![feature(proc_macro_hygiene, decl_macro)] // language features needed by Rocket + +// Import the rocket macros +#[macro_use] +extern crate rocket; + +// Create route / that returns "Hello, world!" +#[get("/")] +fn index() -> &'static str { + "Hello, world!" +} + +fn main() { + rocket::ignite().mount("/", routes![index]).launch(); +} +``` + +### Test a build + +Rerun `cargo build`: + +```console +$ cargo build +``` + +This will create the binary at `target/debug/helloworld`. Let's run it locally +and see if it works: + +```console +$ ./target/debug/helloworld & +$ curl http://127.0.0.1:8000 +Hello, world! +$ fg + +``` + +The HTTP service works. We have a binary that is created with the Rust compiler +Nix installed. + +## A simple package build + +Now that we have the HTTP service working, let's put it inside a nix package. We +will need to use [naersk][naersk] to do this. Add naersk to your project with +niv: + +[naersk]: https://github.com/nmattia/naersk + +```console +$ niv add nmattia/naersk +``` + +Now let's create `helloworld.nix`: + +``` +# import niv sources and the pinned nixpkgs +{ sources ? import ./nix/sources.nix, pkgs ? import sources.nixpkgs { }}: +let + # import rust compiler + rust = import ./nix/rust.nix { inherit sources; }; + + # configure naersk to use our pinned rust compiler + naersk = pkgs.callPackage sources.naersk { + rustc = rust; + cargo = rust; + }; + + # tell nix-build to ignore the `target` directory + src = builtins.filterSource + (path: type: type != "directory" || builtins.baseNameOf path != "target") + ./.; +in naersk.buildPackage { + inherit src; + remapPathPrefix = + true; # remove nix store references for a smaller output package +} +``` + +And then build it with `nix-build`: + +```console +$ nix-build helloworld.nix +``` + +This can take a bit to run, but it will do the following things: + +- Download naersk +- Download every Rust crate your HTTP service depends on into the Nix store +- Run your program's tests +- Build your dependencies into a Nix package +- Build your program with those dependencies +- Place a link to the result at `./result` + +Once it is done, let's take a look at the result: + +```console +$ du -hs ./result/bin/helloworld +2.1M ./result/bin/helloworld + +$ ldd ./result/bin/helloworld + linux-vdso.so.1 (0x00007fffae080000) + libdl.so.2 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libdl.so.2 (0x0 +0007f3a01666000) + librt.so.1 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/librt.so.1 (0x0 +0007f3a0165c000) + libpthread.so.0 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libpthread +.so.0 (0x00007f3a0163b000) + libgcc_s.so.1 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libgcc_s.so. +1 (0x00007f3a013f5000) + libc.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libc.so.6 (0x000 +07f3a0123f000) + /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/ld-linux-x86-64.so.2 => /lib6 +4/ld-linux-x86-64.so.2 (0x00007f3a0160b000) + libm.so.6 => /nix/store/wx1vk75bpdr65g6xwxbj4rw0pk04v5j3-glibc-2.27/lib/libm.so.6 (0x000 +07f3a010a9000) +``` + +This means that the Nix build created a 2.1 megabyte binary that only depends on +[glibc][glibc], the implementation of the C language standard library that Nix +prefers. + +[glibc]: https://www.gnu.org/software/libc/ + +For repo cleanliness, add the `result` link to the [gitignore][gitignore]: + +```console +$ echo 'result*' >> .gitignore +``` + +## Shipping it in a Docker image + +Now that we have a package built, let's ship it in a docker image. nixpkgs +provides [dockerTools][dockertools] which helps us create docker images out of +Nix packages. Let's create `default.nix` with the following contents: + +[dockertools]: https://nixos.org/nixpkgs/manual/#sec-pkgs-dockerTools + +```nix +{ system ? builtins.currentSystem }: + +let + sources = import ./nix/sources.nix; + pkgs = import sources.nixpkgs { }; + helloworld = import ./helloworld.nix { inherit sources pkgs; }; + + name = "xena/helloworld"; + tag = "latest"; + +in pkgs.dockerTools.buildLayeredImage { + inherit name tag; + contents = [ helloworld ]; + + config = { + Cmd = [ "/bin/helloworld" ]; + Env = [ "ROCKET_PORT=5000" ]; + WorkingDir = "/"; + }; +} +``` + +And then build it with `nix-build`: + +```console +$ nix-build default.nix +``` + +This will create a tarball containing the docker image information as the result +of the Nix build. Load it into docker using `docker load`: + +```console +$ docker load -i result +``` + +And then run it using `docker run`: + +```console +$ docker run --rm -itp 52340:5000 xena/helloworld +``` + +Now test it using curl: + +```console +$ curl http://127.0.0.1:52340 +Hello, world! +``` + +And now you have a docker image you can run wherever you want. The +`buildLayeredImage` function used in `default.nix` also makes Nix put each +dependency of the package into its own docker layer. This makes new versions of +your program very efficient to upgrade on your clusters, realistically this +reduces the amount of data needed for new versions of the program down to what +changed. If nothing but some resources in their own package were changed, only +those packages get downloaded. + +This is how I start a new project with Nix. I put all of the code described in +this post in [this GitHub repo][helloworldrepo] in case it helps. Have fun and +be well. + +[helloworldrepo]: https://github.com/Xe/helloworld + +--- + +For some "extra credit" tasks, try and see if you can do the following: + +- Use the version of [niv][niv] that niv pinned +- Customize the environment of the container by following the [Rocket + configuration documentation](https://rocket.rs/v0.4/guide/configuration/) +- Add some more routes to the program +- Read the [Nix + documentation](https://nixos.org/nix/manual/#chap-writing-nix-expressions) and + learn more about writing Nix expressions +- Configure your editor/IDE to use the `direnv` path