---
title: Paranoid NixOS on AWS
date: 2021-08-11
author: Heartmender
series: nixos
tags:
- paranix
- aws
- r13y
---
In [the last post](https://christine.website/blog/paranoid-nixos-2021-07-18) we
covered a lot of the base groundwork involved in making a paranoid NixOS setup.
Today we're gonna throw this into prod by making a base NixOS image with it.
[Normally I don't suggest people throw these things into production directly, if
only to have some kind of barrier between you and your money generator; however
today is different. It's probably not completely unsafe to put this in
production, but I really would suggest reading and understanding this article
before doing so.](conversation://Cadey/coffee)
At a high level we are going to do the following:
- Pin production OS versions using [niv](https://github.com/nmattia/niv)
- Create a script to automatically generate a production-ready NixOS image that
you can import into The Cloud
- Manage all this using your favorite buzzwords (Terraform,
Infrastructure-as-Code)
- Install an nginx server reverse proxying to the [Printer facts
service](https://printerfacts.cetacean.club/)
## What is an Image?
Before we yolo this all into prod, let's cover what we're actually doing.
There are a lot of conflicting buzzwords here, so I'm going to go out of my way
to attempt to simplify them down so that we use my arbitrary definitions of
buzzwords instead of what other people will imply they mean. You're reading my
blog, you get my buzzwords; it's as simple as that.
In this post we are going to create a base system that you can build your
production systems on top of. This base system will be crystallized into an
_image_ that AWS will use as the initial starting place for servers.
[So you create the system definition for your base system, then turn that into
an image and put that image into AWS?](conversation://Mara/hmm)
[Yep! The exact steps are a little more complicated but at a high level that's
what we're doing.](conversation://Cadey/enby)
## Base Setup
I'm going to be publishing my work for this post
[here](https://tulpa.dev/cadey/paranix-configs), but you can follow along in
this post to understand the individual steps here.
First, let's set up the environment with
[lorri](https://github.com/nix-community/lorri) and
[niv](https://github.com/nmattia/niv). Lorri will handle creating a cached
nix-shell environment for us to run things in and niv will handle pinning NixOS
to an exact version so you can get a more reproducible production environment.
Set up lorri:
```console
$ lorri init
Aug 11 09:41:50.966 INFO wrote file, path: ./shell.nix
Aug 11 09:41:50.966 INFO wrote file, path: ./.envrc
Aug 11 09:41:50.966 INFO done
direnv: error /home/cadey/code/cadey/paranix-configs/.envrc is blocked. Run `direnv allow` to approve its content
$ direnv allow
direnv: loading ~/code/cadey/paranix-configs/.envrc
Aug 11 09:41:54.581 INFO lorri has not completed an evaluation for this project yet, nix_file: /home/cadey/code/cadey/paranix-configs/shell.nix
direnv: export +IN_NIX_SHELL
```
[Why are you putting the `$` before every command in these examples? It looks
extraneous to me.](conversation://Mara/hacker)
[The `$` is there for two main reasons. First, it allows there to be a clear
delineation between the commands being typed and their output. Secondly it makes
it slightly harder to blindly copy this into your shell without either editing
the `$` out or selecting around it. My hope is that this will make you read the
command and carefully consider whether or not you actually want to run
it.](conversation://Cadey/enby)
Set up niv:
```console
$ niv init
Initializing
Creating nix/sources.nix
Creating nix/sources.json
Importing 'niv' ...
Adding package niv
Writing new sources file
Done: Adding package niv
Importing 'nixpkgs' ...
Adding package nixpkgs
Writing new sources file
Done: Adding package nixpkgs
Done: Initializing
```
[If you don't already have niv in your environment, you can hack around that by
running all the niv commands before you set up `shell.nix` like this:
$ nix-shell -p niv --run 'niv blah'
](conversation://Mara/hacker)
And finally pin nixpkgs to a specific version of NixOS.
[At the time of writing this article, NixOS 21.05 is the stable release, so that
is what is used here.](conversation://Mara/hacker)
```console
$ niv update nixpkgs -b nixos-21.05
Update nixpkgs
Done: Update nixpkgs
$
```
This will become the foundation of our NixOS systems and production images.
You should then set up your `shell.nix` to look like this:
```nix
let
sources = import ./nix/sources.nix;
pkgs = import sources.nixpkgs { };
in pkgs.mkShell {
buildInputs = with pkgs; [
niv
terraform
bashInteractive
];
};
```
### Set Up Unix Accounts
[This step can be omitted if you are grafting this into an existing NixOS
configs repository, however it would be good to read through this to understand
the directory layout at play here.](conversation://Mara/hacker)
It's probably important to be able to have access to production machines. Let's
create a NixOS module that will allow you to SSH into the machine. In your
paranix-configs folder, run this command to make a `common` config directory:
```console
$ mkdir common
$ cd common
```
Now in that common directory, open `default.nix` in ~~emacs~~ your favorite text
editor and copy in this skeleton:
```nix
# common/default.nix
{ config, lib, pkgs, ... }:
{
imports = [ ./users.nix ];
nix.autoOptimiseStore = true;
users.users.root.openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPg9gYKVglnO2HQodSJt4z4mNrUSUiyJQ7b+J798bwD9" ];
services.tailscale.enable = true;
# Tell the firewall to implicitly trust packets routed over Tailscale:
networking.firewall.trustedInterfaces = [ "tailscale0" ];
security.auditd.enable = true;
security.audit.enable = true;
security.audit.rules = [
"-a exit,always -F arch=b64 -S execve"
];
security.sudo.execWheelOnly = true;
environment.defaultPackages = lib.mkForce [];
services.openssh = {
passwordAuthentication = false;
allowSFTP = false; # Don't set this if you need sftp
challengeResponseAuthentication = false;
extraConfig = ''
AllowTcpForwarding yes
X11Forwarding no
AllowAgentForwarding no
AllowStreamLocalForwarding no
AuthenticationMethods publickey
'';
};
# PCI compliance
environment.systemPackages = with pkgs; [ clamav ];
}
```
[Astute readers will notice that this is less paranoid than the last post. This
was pared down after private feedback.](conversation://Mara/hacker)
This will create `common` as a folder that can be imported as a NixOS module
with some basic settings and then tells NixOS to try importing `users.nix` as a
module. This module doesn't exist yet, so it will fail when we try to import it.
Let's fix that by making `users.nix`:
```nix
# common/users.nix
{ config, lib, pkgs, ... }:
with lib;
let
# These options will be used for user account defaults in
# the `mkUser` function.
xeserv.users = {
groups = mkOption {
type = types.listOf types.str;
default = [ "wheel" ];
example = ''[ "wheel" "libvirtd" "docker" ]'';
description =
"The Unix groups that Xeserv staff users should be assigned to";
};
shell = mkOption {
type = types.package;
default = pkgs.bashInteractive;
example = "pkgs.powershell";
description =
"The default shell that Xeserv staff users will be given by default.";
};
};
cfg = config.xeserv.users;
mkUser = { keys, shell ? cfg.shell, extraGroups ? cfg.groups, ... }: {
isNormalUser = true;
inherit extraGroups shell;
openssh.authorizedKeys = {
inherit keys;
};
};
in {
options.xeserv.users = xeserv.users;
config.users.users = {
cadey = mkUser {
keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPg9gYKVglnO2HQodSJt4z4mNrUSUiyJQ7b+J798bwD9" ];
};
twi = mkUser {
keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYr9hiLtDHgd6lZDgQMkJzvYeAXmePOrgFaWHAjJvNU" ];
};
};
}
```
[It's worth noting that `xeserv` in there can be anything you want. It's set to
`xeserv` as we are imagining that this is for the production environment of a
company named Xeserv.](conversation://Mara/hacker)
### Paranoid Settings
Next we're going to set up the paranoid settings from the last post into a
module named `paranoid.nix`. First we'll need to grab
[impermanence](https://github.com/nix-community/impermanence) into our niv
manifest like this:
```console
$ niv add nix-community/impermanence
Adding package impermanence
Writing new sources file
Done: Adding package impermanence
```
Then open `common/default.nix` and change this line:
```nix
imports = [ ./users.nix ];
```
To something like this:
```nix
imports = [ ./paranoid.nix ./users.nix ];
```
Then open `./paranoid.nix` in a text editor and paste in the following:
```nix
# common/paranoid.nix
{ config, pkgs, lib, ... }:
with lib;
let
sources = import ../nix/sources.nix;
impermanence = sources.impermanence;
cfg = config.xeserv.paranoid;
ifNoexec = if cfg.noexec then [ "noexec" ] else [ ];
in {
imports = [ "${impermanence}/nixos.nix" ];
options.xeserv.paranoid = {
enable = mkEnableOption "enables ephemeral filesystems and limited persistence";
noexec = mkEnableOption "enables every mount on the system save /nix being marked as noexec (potentially dangerous at a social level)";
};
config = mkIf cfg.enable {
fileSystems."/" = mkForce {
device = "none";
fsType = "tmpfs";
options = [ "defaults" "size=2G" "mode=755" ] ++ ifNoexec;
};
fileSystems."/etc/nixos".options = ifNoexec;
fileSystems."/srv".options = ifNoexec;
fileSystems."/var/lib".options = ifNoexec;
fileSystems."/var/log".options = ifNoexec;
fileSystems."/boot" = {
device = "/dev/disk/by-label/boot";
fsType = "vfat";
};
fileSystems."/nix" = {
device = "/dev/disk/by-label/nix";
autoResize = true;
fsType = "ext4";
};
boot.cleanTmpDir = true;
environment.persistence."/nix/persist" = {
directories = [
"/etc/nixos" # nixos system config files, can be considered optional
"/srv" # service data
"/var/lib" # system service persistent data
"/var/log" # the place that journald dumps it logs to
];
};
environment.etc."ssh/ssh_host_rsa_key".source =
"/nix/persist/etc/ssh/ssh_host_rsa_key";
environment.etc."ssh/ssh_host_rsa_key.pub".source =
"/nix/persist/etc/ssh/ssh_host_rsa_key.pub";
environment.etc."ssh/ssh_host_ed25519_key".source =
"/nix/persist/etc/ssh/ssh_host_ed25519_key";
environment.etc."ssh/ssh_host_ed25519_key.pub".source =
"/nix/persist/etc/ssh/ssh_host_ed25519_key.pub";
environment.etc."machine-id".source = "/nix/persist/etc/machine-id";
};
}
```
This should give us the base that we need to build the system image for AWS.
## Building The Image
As I mentioned earlier we need to build a system image before we can build the
image. NixOS normally hides a lot of this magic from you, but we're going to
scrape away all that magic and do this by hand. In your `paranix-configs`
folder, create a folder named `images`. This creatively named folder is where we
will store our NixOS image generation scripts.
Copy this code into `build.nix`. This will tell NixOS to create a new system
closure with configuration in `images/configuration.nix`:
```nix
# images/build.nix
let
sources = import ../nix/sources.nix;
pkgs = import sources.nixpkgs { };
sys = (import "${sources.nixpkgs}/nixos/lib/eval-config.nix" {
system = "x86_64-linux";
modules = [ ./configuration.nix ];
});
in sys.config.system.build.toplevel
```
And in `images/configuration.nix` add this skeleton config:
```nix
# images/configuration.nix
{ config, pkgs, lib, modulesPath, ... }:
{
imports = [ ../common (modulesPath + "/virtualisation/amazon-image.nix") ];
xeserv.paranoid.enable = true;
}
```
[You can adapt this to other clouds by changing what module is imported. See the
list of available modules here.](conversation://Mara/hacker)
Then you can kick off the build with `nix-build`:
```console
$ nix-build build.nix
```
It will take a moment to assemble everything together and when you are done you
should have an entire functional system closure in `./result`:
```console
$ cat ./result/nixos-version
21.05pre-git
```
[It has `pre-git` here because we're using a pinned commit of the `nixos-21.05`
git branch. Release channels don't have that suffix there.](conversation://Mara/hacker)
From here we need to put this base system closure into a disk image for AWS.
This process is a bit more involved, but here are the high level things needed
to make a disk image for NixOS (or any Linux system for that matter):
- A virtual hard drive to install the OS to
- A partition mapping on the virtual hard drive
- Essential system files copied over
- A boot configuation
We can model this using a Nix function. This function would need to take in the
system config, some metadata about the kind of image to make and then it would
build the image and return the result. I've made this available
[here](https://tulpa.dev/cadey/paranix-configs/src/branch/main/images/make-image.nix)
so you can grab it into your config folder like this:
```console
$ wget -O make-image.nix https://tulpa.dev/cadey/paranix-configs/raw/branch/main/images/make-image.nix
```
Then we can edit `build.nix` to look like this:
```nix
# images/build.nix
let
sources = import ../nix/sources.nix;
pkgs = import sources.nixpkgs { };
config = (import "${sources.nixpkgs}/nixos/lib/eval-config.nix" {
system = "x86_64-linux";
modules = [ ./configuration.nix ];
});
in import ./make-image.nix {
inherit (config) config pkgs;
inherit (config.pkgs) lib;
format = "vpc"; # change this for other clouds
}
```
Then you can build the AWS image with `nix-build`:
```console
$ nix-build build.nix
```
This will emit the AWS disk image in `./result`:
```console
$ ls ./result/
nixos.vhd
```
[AWS uses Microsoft Virtual PC hard disk files as the preferred input for their
vmimport service. This is probably a legacy thing.](conversation://Mara/hacker)
## Terraforming
[Terraform](https://www.terraform.io/) is not my favorite tool on the planet,
however it is quite useful for beating AWS and other clouds into shape. We will
be using Terraform to do the following:
- Create an S3 bucket to use for storing Terraform states in The Cloud
- Create an S3 bucket for the AMI base images
- Create an IAM role for importing AMIs
- Create an IAM role policy for allowing the AMI importer service to work
- Uploading the image to S3
- Import the image from S3 as an EBS snapshot
- Create an AMI from that EBS snapshot
- Create an example t2.micro virtual machine
- Deploy an example service config for nginx that does nothing
This sounds like a lot, but it's really not as much as it sounds. A lot of this
is boilerplate. The cost associated with these steps should be minimal.
In the root of your `paranix-configs` folder, make a folder called `terraform`,
as this is where our terraform configuration will live:
```console
$ mkdir terraform
$ cd terraform
```
Then you can proceed to the following steps.
### S3 State Bucket
In that folder, make a folder called `bootstrap`, this configuration will
contain the base S3 bucket config for Terraform state:
```console
$ mkdir bootstrap
$ cd bootstrap
```
Copy this terraform code into `main.tf`:
```hcl
# terraform/bootstrap/main.tf
provider "aws" {
region = "us-east-1"
}
resource "aws_s3_bucket" "bucket" {
bucket = "xeserv-tf-state-paranix"
acl = "private"
tags = {
Name = "Terraform State"
}
}
```
Then run `terraform init` to set up the terraform environment:
```console
$ terraform init
```
It will download the AWS provider and run a few tests on your config to make
sure things are correct. Once this is done, you can run `terraform plan`:
```console
$ terraform plan
Terraform used the selected providers to generate the following execution plan. Resource actions
are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_s3_bucket.bucket will be created
+ resource "aws_s3_bucket" "bucket" {
+ acceleration_status = (known after apply)
+ acl = "private"
+ arn = (known after apply)
+ bucket = "xeserv-tf-state-paranoid"
+ bucket_domain_name = (known after apply)
+ bucket_regional_domain_name = (known after apply)
+ force_destroy = false
+ hosted_zone_id = (known after apply)
+ id = (known after apply)
+ region = (known after apply)
+ request_payer = (known after apply)
+ tags = {
+ "Name" = "Terraform State"
}
+ tags_all = {
+ "Name" = "Terraform State"
}
+ website_domain = (known after apply)
+ website_endpoint = (known after apply)
+ versioning {
+ enabled = (known after apply)
+ mfa_delete = (known after apply)
}
}
Plan: 1 to add, 0 to change, 0 to destroy.
Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take
exactly these actions if you run "terraform apply" now.
```
Terraform is very pedantic about what the state of the world is. In this case
nothing in the associated state already exists, so it is saying that it needs to
create the S3 bucket that we will use for our Terraform states in the future. We
can apply this with `terraform apply`:
```console
$ terraform apply