diff --git a/blog/paranoid-nixos-aws-2021-08-11.markdown b/blog/paranoid-nixos-aws-2021-08-11.markdown new file mode 100644 index 0000000..0059844 --- /dev/null +++ b/blog/paranoid-nixos-aws-2021-08-11.markdown @@ -0,0 +1,1292 @@ +--- +title: Paranoid NixOS on AWS +date: 2021-08-11 +author: Heartmender +series: nixos +tags: + - paranix + - aws + - r13y +--- + +In [the last post](https://christine.website/blog/paranoid-nixos-2021-07-18) we +covered a lot of the base groundwork involved in making a paranoid NixOS setup. +Today we're gonna throw this into prod by making a base NixOS image with it. + +[Normally I don't suggest people throw these things into production directly, if +only to have some kind of barrier between you and your money generator; however +today is different. It's probably not completely unsafe to put this in +production, but I really would suggest reading and understanding this article +before doing so.](conversation://Cadey/coffee) + +At a high level we are going to do the following: + +- Pin production OS versions using [niv](https://github.com/nmattia/niv) +- Create a script to automatically generate a production-ready NixOS image that + you can import into The Cloud +- Manage all this using your favorite buzzwords (Terraform, + Infrastructure-as-Code) +- Install an nginx server reverse proxying to the [Printer facts + service](https://printerfacts.cetacean.club/) + +## What is an Image? + +Before we yolo this all into prod, let's cover what we're actually doing. +There are a lot of conflicting buzzwords here, so I'm going to go out of my way +to attempt to simplify them down so that we use my arbitrary definitions of +buzzwords instead of what other people will imply they mean. You're reading my +blog, you get my buzzwords; it's as simple as that. + +In this post we are going to create a base system that you can build your +production systems on top of. This base system will be crystallized into an +_image_ that AWS will use as the initial starting place for servers. + +[So you create the system definition for your base system, then turn that into +an image and put that image into AWS?](conversation://Mara/hmm) + +[Yep! The exact steps are a little more complicated but at a high level that's +what we're doing.](conversation://Cadey/enby) + +## Base Setup + +I'm going to be publishing my work for this post +[here](https://tulpa.dev/cadey/paranix-configs), but you can follow along in +this post to understand the individual steps here. + +First, let's set up the environment with +[lorri](https://github.com/nix-community/lorri) and +[niv](https://github.com/nmattia/niv). Lorri will handle creating a cached +nix-shell environment for us to run things in and niv will handle pinning NixOS +to an exact version so you can get a more reproducible production environment. + +Set up lorri: + +```console +$ lorri init +Aug 11 09:41:50.966 INFO wrote file, path: ./shell.nix +Aug 11 09:41:50.966 INFO wrote file, path: ./.envrc +Aug 11 09:41:50.966 INFO done +direnv: error /home/cadey/code/cadey/paranix-configs/.envrc is blocked. Run `direnv allow` to approve its content +$ direnv allow +direnv: loading ~/code/cadey/paranix-configs/.envrc +Aug 11 09:41:54.581 INFO lorri has not completed an evaluation for this project yet, nix_file: /home/cadey/code/cadey/paranix-configs/shell.nix +direnv: export +IN_NIX_SHELL +``` + +[Why are you putting the `$` before every command in these examples? It looks +extraneous to me.](conversation://Mara/hacker) + +[The `$` is there for two main reasons. First, it allows there to be a clear +delineation between the commands being typed and their output. Secondly it makes +it slightly harder to blindly copy this into your shell without either editing +the `$` out or selecting around it. My hope is that this will make you read the +command and carefully consider whether or not you actually want to run +it.](conversation://Cadey/enby) + +Set up niv: + +```console +$ niv init +Initializing + Creating nix/sources.nix + Creating nix/sources.json + Importing 'niv' ... + Adding package niv + Writing new sources file + Done: Adding package niv + Importing 'nixpkgs' ... + Adding package nixpkgs + Writing new sources file + Done: Adding package nixpkgs +Done: Initializing +``` + +[If you don't already have niv in your environment, you can hack around that by +running all the niv commands before you set up `shell.nix` like this: 
$ nix-shell -p niv --run 'niv blah'
](conversation://Mara/hacker) + +And finally pin nixpkgs to a specific version of NixOS. + +[At the time of writing this article, NixOS 21.05 is the stable release, so that +is what is used here.](conversation://Mara/hacker) + +```console +$ niv update nixpkgs -b nixos-21.05 +Update nixpkgs +Done: Update nixpkgs +$ +``` + +This will become the foundation of our NixOS systems and production images. + +You should then set up your `shell.nix` to look like this: + +```nix +let + sources = import ./nix/sources.nix; + pkgs = import ./sources.nixpkgs { }; +in pkgs.mkShell { + buildInputs = with pkgs; [ + niv + terraform + + bashInteractive + ]; +}; +``` + +### Set Up Unix Accounts + +[This step can be omitted if you are grafting this into an existing NixOS +configs repository, however it would be good to read through this to understand +the directory layout at play here.](conversation://Mara/hacker) + +It's probably important to be able to have access to production machines. Let's +create a NixOS module that will allow you to SSH into the machine. In your +paranix-configs folder, run this command to make a `common` config directory: + +```console +$ mkdir common +$ cd common +``` + +Now in that common directory, open `default.nix` in ~~emacs~~ your favorite text +editor and copy in this skeleton: + +```nix +# common/default.nix + +{ config, lib, pkgs, ... }: + +{ + imports = [ ./users.nix ]; + + nix.autoOptimiseStore = true; + + users.users.root.openssh.authorizedKeys.keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPg9gYKVglnO2HQodSJt4z4mNrUSUiyJQ7b+J798bwD9" ]; + + services.tailscale.enable = true; + + # Tell the firewall to implicitly trust packets routed over Tailscale: + networking.firewall.trustedInterfaces = [ "tailscale0" ]; + + security.auditd.enable = true; + security.audit.enable = true; + security.audit.rules = [ + "-a exit,always -F arch=b64 -S execve" + ]; + + security.sudo.execWheelOnly = true; + environment.defaultPackages = lib.mkForce []; + + services.openssh = { + passwordAuthentication = false; + allowSFTP = false; # Don't set this if you need sftp + challengeResponseAuthentication = false; + extraConfig = '' + AllowTcpForwarding yes + X11Forwarding no + AllowAgentForwarding no + AllowStreamLocalForwarding no + AuthenticationMethods publickey + ''; + }; + + # PCI compliance + environment.systemPackages = with pkgs; [ clamav ]; +} +``` + +[Astute readers will notice that this is less paranoid than the last post. This +was pared down after private feedback.](conversation://Mara/hacker) + +This will create `common` as a folder that can be imported as a NixOS module +with some basic settings and then tells NixOS to try importing `users.nix` as a +module. This module doesn't exist yet, so it will fail when we try to import it. +Let's fix that by making `users.nix`: + +```nix +# common/users.nix + +{ config, lib, pkgs, ... }: + +with lib; + +let + # These options will be used for user account defaults in + # the `mkUser` function. + xeserv.users = { + groups = mkOption { + type = types.listOf types.str; + default = [ "wheel" ]; + example = ''[ "wheel" "libvirtd" "docker" ]''; + description = + "The Unix groups that Xeserv staff users should be assigned to"; + }; + + shell = mkOption { + type = types.package; + default = pkgs.bashInteractive; + example = "pkgs.powershell"; + description = + "The default shell that Xeserv staff users will be given by default."; + }; + }; + + cfg = config.xeserv.users; + + mkUser = { keys, shell ? cfg.shell, extraGroups ? cfg.groups, ... }: { + isNormalUser = true; + inherit extraGroups shell; + openssh.authorizedKeys = { + inherit keys; + }; + }; +in { + options.xeserv.users = xeserv.users; + + config.users.users = { + cadey = mkUser { + keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPg9gYKVglnO2HQodSJt4z4mNrUSUiyJQ7b+J798bwD9" ]; + }; + twi = mkUser { + keys = [ "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYr9hiLtDHgd6lZDgQMkJzvYeAXmePOrgFaWHAjJvNU" ]; + }; + }; +} +``` + +[It's worth noting that `xeserv` in there can be anything you want. It's set to +`xeserv` as we are imagining that this is for the production environment of a +company named Xeserv.](conversation://Mara/hacker) + +### Paranoid Settings + +Next we're going to set up the paranoid settings from the last post into a +module named `paranoid.nix`. First we'll need to grab +[impermanence](https://github.com/nix-community/impermanence) into our niv +manifest like this: + +```console +$ niv add nix-community/impermanence +Adding package impermanence + Writing new sources file +Done: Adding package impermanence +``` + +Then open `common/default.nix` and change this line: + +```nix +imports = [ ./users.nix ]; +``` + +To something like this: + +```nix +imports = [ ./paranoid.nix ./users.nix ]; +``` + +Then open `./paranoid.nix` in a text editor and paste in the following: + +```nix +# common/paranoid.nix + +{ config, pkgs, lib, ... }: + +with lib; + +let + sources = import ../nix/sources.nix; + impermanence = sources.impermanence; + cfg = config.xeserv.paranoid; + + ifNoexec = if cfg.noexec then [ "noexec" ] else [ ]; +in { + imports = [ "${impermanence}/nixos.nix" ]; + + options.xeserv.paranoid = { + enable = mkEnableOption "enables ephemeral filesystems and limited persistence"; + noexec = mkEnableOption "enables every mount on the system save /nix being marked as noexec (potentially dangerous at a social level)"; + }; + + config = mkIf cfg.enable { + fileSystems."/" = mkForce { + device = "none"; + fsType = "tmpfs"; + options = [ "defaults" "size=2G" "mode=755" ] ++ ifNoexec; + }; + + fileSystems."/etc/nixos".options = ifNoexec; + fileSystems."/srv".options = ifNoexec; + fileSystems."/var/lib".options = ifNoexec; + fileSystems."/var/log".options = ifNoexec; + + fileSystems."/boot" = { + device = "/dev/disk/by-label/boot"; + fsType = "vfat"; + }; + + fileSystems."/nix" = { + device = "/dev/disk/by-label/nix"; + autoResize = true; + fsType = "ext4"; + }; + + boot.cleanTmpDir = true; + + environment.persistence."/nix/persist" = { + directories = [ + "/etc/nixos" # nixos system config files, can be considered optional + "/srv" # service data + "/var/lib" # system service persistent data + "/var/log" # the place that journald dumps it logs to + ]; + }; + + environment.etc."ssh/ssh_host_rsa_key".source = + "/nix/persist/etc/ssh/ssh_host_rsa_key"; + environment.etc."ssh/ssh_host_rsa_key.pub".source = + "/nix/persist/etc/ssh/ssh_host_rsa_key.pub"; + environment.etc."ssh/ssh_host_ed25519_key".source = + "/nix/persist/etc/ssh/ssh_host_ed25519_key"; + environment.etc."ssh/ssh_host_ed25519_key.pub".source = + "/nix/persist/etc/ssh/ssh_host_ed25519_key.pub"; + environment.etc."machine-id".source = "/nix/persist/etc/machine-id"; + }; +} +``` + +This should give us the base that we need to build the system image for AWS. + +## Building The Image + +As I mentioned earlier we need to build a system image before we can build the +image. NixOS normally hides a lot of this magic from you, but we're going to +scrape away all that magic and do this by hand. In your `paranix-configs` +folder, create a folder named `images`. This creatively named folder is where we +will store our NixOS image generation scripts. + +Copy this code into `build.nix`. This will tell NixOS to create a new system +closure with configuration in `images/configuration.nix`: + +```nix +# images/build.nix + +let + sources = import ../nix/sources.nix; + pkgs = import sources.nixpkgs { }; + sys = (import "${sources.nixpkgs}/nixos/lib/eval-config.nix" { + system = "x86_64-linux"; + modules = [ ./configuration.nix ]; + }); +in sys.config.system.build.toplevel +``` + +And in `images/configuration.nix` add this skeleton config: + +```nix +# images/configuration.nix + +{ config, pkgs, lib, modulesPath, ... }: + +{ + imports = [ ../common (modulesPath + "/virtualisation/amazon-image.nix") ]; + + xeserv.paranoid.enable = true; +} +``` + +[You can adapt this to other clouds by changing what module is imported. See the +list of available modules here.](conversation://Mara/hacker) + +Then you can kick off the build with `nix-build`: + +```console +$ nix-build build.nix +``` + +It will take a moment to assemble everything together and when you are done you +should have an entire functional system closure in `./result`: + +```console +$ cat ./result/nixos-version +21.05pre-git +``` + +[It has `pre-git` here because we're using a pinned commit of the `nixos-21.05` +git branch. Release channels don't have that suffix there.](conversation://Mara/hacker) + +From here we need to put this base system closure into a disk image for AWS. +This process is a bit more involved, but here are the high level things needed +to make a disk image for NixOS (or any Linux system for that matter): + +- A virtual hard drive to install the OS to +- A partition mapping on the virtual hard drive +- Essential system files copied over +- A boot configuation + +We can model this using a Nix function. This function would need to take in the +system config, some metadata about the kind of image to make and then it would +build the image and return the result. I've made this available +[here](https://tulpa.dev/cadey/paranix-configs/src/branch/main/images/make-image.nix) +so you can grab it into your config folder like this: + +```console +$ wget -O make-image.nix https://tulpa.dev/cadey/paranix-configs/raw/branch/main/images/make-image.nix +``` + +Then we can edit `build.nix` to look like this: + +```nix +# images/build.nix + +let + sources = import ../nix/sources.nix; + pkgs = import sources.nixpkgs { }; + config = (import "${sources.nixpkgs}/nixos/lib/eval-config.nix" { + system = "x86_64-linux"; + modules = [ ./configuration.nix ]; + }); + +in import ./make-image.nix { + inherit (config) config pkgs; + inherit (config.pkgs) lib; + format = "vpc"; # change this for other clouds +} +``` + +Then you can build the AWS image with `nix-build`: + +```console +$ nix-build build.nix +``` + +This will emit the AWS disk image in `./result`: + +```console +$ ls ./result/ +nixos.vhd +``` + +[AWS uses Microsoft Virtual PC hard disk files as the preferred input for their +vmimport service. This is probably a legacy thing.](conversation://Mara/hacker) + +## Terraforming + +[Terraform](https://www.terraform.io/) is not my favorite tool on the planet, +however it is quite useful for beating AWS and other clouds into shape. We will +be using Terraform to do the following: + +- Create an S3 bucket to use for storing Terraform states in The Cloud +- Create an S3 bucket for the AMI base images +- Create an IAM role for importing AMIs +- Create an IAM role policy for allowing the AMI importer service to work +- Uploading the image to S3 +- Import the image from S3 as an EBS snapshot +- Create an AMI from that EBS snapshot +- Create an example t2.micro virtual machine +- Deploy an example service config for nginx that does nothing + +This sounds like a lot, but it's really not as much as it sounds. A lot of this +is boilerplate. The cost associated with these steps should be minimal. + +In the root of your `paranix-configs` folder, make a folder called `terraform`, +as this is where our terraform configuration will live: + +```console +$ mkdir terraform +$ cd terraform +``` + +Then you can proceed to the following steps. + +### S3 State Bucket + +In that folder, make a folder called `bootstrap`, this configuration will +contain the base S3 bucket config for Terraform state: + +```console +$ mkdir bootstrap +$ cd bootstrap +``` + +Copy this terraform code into `main.tf`: + +```hcl +# terraform/bootstrap/main.tf + +provider "aws" { + region = "us-east-1" +} + +resource "aws_s3_bucket" "bucket" { + bucket = "xeserv-tf-state-paranix" + acl = "private" + + tags = { + Name = "Terraform State" + } +} +``` + +Then run `terraform init` to set up the terraform environment: + +```console +$ terraform init +``` + +It will download the AWS provider and run a few tests on your config to make +sure things are correct. Once this is done, you can run `terraform plan`: + +```console +$ terraform plan +Terraform used the selected providers to generate the following execution plan. Resource actions +are indicated with the following symbols: + + create + +Terraform will perform the following actions: + + # aws_s3_bucket.bucket will be created + + resource "aws_s3_bucket" "bucket" { + + acceleration_status = (known after apply) + + acl = "private" + + arn = (known after apply) + + bucket = "xeserv-tf-state-paranoid" + + bucket_domain_name = (known after apply) + + bucket_regional_domain_name = (known after apply) + + force_destroy = false + + hosted_zone_id = (known after apply) + + id = (known after apply) + + region = (known after apply) + + request_payer = (known after apply) + + tags = { + + "Name" = "Terraform State" + } + + tags_all = { + + "Name" = "Terraform State" + } + + website_domain = (known after apply) + + website_endpoint = (known after apply) + + + versioning { + + enabled = (known after apply) + + mfa_delete = (known after apply) + } + } + +Plan: 1 to add, 0 to change, 0 to destroy. + +Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take +exactly these actions if you run "terraform apply" now. +``` + +Terraform is very pedantic about what the state of the world is. In this case +nothing in the associated state already exists, so it is saying that it needs to +create the S3 bucket that we will use for our Terraform states in the future. We +can apply this with `terraform apply`: + +```console +$ terraform apply + + +Do you want to perform these actions? + Terraform will perform the actions described above. + Only 'yes' will be accepted to approve. + + Enter a value: +``` + +If you want to perform these actions, follow the instructions. + +```console + Enter a value: yes + +aws_s3_bucket.bucket: Creating... +aws_s3_bucket.bucket: Creation complete after 3s [id=xeserv-tf-state-paranoid] + +Apply complete! Resources: 1 added, 0 changed, 0 destroyed. +``` + +Now that we have the state bucket, let's use it to create our AMI. + +### Creating the AMI + +In your `terraform` folder, create a new folder called `aws_image`. This is +where the terraform configuration for uploading our disk image to AWS will live. + +```console +$ mkdir aws_image +$ cd aws_image +``` + +[This part of the config is modified from the instructions on how to create an +AMI from a locally created VM image here.](conversation://Mara/hacker) + +Make a file called `main.tf` and we'll add to it as we go through this section. + +In `main.tf`, add the following boilerplate to make the AWS provider use the +terraform state bucket we just created: + +```hcl +# terraform/aws_image/main.tf + +provider "aws" { + region = "us-east-1" +} + +terraform { + backend "s3" { + bucket = "xeserv-tf-state-paranoid" + key = "aws_image" + region = "us-east-1" + } +} +``` + +This will tell the AWS provider to use the S3 bucket we just made, but also to +put the terraform state in a key called `aws_image`. We will reuse this state +later for making our printer facts host. After we do this, we should run +`terraform init` to make sure that the state bucket is working: + +```console +$ terraform init +Initializing the backend... + +Initializing provider plugins... +- Finding latest version of hashicorp/aws... +- Installing hashicorp/aws v3.53.0... +- Installed hashicorp/aws v3.53.0 (signed by HashiCorp) + +Terraform has created a lock file .terraform.lock.hcl to record the provider +selections it made above. Include this file in your version control repository +so that Terraform can guarantee to make the same selections by default when +you run "terraform init" in the future. + +Terraform has been successfully initialized! + +You may now begin working with Terraform. Try running "terraform plan" to see +any changes that are required for your infrastructure. All Terraform commands +should now work. + +If you ever set or change modules or backend configuration for Terraform, +rerun this command to reinitialize your working directory. If you forget, other +commands will detect it and remind you to do so if necessary. +``` + +Now let's create the S3 bucket that we will put our NixOS image in: + +```hcl +# terraform/aws_image/main.tf + +resource "aws_s3_bucket" "images" { + bucket = "xeserv-ami-images" + acl = "private" + + tags = { + Name = "Xeserv AMI Images" + } +} +``` + +Then let's create the IAM role and policy that allows the VM importer service +to import objects from S3 into EBS snapshots that we use to create an AMI. + +In the `aws_image` folder, copy this trust policy statement into +`vmie-trust-policy.json`: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Principal": { "Service": "vmie.amazonaws.com" }, + "Action": "sts:AssumeRole", + "Condition": { + "StringEquals":{ + "sts:Externalid": "vmimport" + } + } + } + ] +} +``` + +This will be used to give the VM import service permission to act against AWS on +your behalf. + +In `main.tf`, add the following role and policy to the configuration: + +```hcl +# terraform/aws_image/main.tf + +resource "aws_iam_role" "vmimport" { + name = "vmimport" + assume_role_policy = file("./vmie-trust-policy.json") +} + +resource "aws_iam_role_policy" "vmimport_policy" { + name = "vmimport" + role = aws_iam_role.vmimport.id + policy = <ARN, +and in order to give the vmimport service the minimal possible permissions, we +need to template out that policy JSON file, and doing this inline in Terraform +is a lot simpler.](conversation://Cadey/enby) + +And now we should run `terraform plan` and `terraform apply` to make sure +everything works okay: + +```console +$ terraform plan + +Plan: 3 to add, 0 to change, 0 to destroy. + +$ terraform apply + +Apply complete! Resources: 3 added, 0 changed, 0 destroyed. +``` + +Perfect! Now we need to upload the image to S3. You are going to have to build +the NixOS image outside of terraform, so run `nix-build`: + +```console +$ nix-build ../../build.nix +``` + +This should largely be a no-op and will put the correct `result` symlink in your +`aws_image` folder so terraform can read the image metadata. + +[Practically you would want to make a script to run terraform, and in the script +for this folder you would probably want to add that `nix-build` command to that +script. However this is trivial and is thus an exercise for the +reader.](conversation://Mara/hacker) + +In your `main.tf` file, add this: + +```hcl +# terraform/aws_image/main.tf + +resource "aws_s3_bucket_object" "nixos_21_05" { + bucket = aws_s3_bucket.images.bucket + key = "nixos-21.05-paranoid.vhd" + + source = "./result/nixos.vhd" + etag = filemd5("./result/nixos.vhd") +} +``` + +Now we need to create the EBS snapshot. Copy this into your `main.tf`: + +```hcl +# terraform/aws_image/main.tf + +resource "aws_ebs_snapshot_import" "nixos_21_05" { + disk_container { + format = "VHD" + user_bucket { + s3_bucket = aws_s3_bucket.images.bucket + s3_key = aws_s3_bucket_object.nixos_21_05.key + } + } + + role_name = aws_iam_role.vmimport.name + + tags = { + Name = "NixOS-21.05" + } +} +``` + +This step may take a while (more than 5 minutes), so let's run `terraform plan` +and then `terraform apply`: + +```console +$ terraform plan +Plan: 2 to add, 0 to change, 0 to destroy. + +$ terraform apply +Apply complete! Resources: 2 added, 0 changed, 0 destroyed. +``` + +Finally you can create the AMI and export the AMI ID like this: + +```hcl +# terraform/aws_image/main.tf + +resource "aws_ami" "nixos_21_05" { + name = "nixos_21_05" + architecture = "x86_64" + virtualization_type = "hvm" + root_device_name = "/dev/xvda" + ena_support = true + sriov_net_support = "simple" + + ebs_block_device { + device_name = "/dev/xvda" + snapshot_id = aws_ebs_snapshot_import.nixos_21_05.id + volume_size = 40 # you can go as low as 8 GB, but 40 is a nice number + delete_on_termination = true + volume_type = "gp3" + } +} + +output "nixos_21_05_ami" { + value = aws_ami.nixos_21_05.id +} +``` + +Then run `terraform plan` and `terraform apply`: + +```console +$ terraform plan +Plan: 1 to add, 0 to change, 0 to destroy. + +$ terraform apply +Apply complete! Resources: 1 added, 0 changed, 0 destroyed. + +Outputs: + +nixos_21_05_ami = "ami-0f43f74cbbdd1ddef" +``` + +Et voila! We have a NixOS base image that we can use for production workloads. +Let's use it to create a NixOS server running the [printer facts +service](https://printerfacts.cetacean.club/). + +[**KEEP IN MIND** that this configuration means that every time you +rebuild and upload this image you potentially risk breaking production machines. +Don't rebuild this config more than once every 6 months (or when you bump to a +new release of NixOS) at most.](conversation://Mara/hacker) + +### Using the AMI + +Let's make a new folder in the `terraform` folder called `printerfacts`. In this +folder we're going to set up a new terraform state that imports the AMI state we +just made and then we will use that AMI to run the printer facts service. + +```console +$ mkdir printerfacts +$ cd printerfacts +``` + +In `main.tf`, copy the following: + +```hcl +# terraform/printerfacts/main.tf + +provider "aws" { + region = "us-east-1" +} + +terraform { + backend "s3" { + bucket = "xeserv-tf-state-paranoid" + key = "printerfacts" + region = "us-east-1" + } +} +``` + +Now you can `terraform init` as normal to ensure everything is working as we +expect: + +```console +$ terraform init +Terraform has been successfully initialized! +``` + +Then let's add the `aws_image` state as a data source. This will let us +reference the AMI ID from the remote state file instead of having to build it +from scratch every time. + +```hcl +# terraform/printerfacts/main.tf + +data "terraform_remote_state" "aws_image" { + backend = "s3" + + config = { + bucket = "xeserv-tf-state-paranoid" + key = "aws_image" + region = "us-east-1" + } +} +``` + +AWS wants us to create a keypair for the instance, so to make AWS happy we will +make a keypair like this: + +```hcl +# terraform/printerfacts/main.tf + +resource "tls_private_key" "state_ssh_key" { + algorithm = "RSA" +} + +resource "aws_key_pair" "generated_key" { + key_name = "generated-key-${sha256(tls_private_key.state_ssh_key.public_key_openssh)}" + public_key = tls_private_key.state_ssh_key.public_key_openssh +} +``` + +[You will need to `terraform init` after this step.](conversation://Mara/hacker) + +Now we need to create a security group for this instance. This security group +should do the following: + +- Allow port 22 (ssh) ingress +- Allow port 80 (http) ingress +- Allow ICMP (ping) ingress +- Allow ICMP (ping) egress +- Allow TCP egress on all ports to everywhere +- Allow UDP egress on all ports to everywhere + +You can do this with this terraform fragment: + +```hcl +# terraform/printerfacts/main.tf + +resource "aws_security_group" "printerfacts" { + ingress { + from_port = 22 + to_port = 22 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + ingress { + from_port = 80 + to_port = 80 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + ingress { + from_port = -1 + to_port = -1 + protocol = "icmp" + cidr_blocks = ["0.0.0.0/0"] + } + egress { + from_port = -1 + to_port = -1 + protocol = "icmp" + cidr_blocks = ["0.0.0.0/0"] + } + egress { + from_port = 0 + to_port = 65535 + protocol = "tcp" + cidr_blocks = ["0.0.0.0/0"] + } + egress { + from_port = 0 + to_port = 65535 + protocol = "udp" + cidr_blocks = ["0.0.0.0/0"] + } +} +``` + +Then we can create the AWS instance using our AMI, keypair and security group: + +```hcl +# terraform/printerfacts/main.tf + +resource "aws_instance" "printerfacts" { + ami = data.terraform_remote_state.aws_image.outputs.nixos_21_05_ami + instance_type = "t3.micro" + security_groups = [ + aws_security_group.printerfacts.name, + ] + key_name = aws_key_pair.generated_key.key_name + + root_block_device { + volume_size = 40 # GiB + } + + tags = { + Name = "xe-printerfacts" + } +} +``` + +And then we can create a NixOS deploy config with the fantastic +[deploy_nixos](https://github.com/tweag/terraform-nixos/tree/master/deploy_nixos) +module from Tweag. Copy this into your `main.tf`: + +```hcl +# terraform/printerfacts/main.tf + +module "deploy_printerfacts" { + source = "git::https://github.com/Xe/terraform-nixos.git//deploy_nixos?ref=1b49f2c6b4e7537cca6dd6d7b530037ea81e8268" + nixos_config = "${path.module}/printerfacts.nix" + hermetic = true + target_user = "root" + target_host = aws_instance.printerfacts.public_ip + ssh_private_key = tls_private_key.state_ssh_key.private_key_pem + ssh_agent = false + build_on_target = false +} +``` + +[You will need to `terraform init` again after this +step.](conversation://Mara/hacker) + +Now let's make the `printerfacts.nix` host definition. We're going to start with +a simple config to begin with. This will start nginx in a mostly broken but +still semi-functional state on port 80. + +```nix +# terraform/printerfacts/printerfacts.nix + +let + sources = import ../../nix/sources.nix; + pkgs = import sources.nixpkgs { }; + system = "x86_64-linux"; + + configuration = { config, lib, pkgs, ... }: { + imports = [ + ../../common + "${sources.nixpkgs}/nixos/modules/virtualisation/amazon-image.nix" + ]; + + networking.firewall.allowedTCPPorts = [ 22 80 ]; + + xeserv.paranoid.enable = true; + + services.nginx.enable = true; + }; +in import "${sources.nixpkgs}/nixos" { inherit system configuration; } +``` + +[What is up with that config? It doesn't look like a normal NixOS module at +all.](conversation://Mara/hmm) + +[That is a NixOS config that will use the pinned version of nixpkgs with niv in +order to build everything. It won't work everywhere, however the `hermetic` flag +in the `deploy_nixos` Terraform module will make this +work.](conversation://Cadey/enby) + +Now let's deploy all this and see if it works! + +```console +$ terraform init + +$ terraform plan + +$ terraform apply +``` + +### Printerfacts Install + +Now we can add the printerfacts service to the VM. First, add the printerfacts +repo to niv: + +```console +$ niv add git -n printerfacts --repo https://tulpa.dev/cadey/printerfacts +Done: Adding package printerfacts +``` + +Then create a service definition for it in your `common` folder. First create +the folder `common/services`: + +``` +$ cd ../.. +$ cd common +$ mkdir services +$ cd services +``` + +Then create a `default.nix` file with the following contents: + +```nix +# common/services/default.nix + +{ ... }: + +{ + imports = [ ./printerfacts.nix ]; +} +``` + +And create `./printerfacts.nix` with this service boilerplate: + +```nix +# common/services/printerfacts.nix + +{ config, pkgs, lib, ... }: + +with lib; + +let + sources = import ../../nix/sources.nix; + pkg = pkgs.callPackage sources.printerfacts { }; + cfg = config.xeserv.services.printerfacts; +in +{ + options.xeserv.services.printerfacts = { + enable = mkEnableOption "enable Printerfacts"; + useACME = mkEnableOption "enable ACME certs"; + + domain = mkOption { + type = types.str; + default = "printerfacts.akua"; + example = "printerfacts.cetacean.club"; + description = + "The domain name that nginx should check against for HTTP hostnames"; + }; + + port = mkOption { + type = types.int; + default = 28318; + example = 9001; + description = + "The port number printerfacts should listen on for HTTP traffic"; + }; + }; + + config = mkIf cfg.enable { + systemd.services.printerfacts = { + wantedBy = [ "multi-user.target" ]; + + script = '' + export PORT=${toString cfg.port} + export DOMAIN=${toString cfg.domain} + export RUST_LOG=info + exec ${pkg}/bin/printerfacts + ''; + + serviceConfig = { + Restart = "always"; + RestartSec = "30s"; + WorkingDirectory = "${pkg}"; + RuntimeDirectory = "printerfacts"; + RuntimeDirectoryMode = "0755"; + StateDirectory = "tailscale"; + StateDirectoryMode = "0750"; + CacheDirectory = "tailscale"; + CacheDirectoryMode = "0750"; + DynamicUser = "yes"; + }; + }; + + services.nginx.virtualHosts."${cfg.domain}" = { + locations."/" = { + proxyPass = "http://127.0.0.1:${toString cfg.port}"; + proxyWebsockets = true; + }; + enableACME = cfg.useACME; + }; + }; +} +``` + +Then wire up `common/default.nix` with this: + +```nix +# common/default.nix + +imports = [ ./paranoid.nix ./users.nix ./services ]; +``` + +Then you can add this to your machine config in the terraform directory: + +```nix +# terraform/printerfacts/printerfacts.nix + +configuration = { config, lib, pkgs, ... }: { + # ... + + xeserv.services.printerfacts = { + enable = true; + domain = "3.237.88.228"; # replace this with the IP of your AWS instance + }; +}; +``` + +Then `terraform plan` and `terraform apply`: + +```console +$ terraform plan + +$ terraform apply +``` + +And finally get yourself a hard-earned printer fact: + +```console +$ curl http://3.237.88.228/fact +In 1987 printers overtook scanners as the number one pet in America. +``` + +--- + +We have gone from nothing to a fully production-ready NixOS deployment including +a custom AMI pinned to an exact version of NixOS and an additional service added +from its git repo. This will allow you to create a NixOS deployment that can be +used by multiple people but will also stay pinned to an exact version of NixOS. +Terraform will do all of the NixOS building and ensure that things are kept up +to date, meaning that your infrastructure is all configured using the same +workflow. + +This post outlines boilerplate and templates. I'm sure that you could easily +adapt these templates for other things as well. If you need to store persistent +data, make sure its being put in `/var/lib` so that it isn't wiped on reboot. +This took at least a week of research, banging my head against the wall and so +many failures to implement this. Many thanks to [Graham +Christensen](https://twitter.com/grhmc) for unblocking me on this and pulling me +back from the chasm a few times. + +Hope this helps your prod NixOS adventures!