--- title: Unix Domain Sockets for Serving HTTP in Production date: 2021-04-01 series: howto tags: - unix - nginx - devops - nixos - systemd --- # Unix Domain Sockets for Serving HTTP in Production Securing production servers can be a chore. It is a seemingly endless game of balancing risks with convenience and not breaking what you want to do. Small, incremental gains are usually a very good idea however. Today we'll learn how to use [Unix Domain Sockets](https://en.wikipedia.org/wiki/Unix_domain_socket) to host your HTTP services. This allows you to run your services like normal on production machines without there being a risk of people being able to access the raw HTTP port. [Wait, what. You're having a _service_ listen on a _file_? Why would you want to do this?](conversation://Mara/hmm?smol) Mostly to prevent you from messing up and accidentally exposing your backend port to the internet. Firewall configuration is probably the most "correct" way to solve that concern, however this lets you also take advantage of filesystem permissions to fine-tune access down to the exact users and groups that should have access to the socket. In our case we only want ngnix to access this socket, so we can use filesystem permissions (and a unix group) to ensure this. Attackers can't connect to anything they aren't able to connect to. [I see. How do you do this?](conversation://Mara/aha?smol) At a high level every file in a unix filesystem has 3 kinds of permissions: user, group and "other". Every file has an owner and a UNIX group associated with it. Here's an example using the [Cargo.toml](https://github.com/Xe/site/blob/main/Cargo.toml) of this website's app server: ```console $ stat ./Cargo.toml File: ./Cargo.toml Size: 1572 Blocks: 8 IO Block: 4096 regular file Device: 10301h/66305d Inode: 20447261 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 1001/ cadey) Gid: ( 100/ users) Access: 2021-04-01 19:48:44.791162535 -0400 Modify: 2021-04-01 19:48:44.786162545 -0400 Change: 2021-04-01 19:48:44.786162545 -0400 Birth: 2021-03-25 09:09:35.490311674 -0400 ``` [The `stat(1)` command lets you query the filesystem for common types of metadata about a given file.](conversation://Mara/hacker?smol) In this case the permissions of this file are `0644`, which is a base-8 (octal) number that describes the permissions for the user, group and others. It breaks up something like this:

unix permissions pic.twitter.com/2WcL6w44FR

— 🔎Julia Evans🔍 (@b0rk) April 7, 2018
If we wanted to create a socket that only nginx can access, assuming we share a group with nginx we would need a socket with something like `0770` (user and group can read, write and "execute", everyone else gets denied) for its permissions. Then we would need to chuck it somewhere that both the app backend and nginx have access to and finally configure nginx to do this. So let's do it! Let's take the venerable [printer facts server](https://tulpa.dev/cadey/printerfacts) server and make it listen on a Unix socket. Right now it uses something like this to listen for requests: ```rust warp::serve( fact_handler .or(index_handler) .or(files) .or(not_found_handler) .with(warp::log(APPLICATION_NAME)), ) .run(([0, 0, 0, 0], port)) .await; ``` This configures [warp](https://github.com/seanmonstar/warp/) (the HTTP framework that I'm using for the printer facts server) to listen over TCP on some port. This is hard-coded to listen on `0.0.0.0`, which means that TCP sessions from _any_ network interface can connect to the service. This is _very_ convenient for development, so we are going to want to keep this behaviour in some way. Fortunately warp [has an example](https://github.com/seanmonstar/warp/blob/master/examples/unix_socket.rs) for listening on a unix socket. Let's make the service listen on `./printerfacts.sock` so we can make sure that everything still works: ```rust let server = warp::serve( fact_handler .or(index_handler) .or(files) .or(not_found_handler) .with(warp::log(APPLICATION_NAME)), ); if let Ok(sockpath) = std::env::var("SOCKPATH") { use tokio::net::UnixListener; use tokio_stream::wrappers::UnixListenerStream; let listener = UnixListener::bind(sockpath).unwrap(); let incoming = UnixListenerStream::new(listener); server.run_incoming(incoming).await; } else { server.run(([0, 0, 0, 0], port)); } ``` Then we can launch the service with a domain socket using a command like this: ```console $ env SOCKPATH=./printerfacts.sock cargo run ``` Let's see how the output of `stat(1)` changed compared to when we ran it on a file: ```console $ stat ./printerfacts.sock File: ./printerfacts.sock Size: 0 Blocks: 0 IO Block: 4096 socket Device: 10301h/66305d Inode: 23858442 Links: 1 Access: (0755/srwxr-xr-x) Uid: ( 1001/ cadey) Gid: ( 100/ users) Access: 2021-04-01 21:00:51.558219253 -0400 Modify: 2021-04-01 21:00:51.558219253 -0400 Change: 2021-04-01 21:00:51.558219253 -0400 Birth: 2021-04-01 21:00:51.558219253 -0400 ``` `stat(1)` reports that the file is a socket! Let's see if everything still works by using `curl --unix-socket` to connect to the service and retrieve an amusing fact about printers: ```console $ curl --unix-socket ./printerfacts.sock http://foo/fact The strongest climber among the big printers, a leopard can carry prey twice its weight up a tree. ``` [Why do you have `foo` as the HTTP hostname for the request?](conversation://Mara/hmm?smol) Because it doesn't matter! I could have anything there, but foo is fast for me to type. The URL host information usually tells curl where to connect, but the `--unix-socket` flag overrides this logic. [Wait, what the heck are printer facts?](conversation://Mara/wat?smol) Blame Foone and #infoforcefeed. Anyways, let's make the TCP logic a bit more clean in the process. Right now it only listens on IPv4 and it would be nice if it listened on IPv6 too. Let's replace that last `else` body with this: ```rust } else { server .run((std::net::IpAddr::from_str("::").unwrap(), port)) .await; } ``` [`::` is the IPv6 version of `0.0.0.0`, or the unspecified address. It tells most IP stacks to allow traffic from any network interface.](conversation://Mara/hacker?smol) Now let's re-build the printer facts service and re-run it to make sure it still works: ```console $ env SOCKPATH=./printerfacts.sock cargo run Finished dev [unoptimized + debuginfo] target(s) in 0.04s Running `target/debug/printerfacts` thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 98, kind: AddrInUse, message: "Address already in use" }', src/main.rs:73:53 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ``` [Wait, what. Isn't this serving HTTP from a file? Why would it be an address in use error?](conversation://Mara/wat?smol) Even though it looks like a file to us humans, it's still a socket under the hood. In this case it means the filename is already in use. Working around this is simple though, all we need to do is- [DELETE THIS!](conversation://Numa/delet?smol) [Where the hell did you come from?](conversation://Cadey/angy?smol) But yes, we do need to delete the socket file if it doesn't already exist. Let's sneak this bit of code in before we listen on the Unix socket: ```rust if let Ok(sockpath) = std::env::var("SOCKPATH") { let _ = std::fs::remove_file(&sockpath); // nuke the socket let listener = UnixListener::bind(sockpath).unwrap(); let incoming = UnixListenerStream::new(listener); server.run_incoming(incoming).await; } else { server .run((std::net::IpAddr::from_str("::").unwrap(), port)) .await; } ``` [Didn't you just say "if it doesn't already exist"? Why delete it unconditionally and throw away any errors?](conversation://Mara/hmm?smol) Two reasons: 1. Statistically if the file doesn't exist and the service can't create it when it binds to that path, you probably have bigger problems and it's probably better for the program to explode there. 2. The filename is passed in as an environment variable. If your environment variable is wrong, we can treat this as a fundamental assertion error and blow up when the file fails to bind. Let's define this in the [NixOS module for the printerfacts service](https://github.com/Xe/nixos-configs/blob/master/common/services/printerfacts.nix). First we will need to add a configuration option for the socket path: ```nix let cfg = config.within.services.printerfacts; in { options.within.services.printerfacts = { # ... sockPath = mkOption rec { type = types.str; default = "/tmp/printerfacts.sock"; example = default; description = "The unix domain socket that printerfacts should listen on"; }; }; # ... } ``` This creates an option at `cfg.sockPath` that we can pipe through elsewhere, such as the start script for the service: ```nix # inside script = let site = pkgs.tulpa.dev.cadey.printerfacts; in '' export SOCKPATH=${cfg.sockPath} export DOMAIN=${toString cfg.domain} export RUST_LOG=info cd ${site} exec ${site}/bin/printerfacts ''; ``` And then we can go on to setting up nginx. First, let's figure out how to reverse proxy to a unix socket. In nginx configuration land, [`proxy_pass`](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) is the name of the configuration directive that lets you tell nginx to reverse proxy to somewhere. There's an example with a unix socket! This would let us reverse proxy a unix socket to a TCP port like this: ```nginx server { listen 127.0.0.1:9000; location / { proxy_pass http://unix:/tmp/printerfacts.sock; } } ``` For comparison here's how you'd reverse proxy to a HTTP server running on port `42069`: ```nginx server { listen 127.0.0.1:9001; location / { proxy_pass http://127.0.0.1:42069; } } ``` So, we just need to change where nginx reverse proxies to in the NixOS config. Let's look down at the nginx config for `printerfacts`: ```nix # ... services.nginx.virtualHosts."${cfg.domain}" = { locations."/" = { proxyPass = "http://127.0.0.1:${toString cfg.port}"; proxyWebsockets = true; }; forceSSL = cfg.useACME; useACMEHost = "cetacean.club"; extraConfig = '' access_log /var/log/nginx/printerfacts.access.log; ''; }; ``` The `proxyPass` option directly translates to a `proxy_pass` directive, so we can get away with something like this: ```nix # ... proxyPass = "http://unix:${cfg.sockPath}"; ``` And now we can deploy the service and everything should work right? printerfacts provides a unix socket at the given path and then nginx is configured to use that socket to send back printer facts. Let's deploy it and see what happens:
![A picture of the nginx "502 Bad Gateway" error message with a man scolding a router](https://cdn.christine.website/file/christine-static/blog/57f66e907bb62.jpeg)
Oh no. Let's see what `journalctl -fu nginx` has to say: ```console $ journalctl -fu nginx Apr 01 23:29:58 lufta nginx[15396]: 2021/04/01 23:29:58 [crit] 15396#15396: *198 connect() to unix:/tmp/printerfacts.sock failed (13: Permission denied) while connecting to upstream, client: lol.no.ip.here, server: printerfacts.cetacean.club, request: "GET / HTTP/2.0", upstream: "http://unix:/tmp/printerfacts.sock:/", host: "printerfacts.cetacean.club" ``` [Wait, what. Isn't `/tmp` guaranteed by the filesystem hierarchy standards to always be readable and writable by any user?](conversation://Mara/wat?smol) Normally, yes. However we are running nginx inside systemd, and one of the things you can do with systemd is make `/tmp` isolated for given services. This allows you to prevent a service from being able to exfiltrate data inside `/tmp`. However, this is definitely NOT the behaviour we want in this case. Let's change the systemd unit for nginx to disable this and also make nginx run as the same group as the printerfacts service: ```nix systemd.services.nginx.serviceConfig = { PrivateTmp = lib.mkForce "false"; SupplementaryGroups = "within"; }; ``` [In NixOS, most of the time if the same option is declared in multiple places it will result in a build error. `lib.mkForce` disables this behaviour and instead "forcibly" sets this value.](conversation://Mara/hacker?smol) Now nginx has the same `/tmp` as the printerfacts service, everything will work as we expect. Users are none the wiser that I'm using a domain socket here. I get to have another service not bound to the network and I have moved towards better security on my machine! [What about Prometheus? Doesn't it need a direct line of fire to the service to scrape metrics?](conversation://Mara/hmm?smol) ...Time for some percussive maintenance!
![](https://cdn.christine.website/file/christine-static/stickers/cadey/percussive-maintenance.png)
--- I'm experimenting with a new "smol" mode for the Mara interludes as well as introducing a few more characters to the christine dot website cinematic universe. Please do let me know how this works out for you. I think I have the sizes optimized for mobile usage better, but [contributions to fix my horrible CSS](https://github.com/Xe/site/blob/main/css/shim.css) would really, really, really be appreciated. I'm considering moving over all of the Mara interludes to use smol mode. If you have opinions about this please let me know them.