---
title: The Surreal Horror of PAM
date: 2021-11-09
slides_link: /static/talks/surreal-horror-pam.pdf
tags:
 - alpinelinux
 - pam
 - satire
---

<iframe width="1043" height="587" src="https://www.youtube.com/embed/INjCiHUIjgg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>

[https://youtu.be/INjCiHUIjgg](https://youtu.be/INjCiHUIjgg)

---

![](/static/talks/surreal-horror-pam/001.jpeg)

Hi, I’m Xe. You know this because that is what your computer tells you. But how
does it know that?

This is a partially satirical talk. It is intended to be mostly factually
accurate, however some of the details are stretched for comedic effect. This
talk may contain opinions, none of these opinions are the opinions of my
employer. I hope you enjoy this catharsis.

You may want to make sure you all are muted, as this is probably going to make
you laugh and I’ll get speech jammed if I hear it. I’m also planning on
publishing this publicly, so please avoid mentioning privileged information in
the Q&A section at the end.

![](/static/talks/surreal-horror-pam/002.jpeg)

So before we talk about complicated things, let’s start with the basics. This is
how UNIX systems authenticate. They have some files in /etc/ that are
effectively plaintext databases for usernames, ids, groups and password hashes,
and those are what are used for this legacy authentication flow.

You start with a login program running as root (such as /bin/login) and then it
gets your username and password. Then it checks /etc/passwd to see if your user
account exists. If it does it grabs the user ID and uses that to look up more
information from /etc/groups to build up your dossier. Then it takes your
password and does some crazy math to it to compare it to the hashed password in
/etc/shadow. If it matches (or if someone forces it to match through malicious
means), then the login program forks a child process, impersonates your user
account based on that dossier from earlier, creates a login shell and finally
sends you off to do whatever it is you want to the poor computer.

That’s it. That’s how the classic System V authentication stack works.
Technically I’m stretching things a bit as /etc/shadow was a fairly recent
addition (mostly because /etc/passwd is world-readable by design for some arcane
reason that I can’t find on Google), but it’s basically that. There’s a few
steps I’m leaving out for brevity, but they are boring things that only nerds
care about.

![](/static/talks/surreal-horror-pam/003.jpeg)

This UNIX authentication model is really that simple. You can explain the high
level details on a single slide that was hastily written at 8 am in about 5
minutes. However, because of this simplicity it leaves attackers with a small
list of targets to try to trick the computer into mucking with. But it does
work, mostly.

Some of the huge downsides are that it only works on one machine at the time.
This made sense for when UNIX was created as the model was to have a big ol
mainframe for a company and then have everyone connect to it, but in the
meantime we’ve gone around carrying supercomputers in our watches and always
having a calculator in our pockets (Miss Van Hamme, you should have been more
forward thinking than to insinuate otherwise in my second grade math class!).
Because of this (and the fact that said supercomputer watches also run a full
fledged UNIX kernel), we can’t really rely on a model created for 1970’s
mainframe technology to get the job done in this day and age of hyperconverged
cloud federated femtoservices.

![](/static/talks/surreal-horror-pam/004.jpeg)

The last sentence probably set off the beard twitching alarms, so yes there are
some workarounds here.

You could JUST put the files on a network filesystem. That would make them
immutable to whoever tries to mess with them on an individual machine. However
the peak of network filesystem security on Linux is “don’t get your network
hacked lol”. They won’t add the ability for a kernel mode filesystem driver to
make TLS validated sessions so you can use this thing called cryptography to
secure access to the filesystem and data on the wire. They are busy arguing with
people about how to send plain-text email and the like. You could also put those
files on a CD, set the immutable flag or something, but all that will do is
making changing passwords more expensive, annoying and filled with anger.

![](/static/talks/surreal-horror-pam/005.jpeg)

What’s that? I think I hear something coming.

![](/static/talks/surreal-horror-pam/006.jpeg)

It’s sshd! Turns out that we do in fact need something more complicated because
we have networks and the cloud and complicated mutifactor auth requirements for
acronym compliance! We can’t really do that with UNIX authentication because it
was designed before such things were even a glimmer in the eye of security
professionals.

Surely there has to be a better option out there _somewhere_.

![](/static/talks/surreal-horror-pam/007.jpeg)

Et voila! C’est le PAM! Turns out someone else a long time ago had the same
problems and somehow got legal to sign off on making it open source! PAM is a
modular system for making authentication and authorization work.

![](/static/talks/surreal-horror-pam/008.jpeg)

For reference, authentication and authorization are being split up into two
concepts here (like they are in a lot of the industry). We’re gonna take a page
out of the white hat’s guide to security here and call these concepts
authentication (who you are and how we know who you are) and authorization (can
you _really_ take all the money out of the bank account?). It is a solid 90’s
solution to a 70’s problem and good god it shows.

![](/static/talks/surreal-horror-pam/009.jpeg)

PAM was made in the 90’s by this little startup nobody here has heard of called
Sun Microsystems. They had a problem where they had a bunch of machines to apply
complicated authentication rules to (all thanks to those pesky enterprise
contracts) and no way to really do it. Money won this valiant fight between
engineering and sales, so we ended up with PAM.

![](/static/talks/surreal-horror-pam/010.jpeg)

So you’re probably wondering something along the lines of “how does this thing
work?”. Carefully, that's how.

![](/static/talks/surreal-horror-pam/011.jpeg)

This is a screenshot of a text file (a common thing to do these days) of the
main PAM configuration file in a distribution called Alpine Linux. I’m using
Alpine Linux here because it is the simpler option for getting PAM to work and I
really do not want to spend all day debugging PAM with gdb and strace on Ubuntu
to demonstrate it with that. PAM has a few kinds of modules:

* authentication, this is not just checking your password, but also making sure
  that your account is allowed to be logged into and setting up things like your
  preferred login shell
* account, the things that assign a user an account based on the circumstances
  of their authentication or validate that somehow (this is also where an LDAP
  server would get thrown into the mix if you really hate yourself)
* password, the things that check passwords or do other kinds of validation like
  that (if you want to use Google Authenticator TOTP codes, you’d do that here)
* session, these things handle other system errata like making sure the
  message-of-the-day (MOTD) is shown when you log in or letting logind know
  about the session so it can make a cgroup for you

All of these modules are implemented as dynamically linked libraries in C (the
HEIGHT of modern programming security, as we all know) and PAM works by loading
all of these files out of /lib/security, throwing them directly into ram and
then executing arbitrary C ABI functions out of them to see what they return.

Yes, really. I am still surprised that the modules are written in C and not Java
given it was from Sun.

If you typo this configuration file and don’t have a root session open with your
box, it is even worse than typoing the /etc/sudoers file. Typoing /etc/sudoers
will just make it impossible to use sudo. Your system can limp along in the
meantime or you can directly login as root or something to mend the situation,
but typoing the PAM file will cause glibc to hold your wife and children hostage
until you forcibly reboot the system and hack back into it so you can regain
control.

![](/static/talks/surreal-horror-pam/012.jpeg)

How is this relevant to us? Well, I have a bit of a side project going on. I’ve
been trying to write a PAM module that would use Tailscale as its authentication
method.

When you are on a Tailscale network, you are already past a two-factor auth
trust barrier. If we know who you are, and you are authorized to connect to the
server by its ACLs, why should we subject you to the surreal horror of local
authentication logic in order to let you SSH into the server? We know who you
are. You’re allowed to connect to it, so why stop you?

![](/static/talks/surreal-horror-pam/013.jpeg)

The heart of the PAM module I’ve been writing looks like this right now. It sets
up syslog for its log sink (this is really your only good option in PAM land)
with a syslog client, grabs the status of the network from tailscaled, and
finally makes sure that the IP address is in the tailnet. This probably should
be more complicated in the future, I’ve had ideas for sending a TSMP message to
the source machine to prompt you with a “are you sure you want to do that” style
message, but they are just ideas right now. But yes, the rest is a bunch of
random boilerplate code to deal with PAM’s complexities, making sure that the C
ABI functions are exposed correctly and other helpers to grab things from
tailscaled with unix sockets.

[https://github.com/tailscale/pam](https://github.com/tailscale/pam)

This is written in Rust because I personally believe that writing security
critical components that we would ship with the operating system in C is a
massive disservice to our users. Go also doesn’t really have a good story to do
interoperability with core C system components like this (the Go runtime is
_massive_ and as of writing this post the entire PAM module I’ve written is
smaller than the Go runtime, even with a statically compiled copy of libcurl).

Plus I also get to use this to point out the little question mark at the end of
the third line of this code blurb. See that question mark? It is an “if err !=
nil, return nil, err” statement. It’s handled at compile time and it will even
return the Ok side of the result if there is one. God I understand why Go can’t
have that nice thing but it would take at least 7 lines of code out of my
keyboard firmware if we had that nice thing in Go (not to mention countless
editor macros for other people).

If you want to peek around the C module part of the PAM project, the QR code
will take you there.

![](/static/talks/surreal-horror-pam/014.jpeg)

I’m fairly sure that I can get away with this (I made my appropriate sacrifices
to the demo gods this morning), so let’s try SSHing into a VM on my laptop. If
you are watching the recording of this talk or you are not in the corp tailnet,
that command will not work. However you should see something like this:

![](/static/talks/surreal-horror-pam/015.jpeg)

It would be really cool to flesh this out as a full product. I feel this could
really make people’s lives a lot easier. The hard part is going to be making
sure that this absolutely has security experts pore over this to make sure that
this is _actually_ safe. I’m fairly sure that it is safe as it is, but right now
this is an uberhammer that lets you log in as root if you get SSH access to a
system. I would love to have this send a TSMP message to have a GUI prompt
validate that you want to do this as a kind of second factor for authentication,
but even in this limited state I feel it has a lot of value as is.

![](/static/talks/surreal-horror-pam/016.jpeg)

Something you may wonder (and something I had to wonder too) is how do you debug
PAM?

![](/static/talks/surreal-horror-pam/017.jpeg)

It ain’t easy. I’m currently trying to get this thing to work on Ubuntu and all
of the paths I have taken are fraught with despair. I have luckily not managed
to lock myself out of the system yet, but it is really fighting me. You know
you’re in for a ride when obscure PDFs of ring-binder manuals that have been
poorly maintained tell you to do things that literally do not exist anymore.
I’ve had to use a combination of a debugger and a system call tracing tool to
get anywhere with it. PAM is a surreal horror because the most terrifying part
is that it works and that there’s not really any good other options.

![](/static/talks/surreal-horror-pam/018.jpeg)

This is not OpenBSD or Plan 9. This is Linux and macOS. Those exist but we can’t
use them because we are cursed into using PAM. Especially so if we want to do
this on arbitrary customer machines.

![](/static/talks/surreal-horror-pam/019.jpeg)

That’s the end of the talk! I want to give special thanks to the council of
elders that I summoned the help of in order to get this far. Without their help
(and at least 800 bing points worth of searching) I would have never been able
to understand this at all. If you have any questions, you can ask them now; just
remember that you probably are still on mute.

---

As a note to people who are reading: if you want my wit, charm and/or smarmy
style to grace your conference of choice, please [get in contact with
me](/contact) and I'll see what I can do to make it happen.
