</kubernetes>
Signed-off-by: Christine Dodrill <me@christine.website>
This commit is contained in:
parent
951542ccf2
commit
1ae1cc2945
|
|
@ -0,0 +1,229 @@
|
||||||
|
---
|
||||||
|
title: "</kubernetes>"
|
||||||
|
date: 2021-01-03
|
||||||
|
---
|
||||||
|
|
||||||
|
# </kubernetes>
|
||||||
|
|
||||||
|
Well, since I posted [that last post](/blog/k8s-pondering-2020-12-31) I have had
|
||||||
|
an adventure. A good friend pointed out a server host that I had missed when I
|
||||||
|
was looking for other places to use, and now I have migrated my blog to this new
|
||||||
|
server. As of yesterday, I now run my website on a dedicated server in the
|
||||||
|
Netherlands. Here is the story of my journey to migrate 6 years of cruft and
|
||||||
|
technical debt to this new server.
|
||||||
|
|
||||||
|
Let's talk about this goliath of a server. This server is an AX41 from Hetzner.
|
||||||
|
It has 64 GB of ram, a 512 GB nvme drive, 3 2 TB drives, and a Ryzen 3600. For
|
||||||
|
all practical concerns, this beast is beyond overkill and rivals my workstation
|
||||||
|
tower in everything but the GPU power. I have named it `lufta`, which is the
|
||||||
|
word for feather in [L'ewa](https://lewa.within.website/dictionary.html).
|
||||||
|
|
||||||
|
## Assimilation
|
||||||
|
|
||||||
|
For my server setup process, the first step it to assimilate it. In this step I
|
||||||
|
get a base NixOS install on it somehow. Since I was using Hetzner, I was able to
|
||||||
|
boot into a NixOS install image using the process documented
|
||||||
|
[here](https://nixos.wiki/wiki/Install_NixOS_on_Hetzner_Online). Then I decided
|
||||||
|
that it would also be cool to have this server use
|
||||||
|
[zfs](https://en.wikipedia.org/wiki/ZFS) as its filesystem to take advantage of
|
||||||
|
its legendary subvolume and snapshotting features.
|
||||||
|
|
||||||
|
So I wrote up a bootstrap system definition like the Hetzner tutorial said and
|
||||||
|
ended up with `hosts/lufta/bootstrap.nix`:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
{ pkgs, ... }:
|
||||||
|
|
||||||
|
{
|
||||||
|
services.openssh.enable = true;
|
||||||
|
users.users.root.openssh.authorizedKeys.keys = [
|
||||||
|
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPg9gYKVglnO2HQodSJt4z4mNrUSUiyJQ7b+J798bwD9 cadey@shachi"
|
||||||
|
];
|
||||||
|
|
||||||
|
networking.usePredictableInterfaceNames = false;
|
||||||
|
systemd.network = {
|
||||||
|
enable = true;
|
||||||
|
networks."eth0".extraConfig = ''
|
||||||
|
[Match]
|
||||||
|
Name = eth0
|
||||||
|
[Network]
|
||||||
|
# Add your own assigned ipv6 subnet here here!
|
||||||
|
Address = 2a01:4f9:3a:1a1c::/64
|
||||||
|
Gateway = fe80::1
|
||||||
|
# optionally you can do the same for ipv4 and disable DHCP (networking.dhcpcd.enable = false;)
|
||||||
|
Address = 135.181.162.99/26
|
||||||
|
Gateway = 135.181.162.65
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
|
||||||
|
boot.supportedFilesystems = [ "zfs" ];
|
||||||
|
|
||||||
|
environment.systemPackages = with pkgs; [ wget vim zfs ];
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Then I fired up the kexec tarball and waited for the server to boot into a NixOS
|
||||||
|
live environment. A few minutes later I was in. I started formatting the drives
|
||||||
|
according to the [NixOS install
|
||||||
|
guide](https://nixos.org/manual/nixos/stable/index.html#sec-installation) with
|
||||||
|
one major difference: I added a `/boot` ext4 partition on the SSD. This allows
|
||||||
|
me to have the system root device on zfs. I added the disks to a `raidz1` pool
|
||||||
|
and created a few volumes. I also added the SSD as a log device so I get SSD
|
||||||
|
caching.
|
||||||
|
|
||||||
|
From there I installed NixOS as normal and rebooted the server. It booted
|
||||||
|
normally. I had a shiny new NixOS server in the cloud! I noticed that the server
|
||||||
|
had booted into NixOS unstable as opposed to NixOS 20.09 like my other nodes. I
|
||||||
|
thought "ah, well, that probably isn't a problem" and continued to the
|
||||||
|
configuration step.
|
||||||
|
|
||||||
|
[That's ominous...](conversation://Mara/hmm)
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Now that the server was assimilated and I could SSH into it, the next step was
|
||||||
|
to configure it to run my services. While I was waiting for Hetzner to provision
|
||||||
|
my server I ported a bunch of my services over to Nixops services [a-la this
|
||||||
|
post](/blog/nixops-services-2020-11-09) in [this
|
||||||
|
folder](https://github.com/Xe/nixos-configs/tree/master/common/services) of my
|
||||||
|
configs repo.
|
||||||
|
|
||||||
|
Now that I had them, it was time to add this server to my Nixops setup. So I
|
||||||
|
opened the [nixops definition
|
||||||
|
folder](https://github.com/Xe/nixos-configs/tree/master/nixops/hexagone) and
|
||||||
|
added the metadata for `lufta`. Then I added it to my Nixops deployment with
|
||||||
|
this command:
|
||||||
|
|
||||||
|
```console
|
||||||
|
$ nixops modify -d hexagone -n hexagone *.nix
|
||||||
|
```
|
||||||
|
|
||||||
|
Then I copied over the autogenerated config from `lufta`'s `/etc/nixos/` folder
|
||||||
|
into
|
||||||
|
[`hosts/lufta`](https://github.com/Xe/nixos-configs/tree/master/hosts/lufta) and
|
||||||
|
ran a `nixops deploy` to add some other base configuration.
|
||||||
|
|
||||||
|
## Migration
|
||||||
|
|
||||||
|
Once that was done, I started enabling my services and pushing configs to test
|
||||||
|
them. After I got to a point where I thought things would work I opened up the
|
||||||
|
Kubernetes console and started deleting deployments on my kubernetes cluster as
|
||||||
|
I felt "safe" to migrate them over. Then I saw the deployments come back. I
|
||||||
|
deleted them again and they came back again.
|
||||||
|
|
||||||
|
Oh, right. I enabled that one Kubernetes service that made it intentionally hard
|
||||||
|
to delete deployments. One clever set of scale-downs and kills later and I was
|
||||||
|
able to kill things with wild abandon.
|
||||||
|
|
||||||
|
I copied over the gitea data with `rsync` running in the kubernetes deployment.
|
||||||
|
Then I killed the gitea deployment, updated DNS and reran a whole bunch of gitea
|
||||||
|
jobs to resanify the environment. I did a test clone on a few of my repos and
|
||||||
|
then I deleted the gitea volume from DigitalOcean.
|
||||||
|
|
||||||
|
Moving over the other deployments from Kubernetes into NixOS services was
|
||||||
|
somewhat easy, however I did need to repackage a bunch of my programs and static
|
||||||
|
sites for NixOS. I made the
|
||||||
|
[`pkgs`](https://github.com/Xe/nixos-configs/tree/master/pkgs) tree a bit more
|
||||||
|
fleshed out to compensate.
|
||||||
|
|
||||||
|
[Okay, packaging static sites in NixOS is beyond overkill, however a lot of them
|
||||||
|
need some annoyingly complicated build steps and throwing it all into Nix means
|
||||||
|
that we can make them reproducible and use one build system to rule them
|
||||||
|
all. Not to mention that when I need to upgrade the system, everything will
|
||||||
|
rebuild with new system libraries to avoid the <a
|
||||||
|
href="https://blog.tidelift.com/bit-rot-the-silent-killer">Docker bitrot
|
||||||
|
problem</a>.](conversation://Mara/hacker)
|
||||||
|
|
||||||
|
## Reboot Test
|
||||||
|
|
||||||
|
After a significant portion of the services were moved over, I decided it was
|
||||||
|
time to do the reboot test. I ran the `reboot` command and then...nothing.
|
||||||
|
My continuous ping test was timing out. My phone was blowing up with downtime
|
||||||
|
messages from NodePing. Yep, I messed something up.
|
||||||
|
|
||||||
|
I was able to boot the server back into a NixOS recovery environment using the
|
||||||
|
kexec trick, and from there I was able to prove the following:
|
||||||
|
|
||||||
|
- The zfs setup is healthy
|
||||||
|
- I can read some of the data I migrated over
|
||||||
|
- I can unmount and remount the ZFS volumes repeatedly
|
||||||
|
|
||||||
|
I was confused. This shouldn't be happening. After half an hour of
|
||||||
|
troubleshooting, I gave in and ordered an IPKVM to be installed in my server.
|
||||||
|
|
||||||
|
Once that was set up (and I managed to trick MacOS into letting me boot a .jnlp
|
||||||
|
web start file), I rebooted the server so I could see what error I was getting
|
||||||
|
on boot. I missed it the first time around, but on the second time I was able to
|
||||||
|
capture this screenshot:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Then it hit me. I did the install on NixOS unstable. My other servers use NixOS
|
||||||
|
20.09. I had downgraded zfs and the older version of zfs couldn't mount the
|
||||||
|
volume created by the newer version of zfs in read/write mode. One more trip to
|
||||||
|
the recovery environment later to install NixOS unstable in a new generation.
|
||||||
|
|
||||||
|
Then I switched my tower's default NixOS channel to the unstable channel and ran
|
||||||
|
`nixops deploy` to reactivate my services. After the NodePing uptime
|
||||||
|
notifications came in, I ran the reboot test again while looking at the console
|
||||||
|
output to be sure.
|
||||||
|
|
||||||
|
It booted. It worked. I had a stable setup. Then I reconnected to IRC and passed
|
||||||
|
out.
|
||||||
|
|
||||||
|
## Services Migrated
|
||||||
|
|
||||||
|
Here is a list of all of the services I have migrated over from my old dedicated
|
||||||
|
server, my kubernetes cluster and my dokku server:
|
||||||
|
|
||||||
|
- aerial -> discord chatbot
|
||||||
|
- goproxy -> go modules proxy
|
||||||
|
- lewa -> https://lewa.within.website
|
||||||
|
- hlang -> https://h.christine.website
|
||||||
|
- mi -> https://mi.within.website
|
||||||
|
- printerfacts -> https://printerfacts.cetacean.club
|
||||||
|
- xesite -> https://christine.website
|
||||||
|
- graphviz -> https://graphviz.christine.website
|
||||||
|
- idp -> https://idp.christine.website
|
||||||
|
- oragono -> ircs://irc.within.website:6697/
|
||||||
|
- tron -> discord bot
|
||||||
|
- withinbot -> discord bot
|
||||||
|
- withinwebsite -> https://within.website
|
||||||
|
- gitea -> https://tulpa.dev
|
||||||
|
- other static sites
|
||||||
|
|
||||||
|
Doing this migration is a bit of an archaeology project as well. I was
|
||||||
|
continuously discovering services that I had littered over my machines with very
|
||||||
|
poorly documented requirements and configuration. I hope that this move will let
|
||||||
|
the next time I do this kind of migration be a lot easier by comparison.
|
||||||
|
|
||||||
|
I still have a few other services to move over, however the ones that are left
|
||||||
|
are much more annoying to set up properly. I'm going to get to deprovision 5
|
||||||
|
servers in this migration and as a result get this stupidly powerful goliath of
|
||||||
|
a server to do whatever I want with and I also get to cut my monthly server
|
||||||
|
costs by over half.
|
||||||
|
|
||||||
|
I am very close to being able to turn off the Kubernetes cluster and use NixOS
|
||||||
|
for everything. A few services that are still on the Kubernetes cluster are
|
||||||
|
resistant to being nixified, so I may have to use the Docker containers for
|
||||||
|
that. I was hoping to be able to cut out Docker entirely, however we don't seem
|
||||||
|
to be that lucky yet.
|
||||||
|
|
||||||
|
Sure, there is some added latency with the server being in Europe instead of
|
||||||
|
Montreal, however if this ever becomes a practical issue I can always launch a
|
||||||
|
cheap DigitalOcean VPS in Toronto to act as a DNS server for my WireGuard setup.
|
||||||
|
|
||||||
|
Either way, I am now off Kubernetes for my highest traffic services. If services
|
||||||
|
of mine need to use the disk, they can now just use the disk. If I really care
|
||||||
|
about the data, I can add the service folders to the list of paths to back up to
|
||||||
|
`rsync.net` (I have a post about how this backup process works in the drafting
|
||||||
|
stage) via [borgbackup](https://www.borgbackup.org/).
|
||||||
|
|
||||||
|
Let's hope it stays online!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Many thanks to [Graham Christensen](https://twitter.com/grhmc), [Dave
|
||||||
|
Anderson](https://twitter.com/dave_universetf) and everyone else who has been
|
||||||
|
helping me along this journey. I would be lost without them.
|
||||||
|
|
@ -9,6 +9,13 @@ fn main() -> Result<()> {
|
||||||
.output()
|
.output()
|
||||||
.unwrap();
|
.unwrap();
|
||||||
let git_hash = String::from_utf8(output.stdout).unwrap();
|
let git_hash = String::from_utf8(output.stdout).unwrap();
|
||||||
println!("cargo:rustc-env=GITHUB_SHA={}", git_hash);
|
println!(
|
||||||
|
"cargo:rustc-env=GITHUB_SHA={}",
|
||||||
|
if git_hash.as_str() == "" {
|
||||||
|
env!("out").into()
|
||||||
|
} else {
|
||||||
|
git_hash
|
||||||
|
}
|
||||||
|
);
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@
|
||||||
<blockquote>Copyright 2020 Christine Dodrill. Any and all opinions listed here are my own and not representative of my employers; future, past and present.</blockquote>
|
<blockquote>Copyright 2020 Christine Dodrill. Any and all opinions listed here are my own and not representative of my employers; future, past and present.</blockquote>
|
||||||
<!--<p>Like what you see? Donate on <a href="https://www.patreon.com/cadey">Patreon</a> like <a href="/patrons">these awesome people</a>!</p>-->
|
<!--<p>Like what you see? Donate on <a href="https://www.patreon.com/cadey">Patreon</a> like <a href="/patrons">these awesome people</a>!</p>-->
|
||||||
<p>Looking for someone for your team? Take a look <a href="/signalboost">here</a>.</p>
|
<p>Looking for someone for your team? Take a look <a href="/signalboost">here</a>.</p>
|
||||||
<p>Served by @APP running commit <a href="https://github.com/Xe/site/commit/@env!("GITHUB_SHA")">@env!("GITHUB_SHA")</a>, see <a href="https://github.com/Xe/site">source code here</a>.</p>
|
<p>Served by @APP running @env!("out")/bin/xesite</a>, see <a href="https://github.com/Xe/site">source code here</a>.</p>
|
||||||
</footer>
|
</footer>
|
||||||
|
|
||||||
</div>
|
</div>
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue