From 43057536ad09252121779461021406a0833cdcde Mon Sep 17 00:00:00 2001 From: Christine Dodrill Date: Thu, 31 Dec 2020 12:10:37 -0500 Subject: [PATCH] blog: Kubernetes pondering post Signed-off-by: Christine Dodrill --- blog/k8s-pondering-2020-12-31.markdown | 160 +++++++++++++++++++++++++ templates/header.rs.html | 2 +- 2 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 blog/k8s-pondering-2020-12-31.markdown diff --git a/blog/k8s-pondering-2020-12-31.markdown b/blog/k8s-pondering-2020-12-31.markdown new file mode 100644 index 0000000..21b044c --- /dev/null +++ b/blog/k8s-pondering-2020-12-31.markdown @@ -0,0 +1,160 @@ +--- +title: Kubernetes Pondering +date: 2020-12-31 +tags: + - k8s + - kubernetes + - soyoustart + - kimsufi + - digitalocean + - vultr +--- + +# Kubernetes Pondering + +Right now I am using a freight train to mail a letter when it comes to hosting +my web applications. If you are reading this post on the day it comes out, then +you are connected to one of a few replicas of my site code running across at +least 3 machines in my Kubernetes cluster. This certainly _works_, however it is +not very ergonomic and ends up being quite expensive. + +I think I made a mistake when I decided to put my cards into Kubernetes for my +personal setup. It made sense at the time (I was trying to learn Kubernetes and +I am cursed into learning by doing), however I don't think it is really the best +choice available for my needs. I am not a large company. I am a single person +making things that are really targeted for myself. I would like to replace this +setup with something more at my scale. Here are a few options I have been +exploring combined with their pros and cons. + +Here are the services I currently host on my Kubernetes cluster: + +- [this site](/) +- [my git server](https://tulpa.dev) +- [hlang](https://h.christine.website) +- A few personal services that I've been meaning to consolidate +- The [olin demo](https://olin.within.website/) +- The venerable [printer facts server](https://printerfacts.cetacean.club) +- A few static websites +- An IRC server (`irc.within.website`) + +My goal in evaluating other options is to reduce cost and complexity. Kubernetes +is a very complicated system and requires a lot of hand-holding and rejiggering +to make it do what you want. NixOS, on the other hand, is a lot simpler overall +and I would like to use it for running my services where I can. + +Cost is a huge factor in this. My Kubernetes setup is a money pit. I want to +prioritize cost reduction as much as possible. + +## Option 1: Do Nothing + +I could do nothing about this and eat the complexity as a cost of having this +website and those other services online. However over the year or so I've been +using Kubernetes I've had to do a lot of hacking at it to get it to do what I +want. + +I set up the cluster using Terraform and Helm 2. Helm 3 is the current +(backwards-incompatible) release, and all of the things that are managed by Helm +2 have resisted being upgraded to Helm 3. + +I'm going to say something slightly controversial here, but YAML is a HORRIBLE +format for configuration. I can't trust myself to write unambiguous YAML. I have +to reference the spec constantly to make sure I don't have an accidental +Norway/Ontario bug. I have a Dhall package that takes away most of the pain, +however it's not flexible enough to describe the entire scope of what my +services need to do (IE: pinging Google/Bing to update their indexes on each +deploy), and I don't feel like putting in the time to make it that flexible. + +[This is the regex for determining what is a valid boolean value in YAML: +`y|Y|yes|Yes|YES|n|N|no|No|NO|true|True|TRUE|false|False|FALSE|on|On|ON|off|Off|OFF`. +This can bite you eventually. See the Norway +Problem for more information.](conversation://Mara/hacker) + +I have a tor hidden service endpoint for a few of my services. I have to use an +[unmaintained tool](https://github.com/kragniz/tor-controller) to manage these +on Kubernetes. It works _today_, but the Kubernetes operator API could change at +any time (or the API this uses could be deprecated and removed without much +warning) and leave me in the dust. + +I could live with all of this, however I don't really think it's the best idea +going forward. There's a bunch of services that I added on top of Kubernetes +that are dangerous to upgrade and very difficult (if not impossible) to +downgrade when something goes wrong during the upgrade. + +One of the big things that I have with this setup that I would have to rebuild +in NixOS is the continuous deployment setup. However I've done that before and +it wouldn't really be that much of an issue to do it again. + +NixOS fixes all the jank I mentioned above by making my specifications not have +to include the version numbers of everything the system already provides. You +can _actually trust the package repos to have up to date packages_. I don't +have to go around and bump the versions of shims and pray they work, because +with NixOS I don't need them anymore. + +## Option 2: NixOS on top of SoYouStart or Kimsufi + +This is a doable option. The main problem here would be doing the provision +step. SoYouStart and Kimsufi (both are offshoot/discount brands of OVH) have +very little in terms of customization of machine config. They work best when you +are using "normal" distributions like Ubuntu or CentOS and leave them be. I +would want to run NixOS on it and would have to do several trial and error runs +with a tool such as [nixos-infect](https://github.com/elitak/nixos-infect) to +assimilate the server into running NixOS. + +With this option I would get the most storage out of any other option by far. 4 +TB is a _lot_ of space. However, SoYouStart and Kimsufi run decade-old hardware +at best. I would end up paying a lot for very little in the CPU department. For +most things I am sure this would be fine, however some of my services can have +CPU needs that might exceed what second-generation Xeons can provide. + +SoYouStart and Kimsufi have weird kernel versions though. The last SoYouStart +dedi I used ran Fedora and was gimped with a grsec kernel by default. I had to +end up writing [this gem of a systemd service on +boot](https://github.com/Xe/dotfiles/blob/master/ansible/roles/soyoustart/files/conditional-kexec.sh) +which did a [`kexec`](https://en.wikipedia.org/wiki/Kexec) to boot into a +non-gimped kernel on boot. It was a huge hack and somehow worked every time. I +was still afraid to reboot the machine though. + +Sure is a lot of ram for the cost though. + +## Option 3: NixOS on top of Digital Ocean + +This shares most of the problems as the SoYouStart or Kimsufi nodes. However, +nixos-infect is known to have a higher success rate on Digital Ocean droplets. +It would be really nice if Digital Ocean let you upload arbitrary ISO files and +go from there, but that is apparently not the world we live in. + +8 GB of ram would be _way more than enough_ for what I am doing with these +services. + +## Option 4: NixOS on top of Vultr + +Vultr is probably my top pick for this. You can upload an arbitrary ISO file, +kick off your VPS from it and install it like normal. I have a little shell +server shared between some friends built on top of such a Vultr node. It works +beautifully. + +The fact that it has the same cost as the Digital Ocean droplet just adds to the +perfection of this option. + +## Costs + +Here is the cost table I've drawn up while comparing these options: + +| Option | Ram | Disk | Cost per month | Hacks | +| :--------- | :----------------- | :------------------------------------ | :-------------- | :----------- | +| Do nothing | 6 GB (4 GB usable) | Not really usable, volumes cost extra | $60/month | Very Yes | +| SoYouStart | 32 GB | 2x2TB SAS | $40/month | Yes | +| Kimsufi | 32 GB | 2x2TB SAS | $35/month | Yes | +| Digital Ocean | 8 GB | 160 GB SSD | $40/month | On provision | +| Vultr | 8 GB | 160 GB SSD | $40/month | No | + +I think I am going to go with the Vultr option. I will need to modernize some of +my services to support being deployed in NixOS in order to do this, however I +think that I will end up creating a more robust setup in the process. At least I +will create a setup that allows me to more easily maintain my own backups rather +than just relying on DigitalOcean snapshots and praying like I do with the +Kubernetes setup. + +Thanks farcaller, Marbles, John Rinehart and others for reviewing this post +prior to it being published. diff --git a/templates/header.rs.html b/templates/header.rs.html index ca49a36..a1e2d4d 100644 --- a/templates/header.rs.html +++ b/templates/header.rs.html @@ -15,7 +15,7 @@ - @if Utc::now().month() == 12 { } + @if Utc::now().month() == 12 || Utc::now().month() == 1 || Utc::now().month() == 2 { }