blog: Kubernetes pondering post
Signed-off-by: Christine Dodrill <me@christine.website>
This commit is contained in:
parent
2389af7ee5
commit
43057536ad
|
|
@ -0,0 +1,160 @@
|
||||||
|
---
|
||||||
|
title: Kubernetes Pondering
|
||||||
|
date: 2020-12-31
|
||||||
|
tags:
|
||||||
|
- k8s
|
||||||
|
- kubernetes
|
||||||
|
- soyoustart
|
||||||
|
- kimsufi
|
||||||
|
- digitalocean
|
||||||
|
- vultr
|
||||||
|
---
|
||||||
|
|
||||||
|
# Kubernetes Pondering
|
||||||
|
|
||||||
|
Right now I am using a freight train to mail a letter when it comes to hosting
|
||||||
|
my web applications. If you are reading this post on the day it comes out, then
|
||||||
|
you are connected to one of a few replicas of my site code running across at
|
||||||
|
least 3 machines in my Kubernetes cluster. This certainly _works_, however it is
|
||||||
|
not very ergonomic and ends up being quite expensive.
|
||||||
|
|
||||||
|
I think I made a mistake when I decided to put my cards into Kubernetes for my
|
||||||
|
personal setup. It made sense at the time (I was trying to learn Kubernetes and
|
||||||
|
I am cursed into learning by doing), however I don't think it is really the best
|
||||||
|
choice available for my needs. I am not a large company. I am a single person
|
||||||
|
making things that are really targeted for myself. I would like to replace this
|
||||||
|
setup with something more at my scale. Here are a few options I have been
|
||||||
|
exploring combined with their pros and cons.
|
||||||
|
|
||||||
|
Here are the services I currently host on my Kubernetes cluster:
|
||||||
|
|
||||||
|
- [this site](/)
|
||||||
|
- [my git server](https://tulpa.dev)
|
||||||
|
- [hlang](https://h.christine.website)
|
||||||
|
- A few personal services that I've been meaning to consolidate
|
||||||
|
- The [olin demo](https://olin.within.website/)
|
||||||
|
- The venerable [printer facts server](https://printerfacts.cetacean.club)
|
||||||
|
- A few static websites
|
||||||
|
- An IRC server (`irc.within.website`)
|
||||||
|
|
||||||
|
My goal in evaluating other options is to reduce cost and complexity. Kubernetes
|
||||||
|
is a very complicated system and requires a lot of hand-holding and rejiggering
|
||||||
|
to make it do what you want. NixOS, on the other hand, is a lot simpler overall
|
||||||
|
and I would like to use it for running my services where I can.
|
||||||
|
|
||||||
|
Cost is a huge factor in this. My Kubernetes setup is a money pit. I want to
|
||||||
|
prioritize cost reduction as much as possible.
|
||||||
|
|
||||||
|
## Option 1: Do Nothing
|
||||||
|
|
||||||
|
I could do nothing about this and eat the complexity as a cost of having this
|
||||||
|
website and those other services online. However over the year or so I've been
|
||||||
|
using Kubernetes I've had to do a lot of hacking at it to get it to do what I
|
||||||
|
want.
|
||||||
|
|
||||||
|
I set up the cluster using Terraform and Helm 2. Helm 3 is the current
|
||||||
|
(backwards-incompatible) release, and all of the things that are managed by Helm
|
||||||
|
2 have resisted being upgraded to Helm 3.
|
||||||
|
|
||||||
|
I'm going to say something slightly controversial here, but YAML is a HORRIBLE
|
||||||
|
format for configuration. I can't trust myself to write unambiguous YAML. I have
|
||||||
|
to reference the spec constantly to make sure I don't have an accidental
|
||||||
|
Norway/Ontario bug. I have a Dhall package that takes away most of the pain,
|
||||||
|
however it's not flexible enough to describe the entire scope of what my
|
||||||
|
services need to do (IE: pinging Google/Bing to update their indexes on each
|
||||||
|
deploy), and I don't feel like putting in the time to make it that flexible.
|
||||||
|
|
||||||
|
[This is the regex for determining what is a valid boolean value in YAML:
|
||||||
|
`y|Y|yes|Yes|YES|n|N|no|No|NO|true|True|TRUE|false|False|FALSE|on|On|ON|off|Off|OFF`.
|
||||||
|
This can bite you eventually. See the <a
|
||||||
|
href="https://hitchdev.com/strictyaml/why/implicit-typing-removed/">Norway
|
||||||
|
Problem</a> for more information.](conversation://Mara/hacker)
|
||||||
|
|
||||||
|
I have a tor hidden service endpoint for a few of my services. I have to use an
|
||||||
|
[unmaintained tool](https://github.com/kragniz/tor-controller) to manage these
|
||||||
|
on Kubernetes. It works _today_, but the Kubernetes operator API could change at
|
||||||
|
any time (or the API this uses could be deprecated and removed without much
|
||||||
|
warning) and leave me in the dust.
|
||||||
|
|
||||||
|
I could live with all of this, however I don't really think it's the best idea
|
||||||
|
going forward. There's a bunch of services that I added on top of Kubernetes
|
||||||
|
that are dangerous to upgrade and very difficult (if not impossible) to
|
||||||
|
downgrade when something goes wrong during the upgrade.
|
||||||
|
|
||||||
|
One of the big things that I have with this setup that I would have to rebuild
|
||||||
|
in NixOS is the continuous deployment setup. However I've done that before and
|
||||||
|
it wouldn't really be that much of an issue to do it again.
|
||||||
|
|
||||||
|
NixOS fixes all the jank I mentioned above by making my specifications not have
|
||||||
|
to include the version numbers of everything the system already provides. You
|
||||||
|
can _actually trust the package repos to have up to date packages_. I don't
|
||||||
|
have to go around and bump the versions of shims and pray they work, because
|
||||||
|
with NixOS I don't need them anymore.
|
||||||
|
|
||||||
|
## Option 2: NixOS on top of SoYouStart or Kimsufi
|
||||||
|
|
||||||
|
This is a doable option. The main problem here would be doing the provision
|
||||||
|
step. SoYouStart and Kimsufi (both are offshoot/discount brands of OVH) have
|
||||||
|
very little in terms of customization of machine config. They work best when you
|
||||||
|
are using "normal" distributions like Ubuntu or CentOS and leave them be. I
|
||||||
|
would want to run NixOS on it and would have to do several trial and error runs
|
||||||
|
with a tool such as [nixos-infect](https://github.com/elitak/nixos-infect) to
|
||||||
|
assimilate the server into running NixOS.
|
||||||
|
|
||||||
|
With this option I would get the most storage out of any other option by far. 4
|
||||||
|
TB is a _lot_ of space. However, SoYouStart and Kimsufi run decade-old hardware
|
||||||
|
at best. I would end up paying a lot for very little in the CPU department. For
|
||||||
|
most things I am sure this would be fine, however some of my services can have
|
||||||
|
CPU needs that might exceed what second-generation Xeons can provide.
|
||||||
|
|
||||||
|
SoYouStart and Kimsufi have weird kernel versions though. The last SoYouStart
|
||||||
|
dedi I used ran Fedora and was gimped with a grsec kernel by default. I had to
|
||||||
|
end up writing [this gem of a systemd service on
|
||||||
|
boot](https://github.com/Xe/dotfiles/blob/master/ansible/roles/soyoustart/files/conditional-kexec.sh)
|
||||||
|
which did a [`kexec`](https://en.wikipedia.org/wiki/Kexec) to boot into a
|
||||||
|
non-gimped kernel on boot. It was a huge hack and somehow worked every time. I
|
||||||
|
was still afraid to reboot the machine though.
|
||||||
|
|
||||||
|
Sure is a lot of ram for the cost though.
|
||||||
|
|
||||||
|
## Option 3: NixOS on top of Digital Ocean
|
||||||
|
|
||||||
|
This shares most of the problems as the SoYouStart or Kimsufi nodes. However,
|
||||||
|
nixos-infect is known to have a higher success rate on Digital Ocean droplets.
|
||||||
|
It would be really nice if Digital Ocean let you upload arbitrary ISO files and
|
||||||
|
go from there, but that is apparently not the world we live in.
|
||||||
|
|
||||||
|
8 GB of ram would be _way more than enough_ for what I am doing with these
|
||||||
|
services.
|
||||||
|
|
||||||
|
## Option 4: NixOS on top of Vultr
|
||||||
|
|
||||||
|
Vultr is probably my top pick for this. You can upload an arbitrary ISO file,
|
||||||
|
kick off your VPS from it and install it like normal. I have a little shell
|
||||||
|
server shared between some friends built on top of such a Vultr node. It works
|
||||||
|
beautifully.
|
||||||
|
|
||||||
|
The fact that it has the same cost as the Digital Ocean droplet just adds to the
|
||||||
|
perfection of this option.
|
||||||
|
|
||||||
|
## Costs
|
||||||
|
|
||||||
|
Here is the cost table I've drawn up while comparing these options:
|
||||||
|
|
||||||
|
| Option | Ram | Disk | Cost per month | Hacks |
|
||||||
|
| :--------- | :----------------- | :------------------------------------ | :-------------- | :----------- |
|
||||||
|
| Do nothing | 6 GB (4 GB usable) | Not really usable, volumes cost extra | $60/month | Very Yes |
|
||||||
|
| SoYouStart | 32 GB | 2x2TB SAS | $40/month | Yes |
|
||||||
|
| Kimsufi | 32 GB | 2x2TB SAS | $35/month | Yes |
|
||||||
|
| Digital Ocean | 8 GB | 160 GB SSD | $40/month | On provision |
|
||||||
|
| Vultr | 8 GB | 160 GB SSD | $40/month | No |
|
||||||
|
|
||||||
|
I think I am going to go with the Vultr option. I will need to modernize some of
|
||||||
|
my services to support being deployed in NixOS in order to do this, however I
|
||||||
|
think that I will end up creating a more robust setup in the process. At least I
|
||||||
|
will create a setup that allows me to more easily maintain my own backups rather
|
||||||
|
than just relying on DigitalOcean snapshots and praying like I do with the
|
||||||
|
Kubernetes setup.
|
||||||
|
|
||||||
|
Thanks farcaller, Marbles, John Rinehart and others for reviewing this post
|
||||||
|
prior to it being published.
|
||||||
|
|
@ -15,7 +15,7 @@
|
||||||
<link rel="stylesheet" href="/css/gruvbox-dark.css" />
|
<link rel="stylesheet" href="/css/gruvbox-dark.css" />
|
||||||
<link rel="stylesheet" href="/css/shim.css" />
|
<link rel="stylesheet" href="/css/shim.css" />
|
||||||
<link rel="stylesheet" href="https://cdn.christine.website/file/christine-static/prism/prism.css" />
|
<link rel="stylesheet" href="https://cdn.christine.website/file/christine-static/prism/prism.css" />
|
||||||
@if Utc::now().month() == 12 { <link rel="stylesheet" href="/css/snow.css" /> }
|
@if Utc::now().month() == 12 || Utc::now().month() == 1 || Utc::now().month() == 2 { <link rel="stylesheet" href="/css/snow.css" /> }
|
||||||
<link rel="manifest" href="/static/manifest.json" />
|
<link rel="manifest" href="/static/manifest.json" />
|
||||||
|
|
||||||
<link rel="alternate" title="Christine Dodrill's Blog" type="application/rss+xml" href="https://christine.website/blog.rss" />
|
<link rel="alternate" title="Christine Dodrill's Blog" type="application/rss+xml" href="https://christine.website/blog.rss" />
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue