forked from cadey/xesite
359 lines
14 KiB
Markdown
359 lines
14 KiB
Markdown
---
|
|
title: The Cult of Kubernetes
|
|
date: 2019-09-07
|
|
---
|
|
|
|
# The Cult of Kubernetes
|
|
|
|
or: How I got my blog onto it with autodeployment via GitHub Actions.
|
|
|
|
The world was once a simple place. Things used to make sense, or at least there
|
|
weren't so many layers that it became difficult to tell what the hell is going
|
|
on.
|
|
|
|
Then complexity happened. This is a tale of how I literally recreated this meme:
|
|
|
|
<center><blockquote class="twitter-tweet"><p lang="en" dir="ltr">Deployed my blog on Kubernetes <a href="https://t.co/XHXWLrmYO4">pic.twitter.com/XHXWLrmYO4</a></p>— DevOps Thought Liker (@dexhorthy) <a href="https://twitter.com/dexhorthy/status/856639005462417409?ref_src=twsrc%5Etfw">April 24, 2017</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></center>
|
|
|
|
This is how I deployed my blog (the one you are reading right now) to Kubernetes.
|
|
|
|
## The Old State of the World
|
|
|
|
Before I deployed my blog to Kubernetes, I used [Dokku][dokku], as I had been
|
|
for years. Dokku is great. It emulates most of the Heroku "git push; don't care"
|
|
workflow, but on your own server that you can self-manage.
|
|
|
|
This is a blessing and a curse.
|
|
|
|
The real advantage of managed services like Heroku is that you literally just
|
|
_HAND OFF_ operations to Heroku's team. This is not the case with Dokku. Unless
|
|
you pay someone a lot of money, you are going to have to manage the server
|
|
yourself. My dokku server was unmanaged, and I run _many_ apps on it (this
|
|
listing was taken after I started to move apps over):
|
|
|
|
```
|
|
=====> My Apps
|
|
bsnk
|
|
cinemaquestria
|
|
fordaplot-backup
|
|
graphviz.christine.website
|
|
identicond
|
|
ilo-kesi
|
|
johaus
|
|
maison
|
|
olin
|
|
printerfacts
|
|
since
|
|
tulpaforce.tk
|
|
tulpanomicon
|
|
```
|
|
|
|
This is enough apps (plus 5 more that I've already migrated) that it really
|
|
doesn't make sense paying for something like Heroku; nor does it really make
|
|
sense to use the free tier either.
|
|
|
|
So, I decided that it was time for me to properly learn how to Kubernetes, and I
|
|
set off to create a cluster via [DigitalOcean managed Kubernetes][dok8s].
|
|
|
|
## The Cluster
|
|
|
|
I decided it would be a good idea to create my cluster using
|
|
[Terraform][terraform], mostly because I wanted to learn how to use it better.
|
|
I use Terraform at work, so I figured this would also be a way to level up my
|
|
skills in a mostly sane environment.
|
|
|
|
<center><blockquote class="twitter-tweet"><p lang="en" dir="ltr">Terraform is suffering as a service</p>— Cadey Ratio 🌐 (@theprincessxena) <a href="https://twitter.com/theprincessxena/status/1165390942679048192?ref_src=twsrc%5Etfw">August 24, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></center>
|
|
|
|
I have been creating and playing with a small Terraform wrapper tool called
|
|
[dyson][dyson]. This tool is probably overly simplistic and is written in Nim.
|
|
With the config in `~/.config/dyson/dyson.ini`, I can simplify my Terraform
|
|
usage by moving my secrets _out of_ the Terraform code directly. I also avoid
|
|
having my API tokens exposed in my shell to avoid accidental exposure of the
|
|
secrets.
|
|
|
|
Dyson is very simple to use:
|
|
|
|
```console
|
|
$ dyson
|
|
Usage:
|
|
dyson {SUBCMD} [sub-command options & parameters]
|
|
where {SUBCMD} is one of:
|
|
help print comprehensive or per-cmd help
|
|
apply apply Terraform code to production
|
|
destroy destroy resources managed by Terraform
|
|
env dump envvars
|
|
init init Terraform
|
|
manifest generate a somewhat sane manifest for a kubernetes app based on the arguments.
|
|
plan plan a future Terraform run
|
|
slug2docker converts a heroku/dokku slug to a docker image
|
|
|
|
dyson {-h|--help} or with no args at all prints this message.
|
|
dyson --help-syntax gives general cligen syntax help.
|
|
Run "dyson {help SUBCMD|SUBCMD --help}" to see help for just SUBCMD.
|
|
Run "dyson help" to get *comprehensive* help.
|
|
```
|
|
|
|
So I wrote up my config:
|
|
|
|
```
|
|
# main.tf
|
|
provider "digitalocean" {}
|
|
|
|
resource "digitalocean_kubernetes_cluster" "main" {
|
|
name = "kubermemes"
|
|
region = "${var.region}"
|
|
version = "${var.kubernetes_version}"
|
|
|
|
node_pool {
|
|
name = "worker-pool"
|
|
size = "${var.node_size}"
|
|
node_count = 2
|
|
}
|
|
}
|
|
```
|
|
|
|
```
|
|
# variables.tf
|
|
variable "region" {
|
|
type = "string"
|
|
default = "nyc3"
|
|
}
|
|
|
|
variable "kubernetes_version" {
|
|
type = "string"
|
|
default = "1.15.3-do.1"
|
|
}
|
|
|
|
variable "node_size" {
|
|
type = "string"
|
|
default = "s-1vcpu-2gb"
|
|
}
|
|
```
|
|
|
|
and ran it:
|
|
|
|
```console
|
|
$ dyson plan
|
|
<... many lines of plan output ...>
|
|
$ dyson apply
|
|
<... many lines of apply output ...>
|
|
```
|
|
|
|
Then I had a working but mostly unconfigured Kubernetes cluster.
|
|
|
|
## Configuration
|
|
|
|
This is where things started to go downhill. I wanted to do a few things with
|
|
this cluster so I could consider it "ready" for me to use for deploying applications
|
|
to.
|
|
|
|
I wanted to do the following:
|
|
|
|
- setup [helm][helm] to install packages for things like DNS management and HTTP/HTTPS ingress
|
|
- setup [automatic certificate management][certmanager] with [Let's Encrypt][letsencrypt]
|
|
- setup HTTP/HTTPS request ingress with [nginx-ingress][nginxingress]
|
|
- setup [automatic DNS management][autodns] because the external IP addresses of Kubernetes nodes can and will change
|
|
|
|
After a lot of trial, error, pain, suffering and the like, I created
|
|
[this script][setupdotsh] which I am not pasting here. Look at it if you want to
|
|
get a streamlined overview of how to set these things up.
|
|
|
|
Now that all of this is set up, I can deploy an [example app][exanple] with a
|
|
manifest that looks something like [this][ingresstestdotyaml]:
|
|
|
|
```yaml
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: hello-kubernetes-first
|
|
annotations:
|
|
external-dns.alpha.kubernetes.io/hostname: exanple.within.website
|
|
external-dns.alpha.kubernetes.io/ttl: "120" #optional
|
|
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
|
spec:
|
|
type: ClusterIP
|
|
ports:
|
|
- port: 80
|
|
targetPort: 8080
|
|
selector:
|
|
app: hello-kubernetes-first
|
|
|
|
---
|
|
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: hello-kubernetes-first
|
|
spec:
|
|
replicas: 1
|
|
selector:
|
|
matchLabels:
|
|
app: hello-kubernetes-first
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app: hello-kubernetes-first
|
|
spec:
|
|
containers:
|
|
- name: hello-kubernetes
|
|
image: paulbouwer/hello-kubernetes:1.5
|
|
ports:
|
|
- containerPort: 8080
|
|
env:
|
|
- name: MESSAGE
|
|
value: Henlo this are an exanple deployment
|
|
|
|
---
|
|
|
|
apiVersion: extensions/v1beta1
|
|
kind: Ingress
|
|
metadata:
|
|
name: hello-kubernetes-ingress
|
|
annotations:
|
|
kubernetes.io/ingress.class: nginx
|
|
certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
|
|
spec:
|
|
tls:
|
|
- hosts:
|
|
- exanple.within.website
|
|
secretName: prod-certs
|
|
rules:
|
|
- host: exanple.within.website
|
|
http:
|
|
paths:
|
|
- backend:
|
|
serviceName: hello-kubernetes-first
|
|
servicePort: 80
|
|
```
|
|
|
|
<center><blockquote class="twitter-tweet"><p lang="en" dir="ltr">Nope, I was wrong, Kubernetes is the real suffering as a service</p>— Cadey Ratio 🌐 (@theprincessxena) <a href="https://twitter.com/theprincessxena/status/1169997202971930624?ref_src=twsrc%5Etfw">September 6, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></center>
|
|
|
|
It was about this time when I wondered if I was making a mistake moving off of
|
|
Dokku. Dokku really does a lot to abstract almost everything involved with nginx
|
|
away from you, and it _really shows_.
|
|
|
|
However, as a side effect of everything being so declarative and Kubernetes really
|
|
not assuming anything, you have _a lot_ more freedom to do basically anything
|
|
you want. You don't have to have specially magic names for tasks like `web` or
|
|
`worker` like you do in Heroku/Dokku. You just have a deployment that belongs to
|
|
an "app" that just so happens to expose a TCP port that just so happens to have
|
|
a correlating ingress associated with it.
|
|
|
|
Lucky for me, most of the apps I write fit into that general format, and the ones
|
|
that don't can mostly use the same format without the ingress.
|
|
|
|
So I [templated][deploymenttemplateyaml] that sucker as a subcommand in dyson.
|
|
This lets me do commands like [this][exampledysonmanifestcommand]:
|
|
|
|
```console
|
|
$ dyson manifest \
|
|
--name=hlang \
|
|
--domain=h.christine.website \
|
|
--dockerImage=docker.pkg.github.com/xe/x/h:v1.1.8 \
|
|
--containerPort=5000 \
|
|
--replicas=1 \
|
|
--useProdLE=true | kubectl apply -f-
|
|
```
|
|
|
|
And the service gets shunted into the cloud without any extra effort on my part.
|
|
This also automatically sets up Let's Encrypt, DNS and other things that were
|
|
manual in my Dokku setup. This saves me time for when I want to go add services
|
|
in the future. All I have to do is create a docker image somehow, identify what
|
|
port should be exposed, give it a domain name and number of replicas and just
|
|
send it on its merry way.
|
|
|
|
## GitHub Actions
|
|
|
|
This does however mean that deployment is no longer as simple as
|
|
"git push; don't care". This is where [GitHub Actions][actions] come into play.
|
|
They claimed to have the ability to run full end-to-end CI/CD on my applications.
|
|
|
|
I have been using them for a while for [CI on my website][sitegoci] and have
|
|
been pleased with them, so I decided to give it a try and set up continuous
|
|
deployment with them.
|
|
|
|
As [the commit log for the deployment manifest can tell][kubernetescddotyml],
|
|
this took a lot of trial and error. One of the main sources of problems here
|
|
was that GitHub Actions had recently had _a lot_ of changes made to
|
|
configuration and usage as compared to when it was in private beta. This
|
|
included changing the configuration schema from [HCL][hcl] to [YAML][yaml].
|
|
|
|
Of course, all of the documentation (outside of GitHub's
|
|
[quite excellent documentation][githubactionsdocs]) was out of date and wrong.
|
|
I tried following a tutorial by [DigitalOcean themselves][dotutorialkube] on
|
|
how to do this exact thing I wanted to do, but it referenced the old HCL syntax
|
|
for GitHub Actions and did not work. To make things worse, examples
|
|
[in the marketplace READMEs][marketplacereadmeexample] simply DID NOT WORK because
|
|
they were written for the old GitHub Actions syntax.
|
|
|
|
This was frustrating to say the least.
|
|
|
|
After trying to make them work anyways with a combination of the "Use Latest
|
|
Version" button in the marketplace, prayer and gratuitous use of the `with.args`
|
|
field in steps I gave up and decided to manually download the tools I needed
|
|
from their upstream providers and execute them by hand.
|
|
|
|
This is how I ended up with [this monstrosity][monster]:
|
|
|
|
```yaml
|
|
- name: Configure/Deploy/Verify Kubernetes
|
|
run: |
|
|
curl -L https://github.com/digitalocean/doctl/releases/download/v1.30.0/doctl-1.30.0-linux-amd64.tar.gz | tar xz
|
|
./doctl auth init -t $DIGITALOCEAN_ACCESS_TOKEN
|
|
./doctl kubernetes cluster kubeconfig show kubermemes > .kubeconfig
|
|
|
|
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
|
|
chmod +x kubectl
|
|
./kubectl --kubeconfig .kubeconfig apply -n apps -f deploy.yml
|
|
sleep 2
|
|
./kubectl --kubeconfig .kubeconfig rollout -n apps status deployment/christinewebsite
|
|
env:
|
|
DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_TOKEN }}
|
|
```
|
|
|
|
I am almost _certain_ that I am doing it wrong here, I don't know how robust this
|
|
is and I'm very sure that this can and should be done another way; but this is
|
|
the only thing I could get working (for some definition of "working").
|
|
|
|
<center><blockquote class="twitter-tweet"><p lang="hu" dir="ltr">kubernetes is a cult</p>— Andrew Kelley (@andy_kelley) <a href="https://twitter.com/andy_kelley/status/1169999209438859264?ref_src=twsrc%5Etfw">September 6, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></center>
|
|
|
|
---
|
|
|
|
Now when I git push things to the master branch of my blog repo, it will
|
|
automatically get deployed to my Kubernetes cluster.
|
|
|
|
If you work at DigitalOcean and are reading this post. Please get someone to
|
|
update [this tutorial][dotutorialkube] and the README of [this repo][marketplacereadmeexample].
|
|
The examples listed _DO NOT WORK_ for me because I was not in the private beta
|
|
of GitHub Actions. It would also be nice if you had better documentation on how
|
|
to use [your premade action][doctlgithubaction] for usecases like mine. I just
|
|
wanted to download the kubernetes configuration file and run apply against a yaml
|
|
file.
|
|
|
|
Thanks for reading, I hope this was entertaining. Be well.
|
|
|
|
[dokku]: http://dokku.viewdocs.io/dokku/
|
|
[dok8s]: https://www.digitalocean.com/products/kubernetes/
|
|
[terraform]: https://www.terraform.io
|
|
[dyson]: https://github.com/Xe/within-terraform/tree/master/dyson
|
|
[helm]: https://helm.sh
|
|
[certmanager]: https://docs.cert-manager.io/en/latest/
|
|
[letsencrypt]: https://letsencrypt.org
|
|
[nginxingress]: https://kubernetes.github.io/ingress-nginx/
|
|
[autodns]: https://github.com/kubernetes-incubator/external-dns
|
|
[setupdotsh]: https://github.com/Xe/within-terraform/blob/master/do/setup.sh
|
|
[exanple]: https://exanple.within.website
|
|
[ingresstestdotyaml]: https://github.com/Xe/within-terraform/blob/master/do/ingress_test.yaml
|
|
[deploymenttemplateyaml]: https://github.com/Xe/within-terraform/blob/master/dyson/src/dysonPkg/deployment_with_ingress.yaml
|
|
[exampledysonmanifestcommand]: https://github.com/Xe/within-terraform/blob/master/kube_manifests/h.sh
|
|
[actions]: https://github.com/features/actions
|
|
[sitegoci]: https://github.com/Xe/site/blob/master/.github/workflows/go.yml
|
|
[githubactionsdocs]: https://help.github.com/en/articles/about-github-actions
|
|
[kubernetescddotyml]: https://github.com/Xe/site/commits/master/.github/workflows/kubernetes-cd.yml
|
|
[hcl]: https://github.com/hashicorp/hcl
|
|
[yaml]: https://yaml.org
|
|
[marketplacereadmeexample]: https://github.com/marketplace/actions/github-action-for-digitalocean-doctl
|
|
[monster]: https://github.com/Xe/site/blob/master/.github/workflows/kubernetes-cd.yml#L53-L65
|
|
[dotutorialkube]: https://blog.digitalocean.com/how-to-deploy-to-digitalocean-kubernetes-with-github-actions/
|
|
[dogithubaction]: https://github.com/digitalocean/action-doctl
|