forked from cadey/xesite
blog: The Cult of Kubernetes (#72)
* blog: The Cult of Kubernetes * caps
This commit is contained in:
GitHub
parent
a3d0441ff6
commit
b955b5a130
|
|
@ -0,0 +1,358 @@
|
|||
---
|
||||
title: The Cult of Kubernetes
|
||||
date: 2019-09-07
|
||||
---
|
||||
|
||||
# The Cult of Kubernetes
|
||||
|
||||
or: How I got my blog onto it with autodeployment via GitHub Actions.
|
||||
|
||||
The world was once a simple place. Things used to make sense, or at least there
|
||||
weren't so many layers that it became difficult to tell what the hell is going
|
||||
on.
|
||||
|
||||
Then complexity happened. This is a tale of how I literally recreated this meme:
|
||||
|
||||
<center><blockquote class="twitter-tweet"><p lang="en" dir="ltr">Deployed my blog on Kubernetes <a href="https://t.co/XHXWLrmYO4">pic.twitter.com/XHXWLrmYO4</a></p>— DevOps Thought Liker (@dexhorthy) <a href="https://twitter.com/dexhorthy/status/856639005462417409?ref_src=twsrc%5Etfw">April 24, 2017</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></center>
|
||||
|
||||
This is how I deployed my blog (the one you are reading right now) to Kubernetes.
|
||||
|
||||
## The Old State of the World
|
||||
|
||||
Before I deployed my blog to Kubernetes, I used [Dokku][dokku], as I had been
|
||||
for years. Dokku is great. It emulates most of the Heroku "git push; don't care"
|
||||
workflow, but on your own server that you can self-manage.
|
||||
|
||||
This is a blessing and a curse.
|
||||
|
||||
The real advantage of managed services like Heroku is that you literally just
|
||||
_HAND OFF_ operations to Heroku's team. This is not the case with Dokku. Unless
|
||||
you pay someone a lot of money, you are going to have to manage the server
|
||||
yourself. My dokku server was unmanaged, and I run _many_ apps on it (this
|
||||
listing was taken after I started to move apps over):
|
||||
|
||||
```
|
||||
=====> My Apps
|
||||
bsnk
|
||||
cinemaquestria
|
||||
fordaplot-backup
|
||||
graphviz.christine.website
|
||||
identicond
|
||||
ilo-kesi
|
||||
johaus
|
||||
maison
|
||||
olin
|
||||
printerfacts
|
||||
since
|
||||
tulpaforce.tk
|
||||
tulpanomicon
|
||||
```
|
||||
|
||||
This is enough apps (plus 5 more that I've already migrated) that it really
|
||||
doesn't make sense paying for something like Heroku; nor does it really make
|
||||
sense to use the free tier either.
|
||||
|
||||
So, I decided that it was time for me to properly learn how to Kubernetes, and I
|
||||
set off to create a cluster via [DigitalOcean managed Kubernetes][dok8s].
|
||||
|
||||
## The Cluster
|
||||
|
||||
I decided it would be a good idea to create my cluster using
|
||||
[Terraform][terraform], mostly because I wanted to learn how to use it better.
|
||||
I use Terraform at work, so I figured this would also be a way to level up my
|
||||
skills in a mostly sane environment.
|
||||
|
||||
<center><blockquote class="twitter-tweet"><p lang="en" dir="ltr">Terraform is suffering as a service</p>— Cadey Ratio 🌐 (@theprincessxena) <a href="https://twitter.com/theprincessxena/status/1165390942679048192?ref_src=twsrc%5Etfw">August 24, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></center>
|
||||
|
||||
I have been creating and playing with a small Terraform wrapper tool called
|
||||
[dyson][dyson]. This tool is probably overly simplistic and is written in Nim.
|
||||
With the config in `~/.config/dyson/dyson.ini`, I can simplify my Terraform
|
||||
usage by moving my secrets _out of_ the Terraform code directly. I also avoid
|
||||
having my API tokens exposed in my shell to avoid accidental exposure of the
|
||||
secrets.
|
||||
|
||||
Dyson is very simple to use:
|
||||
|
||||
```console
|
||||
$ dyson
|
||||
Usage:
|
||||
dyson {SUBCMD} [sub-command options & parameters]
|
||||
where {SUBCMD} is one of:
|
||||
help print comprehensive or per-cmd help
|
||||
apply apply Terraform code to production
|
||||
destroy destroy resources managed by Terraform
|
||||
env dump envvars
|
||||
init init Terraform
|
||||
manifest generate a somewhat sane manifest for a kubernetes app based on the arguments.
|
||||
plan plan a future Terraform run
|
||||
slug2docker converts a heroku/dokku slug to a docker image
|
||||
|
||||
dyson {-h|--help} or with no args at all prints this message.
|
||||
dyson --help-syntax gives general cligen syntax help.
|
||||
Run "dyson {help SUBCMD|SUBCMD --help}" to see help for just SUBCMD.
|
||||
Run "dyson help" to get *comprehensive* help.
|
||||
```
|
||||
|
||||
So I wrote up my config:
|
||||
|
||||
```
|
||||
# main.tf
|
||||
provider "digitalocean" {}
|
||||
|
||||
resource "digitalocean_kubernetes_cluster" "main" {
|
||||
name = "kubermemes"
|
||||
region = "${var.region}"
|
||||
version = "${var.kubernetes_version}"
|
||||
|
||||
node_pool {
|
||||
name = "worker-pool"
|
||||
size = "${var.node_size}"
|
||||
node_count = 2
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
# variables.tf
|
||||
variable "region" {
|
||||
type = "string"
|
||||
default = "nyc3"
|
||||
}
|
||||
|
||||
variable "kubernetes_version" {
|
||||
type = "string"
|
||||
default = "1.15.3-do.1"
|
||||
}
|
||||
|
||||
variable "node_size" {
|
||||
type = "string"
|
||||
default = "s-1vcpu-2gb"
|
||||
}
|
||||
```
|
||||
|
||||
and ran it:
|
||||
|
||||
```console
|
||||
$ dyson plan
|
||||
<... many lines of plan output ...>
|
||||
$ dyson apply
|
||||
<... many lines of apply output ...>
|
||||
```
|
||||
|
||||
Then I had a working but mostly unconfigured Kubernetes cluster.
|
||||
|
||||
## Configuration
|
||||
|
||||
This is where things started to go downhill. I wanted to do a few things with
|
||||
this cluster so I could consider it "ready" for me to use for deploying applications
|
||||
to.
|
||||
|
||||
I wanted to do the following:
|
||||
|
||||
- setup [helm][helm] to install packages for things like DNS management and HTTP/HTTPS ingress
|
||||
- setup [automatic certificate management][certmanager] with [Let's Encrypt][letsencrypt]
|
||||
- setup HTTP/HTTPS request ingress with [nginx-ingress][nginxingress]
|
||||
- setup [automatic DNS management][autodns] because the external IP addresses of Kubernetes nodes can and will change
|
||||
|
||||
After a lot of trial, error, pain, suffering and the like, I created
|
||||
[this script][setupdotsh] which I am not pasting here. Look at it if you want to
|
||||
get a streamlined overview of how to set these things up.
|
||||
|
||||
Now that all of this is set up, I can deploy an [example app][exanple] with a
|
||||
manifest that looks something like [this][ingresstestdotyaml]:
|
||||
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: hello-kubernetes-first
|
||||
annotations:
|
||||
external-dns.alpha.kubernetes.io/hostname: exanple.within.website
|
||||
external-dns.alpha.kubernetes.io/ttl: "120" #optional
|
||||
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||||
spec:
|
||||
type: ClusterIP
|
||||
ports:
|
||||
- port: 80
|
||||
targetPort: 8080
|
||||
selector:
|
||||
app: hello-kubernetes-first
|
||||
|
||||
---
|
||||
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: hello-kubernetes-first
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: hello-kubernetes-first
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: hello-kubernetes-first
|
||||
spec:
|
||||
containers:
|
||||
- name: hello-kubernetes
|
||||
image: paulbouwer/hello-kubernetes:1.5
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
env:
|
||||
- name: MESSAGE
|
||||
value: Henlo this are an exanple deployment
|
||||
|
||||
---
|
||||
|
||||
apiVersion: extensions/v1beta1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: hello-kubernetes-ingress
|
||||
annotations:
|
||||
kubernetes.io/ingress.class: nginx
|
||||
certmanager.k8s.io/cluster-issuer: "letsencrypt-prod"
|
||||
spec:
|
||||
tls:
|
||||
- hosts:
|
||||
- exanple.within.website
|
||||
secretName: prod-certs
|
||||
rules:
|
||||
- host: exanple.within.website
|
||||
http:
|
||||
paths:
|
||||
- backend:
|
||||
serviceName: hello-kubernetes-first
|
||||
servicePort: 80
|
||||
```
|
||||
|
||||
<center><blockquote class="twitter-tweet"><p lang="en" dir="ltr">Nope, I was wrong, Kubernetes is the real suffering as a service</p>— Cadey Ratio 🌐 (@theprincessxena) <a href="https://twitter.com/theprincessxena/status/1169997202971930624?ref_src=twsrc%5Etfw">September 6, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></center>
|
||||
|
||||
It was about this time when I wondered if I was making a mistake moving off of
|
||||
Dokku. Dokku really does a lot to abstract almost everything involved with nginx
|
||||
away from you, and it _really shows_.
|
||||
|
||||
However, as a side effect of everything being so declarative and Kubernetes really
|
||||
not assuming anything, you have _a lot_ more freedom to do basically anything
|
||||
you want. You don't have to have specially magic names for tasks like `web` or
|
||||
`worker` like you do in Heroku/Dokku. You just have a deployment that belongs to
|
||||
an "app" that just so happens to expose a TCP port that just so happens to have
|
||||
a correlating ingress associated with it.
|
||||
|
||||
Lucky for me, most of the apps I write fit into that general format, and the ones
|
||||
that don't can mostly use the same format without the ingress.
|
||||
|
||||
So I [templated][deploymenttemplateyaml] that sucker as a subcommand in dyson.
|
||||
This lets me do commands like [this][exampledysonmanifestcommand]:
|
||||
|
||||
```console
|
||||
$ dyson manifest \
|
||||
--name=hlang \
|
||||
--domain=h.christine.website \
|
||||
--dockerImage=docker.pkg.github.com/xe/x/h:v1.1.8 \
|
||||
--containerPort=5000 \
|
||||
--replicas=1 \
|
||||
--useProdLE=true | kubectl apply -f-
|
||||
```
|
||||
|
||||
And the service gets shunted into the cloud without any extra effort on my part.
|
||||
This also automatically sets up Let's Encrypt, DNS and other things that were
|
||||
manual in my Dokku setup. This saves me time for when I want to go add services
|
||||
in the future. All I have to do is create a docker image somehow, identify what
|
||||
port should be exposed, give it a domain name and number of replicas and just
|
||||
send it on its merry way.
|
||||
|
||||
## GitHub Actions
|
||||
|
||||
This does however mean that deployment is no longer as simple as
|
||||
"git push; don't care". This is where [GitHub Actions][actions] come into play.
|
||||
They claimed to have the ability to run full end-to-end CI/CD on my applications.
|
||||
|
||||
I have been using them for a while for [CI on my website][sitegoci] and have
|
||||
been pleased with them, so I decided to give it a try and set up continuous
|
||||
deployment with them.
|
||||
|
||||
As [the commit log for the deployment manifest can tell][kubernetescddotyml],
|
||||
this took a lot of trial and error. One of the main sources of problems here
|
||||
was that GitHub Actions had recently had _a lot_ of changes made to
|
||||
configuration and usage as compared to when it was in private beta. This
|
||||
included changing the configuration schema from [HCL][hcl] to [YAML][yaml].
|
||||
|
||||
Of course, all of the documentation (outside of GitHub's
|
||||
[quite excellent documentation][githubactionsdocs]) was out of date and wrong.
|
||||
I tried following a tutorial by [DigitalOcean themselves][dotutorialkube] on
|
||||
how to do this exact thing I wanted to do, but it referenced the old HCL syntax
|
||||
for GitHub Actions and did not work. To make things worse, examples
|
||||
[in the marketplace READMEs][marketplacereadmeexample] simply DID NOT WORK because
|
||||
they were written for the old GitHub Actions syntax.
|
||||
|
||||
This was frustrating to say the least.
|
||||
|
||||
After trying to make them work anyways with a combination of the "Use Latest
|
||||
Version" button in the marketplace, prayer and gratuitous use of the `with.args`
|
||||
field in steps I gave up and decided to manually download the tools I needed
|
||||
from their upstream providers and execute them by hand.
|
||||
|
||||
This is how I ended up with [this monstrosity][monster]:
|
||||
|
||||
```yaml
|
||||
- name: Configure/Deploy/Verify Kubernetes
|
||||
run: |
|
||||
curl -L https://github.com/digitalocean/doctl/releases/download/v1.30.0/doctl-1.30.0-linux-amd64.tar.gz | tar xz
|
||||
./doctl auth init -t $DIGITALOCEAN_ACCESS_TOKEN
|
||||
./doctl kubernetes cluster kubeconfig show kubermemes > .kubeconfig
|
||||
|
||||
curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl
|
||||
chmod +x kubectl
|
||||
./kubectl --kubeconfig .kubeconfig apply -n apps -f deploy.yml
|
||||
sleep 2
|
||||
./kubectl --kubeconfig .kubeconfig rollout -n apps status deployment/christinewebsite
|
||||
env:
|
||||
DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_TOKEN }}
|
||||
```
|
||||
|
||||
I am almost _certain_ that I am doing it wrong here, I don't know how robust this
|
||||
is and I'm very sure that this can and should be done another way; but this is
|
||||
the only thing I could get working (for some definition of "working").
|
||||
|
||||
---
|
||||
|
||||
Now when I git push things to the master branch of my blog repo, it will
|
||||
automatically get deployed to my Kubernetes cluster.
|
||||
|
||||
If you work at DigitalOcean and are reading this post. Please get someone to
|
||||
update [this tutorial][dotutorialkube] and the README of [this repo][marketplacereadmeexample].
|
||||
The examples listed _DO NOT WORK_ for me because I was not in the private beta
|
||||
of GitHub Actions. It would also be nice if you had better documentation on how
|
||||
to use [your premade action][doctlgithubaction] for usecases like mine. I just
|
||||
wanted to download the kubernetes configuration file and run apply against a yaml
|
||||
file.
|
||||
|
||||
Thanks for reading, I hope this was entertaining. Be well.
|
||||
|
||||
<center><blockquote class="twitter-tweet"><p lang="hu" dir="ltr">kubernetes is a cult</p>— Andrew Kelley (@andy_kelley) <a href="https://twitter.com/andy_kelley/status/1169999209438859264?ref_src=twsrc%5Etfw">September 6, 2019</a></blockquote> <script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script></center>
|
||||
|
||||
[dokku]: http://dokku.viewdocs.io/dokku/
|
||||
[dok8s]: https://www.digitalocean.com/products/kubernetes/
|
||||
[terraform]: https://www.terraform.io
|
||||
[dyson]: https://github.com/Xe/within-terraform/tree/master/dyson
|
||||
[helm]: https://helm.sh
|
||||
[certmanager]: https://docs.cert-manager.io/en/latest/
|
||||
[letsencrypt]: https://letsencrypt.org
|
||||
[nginxingress]: https://kubernetes.github.io/ingress-nginx/
|
||||
[autodns]: https://github.com/kubernetes-incubator/external-dns
|
||||
[setupdotsh]: https://github.com/Xe/within-terraform/blob/master/do/setup.sh
|
||||
[exanple]: https://exanple.within.website
|
||||
[ingresstestdotyaml]: https://github.com/Xe/within-terraform/blob/master/do/ingress_test.yaml
|
||||
[deploymenttemplateyaml]: https://github.com/Xe/within-terraform/blob/master/dyson/src/dysonPkg/deployment_with_ingress.yaml
|
||||
[exampledysonmanifestcommand]: https://github.com/Xe/within-terraform/blob/master/kube_manifests/h.sh
|
||||
[actions]: https://github.com/features/actions
|
||||
[sitegoci]: https://github.com/Xe/site/blob/master/.github/workflows/go.yml
|
||||
[githubactionsdocs]: https://help.github.com/en/articles/about-github-actions
|
||||
[kubernetescddotyml]: https://github.com/Xe/site/commits/master/.github/workflows/kubernetes-cd.yml
|
||||
[hcl]: https://github.com/hashicorp/hcl
|
||||
[yaml]: https://yaml.org
|
||||
[marketplacereadmeexample]: https://github.com/marketplace/actions/github-action-for-digitalocean-doctl
|
||||
[monster]: https://github.com/Xe/site/blob/master/.github/workflows/kubernetes-cd.yml#L53-L65
|
||||
[dotutorialkube]: https://blog.digitalocean.com/how-to-deploy-to-digitalocean-kubernetes-with-github-actions/
|
||||
[dogithubaction]: https://github.com/digitalocean/action-doctl
|
||||
Loading…
Reference in New Issue