diff --git a/blog/the-cult-of-kubernetes-2019-09-07.markdown b/blog/the-cult-of-kubernetes-2019-09-07.markdown new file mode 100644 index 0000000..dea40e2 --- /dev/null +++ b/blog/the-cult-of-kubernetes-2019-09-07.markdown @@ -0,0 +1,358 @@ +--- +title: The Cult of Kubernetes +date: 2019-09-07 +--- + +# The Cult of Kubernetes + +or: How I got my blog onto it with autodeployment via GitHub Actions. + +The world was once a simple place. Things used to make sense, or at least there +weren't so many layers that it became difficult to tell what the hell is going +on. + +Then complexity happened. This is a tale of how I literally recreated this meme: + +

Deployed my blog on Kubernetes pic.twitter.com/XHXWLrmYO4

— DevOps Thought Liker (@dexhorthy) April 24, 2017
+ +This is how I deployed my blog (the one you are reading right now) to Kubernetes. + +## The Old State of the World + +Before I deployed my blog to Kubernetes, I used [dokku][dokku], as I had been +for years. Dokku is great. It emulates most of the Heroku "git push; don't care" +workflow, but on your own server that you can self-manage. + +This is a blessing and a curse. + +The real advantage of managed services like Heroku is that you literally just +_HAND OFF_ operations to Heroku's team. This is not the case with Dokku. Unless +you pay someone a lot of money, you are going to have to manage the server +yourself. My dokku server was unmanaged, and I run _many_ apps on it (this +listing was taken after I started to move apps over): + +``` +=====> My Apps +bsnk +cinemaquestria +fordaplot-backup +graphviz.christine.website +identicond +ilo-kesi +johaus +maison +olin +printerfacts +since +tulpaforce.tk +tulpanomicon +``` + +This is enough apps (plus 5 more that I've already migrated) that it really +doesn't make sense paying for something like Heroku; nor does it really make +sense to use the free tier either. + +So, I decided that it was time for me to properly learn how to Kubernetes, and I +set off to create a cluster via [DigitalOcean managed Kubernetes][dok8s]. + +## The Cluster + +I decided it would be a good idea to create my cluster using +[Terraform][terraform], mostly because I wanted to learn how to use it better. +I use Terraform at work, so I figured this would also be a way to level up my +skills in a mostly sane environment. + +

Terraform is suffering as a service

— Cadey Ratio 🌐 (@theprincessxena) August 24, 2019
+ +I have been creating and playing with a small Terraform wrapper tool called +[dyson][dyson]. This tool is probably overly simplistic and is written in Nim. +With the config in `~/.config/dyson/dyson.ini`, I can simplify my Terraform +usage by moving my secrets _out of_ the Terraform code directly. I also avoid +having my API tokens exposed in my shell to avoid accidental exposure of the +secrets. + +Dyson is very simple to use: + +```console +$ dyson +Usage: + dyson {SUBCMD} [sub-command options & parameters] +where {SUBCMD} is one of: + help print comprehensive or per-cmd help + apply apply Terraform code to production + destroy destroy resources managed by Terraform + env dump envvars + init init Terraform + manifest generate a somewhat sane manifest for a kubernetes app based on the arguments. + plan plan a future Terraform run + slug2docker converts a heroku/dokku slug to a docker image + +dyson {-h|--help} or with no args at all prints this message. +dyson --help-syntax gives general cligen syntax help. +Run "dyson {help SUBCMD|SUBCMD --help}" to see help for just SUBCMD. +Run "dyson help" to get *comprehensive* help. +``` + +So I wrote up my config: + +``` +# main.tf +provider "digitalocean" {} + +resource "digitalocean_kubernetes_cluster" "main" { + name = "kubermemes" + region = "${var.region}" + version = "${var.kubernetes_version}" + + node_pool { + name = "worker-pool" + size = "${var.node_size}" + node_count = 2 + } +} +``` + +``` +# variables.tf +variable "region" { + type = "string" + default = "nyc3" +} + +variable "kubernetes_version" { + type = "string" + default = "1.15.3-do.1" +} + +variable "node_size" { + type = "string" + default = "s-1vcpu-2gb" +} +``` + +and ran it: + +```console +$ dyson plan +<... many lines of plan output ...> +$ dyson apply +<... many lines of apply output ...> +``` + +Then I had a working but mostly unconfigured Kubernetes cluster. + +## Configuration + +This is where things started to go downhill. I wanted to do a few things with +this cluster so I could consider it "ready" for me to use for deploying applications +to. + +I wanted to do the following: + +- setup [helm][helm] to install packages for things like DNS management and HTTP/HTTPS ingress +- setup [automatic certificate management][certmanager] with [Let's Encrypt][letsencrypt] +- setup HTTP/HTTPS request ingress with [nginx-ingress][nginxingress] +- setup [automatic DNS management][autodns] because the external IP addresses of Kubernetes nodes can and will change + +After a lot of trial, error, pain, suffering and the like, I created +[this script][setupdotsh] which I am not pasting here. Look at it if you want to +get a streamlined overview of how to set these things up. + +Now that all of this is set up, I can deploy an [example app][exanple] with a +manifest that looks something like [this][ingresstestdotyaml]: + +```yaml +apiVersion: v1 +kind: Service +metadata: + name: hello-kubernetes-first + annotations: + external-dns.alpha.kubernetes.io/hostname: exanple.within.website + external-dns.alpha.kubernetes.io/ttl: "120" #optional + external-dns.alpha.kubernetes.io/cloudflare-proxied: "false" +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 8080 + selector: + app: hello-kubernetes-first + +--- + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: hello-kubernetes-first +spec: + replicas: 1 + selector: + matchLabels: + app: hello-kubernetes-first + template: + metadata: + labels: + app: hello-kubernetes-first + spec: + containers: + - name: hello-kubernetes + image: paulbouwer/hello-kubernetes:1.5 + ports: + - containerPort: 8080 + env: + - name: MESSAGE + value: Henlo this are an exanple deployment + +--- + +apiVersion: extensions/v1beta1 +kind: Ingress +metadata: + name: hello-kubernetes-ingress + annotations: + kubernetes.io/ingress.class: nginx + certmanager.k8s.io/cluster-issuer: "letsencrypt-prod" +spec: + tls: + - hosts: + - exanple.within.website + secretName: prod-certs + rules: + - host: exanple.within.website + http: + paths: + - backend: + serviceName: hello-kubernetes-first + servicePort: 80 +``` + +

Nope, I was wrong, Kubernetes is the real suffering as a service

— Cadey Ratio 🌐 (@theprincessxena) September 6, 2019
+ +It was about this time when I wondered if I was making a mistake moving off of +Dokku. Dokku really does a lot to abstract almost everything involved with nginx +away from you, and it _really shows_. + +However, as a side effect of everything being so declarative and Kubernetes really +not assuming anything, you have _a lot_ more freedom to do basically anything +you want. You don't have to have specially magic names for tasks like `web` or +`worker` like you do in Heroku/Dokku. You just have a deployment that belongs to +an "app" that just so happens to expose a TCP port that just so happens to have +a correlating ingress associated with it. + +Lucky for me, most of the apps I write fit into that general format, and the ones +that don't can mostly use the same format without the ingress. + +So I [templated][deploymenttemplateyaml] that sucker as a subcommand in dyson. +This lets me do commands like [this][exampledysonmanifestcommand]: + +```console +$ dyson manifest \ + --name=hlang \ + --domain=h.christine.website \ + --dockerImage=docker.pkg.github.com/xe/x/h:v1.1.8 \ + --containerPort=5000 \ + --replicas=1 \ + --useProdLE=true | kubectl apply -f- +``` + +And the service gets shunted into the cloud without any extra effort on my part. +This also automatically sets up Let's Encrypt, DNS and other things that were +manual in my Dokku setup. This saves me time for when I want to go add services +in the future. All I have to do is create a docker image somehow, identify what +port should be exposed, give it a domain name and number of replicas and just +send it on its merry way. + +## GitHub Actions + +This does however mean that deployment is no longer as simple as +"git push; don't care". This is where [GitHub Actions][actions] come into play. +They claimed to have the ability to run full end-to-end CI/CD on my applications. + +I have been using them for a while for [CI on my website][sitegoci] and have +been pleased with them, so I decided to give it a try and set up continuous +deployment with them. + +As [the commit log for the deployment manifest can tell][kubernetescddotyml], +this took a lot of trial and error. One of the main sources of problems here +was that GitHub Actions had recently had _a lot_ of changes made to +configuration and usage as compared to when it was in private beta. This +included changing the configuration schema from [HCL][hcl] to [YAML][yaml]. + +Of course, all of the documentation (outside of GitHub's +[quite excellent documentation][githubactionsdocs]) was out of date and wrong. +I tried following a tutorial by [DigitalOcean themselves][dotutorialkube] on +how to do this exact thing I wanted to do, but it referenced the old HCL syntax +for GitHub Actions and did not work. To make things worse, examples +[in the marketplace READMEs][marketplacereadmeexample] simply DID NOT WORK because +they were written for the old GitHub Actions syntax. + +This was frustrating to say the least. + +After trying to make them work anyways with a combination of the "Use Latest +Version" button in the marketplace, prayer and gratuitous use of the `with.args` +field in steps I gave up and decided to manually download the tools I needed +from their upstream providers and execute them by hand. + +This is how I ended up with [this monstrosity][monster]: + +```yaml +- name: Configure/Deploy/Verify Kubernetes + run: | + curl -L https://github.com/digitalocean/doctl/releases/download/v1.30.0/doctl-1.30.0-linux-amd64.tar.gz | tar xz + ./doctl auth init -t $DIGITALOCEAN_ACCESS_TOKEN + ./doctl kubernetes cluster kubeconfig show kubermemes > .kubeconfig + + curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl + chmod +x kubectl + ./kubectl --kubeconfig .kubeconfig apply -n apps -f deploy.yml + sleep 2 + ./kubectl --kubeconfig .kubeconfig rollout -n apps status deployment/christinewebsite + env: + DIGITALOCEAN_ACCESS_TOKEN: ${{ secrets.DIGITALOCEAN_TOKEN }} +``` + +I am almost _certain_ that I am doing it wrong here, I don't know how robust this +is and I'm very sure that this can and should be done another way; but this is +the only thing I could get working (for some definition of "working"). + +--- + +Now when I git push things to the master branch of my blog repo, it will +automatically get deployed to my Kubernetes cluster. + +If you work at DigitalOcean and are reading this post. Please get someone to +update [this tutorial][dotutorialkube] and the README of [this repo][marketplacereadmeexample]. +The examples listed _DO NOT WORK_ for me because I was not in the private beta +of GitHub Actions. It would also be nice if you had better documentation on how +to use [your premade action][doctlgithubaction] for usecases like mine. I just +wanted to download the kubernetes configuration file and run apply against a yaml +file. + +Thanks for reading, I hope this was entertaining. Be well. + +

kubernetes is a cult

— Andrew Kelley (@andy_kelley) September 6, 2019
+ +[dokku]: http://dokku.viewdocs.io/dokku/ +[dok8s]: https://www.digitalocean.com/products/kubernetes/ +[terraform]: https://www.terraform.io +[dyson]: https://github.com/Xe/within-terraform/tree/master/dyson +[helm]: https://helm.sh +[certmanager]: https://docs.cert-manager.io/en/latest/ +[letsencrypt]: https://letsencrypt.org +[nginxingress]: https://kubernetes.github.io/ingress-nginx/ +[autodns]: https://github.com/kubernetes-incubator/external-dns +[setupdotsh]: https://github.com/Xe/within-terraform/blob/master/do/setup.sh +[exanple]: https://exanple.within.website +[ingresstestdotyaml]: https://github.com/Xe/within-terraform/blob/master/do/ingress_test.yaml +[deploymenttemplateyaml]: https://github.com/Xe/within-terraform/blob/master/dyson/src/dysonPkg/deployment_with_ingress.yaml +[exampledysonmanifestcommand]: https://github.com/Xe/within-terraform/blob/master/kube_manifests/h.sh +[actions]: https://github.com/features/actions +[sitegoci]: https://github.com/Xe/site/blob/master/.github/workflows/go.yml +[githubactionsdocs]: https://help.github.com/en/articles/about-github-actions +[kubernetescddotyml]: https://github.com/Xe/site/commits/master/.github/workflows/kubernetes-cd.yml +[hcl]: https://github.com/hashicorp/hcl +[yaml]: https://yaml.org +[marketplacereadmeexample]: https://github.com/marketplace/actions/github-action-for-digitalocean-doctl +[monster]: https://github.com/Xe/site/blob/master/.github/workflows/kubernetes-cd.yml#L53-L65 +[dotutorialkube]: https://blog.digitalocean.com/how-to-deploy-to-digitalocean-kubernetes-with-github-actions/ +[dogithubaction]: https://github.com/digitalocean/action-doctl