434 lines
13 KiB
Markdown
434 lines
13 KiB
Markdown
|
---
|
||
|
|
title: Dhall for Kubernetes
|
||
|
|
date: 2020-01-25
|
||
|
|
tags:
|
||
|
|
- dhall
|
||
|
|
- kubernetes
|
||
|
|
- witchcraft
|
||
|
|
---
|
||
|
|
|
||
|
|
Kubernetes is a surprisingly complicated software package. Arguably, it has to
|
||
|
|
be that complicated as a result of the problems it solves being complicated; but
|
||
|
|
managing yaml configuration files for Kubernetes is a complicated task. [YAML][yaml]
|
||
|
|
doesn't have support for variables or type metadata. This means that the
|
||
|
|
validity (or sensibility) of a given Kubernetes configuration file (or files)
|
||
|
|
isn't easy to figure out without using a Kubernetes server.
|
||
|
|
|
||
|
|
[yaml]: https://yaml.org
|
||
|
|
|
||
|
|
In my [last post][cultk8s] about Kubernetes, I mentioned I had developed a tool
|
||
|
|
named [dyson][dyson] in order to help me manage Terraform as well as create
|
||
|
|
Kubernetes manifests from [a template][template]. This works for the majority of
|
||
|
|
my apps, but it is difficult to extend at this point for a few reasons:
|
||
|
|
|
||
|
|
[cultk8s]: https://christine.website/blog/the-cult-of-kubernetes-2019-09-07
|
||
|
|
[dyson]: https://github.com/Xe/within-terraform/tree/master/dyson
|
||
|
|
[template]: https://github.com/Xe/within-terraform/blob/master/dyson/src/dysonPkg/deployment_with_ingress.yaml
|
||
|
|
|
||
|
|
- It assumes that everything passed to it are already valid yaml terms
|
||
|
|
- It doesn't assert the type of any values passed to it
|
||
|
|
- It is difficult to add another container to a given deployment
|
||
|
|
- Environment variables implicitly depend on the presence of a private git repo
|
||
|
|
- It depends on the template being correct more than the output being correct
|
||
|
|
|
||
|
|
So, this won't scale. People in the community have created other solutions for
|
||
|
|
this like [Helm][helm], but a lot of them have some of the same basic problems.
|
||
|
|
Helm also assumes that your template is correct. [Kustomize][kustomize] does
|
||
|
|
help with a lot of the type-safe variable replacements, but it doesn't have the
|
||
|
|
ability to ensure your manifest is valid.
|
||
|
|
|
||
|
|
[helm]: https://helm.sh
|
||
|
|
[kustomize]: https://kustomize.io
|
||
|
|
|
||
|
|
I looked around for alternate solutions for a while and eventually found
|
||
|
|
[Dhall][dhall] thanks to a friend. Dhall is a _statically typed_ configuration
|
||
|
|
language. This means that you can ensure that inputs are _always_ the correct
|
||
|
|
type or the configuration file won't load. There's also a built-in
|
||
|
|
[dhall-to-yaml][dhallyaml] tool that can be used with the [Kubernetes
|
||
|
|
package][dhallk8s] in order to declare Kubernetes manifests in a type-safe way.
|
||
|
|
|
||
|
|
[dhall]: https://dhall-lang.org
|
||
|
|
[dhallyaml]: https://github.com/dhall-lang/dhall-haskell/tree/master/dhall-yaml#dhall-yaml
|
||
|
|
[dhallk8s]: https://github.com/dhall-lang/dhall-kubernetes
|
||
|
|
|
||
|
|
Here's a small example of Dhall and the yaml it generates:
|
||
|
|
|
||
|
|
```dhall
|
||
|
|
-- Mastodon usernames
|
||
|
|
[ { name = "Cadey", mastodon = "@cadey@mst3k.interlinked.me" }
|
||
|
|
, { name = "Nicole", mastodon = "@sharkgirl@mst3k.interlinked.me" }
|
||
|
|
]
|
||
|
|
```
|
||
|
|
|
||
|
|
Which produces:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
- mastodon: "@cadey@mst3k.interlinked.me"
|
||
|
|
name: Cadey
|
||
|
|
- mastodon: "@sharkgirl@mst3k.interlinked.me"
|
||
|
|
name: Nicole
|
||
|
|
```
|
||
|
|
|
||
|
|
Which is fine, but we still have the type-safety problem that you would have in
|
||
|
|
normal yaml. Dhall lets us define [record types][dhallrecord] for this data like
|
||
|
|
this:
|
||
|
|
|
||
|
|
[dhallrecord]: http://www.haskellforall.com/2020/01/dhall-year-in-review-2019-2020.html
|
||
|
|
|
||
|
|
```dhall
|
||
|
|
let User =
|
||
|
|
{ Type = { name : Text, mastodon : Optional Text }
|
||
|
|
, default = { name = "", mastodon = None }
|
||
|
|
}
|
||
|
|
|
||
|
|
let users =
|
||
|
|
[ User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me" }
|
||
|
|
, User::{
|
||
|
|
, name = "Nicole"
|
||
|
|
, mastodon = Some "@sharkgirl@mst3k.interlinked.me"
|
||
|
|
}
|
||
|
|
]
|
||
|
|
|
||
|
|
in users
|
||
|
|
```
|
||
|
|
|
||
|
|
Which produces:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
- mastodon: "@cadey@mst3k.interlinked.me"
|
||
|
|
name: Cadey
|
||
|
|
- mastodon: "@sharkgirl@mst3k.interlinked.me"
|
||
|
|
name: Nicole
|
||
|
|
```
|
||
|
|
|
||
|
|
This is type-safe because you cannot add arbitrary fields to User instances
|
||
|
|
without the compiler rejecting it. Let's add an invalid "preferred_language"
|
||
|
|
field to Cadey's instance:
|
||
|
|
|
||
|
|
```
|
||
|
|
-- ...
|
||
|
|
let users =
|
||
|
|
[ User::{
|
||
|
|
, name = "Cadey"
|
||
|
|
, mastodon = Some "@cadey@mst3k.interlinked.me"
|
||
|
|
, preferred_language = "en-US"
|
||
|
|
}
|
||
|
|
-- ...
|
||
|
|
]
|
||
|
|
```
|
||
|
|
|
||
|
|
Which gives us:
|
||
|
|
|
||
|
|
```
|
||
|
|
$ dhall-to-yaml --file example.dhall
|
||
|
|
Error: Expression doesn't match annotation
|
||
|
|
|
||
|
|
{ + preferred_language : …
|
||
|
|
, …
|
||
|
|
}
|
||
|
|
|
||
|
|
4│ User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me",
|
||
|
|
5│ preferred_language = "en-US" }
|
||
|
|
|
||
|
|
example.dhall:4:9
|
||
|
|
```
|
||
|
|
|
||
|
|
Or [this more detailed explanation][explanation] if you add the `--explain` flag
|
||
|
|
to the `dhall-to-yaml` call.
|
||
|
|
|
||
|
|
[explanation]: https://clbin.com/JtVWT
|
||
|
|
|
||
|
|
We tried to do something that violated the contract that the type specified.
|
||
|
|
This means that it's an invalid configuration and is therefore rejected and no
|
||
|
|
yaml file is created.
|
||
|
|
|
||
|
|
The Dhall Kubernetes package specifies record types for _every_ object available
|
||
|
|
by default in Kubernetes. This does mean that the package is incredibly large,
|
||
|
|
but it also makes sure that _everything_ you could possibly want to do in
|
||
|
|
Kubernetes matches what it expects. In the [package
|
||
|
|
documentation][k8sdhalldocs], they give an example where a
|
||
|
|
[Deployment][k8sdeployment] is created.
|
||
|
|
|
||
|
|
[k8sdhalldocs]: https://github.com/dhall-lang/dhall-kubernetes/tree/master/1.15#quickstart---a-simple-deployment
|
||
|
|
[k8sdeployment]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
|
||
|
|
|
||
|
|
``` dhall
|
||
|
|
-- examples/deploymentSimple.dhall
|
||
|
|
|
||
|
|
-- Importing other files is done by specifying the HTTPS URL/disk location of
|
||
|
|
-- the file. Attaching a sha256 hash (obtained with `dhall freeze`) allows
|
||
|
|
-- the Dhall compiler to cache these files and speed up configuration loads
|
||
|
|
-- drastically.
|
||
|
|
let kubernetes =
|
||
|
|
https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/1.15/master/package.dhall
|
||
|
|
sha256:4bd5939adb0a5fc83d76e0d69aa3c5a30bc1a5af8f9df515f44b6fc59a0a4815
|
||
|
|
|
||
|
|
let deployment =
|
||
|
|
kubernetes.Deployment::{
|
||
|
|
, metadata = kubernetes.ObjectMeta::{ name = "nginx" }
|
||
|
|
, spec =
|
||
|
|
Some
|
||
|
|
kubernetes.DeploymentSpec::{
|
||
|
|
, replicas = Some 2
|
||
|
|
, template =
|
||
|
|
kubernetes.PodTemplateSpec::{
|
||
|
|
, metadata = kubernetes.ObjectMeta::{ name = "nginx" }
|
||
|
|
, spec =
|
||
|
|
Some
|
||
|
|
kubernetes.PodSpec::{
|
||
|
|
, containers =
|
||
|
|
[ kubernetes.Container::{
|
||
|
|
, name = "nginx"
|
||
|
|
, image = Some "nginx:1.15.3"
|
||
|
|
, ports =
|
||
|
|
[ kubernetes.ContainerPort::{
|
||
|
|
, containerPort = 80
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|
||
|
|
]
|
||
|
|
}
|
||
|
|
}
|
||
|
|
}
|
||
|
|
}
|
||
|
|
|
||
|
|
in deployment
|
||
|
|
```
|
||
|
|
|
||
|
|
Which creates the following yaml:
|
||
|
|
|
||
|
|
```
|
||
|
|
apiVersion: apps/v1
|
||
|
|
kind: Deployment
|
||
|
|
metadata:
|
||
|
|
name: nginx
|
||
|
|
spec:
|
||
|
|
replicas: 2
|
||
|
|
template:
|
||
|
|
metadata:
|
||
|
|
name: nginx
|
||
|
|
spec:
|
||
|
|
containers:
|
||
|
|
- image: nginx:1.15.3
|
||
|
|
name: nginx
|
||
|
|
ports:
|
||
|
|
- containerPort: 80
|
||
|
|
```
|
||
|
|
|
||
|
|
Dhall's lambda functions can help you break this into manageable chunks. For
|
||
|
|
example, here's a Dhall function that helps create a docker image reference:
|
||
|
|
|
||
|
|
```
|
||
|
|
let formatImage
|
||
|
|
: Text -> Text -> Text
|
||
|
|
= \(repository : Text) -> \(tag : Text) ->
|
||
|
|
"${repository}:${tag}"
|
||
|
|
|
||
|
|
in formatImage "xena/christinewebsite" "latest"
|
||
|
|
```
|
||
|
|
|
||
|
|
Which outputs `xena/christinewebsite:latest` when passed to `dhall text`.
|
||
|
|
|
||
|
|
All of this adds up into a powerful toolset that lets you express Kubernetes
|
||
|
|
configuration in a way that does what you want without as many headaches.
|
||
|
|
|
||
|
|
Most of my apps on Kubernetes need only a few generic bits of configuration:
|
||
|
|
|
||
|
|
- Their name
|
||
|
|
- What port should be exposed
|
||
|
|
- The domain that this service should be exposed on
|
||
|
|
- How many replicas of the service are needed
|
||
|
|
- Which Let's Encrypt Issuer to use (currently only `"prod"` or `"staging"`)
|
||
|
|
- The [configuration variables of the service][12factorconfig]
|
||
|
|
- Any other containers that may be needed for the service
|
||
|
|
|
||
|
|
[12factorconfig]: https://12factor.net/config
|
||
|
|
|
||
|
|
From here, I defined all of the [bits and pieces][kubermemeshttp] for the
|
||
|
|
Kubernetes manifests that Dyson produces and then created a `Config` type that
|
||
|
|
helps to template them out. Here's my [`Config` type
|
||
|
|
definition][configdefinition]:
|
||
|
|
|
||
|
|
[kubermemeshttp]: https://tulpa.dev/cadey/kubermemes/src/branch/master/k8s/http
|
||
|
|
[configdefinition]: https://tulpa.dev/cadey/kubermemes/src/branch/master/k8s/app/config.dhall
|
||
|
|
|
||
|
|
```dhall
|
||
|
|
let kubernetes = ../kubernetes.dhall
|
||
|
|
|
||
|
|
in { Type =
|
||
|
|
{ name : Text
|
||
|
|
, appPort : Natural
|
||
|
|
, image : Text
|
||
|
|
, domain : Text
|
||
|
|
, replicas : Natural
|
||
|
|
, leIssuer : Text
|
||
|
|
, envVars : List kubernetes.EnvVar.Type
|
||
|
|
, otherContainers : List kubernetes.Container.Type
|
||
|
|
}
|
||
|
|
, default =
|
||
|
|
{ name = ""
|
||
|
|
, appPort = 5000
|
||
|
|
, image = ""
|
||
|
|
, domain = ""
|
||
|
|
, replicas = 1
|
||
|
|
, leIssuer = "staging"
|
||
|
|
, envVars = [] : List kubernetes.EnvVar.Type
|
||
|
|
, otherContainers = [] : List kubernetes.Container.Type
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
Then I defined a `makeApp` function that creates everything I need to deploy my
|
||
|
|
stuff on Kubernetes:
|
||
|
|
|
||
|
|
```dhall
|
||
|
|
let Prelude = ../Prelude.dhall
|
||
|
|
|
||
|
|
let kubernetes = ../kubernetes.dhall
|
||
|
|
|
||
|
|
let typesUnion = ../typesUnion.dhall
|
||
|
|
|
||
|
|
let deployment = ../http/deployment.dhall
|
||
|
|
|
||
|
|
let ingress = ../http/ingress.dhall
|
||
|
|
|
||
|
|
let service = ../http/service.dhall
|
||
|
|
|
||
|
|
let Config = ../app/config.dhall
|
||
|
|
|
||
|
|
let K8sList = ../app/list.dhall
|
||
|
|
|
||
|
|
let buildService =
|
||
|
|
\(config : Config.Type)
|
||
|
|
-> let myService = service config
|
||
|
|
|
||
|
|
let myDeployment = deployment config
|
||
|
|
|
||
|
|
let myIngress = ingress config
|
||
|
|
|
||
|
|
in K8sList::{
|
||
|
|
, items =
|
||
|
|
[ typesUnion.Service myService
|
||
|
|
, typesUnion.Deployment myDeployment
|
||
|
|
, typesUnion.Ingress myIngress
|
||
|
|
]
|
||
|
|
}
|
||
|
|
|
||
|
|
in buildService
|
||
|
|
```
|
||
|
|
|
||
|
|
And used it to deploy the [h language website][hlang]:
|
||
|
|
|
||
|
|
[hlang]: https://h.christine.website
|
||
|
|
|
||
|
|
```dhall
|
||
|
|
let makeApp = ../app/make.dhall
|
||
|
|
|
||
|
|
let Config = ../app/config.dhall
|
||
|
|
|
||
|
|
let cfg =
|
||
|
|
Config::{
|
||
|
|
, name = "hlang"
|
||
|
|
, appPort = 5000
|
||
|
|
, image = "xena/hlang:latest"
|
||
|
|
, domain = "h.christine.website"
|
||
|
|
, leIssuer = "prod"
|
||
|
|
}
|
||
|
|
|
||
|
|
in makeApp cfg
|
||
|
|
```
|
||
|
|
|
||
|
|
Which produces the following Kubernetes config:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
apiVersion: v1
|
||
|
|
items:
|
||
|
|
- apiVersion: v1
|
||
|
|
kind: Service
|
||
|
|
metadata:
|
||
|
|
annotations:
|
||
|
|
external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
|
||
|
|
external-dns.alpha.kubernetes.io/hostname: h.christine.website
|
||
|
|
external-dns.alpha.kubernetes.io/ttl: "120"
|
||
|
|
labels:
|
||
|
|
app: hlang
|
||
|
|
name: hlang
|
||
|
|
namespace: apps
|
||
|
|
spec:
|
||
|
|
ports:
|
||
|
|
- port: 5000
|
||
|
|
targetPort: 5000
|
||
|
|
selector:
|
||
|
|
app: hlang
|
||
|
|
type: ClusterIP
|
||
|
|
- apiVersion: apps/v1
|
||
|
|
kind: Deployment
|
||
|
|
metadata:
|
||
|
|
name: hlang
|
||
|
|
namespace: apps
|
||
|
|
spec:
|
||
|
|
replicas: 1
|
||
|
|
selector:
|
||
|
|
matchLabels:
|
||
|
|
app: hlang
|
||
|
|
template:
|
||
|
|
metadata:
|
||
|
|
labels:
|
||
|
|
app: hlang
|
||
|
|
name: hlang
|
||
|
|
spec:
|
||
|
|
containers:
|
||
|
|
- image: xena/hlang:latest
|
||
|
|
imagePullPolicy: Always
|
||
|
|
name: web
|
||
|
|
ports:
|
||
|
|
- containerPort: 5000
|
||
|
|
imagePullSecrets:
|
||
|
|
- name: regcred
|
||
|
|
- apiVersion: networking.k8s.io/v1beta1
|
||
|
|
kind: Ingress
|
||
|
|
metadata:
|
||
|
|
annotations:
|
||
|
|
certmanager.k8s.io/cluster-issuer: letsencrypt-prod
|
||
|
|
kubernetes.io/ingress.class: nginx
|
||
|
|
labels:
|
||
|
|
app: hlang
|
||
|
|
name: hlang
|
||
|
|
namespace: apps
|
||
|
|
spec:
|
||
|
|
rules:
|
||
|
|
- host: h.christine.website
|
||
|
|
http:
|
||
|
|
paths:
|
||
|
|
- backend:
|
||
|
|
serviceName: hlang
|
||
|
|
servicePort: 5000
|
||
|
|
tls:
|
||
|
|
- hosts:
|
||
|
|
- h.christine.website
|
||
|
|
secretName: prod-certs-hlang
|
||
|
|
kind: List
|
||
|
|
```
|
||
|
|
|
||
|
|
And when I applied it on my Kubernetes cluster, it worked the first time and had
|
||
|
|
absolutely no effect on the existing configuration.
|
||
|
|
|
||
|
|
In the future, I hope to expand this to allow for multiple deployments (IE: a
|
||
|
|
chatbot running in a separate deployment than a web API the chatbot depends on
|
||
|
|
or non-web projects in general) as well as supporting multiple Kubernetes
|
||
|
|
namespaces.
|
||
|
|
|
||
|
|
Dhall is probably the most viable replacement to Helm or other Kubernetes
|
||
|
|
templating tools I have found in recent memory. I hope that it will be used by
|
||
|
|
more people to help with configuration management, but I can understand that
|
||
|
|
that may not happen. At least it works for me.
|
||
|
|
|
||
|
|
If you want to learn more about Dhall, I suggest checking out the following
|
||
|
|
links:
|
||
|
|
|
||
|
|
- [The Dhall Language homepage](https://dhall-lang.org)
|
||
|
|
- [Learn Dhall in Y Minutes](https://learnxinyminutes.com/docs/dhall/)
|
||
|
|
- [The Dhall Language GitHub Organization](https://github.com/dhall-lang)
|
||
|
|
|
||
|
|
I hope this was helpful and interesting. Be well.
|