--- title: Dhall for Kubernetes date: 2020-01-25 tags: - dhall - kubernetes - witchcraft --- # Dhall for Kubernetes Kubernetes is a surprisingly complicated software package. Arguably, it has to be that complicated as a result of the problems it solves being complicated; but managing yaml configuration files for Kubernetes is a complicated task. [YAML][yaml] doesn't have support for variables or type metadata. This means that the validity (or sensibility) of a given Kubernetes configuration file (or files) isn't easy to figure out without using a Kubernetes server. [yaml]: https://yaml.org In my [last post][cultk8s] about Kubernetes, I mentioned I had developed a tool named [dyson][dyson] in order to help me manage Terraform as well as create Kubernetes manifests from [a template][template]. This works for the majority of my apps, but it is difficult to extend at this point for a few reasons: [cultk8s]: https://christine.website/blog/the-cult-of-kubernetes-2019-09-07 [dyson]: https://github.com/Xe/within-terraform/tree/master/dyson [template]: https://github.com/Xe/within-terraform/blob/master/dyson/src/dysonPkg/deployment_with_ingress.yaml - It assumes that everything passed to it are already valid yaml terms - It doesn't assert the type of any values passed to it - It is difficult to add another container to a given deployment - Environment variables implicitly depend on the presence of a private git repo - It depends on the template being correct more than the output being correct So, this won't scale. People in the community have created other solutions for this like [Helm][helm], but a lot of them have some of the same basic problems. Helm also assumes that your template is correct. [Kustomize][kustomize] does help with a lot of the type-safe variable replacements, but it doesn't have the ability to ensure your manifest is valid. [helm]: https://helm.sh [kustomize]: https://kustomize.io I looked around for alternate solutions for a while and eventually found [Dhall][dhall] thanks to a friend. Dhall is a _statically typed_ configuration language. This means that you can ensure that inputs are _always_ the correct type or the configuration file won't load. There's also a built-in [dhall-to-yaml][dhallyaml] tool that can be used with the [Kubernetes package][dhallk8s] in order to declare Kubernetes manifests in a type-safe way. [dhall]: https://dhall-lang.org [dhallyaml]: https://github.com/dhall-lang/dhall-haskell/tree/master/dhall-yaml#dhall-yaml [dhallk8s]: https://github.com/dhall-lang/dhall-kubernetes Here's a small example of Dhall and the yaml it generates: ```dhall -- Mastodon usernames [ { name = "Cadey", mastodon = "@cadey@mst3k.interlinked.me" } , { name = "Nicole", mastodon = "@sharkgirl@mst3k.interlinked.me" } ] ``` Which produces: ```yaml - mastodon: "@cadey@mst3k.interlinked.me" name: Cadey - mastodon: "@sharkgirl@mst3k.interlinked.me" name: Nicole ``` Which is fine, but we still have the type-safety problem that you would have in normal yaml. Dhall lets us define [record types][dhallrecord] for this data like this: [dhallrecord]: http://www.haskellforall.com/2020/01/dhall-year-in-review-2019-2020.html ```dhall let User = { Type = { name : Text, mastodon : Optional Text } , default = { name = "", mastodon = None } } let users = [ User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me" } , User::{ , name = "Nicole" , mastodon = Some "@sharkgirl@mst3k.interlinked.me" } ] in users ``` Which produces: ```yaml - mastodon: "@cadey@mst3k.interlinked.me" name: Cadey - mastodon: "@sharkgirl@mst3k.interlinked.me" name: Nicole ``` This is type-safe because you cannot add arbitrary fields to User instances without the compiler rejecting it. Let's add an invalid "preferred_language" field to Cadey's instance: ``` -- ... let users = [ User::{ , name = "Cadey" , mastodon = Some "@cadey@mst3k.interlinked.me" , preferred_language = "en-US" } -- ... ] ``` Which gives us: ``` $ dhall-to-yaml --file example.dhall Error: Expression doesn't match annotation { + preferred_language : … , … } 4│ User::{ name = "Cadey", mastodon = Some "@cadey@mst3k.interlinked.me", 5│ preferred_language = "en-US" } example.dhall:4:9 ``` Or [this more detailed explanation][explanation] if you add the `--explain` flag to the `dhall-to-yaml` call. [explanation]: https://clbin.com/JtVWT We tried to do something that violated the contract that the type specified. This means that it's an invalid configuration and is therefore rejected and no yaml file is created. The Dhall Kubernetes package specifies record types for _every_ object available by default in Kubernetes. This does mean that the package is incredibly large, but it also makes sure that _everything_ you could possibly want to do in Kubernetes matches what it expects. In the [package documentation][k8sdhalldocs], they give an example where a [Deployment][k8sdeployment] is created. [k8sdhalldocs]: https://github.com/dhall-lang/dhall-kubernetes/tree/master/1.15#quickstart---a-simple-deployment [k8sdeployment]: https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ ``` dhall -- examples/deploymentSimple.dhall -- Importing other files is done by specifying the HTTPS URL/disk location of -- the file. Attaching a sha256 hash (obtained with `dhall freeze`) allows -- the Dhall compiler to cache these files and speed up configuration loads -- drastically. let kubernetes = https://raw.githubusercontent.com/dhall-lang/dhall-kubernetes/1.15/master/package.dhall sha256:4bd5939adb0a5fc83d76e0d69aa3c5a30bc1a5af8f9df515f44b6fc59a0a4815 let deployment = kubernetes.Deployment::{ , metadata = kubernetes.ObjectMeta::{ name = "nginx" } , spec = Some kubernetes.DeploymentSpec::{ , replicas = Some 2 , template = kubernetes.PodTemplateSpec::{ , metadata = kubernetes.ObjectMeta::{ name = "nginx" } , spec = Some kubernetes.PodSpec::{ , containers = [ kubernetes.Container::{ , name = "nginx" , image = Some "nginx:1.15.3" , ports = [ kubernetes.ContainerPort::{ , containerPort = 80 } ] } ] } } } } in deployment ``` Which creates the following yaml: ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: replicas: 2 template: metadata: name: nginx spec: containers: - image: nginx:1.15.3 name: nginx ports: - containerPort: 80 ``` Dhall's lambda functions can help you break this into manageable chunks. For example, here's a Dhall function that helps create a docker image reference: ``` let formatImage : Text -> Text -> Text = \(repository : Text) -> \(tag : Text) -> "${repository}:${tag}" in formatImage "xena/christinewebsite" "latest" ``` Which outputs `xena/christinewebsite:latest` when passed to `dhall text`. All of this adds up into a powerful toolset that lets you express Kubernetes configuration in a way that does what you want without as many headaches. Most of my apps on Kubernetes need only a few generic bits of configuration: - Their name - What port should be exposed - The domain that this service should be exposed on - How many replicas of the service are needed - Which Let's Encrypt Issuer to use (currently only `"prod"` or `"staging"`) - The [configuration variables of the service][12factorconfig] - Any other containers that may be needed for the service [12factorconfig]: https://12factor.net/config From here, I defined all of the [bits and pieces][kubermemeshttp] for the Kubernetes manifests that Dyson produces and then created a `Config` type that helps to template them out. Here's my [`Config` type definition][configdefinition]: [kubermemeshttp]: https://tulpa.dev/cadey/kubermemes/src/branch/master/k8s/http [configdefinition]: https://tulpa.dev/cadey/kubermemes/src/branch/master/k8s/app/config.dhall ```dhall let kubernetes = ../kubernetes.dhall in { Type = { name : Text , appPort : Natural , image : Text , domain : Text , replicas : Natural , leIssuer : Text , envVars : List kubernetes.EnvVar.Type , otherContainers : List kubernetes.Container.Type } , default = { name = "" , appPort = 5000 , image = "" , domain = "" , replicas = 1 , leIssuer = "staging" , envVars = [] : List kubernetes.EnvVar.Type , otherContainers = [] : List kubernetes.Container.Type } } ``` Then I defined a `makeApp` function that creates everything I need to deploy my stuff on Kubernetes: ```dhall let Prelude = ../Prelude.dhall let kubernetes = ../kubernetes.dhall let typesUnion = ../typesUnion.dhall let deployment = ../http/deployment.dhall let ingress = ../http/ingress.dhall let service = ../http/service.dhall let Config = ../app/config.dhall let K8sList = ../app/list.dhall let buildService = \(config : Config.Type) -> let myService = service config let myDeployment = deployment config let myIngress = ingress config in K8sList::{ , items = [ typesUnion.Service myService , typesUnion.Deployment myDeployment , typesUnion.Ingress myIngress ] } in buildService ``` And used it to deploy the [h language website][hlang]: [hlang]: https://h.christine.website ```dhall let makeApp = ../app/make.dhall let Config = ../app/config.dhall let cfg = Config::{ , name = "hlang" , appPort = 5000 , image = "xena/hlang:latest" , domain = "h.christine.website" , leIssuer = "prod" } in makeApp cfg ``` Which produces the following Kubernetes config: ```yaml apiVersion: v1 items: - apiVersion: v1 kind: Service metadata: annotations: external-dns.alpha.kubernetes.io/cloudflare-proxied: "false" external-dns.alpha.kubernetes.io/hostname: h.christine.website external-dns.alpha.kubernetes.io/ttl: "120" labels: app: hlang name: hlang namespace: apps spec: ports: - port: 5000 targetPort: 5000 selector: app: hlang type: ClusterIP - apiVersion: apps/v1 kind: Deployment metadata: name: hlang namespace: apps spec: replicas: 1 selector: matchLabels: app: hlang template: metadata: labels: app: hlang name: hlang spec: containers: - image: xena/hlang:latest imagePullPolicy: Always name: web ports: - containerPort: 5000 imagePullSecrets: - name: regcred - apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: annotations: certmanager.k8s.io/cluster-issuer: letsencrypt-prod kubernetes.io/ingress.class: nginx labels: app: hlang name: hlang namespace: apps spec: rules: - host: h.christine.website http: paths: - backend: serviceName: hlang servicePort: 5000 tls: - hosts: - h.christine.website secretName: prod-certs-hlang kind: List ``` And when I applied it on my Kubernetes cluster, it worked the first time and had absolutely no effect on the existing configuration. In the future, I hope to expand this to allow for multiple deployments (IE: a chatbot running in a separate deployment than a web API the chatbot depends on or non-web projects in general) as well as supporting multiple Kubernetes namespaces. Dhall is probably the most viable replacement to Helm or other Kubernetes templating tools I have found in recent memory. I hope that it will be used by more people to help with configuration management, but I can understand that that may not happen. At least it works for me. If you want to learn more about Dhall, I suggest checking out the following links: - [The Dhall Language homepage](https://dhall-lang.org) - [Learn Dhall in Y Minutes](https://learnxinyminutes.com/docs/dhall/) - [The Dhall Language GitHub Organization](https://github.com/dhall-lang) I hope this was helpful and interesting. Be well.