waifud progress report number 1
Signed-off-by: Xe Iaso <me@christine.website>
This commit is contained in:
parent
b3d341d954
commit
0db69641a5
|
|
@ -0,0 +1,272 @@
|
|||
---
|
||||
title: "waifud Progress Report #1"
|
||||
date: 2022-02-06
|
||||
tags:
|
||||
- waifud
|
||||
- zfs
|
||||
- libvirt
|
||||
---
|
||||
|
||||
In [June of last year](/blog/waifud-plans-2021-06-19) I wrote out my plans for
|
||||
making a tool to manage virtual machines on my homelab called
|
||||
[waifud](https://github.com/Xe/waifud). I have gotten almost nothing done on it
|
||||
for a while due to Facts and Circumstances getting in the way. However, I have
|
||||
broken through the dread and gotten to a point where waifud is an actual daemon
|
||||
instead of a prototype of a tool written in Go.
|
||||
|
||||
Currently it allows you to make an instance (virtual machine), list the
|
||||
instances that waifud manages, delete instances, get information from libvirt
|
||||
for every machine that libvirt manages and manage distribution base image
|
||||
snapshots.
|
||||
|
||||
Here is the overall architecture:
|
||||
|
||||
- waifud runs somewhere and has the ability to SSH into machines with limited
|
||||
sudo powers.
|
||||
- waifuctl runs on your development machine and can hit the waifud API to do
|
||||
things.
|
||||
- Runner machines use [libvirtd](https://libvirt.org/index.html) and zfs, they
|
||||
are given instructions over SSH.
|
||||
- waifud stores all state in SQLite for easy backup and replication.
|
||||
- waifud is configured using a [Dhall](https://dhall-lang.org/) file named
|
||||
`config.dhall` that it expects to find in its current working directory.
|
||||
|
||||
I have managed to totally refactor out the requirement for waifud-metadatad at
|
||||
this time. I am glad this is the case, doing this will allow me to implement
|
||||
everything in Rust and reduce complexity in developing this tool.
|
||||
|
||||
Here are the downsides of the currently implemented feature set:
|
||||
|
||||
- Linux distributions love breaking links to their images and you have to
|
||||
constantly update your cached image versions against upstream or you will
|
||||
randomly get failures.
|
||||
- It only currently runs on my machines and will need you to buy into my
|
||||
incredibly opinionated views on filesystem choice (zfs) and virtualization
|
||||
engine choice (kvm on libvirtd).
|
||||
- waifud tells other machines what to do and will blow up instantly if things go
|
||||
sour in ways that require manual editing of database state and god knows what
|
||||
else depending on where it failed in the chain (hope you're good to use the
|
||||
`zfs` command and know your way around `virsh` enough to reset things back to
|
||||
how they should be). You may also need to edit the XML that the templates
|
||||
generate in the worst case, so being familiar with that can't hurt.
|
||||
|
||||
In a production environment, it would be reasonable to describe these downsides
|
||||
as "catastrophic". However for something that's only really started to exist in
|
||||
its current form less than 48 hours ago, I'd prefer to describe that as "having
|
||||
promise".
|
||||
|
||||
I still plan to make waifud under a dual pay-me-for-commercial-support/Be Gay Do
|
||||
Crimes license. The latter is not accepted by the OSI, so I am fairly sure that
|
||||
I will be able to avoid getting this software packaged by any distributions
|
||||
until it is more stable. Then I can relicense it as reality demands.
|
||||
|
||||
If you want to test this and have NixOS machines that run zfs and libvirtd, here
|
||||
is how you can set it up:
|
||||
|
||||
- On each target machine, run this command to make the image cache folder:
|
||||
`mkdir -p $HOME/.cache/within/mkvm/qcow2`.
|
||||
- Set up things so that you have passwordless sudo on the remote host you want
|
||||
to run VMs on.
|
||||
- Set up each VM host to have its VM subnet unique within your network.
|
||||
Advertise a subnet route to that subnet over Tailscale or something.
|
||||
- Set up each VM host to advertise a subnet
|
||||
- Edit `config.dhall` in the root of the waifud repo to contain the host/hosts
|
||||
you want to puppet with waifud.
|
||||
- Edit `config.dhall` to point the base URL to the IP address of the server
|
||||
running waifud. This is used with cloud-init to load your user-data into
|
||||
virtual machines. You probably want to make sure this works with plain HTTP, I
|
||||
don't know if all the VM images that I ship by default come with CA
|
||||
certificates.
|
||||
- Run `cargo run --bin waifud` in one terminal window.
|
||||
- Edit `cargo run --bin waifuctl -- --host http://[::]:23818 create --distro
|
||||
ubuntu-20.04 --host pneuma --zvol fast/vms --disk-size 20 --user-data
|
||||
/path/to/config.yaml` to contain one of your hostnames, the right zfs parent
|
||||
dataset and your cloudconfig user data, then hope it works.
|
||||
- SSH into the VM and laugh at the haters for doubting you.
|
||||
|
||||
[The default set of hosts are the MagicDNS names of my homelab machines. You
|
||||
will definitely want to replace them with your server/s unless you also have an
|
||||
affinity for Xenoblade-inspired server names.](conversation://Cadey/enby)
|
||||
|
||||
The core of this is built on [Axum](https://github.com/tokio-rs/axum), a Rust
|
||||
web application framework that I want to base more things on in the future.
|
||||
Previously I have liked Rocket and Warp, however Axum seems more likely to stick
|
||||
around for a very long time and is actively maintained.
|
||||
|
||||
In its current state, here's what it looks like to create a VM running Arch
|
||||
Linux.
|
||||
|
||||
[Arch Linux is notoriously annoying to install, so let's see how hard it is with waifud.](conversation://Mara/hmm)
|
||||
|
||||
```console
|
||||
$ waifuctl --host http://[::]:23818 \
|
||||
create --distro arch --host pneuma \
|
||||
--zvol fast/vms --disk-size 20 \
|
||||
--memory 1024 --cpus 4 \
|
||||
--user-data ./var/xe-base.yaml
|
||||
created instance sunspot on pneuma, waiting for IP address
|
||||
IP address: 10.77.131.97
|
||||
```
|
||||
|
||||
And then you can SSH into that instance as normal:
|
||||
|
||||
```console
|
||||
$ ssh xe@10.77.131.97
|
||||
Warning: Permanently added '10.77.131.97' (ED25519) to the list of known hosts.
|
||||
[xe@sunspot ~]$ uname -av
|
||||
Linux sunspot 5.16.5-arch1-1 #1 SMP PREEMPT Tue, 01 Feb 2022 21:42:50 +0000 x86_64 GNU/Linux
|
||||
|
||||
[xe@sunspot ~]$ cat /etc/os-release | head -n1
|
||||
NAME="Arch Linux"
|
||||
```
|
||||
|
||||
Then when you get bored of it, you can remove it just as easily:
|
||||
|
||||
```console
|
||||
$ waifuctl --host http://[::]:23818 delete sunspot
|
||||
```
|
||||
|
||||
No more having to remember the baroque pacstrap flags. No more having to figure
|
||||
out how to add your SSH keys places. No more spending hours setting up and
|
||||
maintaining VMs that you may only need for a few minutes a month. No more having
|
||||
to come up with names for your VMs. No more remembering what you did to VMs when
|
||||
you later need to debug what is going on. No more of any of that. Spend less time
|
||||
fighting whatever you did to OpenSUSE and more time doing things that matter to
|
||||
you, like cuddling waifus.
|
||||
|
||||
[Marin Kitagawa is an S-tier waifu and I will fight you for thinking
|
||||
otherwise!](conversation://Numa/delet)
|
||||
|
||||
Out of the box, waifud ships with templates for the following distributions:
|
||||
|
||||
- [Alpine Linux](https://alpinelinux.org/)
|
||||
- [Amazon Linux](https://aws.amazon.com/amazon-linux-2/)
|
||||
- [Arch Linux](https://archlinux.org/)
|
||||
- [CentOS and CentOS Stream](https://www.centos.org/)
|
||||
- [Fedora](https://getfedora.org/)
|
||||
- [OpenSUSE Leap](https://get.opensuse.org/leap/)
|
||||
- [OpenSUSE Tumbleweed](https://get.opensuse.org/tumbleweed/)
|
||||
- [Rocky Linux](https://rockylinux.org/)
|
||||
- [Ubuntu](https://ubuntu.com/)
|
||||
|
||||
You can query the list of templates with the following command:
|
||||
|
||||
```console
|
||||
$ waifuctl --host http://[::]:23818 distro list
|
||||
alpine-3.13
|
||||
alpine-3.14
|
||||
alpine-3.15
|
||||
amazon-linux
|
||||
arch
|
||||
centos-7
|
||||
centos-8
|
||||
centos-stream-9
|
||||
fedora-35
|
||||
opensuse-leap-15.3
|
||||
opensuse-leap-15.4
|
||||
opensuse-tumbleweed
|
||||
rocky-linux-8
|
||||
ubuntu-18.04
|
||||
ubuntu-20.04
|
||||
```
|
||||
|
||||
[Why isn't there a NixOS image here?](conversation://Mara/hmm)
|
||||
|
||||
[NixOS in particular is going to require a bit of thought to do right here.
|
||||
NixOS is weird because it requires you to specify the system state entirely.
|
||||
`mkvm` currently will create a custom VM image per invocation based on the
|
||||
module you pass it, however I may be able to figure out a decently generic base
|
||||
image that you can layer your own config on top of. Follow <a
|
||||
href="https://github.com/Xe/waifud/issues/6">this GitHub issue</a> for more
|
||||
information as I figure out better ways to do
|
||||
this.](conversation://Cadey/coffee)
|
||||
|
||||
## How It Works
|
||||
|
||||
Previously `mkvm` created a cloudconfig seed file. This is basically an ISO file
|
||||
that gets mounted to the VM and then cloud-init picks up on it and does what it
|
||||
says. This is an incredibly cursed affair, but it does work.
|
||||
|
||||
waifud instead uses the
|
||||
[nocloud-net](https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html)
|
||||
data source to fetch the cloudconfig over HTTP. When you create an instance in
|
||||
waifud, waifuctl uploads the user-data to the server. This is then stored in
|
||||
SQLite and queried from when a VM boots. cloud-init picks up on this data and
|
||||
then executes it, just like it would when you're using a cloudconfig seed from a
|
||||
CD.
|
||||
|
||||
```console
|
||||
$ curl http://127.0.0.1:23818/api/cloudinit/e81ddefc-3fa1-4fdf-9809-ee19f06f9675/meta-data
|
||||
instance-id: e81ddefc-3fa1-4fdf-9809-ee19f06f9675
|
||||
local-hostname: manaphy
|
||||
```
|
||||
|
||||
[The list of names that waifud uses includes sources like every Pokemon up to
|
||||
Sword and Shield! More sources will come in the future!](conversation://Mara/hacker)
|
||||
|
||||
```console
|
||||
$ curl http://127.0.0.1:23818/api/cloudinit/e81ddefc-3fa1-4fdf-9809-ee19f06f9675/user-data
|
||||
#cloud-config
|
||||
#vim:syntax=yaml
|
||||
|
||||
users:
|
||||
- name: xe
|
||||
groups: [ wheel ]
|
||||
sudo: [ "ALL=(ALL) NOPASSWD:ALL" ]
|
||||
shell: /bin/bash
|
||||
ssh-authorized-keys:
|
||||
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPg9gYKVglnO2HQodSJt4z4mNrUSUiyJQ7b+J798bwD9
|
||||
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYr9hiLtDHgd6lZDgQMkJzvYeAXmePOrgFaWHAjJvNU
|
||||
```
|
||||
|
||||
This successfully allows me to refactor `genisoimage` out of the critical path
|
||||
of creating virtual machines. This should also allow me to migrate VMs between
|
||||
hosts with `zfs send` in the future without having to copy around the
|
||||
cloudconfig seeds. I should be able to enable editing cloudconfig data too.
|
||||
|
||||
---
|
||||
|
||||
I hope this preview of what is to come with waifud was interesting. Future tasks
|
||||
will include making a management panel with [Xeact](https://github.com/Xe/Xeact)
|
||||
and [Xess](https://github.com/Xe/Xess), making NixOS modules to automate
|
||||
installation/configuration of machines and so much documentation.
|
||||
|
||||
Even in this minimal state though, waifud shows a lot of promise to be used even
|
||||
more than the `mkvm` prototype tool. If I need an Ubuntu, I can get one in
|
||||
seconds. Faster than AWS even. waifud is the future of my infrastructure.
|
||||
|
||||
waifud makes an init snapshot every time it creates an instance. This will be
|
||||
used to roll back instances to the "fresh out of the box" state if you need to
|
||||
undo everything and go back to a fresh state as an emergency hammer. When you do
|
||||
the kinds of cursed distro testing that I do on a regular basis, this is a very
|
||||
useful hammer to beat your infrastructure with when you need it.
|
||||
|
||||
<center>
|
||||
|
||||
|
||||
|
||||
</center>
|
||||
|
||||
This is going to be instrumental to how my future clusters work. Originally I
|
||||
was going to pair this with [assimil8](https://github.com/Xe/assimil8), but
|
||||
cloud-init unfortunately has mass market adoption and I don't feel like fighting
|
||||
that. The only images I have to build for myself are the Alpine Linux ones.
|
||||
Every other image is the unmodified upstream image that is shipped to cloud
|
||||
providers.
|
||||
|
||||
[I guess I can call Xeserv a cloud provider now!](conversation://Cadey/enby)
|
||||
|
||||
As a fun added bonus/easter egg, whenever you set up an Ubuntu machine with
|
||||
waifud you get an extra message in your MOTD:
|
||||
|
||||
```
|
||||
Welcome to waifud <3
|
||||
```
|
||||
|
||||
I will probably remove or edit this in the future. I'm still figuring out how
|
||||
[cloud-init vendor
|
||||
data](https://cloudinit.readthedocs.io/en/latest/topics/vendordata.html) works.
|
||||
The total lack of useful documentation and examples here is quite annoying. If
|
||||
you have any examples to give me, please do.
|
||||
|
||||
More to come when I have more things to write about.
|
||||
Loading…
Reference in New Issue