Xeorg
/
site
2
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
site/blog/lokahi-2018-02-08.markdown

300 lines
9.7 KiB
Markdown
Raw Blame History

---
title: Introducing Lokahi
date: 2018-02-08
github_issue: https://github.com/Xe/lokahi/issues/15
---
# Introducing Lokahi
This week at Heroku, there was a hackweek. I decided to tackle a few problems at
once and this is the result. The two big things I wanted to tackle were building
a scalable HTTP health checking service and unlocking the "flow" state of
consciousness to make developing, understanding and improving this project a lot
easier.
## lokahi
Lokahi is a http service uptime checking and notification service. Currently
lokahi does very little. Given a URL and a webhook URL, lokahi runs checks every
minute on that URL and ensures it's up. If the URL goes down or the health
workers have trouble getting to the URL, the service is flagged as down and a
webhook is sent out.
### Stack
| What | Role |
| :-------- | :------------ |
| Postgres | Database |
| Go | Language |
| [Twirp](https://twitchtv.github.io/twirp/docs/intro.html) | API layer |
| Protobuf | Serialization |
| Nats | Message queue |
| Cobra | CLI |
### Components
Interrelation graph:
![interrelation graph of lokahi components, see /static/img/lokahi.dot for the graphviz]("/static/img/lokahi.png")
#### lokahictl
The command line interface, currently outputs everything in JSON. It currently
has a few options:
```console
$ ./bin/lokahictl
See https://github.com/Xe/lokahi for more information
Usage:
lokahictl [command]
Available Commands:
create creates a check
create_load creates a bunch of checks
delete deletes a check
get dumps information about a check
help Help about any command
list lists all checks that you have permission to access
put puts updates to a check
run runs a check
runstats gets performance information
Flags:
-h, --help help for lokahictl
--server string http url of the lokahid instance (default "http://AzureDiamond:hunter2@127.0.0.1:24253")
Use "lokahictl [command] --help" for more information about a command.
```
Each of these subcommands has help and most of them have additional flags.
#### lokahid
This is the main API server. It exposes twirp services defined in [`xe.github.lokahi`](https://github.com/Xe/lokahi/blob/master/rpc/lokahi/lokahi.proto)
and [`xe.github.lokahi.admin`](https://github.com/Xe/lokahi/blob/master/rpc/lokahiadmin/lokahiadmin.proto).
It is configured using environment variables like so:
```shell
# Username and password to use for checking authentication
# http://bash.org/?244321
USERPASS=AzureDiamond:hunter2
# Postgres database URL in heroku-ish format
DATABASE_URL=postgres://postgres:hunter2@127.0.0.1:5432/postgres?sslmode=disable
# Nats queue URL
NATS_URL=nats://127.0.0.1:4222
# TCP port to listen on for HTTP traffic
PORT=9001
```
Every minute, lokahid will scan for every check that is set to run minutely and
run them. Running checks any time but minutely is currently unsupported.
#### healthworker
healthworker listens on nats queue `check.run` and returns health information
about that service.
#### webhookworker
webhookworker listens on nats queue `webhook.egress` and sends webhooks based on
the input it's given.
### Challenges Faced During Development
#### ORM Issues
Initially, I implemented this using [gorm](https://github.com/jinzhu/gorm) and
started to run into a lot of problems when using it in anything but small
scale circumstances. Gorm spun up way too many database connections (as many as
a new one for every operation!) and quickly exhausted postgres' pool of client.
connections.
I rewrote this to use [`database/sql`](https://godoc.org/database/sql) and
[`sqlx`](https://godoc.org/github.com/jmoiron/sqlx) and all of the tests passed
the first time I tried to run this, no joke.
#### Scaling to 50,000 Checks
This one was actually a lot harder than I thought it would be, and not for the
reasons I thought it would be. One of the main things that I discovered when
I was trying to scale this was that I was putting way too much load on the
database way too quickly.
The solution to this was to use [bundler](https://godoc.org/google.golang.org/api/support/bundler)
to batch-write the most frequently written database items, see [here](https://github.com/Xe/lokahi/blob/7fc03120f731def3a351ddd516430feb635345b4/internal/lokahiadminserver/local_run.go#L245).
Even then, [database connection count limiting](https://godoc.org/database/sql#DB.SetMaxOpenConns)
was also needed in order to scale to the full 50,000 checks needed for this
to exist as more than a proof of concept.
This service can handle 50,000 HTTP checks in a minute. The only part that gets
backed up currently is webhook egress, but that is likely fixable with further
optimization on the HTTP checking and webhook egress paths.
### Basic Usage
To set up an instance of lokahi on a machine with [Docker Compose](https://docs.docker.com/compose/)
installed, create a docker compose manifest with the following in it:
```yaml
version: "3.1"
services:
# The postgres database where all lokahi data is stored.
db:
image: postgres:alpine
restart: always
environment:
POSTGRES_PASSWORD: hunter2
command: postgres -c max_connections=1000
# The message queue for lokahid and its workers.
nats:
image: nats:1.0.4
# The service that runs http healthchecks. This is its own service so it can
# be scaled independently.
healthworker:
image: xena/lokahi:latest
restart: always
depends_on:
- "db"
- "nats"
environment:
NATS_URL: nats://nats:4222
DATABASE_URL: postgres://postgres:hunter2@db:5432/postgres?sslmode=disable
command: healthworker
# The service that sends out webhooks in response to http healthchecks. This
# is also its own service so it can be scaled independently.
webhookworker:
image: xena/lokahi:latest
restart: always
depends_on:
- "db"
- "nats"
environment:
NATS_URL: nats://nats:4222
DATABASE_URL: postgres://postgres:hunter2@db:5432/postgres?sslmode=disable
command: webhookworker
# The main API server. This is what you port forward to.
lokahid:
image: xena/lokahi:latest
restart: always
depends_on:
- "db"
- "nats"
environment:
USERPASS: AzureDiamond:hunter2 # want ideas? https://strongpasswordgenerator.com/
NATS_URL: nats://nats:4222
DATABASE_URL: postgres://postgres:hunter2@db:5432/postgres?sslmode=disable
PORT: 24253
ports:
- 24253:24253
# This is a sample webhook server that prints information about incoming
# webhooks.
samplehook:
image: xena/lokahi:latest
restart: always
depends_on:
- "lokahid"
environment:
PORT: 9001
command: sample_hook
# Duke is a service that gets approximately 50% uptime by changing between up
# and down every minute. When it's up, it responds to every HTTP request with
# 200. When it's down, it responds to every HTTP request with 500.
duke:
image: xena/lokahi:latest
restart: always
depends_on:
- "samplehook"
environment:
PORT: 9001
command: duke-of-york
```
Start this with `docker-compose up -d`.
#### Configuration
Open `~/.lokahictl.hcl` and enter in the following:
```hcl
server = "http://AzureDiamond:hunter2@127.0.0.1:24253"
```
Save this and then lokahictl is now configured to work with the local copy of lokahi.
#### Creating a check
To create a check against duke reporting to samplehook:
```
$ lokahictl create \
--every 60 \
--webhook-url http://samplehook:9001/twirp/github.xe.lokahi.Webhook/Handle \
--url http://duke:9001 \
--playbook-url https://github.com/Xe/lokahi/wiki/duke-of-york-Playbook
{
"id": "a5c7179a-0d3a-11e8-b53d-8faa88cfa70c",
"url": "http://duke:9001",
"webhook_url": "http://samplehook:9001/twirp/github.xe.lokahi.Webhook/Handle",
"every": 60,
"playbook_url": "https://github.com/Xe/lokahi/wiki/duke-of-york-Playbook"
}
```
Now attach to samplehook's logs and wait for it:
```
$ docker-compose -f samplehook
2018/02/09 06:27:15 check id: a5c7179a-0d3a-11e8-b53d-8faa88cfa70c,
state: DOWN, latency: 2.265561ms, status code: 500,
playbook url: https://github.com/Xe/lokahi/wiki/duke-of-york-Playbook
```
### Webhooks
Webhooks get a HTTP POST of a protobuf-encoded [`xe.github.lokahi.CheckStatus`](https://github.com/Xe/lokahi/blob/13bc98ff0665ab13044f08d51ed2141ca0c38647/rpc/lokahi/lokahi.proto#L83)
with the following additional HTTP headers:
| Key | Value |
| :------------- | :------------------------------------------- |
| `Accept` | `application/protobuf` |
| `Content-Type` | `application/protobuf` |
| `User-Agent` | `lokahi/dev (+https://github.com/Xe/lokahi)` |
Webhook server implementations should probably store check ID's in a database of
some kind and trigger additional logic, such as Pagerduty API calls or similar
things. The lokahi standard distribution includes [Discord](https://github.com/Xe/lokahi/tree/master/cmd/discord_hook)
and [Slack](https://github.com/Xe/lokahi/tree/master/cmd/slack_hook) webhook
receivers.
JSON webhook support is not currently implemented, but is being tracked at
[this github issue](https://github.com/Xe/lokahi/issues/4).
### Call for Contributions
Lokahi is pretty great as it is, but to be even better lokahi needs a bunch
of work, experience reports and people willing to contribute to the project.
If making a better HTTP uptime service sounds like something you want to do with
your free time, please get involved! Ask questions, fix issues, help newcomers
and help us all work together to make the best HTTP uptime service we can.
---
Social media links for discussion on this article:
Mastodon:
Reddit:
Hacker News:
Twitter:
Reference in New Issue View Git Blame Copy Permalink