From c2c2b486a6086ef25afc74ee555df051aeff960b Mon Sep 17 00:00:00 2001 From: Christine Dodrill Date: Sat, 1 Sep 2018 19:42:43 -0700 Subject: [PATCH] blog: add first olin post --- blog/olin-1-why-09-1-2018.markdown | 267 +++++++++++++++++++++++++++++ 1 file changed, 267 insertions(+) create mode 100644 blog/olin-1-why-09-1-2018.markdown diff --git a/blog/olin-1-why-09-1-2018.markdown b/blog/olin-1-why-09-1-2018.markdown new file mode 100644 index 0000000..8e5a601 --- /dev/null +++ b/blog/olin-1-why-09-1-2018.markdown @@ -0,0 +1,267 @@ +--- +title: "Olin: 1: Why" +date: 2018-09-01 +--- + +# [Olin][olin]: 1: Why + +[Olin][olin] is an attempt at defining a radically new primitive to make it +easier to reason about, deploy and operate event-driven services that are +independent of the OS or CPU of the computer they are running on. It will have +components that take care of the message queue offsetting, retry logic, +parallelism and most other concerns except for your application's state layer. + +Olin is designed to work top on two basic concepts: types and handlers. Types +are some bit of statically defined data that has a meaning to humans. An example +type could be the following: + +``` +package example; + +message UserLoginEvent { + string user_id = 1; + string user_ip_address = 2; + string device = 3; + int64 timestamp_utc_unix = 4; +} +``` + +and when matching data is written to the queue for the event type `example.UserLoginEvent`, +all of the handlers registered to that data type will run with serialized protocol +buffer bytes as its standard input. If the handlers return a nonzero exit status, +they are retried up to three times, exponentially backing off. +Handlers need to deal with the fact they can be run out of order. + +## Background + +Very frequently, I end up needing to write applications that basically end up +waiting forever to make sure things get put in the right place and then the +right code runs as a response. I then have to make sure these things get put +in the right places and then that the right versions of things are running for +each of the relevant services. This doesn't scale very well, not to mention is +hard to secure. This leads to a lot of duplicate infrastructure over time and +as things grow. Not to mention adding in tracing, metrics and log aggreation. + +I would like to change this. + +I would like to make a perscriptive environment kinda like [Google Cloud Functions][gcf] +or [AWS Lambda][lambda] backed by a durable message queue and with handlers +compiled to webassembly to ensure forward compatibility. As such, the ABI +involved will be versioned, documented and tested. Multiple ABI's will eventually +need to be maintained in parallel, so it might be good to get used to that early +on. + +You should not have to write ANY code but the bare minimum needed in order to +perform your buisiness logic. You don't need to care about distributed tracing. +You don't need to care about logging. + +I want this project to last decades. I want the binary modules any user of Olin +would upload today to be still working, untouched, in 5 years, assuming its +dependencies outside of the module still work. + +Since this requires a stable ABI in the long run, I would like to propose the +following _unstable_ ABI as a particularly minimal starting point to work out +the ideas at play, and see how little of a surface area we can expose while +still allowing for useful programs to be created and run. + +## Dagger + +> The dagger of light that renders your self-importance a decisive death + +Dagger is the first ABI that will be used for interfacing with the outside world. +This will be mostly for an initial spike out of the basic ideas to see what it's +like while the rest of the plan is being stabilized and implemented. +The core idea is that everything is a file, to the point that the file descriptor +and file handle array are the only real bits of persistent state for the process. +HTTP sessions, logging writers, TCP sockets, operating system files, cryptographic +random readers, everything is done via filesystem system calls. + +Consider this the first draft of Dagger, everything here is subject to change. +This is going to be the experimental phase. + +### Base Environment + +When a dagger process is opened, the following files are open: + +- 0: standard input: the semantic "input" of the program. +- 1: standard output: the standard output of the program. +- 2: standard error: error output for the program. + +Any memory address above byte 4096 is free for implementing applications to use. +Memory addresses below byte 4096 are reserved for future internal-only use. + +### File Handlers + +In the open call (defined later), a file URL is specified instead of a file name. +This allows for Dagger to natively offer programs using it quick access to common +services like HTTP, logging or pretty much anything else. + +I'm playing with the following handlers currently: + +- http and https (Write request as http/1.1 request and sync(), Read response as http/1.1 response and close()) `http://ponyapi.apps.xeserv.us/newest` + +I'd like to add the following handlers in the future: + +- file - filesystem files on the host OS (dangerous!) `file:///etc/hostname` +- tcp - TCP connections `tcp://10.0.0.39:1337` +- tcp+tls - TCP connections with TLS `tcp+tls://10.0.0.39:31337` +- meta - metadata about the runtime or the event `meta://host/hostname`, `meta://event/created_at` +- project - writers of other event types for this project (more on this, again, in future posts) `project://example.UserLoginEvent` +- rand - cryptographically secure random data good for use in crypto keys `rand://` +- time - unix timestamp in a little-endian encoded int64 on every read() - `time://utc` + +### Handler Function + +Each Dagger module can only handle one data type. This is intentional. This +forces users to make a separate handler for each type of data they want to +handle. The handler function reads its input from standard input and then +returns `0` if whatever it needs to do "worked" (for some definition of success). +Each ABI, unfortunately, will have to have its own "main" semantics. For Dagger, +these semantics are used: + +- The entrypoint is exposed func `handle` that takes no arguments and returns an int32. +- The input message packet is on standard input implicitly. +- Returning 0 from func `handle` will mark the event as a success, returning anything else will mark it as a failure and trigger an automatic retry. + +In clang in C mode, you could define the entrypoint for a handler module like this: + +```c +// handle_nothing.c + +__attribute__ ((visibility ("default"))) +int handle() { + // read all of standard input to memory and handle it + return 0; // success +} +``` + +### System Calls + +A [system call][syscall] is how computer programs interface with the outside +world. When a dagger program makes a system call, the amount of time the program +spends waiting for that system call is collected and recorded based on what +underlying resource took care of the call. This means, in theory, users of olin +could alert on HTTP requests from one service to another taking longer amounts +of time very trivially. + +Dagger uses the following system calls: + +- open +- close +- read +- write +- sync + +Each of the system calls will be documented with their C and WebAssembly Text format +type/import definitions and a short bit of prose explaining them. A future +blogpost will outline the implementation of Dagger's system calls and why the +choices made in its design were made. + +#### open + +```c +extern int open(const char *furl, int flags); +(func $open (import "dagger" "open") (param i32 i32) (result i32)) +``` + +This opens a file with the given file URL and flags. The flags are only relevant +for some backend schemes. Most of the time, the flags argument can be set to `0`. + +#### close + +```c +extern int close(int fd); +(func $close (import "dagger" "close") (param i32) (result i32)) +``` + +Close closes a file and returns if it failed or not. If this call returns nonzero, +you don't know what state the world is in. Panic. + +#### read + +```c +extern int read(int fd, void *buf, int nbyte); +(func $read (import "dagger" "read") (param i32 i32 i32) (result i32)) +``` + +Read attempts to read up to count bytes from file descriptor fd into the buffer +starting at buf. + +#### write + +```c +extern int write(int fd, void *buf, int nbyte); +(func $write (import "dagger" "write") (param i32 i32 i32) (result i32)) +``` + +Write writes up to count bytes from the buffer starting at buf to the file +referred to by the file descriptor fd. + +#### sync + +```c +extern int sync(int fd); +(func $sync (import "dagger" "sync") (param i32) (result i32)) +``` + +This is for some backends to forcibly make async operations into sync operations. +With the HTTP backend, for example, calling sync actually kicks off the +dependent HTTP request. + +## Go ABI + +Olin also includes support for running webassembly modules created by [Go 1.11's webassembly support](https://golang.org/wiki/WebAssembly). +It uses [the `wasmgo` ABI][wasmgo] package in order to do things. Right now +this is incredibly basic, but should be extendable to more things in the future. + +As an example: + +```go +// +build js,wasm ignore +// hello_world.go + +package main + +func main() { + println("Hello, world!") +} +``` + +when compiled like this: + +```console +$ GOARCH=wasm GOOS=js go1.11 build -o hello_world.wasm hello_world.go +``` + +produces the following output when run with the testing shim: + +``` +=== RUN TestWasmGo/github.com/Xe/olin/internal/abi/wasmgo.testHelloWorld +Hello, world! +--- PASS: TestWasmGo (1.66s) + --- PASS: TestWasmGo/github.com/Xe/olin/internal/abi/wasmgo.testHelloWorld (1.66s) +``` + +Currently Go binaries cannot interface with the Dagger ABI. There is [an issue](https://github.com/Xe/olin/issues/5) +open to track the solution to this. + +Future posts will include more detail about using Go on top of Olin, including +how support for Go's compiled webassembly modules was added to Olin. + +## Project Meta + +To follow the project, check it on GitHub [here][olin]. To talk about it on Slack, +join the [Go community Slack][goslack] and join `#olin`. + +Thank you for reading this post, I hope it wasn't too technical too fast, but +there is a lot of base context required with this kind of technology. I will +attempt to make things more detailed and clear in future posts as I come up with +ways to explain this easier. Please consider this the 10,000 mile overview of +a very long-term project that radically redesigns how software should be written. + +[gcf]: https://cloud.google.com/functions/ +[lambda]: https://aws.amazon.com/lambda/ +[syscall]: https://en.wikipedia.org/wiki/System_call +[olin]: https://github.com/Xe/olin +[goslack]: https://invite.slack.golangbridge.org +[wasmgo]: https://github.com/Xe/olin/tree/master/internal/abi/wasmgo