site/blog/cinemaquestria-orchestration-2015-03-13.markdown

CinemaQuestria Orchestration
============================

### Or: Continuous Defenstration in a Container-based Ecosystem

I've been a core member of the staff for [CinemaQuestria](http://cinemaquestria.com) 
for many months. In that time we have gone from shared hosting (updated by hand 
with FTP) to a git-based deployment system that has won over the other 
staffers.

In this blogpost I'm going to take a look at what it was, what it is, and what 
it will be as well as some challenges that have been faced or will be faced as 
things advance into the future.

The Past
--------

The site for CinemaQuestria is mostly static HTML. This was chosen mainly 
because it made the most sense for the previous shared hosting environment as 
it was the least surprising to set up and test.

The live site content is about 50 MB of data including PDF transcripts of 
previous podcast episodes and for a long time was a Good Enough solution that 
we saw no need to replace it.

However, being on shared hosting it meant that there was only one set of 
authentication credentials and they had to be shared amongst ourselves. This 
made sense as we were small but as we started to grow it didn't make much 
sense. Combined with the fact that the copy of the site on the live server *was 
pretty much the only copy of the site* we also lost disaster recovery points.

Needless to say, I started researching into better solutions for this.

The first solution I took a look at was AWS S3. It would let us host the CQ 
site for about 0 dollars per month. On paper this looked amazing, until we 
tried it and everyone was getting huge permissions issues. The only way to have 
fixed this would have been to have everyone use the same username/password or 
to have only one person do the deploys. In terms of reducing the [Bus 
factor](https://en.wikipedia.org/wiki/Bus_factor) of the site's staff, this was 
also unacceptable.

I had done a lot of work with [Dokku-alt](https://github.com/dokku-alt/dokku-alt) 
for hosting my personal things (this site is one of many hosted on this 
server), so I decided to give it a try with us.

The Present
-----------

Presently the CQ website is hosted on a Dokku-alt server inside a container. 
For a while while I was working on getting the warts out only I had access to 
deploy code to the server, but quickly on I set up a private repo on my git 
server for us to be able to track changes.

Once the other staffers realized the enormous amount of flexibility being on 
git gave us they loved it. From the comments I received the things they liked 
the most were:

 - Accountability for who made what change
 - The ability to rollback changes if need be
 - Everyone being able to have an entire copy of the site and its history

After the warts were worked out I gave the relevant people access to the dokku 
server in the right way and the productivity has skyrocketed. Not only have 
people loved how simple it is to push out new changes but they love how 
consistent it is and the brutal simplicity of it.

Mind you these are not all super-technically gifted people, but the command 
line git client was good enough that not only were they able to commit and make 
changes to the site, but they also took initiative and *corrected things they 
messed up* and made sure things were consistent and correct.

When I saw those commits in the news feed, I almost started crying tears of 
happy.

Nowadays our site is hosted inside a simple [nginx 
container](https://registry.hub.docker.com/_/nginx/). In fact, I'll even paste 
the entire Dockerfile for the site below:

```Dockerfile
FROM nginx

COPY . /usr/share/nginx/html
```

That's it. When someone pushes a new change to the server it figures out 
everything from just those two lines of code.

Of course, this isn't to say this system is completely free of warts. I'd love 
to someday be able to notify the backrooms on skype every time a push to the 
live server is made, but that might be for another day.

The Future
----------

In terms of future expansion I am split mentally. On one hand the existing 
static HTML is *hysterically fast* and efficient on the server, meaning that 
anything such as a Go binary, Lua/Lapis environment or other web application 
framework would have a very tough reputation to beat.

I have looked into using Lapis for [this beta test site](http://cqsite-beta.apps.xeserv.us/),
but the fact that HTML is so dead easy to modify made that idea lose out.

Maybe this is in the realm of something like [jekyll](http://jekyllrb.com/), 
[Hugo](http://gohugo.io/) or [sw](https://github.com/jroimartin/sw) to take 
care of. I'd need to do more research into this when I have the time.

If you look at the website code currently a lot of it is heavily duplicated 
code because the shared hosting version used to use Apache server-side 
includes. I think a good place to apply these would be in the build in the 
future. Maybe with a nice husking operation on build.

---

Anyways, I hope this was interesting and a look into a side of CinemaQuestria 
that most of you haven't seen before. The Season 5 premiere is coming up soon 
and this poor server is going to get hammered like nothing else, so that will 
be a nice functional test of Dokku-alt in a production setting.