How Yeller does deploys: CircleCI, Hubot and Haskell

This is a blog about the development of Yeller, The Exception Tracker with Answers

Read more about Yeller here

Yeller’s deploy system is somewhat interesting. As a project with only one developer, almost anything could work. However, there are a few unique things about my situation that lead to me building something a bit more fancy than “run bash commands over ssh from your laptop” (which is what many many deploy systems come down to).

Constraint 1: Fat Jars

There are two things that drove me to building a more integrated deploy system for yeller:

Yeller ships “fat” jars, which include all dependencies. This is great from many standpoints - all you need to do to run a piece of yeller is have the appropriate jar file in a folder, a config file, and be behind the firewalls to talk to the database. This contrasts well with a lot of other dynamic languages, where you ship source code, then have to do some kind of installation of dependency source code (e.g. bundler, virtualenv) in production.

However, the thing with these jarfiles is that they get relatively large - yeller’s api jar, for example is nearly 80MB. Whilst uploading 80MB * server count is relatively tenable for most home broadband connections, it still takes a while.

Constraint 2: Deploys whilst on 3g

I very often find myself working from coffee shops, trains, etc. Places that tend to have very spotty internet connections. This isn’t so great when you want to push up these large jar files, and you often lose connection completely.

All these issues hit me pretty hard while trying to deploy a bugfix from a train one particular time. Whilst I did indeed get the upload done, it took 40 minutes or so, which isn’t really acceptable. So, I set out to build something better.

Yeller’s current deploy system works something like this:

  • After each git push, CircleCI runs and builds the fat jar files
  • When I tell the system to deploy (which is done via hubot):
  • a (tiny) haskell app hits CircleCI’s api, and downloads jar files
  • it then uploads them to each server, using rsync (which is far faster than scp, and still works over ssh)
  • if the build failed, or the jar files aren’t there, the app logs to chat why it isn’t deploying
  • otherwise, it runs some bash commands over ssh to restart the JVMs (doing a rolling restart)
  • services are deployed separately.

Here’s what a full deploy looks like:

(squirrel is the name of the haskell deploy app that talks to CircleCI and is controlled entirely via hubot, and logs to the chatroom).

CircleCI

The biggest component of this, and one that I’ve been very happy outsourcing, is building the jar files. I don’t exactly want to build them on my laptop, as then I still have to worry about uploading them somewhere, which is painful over 3g.

I use CircleCI, for building jar files (and running tests) and have been amazingly happy with them. Their support has astounded me on several occasions, and their feature that lets you ssh into a CI run has saved a large amount of debugging time on bad builds. They also directly host my jar files (build artifacts), so I don’t have to worry about the security implications of pushing them to s3 or another storage service, which is invaluable for this deploy infrastructure.

Tradeoffs

As with all software development, there are tradeoffs here. Specifically, now deploy relies on the correctness of the deployment app (that fetches jars, reads off of CircleCI’s api etc). This means from time to time, I’ve spent time debugging the deployment app. It’s written in haskell though - so much of the usual worries about basic correctness are solved by the type system.

All in all, I’m pretty happy with this system. It took maybe a couple of days to write the haskell app, about 30 seconds to setup CircleCI and have it generate jarfiles.

This is a blog about the development of Yeller, the Exception Tracker with Answers.

Read more about Yeller here

Looking for more about running production applications, debugging, Clojure development and distributed systems? Subscribe to our newsletter: