Bash Tips to make your CI runs better

This is a blog about the development of Yeller, The Exception Tracker with Answers

Read more about Yeller here

I’ve been tweeting a bunch of bash tips recently. Most of this has come out of making Yeller’s CI build faster and easier to fix when it breaks. Bash is a pretty powerful tool for improving your CI build, and adding a bit more complexity can speed up your debugging a whole lot. I thought I’d write up some specific bash lessons I’ve learned in more detail, with some explanation.

Debugging Slow Builds with Timestamps

By far the biggest win for speeding up CI runs in my experience is prepending timestamps to each line of the build. Fast builds are important for feedback cycles, they let you deploy faster after making a change (assuming your deploys are gated on CI), and a whole bunch more. I aim for 10 seconds for my builds in CI (though yeller is nowhere near there yet, because the clojure compiler is slow). Having timestamps on each phase lets you at least see which bit was slow:

with_timestamps() {
    while read -r line; do
        echo -e "$(date +%T)\t$line"
    done
}

build() {
  # do your actual build here
}

build | with_timestamps

pipefail

There’s an issue with the above code though - when the build function fails, the whole CI script won’t actually fail. Bash has a very handy option to alleviate this:

set -o pipefail

pipefail means that the exit status of a pipe will only be 0 if every stage in the pipe was 0. It’s a very useful option for this case - you can just set it and then your CI script will fail when it should.

Concurrency

Does your build have multiple, slow stages that aren’t dependent on each other? If so, you can sometimes get dramatic speedups by running the stages in parallel. For example, rails apps often do asset compilation during CI, because otherwise your deploy could fail if you messed up the asset pipeline somewhere. You can run the asset compilation and your test suite in parallel, and as long as the overall build time isn’t super dominated by one of the steps (via Amdahl’s Law), you can get dramatic speedups from this. Here’s a basic pattern, for starters:

bundle exec rspec spec &
rspec_pid=$!
RAILS_ENV=production bundle exec rake assets:precompile:primary &
assets_pid=$!

wait $rspec_pid &&
    wait $assets_pid

You can get dramatically more complicated than this if/when you want. Some of the builds I’ve worked on put the output from the concurrent commands into a file in /tmp, so you don’t get interleaved output.

set -x for better debugging

Bash has this handy option, set -x, which logs all commands before they’re run, with a full expansion. This isn’t just useful when you’re initially writing your CI script, it can help you and your team debug failures when they do happen.

set -e to stop && explosion

Many CI scripts are relatively simple - they just run N shell commands in sequence. To start with, using && works fine for this. But, there’s a better way: set -e. set -e means that as soon as one command exits with a failure, the entire script exits with a failure. This means you can just write your straight line script, without using && at the end of every line

Putting it all together

Yeller’s CI script (roughly) is here. I do some JVM specific things in there (like, I compile my production jar, and my test files into another jar, then run the tests by shoving bolth of those on the classpath), but apart from that, it’s all bash. This script gives me timing on how long things take, good debugging output, and runs as fast as it can without optimizing the clojure compiler.

CI servers running continous builds are incredibly useful. Bash is a powerful tool for getting the most of your CI server, speeding up your builds, making them easier to debug and tune and so on. Making your CI more useful requires a bit of work, but it pays off when you have a failure, or a suddenly slow build, and can debug the problem quickly.

This is a blog about the development of Yeller, the Exception Tracker with Answers.

Read more about Yeller here

Looking for more about running production applications, debugging, Clojure development and distributed systems? Subscribe to our newsletter: