Bolth: a faster, more humane runner for clojure.test

Clojure’s built in clojure.test test framework is good enough, most of the time. It’s small, very well known, and ships with the language. But it isn’t very human friendly to actually use, day to day. Test failures, especially for larger data structures get pretty hard to read. And the stacktraces it outputs on an exception are very gnarly. Also, folk using it on larger projects often end up writing a bunch of supporting code to do things like diagnose slower test runs, or exit with the right exit code in CI when the tests fail. Lastly, it doesn’t provide any support for running tests in parallel.

Bolth

So, today, I’m releasing bolth, an alternative test runner for clojure.test which solves all of the above problems:

• it uses the same data diffing method that humane-test-ouput uses to provide better test failures
• it uses io.aviso.exception for pretty printing exception stacktraces. Aviso is a huge improvement over the standard jvm exception formatting - stacktraces are printed in the right order, and have colors and such.
• it includes a load of formatting and reporting options (see the docstring for more on that)

Here’s what it looks like in action, on yeller’s full test suite:

Note:

1. The full run is very fast. This is partly through the parallelization provided with bolth, partly through careful design. Fast feedback cycles are still not quite understood as the big deal that they are: with fast feedback, you can experiment on code with tests just as fast as you do in the repl, and you end up with real regression suites as well.
2. With my current setup, test run output sticks around in the repl (on the right of the screen). This means you’re not rerunning tests just to look at the failure message (or worse: scrolling up with your mouse/tmux to read things).
3. The whole repl pane is “cleared” (by printing a load of blank newlines) before a run (this is optional with bolth, but just an extra option to the runner). This stops the eyes getting distracted by previous test output.

Now let’s see what a failing run looks like:

And with an exception:

Another minor helper bolth provides wraps clojure.tools.namespace.repl/refresh to pretty print compilation exceptions. Here’s what that looks like:

Much nicer.

Showing slow tests

Bolth provides a lot of formatting and reporting options. By far and away my favorite though, is the reporting of slow tests it can do:

This just requires adding a single option :show-slow-tests true to the runner. Typically test suite runtimes are completely dominated by a few slow tests. Yeller’s suite (before this option) took about 3 seconds, with 2.5 seconds of that time entirely contributed by 4 tests.

Fin

So, that’s it. The code’s up on github, the jar is on clojars. It should be relatively trivial to convert any existing clojure.test workflow to bolth.