Yeller: The Exception Tracker with Answers
Articles Pricing Login Try it Free

Effective Test.Check

This is a blog about the development of Yeller, The Exception Tracker with Answers

Read more about Yeller here

Clojure’s test.check library is a wonderful complement to traditional unit testing. It encourages you to write “properties” that hold globally true for a part of your software. But because of it’s different approach, it’s usually found harder to get started with than unit testing (which many developers are already familiar with). This means developers new to the tool often get stumped with “what kinds of things do I test”.

Test.check isn’t going to eat you. It just wants to break your code

An example would be handy right about now

One of the best ways I’ve found of overcoming this obstacle is a load of examples. With enough good examples, your brain starts generalizing well, and you can generalize and then figure out how you can use test.check’s powerful abilities on your own software.

These examples aren’t an introduction to test.check - the README does a great job of that. These examples solve “what kinds of things do I test?”, not “how do I test”

So, I’ve gathered together all the examples of generative testing I could find, so you can generalize from them:

Invariants

The best use for test.check testing is to take any phrase of the sort “it should NEVER” or “it should ALWAYS” about the software to be tested, and write a test.check test for that. A very simple (and well known) example set of properties can come from effective java: the assumptions Java’s standard libraries make about equals and hashCode methods (thanks to this blog):

equality:

  1. It is reflexive: For any reference value x, x.equals(x) must return true.
  2. It is symmetric: For any reference values x and y, x.equals(y) must return true if and only if y.equals(x) return true.
  3. It is transitive. For any reference values x, y, and z, if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) must return true.
  4. It is consistent. For any reference values x and y, multiple invocations of x.equals(y) consistently return true or consistently return false, provided no information used in equals comparisons on the object is modified.

hashCode:

  1. Whenever it is invoked on the same object more than once during an execution of an application, the hashCode method must consistently return the same integer, provided no information used in equals comparisons on the object is modified. This integer need not remain consistent from one execution of an application to another execution of the same application.
  2. If two objects are equal according to the equals(Object) method, then calling the hashCode method on each of the two objects must produce the same integer result.
  3. It is not required that if two objects are unequal according to the equals (Object) method, then calling the hashCode method on each of the two Objects must produce distinct integer results. However, the programmer should be aware that producing distinct integer results for unequal objects may improve the performance of hash tables.

Clojure developers don’t often implement equality themselves, but if you are, you’ve got to obey these rules (or weird things will happen to your programs). Test.check provides a very cheap way to check these contracts hold.

Checking Algebraic Properties

Riak requires that merge functions have particular properties, they have to be idempotent, commutative, and associative. Testing these properties hold true of your merge function is crucial - without them you’ll causing things like missing user data, unexpected counter values, and so on. I’ve written about this in the past, go read it.

Serialization

Any time you have two functions, a -> b and b -> a, and they are meant to be exact inverses of each other, you can test this pretty easily with test.check. By far the biggest payoff I’ve found for this kind of property test though, is serialization logic - you wanna check you can roundtrip perfectly, and test.check makes that trivial.

Custom Collections

Quite a few libraries implement custom persistent data structures that implement Clojure’s core interfaces. Zach Tellman has a wonderful test.check based library for making sure you don’t violate the assumptions those interfaces hold: collection-check.

Stateful checks:

Ok cool, those are some decent examples for pure functions. But what about real world systems? Stateful stuff often has many more bugs than pure functional code, but can be a bit more tricky to test. There are two libraries on top of test.check that help with testing stateful code - both use a pure “model” of the system to model a state machine and generate actions, then check pre/post conditions on each action against the real code to check that it worked as expected:

Clojure Talks

  • Reid Draper, the author of test.check gave a great talk at Clojure West about it.
  • Ashton Kemerling gave an excellent talk about testing legacy JavaScript apps with test.check
  • Philip Potter gave a nice talk at Euroclojure about test.check in general

QuickCheck

So, that’s all the test.check examples I could find out there (plus some ideas I’ve used in Yeller). But that’s not all. test.check was directly inspired by Haskell’s QuickCheck, and that community has been teaching how to effectively use generative testing for years. Here’s a collection of resources from them, and from folk using similar ideas in other languages:

Talks:

Blog posts and github issues:

Attributions

This is a blog about the development of Yeller, the Exception Tracker with Answers.

Read more about Yeller here

Looking for more about running production applications, debugging, Clojure development and distributed systems? Subscribe to our newsletter: