Effective Test.Check
This is a blog about the development of Yeller, The Exception Tracker with Answers
Clojure’s test.check library is a wonderful complement to traditional unit testing. It encourages you to write “properties” that hold globally true for a part of your software. But because of it’s different approach, it’s usually found harder to get started with than unit testing (which many developers are already familiar with). This means developers new to the tool often get stumped with “what kinds of things do I test”.
Test.check isn’t going to eat you. It just wants to break your code
An example would be handy right about now
One of the best ways I’ve found of overcoming this obstacle is a load of examples. With enough good examples, your brain starts generalizing well, and you can generalize and then figure out how you can use test.check’s powerful abilities on your own software.
These examples aren’t an introduction to test.check - the README does a great job of that. These examples solve “what kinds of things do I test?”, not “how do I test”
So, I’ve gathered together all the examples of generative testing I could find, so you can generalize from them:
Invariants
The best use for test.check testing is to take any phrase of the sort “it should NEVER” or “it should ALWAYS” about the software to be tested, and write a test.check test for that. A very simple (and well known) example set of properties can come from effective java: the assumptions Java’s standard libraries make about equals
and hashCode
methods (thanks to this blog):
equality:
- It is reflexive: For any reference value x, x.equals(x) must return true.
- It is symmetric: For any reference values x and y, x.equals(y) must return true if and only if y.equals(x) return true.
- It is transitive. For any reference values x, y, and z, if x.equals(y) returns true and y.equals(z) returns true, then x.equals(z) must return true.
- It is consistent. For any reference values x and y, multiple invocations of x.equals(y) consistently return true or consistently return false, provided no information used in equals comparisons on the object is modified.
hashCode:
- Whenever it is invoked on the same object more than once during an execution of an application, the hashCode method must consistently return the same integer, provided no information used in equals comparisons on the object is modified. This integer need not remain consistent from one execution of an application to another execution of the same application.
- If two objects are equal according to the equals(Object) method, then calling the hashCode method on each of the two objects must produce the same integer result.
- It is not required that if two objects are unequal according to the equals (Object) method, then calling the hashCode method on each of the two Objects must produce distinct integer results. However, the programmer should be aware that producing distinct integer results for unequal objects may improve the performance of hash tables.
Clojure developers don’t often implement equality themselves, but if you are, you’ve got to obey these rules (or weird things will happen to your programs). Test.check provides a very cheap way to check these contracts hold.
Checking Algebraic Properties
Riak requires that merge functions have particular properties, they have to be idempotent, commutative, and associative. Testing these properties hold true of your merge function is crucial - without them you’ll causing things like missing user data, unexpected counter values, and so on. I’ve written about this in the past, go read it.
Serialization
Any time you have two functions, a -> b
and b -> a
, and they are meant to be exact inverses of each other, you can test this pretty easily with test.check
. By far the biggest payoff I’ve found for this kind of property test though, is serialization logic - you wanna check you can roundtrip perfectly, and test.check
makes that trivial.
Custom Collections
Quite a few libraries implement custom persistent data structures that implement Clojure’s core interfaces. Zach Tellman has a wonderful test.check
based library for making sure you don’t violate the assumptions those interfaces hold: collection-check.
Stateful checks:
Ok cool, those are some decent examples for pure functions. But what about real world systems? Stateful stuff often has many more bugs than pure functional code, but can be a bit more tricky to test. There are two libraries on top of test.check
that help with testing stateful code - both use a pure “model” of the system to model a state machine and generate actions, then check pre/post conditions on each action against the real code to check that it worked as expected:
Clojure Talks
- Reid Draper, the author of
test.check
gave a great talk at Clojure West about it. - Ashton Kemerling gave an excellent talk about testing legacy JavaScript apps with
test.check
- Philip Potter gave a nice talk at Euroclojure about
test.check
in general
QuickCheck
So, that’s all the test.check
examples I could find out there (plus some ideas I’ve used in Yeller). But that’s not all. test.check
was directly inspired by Haskell’s QuickCheck
, and that community has been teaching how to effectively use generative testing for years. Here’s a collection of resources from them, and from folk using similar ideas in other languages:
Talks:
- John Hughes at Clojure West. If you’ve only got time to watch one talk from this post, this is it. John invented QuickCheck, and details many advanced uses of it, whilst remaining approachable.
- 29 GIFs only ScalaCheck witches will understand. I love this talk (and the accompanying blog post), and there’s quite a few more examples beyond this post in there.
Blog posts and github issues:
- a basho blog post about how their worker pool was very broken before they quickchecked it
- a high priority bug in riak_core that was discovered using QuickCheck
- An intro to the haskell QuickCheck from a the Stanford Haskell course
- A nice look into how writing good test data generators will magnify the effectiveness of your generative testing suite
- A discussion on the clojure mailing list about using test.check to test
group-by
- Some lessons learned testing an optimizer in Haskell with QuickCheck
- An amazing post about how hardware engineers are miles ahead of us with generative testing, and the ways we should be learning from them
Attributions
This is a blog about the development of Yeller, the Exception Tracker with Answers.