Property testing in JavaScript with quick_check.js

Property testing is a way to test our code with unexpected corner cases and allow us to find bugs.

At RightScale we use unit testing and integration testing heavily. Integration tests are great, but often, especially with complex applications, very slow. Unit tests are fast, but their chief disadvantage is that they rarely find actual bugs in our code.

The solution can be to use property based testing, where we assert logical properties that our functions should fulfil quantified across any reasonable input the function can take. We then allow our test framework to try its best to falsify the property.

Let’s start with a very simple example. Imagine I need to implement a function to detect odd numbers:

function odd(n) {
  return n % 2 === 1;
}

Unless you are quite a seasoned JavaScript veteran, this code will likely look fine to you. But since you are extra careful, you might write some unit tests:

And all these tests will pass. Looks like our code is solid. Wrong. We can write a property test that expresses a property our function should have:

This code will fail with something like the following message:

Error: Falsified after 3 attempts. Counter-example: -3

Hah! This property test found a bug in our code that we had no idea about. Turns out the % operator returns a negative number when the first operand is negative, rather than the second, unlike other languages like Ruby or Python.

So what happened in the code above? We used a library we have open sourced recently called quick_check.js, which is inspired of course by the famous Haskell library QuickCheck.

In Quick Check you use code to specify certain properties that your functions should obey. A property is expressed as a function that takes in some data and returns a boolean: true when the property holds and false when the property is falsified. For example any stack implementation should have the property that if we push any item on the stack and then pop it, we should arrive at a stack with the same length as the original stack:

expect(function(stack, item) {
  var originalLength = stack.length;
  stack.push(item).pop();
  return stack.length === originalLength;
}).forAll(stacks, qc.any);

Quick Check then proceeds to try to falsify the property by generating random data (called examples) and calling the property with this random data. If the property ends up false, Quick Check will fail the test and print out the example data which falsified the property. As such we can view a Quick Check test as a universally quantified statement that the test library attempts to disprove by counter-example.

The final piece of the puzzle are the data generators that we pass into the forAll call. The quick_check.js library ships with a large number of generators for common types. With these we can quickly derive our own generators, as an example see above the evenNumbers and oddNumbers generators, or this generator for our custom stack implementation:

var stacks = qc.fromFunction(function(ary) {
  var stack = new Stack();
  for(var i = 0, len = ary.length; i < len; i++) {
    stack.push(ary[i]);
  }
  return stack;
}, qc.array);

For a more in depth introduction, please see my talk about this library:

Or check out the project page or the annotated source code.

Also I wrote a follow up blog post that shows how to test Lo-Dash extensions with quick_chech.js.