A (Gentle) Introduction to Property-based Testing

posted in How-to by Attie

Code to Test Code

Over the last few decades there has been an increasing shift towards the automation of application testing.  From low-level unit tests to integration test suites and everything in-between, testing has become the topic of much research and debate.  In this post I'll focus specifically on Unit Testing and what can be done to improve the efficacy and coverage of these types of tests, with implementation provided for .NET.

Unit Tests

Traditional unit tests consume a small piece of functionality using a single set of input parameters.  The result of this call is then compared to an expected value or side-effect.  These tests often require a lot of work to write and only test a small subset of the possible input and output values.  Some unit test runners allow for multiple iterations of these tests using a collection of input and output values, but ultimately it is still up to the developer to manually select the suite of input values and calculate the correct output values.  This process, known as example-based testing, can often result in missed edge cases or calculation errors by the developer, leading to unnecessary test failures and false negatives.  Most of these types of tests also do not significantly contribute to the documentation of the desired functionality, but rather focuses on verifying results.

Generating Test Cases

What if these test suites could be generated for us instead?  If we were able to establish a series of properties that define the relationship between the input and output values of a function, we could (in theory) randomly generate a series of input values and simply check whether these properties hold true for the output values produced.  In this way we're able to supplement the example-based tests and more accurately document the desired functionality (as a series of properties) whilst automating the generation of test cases.


A "property" in this context does not refer to a field wrapped in a getter/setter, a value in a property bag or any other programming construct that uses the same nomenclature.  Instead, a property is more akin to a physical attribute or rule that describes behaviour.  Some examples in everyday life include:

  • Adding items to a shopping cart increases the number of items in the cart
  • Multiplying any real number by zero results in a zero value
  • Executing any idempotent operation more than once produces the same result as executing it only once
  • The associative property of addition : a + b = b + a

Automating test generation

Writing your own automated test generator for an OO language would roughly involve the following steps: 

  • Reflecting on input types (from primitives to complex objects)
  • Generating reasonable (and unreasonable) values for those inputs
  • Hooking into the selected Unit Testing framework to loop over the test, passing in each generated input set
  • Aggregating the success/failure messages for each property (including the input set for failure conditions)

Whilst this may seem like a reasonable amount of effort to implement yourself, the real complexity comes into providing extension points that allow the user to constrain the input values or provide custom values, configure the number of test iterations, etc.  This code soon starts to resemble a framework, so the right thing to do is to delete all of it.  What we need is a property-based testing framework*.

Enter FsCheck

FsCheck is an F# port of the excellent QuickCheck Property-based Testing Framework for Haskell.  It integrates with most .NET unit test libraries, but the most popular F# one is xUnit, so we'll be using that for this tutorial.  The integration between F# and C# is (mostly) hassle-free, so adding this to an existing C# solution is trivial for the most part.  If you're a C# developer who hasn't looked at F# before, I highly recommend that you do so.  The beauty and readability of well-written functional code is something that you'll have to experience first-hand in order to truly appreciate it.  It's also a great fit for writing unit tests due to its terseness and implicit typing.

Testing Addition

Since this is an introductory post, let's keep things simple.  One of the most basic operations possible is the addition of 2 numbers.

Let's assume that the following static method exists inside a C# project called PBTExample.Library:

In order to separate our tests from the implementation (and since F# code cannot coexist with C# code within the same project), let's create a new F# library project to host our unit tests:

At this point your solution should look something like this:

Next we remove the unnecessary Script.fsx and rename "Library1.fs" to "Math.js" to simply match the source file under test:

In order to test the Library functionality from within our Test project, we need to add a project reference to the C# project:

Finally, in order to add FsCheck and XUnit to our solution we install the FsCheck.XUnit package via NuGet.  This contains both these libraries and the integration between them.  In order to enable Visual Studio's built-in test runner to execute XUnit tests we also install a package now to act as an adapter.  This can be done via the Project -> "Manage NuGet Packages" dialog or via the Package Manager Console with the following commands:

We're now ready to start implementing some Property-based Tests.

Properties of Addition

At this point we need to take some time to think about what the properties are that we actually want to test.  Luckily for us addition is a well-defined mathematical operation, so its properties are well-known and documented.  For other functionality we have to really think about what we're testing and what pre- and post-conditions we have to satisfy and what the relationship between these are.  This obviously requires a lot more thought than standard example-based testing, but it results in a much higher quality of test.

Some properties of addition:

  • Identity Property
    • a + 0 = a
  • Commutative Property
    • a + b = b + a
  • Associative Property
    • (a + b) + c = a + (b + c)
  • Distributive Property
    • a * (b + c) = a*b + a*c

Testing the Identity Property of Addition

In order to test the Identity property of Addition, we can simply test addition using an arbitrary number + 0 and see whether the result is still our original number.  The code for implementing this is extremely compact thanks to FsCheck, which does all the heavy lifting for us:

This may be a first look at F# code for some readers, so I'll go through this line by line to explain exactly what's happening here.

This line simply declares a new module called Math in the PBTExample.Tests namespace.  F# modules are roughly equivalent to a static class in C# and simply serves as an organizational construct for code elements.

The open statement in F# is equivalent to C#'s using statement.  It imports namespaces in order to save you from typing out fully qualified identifiers each time a member of that namespace is used.  In this case we're importing the testing library and our C# math library.

Line 1

The [<...>] element is used to assign attributes to members.  In this case we're assigning the FsCheck.Xunit.PropertyAttribute to the function we're declaring underneath this element.  This tells FsCheck to treat this function as a property-based test.

Line 2

The let keyword in F# is used to associate an identifier with a new value or function expression.  I.e., it allows us to declare new "variables" and functions.  In this case we're declaring a new function expression with the rather verbose name of "Adding 0 to any number should return the same number".  Note the use of the double-backtick to escape "friendly" identifiers, i.e., identifiers containing otherwise illegal characters such as spaces and punctuation.  This allows us to use full sentences for test names rather than an obscure naming pattern such as Pascal casing or underscores.

The parameter list follows after the function expression name.  In this case we have a single parameter named a.  Note that F# uses implicit typing to infer the type of this parameter based on its usage, so there's no need for us to explicitly provide any type information here.  You can also view the inferred type for each identifier by hovering over it in Visual Studio.  Also note that F# doesn't require any extraneous syntax such as brackets or commas to contain the parameter list for function expressions.  This is related to the nature of function expressions, which allows for partial application or "currying".  In this case we could also have surrounded the parameter list with brackets without any side-effects.

Finally, the = sign signifies the start of the function expression's body.

Line 3

This is the first line of the body of our test.  Note the indentation of this line and the succeeding lines of the body.  Indentation in F# code is very important, since this is used to denote the scope of lines.  In this case we're indicating that this line actually forms part of the line above it and is part of the declaration.

The let keyword is used here to declare a value called result.  Result is defined as being the output from the invocation of the Add method in our C# library, using the parameter x and the constant value zero.  Note that no explicit type is specified for result either, since this is inferred from the return type of our library call.  In this case it behaves exactly like the var keyword in C#.  Also note the absence of any unnecessary semicolons or other ceremony around this invocation.

Line 4

In F# the last line of a function expression is used as the return value by default.  In this case we're comparing result with x to determine whether adding 0 actually left the value unmodified.  Note that comparison uses the same single-equals operator as assignment.  The absence of the let keyword indicates that this is a comparison rather than an assignment.  Since F# favors immutable values, assignment is the exception rather than the rule, so the equals operator can serve double-duty as both declaration and comparison.

The result from this function is interpreted by FsCheck to indicate whether the test passed or failed.  True = pass, False = fail.  In case of failure, FsCheck generates an appropriate failure message, including the input values used.

Running the Tests

Build the solution now using F6 or your favourite mouse action.  Since we installed the VS Test Runner adapter for XUnit, we can simply run this test using the out-of-the-box functionality.  Use Ctrl R, A or Test -> Run -> All Tests from the menu bar to run all tests in the solution.

If all went well we should see the following result:

You might also notice the "Ok, passed 100 tests" message in the output.  By default FsCheck generates 100 input value sets for us per property, using an intelligent range of values based on the parameter types.  This is of course configurable, but most of the time 100 is more than adequate.  The input values are only printed in the case of failed tests, so we don't have any insight into which values were actually generated for us in this case.

Of course, we can easily remedy that by printing the values manually in our test as follows:

We can now see the generated values in the test output:

As you can see, FsCheck generates a wide range of values covering most/all of the edge cases better than a human programmer might.

Let's make this test fail now to see what that looks like.  By simply changing the test to this:

We should get an output along these lines:

Our test failed after only 4 tries using a value of 73786976299.133173762M.  FsCheck also uses a process called Shrinking to try and simplify the failure conditions to a more human-readable form.  For decimal values it reruns the test multiple times using progressively fewer decimal digits until the test passes again or it ends up with an integer.  For other data types it uses other techniques to achieve the same effect.

Testing the Commutative Property

To test the Commutative property, we need to pass 2 randomly generated numbers into the test function and then determine whether adding them in reverse order produces the same result as adding them in the original order.  The code for this is trivial:

The only real difference between this test and the first one is that we now accept 2 parameters rather than just 1.  FsCheck generates these for us and passes them in as parameters to this function for each run.  The other tests follow a similar pattern and is equally trivial to implement.  The full test suite is provided below for completeness.






* In this tutorial I focus on implementing Property-based Testing for the .NET framework, but there are similar JVM solutions as well.  See ScalaCheck for a Scala port of QuickCheck, for example.

Property-based Testing F# FsCheck Unit Testing



There are no comments yet.