This blog is structured into three segments, each addressing key aspects of this innovative testing approach. In Part 1, we cover the fundamentals and highlight the distinctions between property-based and example-based tests. Part 2 showcases the advantages and provides practical templates for integration. Finally in Part 3, we'll analyze a real-world example and explore the nuances of debugging with property-based tests. Join us on this journey as we connect the dots and offer a first dive into the power behind property-based testing.
Understanding the difference between example tests and property-based tests
Testing has become a must-have in software development to ensure the correctness of our code and help us develop functions with methodologies like TDD or BDD. We even have defined the granularity of the tests we must create with the test pyramid or trophy.
But if you think about it, most of the test types, like unit testing, integration testing, and end-to-end testing, are based on patterns or examples that we give to the test framework to run or test against. This is called example-based testing.
Now, property-based testing is a powerful approach that has gained considerable popularity in recent years. Let’s make an analogy to start understanding what property-based testing is and why it might be useful.
Analogy
Imagine you are a chef trying out a new recipe for a cheesecake.
Example-based testing:
In this analogy:
Each recipe represents an individual example-based test. The ingredients and instructions are like specific inputs and expected outputs for your function.
Tasting the cakes is equivalent to running the test and checking if the function behaves correctly for those specific inputs.
Property-based Testing
In this analogy:
- The magic oven represents the property-based testing framework, which generates random inputs for your function.
- The requirements for the cheesecake are the properties or invariants. These are the conditions that should hold in order to test the correctness of your function.
- The taste test of the random cakes is equivalent to running the property-based test and checking if the function meets the defined properties for all the randomly generated inputs.
Comparison:
- In example-based testing, developers provide specific inputs and test cases that act as predefined scenarios for evaluating their code. You can think of these test cases as "recipes" in the context of the analogy. Typically, these tests focus on common and expected situations, often referred to as the "happy path," or they target inputs that are likely to cause failures. It's important to note that the effectiveness of these test cases relies on the developer's ability to think carefully and thoroughly when designing and creating them.
- In property-based testing we don’t specify the recipes; instead we let our magic oven generate the recipes or inputs that we will use later in our tests. Defining the requirements or properties that the function should satisfy it’s our challenge as developers.
What is it:
Now that we have an idea of what property-based tests are and how these are different from the most common tests that we use, let’s dig into some technical concepts using the JS property-based tests library fast-check.
What's property-based testing
It is a testing strategy that verifies that a function, UI flow, or any system abides or holds certain characteristics or requirements called properties. This kind of test helps the developers generate random inputs according to the specifications.
It allows users to focus on the behaviors they want to assess, rather than the specific values required to assess them. We care about the requirements for the cheesecake (“should be moist and delicious”) instead of the ingredients. This strategy achieves this by generating random inputs and applying them to the code being tested.
How do we get it to do that? We basically need three elements: a property, arbitraries, and a runner.
The property: This is the core of our test. We need to define it with two main elements:
- Predicate: A condition that represents one of our cake requirements; the magic oven will use arbitraries to test randomly generated inputs against the predicate.
- Arbitraries: Functions that we can use so that the magic oven can generate the random values we want as input for our test, like generating random ingredients for the magic oven
A property can be expressed as follows:
Above is the definition of a property for a function that should capitalize any string, meaning that the first letter of that string should always be uppercase and the rest lowercase. The complete property test for our function in fast-check looks like this:
test('should capitalize any string', () => {
fc.assert(
fc.property(fc.string(), (data) => {
const result = capitalize(data);
expect(
data.length === 0
|| result[0] === result[0].toUpperCase()
).toBeTruthy()
})
);
});
Analyzing the code above:
fc.assert = Is the runner responsible for interpreting and executing the property. It basically helps the test framework to run a property test.
fc.property = Is a callback function, it receives the arbitraries, you can pass n arbitraries of any type like:
fc.property(fc.string(), fc.integer(), fc.array(fc.integer()), (data) => {
…
})
The fc.assert is one of the main parts of fast-check’s “magic oven”: it is the one that helps to run and orchestrate the cooking inside of it. We define the arbitraries, we define the property that should hold, and fc.assert takes care of running the test multiple times and with all sorts of different values for our arbitraries trying to cover edge cases and potential issues that may not have been considered with example-based tests. The default configuration runs the property against 100 generated inputs.
Next Steps:
We have covered what property-based tests are and the difference between example-based tests. Also now we know the basic elements of a property and how to define one using fast-check. The next part of this blog will talk about why we should use it and define a template to approach property-based testing.
References:
