Introduction
Testing is important no matter which programming language or framework you use. hypothesis is an advanced testing library for Python. It allows writing test cases with parameters, and then generating easy-to-understand test data that makes the test fail. The testing library covers most cases and can really help you find bugs in your code.
This article shows how to use Hypothesis for testing in Python, and provides some examples.
How do we differentiate the tests?
Before we get started with attribute-based testing, we need to know the general differences in testing. There are different approaches to group testing, the two most common approaches are based on test methods and test levels. Let's start with the test levels that most people have already heard of. Essentially, there are four test levels (although one may know or define others as well)：
-
Unit Testing
-
Integration Testing
-
System Testing
-
End-to-end testing
Different test levels focus on different things. Unit testing focuses on a specific part or feature of the software. This can be a single feature or part of a feature. Conversely, integration testing focuses on collaboration through the interfaces of software components. System testing goes even further and can test the entire system.
Now, we will look at the wide variety of testing methods that exist.
The most common and known are static and dynamic testing. Static testing is the process of statically checking for possible errors in the program code, interface or documentation without actually running the software under test. If the software or its parts are actually executed, we call it dynamic testing. Writing unit tests and integration tests are dynamic tests.
Another common approach is the boxed approach. Basically, it can be divided into white-box testing and black-box testing (and gray-box testing as a mix of both). White-box testing verifies the internal structure or working of a program. Black-box testing is the opposite, in which the application is treated as a black box and tested for its interactions. This means testing functionality without knowledge of the internal implementation.
What is attribute-based testing?
Now that we've taken a quick look at how to differentiate tests, you may be asking: What is property-based testing?
Property-based testing refers to writing logical statements (i.e., "properties") that are true for your code, and then using an automated tool to generate test input (generally, some specific type of randomly generated input data) and observe whether the program whether the properties remain unchanged when the program accepts that input. If an input violates a property, the user proves that there is an error in the program and finds a convenient example that demonstrates the error.
Attribute based testing using Hypothesis
Let's take a simple example. Suppose you have two functions crement() and decrement(). A sample implementation might look like this：
# increment_decrement.py
def increment(number: int) -> int:
return number + 1
def decrement(number: int) -> int:
return number - 1
You might write unit test code for both, as follows：
# test_increment_decrement_pytest.py
from increment_decrement import decrement
from increment_decrement import increment
def test_increment():
x = 5
expected = 6
actual = increment(x)
assert actual == expected
def test_decrement():
x = 5
expected = 4
actual = decrement(x)
assert actual == expected
Note: The test code is written using the pytest framework.
Of course, you can write more test scripts to test two functions with different values, or even parameterize the tests. However, in the end you will test both functions using predefined values.
Writing tests using attribute-based test libraries (e.g. Hypothesis) is different. Here, you specify the type to be tested and how the software works or behaves. The library then generates random values based on the specified types to actually test the functionality.
Let's see how we can use Hypothesis to test two of our features.
# test_increment_decrement_hypothesis.py
from hypothesis import given
import hypothesis.strategies as st
from increment_decrement import decrement
from increment_decrement import increment
@given(st.integers())
def test_increment(x):
expected = x + 1
actual = increment(x)
assert actual == expected
@given(st.integers())
def test_decrement(x):
expected = x - 1
actual = decrement(x)
assert actual == expected
As you can see, both test scripts have an argument x. The value of x is generated by Hypothesis using the integers() method. Hypothesis provides various methods. Essentially, these methods correspond to built-in types or other structures, and generate random data that matches the given type.
Sounds good, doesn't it? But what if we want to test a function with a particular value to make sure it can also use that value? Hypothesis provides an @example() decorator in which you can define a value that can be passed to the corresponding function even if that value does not belong to the randomly generated test data set.
Let's take a simple example：
# div.py
def div(dividend: int, divisor: int) -> int:
return dividend // divisor
We define a function div() that takes a divisor and a divisor and returns the quotient of both. Note that both arguments are integer data, so the result should also be integer data, and we use Python's // operator to perform integer division.
To test the div() function, we created a new test file test_div.py and wrote a test script called test_div().
# test_div.py
from hypothesis import example
from hypothesis import given
import hypothesis.strategies as st
from div import div
@given(dividend=st.integers(), divisor=st.integers())
def test_div(dividend, divisor):
if divisor == 0:
expected = -1
else:
expected = dividend // divisor
actual = div(dividend, divisor)
assert actual == expected
Similarly, we use Hypothesis' integers() method to generate the values of the divisor and the divisor. The test script we write may or may not pass, depending on the values generated by Hypothesis at the time of execution. To ensure that the value 0 is always passed to the div() function, we add @example(1, 0) to the test_div() function. Thus, even if div() is not in the randomly generated dataset, it will be called at least once with the value 0 of the divisor.
If we run the test script as is, test_div() will always fail. So, let's modify the div() function to handle this situation and make the test pass：
# div.py
def div(dividend: int, divisor: int) -> int:
if divisor == 0:
return -1
return dividend // divisor
Outline
This article focuses on what attribute-based tests are and why they are useful. In addition, you take a quick look at the Hypothesis library, which enables you to write attribute-based tests and execute them together with pytest tests.