Sharing Hypothesis example database from Github Actions

Hypothesis is an awesome framework for producing fuzz testing. Investment in using the framework has definitely paid off with flapping tests showing logic errors. When running Hypothesis via Github Actions with Pytest directly you lose the database unless you upload it. Meaning you can reproduce the error only if you are lucky enough to generate the exact same example.

NOTE: The examples database is not always generated. Only when Hypothesis deems the examples complicated enough does the framework choose to generate this database. If you set CI to any value at all it will produce a long entry with something along the lines of the following. You should prefer this method this approach:

assert 128 <= 127 Falsifying example: test_broken( value=128, ) You can reproduce this example by temporarily adding @reproduce_failure(‘6.119.3’, b’AAAAgA==’) as a decorator on your test case

Ideally we publish the database as an Artifact of the test pipeline. This would allow an engineer to download the failing run and investigate the results.

Using the failure database

On a practical level this involves grabbing .hypothesis/examples and placing it in your local file system. As a best practice I would recommend .hypothesis/{branch-name}. From here you would annotate your failing test with something like the following:

from hypothesis import given, settings
from hypothesis.database import DirectoryBasedExampleDatabase

#
# Loads the failing example database
#
branch_name="branch-name"
failed_db = DirectoryBasedExampleDatabase(path=f".hypothesis/{branch_name}")

#
# Actual test which failed
#
@given(value=integers(min_value=0, max_value=128))
@settings(database=failed_db)
def test_value(value:int) -> None:
    assert value > 0
    assert value < (2 << 8) - 1

Ideally in the future I can find a mechanism to avoid having to manually create a database and just add to the local test cases Hypothesis knows about.

Publishing the Examples Database

Now the question of retrieving the database when the database fails. Uploading artifacts is the easy part and should look like the following. The tricky part is adding an if statement which runs correctly.

jobs:
  python_unit_tests:
    steps:
      - name: Run unit tests
        id: unit_tests # important so we may reference below
      - name: Publish Hypothesis Database on failure
        id: publish_hypothesis_database
        uses: actions/upload-artifact@v4
        if: "${{ failure() && steps.unit_tests.conclusion == 'failure' }}"
        with:
          name: hypothesis-database
          path: ".hypothesis/examples"

if and only if failure

By default, Github Actions will append success() to all steps. Meaning all prior steps in the job must succeed for the step to be run. To work around this, the if statement must include failure().

To reference the unit_test step result the filter steps.<id>.outcome is utilized. outcome is the result of the step, meaning failure when the step did not complete. While conclusion is overridden by the continue-on-error if outcome is failure.

Testing on Github Actions

For verification there needs to be a failing test. Easiest way is to set a variable for to flag for explicit failure. As a result unit_test step now has a export CI_BREAK_TESTS=yes prior to running pytest.

Using the skipif annotation makes this easy with some cheating on condition. When True the test is skipped which is a bit confusing. Either way the ideal implementation is getenv("CI_BREAK_TESTS", None) is None meaning we skip if the environment variable does not exist.

# cicd/broken_test.py
from os import getenv
from pytest import mark


@mark.skipif(
    condition=getenv("CI_BREAK_TESTS", None) is None,
    reason="CI_BREAK_TESTS environment variable not set; set to verify breaking tests produce desired behavior",
)
def test_broken():
    assert False, "intentional failure to ensure machinery works as expected"

Considered Alternative - GitHubArtifact

Ideal solution until each developer needs GITHUB_TOKEN. Effectively this pulls the examples directly from Github’s uploaded artifact.

https://hypothesis.readthedocs.io/en/latest/database.html#hypothesis.database.GitHubArtifactDatabase