Test isolation and determinism

Tests should be deterministic, isolated, and assert the intended behavior (not just a coincidence or shared prior state). Concretely: - Reset external/shared state for every test (especially Redis, quotas, counters). Do not rely on the order tests run.

copy reviewer prompt

Prompt

Reviewer Prompt

Tests should be deterministic, isolated, and assert the intended behavior (not just a coincidence or shared prior state). Concretely:

  • Reset external/shared state for every test (especially Redis, quotas, counters). Do not rely on the order tests run.
  • Avoid vacuous assertions and shared artifacts (e.g., “last line in a shared file”) that can match data from earlier blocks.
  • When testing stateful backends, prefer multi-worker coverage so race/state behavior is exercised.
  • Keep test scaffolding minimal—remove dead/irrelevant setup that doesn’t actually reach the code under test.
  • Use fixed test inputs/identifiers where applicable to avoid flakiness.

Example (state reset + independent setup):

add_block_preprocessor(sub {
    my ($block) = @_;

    # Ensure Redis-backed tests start from a known state
    require("lib.test_redis").flush_all();

    # Optionally set a deterministic default request
    if (!$block->request) {
        $block->set_value("request", "GET /t");
    }
});

Example (avoid shared-state/vacuous “tail -n 1”):

  • Don’t assert against a shared “last export” file unless you can correlate the export to the current request.
  • If you must pattern-match, make the pattern reject the forbidden value (e.g., all-zero) instead of only matching a broad UUID/hex shape.

Apply this standard when adding or modifying tests in the Testing category to reduce CI flakiness and increase confidence in coverage.

Source discussions