Awesome Reviewers

Verification · Feedback and evaluation

CI failure analysis

Reads failing CI runs, separates real failures from flaky ones, and maps each to the job, step and change that caused it.

Sub-techniques

  • Flake detection V1070.001

    Re-run and compare to tell flaky tests from real failures.

  • Workflow structure analysis V1070.002

    Map jobs and steps to the checks they run (the scanner behind this site does this).

How to build it

  1. Record test outcomes per run, not just the job status.
  2. Quarantine tests that fail and pass on the same commit, and file an issue.
  3. Never skip a test to get a green run.

Signals in a repository

  • Retry settings, flaky markers and test result uploads in CI.