CI failure analysis
Reads failing CI runs, separates real failures from flaky ones, and maps each to the job, step and change that caused it.
Sub-techniques
2
| ID | Name | Description |
|---|---|---|
V1070.001 | Flake detection | Re-run and compare to tell flaky tests from real failures. |
V1070.002 | Workflow structure analysis | Map jobs and steps to the checks they run (the scanner behind this site does this). |
Craft it
- Record test outcomes per run, not just the job status.
- Quarantine tests that fail and pass on the same commit, and file an issue.
- Never skip a test to get a green run.
Reverse engineer it
Signs that a codebase already runs this technique, and where to read how it is configured.
- Retry settings, flaky markers and test result uploads in CI.
Open-source packages
2
- actionlint rhysd/actionlint
- pytest pytest-dev/pytest