Verification · Feedback and evaluation · Reviewer evaluation
Labelled benchmark
PRs with known bugs and known clean changes.
How to build it
- Collect changes that shipped with a known bug and changes that needed no comments.
- Label each case with the expected findings, or with none for the clean cases.
- Report recall on the bugs and precision on the findings for each run.
Signals in a repository
- A benchmarks or eval directory with labelled case files.
- A CI job that runs the benchmark on each change to the reviewer.
- A results file that lists recall and precision per run.
In practice
No scanned repository runs a tool for this check. The steps above are the only guidance.