Reviewer evaluation
Benchmarks reviewers on labelled changes: did they find the known bugs, and how many findings were wrong.
Sub-techniques
3
| ID | Name | Description |
|---|---|---|
V1067.001 | Labelled benchmark | PRs with known bugs and known clean changes. |
V1067.002 | Model as judge | A judge model matches reviewer findings to the labelled bugs. |
V1067.003 | Side-by-side comparison | Run two reviewer versions on the same set and compare. |
Craft it
- Build the set from real escaped bugs and from changes that needed no comments.
- Score recall on bugs and precision on findings.
- Gate reviewer changes on the benchmark.
Reverse engineer it
Signs that a codebase already runs this technique, and where to read how it is configured.
- eval/ or benchmarks/ directories with labelled cases.