Awesome Reviewers

Verification · Feedback and evaluation

Reviewer evaluation

Benchmarks reviewers on labelled changes: did they find the known bugs, and how many findings were wrong.

Sub-techniques

  • Labelled benchmark V1067.001

    PRs with known bugs and known clean changes.

  • Model as judge V1067.002

    A judge model matches reviewer findings to the labelled bugs.

  • Side-by-side comparison V1067.003

    Run two reviewer versions on the same set and compare.

How to build it

  1. Build the set from real escaped bugs and from changes that needed no comments.
  2. Score recall on bugs and precision on findings.
  3. Gate reviewer changes on the benchmark.

Signals in a repository

  • eval/ or benchmarks/ directories with labelled cases.