Verification · Feedback and evaluation · Reviewer evaluation
Model as judge
A judge model matches reviewer findings to the labelled bugs.
How to build it
- Write a grading prompt that matches each reviewer finding to a labelled bug, or to none.
- Check the grader against a set of hand-labelled matches first.
- Report grader agreement with the hand labels alongside each score.
Signals in a repository
- A grader prompt or rubric file in the eval directory.
- A config that sets a rubric-based grader assertion.
- A file of hand-labelled matches used to check the grader.
In practice
No scanned repository runs a tool for this check. The steps above are the only guidance.