Verification · Feedback and evaluation · Missed bug analysis
Eval set growth
Add each missed bug to an evaluation set for the reviewers.
How to build it
- Add each missed bug as a test case with the diff and the expected finding.
- Keep the cases in the evaluation dataset that the reviewers run against.
- Re-run the reviewers on the set after every change to a rule.
Signals in a repository
- An evaluation dataset file or configuration, such as promptfoo test cases or a Langfuse dataset.
- Test cases named after the bug or pull request they came from.
In practice
No scanned repository runs a tool for this check. The steps above are the only guidance.