Awesome Reviewers

Verification · Feedback and evaluation

Reviewer optimization

Tunes reviewer prompts and instructions against the evaluation set instead of by hand.

Sub-techniques

  • Prompt optimization V1068.001

    Search prompt variants against a metric.

  • Instruction updates from feedback V1068.002

    Update reviewer instructions from accepted and rejected findings.

How to build it

  1. Define the metric first (from reviewer evaluation).
  2. Optimize on a train split; report on a held-out split.
  3. Review every prompt change like code.

Signals in a repository

  • Optimizer scripts and versioned prompt files in the repository.