Awesome Reviewers

Verification · Finding validation

Confidence calibration

Maps a reviewer's confidence to the observed rate of accepted findings and sets the posting threshold from that curve.

Sub-techniques

  • Calibration curve V1049.001

    Fit confidence against accepted or rejected outcomes.

  • Threshold per reviewer V1049.002

    Each reviewer gets its own threshold.

How to build it

  1. Log confidence and the human outcome for every posted finding.
  2. Fit a calibration model monthly.
  3. Pick the threshold that meets a target precision.

Signals in a repository

  • Thumbs up or down reactions collected on bot comments.