Confidence calibration
Maps a reviewer's confidence to the observed rate of accepted findings and sets the posting threshold from that curve.
Sub-techniques
2
| ID | Name | Description |
|---|---|---|
V1049.001 | Calibration curve | Fit confidence against accepted or rejected outcomes. |
V1049.002 | Threshold per reviewer | Each reviewer gets its own threshold. |
Craft it
- Log confidence and the human outcome for every posted finding.
- Fit a calibration model monthly.
- Pick the threshold that meets a target precision.
Reverse engineer it
Signs that a codebase already runs this technique, and where to read how it is configured.
- Thumbs up or down reactions collected on bot comments.
Open-source packages
1
- scikit-learn scikit-learn/scikit-learn