Awesome Reviewers

Verification · Feedback and evaluation

Developer feedback capture

Records how developers respond to findings (fixed, dismissed, argued) and the sentiment of replies, as the ground truth for every other loop.

Sub-techniques

  • Outcome tracking V1069.001

    Was the finding fixed in a later commit, dismissed or ignored.

  • Reply sentiment V1069.002

    Classify replies as agreeing, disputing or neutral.

How to build it

  1. Store every finding with its later outcome.
  2. Sample disputed findings for human labelling.
  3. Feed labels to calibration and evaluation.

Signals in a repository

  • Reactions and replies on bot comments, collected by a webhook.