Calibration: confirm findings adversarially before surfacing them #269

Closed
opened 2026-10-27 13:35:38 +00:00 by sam · 0 comments
Owner

The risk that kills this feature is a reviewer that cries wolf. A findings list padded with speculation trains the operator to skim past it, and then the feature is worse than nothing because it consumes attention while lowering trust.

Before a finding is surfaced, it must be independently checked and marked CONFIRMED or PLAUSIBLE, with only CONFIRMED able to block. The reviewer must also be able to return zero findings, and to state which properties it checked and cleared -- a review that reports nothing is a valid and welcome result, and saying so explicitly is what distinguishes 'I checked and it is fine' from 'I did not look'.

Prior art from the 2026-08-20/21 sessions on PR #530: three passes each separated confirmed from plausible, one explicitly listed properties it had verified clean, and the confirmed findings were reproduced by mutation before being trusted. That calibration is the feature, not the finding count.

Acceptance criteria

Only CONFIRMED findings block PR creation; a zero-finding review is representable and reported as such; the reviewer records which properties it checked and cleared; padding with style nits is explicitly out of contract.

Close reason

Implemented review calibration: ReviewReport.Checked []ReviewCheck{Property,Evidence} (Evidence required), typed record_checked capture tool, PR-body rendering of cleared properties, and style nits explicitly out of contract (type doc + reviewer prompt). Shipped contract already had confirmed-only blocking and zero-findings-vs-not-run; this adds the 'which properties it checked and cleared' half. PR #787. Design: docs/prds/review-calibration.md.


Imported from beads archie-core-h019.5 · GitHub #555 · type feature · priority P2 · created 2026-08-21 · closed 2026-09-11

The risk that kills this feature is a reviewer that cries wolf. A findings list padded with speculation trains the operator to skim past it, and then the feature is worse than nothing because it consumes attention while lowering trust. Before a finding is surfaced, it must be independently checked and marked CONFIRMED or PLAUSIBLE, with only CONFIRMED able to block. The reviewer must also be able to return zero findings, and to state which properties it checked and cleared -- a review that reports nothing is a valid and welcome result, and saying so explicitly is what distinguishes 'I checked and it is fine' from 'I did not look'. Prior art from the 2026-08-20/21 sessions on PR #530: three passes each separated confirmed from plausible, one explicitly listed properties it had verified clean, and the confirmed findings were reproduced by mutation before being trusted. That calibration is the feature, not the finding count. ### Acceptance criteria Only CONFIRMED findings block PR creation; a zero-finding review is representable and reported as such; the reviewer records which properties it checked and cleared; padding with style nits is explicitly out of contract. ### Close reason Implemented review calibration: ReviewReport.Checked []ReviewCheck{Property,Evidence} (Evidence required), typed record_checked capture tool, PR-body rendering of cleared properties, and style nits explicitly out of contract (type doc + reviewer prompt). Shipped contract already had confirmed-only blocking and zero-findings-vs-not-run; this adds the 'which properties it checked and cleared' half. PR #787. Design: docs/prds/review-calibration.md. --- _Imported from beads `archie-core-h019.5` · GitHub #555 · type feature · priority P2 · created 2026-08-21 · closed 2026-09-11_
sam added the mediumfeature labels 2026-10-27 13:35:38 +00:00
sam self-assigned this 2026-10-27 13:35:38 +00:00
sam closed this issue 2026-10-27 13:35:38 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#269