The auditor as the first guard, with a measured judge #1987
Notifications
Due Date
No due date set.
Blocks
Depends on
Reference: sam/archie#1987
Reference in new issue
Block a user
Adapt the auditor (deterministic lens plus System One judge) to the guard contract as the Integrity guard. Measure its judge with its eval against a recorded pass threshold. The judge stays advisory until it passes, and only while it keeps passing.
Done when: the auditor runs as a guard extension and an eval result with its threshold is on record.
Spec: section 5, R23.
Done in archipelago 3e507ed (guards/integrity) and running on staging as guard 'integrity'. The deterministic lens always sets the ceiling (proven violation: escalate; signal: propose and wait). The System One judge is advisory unless guards/integrity/eval_result.json passes its threshold for the same model, with no judge errors and a result under 90 days old. Recorded eval, clef-flash, 34 cases: judged recall 0.73, precision 0.80, false block 0.087 against the threshold recall >= 0.80, precision >= 0.80, false block <= 0.05. It fails, so the judge ships advisory; it fixed and broke 0 decisions against the lens alone. The false-block metric in the copied eval now counts only blocks whose label wasn't block. Staging runs the lens only (judge_base_url unset). Re-measure with go run ./cmd/eval in guards/integrity.