Policy and Confidence guards, advisory until measured #2006

Open
opened 2026-11-14 16:02:45 +00:00 by sam · 1 comment
Owner

Model-based guards: Policy reads plain-language policy; Confidence checks the agent's evidence for its diagnosis. Both answer with levels and reasons, and stay advisory (recorded, no ceiling) until their judge passes a measured evaluation against a recorded threshold.

Done when: both guards' verdicts appear in the decision trail as advisory.

Spec: section 5 rule 5, R37.

Model-based guards: Policy reads plain-language policy; Confidence checks the agent's evidence for its diagnosis. Both answer with levels and reasons, and stay advisory (recorded, no ceiling) until their judge passes a measured evaluation against a recorded threshold. Done when: both guards' verdicts appear in the decision trail as advisory. Spec: section 5 rule 5, R37.
sam added the featuremedium labels 2026-11-14 16:02:45 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
sam added a new dependency 2026-11-14 16:02:46 +00:00
Author
Owner

Built: archipelago guards/policy and guards/confidence (bd087e3) with shared sdk/judgegate and sdk/systemone; core 88afd1ebc gives guards the agent's diagnosis and the evidence it read (guard.v1 JudgeRequest diagnosis and evidence, from the run journal), b6f705921 adds a multi-line text setting so the policy text is edited on the guard's settings page. Both are advisory until their recorded eval passes. clef-flash, threshold recall >= 0.8, precision >= 0.8, false flag <= 0.05: policy 35 cases recall 0.92, precision 0.85, false flag 0.09 (fails on false flags); confidence 29 cases recall 0.73, precision 0.80, false flag 0.11 (fails). Not yet installed on staging, so the done-when (verdicts in the trail as advisory) isn't shown: the guards need a judge endpoint the staging host can reach.

Built: archipelago guards/policy and guards/confidence (bd087e3) with shared sdk/judgegate and sdk/systemone; core 88afd1ebc gives guards the agent's diagnosis and the evidence it read (guard.v1 JudgeRequest diagnosis and evidence, from the run journal), b6f705921 adds a multi-line text setting so the policy text is edited on the guard's settings page. Both are advisory until their recorded eval passes. clef-flash, threshold recall >= 0.8, precision >= 0.8, false flag <= 0.05: policy 35 cases recall 0.92, precision 0.85, false flag 0.09 (fails on false flags); confidence 29 cases recall 0.73, precision 0.80, false flag 0.11 (fails). Not yet installed on staging, so the done-when (verdicts in the trail as advisory) isn't shown: the guards need a judge endpoint the staging host can reach.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#2006