From df3d86acc4ea3307143431442ac502d65f8bdcc2 Mon Sep 17 00:00:00 2001 From: Sam Catlow Date: Tue, 25 Aug 2026 00:56:10 +1000 Subject: [PATCH] docs(prds): decide reviewer isolation via agent.Subagent, fail-closed Section 1 of the adversarial self-review decision was superseded on 2026-08-24: it assumed daemon-owned in-process execution and a daemon-side second Pool.Acquire, both removed by archie-core-q15y. It is now re-decided rather than merely marked dead, so the document is Active again. The reviewer is an agent.Subagent run from inside the task-scoped worker. ai-sdk v0.1.22 is already pinned, so this needs no SDK bump, and it adds no host executor, no second container acquire and no nested container lifecycle -- the open question the supersession left. Subagent.Run passes Prompt and never populates Messages, so conversation isolation is a construction fact rather than prompt text. It does NOT give filesystem isolation: the sub-agent shares the parent's process, working directory and credentials, so the .git-stripped snapshot survives from the original design and is load-bearing, and the reviewer's toolset must be read-only, rooted at the snapshot, and must exclude the worktreerpc publication grant -- otherwise a reviewer could push the branch it exists to block. Run() not Tool(), so the implementer cannot elect to review itself. Findings return through a closure-captured tool because Run yields only text, and are read even when Run errors, since captured findings survive. Also decides fail-closed: a review that fails to RUN parks the task rather than opening an unreviewed PR. ReviewEnabled is opt-in, so an operator who turned the gate on wants the gate. This is why h019.2 models "ran and found nothing" separately from "did not run" -- collapsing them would make an outage indistinguishable from a clean review. h019.3 is unblocked and rewritten with the construction rules. h019.4's description said the task parks as waiting_human, contradicting section 4's decision of StatusParked; corrected, and the fail-closed rule added. --- .beads/issues.jsonl | 4 +- docs/prds/adversarial-self-review.md | 140 +++++++++++++++++++-------- 2 files changed, 101 insertions(+), 43 deletions(-) diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 3096c5ac5..0b4138752 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -38,8 +38,8 @@ {"_type":"issue","id":"archie-core-7d5u.2","title":"Reaction delivery via embedded NATS, no external server required","description":"Reactions need at-least-once delivery distinct from the lossy events.Bus (see epic description). The delivery mechanism is NATS -- that part is settled, not a design question. What's open is deployment shape.\n\nVERIFIED 2026-08-22: config.NATSConfig.URL empty currently means 'no NATS, use SQLite ClaimNext' (internal/config/config.go:573-582). That fallback was designed for task distribution working without a broker at all. It was never meant to generalise to 'archie has no NATS' as a permanent architectural stance -- the intended second deployment shape alongside an external NATS server was always NATS EMBEDDED (in-process), not a different eventbus.\n\ngithub.com/nats-io/nats-server/v2 is already a direct dependency and is proven to work in this codebase: internal/infrastructure/agenttransport/nats/transport_test.go and others already start a real embedded server via nats-server/v2/test for tests. It has just never been wired into production startup -- cmd/archied only knows 'external URL configured' vs 'fall back to SQLite'. No new library, no new bus, no architectural gap to design around: this is 'wire the dependency we already have and already test against into the daemon's own startup path.'\n\nDesign questions that remain, now correctly scoped: does an embedded NATS instance replace the SQLite fallback outright for single-process deployments, or run alongside it; what config controls embedded vs external vs disabled; does the daemon own the embedded server's lifecycle the same way it owns everything else (start/stop paired with daemon start/stop, per the shutdown-ordering rules in ARCHITECTURE.md).","acceptance_criteria":"A single-process deployment with no external NATS URL configured still gets reaction delivery via an embedded NATS instance; the daemon starts and stops the embedded server as part of its own lifecycle; no second eventbus type is introduced; internal/infrastructure/eventbus/nats is reused as the transport rather than duplicated.","notes":"2026-08-22, from docs/prds/event-sources-and-reactions.md: do not reuse the existing nats.Client's stream creation as-is. internal/infrastructure/eventbus/nats/client.go:65 hardcodes Retention: jetstream.WorkQueuePolicy -- correct for ARCHIE_TASKS (task distribution, one consumer claims each message) and wrong for reactions, which need fan-out (every registered reaction sees every matching event). Two reactions on overlapping subjects under WorkQueuePolicy hit the exact trap CLAUDE.md documents: the second consumer silently gets nothing. Config.StreamName/Config.Subjects are already composition-supplied (config.go:30-38); Retention is not parameterized and needs to become a Config field, defaulting to today's WorkQueuePolicy so ARCHIE_TASKS is unaffected, with reactions getting a second stream under a fan-out-capable policy and their own subject namespace.","status":"closed","priority":1,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:26:26Z","created_by":"Sam Catlow","updated_at":"2026-08-22T09:25:10Z","started_at":"2026-08-21T18:57:41Z","closed_at":"2026-08-21T19:18:55Z","close_reason":"Embedded NATS wired into production startup: mode=embedded/external/off resolves from url, daemon owns the in-process server lifecycle, Retention parameterized for a future fan-out reaction stream. task check + lint clean; adversarial review zero surviving findings. Committed 2c7db34.","external_ref":"https://github.com/samcharles93/archie-core/issues/545","dependencies":[{"issue_id":"archie-core-7d5u.2","depends_on_id":"archie-core-7d5u","type":"parent-child","created_at":"2026-08-21T23:26:26Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-7d5u.2","depends_on_id":"archie-core-7d5u.1","type":"blocks","created_at":"2026-08-21T23:26:57Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"archie-core-7d5u.3","title":"Shared webhook mechanics: HMAC verification, per-source rate limiting, payload redaction","description":"NARROWED 2026-08-22 from this bead's prior scope, which bundled capture-storage and an approval state machine into a single 'shared mechanics' package. On reflection those are not genuinely shared yet:\n\n- Capture STORAGE is t2db.1's job -- it depends on the NATS-vs-SQLite decision 7d5u.2 owns, and building a storage engine here would either duplicate or contradict that. This package may be USED by t2db.1's capture path, but does not own persistence.\n- The draft/pending_approval/armed state machine belongs to the entity that has that state -- t2db.4's binding -- not a generic package. There is exactly one consumer (bindings) today; per 'extract into a shared primitive on the second real consumer, never in anticipation of one,' building it here now would be speculative.\n\nWhat remains, and is genuinely storage-agnostic and reusable by any inbound HTTP source regardless of what consumes it:\n\n1. HMAC-SHA256 verification -- generalize internal/channels/webhook/webhook.go's existing unexported validateHMAC into an exported function here, and have that package call into it instead of keeping its own copy. One implementation of signature checking, not two that can drift.\n2. Per-source rate limiting -- a token bucket keyed by source ID (not global), so one noisy sender cannot exhaust another's budget. Limit-exceeded is an observable rejection, not a silent drop, per t2db.5's decision.\n3. Payload redaction -- generalize internal/gateway/stream.go's unexported sensitiveParameterKey/redactToolParameters heuristic (matches token/secret/password/passwd/api_key/apikey/authorization/credential/cookie/private_key against normalized JSON keys) into a reusable, storage-agnostic transform any capture path can apply before persisting.","acceptance_criteria":"webhookguard.VerifyHMAC replaces internal/channels/webhook's private copy with no behaviour change (its existing tests still pass); a per-source rate limiter rejects over-budget requests observably; a redaction function matches the same key heuristic already proven in gateway/stream.go; the package has zero imports of any domain, infrastructure or app package.","notes":"--notes=2026-08-22: merged to main as f3d3d32 (PR #534, squash of aa23ac9+46401ab+ce4f945).","status":"closed","priority":1,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:26:26Z","created_by":"Sam Catlow","updated_at":"2026-08-22T09:25:11Z","started_at":"2026-08-21T15:19:09Z","closed_at":"2026-08-21T15:26:01Z","close_reason":"Built internal/webhookguard: VerifyHMAC (behavior-preserving extraction from internal/channels/webhook, which now calls it instead of keeping a private copy), RateLimiter (per-source token bucket, independent budgets, observable rejection), RedactPayload (key-name heuristic matching gateway/stream.go's marker list, but -- deliberately unlike that function -- redacts only values under sensitive keys and leaves ordinary strings/numbers/bools untouched, since a captured payload exists to be inspected for field mapping in t2db.3).\n\nAll four acceptance criteria met and independently confirmed by a fresh adversarial review with zero surviving findings: HMAC extraction is byte-for-byte behavior-preserving (diffed against the deleted function); rate limiter hand-traced correct on two scenarios plus race-clean under -race; redaction's divergence from gateway/stream.go verified by reading both functions side by side; zero non-stdlib imports confirmed by grep. The reviewer also mutation-tested both the redaction and rate-limit guards (disabled each in turn) and confirmed the test suite catches both, then reverted and re-verified go test -race and git status were clean.\n\ntask check exit 0, golangci-lint 0 issues (one forcetypeassert finding was caught and fixed properly during build, not suppressed), gofumpt clean. All 10 pre-existing internal/channels/webhook tests pass unmodified; the one now-redundant unit test was removed with a comment pointing to where its coverage relocated.","external_ref":"https://github.com/samcharles93/archie-core/issues/546","dependencies":[{"issue_id":"archie-core-7d5u.3","depends_on_id":"archie-core-1786637497036-275-8838629e","type":"relates-to","created_at":"2026-08-21T23:27:06Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-7d5u.3","depends_on_id":"archie-core-7d5u","type":"parent-child","created_at":"2026-08-21T23:26:26Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-7d5u.3","depends_on_id":"archie-core-7d5u.1","type":"blocks","created_at":"2026-08-21T23:26:58Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"archie-core-7d5u","title":"[EPIC] Event sources and typed reactions: archie as an extensible platform","description":"Goal: let archie react to things happening -- forge activity, inbound callers, schedules -- and let extensions hook those reactions, without a generic untyped event bus.\n\nTwo findings shape this, both verified in the tree on 2026-08-21.\n\n1. ARCHIE HAS TWO EVENT NEEDS AND ONE BUS. internal/events is documented as 'archied's in-process observability stream' with 'bounded per-subscriber buffers that DROP on overflow, so a stalled dashboard connection can never apply backpressure to the task engine'. Publish is fire-and-forget; subscribers get a read-only channel and cannot influence anything. Correct for the dashboard, wrong for reactions: a hook that silently misses events under load and cannot affect an outcome is worse than no hook, because it looks like it ran. Reactions need at-least-once delivery and the ability to change what happens. Do not bolt reactions onto events.Bus.\n\n2. THE OBVIOUS DESIGN IS PROHIBITED, DELIBERATELY. The tau model this is drawn from dispatches generically: DispatchEvent(ctx, eventName string, sessionID, ...) to out-of-process gRPC plugins. archie's approved foundation (docs/architecture/plugins-and-extensions.md) forbids that shape -- the generic plugin contract stays metadata-only and MUST NOT expose untyped hooks. plugin.Plugin is Name()/Version() only, enforced by internal/plugin/architecture_test.go and by the plugin engine rule in ARCHITECTURE.md. tau already carries the scar the rule avoids: TestUnload_DoesNotWaitForInFlightDispatch exists because a stuck plugin blocked plugin unload.\n\nThe reconciliation, and why this is an epic rather than a feature: separate INBOUND (callers push work in -- forge webhook, chat webhook, cron, channel) from REACTION (extensions respond to something that happened). Inbound becomes a typed source/intake capability family with an owning registry. Reactions become typed capability families owned by the domain whose semantics they affect. 'Webhooks aligned to types of callers' is exactly the typed-family model applied to intake: the instinct is right, the generic bus is what has to go.\n\nNear-term payoff: archie reacts to an issue being labelled the moment it happens, instead of up to poll_interval later.","acceptance_criteria":"A forge webhook triggers work behind a feature flag with polling still available as a backstop; inbound callers register through one typed family rather than bespoke wiring per caller; reactions have delivery guarantees distinct from the lossy observability bus; the generic plugin contract remains metadata-only and internal/plugin/architecture_test.go still passes.","notes":"CORRECTION 2026-08-22 (second): the earlier summary I gave characterized 'NATS is currently optional' as an open architectural gap for this epic to navigate around. That's wrong, and Sam corrected it directly: NATS was never meant to be optional as a technology. What's optional is running a standalone NATS SERVER -- config.NATSConfig.URL empty falls back to the existing SQLite ClaimNext flow (internal/config/config.go:573-582, internal/infrastructure/eventbus/nats/doc.go:24). The intended second deployment shape is NATS EMBEDDED (in-process, no external server to stand up), not 'no NATS, use some other eventbus'. There is no other eventbus in scope, was never meant to be one, and this epic must not introduce one.\n\nCurrent code state, verified: github.com/nats-io/nats-server/v2 (the embeddable NATS server package) is already a direct, non-indirect dependency, and is genuinely used to spin up an in-process server -- but only inside _test.go files (internal/infrastructure/agenttransport/nats/transport_test.go and others use github.com/nats-io/nats-server/v2/test to run tests against a real embedded server). It has never been wired into production as an actual deployment mode. So embedded NATS is proven to work in this codebase's own test harness, and the gap is purely that production startup only knows 'external URL configured' vs 'no NATS, fall back to SQLite' -- there is no production code path that starts nats-server in-process.\n\nThis changes 7d5u.2 (reaction delivery) materially: 'ride the existing NATS eventbus, or a second in-process bus' was the wrong framing. The actual question is whether reaction delivery requires an external NATS deployment or can run on an embedded NATS instance so single-process deployments still get it -- not whether to use NATS at all versus some other bus. If embedded NATS is wired for reaction delivery, capture (t2db.1) gets it too essentially for free, since they were flagged as needing the same underlying decision.","status":"open","priority":1,"issue_type":"epic","owner":"sam@catlow.tech","created_at":"2026-08-21T13:25:57Z","created_by":"Sam Catlow","updated_at":"2026-08-22T09:25:11Z","external_ref":"https://github.com/samcharles93/archie-core/issues/547","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"archie-core-h019.3","title":"Reviewer runs in a context that provably never saw the implementation","description":"This is the whole feature. A reviewer that shares the implementer's context is a second opinion from the same mind, and it will rationalise the same mistakes -- which is exactly the failure mode CLAUDE.md's 'never self-certified by the implementer' rule exists to prevent.\n\nArchie already has the machinery: each task runs in an isolated container against its own worktree. Launch the reviewer the same way, seeded only with the diff and the originating issue. It must not receive the implementer's conversation, its notes, its reasoning, or its own claims about what it did.\n\nPer 'environmental enforcement over prompt rules', isolation must be a property of construction, not a sentence in the prompt. The agent cannot talk its way out of a context it was never given. Assert it in a test rather than assuming it: a test that would fail if implementer context were threaded through.","acceptance_criteria":"The reviewer is launched with only diff + issue; a test fails if implementer context reaches it; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses the existing container/worktree isolation rather than inventing a second one.","notes":"BLOCKED 2026-08-24: not implementable as specified. docs/prds/adversarial-self-review.md is marked 'Superseded; requires redesign before implementation' -- superseded by the full-task worker boundary landed in archie-core-q15y. Its mechanism for this bead was a daemon-side second container.Pool.Acquire plus an in-process agentexec path; q15y.2 and q15y.3 removed BOTH call sites, so the isolation mechanism this bead reuses no longer exists. Needs a design decision first: how a fresh reviewer receives a .git-stripped snapshot from INSIDE the existing task-scoped archie-agent worker, without introducing a host executor or a nested container lifecycle. Do not implement against the superseded PRD. The sibling bead h019.2 (findings contract) is unaffected and is being implemented, since it is a pure domain type with no execution dependency.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","updated_at":"2026-08-24T08:58:06Z","external_ref":"https://github.com/samcharles93/archie-core/issues/548","dependencies":[{"issue_id":"archie-core-h019.3","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.3","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} -{"_type":"issue","id":"archie-core-h019.4","title":"Wire review as an implement-workflow stage: after gates, before PR open","description":"Sequence matters. The review runs after gates pass -- there is no point reviewing code that does not build or test -- and before the PR is opened, because the point is to not spend the operator's attention on a PR that is not ready.\n\nA surviving CONFIRMED finding must not open a PR. It parks the task as waiting_human with the findings attached, which is the existing lifecycle state for 'a human needs to look at this' and needs no new state. Per CLAUDE.md, a finding that survives blocks closing the work; it does not get waved through.\n\nScope to the implement workflow first. Routing to tdd/bootstrap/feasibility is a later decision -- feasibility in particular produces a document rather than a diff, so 'review the diff' may not even apply.","acceptance_criteria":"An implement task with a clean gate and no confirmed findings opens a PR as it does today; one with a surviving confirmed finding parks as waiting_human with findings recorded and opens no PR; the stage is observable in the task's running(workflow:stage) status; retry after operator action re-runs the review.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","updated_at":"2026-08-22T09:25:13Z","external_ref":"https://github.com/samcharles93/archie-core/issues/549","dependencies":[{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.2","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.3","type":"blocks","created_at":"2026-08-21T13:03:26Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":3,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"archie-core-h019.3","title":"Reviewer runs in a context that provably never saw the implementation","description":"A reviewer that shares the implementer's context is a second opinion from the same mind, and it will rationalise the same mistakes -- the failure mode CLAUDE.md's \"never self-certified by the implementer\" rule exists to prevent.\n\nUNBLOCKED 2026-08-25. The mechanism is decided in docs/prds/adversarial-self-review.md section 1 (rewritten, superseding the version q15y invalidated): the reviewer is an `agent.Subagent` run from inside the task-scoped archie-agent worker. ai-sdk v0.1.22 is already the pinned version, so no SDK bump. This adds no host executor, no second Pool.Acquire and no nested container lifecycle -- the open question the supersession left.\n\nTwo isolations are required and only one is free:\n\n- CONVERSATION isolation is structural. `Subagent.Run` (agent/subagent.go:76-96) calls GenerateText with Prompt set and Messages never populated, so the implementer's history has no code path into the nested run. This is the property to assert in the required test.\n- WORKSPACE isolation is ours to build. A sub-agent shares the parent's process, working directory and credentials, so the `.git`-free snapshot survives from the original design and is load-bearing: `git archive \u003ccommit\u003e | tar -x` into a fresh scratch directory BESIDE the task worktree, plus `diff.patch` and `issue.md`. Stripping `.git` is what makes commit messages, branch name and reflog structurally unreachable rather than merely undisclosed.\n\nThree construction rules:\n1. `Subagent.Tools` gets a distinct read-only toolset rooted at the snapshot. NEVER the worker's own toolset -- it carries the worktreerpc publication grant, so a reviewer holding it could push the branch it exists to block.\n2. Call `Run()`, not `Tool()`. `Tool()` would let the implementer's own model elect to review itself, which is the self-certification this feature prevents. It also avoids Tool()'s two-step-parent and recursion footguns.\n3. Findings return via a closure-captured capture tool, not the return value -- `Run` yields only text. Because the sub-agent is in-process, the capture tool's Go func appends to a []workflow.ReviewFinding the stage owns. Read that slice EVEN WHEN `Run` returns an error: findings already captured survive, so a reviewer that logged three defects then hit its step limit still yields those three.\n\n## Acceptance Criteria\n\nThe reviewer is launched with only the snapshot, diff.patch and issue.md; a test fails if implementer context reaches it, asserting isolation as a construction fact rather than prompt text; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses agent.Subagent inside the existing worker rather than inventing a second execution path; the reviewer's toolset is read-only, rooted at the snapshot, and provably excludes the worktreerpc publication grant; captured findings are recovered even when Run returns an error.\n","acceptance_criteria":"The reviewer is launched with only diff + issue; a test fails if implementer context reaches it; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses the existing container/worktree isolation rather than inventing a second one.","notes":"BLOCKED 2026-08-24: not implementable as specified. docs/prds/adversarial-self-review.md is marked 'Superseded; requires redesign before implementation' -- superseded by the full-task worker boundary landed in archie-core-q15y. Its mechanism for this bead was a daemon-side second container.Pool.Acquire plus an in-process agentexec path; q15y.2 and q15y.3 removed BOTH call sites, so the isolation mechanism this bead reuses no longer exists. Needs a design decision first: how a fresh reviewer receives a .git-stripped snapshot from INSIDE the existing task-scoped archie-agent worker, without introducing a host executor or a nested container lifecycle. Do not implement against the superseded PRD. The sibling bead h019.2 (findings contract) is unaffected and is being implemented, since it is a pure domain type with no execution dependency.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","updated_at":"2026-08-24T14:55:25Z","external_ref":"https://github.com/samcharles93/archie-core/issues/548","dependencies":[{"issue_id":"archie-core-h019.3","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.3","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"archie-core-h019.4","title":"Wire review as an implement-workflow stage: after gates, before PR open","description":"Sequence matters. The review runs after gates pass -- there is no point reviewing code that does not build or test -- and before the PR is opened, because the point is to not spend the operator's attention on a PR that is not ready. The slot is between StageDiffCap() and StageOpenPR() in internal/domain/workflow/implement.go.\n\nA surviving CONFIRMED finding must not open a PR. Per CLAUDE.md, a finding that survives blocks closing the work; it does not get waved through.\n\nLIFECYCLE CORRECTION (2026-08-25): this bead previously said \"parks the task as waiting_human\", which names two different states in the actual machine (internal/taskstate/taskstate.go:44-59). docs/prds/adversarial-self-review.md section 4 decided this: use store.StatusParked, following StageDiffCap's existing precedent (steps.go:91-94) -- a stage-computed policy stop with a Detail explaining why, before PR open. Parked's available actions, retry and abandon, are the right shape: fix the flagged code and retry, or abandon/override a false positive. WaitingHuman's approve/reject actions have no referent here because there is no PR yet to approve or reject.\n\nBlocking is workflow.ReviewBlocking (shipped in h019.2): Verdict == confirmed AND Level == error. A PLAUSIBLE finding never blocks, however severe its level claims to be.\n\nFAIL-CLOSED (decided 2026-08-25): a review that fails to RUN is distinct from a review that ran and found nothing, and it must not open a PR. A sub-agent that errors, is truncated, or produces no conclusion yields ReviewStatusNotRun, for which Passed() is false and Blocking() is also false -- the contract deliberately refuses to guess, and this stage resolves it as park. Rationale: Repo.ReviewEnabled is opt-in, so an operator who turned the gate on wants the gate, and a provider outage silently degrading into \"PR opened unreviewed\" is what they were buying protection against. Accepted cost: a transient model error parks a task, and Parked's retry action is the remedy. The park Detail must say the review did not run, never that it passed.\n\nScope to the implement workflow first. Routing to tdd/bootstrap/feasibility is a later decision -- feasibility in particular produces a document rather than a diff, so \"review the diff\" may not even apply.\n\nAlso adds to internal/config: Models[\"reviewer\"] (falling back to Models[\"builder\"], no schema change) and Repo.ReviewEnabled bool (default false).\n\n## Acceptance Criteria\n\nAn implement task with a clean gate and no confirmed findings opens a PR as it does today; one with a surviving confirmed finding parks as store.StatusParked with findings recorded in Detail and opens no PR; a review that fails to run also parks, with a Detail that says so rather than implying a pass; a warn-level or plausible finding does not block; the stage is observable in the task's running(workflow:stage) status and emits task events despite the nested run being non-streaming; retry after operator action re-runs the stage; the stage is inert unless Repo.ReviewEnabled is true.\n","acceptance_criteria":"An implement task with a clean gate and no confirmed findings opens a PR as it does today; one with a surviving confirmed finding parks as waiting_human with findings recorded and opens no PR; the stage is observable in the task's running(workflow:stage) status; retry after operator action re-runs the review.","status":"open","priority":1,"issue_type":"feature","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","updated_at":"2026-08-24T14:55:47Z","external_ref":"https://github.com/samcharles93/archie-core/issues/549","dependencies":[{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:54Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.2","type":"blocks","created_at":"2026-08-21T13:03:25Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.4","depends_on_id":"archie-core-h019.3","type":"blocks","created_at":"2026-08-21T13:03:26Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":3,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"archie-core-h019.1","title":"Design: adversarial self-review stage (one-page decision in docs/prds/)","description":"No settled design exists for this in docs/architecture/, so under the scope-discipline rule merged as PR #533 the design comes first -- briefly and decisively -- and is not finished until a diff follows it.\n\nDecide and record, in one page under docs/prds/: where the reviewer runs (reuse the per-task container pool, or a separate short-lived container); what it is given (diff plus originating issue, and explicitly what it must NOT be given); the findings contract and who owns the type; what a surviving finding does to the task lifecycle; and how the reviewer's model/effort is configured.\n\nConstraints from the repo that the design must respect: dependency direction is cmd -\u003e app -\u003e {domain, infrastructure}, so the reviewer contract belongs to the workflow domain and the execution to infrastructure/app. Agent execution is a data boundary. The model never runs git. Prefer environmental enforcement over prompt rules -- 'the reviewer cannot see the implementation' must be a property of how it is launched, not an instruction in its prompt.\n\nDo not write an architecture document, migration plan or review campaign. One page that decides.","acceptance_criteria":"A one-page decision exists under docs/prds/ answering: reviewer isolation mechanism, inputs and forbidden inputs, findings contract owner, lifecycle effect of a surviving finding, and model/effort configuration. It decides rather than surveys, and cites the packages it will touch.","status":"closed","priority":1,"issue_type":"decision","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:39Z","created_by":"Sam Catlow","updated_at":"2026-08-24T07:12:05Z","started_at":"2026-08-22T09:30:19Z","closed_at":"2026-08-24T07:12:05Z","close_reason":"One-page decision written at docs/prds/adversarial-self-review.md: reviewer runs in a second workspace (container or in-process) pointed at a .git-stripped snapshot of the reviewed commit, never the implementer's worktree; findings contract is workflow.ReviewFinding, owned by internal/domain/workflow, reported via a structured capture tool; a surviving error-level finding sets StatusParked (not WaitingHuman -- epic wording named two different states, this resolves which); model/effort reuse the existing Models[role] map and per-stage budget override, gated by a new opt-in Repo.ReviewEnabled so h019.4 doesn't gate every repo on an uncalibrated reviewer before h019.5 lands. [Closure restored 2026-08-24: originally closed 2026-08-22, lost when the beads DB was rebuilt (.beads.backup-pre-recovery).]","external_ref":"https://github.com/samcharles93/archie-core/issues/550","metadata":null,"dependencies":[{"issue_id":"archie-core-h019.1","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:39Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":0,"dependent_count":5,"comment_count":0} {"_type":"issue","id":"archie-core-h019.2","title":"Findings contract: a structured review result the workflow can act on","description":"The reviewer's output must be data the workflow can branch on, not prose a human has to read to find out whether the PR is safe to open. Prose is what makes a review advisory; structure is what lets a finding block.\n\nDefine the finding type in the workflow domain (the domain declares the contract it needs; it never names who implements it). At minimum a finding carries: file and line, a one-sentence defect, a concrete failure scenario (inputs/state -\u003e wrong output), a verdict distinguishing CONFIRMED from PLAUSIBLE, and a disposition once acted on (fixed / skipped / no_change_needed).\n\nThe CONFIRMED/PLAUSIBLE split is load-bearing and must not be collapsed into a single boolean: only confirmed findings may block, or the feature degrades into crying wolf and the operator learns to skim past it.","acceptance_criteria":"A finding type exists in internal/domain/workflow with file, line, summary, failure scenario, verdict and disposition; it is serialisable across the agent-execution data boundary; a review result with zero findings is representable and distinct from a review that did not run.","status":"closed","priority":1,"issue_type":"feature","assignee":"Sam Catlow","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:39Z","created_by":"Sam Catlow","updated_at":"2026-08-24T09:15:56Z","started_at":"2026-08-24T08:57:20Z","closed_at":"2026-08-24T09:15:56Z","close_reason":"Implemented: workflow.ReviewFinding + ReviewReport in internal/domain/workflow/review_findings.go, merged 53bb841. The domain owns the type; gate.Finding was used as a shape precedent only, not reused, so the review contract is not coupled to the Yaegi gate's evolution. Blocking requires Verdict==confirmed AND Level==error, so a PLAUSIBLE finding never blocks however severe it claims to be. ReviewStatus makes 'ran and found nothing' structurally distinct from 'never ran' -- Passed() is false for a zero-value report, and the distinction does not rest on nil-vs-empty-slice. JSON round-trip covered; Line 0 is preserved (no omitempty) for whole-file findings. Review found and fixed a duplicate package doc comment and a redundant ReviewBlocking/BlockingReviewFindings alias pair. task check green. Fresh adversarial review clean across 8 dimensions; I independently mutation-tested the blocking rule: dropping the verdict check fails 5 tests (incl. plausible_error_MUST_NOT_block) and dropping the level check fails 5 more, so the tests genuinely pin both halves.","external_ref":"https://github.com/samcharles93/archie-core/issues/551","dependencies":[{"issue_id":"archie-core-h019.2","depends_on_id":"archie-core-h019","type":"parent-child","created_at":"2026-08-21T13:02:39Z","created_by":"Sam Catlow","metadata":"{}"},{"issue_id":"archie-core-h019.2","depends_on_id":"archie-core-h019.1","type":"blocks","created_at":"2026-08-21T13:03:24Z","created_by":"Sam Catlow","metadata":"{}"}],"dependency_count":1,"dependent_count":3,"comment_count":0} {"_type":"issue","id":"archie-core-h019","title":"[EPIC] Adversarial self-review before archie opens a PR","description":"Archie enforces gates on every PR it opens, but it does not run the adversarial review CLAUDE.md makes mandatory for every change in this repo: a fresh reviewer that did not write the code, told to assume every line is wrong, working a defined checklist, where a surviving finding blocks the work. It is the one discipline the project requires of its contributors and exempts itself from.\n\nWhy it matters, with evidence rather than assertion: on 2026-08-20/21 the update-verification change (PR #530) passed task check at every checkpoint -- exit 0, golangci-lint 0 issues, gofumpt clean -- and three adversarial passes still found six real defects, two of them blocking. A 'dev' version placeholder would have reported healthy updates as failures, and an 'unknown' install claim was still being worded as confirmed success. The gates were green throughout and would have stayed green. This is CLAUDE.md's own recorded lesson: tests catch behaviour that breaks, not behaviour that quietly starts meaning something else.\n\nArchie is unusually well placed to do this. It already runs each task in an isolated container and worktree, so it can spawn a reviewer that genuinely never saw the implementation -- not a second opinion from the same context pretending to be fresh, which is the failure mode of every single-context tool. The isolation boundary built for safety is what makes honest self-review possible.\n\nWhat the operator gets: a PR that arrives with a findings section -- what was checked, what was found, what was fixed, what survived. A surviving finding parks the task as waiting_human instead of opening a PR that looks ready and is not. Review shifts from hunting for bugs to adjudicating findings.\n\nThe risk that kills the feature is a reviewer that cries wolf. Findings must be adversarially confirmed before they surface, and the reviewer must be able to report zero findings on properties it checked and cleared. Calibration is the feature, not the finding count.","acceptance_criteria":"An implement-workflow task runs an adversarial review after gates pass and before the PR is opened; the reviewer provably cannot see the implementer's context; a surviving confirmed finding blocks PR creation and parks the task; findings and their disposition are visible on the PR and the dashboard.","status":"open","priority":1,"issue_type":"epic","owner":"sam@catlow.tech","created_at":"2026-08-21T13:02:17Z","created_by":"Sam Catlow","updated_at":"2026-08-22T09:25:15Z","external_ref":"https://github.com/samcharles93/archie-core/issues/552","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/docs/prds/adversarial-self-review.md b/docs/prds/adversarial-self-review.md index 87aafb1c8..58f36111f 100644 --- a/docs/prds/adversarial-self-review.md +++ b/docs/prds/adversarial-self-review.md @@ -1,24 +1,24 @@ # Adversarial self-review stage -- decision -**Status:** Superseded; requires redesign before implementation -**Date:** 2026-08-22 +**Status:** Active +**Date:** 2026-08-22, section 1 replaced 2026-08-25 **Beads issue:** `archie-core-h019.1`, blocks `h019.2/.3/.4/.5` -> **Superseded 2026-08-24 by the full-task worker boundary.** This document is -> retained as the record of the earlier decision, not as an implementation -> contract. It assumes daemon-owned in-process execution and a daemon-side -> second `Pool.Acquire`; both call sites were removed by `archie-core-q15y`. -> The still-useful findings and lifecycle decisions below do not authorize an -> implementation. The feature must first decide how a fresh reviewer receives -> an isolated snapshot from inside the existing task-scoped worker without -> creating a host executor or a nested container lifecycle. +> **Section 1 was superseded on 2026-08-24 and re-decided on 2026-08-25.** +> The original execution mechanism assumed daemon-owned in-process execution +> and a daemon-side second `Pool.Acquire`; `archie-core-q15y` removed both +> call sites, leaving one open question: how a fresh reviewer receives an +> isolated snapshot from inside the task-scoped worker, without a host +> executor or a nested container lifecycle. Section 1 below now answers it +> with `agent.Subagent`. Sections 2-5 were never invalidated and still hold; +> section 3's contract shipped as `archie-core-h019.2`. Answers the five questions in `h019.1`'s description against the actual implement workflow, not designed from scratch: the gate pattern, the container pool, and the workflow engine's `Outcome` mechanism already answer most of this. -## 1. Where it runs: a second workspace, not a second worktree +## 1. Where it runs: `agent.Subagent`, inside the task worker The implement workflow's stage list (`internal/domain/workflow/implement.go:110-185`) already has the right slot: @@ -26,30 +26,57 @@ after `StageDiffCap()` (line 178), before `StageOpenPR()` (line 179) -- gates have passed, the diff is final, and nothing has told the forge about this change yet. -**The isolating property is the workspace path, not container-vs-in-process.** -Both execution modes already scope an agent's tools to a directory -- -`agentexec.Runner.Run(ctx, workspace string, ...)` -(`internal/agentexec/inprocess.go:20-26`) roots tool access there in-process, -and container mode mounts only what `Pool.Acquire` -(`internal/container/pool.go:161-177`) is given. So "the reviewer cannot see -the implementation" becomes an environmental fact by construction, in both -deployment shapes, as long as the reviewer's workspace is never -`tc.WorktreeDir`. +**The reviewer is an `agent.Subagent` run, not a second container.** ai-sdk +`v0.1.22` (already the pinned version) added `agent/subagent.go`: a nested, +synchronous `core.GenerateText` call in the same process, with its own model, +system prompt, toolset and step budget. That is what makes this feature +buildable at all after `q15y` -- the reviewer runs inside the task-scoped +`archie-agent` worker that is already executing the workflow, so there is no +host executor, no second `Pool.Acquire`, and no nested container lifecycle. -**What the workspace is:** before running the reviewer, the app-layer code -that owns this stage exports a `.git`-free snapshot of the reviewed commit -(`git archive | tar -x`) into a fresh scratch directory next to, not -inside, the task's real worktree, then writes two files alongside it: -`diff.patch` (the unified diff already computed for `StageYaegiGate`'s -`Trees.Diff()`) and `issue.md` (the originating issue's title and body, read -from `store.Task`, nothing else). The reviewer's `agentexec.Request.Workspace` -points at this directory. Stripping `.git` is deliberate: it is what makes -commit messages, branch name, and reflog -- the actual channel a multi-commit -implementer's back-and-forth reasoning would leak through -- structurally -unreachable rather than merely undisclosed. Container mode acquires a second, -distinct container for this (`Pool.Acquire` again, different mounts); in-process -mode builds a second `agentexec.Request` with this workspace. Neither needs a -new execution mechanism. +**Two isolations are needed, and Subagent only provides one of them.** + +*Conversation isolation is structural, and free.* `Subagent.Run` +(`agent/subagent.go:76-96`) calls `GenerateText` with `Prompt` set and +`Messages` never populated. The implementer's history has no code path into +the nested run; only the prompt string crosses, and only text comes back. +This is the literal form of "environmental enforcement over prompt rules": +the reviewer is not told to ignore the implementer's reasoning, because +nothing hands it any. `h019.3`'s required test asserts this as a construction +fact. + +*Workspace isolation is still ours to build.* A sub-agent shares the parent's +process, working directory and credentials. So the `.git`-free snapshot from +the original decision survives unchanged and is load-bearing: export the +reviewed commit (`git archive | tar -x`) into a fresh scratch +directory beside, not inside, the task worktree, and write `diff.patch` (the +diff already computed for `StageYaegiGate`'s `Trees.Diff()`) and `issue.md` +(the originating issue's title and body, nothing else) alongside it. Stripping +`.git` is what makes commit messages, branch name and reflog -- the actual +channel a multi-commit implementer's reasoning leaks through -- structurally +unreachable rather than merely undisclosed. + +`Subagent.Tools` therefore takes a **distinct read-only toolset rooted at the +snapshot**. Never the worker's own toolset: it carries the `worktreerpc` +publication grant, so a reviewer holding it could push the branch it exists to +block. + +**Call `Run()`, not `Tool()`.** `Tool()` would register delegation as a tool +the *implementer's* model may choose to invoke -- self-certification by the +implementer, which is the exact failure this feature exists to prevent. The +stage calls `Run()` unconditionally. That also sidesteps both of `Tool()`'s +documented footguns: the parent's two-step minimum and the recursion risk of +passing a toolset back to itself. + +**Findings return through a closure, not the return value.** `Run` yields only +final text, so section 3's structured contract cannot ride the return. Because +the sub-agent is in-process, the capture tool's Go implementation appends +directly to a `[]workflow.ReviewFinding` the stage owns. This preserves the +"cannot end its turn without calling a typed tool" property, and it defeats +`Subagent`'s documented "no partial results on failure" limitation: findings +already captured survive an error return, so **the stage reads the captured +slice even when `Run` returns an error**. A reviewer that logged three defects +and then hit its step limit still yields those three. ## 2. What it's given, and what it must never be given @@ -122,6 +149,23 @@ returns `nil`, so `StageOpenPR` runs next. Attaching warn-level findings to the task record for PR-body/dashboard display is `h019.6`'s job, not decided here. +**A review that fails to run is fail-closed** (decided 2026-08-25). The +blocking rule is `Verdict == confirmed && Level == error`, so only a confirmed +finding blocks -- but that governs findings, not the reviewer's own failure. A +sub-agent that errors, is truncated, or produces no conclusion yields +`ReviewStatusNotRun`, for which `ReviewReport.Passed()` is false and +`Blocking()` is also false; the contract deliberately refuses to guess. This +stage resolves that as **park, do not open the PR**. The reasoning is that +`Repo.ReviewEnabled` is opt-in: an operator who turned the gate on wants the +gate, and a provider outage silently degrading into "PR opened unreviewed" is +the failure they were buying protection against. The cost is accepted -- a +transient model error parks a task, and `Parked`'s retry action is the +remedy. `Detail` must say the review did not run, never that it passed. + +Note this is why `h019.2` models "ran and found nothing" separately from "did +not run": collapsing them to an empty slice would make an outage +indistinguishable from a clean review, and fail-closed unimplementable. + ## 5. Model, effort, and whether a repo even runs this yet **Model:** a new `"reviewer"` key in the existing `Config.Models` map (already @@ -135,7 +179,16 @@ that works," not a mandated model-diversity policy. **Effort/budget:** reuse the per-stage budget override `AgentStage` already supports (`agent.go:35`), not a new config surface -- a repo wanting a cheaper or more thorough reviewer sets `MaxSteps` the same way -the plan stage already can. +the plan stage already can. Map that budget onto `Subagent.MaxSteps` and +`Subagent.MaxTokens` deliberately: **`MaxTokens` is a per-call ceiling, not a +cumulative budget**, so an N-step reviewer can emit up to N x `MaxTokens`. +Setting it as though it bounded the whole run is a costing error, not a +safety one. `Subagent.MaxSteps` defaults to 10 when unset. + +**Observability:** the nested run is synchronous and non-streaming -- nothing +is emitted between the parent's call and its result, so a long review reads as +a stalled stage. Emit task events around the call so the dashboard shows the +reviewer working rather than a gap. **Opt-in, not automatic:** `h019.5` (calibration -- confirm findings adversarially before they surface) is a separate, lower-priority bead than @@ -150,14 +203,19 @@ setting. ## Packages this touches - `internal/domain/workflow` (`h019.2`, `h019.4`): `ReviewFinding`, - `ReviewBlocking`, and a `ReviewStage` inserted between `StageDiffCap` and - `StageOpenPR` in `implement.go`. Contract only; no execution mechanism. + `ReviewReport`, `ReviewBlocking` -- shipped -- and a `ReviewStage` inserted + between `StageDiffCap` and `StageOpenPR` in `implement.go`. The domain owns + the contract and the pure blocking decision, and declares the reviewer + interface it needs; it never names `agent.Subagent`. - `internal/worktree` or a sibling package (`h019.3`): the clean-snapshot export (`git archive`, no `.git`) building the reviewer's isolated - workspace. -- `internal/agentexec` / `internal/container` (`h019.3`): no new mechanism -- - a second `Request`/`Pool.Acquire` scoped to the snapshot directory reuses - the existing `Runner` interface. + workspace. Still required -- `Subagent` isolates conversation, not + filesystem. +- `internal/app/agentworker` (`h019.3`): constructs the `agent.Subagent` from + the worker's own resolved provider (`agentexec.NewRuntime`), with a + read-only toolset rooted at the snapshot and the findings capture tool. This + is app-layer wiring: the domain gets an implementation of its reviewer + interface, not an SDK type. - `internal/config` (`h019.4`): `Models["reviewer"]` (no type change) and `Repo.ReviewEnabled bool`. - `internal/store`/`internal/taskstate`: unchanged; reuses `StatusParked`.