Reviewer runs in a context that provably never saw the implementation #267

Closed
opened 2026-10-27 13:35:38 +00:00 by sam · 0 comments
Owner

A reviewer that shares the implementer's context is a second opinion from the same mind, and it will rationalise the same mistakes -- the failure mode CLAUDE.md's "never self-certified by the implementer" rule exists to prevent.

UNBLOCKED 2026-08-25. The mechanism is decided in docs/prds/adversarial-self-review.md section 1 (rewritten, superseding the version q15y invalidated): the reviewer is an agent.Subagent run from inside the task-scoped archie-agent worker. ai-sdk v0.1.22 is already the pinned version, so no SDK bump. This adds no host executor, no second Pool.Acquire and no nested container lifecycle -- the open question the supersession left.

Two isolations are required and only one is free:

  • CONVERSATION isolation is structural. Subagent.Run (agent/subagent.go:76-96) calls GenerateText with Prompt set and Messages never populated, so the implementer's history has no code path into the nested run. This is the property to assert in the required test.
  • WORKSPACE isolation is ours to build. A sub-agent shares the parent's process, working directory and credentials, so the .git-free snapshot survives from the original design and is load-bearing: git archive <commit> | tar -x into a fresh scratch directory BESIDE the task worktree, plus diff.patch and issue.md. Stripping .git is what makes commit messages, branch name and reflog structurally unreachable rather than merely undisclosed.

Three construction rules:

  1. Subagent.Tools gets a distinct read-only toolset rooted at the snapshot. NEVER the worker's own toolset -- it carries the worktreerpc publication grant, so a reviewer holding it could push the branch it exists to block.
  2. Call Run(), not Tool(). Tool() would let the implementer's own model elect to review itself, which is the self-certification this feature prevents. It also avoids Tool()'s two-step-parent and recursion footguns.
  3. Findings return via a closure-captured capture tool, not the return value -- Run yields only text. Because the sub-agent is in-process, the capture tool's Go func appends to a []workflow.ReviewFinding the stage owns. Read that slice EVEN WHEN Run returns an error: findings already captured survive, so a reviewer that logged three defects then hit its step limit still yields those three.

Acceptance Criteria

The reviewer is launched with only the snapshot, diff.patch and issue.md; a test fails if implementer context reaches it, asserting isolation as a construction fact rather than prompt text; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses agent.Subagent inside the existing worker rather than inventing a second execution path; the reviewer's toolset is read-only, rooted at the snapshot, and provably excludes the worktreerpc publication grant; captured findings are recovered even when Run returns an error.

Acceptance criteria

The reviewer is launched with only diff + issue; a test fails if implementer context reaches it; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses the existing container/worktree isolation rather than inventing a second one.

Notes

BLOCKED 2026-08-24: not implementable as specified. docs/prds/adversarial-self-review.md is marked 'Superseded; requires redesign before implementation' -- superseded by the full-task worker boundary landed in archie-core-q15y. Its mechanism for this bead was a daemon-side second container.Pool.Acquire plus an in-process agentexec path; q15y.2 and q15y.3 removed BOTH call sites, so the isolation mechanism this bead reuses no longer exists. Needs a design decision first: how a fresh reviewer receives a .git-stripped snapshot from INSIDE the existing task-scoped archie-agent worker, without introducing a host executor or a nested container lifecycle. Do not implement against the superseded PRD. The sibling bead h019.2 (findings contract) is unaffected and is being implemented, since it is a pure domain type with no execution dependency.

Close reason

Implemented as agent.Subagent inside the task-scoped archie-agent worker (internal/app/agentworker/review.go). Conversation isolation is structural (Subagent.Run only sends a prompt, never message history -- verified against a fake chat.Provider). Workspace isolation via new worktree.Manager.Snapshot (.git-free export of HEAD's tracked files into a scratch dir beside the task worktree). Reviewer's toolset is read-only (read/grep/find) rooted at the snapshot, never the worker's own toolset -- verified as a construction fact that the toolset contains no write/edit/shell tool and cannot read outside the snapshot dir. Findings captured via a typed record_finding tool appending to a slice the stage owns, surviving a Subagent.Run error when findings were already captured.


Imported from beads archie-core-h019.3 · GitHub #548 · type feature · priority P1 · created 2026-08-21 · closed 2026-08-31

Duplicate record archie-core-1787489596341-12-bf9443bb

Gitea has one issue for both; the other record's text:

This is the whole feature. A reviewer that shares the implementer's context is a second opinion from the same mind, and it will rationalise the same mistakes -- which is exactly the failure mode CLAUDE.md's 'never self-certified by the implementer' rule exists to prevent.

Archie already has the machinery: each task runs in an isolated container against its own worktree. Launch the reviewer the same way, seeded only with the diff and the originating issue. It must not receive the implementer's conversation, its notes, its reasoning, or its own claims about what it did.

Per 'environmental enforcement over prompt rules', isolation must be a property of construction, not a sentence in the prompt. The agent cannot talk its way out of a context it was never given. Assert it in a test rather than assuming it: a test that would fail if implementer context were threaded through.

Close reason

Duplicate; work tracked under canonical epic

A reviewer that shares the implementer's context is a second opinion from the same mind, and it will rationalise the same mistakes -- the failure mode CLAUDE.md's "never self-certified by the implementer" rule exists to prevent. UNBLOCKED 2026-08-25. The mechanism is decided in docs/prds/adversarial-self-review.md section 1 (rewritten, superseding the version q15y invalidated): the reviewer is an `agent.Subagent` run from inside the task-scoped archie-agent worker. ai-sdk v0.1.22 is already the pinned version, so no SDK bump. This adds no host executor, no second Pool.Acquire and no nested container lifecycle -- the open question the supersession left. Two isolations are required and only one is free: - CONVERSATION isolation is structural. `Subagent.Run` (agent/subagent.go:76-96) calls GenerateText with Prompt set and Messages never populated, so the implementer's history has no code path into the nested run. This is the property to assert in the required test. - WORKSPACE isolation is ours to build. A sub-agent shares the parent's process, working directory and credentials, so the `.git`-free snapshot survives from the original design and is load-bearing: `git archive <commit> | tar -x` into a fresh scratch directory BESIDE the task worktree, plus `diff.patch` and `issue.md`. Stripping `.git` is what makes commit messages, branch name and reflog structurally unreachable rather than merely undisclosed. Three construction rules: 1. `Subagent.Tools` gets a distinct read-only toolset rooted at the snapshot. NEVER the worker's own toolset -- it carries the worktreerpc publication grant, so a reviewer holding it could push the branch it exists to block. 2. Call `Run()`, not `Tool()`. `Tool()` would let the implementer's own model elect to review itself, which is the self-certification this feature prevents. It also avoids Tool()'s two-step-parent and recursion footguns. 3. Findings return via a closure-captured capture tool, not the return value -- `Run` yields only text. Because the sub-agent is in-process, the capture tool's Go func appends to a []workflow.ReviewFinding the stage owns. Read that slice EVEN WHEN `Run` returns an error: findings already captured survive, so a reviewer that logged three defects then hit its step limit still yields those three. ## Acceptance Criteria The reviewer is launched with only the snapshot, diff.patch and issue.md; a test fails if implementer context reaches it, asserting isolation as a construction fact rather than prompt text; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses agent.Subagent inside the existing worker rather than inventing a second execution path; the reviewer's toolset is read-only, rooted at the snapshot, and provably excludes the worktreerpc publication grant; captured findings are recovered even when Run returns an error. ### Acceptance criteria The reviewer is launched with only diff + issue; a test fails if implementer context reaches it; the reviewer's prompt instructs it to assume every line is wrong until proven correct; the mechanism reuses the existing container/worktree isolation rather than inventing a second one. ### Notes BLOCKED 2026-08-24: not implementable as specified. docs/prds/adversarial-self-review.md is marked 'Superseded; requires redesign before implementation' -- superseded by the full-task worker boundary landed in archie-core-q15y. Its mechanism for this bead was a daemon-side second container.Pool.Acquire plus an in-process agentexec path; q15y.2 and q15y.3 removed BOTH call sites, so the isolation mechanism this bead reuses no longer exists. Needs a design decision first: how a fresh reviewer receives a .git-stripped snapshot from INSIDE the existing task-scoped archie-agent worker, without introducing a host executor or a nested container lifecycle. Do not implement against the superseded PRD. The sibling bead h019.2 (findings contract) is unaffected and is being implemented, since it is a pure domain type with no execution dependency. ### Close reason Implemented as agent.Subagent inside the task-scoped archie-agent worker (internal/app/agentworker/review.go). Conversation isolation is structural (Subagent.Run only sends a prompt, never message history -- verified against a fake chat.Provider). Workspace isolation via new worktree.Manager.Snapshot (.git-free export of HEAD's tracked files into a scratch dir beside the task worktree). Reviewer's toolset is read-only (read/grep/find) rooted at the snapshot, never the worker's own toolset -- verified as a construction fact that the toolset contains no write/edit/shell tool and cannot read outside the snapshot dir. Findings captured via a typed record_finding tool appending to a slice the stage owns, surviving a Subagent.Run error when findings were already captured. --- _Imported from beads `archie-core-h019.3` · GitHub #548 · type feature · priority P1 · created 2026-08-21 · closed 2026-08-31_ ### Duplicate record `archie-core-1787489596341-12-bf9443bb` Gitea has one issue for both; the other record's text: This is the whole feature. A reviewer that shares the implementer's context is a second opinion from the same mind, and it will rationalise the same mistakes -- which is exactly the failure mode CLAUDE.md's 'never self-certified by the implementer' rule exists to prevent. Archie already has the machinery: each task runs in an isolated container against its own worktree. Launch the reviewer the same way, seeded only with the diff and the originating issue. It must not receive the implementer's conversation, its notes, its reasoning, or its own claims about what it did. Per 'environmental enforcement over prompt rules', isolation must be a property of construction, not a sentence in the prompt. The agent cannot talk its way out of a context it was never given. Assert it in a test rather than assuming it: a test that would fail if implementer context were threaded through. ### Close reason Duplicate; work tracked under canonical epic
sam added the highfeature labels 2026-10-27 13:35:38 +00:00
sam self-assigned this 2026-10-27 13:35:38 +00:00
sam closed this issue 2026-10-27 13:35:38 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#267