diff --git a/docs/data/generated/dev-docs.json b/docs/data/generated/dev-docs.json index c8d7b5000..27d7868d9 100644 --- a/docs/data/generated/dev-docs.json +++ b/docs/data/generated/dev-docs.json @@ -312,6 +312,12 @@ "section": "PRDs", "body": "# Event sources and reactions -- decision\n\n**Status:** Approved\n**Date:** 2026-08-22\n**Beads issue:** `archie-core-7d5u.1`, blocks `7d5u.2/.3/.4/.5`\n\nAnswers the four questions in `7d5u.1`'s description. Investigated against\nthe actual tree, not designed from scratch: three mechanisms already in this\ncodebase answer most of this, and the job here is picking the right one for\neach question and saying which packages change.\n\n## 1. In-process typed families, not out-of-process dispatch\n\nArchie has no out-of-process extension mechanism. The plugin engine\nrule (`docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`,\nmechanically checked by\n`internal/plugin/architecture_test.go`) keeps every capability family\nin-process behind a narrow typed registrar; NATS RPC exists for\ndaemon\u003c-\u003eagentexec, a different actor with a different trust boundary (task\nexecution, not third-party extension code), not a precedent for generic\nplugin dispatch. Neither epic's near-term payoff -- react to a labelled\nissue, bind a captured webhook to a workflow -- needs extension code in\nanother language or process. Building out-of-process dispatch now would be\ndesigning for a requirement neither epic states.\n\n`docs/architecture/plugins-and-extensions.md` does **not** need amending.\nThe line that looked like a blanket ban on hooks (`archie-core-7d5u`'s notes\nrecord the correction) is scoped to Workflow semantics specifically, and\nnothing decided here touches Workflow's ownership of its own contracts.\n\n## 2. Reaction delivery: `internal/eventbus`'s contract, a NEW stream\n\n`internal/eventbus.Publisher`/`Consumer` already has everything a reaction\nneeds: `PublishUnique(subject, idempotencyKey, payload)` for at-least-once\nwithout duplicate enqueue, `Ack`/`Nak` with \"a handler returning an error\ncauses redelivery.\" Do not design new delivery semantics; use this contract.\n\n**Do not reuse the existing `nats.Client` as-is.** `Config.StreamName` and\n`Config.Subjects` are already composition-supplied (\"the bus must not know\nwhich subjects belong to which domain,\" `config.go`), but the\nretention policy is not parameterised: `New` hardcodes\n`Retention: jetstream.WorkQueuePolicy` (`client.go`), the competing-consumer\npolicy correct for `ARCHIE_TASKS` task distribution -- one message, claimed by\nexactly one consumer -- and wrong for reactions. Two independent reactions\nboth wanting to see the same event under `WorkQueuePolicy` hit exactly the\ntrap `CLAUDE.md` already documents: overlapping filter subjects on a\nwork-queue stream mean the second consumer silently gets nothing. Reactions\nneed fan-out, not claim-once, so the client needs retention made a `Config`\nfield (defaulting to `WorkQueuePolicy` so `ARCHIE_TASKS` is\nunaffected), and a second stream (`ARCHIE_REACTIONS` or similar) configured\nunder a fan-out-capable policy, alongside the existing task-distribution\nstream -- not instead of it.\n\nThis is where `7d5u.2`'s embedded-vs-external axis attaches: whichever NATS\ninstance backs `ARCHIE_TASKS` also backs the reaction stream. NATS is not\noptional as a technology -- the choice is only\nwhether it runs as an external server or embedded in-process\n(`github.com/nats-io/nats-server/v2` is already a direct, test-proven\ndependency, just never wired into production startup). `7d5u.2` wires that;\nthis document does not re-decide it.\n\n## 3. Reactions in these two epics are producer-only\n\nNo veto, no mutation of in-flight work. Two existing shapes cover everything\neither epic actually needs, and neither is a hook that can block something\nelse:\n\n- **Cheap wake, no payload.** `internal/domain/curator/wake.go`'s\n `WakeOnPrimaryInput` subscribes to the lossy `events.Bus` and nudges a\n runtime to check sooner; a dropped event only delays a check-in, never\n loses work, because the trigger decision is a deterministic state read at\n pass time, not the event payload. Model any \"check something sooner\"\n reaction on this, unchanged.\n- **Produce new work, real payload.** `daemon.go`'s existing `pollNATS` -\u003e\n `publishTask` path is already exactly a reaction that observes forge state\n and produces a `workintake.TaskEnvelope`. A webhook source is a second\n producer into the same path, not a new capability -- see question 4.\n\nA reaction that can veto or mutate an _in-flight_ task (block a PR from\nopening, alter a running stage) is a different, harder problem with its own\nordering and failure semantics, and belongs to `archie-core-h019` (adversarial\nself-review) if and when that epic needs one -- not here. Do not build a\ngeneral block-capable hook mechanism for either epic; nothing in their stated\nscope requires it, and it would duplicate whatever h019 eventually needs.\n\n## 4. No single generic intake interface; each domain keeps its own\n\nA `Source` interface generic enough to cover a forge issue, a chat message,\nand a cron tick would need an `any` payload -- the untyped-hook shape the\nplugin engine rule exists to prevent, just moved into a new package instead\nof the old one.\n\n**What's genuinely shared, and belongs in a small cross-cutting package**\n(imports nothing per the cross-cutting rule): the mechanics from\n`archie-core-t2db.5` that apply to any inbound HTTP source regardless of what\nconsumes it -- HMAC verification (already a standalone function,\n`internal/channels/webhook/webhook.go`'s `validateHMAC`), per-source rate\nlimiting, capture/audit, and the draft/pending_approval/armed gate. This\npackage owns none of forge, chat, or workintake's vocabulary.\n\n**What stays domain-owned, per existing convention:**\n\n- Forge-issue intake needs no new type. `internal/domain/workintake.TaskEnvelope`\n already is the typed contract -- `Subject()`, `IdempotencyKey()` (keyed on\n `owner/repo/number`, not on delivery source), `Encode()`. A webhook-sourced\n envelope for the same issue produces the _same_ idempotency key as a\n poll-discovered one, so `PublishUnique` dedups the two paths for free. This\n is `7d5u.5`'s answer, not a new mechanism: reuse `TaskEnvelope` and\n `publishTask`, don't invent a second envelope shape.\n- Chat-webhook intake stays exactly what it is:\n `internal/channels/webhook`'s `RouteConfig` routes into `gateway.Router`,\n documented as chat-scoped (\"External services push messages into\n archie-core's gateway for LLM processing\"). Do not generalise this package\n into a multi-domain receiver -- it already does one thing per \"a package\n owns its own format end to end.\"\n- A forge webhook receiver (`7d5u.4`) is new: it builds on the shared\n mechanics above, decodes a GitHub payload into the _same_\n `workintake.TaskEnvelope`, and calls the _same_ `publishTask` the poller\n calls. It does not get its own dispatch predicate -- `7d5u.4`'s bead\n already requires reusing the poller's label/assignee predicate rather than\n reimplementing it, and this decision extends that to the envelope and\n publish path too.\n\n## Packages this touches\n\n- New: a cross-cutting intake-mechanics package (HMAC, rate limit, capture,\n approval state machine) -- name and exact location decided in `t2db.1`/`t2db.5`.\n- `internal/infrastructure/eventbus/nats`: a second stream/consumer\n configuration for reactions, distinct retention policy from `ARCHIE_TASKS`.\n- `internal/domain/workintake`: unchanged in shape; `TaskEnvelope` is reused,\n not extended.\n- `internal/daemon`: gains a webhook-sourced call path into the existing\n `publishTask`, alongside the existing poll-sourced one.\n- `internal/channels/webhook`: unchanged. Not the forge receiver.\n" }, + { + "url": "/dev/prds/execution-tree-state-machine/", + "title": "Execution tree and enforced state machine", + "section": "PRDs", + "body": "# Execution tree and enforced state machine\n\n**Status:** Draft\n**Authority:** `docs/architecture/agent-system.md` (canonical workflow model);\nsettles the \"enforced WorkflowExecution state machine\" item in\n`docs/architecture/migration-decisions.md` section 4.\n**Compounds with:** `docs/prds/task-run-detail.md` (attempt attribution),\n`docs/prds/state-store-contract.md` (wire contract and task-scoped grants).\n\n## Decision\n\nA WorkflowExecution becomes the root of a tree of StepExecutions. Every stage\nrun, and every agent call a stage makes, is its own durable record with a\nparent, an attempt, a status and an outcome. Both levels move only through a\ndeclared transition table, enforced by the State Store in the same transaction\nthat writes the audit row and the domain event.\n\n## Problem\n\n- A run is one `tasks` row. The only per-step record is the mutable `stage`\n column, overwritten as `workflow.Run` advances, plus observability events.\n A failed attempt cannot say which step failed or what its children did\n without replaying the event stream.\n- `Store.Transition` guards on the expected `from` status, but any `from` to\n any `to` is accepted. A `queued` to `merged` write succeeds.\n- Stop cancels one context in `daemon.runningTasks`. Nothing persisted says\n which work was in flight, so an agent call that outlives its stage (a\n reviewer, a container) is invisible to the store.\n- `workflow.Run` writes `tc.Store.Update` and discards the error, then emits\n `KindStageStart` separately. State and events can diverge.\n\n## Model\n\n```text\nWorkflowExecution (tasks row, id E)\n└── attempt N\n ├── StepExecution stage \"plan\" depth 0\n ├── StepExecution stage \"implement\" depth 0\n │ └── StepExecution agent \"implement\" depth 1\n └── StepExecution stage \"review\" depth 0\n ├── StepExecution agent \"review:1\" depth 1\n └── StepExecution agent \"review:2\" depth 1\n```\n\n- A StepExecution belongs to exactly one WorkflowExecution and one attempt.\n `parent_id` is null for a stage and names the enclosing StepExecution\n otherwise. Depth is derived from the parent, never supplied by the caller.\n- `kind` is `stage` or `agent`. A new kind needs a named consumer before it is\n added.\n- A retry starts a new attempt. Earlier attempts' StepExecutions are never\n rewritten.\n\n## Transition tables\n\nWorkflowExecution statuses are the existing `taskstate` constants, persisted\nstrings unchanged. `taskstate` gains the one table every writer checks:\n\n| From | To |\n| --------------- | ---------------------------------------------------------- |\n| `queued` | `running`, `closed_wont_do` |\n| `running` | `queued` (crash recovery), `waiting_human`, `pr_open`, `completed`, `parked`, `closed_wont_do` |\n| `waiting_human` | `queued` (approve), `closed_wont_do` |\n| `parked` | `queued` (retry), `dead` (abandon), `closed_wont_do` |\n| `pr_open` | `merged`, `rejected`, `queued` (remediation), `closed_wont_do` |\n| terminal | none |\n\nStepExecution statuses:\n\n| From | To |\n| --------- | ------------------------------------------------------ |\n| `pending` | `running`, `cancelled` |\n| `running` | `succeeded`, `failed`, `cancelled`, `interrupted` |\n| terminal | none |\n\n`interrupted` means the process running the step died or shut down. It is\ndistinct from `cancelled`, which an operator or a parent asked for, and from\n`failed`, which the step reported.\n\nA StepExecution cannot reach `running` while its WorkflowExecution is not\n`running`, and cannot start under a terminal parent.\n\n## Enforcement\n\n- The table lives in `taskstate` (no dependencies), so the store, daemon and\n gateway share one copy.\n- The State Store rejects an illegal transition with a new sentinel,\n `ErrIllegalTransition`, carried across the wire through `mapError` like\n `ErrStaleTransition`. A stale `from` stays `ErrStaleTransition`.\n- Each transition is one transaction: guarded row update, audit row, domain\n event row. The events table is written by the store, not emitted beside it.\n `KindStageStart` and `KindStageFinish` are produced by the step transition.\n- `workflow.Run` stops discarding persistence errors. A failed step write\n parks the execution; the step does not run unrecorded.\n\n## Cancellation\n\nStopping a run is one store call, `CancelExecution(E, reason)`. In one\ntransaction it moves every non-terminal StepExecution of the current attempt\nto `cancelled` and the WorkflowExecution to `closed_wont_do` or `parked`,\nwhichever the operator action names. The daemon then cancels the in-memory\ncontext. A worker whose step write returns `ErrStaleTransition` because its\nstep is already `cancelled` stops without writing further.\n\nThe store is the record of cancellation. The context cancel is the delivery\nmechanism, so a lost cancel still leaves the store correct and the worker's\nnext write fails.\n\n## Crash recovery\n\n`RecoverStale` moves every `running` StepExecution of an interrupted\nexecution to `interrupted` in the same transaction that requeues the\nexecution. The next attempt starts with fresh steps.\n\n## Wire contract\n\nNew State Store RPCs, each with a matching Go method on a narrow consumer\nfacade:\n\n- `StartStep(execution, attempt, parent, kind, name) -\u003e step id`\n- `FinishStep(step, from, to, detail, tokens_used)`\n- `CancelExecution(execution, reason, to)`\n- `ListSteps(execution, attempt)`\n\n`TaskGrants.UnaryInterceptor` authorises a task-scoped token for `StartStep`\nand `FinishStep` on its own execution only. The container records its own\nagent calls. `CancelExecution` and `ListSteps` require the administrative\ntoken.\n\n## Storage\n\nOne new table, `step_executions`: `id`, `execution_id`, `attempt`,\n`parent_id`, `depth`, `kind`, `name`, `status`, `detail`, `tokens_used`,\n`started_at`, `finished_at`. Indexed on `(execution_id, attempt, id)`.\nExisting rows gain no backfilled steps. Their history stays in `transitions`\nand `events`.\n\n`tasks.stage` is written from the step transition while any reader still uses\nit, then dropped once the dashboard reads `ListSteps`.\n\n## Out of scope\n\n- Running sibling steps in parallel. The tree records concurrency; changing\n the stage runner is separate work.\n- Cost in currency. `tokens_used` per step is the only usage field.\n- Workflow definition versioning and the `tasks` to `workflow_executions`\n rename.\n- A harness contract for swappable coding agents.\n\n## Plan\n\n1. `taskstate` transition tables with table-driven tests covering every\n legal and a sample of illegal pairs.\n2. `ErrIllegalTransition` enforced in the Postgres store and mapped over the\n wire.\n3. `step_executions` migration, `StartStep`/`FinishStep`, grant rules;\n `workflow.Run` records stages; events written in the transition\n transaction.\n4. Agent-call children recorded from the agent runtime.\n5. `CancelExecution`; daemon stop and chat/dashboard actions route through it.\n6. `RecoverStale` interrupts running steps.\n7. Dashboard run detail reads `ListSteps`; `tasks.stage` dropped.\n\n## Verification\n\n- An illegal WorkflowExecution or StepExecution transition returns\n `ErrIllegalTransition` over gRPC, and `errors.Is` matches on the client.\n- Killing the daemon mid-stage, then restarting, leaves that stage\n `interrupted` and the execution `queued` with a new attempt.\n- Stopping a run during review leaves every review child `cancelled`, and a\n late `FinishStep` from the container returns `ErrStaleTransition`.\n- A task-scoped token cannot `StartStep` on another execution.\n- For every step transition there is exactly one event row, written in the\n same transaction.\n" + }, { "url": "/dev/prds/extensions-center/", "title": "Extensions centre", @@ -727,6 +733,10 @@ "title": "Event sources and reactions -- decision", "url": "/dev/prds/event-sources-and-reactions/" }, + { + "title": "Execution tree and enforced state machine", + "url": "/dev/prds/execution-tree-state-machine/" + }, { "title": "Extensions centre", "url": "/dev/prds/extensions-center/" diff --git a/docs/prds/external-agent-harness.md b/docs/prds/external-agent-harness.md new file mode 100644 index 000000000..ec6ce3c2d --- /dev/null +++ b/docs/prds/external-agent-harness.md @@ -0,0 +1,168 @@ +# External agent harnesses + +**Status:** Draft +**Authority:** `docs/architecture/agent-system.md` (agent execution as a data +boundary); settles the harness non-goal deferred by +`docs/prds/service-decomposition.md`. +**Compounds with:** `docs/prds/execution-tree-state-machine.md` (a harness +run is one `agent` StepExecution). + +## Decision + +A workflow stage can run on an external coding-agent CLI instead of the +built-in agent loop. Supported harnesses are Claude Code, Codex, Pi, OMP and +GitHub Copilot CLI. Each authenticates with either an API key or the +operator's own subscription login, and can run with the operator's own +skills, plugins, subagents and instructions. + +Archie's guarantees do not move into the harness. Git, protected paths, +read-only stages, gates and budgets are enforced by archie around the +harness process and checked against the worktree afterwards. They never +depend on the harness obeying its instructions. + +## Contract + +A harness is a second implementation of `agentexec.Runner`. It takes the same +`Request` and returns the same `Result`, so workflows, the daemon and the +State Store do not know which runner ran a stage. + +Each adapter owns, for one CLI: + +- the non-interactive invocation and its streamed JSON output; +- translating stream events into `ToolCallReporter` calls and `Usage`; +- resuming the same session with follow-up input; +- the capabilities it declares: `mcp`, `resume`, `usage`. + +An adapter that cannot report usage reports zero and says so in +`StopReason`; token budgets for that stage are enforced by wall clock only. +Workflow definition validation rejects a stage whose requirements the chosen +harness does not declare, at load time, not mid-run. + +## Selection + +Order of precedence, first match wins: + +1. the stage's `runner` in the workflow definition; +2. the agent profile's `runner`; +3. `builtin`, the ai-sdk agent loop. + +`runner` names a `[harnesses.]` entry. An entry with no consumer +(named by no stage and no profile) is a validation warning, and an unknown +name is a validation error. + +```toml +[harnesses.claude-sub] +adapter = "claude-code" +image = "registry.local/archie-agent-claude:latest" +home = "claude-sam" # a harness home, below +max_concurrent = 1 + +[harnesses.codex-api] +adapter = "codex" +image = "registry.local/archie-agent-codex:latest" +api_key = { engine = "env", key = "OPENAI_API_KEY" } +max_concurrent = 4 +``` + +## Credentials and customisation + +An entry authenticates one of two ways. Setting both is a validation error. + +- **API key:** `api_key` is a secret reference, injected into the container + as the environment variable the adapter names. The key is never written + into the worktree or a harness home. +- **Subscription:** `home` names a **harness home**, a directory under + archie's state directory that the operator logs into once with + `archied harness login `. That command runs the CLI's own login flow + with its config directory pointed at the home. + +A harness home is also where the operator's own setup lives: skills, +plugins, subagent definitions, global instructions, and the CLI's settings +file. Archie mounts it read-write as the CLI's config directory, because +subscription tokens refresh in place. Archie never edits it. Archie's own +mission and tools reach the harness through the prompt and the invocation, +so they do not touch the operator's files. + +`max_concurrent` caps simultaneous runs per entry, default 1. Runs sharing a +home share its refresh state and its subscription's rate limit; the cap is +per home, summed across entries that name it. A stage waiting for a slot is +`pending`, not `running`. + +## Archie tools inside the harness + +Capture tools, which are how a stage returns structured results, are served +to the harness by `archie-agent mcp`, a stdio MCP server started inside the +container and registered through the adapter's invocation flags. It serves +the stage's `CaptureTools` and the central tools the agent profile allows. +It does not serve the `workspace` toolset, for the same reason the built-in +runner withholds it. + +A harness that does not declare `mcp` can only run stages with no capture +tools. + +## Enforcement around the harness + +The harness runs as a subprocess of `archie-agent` in the task-scoped +container, in the task worktree. Archie holds these properties by +construction and by checking the worktree after the process exits: + +| Property | Enforcement | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| Model never runs git | The container has no forge credential. Archie records `HEAD` and every ref before the run; any change fails the step (`failed`, policy class). The workflow commits the stage's changes. | +| Protected paths | A diff touching a `Protection` path fails the step. | +| Read-only stage | Any worktree diff fails the step. | +| Gate | Archie runs the stage's gate after the harness exits. On failure it resumes the same session with the gate output, up to the stage's iteration budget. A harness without `resume` gets one iteration. | +| Budget | Archie kills the process on wall-clock timeout, or when streamed usage passes the token budget. | +| Cancellation | Cancelling the step kills the process group. | + +Harness settings that deny git or protected paths are added where an +adapter supports them, to save wasted turns. They are never the +enforcement. + +## Images + +The default `archie-agent` image bundles no third-party CLI. A harness entry +names an image that contains both the CLI and the `archie-agent` binary. +The repository ships one Dockerfile build target per supported adapter, +built on the `agent` image. Pull policy follows `containers.pull_policy`. + +## Execution tree + +A harness run is one `agent` StepExecution. Subagents the harness spawns +internally are not recorded as children. Their usage is included in the +step's `tokens_used` when the adapter reports it. + +## Out of scope + +- Chat through a harness. The gateway chat loop stays on the built-in + runtime. +- Recording the harness's internal subagents as StepExecutions. +- Cost in currency. +- Choosing a harness automatically per task. + +## Plan + +1. Harness `Runner` with the Claude Code adapter; worktree checks for git, + protected paths and read-only; budget and cancellation. +2. `archie-agent mcp` serving capture tools; gate-and-resume loop. +3. `[harnesses.*]` config, selection precedence and definition validation. +4. API key credentials; harness homes and `archied harness login`; + per-home concurrency. +5. Codex adapter, then Pi, OMP and GitHub Copilot CLI. +6. Dockerfile targets per adapter. + +## Verification + +- The same workflow definition completes on `builtin` and on each adapter + with the same `Result` shape and captures. +- A harness instructed to `git commit` fails its step, and the task branch + is unchanged. +- A harness edit to a protected test file fails the step in a TDD fix stage. +- A read-only feasibility stage that writes a file fails. +- A stage with a failing gate resumes the same session with the gate output, + then succeeds or exhausts its iterations. +- Two tasks naming the same home with `max_concurrent = 1` run one at a + time; the second waits as `pending`. +- An API key never appears in the worktree, the harness home, the task log, + or the event stream. +- Stopping a run kills the harness process within the step's cancel path.