From 0c8ba2a04709b1e572eae681003378ac8b166379 Mon Sep 17 00:00:00 2001 From: Sam Catlow Date: Tue, 22 Sep 2026 23:49:53 +1000 Subject: [PATCH] docs: delete ARCHITECTURE.md, move its normative content into docs/architecture ARCHITECTURE.md duplicated the docs/architecture tree and had drifted badly against it: "internal/ holds ~36" against 112 packages, two of the seven binaries listed, a Forge table listing `RepliesAfter` (deleted from the tree), and a stated invariant ("No daemon-internal cron") contradicted by the scheduling engine that internal/app/archied/scheduling.go composes and main.go starts. It was also the sole home of three pieces of still-authoritative content, so those move rather than disappear: - plugin engine rule (strict), the memory engine family, and the internal/plugin/architecture_test.go enforcement note -> docs/architecture/plugins-and-extensions.md - intake provenance: forge poll/webhook dedup through TaskEnvelope.IdempotencyKey, binding-stamped chat/playbook enqueue, and the single-identity webhook-intake rejection -> docs/architecture/messaging-and-work-intake.md Live references are repointed from CLAUDE.md (AGENTS.md is a symlink to it), docs/architecture, docs/guides, seven PRDs, and four .agents skills. The heading "## Plugin engine rule (strict)" is preserved in its new home so the existing #plugin-engine-rule-strict citations keep resolving. Generated docs artifacts regenerated. --- .agents/skills/archie-agent-fleet/SKILL.md | 2 +- .../archie-architecture-contract/SKILL.md | 14 +- .agents/skills/archie-build-and-env/SKILL.md | 2 +- .../skills/archie-docs-and-writing/SKILL.md | 5 +- ARCHITECTURE.md | 436 ------------------ CLAUDE.md | 15 +- docs/architecture/bindings.md | 9 +- .../architecture/messaging-and-work-intake.md | 26 ++ docs/architecture/plugins-and-extensions.md | 96 +++- docs/data/generated/dev-docs.json | 20 +- docs/data/generated/docs.json | 2 +- docs/guides/first-playbook.md | 2 +- docs/prds/deep-research-capability.md | 4 +- docs/prds/eda-playbook-engine.md | 3 +- docs/prds/event-sources-and-reactions.md | 3 +- docs/prds/image-capability-contract.md | 5 +- docs/prds/memory-engine-unification.md | 3 +- docs/prds/service-decomposition.md | 2 +- docs/prds/webhook-intake-security.md | 8 +- 19 files changed, 171 insertions(+), 486 deletions(-) delete mode 100644 ARCHITECTURE.md diff --git a/.agents/skills/archie-agent-fleet/SKILL.md b/.agents/skills/archie-agent-fleet/SKILL.md index f5df01062..9553b1003 100644 --- a/.agents/skills/archie-agent-fleet/SKILL.md +++ b/.agents/skills/archie-agent-fleet/SKILL.md @@ -158,7 +158,7 @@ Dolt remote is pushed explicitly (`bd dolt push`) and never assumed to be in syn `.pi/control-plane-run/` holds a run's scaffold, lane plan, prompts, scripts and gate output. It is gitignored and disposable. Treat it as run evidence only: a -fact worth keeping goes into this skill, `docs/`, or `ARCHITECTURE.md`, not into a +fact worth keeping goes into this skill or `docs/`, not into a run directory the next clean deletes. ## Worked example diff --git a/.agents/skills/archie-architecture-contract/SKILL.md b/.agents/skills/archie-architecture-contract/SKILL.md index b69c33121..40410e40f 100644 --- a/.agents/skills/archie-architecture-contract/SKILL.md +++ b/.agents/skills/archie-architecture-contract/SKILL.md @@ -63,13 +63,13 @@ Do not simplify these facts into the target model: | Workflow | `internal/domain/workflow` routes a `task.Task` into a `map[string]Workflow`, mutates a broad `TaskContext` through sequential stages. | `internal/domain/workflow/workflow.go` | | Agent-stage boundary | Versioned `agentexec.Request`/`Result` carries one autonomous stage; validation correlates task, attempt, and stage. | `internal/agentexec/protocol.go` | | Container handoff | `internal/taskrun` carries entire task plus `config.Repo` and `config.TaskConfig`; `internal/app/agentworker` owns routing and sequencing while its NATS adapter owns wire mechanics. | `internal/taskrun/taskrun.go`; `internal/app/agentworker/`; `internal/infrastructure/agenttransport/nats/` | -| Git ownership | Daemon-owned worktree operations clone, commit, push, diff, and clean up. | `internal/worktree/`; `internal/domain/workflow/steps.go`; `ARCHITECTURE.md` | +| Git ownership | Daemon-owned worktree operations clone, commit, push, diff, and clean up. | `internal/worktree/`; `internal/domain/workflow/steps.go`; `docs/architecture/agent-system.md` | | Enforcement | Gates, read-only/protected paths, TDD inverted test gates, and diff caps are represented outside prompt prose. | `internal/gate/`; `internal/domain/workflow/{agent,tdd,steps}.go` | | Configuration | `internal/config.Config`, `IdentityConfig`, `Repo`, and task snapshots mix input decoding with runtime concerns. Decoding itself lives in `internal/infrastructure/configuration`. | `internal/config/config.go`; `internal/infrastructure/configuration/` | | Identity | Configured names and bot usernames act as process-local identity keys. There is no durable Identity domain. | `internal/daemon/daemon.go`; `internal/domain/storecontract/storecontract.go`; `internal/gateway/session.go` | | Messaging | `internal/gateway` contains both the `Message` struct flow and richer `MessageEvent`; channel routing and task control are mixed with conversational behavior. | `internal/gateway/{gateway,messageevent,tasks}.go`; `internal/channels/` | | Plugins | `plugin.Plugin` exposes only `Name` and `Version`; current loading interprets operator-installed Go files with Yaegi. | `internal/plugin/plugin.go` | -| Processes | Agent modes include in-process, subprocess, and NATS. Optional containers add a distinct boundary with known gaps. | `internal/config/config.go`; `internal/app/archied/`; `ARCHITECTURE.md` | +| Processes | Agent modes include in-process, subprocess, and NATS. Optional containers add a distinct boundary with known gaps. | `internal/config/config.go`; `internal/app/archied/`; `docs/architecture/organisation.md` | ### Interpret the current lifecycle honestly @@ -98,13 +98,13 @@ not an open design. | Invariant | Required action | Status and source | | --- | --- | --- | -| The model never runs git | Keep clone, branch, commit, push, diff, and cleanup in deterministic workspace/worktree steps. | CURRENT: `ARCHITECTURE.md`; `internal/worktree/` | -| Deterministic constraints outrank prompts | Enforce gates, protected paths, read-only stages, test protection, and change caps outside model instructions. | CURRENT: `ARCHITECTURE.md`; `internal/gate/`; `internal/domain/workflow/` | -| Agent execution is a data boundary | Send bounded, versioned input; validate correlated output before applying it. | CURRENT foundation: `internal/agentexec/protocol.go`; `ARCHITECTURE.md` | +| The model never runs git | Keep clone, branch, commit, push, diff, and cleanup in deterministic workspace/worktree steps. | CURRENT: `docs/architecture/agent-system.md`; `internal/worktree/` | +| Deterministic constraints outrank prompts | Enforce gates, protected paths, read-only stages, test protection, and change caps outside model instructions. | CURRENT: `docs/architecture/agent-system.md`; `internal/gate/`; `internal/domain/workflow/` | +| Agent execution is a data boundary | Send bounded, versioned input; validate correlated output before applying it. | CURRENT foundation: `internal/agentexec/protocol.go`; `docs/architecture/agent-system.md` | | Every workflow ends explicitly | Preserve terminal outcome or visible park. Preserve crash recovery of interrupted `running` work. | CURRENT: `internal/domain/workflow/workflow.go`; `internal/store/store.go` | | Generic plugins stay metadata-only | Keep `plugin.Plugin` at `Name()`/`Version()`. Behavior on typed capability interface with owning `Registry`/`Manager`. | CURRENT-enforced rule: `internal/plugin/architecture_test.go` | -| Capability engines own lifecycle | Give resource-owning engines explicit start/health/stop semantics and failure isolation. | APPROVED rule: `ARCHITECTURE.md#plugin-engine-rule-strict` | -| Trust is explicit | Treat trusted operator-installed in-process code, repository code, out-of-process integrations, and container-isolated code as different trust classes. | APPROVED rule: `ARCHITECTURE.md#plugin-engine-rule-strict` | +| Capability engines own lifecycle | Give resource-owning engines explicit start/health/stop semantics and failure isolation. | APPROVED rule: `docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict` | +| Trust is explicit | Treat trusted operator-installed in-process code, repository code, out-of-process integrations, and container-isolated code as different trust classes. | APPROVED rule: `docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict` | | Infrastructure stays replaceable | Keep forge, persistence, workspace, transport, container, and model details behind behavior-owned contracts. | APPROVED TARGET: `docs/architecture/dependencies-and-contracts.md` | | Same-repository execution serializes by default | Preserve default unless explicit scheduling policy permits concurrency. | CURRENT: `internal/config/config.go`; `internal/daemon/daemon.go` | diff --git a/.agents/skills/archie-build-and-env/SKILL.md b/.agents/skills/archie-build-and-env/SKILL.md index 2e4053818..d329d3d78 100644 --- a/.agents/skills/archie-build-and-env/SKILL.md +++ b/.agents/skills/archie-build-and-env/SKILL.md @@ -26,7 +26,7 @@ No root `README*` or `CONTRIBUTING*` file exists. Begin with: ```bash sed -n '1,240p' CLAUDE.md -sed -n '1,240p' ARCHITECTURE.md +ls docs/architecture/ task --list ``` diff --git a/.agents/skills/archie-docs-and-writing/SKILL.md b/.agents/skills/archie-docs-and-writing/SKILL.md index f7b17c995..b83bb71cd 100644 --- a/.agents/skills/archie-docs-and-writing/SKILL.md +++ b/.agents/skills/archie-docs-and-writing/SKILL.md @@ -1,6 +1,6 @@ --- name: archie-docs-and-writing -description: Maintain Archie's documentation of record and write durable architecture decisions, migration plans, parity matrices, incident and dead-end records, feature ownership and deprecation records, operational runbooks, and documentation reviews. Use when deciding where an Archie fact belongs; reconciling code with ARCHITECTURE.md, CLAUDE.md, docs/prds, docs/archive, or generated contracts; changing tools/docsgen; documenting a feature so future maintainers can find its owners, consumers, invariants, superseded paths, deletion gates, evidence, and rollback; or checking documentation authority, drift, links, copied content, and source-artifact hygiene. +description: Maintain Archie's documentation of record and write durable architecture decisions, migration plans, parity matrices, incident and dead-end records, feature ownership and deprecation records, operational runbooks, and documentation reviews. Use when deciding where an Archie fact belongs; reconciling code with `CLAUDE.md`, `docs/architecture`, `docs/prds`, `docs/archive`, or generated contracts; changing `tools/docsgen`; documenting a feature so future maintainers can find its owners, consumers, invariants, superseded paths, deletion gates, evidence, and rollback; or checking documentation authority, drift, links, copied content, and source-artifact hygiene. --- # Maintain Archie documentation @@ -42,7 +42,6 @@ Verified on 2026-09-18: |---|---|---| | Live Go code, tests, `Taskfile.yml`, executable config parsing, composition under `internal/app/` (entered from `cmd/`) | `CURRENT` execution evidence | Trace producers and consumers; tests prove only asserted behavior | | `CLAUDE.md` | Current contributor and safety protocol plus compact architecture orientation | Verify operational claims against code and `Taskfile.yml`; `AGENTS.md` is a symlink to it | -| `ARCHITECTURE.md` | Useful architecture history and partial current overview | Corroborate every inventory/status claim; its "Planned" list is stale for implemented skill support | | `docs/prds/01-project-management.md` | Index and approved foundation for target architecture | Add or change target decisions in the focused document it names | | `docs/architecture/*.md` | Focused target decisions, active review procedure, migration inventory | Read each file's status | | `docs/architecture/migration-decisions.md` | `OPEN` migration inventory constrained by approved decisions | Close a question only after code-grounded review | @@ -77,7 +76,7 @@ The repository root has no `README.md` or `CONTRIBUTING.md` as of 2026-09-18. | Superseded design retained for context | `docs/archive/`, with historical status | | Incident or rejected approach | `archie-failure-archaeology`'s chronology | -Do not duplicate a package map across `CLAUDE.md`, `ARCHITECTURE.md`, a PRD, and +Do not duplicate a package map across `CLAUDE.md`, a PRD, and the site. Name the authoritative map and link to it. ## Write for future change diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md deleted file mode 100644 index d23de5d2f..000000000 --- a/ARCHITECTURE.md +++ /dev/null @@ -1,436 +0,0 @@ -# archie-core Architecture - -**archied** is a resident daemon that discovers forge work — GitHub and Gitea -issues assigned to its bot user, carrying its label, or either (`dispatch` -mode, per identity) — by polling (the default) or, single-identity -deployments only, reacting to forge webhook events the instant they arrive -(`Forge.Intake` = `webhook`/`both`, see `docs/prds/event-sources-and- -reactions.md`), works each matched issue in an isolated worktree through a -routed workflow, and opens pull requests for human review. A single daemon can -serve several identities across several forges, each with its own `bot_user`, -label, and repository set. - -## Package Map - -This lists the load-bearing packages, not all of them — `internal/` holds ~36. -The domain migration is in progress (`internal/domain/`, -`internal/infrastructure/`), so **the tree is authoritative over this table**. -See `docs/architecture/organisation.md` for the target structure and -`docs/architecture/migration-decisions.md` for the current position. - -| Package | Role | -| ---------------------------- | ------------------------------------------------------------------------- | -| `cmd/archied/` | Binary entry: config→forge→store→runtime→workflows→daemon | -| `cmd/archie-agent/` | Binary entry: flags, environment, signals, and agentworker invocation | -| `internal/app/agentworker/` | Agent worker composition, lifecycle, routing, and workflow execution | -| `internal/infrastructure/agentboot/` | Worker boot-brief filesystem decoding | -| `internal/infrastructure/agentgit/` | Worker git safe-directory setup | -| `internal/infrastructure/agenttransport/nats/` | Worker core-NATS full-task handoff and RPC client construction | -| `internal/config/` | Config types only — loading lives in `infrastructure/configuration` | -| `internal/daemon/` | Resident loop: poll, enqueue, claim, route, process, reconcile | -| `internal/domain/workflow/` | Engine: stage lists, routing, shared steps, workflow definitions | -| `internal/domain/workflow/task/` | Task-execution vocabulary (`Task`, statuses, mid-run `Store`, `Definition`) importable without the engine | -| `internal/domain/messaging/` | Canonical messages plus the chat wire contracts (`ChatContract`, sessions, turns, streaming) shared by gateway, channels, webui, and archie-ui | -| `internal/channels/status/` | Operator-facing channel lifecycle ledger (`State`/`Descriptor`/`Status`/`Manager`); the plugin contract stays in `internal/channels` | -| `internal/app/archiemessaging/` | Messaging Service composition: Telegram/email/webhook channels, dialling the Gateway's `ChatContract` over gRPC. The daemon runs no channel and builds no `gateway.Router` | -| `internal/domain/workintake/`| `TaskEnvelope`, routing `Kind`, label vocabulary, task subjects | -| `internal/agentexec/` | Worker-local stage protocol and ai-sdk loop runner | -| `internal/forge/` | Forge interface: GitHub and Gitea implementations | -| `internal/forge/webhook/` | Optional forge webhook receiver: GitHub issue events → `workintake.TaskEnvelope`, same publish path as polling | -| `internal/webhookguard/` | Cross-cutting inbound-webhook mechanics: HMAC verification, per-source rate limiting | -| `internal/worktree/` | Git operations: clone, branch, commit, push, diff, cleanup | -| `internal/store/` | Task/capture/mapping/binding SQLite implementation; producer-owned contract interfaces live in `internal/domain/storecontract` (aliases kept here) | -| `internal/domain/storecontract/` | Producer-owned State Store read contracts consumed by webui, intake, and the gRPC adapters (state-store-contract rev. 2d) | -| `internal/events/` | In-process event bus: publish/subscribe for observability | -| `internal/infrastructure/` | Config loading, eventbus transport, model catalog | -| `internal/webui/` | Shared dashboard HTTP layer served by the archie-ui process | -| `internal/gateway/` (18.7k LOC) | Persistent-connection layer between archie and its users (Telegram, web UI, etc.); shared CommandRouter for dispatch | -| `internal/tools/` (20.9k LOC) | Central tool subsystem: built-in tool implementations, approval, execution contract | -| `internal/domain/memory/` | Memory engine contract: four typed scopes, CRUD + revisions, content scanner | -| `internal/infrastructure/memory/` | Builtin memory engine + its `builtin/` markdown store | -| `internal/secret/` | Secret reference resolution across named engines (env, vault, etc.) | -| `internal/plugin/` | Core plugin interface: Yaegi-interpreted `.go` plugins loaded from `~/.config/archie/plugins/` and registered with the daemon's extension registry | -| `internal/skill/` | Parses and discovers agentskills.io `SKILL.md` files | -| `ai-sdk/runtime` | External: provider-agnostic LLM runtime | -| `ai-sdk/agentloop` | External: gated agent loop with budgets and tool jails | -| `ai-sdk/core` | External: tool definitions, generate options | - -## Task Lifecycle - -``` -queued → running(workflow:stage) → pr_open → merged | rejected - ↓ - waiting_human → (human approves) → queued(implement) - ↓ - parked -``` - -**Crash recovery:** tasks left in `running` are re-queued on startup. Parks are -never silent, but they no longer comment on the forge issue: the reason is -recorded on the task as `park_reason` (cleared on requeue) and surfaced by the -dashboard and `/api/tasks`. Retries are an explicit operator action capped by -`max_retries`, not a label a human removes. - -**Adversarial review:** the implement workflow runs a fresh adversarial review -after the deterministic gates and before `pr_open` (see -`docs/architecture/adversarial-review.md`). A surviving blocking finding — or a -review that failed to run — parks the task with the findings as its -`park_reason`; otherwise the PR opens with a findings section on its body. -Opt-in per repo via `repo.review_enabled`. - -**PR reconciliation:** the daemon polls open PRs, checks GitHub state -(`merged`/`closed`), transitions the task, and cleans up worktrees. - -**Provenance by origin:** a task's origin determines which enqueue path -created it, though every origin converges on the same `tasks` table and the -same `Route`/`Run` lifecycle above once queued. Forge-sourced tasks — whether -discovered by polling or, single-identity only, delivered by the forge -webhook receiver (`internal/forge/webhook`) — enqueue via the same -`workintake.TaskEnvelope` and `PublishTask` path (see `docs/prds/event-sources- -and-reactions.md`'s decided contract); `TaskEnvelope.IdempotencyKey` is keyed -on `owner/repo/number`, not delivery source, so a webhook-delivered issue and -a later poll of the same issue dedup for free instead of double-enqueuing. -Playbook-binding-dispatched tasks enqueue -via `EnqueueChatTask`/`EnqueueBindingTask` (`internal/store`), the same direct -path chat-spawned tasks use, stamped with `binding_id`/`binding_version` for -provenance. See `docs/architecture/bindings.md` for the binding model and its -threat model in full. - -## Workflow Engine - -### Core types (`workflow.go`) - -- `Workflow` = named list of `Stage`s -- `Stage` = `Name` + `Run(ctx, *TaskContext) error` -- `TaskContext` = mutable bag: Task, Repo, Config, Forge, Store, Agent runner, - worktree dir/branch, scratch fields -- `Outcome` = terminal decision (status + detail) -- ends the workflow - immediately -- `Registry` = `map[string]Workflow` - -### Routing (`Route`) - -Label-driven, no LLM involved. Pre-assigned workflow wins; then labels decide. -The built-in bindings are: - -- `bug` → `tdd` -- `feature` → `feasibility` -- `bootstrap` → `bootstrap` (diagnostics, no LLM spend) -- Default → `implement` - -Both the closed `bug`/`feature`/`bootstrap` set's target workflow and -arbitrary additional labels are YAML-overridable at daemon startup -(`internal/domain/workflow/routing.go`'s `LoadKindWorkflowsYAML`/ -`LoadLabelWorkflowsYAML`, `WorkflowRoutingFile`/`WorkflowLabelsFile`/ -`PlaybookDirs` config) -- see `docs/prds/eda-playbook-engine.md`. A label -already owned by the closed kind set, or a binding declared twice across -sources, is a reported load failure, not a silent override. - -### Execution (`Run`) - -Stages execute sequentially. After each stage: - -- Task is persisted to SQLite -- Error → park the task (comment on issue, keep worktree) -- Outcome set → finish (transition task, update labels) -- Neither → continue to next stage - -A workflow that ends without an outcome parks -- definition bugs must not vanish -silently. - -## Workflows - -### Bootstrap (diagnostics, no LLM) - -``` -prepare → apply(marker file) → commit-push → diff-cap → open-pr -``` - -Proves the plumbing: clone, push, PR. Label any test issue `bootstrap` to verify -end-to-end. - -### Implement (default) - -``` -prepare → baseline-gate → plan(planner, read-only) → build(builder, gated) → commit-push → diff-cap → open-pr -``` - -- **baseline-gate:** runs the repo's gate at the base commit before anything is - touched -- **plan:** read-only agent explores codebase, produces implementation plan, - posts it as a comment -- **build:** full toolset agent implements against the plan behind the repo's - quality gate -- **diff-cap:** parks oversized changes (>400 lines default) -- **open-pr:** builder's summary becomes the PR body - -### TDD (bug label) - -``` -prepare → baseline-gate → analyse(planner) → repro-tests(builder, inverted gate) → capture-proof → commit-repro → fix(builder, normal gate, test files protected) → commit-push → diff-cap → open-pr + repro-evidence -``` - -- **repro-tests:** the quality gate is **inverted** -- the test command must - FAIL. Proven failure is required. -- **capture-proof:** runs the test command deterministically, captures the - failing output -- **commit-repro:** commits the failing tests as the first commit -- **fix:** test files are **environmentally write-protected** (not - prompt-ruled). The agent fixes code, not tests. -- **open-pr + evidence:** posts both the PR and the captured failure proof - -### Feasibility (feature label) - -``` -prepare → assess(planner, with decide tool) → [close_if_bad_fit | prd(planner)] → deliver → waiting_human -``` - -- **assess:** reads AGENT.md, ROADMAP.md, README. Calls a `decide` tool - (structured output, not parsed prose). Bad fit → close with reasons. -- **prd:** read-only agent writes a PRD: problem, solution, files affected, - acceptance criteria, non-goals, diff estimate -- **deliver:** posts PRD as issue comment + webhook notification (n8n → email). - Sets state to `waiting_human`. -- The daemon watches for human replies, judges them with a triage LLM - (APPROVE/REJECT/UNCLEAR), requeues or closes. - -## Forge Interface - -A thin abstraction over the forge. GitHub (`github.go`) and Gitea (`gitea.go`) -both implement it, plus a no-op (`noop.go`) used when no credential resolves — -see the degradation rule in `docs/architecture/configuration.md`. A multi-forge -deployment is configured per identity; see -`deployments/multi-forge-github-gitea.toml`. - -| Method | Purpose | -| ------------------- | --------------------------------------- | -| `AcceptInvitations` | Auto-accept repo invites | -| `AssignedIssues` | Poll assigned issues (excluding PRs) | -| `IssuesWithLabel` | Poll labelled issues | -| `Comment` | Post issue/PR comment, return ID | -| `RepliesAfter` | Find human replies after a comment ID | -| `CreatePR` | Open pull request | -| `PRState` | Check merged/open/closed | -| `CloseIssue` | Close with final comment | -| `React` | Emoji reaction (pickup acknowledgement) | -| `SetStateLabel` | Replace old state label with new one | -| `VerifyPush` | Confirm push permission at startup | - -State labels (`archie:queued`, `archie:working`, etc.) are created on demand, -removed when transitioning. They report state; they do not drive it. Removing -a label no longer retries a parked task -- retries are an explicit operator -action from the dashboard or chat, capped by `max_retries`. - -### Webhook intake (alternative to polling) - -`internal/forge/webhook` is a separate HTTP receiver, not a `Forge` method: it -verifies a GitHub webhook's HMAC signature (`internal/webhookguard.VerifyHMAC`), -decodes an `issues` event, applies the same dispatch predicate the poller -uses, and calls the same `PublishTask` the poller calls. It is opt-in via -`Forge.Intake` (`poll` default, `webhook`, `both`) and single-identity only: a -config naming `webhook`/`both` together with `[[identities]]`, or setting -intake on an identity, is rejected at validation rather than started with the -setting ignored -- the receiver binds one secret and one dispatch predicate, -so it cannot judge a second identity's events by the right rules. A -multi-identity deployment polls per identity until per-identity intake lands. -See `docs/prds/event-sources-and-reactions.md` for why this stayed a -thin decode-and-reuse layer rather than a second intake mechanism. - -## Worktree Manager - -Every task gets a fresh clone -- no persistent state between runs. The model -never runs git. - -``` -~/.local/share/archie/work/-/issue-/ -``` - -Token auth via a generated `GIT_ASKPASS` helper script -- the token never -appears in `.git/config` or process argv. - -## Config - -Daemon-level TOML (`~/.config/archie/config.toml`): - -- `[forge]` -- type (github/gitea), host, token secret reference; `intake` - (`poll` default, `webhook`, or `both`) with `webhook_secret`/`webhook_addr` - when reacting to forge events instead of, or alongside, polling - (single-identity deployments only; rejected, not ignored, alongside - `[[identities]]` or on a `[identities.forge]` entry) -- `[dispatch]` -- trigger (assignee/label/either), labels, ack reaction -- `[[repos]]` -- owner, name, base branch, gate commands, protected paths, - ecosystem -- `[models]` -- planner, builder, triage (provider/model refs) -- `[providers]` -- LLM provider configs -- `[nats]` -- embedded or external broker deployment -- `[containers]` -- managed `archie-agent` image, limits, storage, and network -- `[budgets]` -- max steps, max tokens, wall clock, gate max failures -- `[web]` -- dashboard listen address, bound by the standalone archie-ui - process (the daemon runs no dashboard listener after the UI cutover) -- `[notify]` -- webhook URL for notifications - -Secrets (forge token, provider API keys, channel tokens) are not embedded in -config files. `internal/secret` defines a pluggable `Engine` interface (`Name`, -`Version`, `Resolve(key) (string, error)`) resolved through a `Registry`; the -built-in `env` engine reads an environment variable by name and the built-in -`bws` engine caches the accessible Bitwarden project at startup. Custom engines -(sops, Vault/OpenBao) are Yaegi-interpreted `.go` plugins loaded from a -directory, following the same convention as `internal/plugin`'s daemon plugins. -Config fields reference a secret as -`{engine, key}` (`secret.SecretRef`), e.g. -`token = { engine = "env", key = "ARCHIE_GITHUB_TOKEN" }`. As of this writing -`Forge.Token` and `Provider.APIKey` use `SecretRef`; resolved provider values -are exported under private, identity-scoped environment names because the SDK -and container boundary consume credential names rather than -plaintext wire fields. The older `Provider.APIKeyEnv`, `NATSConfig.TokenEnv`, and -`TelegramConfig.TokenEnv` field names remain supported. - -Per-repo gates are command lists: `[[repos.gate]] = ["go", "vet", "./..."]`. The -last command is the test runner (by convention -- TDD inverts it). - -## Observability - -Every lifecycle event flows through an internal event bus: - -- SQLite event log (every event stamped with a row ID) -- Live dashboard via SSE (server-sent events) -- Bounded subscriber buffers -- slow consumers drop events, never block the - daemon - -## Plugin engine rule (strict) - -“Engine” is reserved for a typed, lifecycle-managed capability family. It is not -a synonym for a generic plugin hook. Every new plugin engine must satisfy all of -these invariants: - -1. **Typed domain contract.** The engine interface exposes real - capability-specific operations in addition to identity metadata. Examples are - resolving a secret, delivering a message, or running an agent. A plugin that - only exposes `Name` and `Version` is metadata, not an engine. -2. **Owning family manager.** The capability package owns a `Registry` or - `Manager` that controls provider registration and discovery, initialization, - or access. Providers do not register arbitrary callbacks directly on the - daemon. -3. **Lifecycle and health.** When an implementation owns resources or background - work, its typed family contract must expose start, health, and stop - semantics. The owning manager invokes them and defines failure isolation and - shutdown ordering. Stateless providers may make these operations explicit - no-ops. -4. **Narrow host access.** Plugins receive the typed family registrar and only - the host services their declared capability requires. They never receive - `*daemon.Daemon`, a service locator, or an untyped hook map. -5. **Metadata-only generic plugin contract.** `plugin.Plugin` remains limited to - `Name()` and `Version()`. Domain hooks belong on typed engine interfaces; - adding capability methods to the generic interface is forbidden. -6. **Trust boundary stays explicit.** Explicitly trusted, operator-installed - plugins—including third-party secret-engine plugins the operator chooses to - trust—may run in-process and have daemon privileges. Repository-supplied code - runs in the task container. Integrations that are not trusted as daemon code - run out of process through MCP or a versioned protocol and, when a - confidentiality or integrity boundary is required, inside a real container - sandbox. A subprocess boundary alone is not a security sandbox, and calling - Yaegi-loaded code a plugin does not isolate it. - -`internal/plugin/architecture_test.go` enforces the mechanically testable parts -of this rule: the generic plugin method set, the shared agent-instruction -source, and the requirement that engine interfaces have domain behavior plus an -owning registry or manager. - -### Memory engine family - -One engine, addressed by four typed scopes -- `ScopeGlobal`, `ScopeAgent`, -`ScopeUser`, `ScopeAgentUser` (`internal/domain/memory`, `Scope`/`ScopeKind`). -A caller names exactly the scopes it may read or write; the engine applies no -access policy of its own, so isolation falls out mechanically because -`ScopeAgent{a}` and `ScopeAgent{b}` are different storage keys. See -`docs/prds/memory-engine-unification.md` for the decision this contract -implements; it superseded `internal/memory` (`MemoryProvider`/`Manager`), -deleted in that PRD's slice 5. - -`MemoryEngine` (`internal/domain/memory/contract.go`) is the typed family -contract: `Create`, `Get`, `Query`, `List`, `Update`, `Forget`, `Revisions` -- -CRUD with revisions retained, not just create/read/delete. `Update` supersedes -rather than overwrites: the superseded state is appended to the scope's -`HISTORY.md` before the live block is replaced, and `Update` carries no -`Scope` field, so a record can never change scope. The family's content -scanner (prompt-injection / sensitive-data patterns) is applied inside -`Create` and `Update`, the single choke point every producer crosses. - -`internal/domain/memory.Registry` is the owning family registry: registration, -start/health/stop, shutdown ordering, and failure isolation, following the -plugin engine rule (`ARCHITECTURE.md#plugin-engine-rule-strict`). -`internal/infrastructure/memory.NewBuiltinEngine` is the only registered -engine today; it persists through `internal/infrastructure/memory/builtin`, -a markdown store (`MEMORY.md`-shaped per-scope files) that owns its on-disk -format end-to-end. - -Two per-turn seams in `internal/gateway` connect a chat turn to the engine, -each narrowed to exactly the operations it needs: - -- **Read** (`turn_memory.go`, `MemoryStore`): `prepareTurn` runs a - scope-only `Query` over the resolved `Subject`'s readable scopes, - synchronously and before the prompt is built. It never returns an error -- - a panicking or failing engine degrades to an empty `` block rather - than failing the turn -- and is bounded by a per-scope record limit and a - byte cap on the rendered block. -- **Write** (`turn_memory_tool.go`, `MemoryWriteStore`): four tools -- - `memory_create`, `memory_update`, `memory_delete`, `memory_list` -- built - per turn onto the resolved `Subject`'s writable scopes, not registered once - at boot. The model chooses the scope kind; it never chooses the agent or - user id, which come from the turn's resolved `Subject` and have no schema - field to override. `ScopeGlobal` is never model-writable. - -`boot.setupMemoryEngine` (`internal/app/archied/bootstrap.go`) registers and -starts the engine and must run before `setupGatewayChat`, which constructs the -only chat turn runner in production and captures the engine at construction -time (`internal/app/archied/composition_order_test.go` pins the ordering), and -before `setupCurators`, whose `Registrar` captures the engine registry. The session curator (`internal/infrastructure/sessioncurator`) -writes derived records onto `ScopeAgentUser`, making it the engine's other -live producer besides the chat write path. - -## Key Design Decisions - -1. **Environmental constraints over prompt rules.** The gate, test-file - protection, and diff cap are code enforcement -- not "please don't do that" - in the system prompt. -2. **The model never runs git.** Worktree operations are deterministic steps, - not agent tools. -3. **Tokens as cost, not features.** Token usage is tracked per task and emitted - in observability events but never budgeted beyond the agent loop's own - limits. -4. **Label-driven routing.** Workflow selection is a switch statement, not an - LLM call. A triage LLM is used only for the human reply judgment in - waiting_human. -5. **Shared code.** `ai-sdk/agentloop` and `ai-sdk/toolkit` provide the common - agent loop and tool set. -6. **No daemon-internal cron.** Polling is a ticker loop, not cron. The - feasibility PRD notification uses an external webhook (n8n). -7. **Fresh worktree per task.** No persistent state between runs. Crash recovery - just re-queues. -8. **Agent execution is a data boundary.** Autonomous stages receive a versioned - request and return a fenced result; SQLite, forge operations, git, workflow - sequencing, and outcome handling stay daemon-owned. - -## What's Implemented vs Planned - -**Implemented:** - -- Daemon poll loop, enqueue, claim, process -- Forge webhook intake as a feature-flagged alternative/backstop pairing to - polling (single-identity deployments; see `internal/forge/webhook`) -- Bootstrap, implement, TDD, feasibility workflows -- GitHub and Gitea forges (issues, PRs, labels, comments, reactions) -- Worktree isolation (clone, branch, commit, push, diff, cleanup) -- SQLite store with transition audit -- Config with ecosystem support (go, python, node, rust) -- Agent stages with gates, budgets, preflight, protected paths -- Observability (event bus, SQLite log, SSE dashboard) -- Crash recovery (re-queue stale running tasks) -- Human reply judgment (triage LLM for waiting_human) - -**Planned:** - -- Multi-identity webhook intake (currently single-identity only) -- More ecosystems -- Notification integrations beyond n8n webhook -- Skills (SKILL.md support -- see `.archie/skills/`) diff --git a/CLAUDE.md b/CLAUDE.md index 2b2b912ec..119e06999 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,11 +25,13 @@ assigned/labelled to it, works each one in an isolated git worktree through a routed workflow (bootstrap / implement / TDD / feasibility), and opens pull requests for human review. -See `ARCHITECTURE.md` for the package map, workflow engine design, task -lifecycle, and key design decisions. Read it before making non-trivial changes. -Key invariants: env-enforced gates, model never runs git, and agent execution -functions as a strict data boundary. Before adding a new plugin engine, satisfy -`ARCHITECTURE.md#plugin-engine-rule-strict`. +Architecture of record lives in `docs/architecture/` -- `organisation.md` for +the package map, `agent-system.md` for the workflow-execution lifecycle and its +invariants, `plugins-and-extensions.md` for engine families. Read it before +making non-trivial changes. Key invariants: env-enforced gates, model never runs +git, and agent execution functions as a strict data boundary. Before adding a new +plugin engine, satisfy +`docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`. ## Scope Discipline @@ -257,7 +259,8 @@ structures found in legacy packages. - Webhook intake refuses to start when `[[identities]]` is configured (one receiver, one dispatch config). Do not silently enable it for multi-identity deployments without first building per-identity routing -- see - `ARCHITECTURE.md`'s "Webhook intake" section. + `docs/architecture/messaging-and-work-intake.md`'s "Intake provenance" + section. - **`internal/domain/embedding/` + `internal/infrastructure/embedding/`:** Embedding client contract + implementation. Config-driven exactly like chat model roles: `models.embedding = "provider/model"` plus a `[providers.*]` diff --git a/docs/architecture/bindings.md b/docs/architecture/bindings.md index 188410ef7..07dc3b78f 100644 --- a/docs/architecture/bindings.md +++ b/docs/architecture/bindings.md @@ -5,9 +5,9 @@ through `.5`, `.7`, `.9`-`.15`) **Date:** 2026-09-03 This document covers the binding model and the intake threat model portion -of `archie-core-t2db.6`. It does not cover the `ARCHITECTURE.md` task-lifecycle -addendum or the operator walkthrough -- those remain separate, open pieces of -the same ticket. +of `archie-core-t2db.6`. It does not cover the task-lifecycle provenance rule +(`messaging-and-work-intake.md`'s "Intake provenance") or the operator +walkthrough -- that walkthrough remains an open piece of the same ticket. ## What a binding is @@ -177,9 +177,6 @@ execution-time idempotency, now landed (`archie-core-t2db.17` closed): the ## What this document does not cover -- The `ARCHITECTURE.md` task-lifecycle section noting that a - playbook-originated task has different provenance from a forge-issue - task -- open, part of `archie-core-t2db.6`. - An end-to-end operator walkthrough (point an external app at archie, build a first playbook) -- open, part of `archie-core-t2db.6`. - The playbook YAML / CEL / Module/Channel/Forge action-position work diff --git a/docs/architecture/messaging-and-work-intake.md b/docs/architecture/messaging-and-work-intake.md index 22971a809..d7f6b8c93 100644 --- a/docs/architecture/messaging-and-work-intake.md +++ b/docs/architecture/messaging-and-work-intake.md @@ -251,6 +251,32 @@ Current intake is fragmented: - `store.Task` collapses an external source reference and durable execution state. +## Intake provenance + +A task's origin determines which enqueue path created it, though every origin +converges on the same `tasks` table and the same `Route`/`Run` lifecycle once +queued. + +- Forge-sourced work -- discovered by polling or, in single-identity + deployments, delivered by the forge webhook receiver (`internal/forge/webhook`) + -- enqueues through the same `workintake.TaskEnvelope` and `PublishTask` path. + `TaskEnvelope.IdempotencyKey` is keyed on `owner/repo/number` and not on the + delivery source, so a webhook-delivered issue and a later poll of the same + issue dedup through `PublishUnique` instead of double-enqueuing. +- Chat-spawned and playbook-binding-dispatched work enqueues through + `EnqueueChatTask`/`EnqueueBindingTask` (`internal/store`), stamped with + `binding_id`/`binding_version` for provenance. + +Webhook intake is opt-in per deployment (`Forge.Intake` = `poll` (default), +`webhook`, or `both`) and single-identity only: a config naming +`webhook`/`both` alongside `[[identities]]`, or setting intake on a +`[identities.forge]` entry, is rejected at validation rather than started with +the setting ignored. One receiver binds one secret and one dispatch predicate, +so it cannot judge a second identity's events by the right rules. A +multi-identity deployment polls per identity until per-identity intake lands. +The decided contract is `docs/prds/event-sources-and-reactions.md`; the binding +model and its threat model are `docs/architecture/bindings.md`. + ## Current hazards - Treating every agent interaction as a work request would remove Archie's diff --git a/docs/architecture/plugins-and-extensions.md b/docs/architecture/plugins-and-extensions.md index 58c8dfcec..1374a1888 100644 --- a/docs/architecture/plugins-and-extensions.md +++ b/docs/architecture/plugins-and-extensions.md @@ -23,6 +23,97 @@ versioning, definition, and execution contracts. The generic plugin domain may provide plugin identity, discovery, compatibility, and lifecycle mechanics, but it MUST NOT define Workflow semantics or expose untyped Workflow hooks. +## Plugin engine rule (strict) + +“Engine” is reserved for a typed, lifecycle-managed capability family. It is not +a synonym for a generic plugin hook. Every new plugin engine must satisfy all of +these invariants: + +1. **Typed domain contract.** The engine interface exposes real + capability-specific operations in addition to identity metadata. Examples are + resolving a secret, delivering a message, or running an agent. A plugin that + only exposes `Name` and `Version` is metadata, not an engine. +2. **Owning family manager.** The capability package owns a `Registry` or + `Manager` that controls provider registration and discovery, initialization, + or access. Providers do not register arbitrary callbacks directly on the + daemon. +3. **Lifecycle and health.** When an implementation owns resources or background + work, its typed family contract must expose start, health, and stop + semantics. The owning manager invokes them and defines failure isolation and + shutdown ordering. Stateless providers may make these operations explicit + no-ops. +4. **Narrow host access.** Plugins receive the typed family registrar and only + the host services their declared capability requires. They never receive + `*daemon.Daemon`, a service locator, or an untyped hook map. +5. **Metadata-only generic plugin contract.** `plugin.Plugin` remains limited to + `Name()` and `Version()`. Domain hooks belong on typed engine interfaces; + adding capability methods to the generic interface is forbidden. +6. **Trust boundary stays explicit.** Explicitly trusted, operator-installed + plugins—including third-party secret-engine plugins the operator chooses to + trust—may run in-process and have daemon privileges. Repository-supplied code + runs in the task container. Integrations that are not trusted as daemon code + run out of process through MCP or a versioned protocol and, when a + confidentiality or integrity boundary is required, inside a real container + sandbox. A subprocess boundary alone is not a security sandbox, and calling + Yaegi-loaded code a plugin does not isolate it. + +`internal/plugin/architecture_test.go` enforces the mechanically testable parts +of this rule: the generic plugin method set, the shared agent-instruction +source, and the requirement that engine interfaces have domain behavior plus an +owning registry or manager. + +### Memory engine family + +One engine, addressed by four typed scopes -- `ScopeGlobal`, `ScopeAgent`, +`ScopeUser`, `ScopeAgentUser` (`internal/domain/memory`, `Scope`/`ScopeKind`). +A caller names exactly the scopes it may read or write; the engine applies no +access policy of its own, so isolation falls out mechanically because +`ScopeAgent{a}` and `ScopeAgent{b}` are different storage keys. See +`docs/prds/memory-engine-unification.md` for the decision this contract +implements; it superseded `internal/memory` (`MemoryProvider`/`Manager`), +deleted in that PRD's slice 5. + +`MemoryEngine` (`internal/domain/memory/contract.go`) is the typed family +contract: `Create`, `Get`, `Query`, `List`, `Update`, `Forget`, `Revisions` -- +CRUD with revisions retained, not just create/read/delete. `Update` supersedes +rather than overwrites: the superseded state is appended to the scope's +`HISTORY.md` before the live block is replaced, and `Update` carries no +`Scope` field, so a record can never change scope. The family's content +scanner (prompt-injection / sensitive-data patterns) is applied inside +`Create` and `Update`, the single choke point every producer crosses. + +`internal/domain/memory.Registry` is the owning family registry: registration, +start/health/stop, shutdown ordering, and failure isolation, following the +plugin engine rule above. `internal/infrastructure/memory.NewBuiltinEngine` is +the only registered engine today; it persists through +`internal/infrastructure/memory/builtin`, a markdown store (`MEMORY.md`-shaped +per-scope files) that owns its on-disk format end-to-end. + +Two per-turn seams in `internal/gateway` connect a chat turn to the engine, +each narrowed to exactly the operations it needs: + +- **Read** (`turn_memory.go`, `MemoryStore`): `prepareTurn` runs a + scope-only `Query` over the resolved `Subject`'s readable scopes, + synchronously and before the prompt is built. It never returns an error -- + a panicking or failing engine degrades to an empty `` block rather + than failing the turn -- and is bounded by a per-scope record limit and a + byte cap on the rendered block. +- **Write** (`turn_memory_tool.go`, `MemoryWriteStore`): four tools -- + `memory_create`, `memory_update`, `memory_delete`, `memory_list` -- built + per turn onto the resolved `Subject`'s writable scopes, not registered once + at boot. The model chooses the scope kind; it never chooses the agent or + user id, which come from the turn's resolved `Subject` and have no schema + field to override. `ScopeGlobal` is never model-writable. + +`boot.setupMemoryEngine` (`internal/app/archied/bootstrap.go`) registers and +starts the engine and must run before `setupGatewayChat`, which constructs the +only chat turn runner in production and captures the engine at construction +time (`internal/app/archied/composition_order_test.go` pins the ordering), and +before `setupCurators`, whose `Registrar` captures the engine registry. The +session curator (`internal/infrastructure/sessioncurator`) writes derived +records onto `ScopeAgentUser`, making it the engine's other live producer +besides the chat write path. + ## Plugin implementations First-party plugins are self-contained vertical modules under @@ -48,8 +139,9 @@ in-process through typed capability families (this document's model) or out-of-process like a generic `DispatchEvent` dispatcher. Decided 2026-08-22 in `docs/prds/event-sources-and-reactions.md`: in-process typed families, confirming this document's model rather than amending it. The forge webhook -receiver (`internal/forge/webhook`, see `ARCHITECTURE.md`'s "Webhook intake" -section) and the EDA playbook engine's Module action position +receiver (`internal/forge/webhook`; see "Intake provenance" in +`docs/architecture/messaging-and-work-intake.md`) and the EDA playbook engine's +Module action position (`docs/prds/eda-playbook-engine.md`, `internal/domain/eda/module`) are the shipped instances of that decision -- each is a narrow typed contract with an owning registry, not a generic event hook. The playbook engine's Channel and diff --git a/docs/data/generated/dev-docs.json b/docs/data/generated/dev-docs.json index 0a8d39231..4571c236c 100644 --- a/docs/data/generated/dev-docs.json +++ b/docs/data/generated/dev-docs.json @@ -22,7 +22,7 @@ "url": "/dev/architecture/bindings/", "title": "Playbook bindings: model and threat model", "section": "Architecture", - "body": "# Playbook bindings: model and threat model\n\n**Status:** Approved foundation (documents shipped code: `archie-core-t2db.1`\nthrough `.5`, `.7`, `.9`-`.15`)\n**Date:** 2026-09-03\n\nThis document covers the binding model and the intake threat model portion\nof `archie-core-t2db.6`. It does not cover the `ARCHITECTURE.md` task-lifecycle\naddendum or the operator walkthrough -- those remain separate, open pieces of\nthe same ticket.\n\n## What a binding is\n\nA `binding` (`internal/domain/binding.Binding`) ties four things together:\na **matcher** (which captured events it applies to), a **mapping**\n(`internal/domain/mapping.Mapping`, how fields are pulled out of a payload),\na **workflow** (which registered workflow runs on a match), and a **state\nmachine** governing whether it is live.\n\n```go\ntype Binding struct {\n ID, Name // identity\n Matcher // { Source string } -- the webhook path segment, e.g. \"sentry\"\n MappingID // which Mapping resolves payload fields\n Workflow // registered workflow name to dispatch to\n Version // bumped on every edit\n Status // draft | pending_approval | armed\n Secret // HMAC-SHA256 shared secret, encrypted at rest, never returned by GET\n}\n```\n\nBindings live in the store (SQLite), not `config.toml` -- operators author\nthem from the dashboard while the daemon runs, without a restart.\n\n### Matching\n\n`Matcher.Source` is the only predicate today: the path segment a sender\nPOSTs to (e.g. a webhook hitting `.../sentry` matches bindings with\n`Matcher.Source == \"sentry\"`). The storage column is plain `TEXT`, so adding\npredicates later is non-breaking -- extend `Matcher`, don't invent a second\nmatcher shape.\n\n### Mapping\n\n`mapping.Resolve(fields, payload)` walks a captured JSON payload against a\nlist of typed `Field`s (name, JSON path, expected type), returning resolved\nvalues plus a `[]Failure` for anything that didn't parse or type-match. This\nis schema-by-example in practice: a mapping is authored by pointing at a\nreal captured payload in the dashboard, not by guessing a schema in advance.\n\n### Versioning\n\n`Version` increments on every `UpdateBinding` call. This exists so an\neditor's later change can never silently rewrite the historical provenance\nof a task that already fired -- a dispatched task's binding reference is\npinned to the version that was armed when it fired, not \"whatever the\nbinding currently says.\"\n\n### State machine\n\n```\ndraft ---\u003e pending_approval ---\u003e armed\n ^ ^ |\n | | |\n +--------------+-----------------+\n (any edit to an armed binding drops it back to pending_approval)\n```\n\nModeled directly on Telegram's existing `dangerousAction`/`pendingApproval`\nflow (`internal/channels/telegram/dangerous.go`, `approval.go`) -- not a new\napproval mechanism invented for this feature. **Only `armed` evaluates\nagainst incoming events** (`binding.Matches`, `internal/domain/binding/\nbinding.go`); `draft` and `pending_approval` bindings are inert. An edit to\nan already-armed binding cannot bypass approval by silent mutation -- it\ndrops back to `pending_approval` and requires an explicit `Approve` call to\nre-arm.\n\n## The threat model\n\nThis is a public, unauthenticated-by-default intake surface whose output is\nan agent with a repo, a worktree, and tool access. The design brief\n(`archie-core-t2db.5`, `docs/prds/webhook-intake-security.md`) named five\nnon-negotiables; each is verified below against the actual shipped code,\nnot the design intent.\n\n### 1. Per-source authentication\n\nReuses `internal/channels/webhook`'s existing HMAC-SHA256 scheme --\ncapture accepts unsigned events (visible in the dashboard, marked\nunauthenticated), but **only authenticated events can ever match a\nbinding**. `binding.Matches`' doc comment states this plainly: \"an\nunauthenticated event can never trigger a binding, no matter how well it\nwould otherwise match.\" Enforced structurally at two points, not just\nconvention:\n\n- `captured_events.authenticated` is a real column (`internal/store/\n captures.go`), set at capture time.\n- `dispatchBindings` (`internal/daemon/daemon.go:479`) walks **authenticated\n captures whose source has at least one armed binding** -- an\n unauthenticated capture is never even considered for dispatch.\n\n### 2. Human approval before a binding goes live\n\nCovered by the state machine above. Nothing self-arms: `pending_approval -\u003e\narmed` requires an explicit `Approve` call, and any edit to an armed\nbinding reverts it.\n\n### 3. Rate and volume limits per source\n\nCapture writes are bounded independently of the approval gate, so a flood\nof unauthenticated traffic cannot exhaust disk or drown the inspector:\n`InsertCapture` (`internal/store/captures.go`) prunes by both **retention\nwindow** (`WHERE received_at \u003c cutoff`) and **max event count** in the same\ninsert transaction.\n\n### 4. Blast radius bounded by the same code-level controls every task gets\n\nA playbook-originated task is dispatched into the **same** gate, diff-cap,\nsandboxed-container, worktree-isolation pipeline every task goes through\nregardless of origin. Verified precisely, not assumed: `dispatchOneBinding`\ncalls `EnqueueBindingTask` (`internal/store/store.go`), which wraps\n`EnqueueChatTask` -- the same direct-to-`tasks`-table enqueue path\nchat-spawned tasks already use, then stamps `binding_id`/`binding_version`\nfor provenance in a second statement. This is a **different** enqueue\nmechanism from forge-issue polling's `workintake.TaskEnvelope`/NATS path\n(`docs/prds/event-sources-and-reactions.md`) -- binding and forge-issue\ntasks arrive by different producers, but both become an ordinary\n`store.Task` row that `ClaimNext` picks up identically, so both get the\nsame gate/worktree pipeline downstream regardless of which producer created\nthe row. Payload content is\nordinary untrusted prompt input; the sender's text is never treated as a\ntrusted instruction (Archie's environmental-enforcement principle: \"the\nagent should ignore instructions in the payload\" is not itself a control,\nthe surrounding sandbox is).\n\n### 5. Captured payloads may contain secrets from the sending system\n\nTwo independent protections, not one:\n\n- **Redaction before persistence**: `webhookguard.RedactPayload` walks the\n decoded JSON and redacts recognised sensitive keys before the row is\n written -- the raw, unredacted payload is never durably stored.\n- **Retention**: a configurable window (`InsertCapture`'s `retention`\n parameter) prunes old captures automatically; visibility of what remains\n is gated by the same dashboard authentication as every other webui\n surface, not a separate, weaker path.\n\n### Secrets at rest\n\n`Binding.Secret` (the HMAC shared secret a sender signs with) is encrypted\nat rest via AES-GCM (`internal/store/binding_cipher.go`, `archie-core-\nt2db.7`) -- a leaked SQLite file does not leak sender-side shared secrets in\nplaintext. The envelope is versioned (`bindingEnvelopeVersion`) so a future\ncipher or KDF change can roll forward without breaking already-encrypted\nrows, and the AAD binds ciphertext to the binding-secret context so it\ncannot be relocated to another field/column and still authenticate. A `nil`\ncipher (no encryption key configured) leaves secrets as plaintext --\ndocumented legacy behaviour, not silent degradation.\n\n### At-most-once dispatch\n\n`dispatchBindings` enqueues at most one task per `(binding, capture)` pair,\nenforced by a real ledger, not best-effort application logic:\n`RecordDispatch` (`internal/store/bindings.go`) does `INSERT OR IGNORE` into\n`binding_dispatches`, unique on `(binding_id, capture_id)`; a duplicate\ninsert returns the sentinel `ErrAlreadyDispatched`, which the dispatch loop\ntreats as \"another cycle raced us,\" not an error. This guarantee holds\n**across daemon restarts**, not just within one process's lifetime, because\nthe ledger is a durable table, not in-memory state.\n\nThis ledger is also the direct precedent for the EDA playbook engine's\nexecution-time idempotency, now landed (`archie-core-t2db.17` closed): the\n`playbook_dispatches` table in `internal/store/playbook_dispatches.go` copies\n`binding_dispatches`'s `INSERT OR IGNORE` conventions with a\n`(playbook_id, playbook_version, event_id, action_id)` key\n(`docs/prds/eda-playbook-engine.md` gap 2).\n\n## What this document does not cover\n\n- The `ARCHITECTURE.md` task-lifecycle section noting that a\n playbook-originated task has different provenance from a forge-issue\n task -- open, part of `archie-core-t2db.6`.\n- An end-to-end operator walkthrough (point an external app at archie,\n build a first playbook) -- open, part of `archie-core-t2db.6`.\n- The playbook YAML / CEL / Module/Channel/Forge action-position work\n (`docs/prds/eda-playbook-engine.md`, `module-position.md`,\n `channel-forge-positions.md`) -- a different, newer mechanism than the\n binding model described here; the two are related but not the same\n system, and this document does not attempt to unify them.\n" + "body": "# Playbook bindings: model and threat model\n\n**Status:** Approved foundation (documents shipped code: `archie-core-t2db.1`\nthrough `.5`, `.7`, `.9`-`.15`)\n**Date:** 2026-09-03\n\nThis document covers the binding model and the intake threat model portion\nof `archie-core-t2db.6`. It does not cover the task-lifecycle provenance rule\n(`messaging-and-work-intake.md`'s \"Intake provenance\") or the operator\nwalkthrough -- that walkthrough remains an open piece of the same ticket.\n\n## What a binding is\n\nA `binding` (`internal/domain/binding.Binding`) ties four things together:\na **matcher** (which captured events it applies to), a **mapping**\n(`internal/domain/mapping.Mapping`, how fields are pulled out of a payload),\na **workflow** (which registered workflow runs on a match), and a **state\nmachine** governing whether it is live.\n\n```go\ntype Binding struct {\n ID, Name // identity\n Matcher // { Source string } -- the webhook path segment, e.g. \"sentry\"\n MappingID // which Mapping resolves payload fields\n Workflow // registered workflow name to dispatch to\n Version // bumped on every edit\n Status // draft | pending_approval | armed\n Secret // HMAC-SHA256 shared secret, encrypted at rest, never returned by GET\n}\n```\n\nBindings live in the store (SQLite), not `config.toml` -- operators author\nthem from the dashboard while the daemon runs, without a restart.\n\n### Matching\n\n`Matcher.Source` is the only predicate today: the path segment a sender\nPOSTs to (e.g. a webhook hitting `.../sentry` matches bindings with\n`Matcher.Source == \"sentry\"`). The storage column is plain `TEXT`, so adding\npredicates later is non-breaking -- extend `Matcher`, don't invent a second\nmatcher shape.\n\n### Mapping\n\n`mapping.Resolve(fields, payload)` walks a captured JSON payload against a\nlist of typed `Field`s (name, JSON path, expected type), returning resolved\nvalues plus a `[]Failure` for anything that didn't parse or type-match. This\nis schema-by-example in practice: a mapping is authored by pointing at a\nreal captured payload in the dashboard, not by guessing a schema in advance.\n\n### Versioning\n\n`Version` increments on every `UpdateBinding` call. This exists so an\neditor's later change can never silently rewrite the historical provenance\nof a task that already fired -- a dispatched task's binding reference is\npinned to the version that was armed when it fired, not \"whatever the\nbinding currently says.\"\n\n### State machine\n\n```\ndraft ---\u003e pending_approval ---\u003e armed\n ^ ^ |\n | | |\n +--------------+-----------------+\n (any edit to an armed binding drops it back to pending_approval)\n```\n\nModeled directly on Telegram's existing `dangerousAction`/`pendingApproval`\nflow (`internal/channels/telegram/dangerous.go`, `approval.go`) -- not a new\napproval mechanism invented for this feature. **Only `armed` evaluates\nagainst incoming events** (`binding.Matches`, `internal/domain/binding/\nbinding.go`); `draft` and `pending_approval` bindings are inert. An edit to\nan already-armed binding cannot bypass approval by silent mutation -- it\ndrops back to `pending_approval` and requires an explicit `Approve` call to\nre-arm.\n\n## The threat model\n\nThis is a public, unauthenticated-by-default intake surface whose output is\nan agent with a repo, a worktree, and tool access. The design brief\n(`archie-core-t2db.5`, `docs/prds/webhook-intake-security.md`) named five\nnon-negotiables; each is verified below against the actual shipped code,\nnot the design intent.\n\n### 1. Per-source authentication\n\nReuses `internal/channels/webhook`'s existing HMAC-SHA256 scheme --\ncapture accepts unsigned events (visible in the dashboard, marked\nunauthenticated), but **only authenticated events can ever match a\nbinding**. `binding.Matches`' doc comment states this plainly: \"an\nunauthenticated event can never trigger a binding, no matter how well it\nwould otherwise match.\" Enforced structurally at two points, not just\nconvention:\n\n- `captured_events.authenticated` is a real column (`internal/store/\n captures.go`), set at capture time.\n- `dispatchBindings` (`internal/daemon/daemon.go:479`) walks **authenticated\n captures whose source has at least one armed binding** -- an\n unauthenticated capture is never even considered for dispatch.\n\n### 2. Human approval before a binding goes live\n\nCovered by the state machine above. Nothing self-arms: `pending_approval -\u003e\narmed` requires an explicit `Approve` call, and any edit to an armed\nbinding reverts it.\n\n### 3. Rate and volume limits per source\n\nCapture writes are bounded independently of the approval gate, so a flood\nof unauthenticated traffic cannot exhaust disk or drown the inspector:\n`InsertCapture` (`internal/store/captures.go`) prunes by both **retention\nwindow** (`WHERE received_at \u003c cutoff`) and **max event count** in the same\ninsert transaction.\n\n### 4. Blast radius bounded by the same code-level controls every task gets\n\nA playbook-originated task is dispatched into the **same** gate, diff-cap,\nsandboxed-container, worktree-isolation pipeline every task goes through\nregardless of origin. Verified precisely, not assumed: `dispatchOneBinding`\ncalls `EnqueueBindingTask` (`internal/store/store.go`), which wraps\n`EnqueueChatTask` -- the same direct-to-`tasks`-table enqueue path\nchat-spawned tasks already use, then stamps `binding_id`/`binding_version`\nfor provenance in a second statement. This is a **different** enqueue\nmechanism from forge-issue polling's `workintake.TaskEnvelope`/NATS path\n(`docs/prds/event-sources-and-reactions.md`) -- binding and forge-issue\ntasks arrive by different producers, but both become an ordinary\n`store.Task` row that `ClaimNext` picks up identically, so both get the\nsame gate/worktree pipeline downstream regardless of which producer created\nthe row. Payload content is\nordinary untrusted prompt input; the sender's text is never treated as a\ntrusted instruction (Archie's environmental-enforcement principle: \"the\nagent should ignore instructions in the payload\" is not itself a control,\nthe surrounding sandbox is).\n\n### 5. Captured payloads may contain secrets from the sending system\n\nTwo independent protections, not one:\n\n- **Redaction before persistence**: `webhookguard.RedactPayload` walks the\n decoded JSON and redacts recognised sensitive keys before the row is\n written -- the raw, unredacted payload is never durably stored.\n- **Retention**: a configurable window (`InsertCapture`'s `retention`\n parameter) prunes old captures automatically; visibility of what remains\n is gated by the same dashboard authentication as every other webui\n surface, not a separate, weaker path.\n\n### Secrets at rest\n\n`Binding.Secret` (the HMAC shared secret a sender signs with) is encrypted\nat rest via AES-GCM (`internal/store/binding_cipher.go`, `archie-core-\nt2db.7`) -- a leaked SQLite file does not leak sender-side shared secrets in\nplaintext. The envelope is versioned (`bindingEnvelopeVersion`) so a future\ncipher or KDF change can roll forward without breaking already-encrypted\nrows, and the AAD binds ciphertext to the binding-secret context so it\ncannot be relocated to another field/column and still authenticate. A `nil`\ncipher (no encryption key configured) leaves secrets as plaintext --\ndocumented legacy behaviour, not silent degradation.\n\n### At-most-once dispatch\n\n`dispatchBindings` enqueues at most one task per `(binding, capture)` pair,\nenforced by a real ledger, not best-effort application logic:\n`RecordDispatch` (`internal/store/bindings.go`) does `INSERT OR IGNORE` into\n`binding_dispatches`, unique on `(binding_id, capture_id)`; a duplicate\ninsert returns the sentinel `ErrAlreadyDispatched`, which the dispatch loop\ntreats as \"another cycle raced us,\" not an error. This guarantee holds\n**across daemon restarts**, not just within one process's lifetime, because\nthe ledger is a durable table, not in-memory state.\n\nThis ledger is also the direct precedent for the EDA playbook engine's\nexecution-time idempotency, now landed (`archie-core-t2db.17` closed): the\n`playbook_dispatches` table in `internal/store/playbook_dispatches.go` copies\n`binding_dispatches`'s `INSERT OR IGNORE` conventions with a\n`(playbook_id, playbook_version, event_id, action_id)` key\n(`docs/prds/eda-playbook-engine.md` gap 2).\n\n## What this document does not cover\n\n- An end-to-end operator walkthrough (point an external app at archie,\n build a first playbook) -- open, part of `archie-core-t2db.6`.\n- The playbook YAML / CEL / Module/Channel/Forge action-position work\n (`docs/prds/eda-playbook-engine.md`, `module-position.md`,\n `channel-forge-positions.md`) -- a different, newer mechanism than the\n binding model described here; the two are related but not the same\n system, and this document does not attempt to unify them.\n" }, { "url": "/dev/architecture/configuration/", @@ -58,7 +58,7 @@ "url": "/dev/architecture/messaging-and-work-intake/", "title": "Messaging and Work Intake", "section": "Architecture", - "body": "# Messaging and Work Intake\n\n**Status:** Current state researched; foundation decisions in progress \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie is a personal assistant and general agentic system. Its interactions\ninclude conversation, questions, notes, reminders, goals, message handling,\ntroubleshooting, infrastructure operations, server management, source changes,\nand open-ended future capabilities.\n\nA message to an agent is therefore not inherently a work request, and invoking\nan agent is not inherently a WorkflowExecution.\n\nMessaging and work intake are separate from the Agent System's durable workflow\nexecution model:\n\n- messaging preserves interactions with people and external systems;\n- an agent turn interprets and responds to an interaction using its available\n capabilities;\n- work intake handles the optional promotion of an interaction into durable,\n workflow-backed work;\n- the Agent System owns Workflow definitions and creates WorkflowExecutions.\n\nForge issues, Jira issues, Telegram messages, emails, webhooks, and future\nchannels remain peer external message or artifact sources. None becomes an\nexecution model merely because it can lead to work.\n\n## Confirmed boundary\n\nWork intake MUST NOT create a WorkflowExecution directly.\n\nInbound interactions normally route to the Agent selected by the channel\nbinding. The selected Agent retains its own memory, user knowledge, conversation\ncontinuity, specialisation, and capability context. This continuity is\nindependent of the underlying model or provider.\n\nEach selected Agent acts through its own immutable IdentityID. Interaction and\nresulting action records preserve both that acting identity and the initiating\nuser identity when one exists.\n\nAgent memory is private by default. Interaction context MAY additionally use\nuser-wide memory and intentionally global shared memory, but another Agent's\nagent-wide memory is not part of the selected Agent's context.\n\nMultiple users may interact with the same Agent. The channel binding and inbound\nmessage MUST resolve both AgentID and initiating User IdentityID before\nconversation or memory context is loaded. Sharing an Agent MUST NOT implicitly\nshare one user's private memory with another user.\n\nAn interaction may assemble memory from four scopes: global shared, the selected\nAgent's agent-wide memory, the initiating user's authorized user-wide memory,\nand the private Agent-user relationship memory addressed by both IDs.\n\n## Conversation identity\n\nEvery conversation is identified by the complete tuple:\n\n```text\nAgentID\n+ UserIdentityID\n+ ChannelBindingID\n+ ExternalConversationID\n+ ThreadID\n```\n\n`ThreadID` has an explicit empty value for channels without threads. No element\nof this key may be replaced by a display name, configured bot username, channel\ntype, or other mutable label.\n\nAgent and user memory continue across conversations according to their memory\nscope. Immediate conversation history and delivery correlation remain isolated\nto the identified channel conversation or thread. Moving between Telegram,\nemail, or another channel does not create a new Agent or user, but it does\ncreate a distinct conversation unless an explicit linking operation relates\nthem.\n\nConversation and message events support projections for channel-entry metrics,\nuse cases, volumes, outcomes, latency, capability usage, and other product or\noperational analysis. Metrics are derived read models; they MUST NOT become\nmutable counters embedded in the conversation aggregate or change message\nrouting semantics.\n\n## Canonical messages\n\nEvery inbound, assistant, tool-call, and tool-result message has an immutable\napplication-generated MessageID. Platform identifiers are external correlation,\nnot substitutes for the canonical ID.\n\nMessages are addressable because an interaction is more than text. A message may\ncontain assistant content, reasoning data subject to its retention policy, one\nor more tool calls, a tool result with authoritative execution metadata,\nattachments, reply relationships, edits, or other typed content. Stable\nMessageIDs also allow sessions to branch from an exact prior point.\n\nThe canonical session and message implementation must supply:\n\n- application-generated stable message IDs;\n- user, assistant, and tool roles;\n- assistant tool calls and correlated tool-result messages;\n- authoritative tool-result status, timing, error, truncation, and size\n metadata;\n- session lifecycle and active-request state;\n- session-to-agent-instance and parent-session lineage;\n- durable SQLite round trips and timestamp validation.\n\nThese semantics must use the confirmed Agent, user, channel-binding, and\nconversation ownership model.\n\nThe canonical message store MUST preserve messages as immutable records and\nrepresent later edits, deletions, and branches through related records or\nevents.\n\nArchie requires exact branch-point correlation using stable MessageIDs.\n\n## Conversation branches\n\nA branch is a continuation of a previous conversation at an exact message. A\nchild conversation records:\n\n```text\nParentConversationID\nForkMessageID\n```\n\nMessages remain an append-only linear sequence within each conversation. The\nchild's effective initial context is the parent's history through\n`ForkMessageID`; messages after that point in the parent are not inherited.\n\nAfter the fork, parent and child are isolated:\n\n- child messages, tool calls, tool results, capability actions, and errors do\n not appear in the parent conversation;\n- later parent messages do not silently enter the child;\n- returning to the parent resumes its unchanged history;\n- the child may itself be branched using the same rules;\n- branch lineage remains queryable for navigation, audit, and metrics.\n\nThis supports building substantial context about a subject and then performing\none or more contextual subtasks without polluting the original conversation. For\nexample, a user may branch from a project discussion, create Jira issues in the\nchild using inherited context, and then return to the project discussion without\nadding the Jira tool transcript to the parent.\n\nBranching does not copy mutable message rows. History is resolved through the\nimmutable parent conversation and fork MessageID, followed by the child's own\nmessage sequence.\n\nBranch isolation applies to conversation history, not to Agent learning or\nmemory. A memory action performed in a branch has exactly the same semantics as\nthe same action performed in any other conversation. It writes to its selected\nglobal, agent-wide, user-wide, or Agent-user relationship scope.\n\nMemory provenance MAY retain the source ConversationID and MessageIDs for\nexplanation and audit. Retrieval and effect are determined by memory scope and\naccess rules, not by the conversation from which the memory originated.\nReturning to a parent does not insert the child's tool transcript into the\nparent history, but scoped memories learned in the child remain available to\nfuture retrieval wherever that scope is visible.\n\nChanging sessions or branching therefore controls immediate conversational\ncontext without preventing the Agent from learning durable knowledge.\n\n## Target ownership\n\nThe Messaging domain lives at `internal/domain/messaging`. It owns:\n\n- canonical Messages and typed message content;\n- Conversations, branches, and lineage;\n- channel bindings and external source correlation;\n- replies, acknowledgements, and outbound delivery attempts;\n- message and conversation lifecycle events;\n- domain-defined repository and delivery service contracts;\n- projections required for conversation navigation and metrics.\n\nWork intake is a separate cohesive domain at `internal/domain/workintake`. It\nowns admission, validation, deterministic routing, and accepted work requests\nfor the optional transition into durable workflow-backed execution.\n\nTelegram, email, webhook, forge, Jira, and future messaging systems are\ninfrastructure adapters implementing Messaging contracts. They are users of the\ndomain; they do not own Messages, Conversations, work admission, or\nWorkflowExecution semantics.\n\nThe Agent decides whether an ordinary interaction is answered directly, uses a\ncapability, changes or retrieves Agent-owned information, or proposes durable\nworkflow-backed work.\n\nExplicitly configured automation MAY enter work intake without a conversational\nAgent turn when the external trigger and admission rule already unambiguously\nrequest durable work. This avoids spending an LLM turn to rediscover a\ndeterministic routing decision.\n\nWhen an interaction requires durable workflow-backed execution:\n\n1. a channel adapter produces a channel-neutral inbound message;\n2. the acting agent handles or interprets the interaction;\n3. work intake validates, admits, and routes a proposed work request;\n4. work intake emits an accepted work request with its source correlation,\n acting identity, requested outcome, inputs, and routing evidence;\n5. the Agent System consumes that request, selects a specific Workflow version,\n and creates the WorkflowExecution.\n\nThe accepted work request is a cross-domain handoff, not an alternate execution\naggregate. The Agent System is the sole authority that creates a valid\nWorkflowExecution.\n\nAn interaction MAY instead be completed directly by an agent turn, delegated to\na tool or capability, recorded as a note or reminder, answered conversationally,\nor retained for later context. Those paths MUST NOT require a Workflow or\nWorkflowExecution.\n\nA Workflow MAY add reusable, versioned structure to any of those behaviours.\nScheduled personal-assistant routines, message triage, reminders, infrastructure\noperations, and other capability sequences are valid Workflow definitions. The\ndistinction is whether durable structured execution is useful, not whether the\nsubject originated from a forge.\n\n## Canonical execution language\n\n- **Workflow**: reusable, versioned behaviour definition.\n- **WorkflowExecution**: durable execution of a specific Workflow version.\n- **WorkflowStep**: a defined operation in a Workflow.\n- **StepExecution**: the recorded execution and outcome of a WorkflowStep.\n- **Attempt**: a retry within the same WorkflowExecution.\n\nExternal issues, messages, requests, goals, and notes are not synonyms for these\nconcepts.\n\n## Current implementation\n\nCurrent intake is fragmented:\n\n- Telegram, email, and webhook adapters translate payloads into\n `gateway.Message`, while an unused richer `MessageEvent` contract exists\n alongside it.\n- `gateway.Router` combines local channel commands, conversational LLM routing,\n and direct task lifecycle operations.\n- Telegram is the only channel with production LLM and durable conversation\n wiring. Email and webhook free text currently have no LLM responder.\n- Forge polling bypasses the gateway and directly enqueues store rows or NATS\n task messages.\n- Chat `/spawn` directly creates a `store.Task` and invents a synthetic forge\n issue number to satisfy the existing persistence key.\n- Forge replies and chat approval commands implement separate waiting-human\n paths.\n- Message delivery, acknowledgement, retry, deduplication, and source\n correlation are platform-specific side effects rather than one durable model.\n- `store.Task` collapses an external source reference and durable execution\n state.\n\n## Current hazards\n\n- Treating every agent interaction as a work request would remove Archie's\n personal-assistant role and force conversation and immediate capability use\n through an inappropriate workflow lifecycle.\n- Treating every accepted work request as a WorkflowExecution would let intake\n or channel code select versions and construct invalid execution state outside\n the Agent System.\n- Source identity is incomplete: sessions use configured bot usernames, tasks\n use identity strings, and external message IDs are generally absent.\n- SQLite and NATS deduplication are based on forge coordinates rather than a\n channel-neutral source-message reference.\n- Replies and acknowledgements have no durable delivery-attempt lifecycle, so\n partial success and retry behaviour differ by adapter.\n- Production code has two competing channel-neutral message contracts.\n- The existing Archie session implementation lacks the required tool-call\n semantics and exact branch lineage and MUST be brought to the canonical\n contract.\n\n## Decisions still required\n\n- the cohesive owner and final location of messaging behaviour;\n- the cohesive owner and final location of work-intake admission and routing;\n- the canonical inbound message, conversation, and source-correlation model;\n- the boundary between an agent turn, direct capability use, and a proposed\n durable work request;\n- the exact relationship between an Agent, its acting IdentityID, its user, and\n its channel bindings;\n- outbound response, acknowledgement, delivery-attempt, retry, deduplication,\n and failure semantics;\n- the smallest accepted-work-request contract consumed by the Agent System;\n- migration of forge polling, replies, chat commands, email, webhook, Telegram,\n and Jira integrations.\n\n## Non-goal: semantic search over chat messages\n\nSemantic/vector search over chat messages is an **explicit non-goal** until usage\ndata shows a real need.\n\nMessage search is lexical only: session-scoped SQLite FTS5 queries paged via\n`MessageQuery{Query,Limit,Offset}` → `MessagePage`. Do not add embeddings\nconfiguration or vector columns to message records.\n\nThe rationale is cost without a demonstrated use case. Measured 2026-08-03 with\n`embeddinggemma:300m` — note this was never an infrastructure problem, the model\nwas already pulled and `ai-sdk/provider/ollama` already has `embed.go`:\n\n- 121 ms median warm per message on the write path, 12.5 s cold;\n- 768 dims = 3072 bytes per vector, roughly 15× a typical message payload;\n- chronological reads would carry vector data they do not use on the hot path\n of every turn;\n- similarity search requires an explicit indexing and retention design.\n\nIf this is revisited, vectors belong in a sibling table so chronological reads\nnever load them.\n" + "body": "# Messaging and Work Intake\n\n**Status:** Current state researched; foundation decisions in progress \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie is a personal assistant and general agentic system. Its interactions\ninclude conversation, questions, notes, reminders, goals, message handling,\ntroubleshooting, infrastructure operations, server management, source changes,\nand open-ended future capabilities.\n\nA message to an agent is therefore not inherently a work request, and invoking\nan agent is not inherently a WorkflowExecution.\n\nMessaging and work intake are separate from the Agent System's durable workflow\nexecution model:\n\n- messaging preserves interactions with people and external systems;\n- an agent turn interprets and responds to an interaction using its available\n capabilities;\n- work intake handles the optional promotion of an interaction into durable,\n workflow-backed work;\n- the Agent System owns Workflow definitions and creates WorkflowExecutions.\n\nForge issues, Jira issues, Telegram messages, emails, webhooks, and future\nchannels remain peer external message or artifact sources. None becomes an\nexecution model merely because it can lead to work.\n\n## Confirmed boundary\n\nWork intake MUST NOT create a WorkflowExecution directly.\n\nInbound interactions normally route to the Agent selected by the channel\nbinding. The selected Agent retains its own memory, user knowledge, conversation\ncontinuity, specialisation, and capability context. This continuity is\nindependent of the underlying model or provider.\n\nEach selected Agent acts through its own immutable IdentityID. Interaction and\nresulting action records preserve both that acting identity and the initiating\nuser identity when one exists.\n\nAgent memory is private by default. Interaction context MAY additionally use\nuser-wide memory and intentionally global shared memory, but another Agent's\nagent-wide memory is not part of the selected Agent's context.\n\nMultiple users may interact with the same Agent. The channel binding and inbound\nmessage MUST resolve both AgentID and initiating User IdentityID before\nconversation or memory context is loaded. Sharing an Agent MUST NOT implicitly\nshare one user's private memory with another user.\n\nAn interaction may assemble memory from four scopes: global shared, the selected\nAgent's agent-wide memory, the initiating user's authorized user-wide memory,\nand the private Agent-user relationship memory addressed by both IDs.\n\n## Conversation identity\n\nEvery conversation is identified by the complete tuple:\n\n```text\nAgentID\n+ UserIdentityID\n+ ChannelBindingID\n+ ExternalConversationID\n+ ThreadID\n```\n\n`ThreadID` has an explicit empty value for channels without threads. No element\nof this key may be replaced by a display name, configured bot username, channel\ntype, or other mutable label.\n\nAgent and user memory continue across conversations according to their memory\nscope. Immediate conversation history and delivery correlation remain isolated\nto the identified channel conversation or thread. Moving between Telegram,\nemail, or another channel does not create a new Agent or user, but it does\ncreate a distinct conversation unless an explicit linking operation relates\nthem.\n\nConversation and message events support projections for channel-entry metrics,\nuse cases, volumes, outcomes, latency, capability usage, and other product or\noperational analysis. Metrics are derived read models; they MUST NOT become\nmutable counters embedded in the conversation aggregate or change message\nrouting semantics.\n\n## Canonical messages\n\nEvery inbound, assistant, tool-call, and tool-result message has an immutable\napplication-generated MessageID. Platform identifiers are external correlation,\nnot substitutes for the canonical ID.\n\nMessages are addressable because an interaction is more than text. A message may\ncontain assistant content, reasoning data subject to its retention policy, one\nor more tool calls, a tool result with authoritative execution metadata,\nattachments, reply relationships, edits, or other typed content. Stable\nMessageIDs also allow sessions to branch from an exact prior point.\n\nThe canonical session and message implementation must supply:\n\n- application-generated stable message IDs;\n- user, assistant, and tool roles;\n- assistant tool calls and correlated tool-result messages;\n- authoritative tool-result status, timing, error, truncation, and size\n metadata;\n- session lifecycle and active-request state;\n- session-to-agent-instance and parent-session lineage;\n- durable SQLite round trips and timestamp validation.\n\nThese semantics must use the confirmed Agent, user, channel-binding, and\nconversation ownership model.\n\nThe canonical message store MUST preserve messages as immutable records and\nrepresent later edits, deletions, and branches through related records or\nevents.\n\nArchie requires exact branch-point correlation using stable MessageIDs.\n\n## Conversation branches\n\nA branch is a continuation of a previous conversation at an exact message. A\nchild conversation records:\n\n```text\nParentConversationID\nForkMessageID\n```\n\nMessages remain an append-only linear sequence within each conversation. The\nchild's effective initial context is the parent's history through\n`ForkMessageID`; messages after that point in the parent are not inherited.\n\nAfter the fork, parent and child are isolated:\n\n- child messages, tool calls, tool results, capability actions, and errors do\n not appear in the parent conversation;\n- later parent messages do not silently enter the child;\n- returning to the parent resumes its unchanged history;\n- the child may itself be branched using the same rules;\n- branch lineage remains queryable for navigation, audit, and metrics.\n\nThis supports building substantial context about a subject and then performing\none or more contextual subtasks without polluting the original conversation. For\nexample, a user may branch from a project discussion, create Jira issues in the\nchild using inherited context, and then return to the project discussion without\nadding the Jira tool transcript to the parent.\n\nBranching does not copy mutable message rows. History is resolved through the\nimmutable parent conversation and fork MessageID, followed by the child's own\nmessage sequence.\n\nBranch isolation applies to conversation history, not to Agent learning or\nmemory. A memory action performed in a branch has exactly the same semantics as\nthe same action performed in any other conversation. It writes to its selected\nglobal, agent-wide, user-wide, or Agent-user relationship scope.\n\nMemory provenance MAY retain the source ConversationID and MessageIDs for\nexplanation and audit. Retrieval and effect are determined by memory scope and\naccess rules, not by the conversation from which the memory originated.\nReturning to a parent does not insert the child's tool transcript into the\nparent history, but scoped memories learned in the child remain available to\nfuture retrieval wherever that scope is visible.\n\nChanging sessions or branching therefore controls immediate conversational\ncontext without preventing the Agent from learning durable knowledge.\n\n## Target ownership\n\nThe Messaging domain lives at `internal/domain/messaging`. It owns:\n\n- canonical Messages and typed message content;\n- Conversations, branches, and lineage;\n- channel bindings and external source correlation;\n- replies, acknowledgements, and outbound delivery attempts;\n- message and conversation lifecycle events;\n- domain-defined repository and delivery service contracts;\n- projections required for conversation navigation and metrics.\n\nWork intake is a separate cohesive domain at `internal/domain/workintake`. It\nowns admission, validation, deterministic routing, and accepted work requests\nfor the optional transition into durable workflow-backed execution.\n\nTelegram, email, webhook, forge, Jira, and future messaging systems are\ninfrastructure adapters implementing Messaging contracts. They are users of the\ndomain; they do not own Messages, Conversations, work admission, or\nWorkflowExecution semantics.\n\nThe Agent decides whether an ordinary interaction is answered directly, uses a\ncapability, changes or retrieves Agent-owned information, or proposes durable\nworkflow-backed work.\n\nExplicitly configured automation MAY enter work intake without a conversational\nAgent turn when the external trigger and admission rule already unambiguously\nrequest durable work. This avoids spending an LLM turn to rediscover a\ndeterministic routing decision.\n\nWhen an interaction requires durable workflow-backed execution:\n\n1. a channel adapter produces a channel-neutral inbound message;\n2. the acting agent handles or interprets the interaction;\n3. work intake validates, admits, and routes a proposed work request;\n4. work intake emits an accepted work request with its source correlation,\n acting identity, requested outcome, inputs, and routing evidence;\n5. the Agent System consumes that request, selects a specific Workflow version,\n and creates the WorkflowExecution.\n\nThe accepted work request is a cross-domain handoff, not an alternate execution\naggregate. The Agent System is the sole authority that creates a valid\nWorkflowExecution.\n\nAn interaction MAY instead be completed directly by an agent turn, delegated to\na tool or capability, recorded as a note or reminder, answered conversationally,\nor retained for later context. Those paths MUST NOT require a Workflow or\nWorkflowExecution.\n\nA Workflow MAY add reusable, versioned structure to any of those behaviours.\nScheduled personal-assistant routines, message triage, reminders, infrastructure\noperations, and other capability sequences are valid Workflow definitions. The\ndistinction is whether durable structured execution is useful, not whether the\nsubject originated from a forge.\n\n## Canonical execution language\n\n- **Workflow**: reusable, versioned behaviour definition.\n- **WorkflowExecution**: durable execution of a specific Workflow version.\n- **WorkflowStep**: a defined operation in a Workflow.\n- **StepExecution**: the recorded execution and outcome of a WorkflowStep.\n- **Attempt**: a retry within the same WorkflowExecution.\n\nExternal issues, messages, requests, goals, and notes are not synonyms for these\nconcepts.\n\n## Current implementation\n\nCurrent intake is fragmented:\n\n- Telegram, email, and webhook adapters translate payloads into\n `gateway.Message`, while an unused richer `MessageEvent` contract exists\n alongside it.\n- `gateway.Router` combines local channel commands, conversational LLM routing,\n and direct task lifecycle operations.\n- Telegram is the only channel with production LLM and durable conversation\n wiring. Email and webhook free text currently have no LLM responder.\n- Forge polling bypasses the gateway and directly enqueues store rows or NATS\n task messages.\n- Chat `/spawn` directly creates a `store.Task` and invents a synthetic forge\n issue number to satisfy the existing persistence key.\n- Forge replies and chat approval commands implement separate waiting-human\n paths.\n- Message delivery, acknowledgement, retry, deduplication, and source\n correlation are platform-specific side effects rather than one durable model.\n- `store.Task` collapses an external source reference and durable execution\n state.\n\n## Intake provenance\n\nA task's origin determines which enqueue path created it, though every origin\nconverges on the same `tasks` table and the same `Route`/`Run` lifecycle once\nqueued.\n\n- Forge-sourced work -- discovered by polling or, in single-identity\n deployments, delivered by the forge webhook receiver (`internal/forge/webhook`)\n -- enqueues through the same `workintake.TaskEnvelope` and `PublishTask` path.\n `TaskEnvelope.IdempotencyKey` is keyed on `owner/repo/number` and not on the\n delivery source, so a webhook-delivered issue and a later poll of the same\n issue dedup through `PublishUnique` instead of double-enqueuing.\n- Chat-spawned and playbook-binding-dispatched work enqueues through\n `EnqueueChatTask`/`EnqueueBindingTask` (`internal/store`), stamped with\n `binding_id`/`binding_version` for provenance.\n\nWebhook intake is opt-in per deployment (`Forge.Intake` = `poll` (default),\n`webhook`, or `both`) and single-identity only: a config naming\n`webhook`/`both` alongside `[[identities]]`, or setting intake on a\n`[identities.forge]` entry, is rejected at validation rather than started with\nthe setting ignored. One receiver binds one secret and one dispatch predicate,\nso it cannot judge a second identity's events by the right rules. A\nmulti-identity deployment polls per identity until per-identity intake lands.\nThe decided contract is `docs/prds/event-sources-and-reactions.md`; the binding\nmodel and its threat model are `docs/architecture/bindings.md`.\n\n## Current hazards\n\n- Treating every agent interaction as a work request would remove Archie's\n personal-assistant role and force conversation and immediate capability use\n through an inappropriate workflow lifecycle.\n- Treating every accepted work request as a WorkflowExecution would let intake\n or channel code select versions and construct invalid execution state outside\n the Agent System.\n- Source identity is incomplete: sessions use configured bot usernames, tasks\n use identity strings, and external message IDs are generally absent.\n- SQLite and NATS deduplication are based on forge coordinates rather than a\n channel-neutral source-message reference.\n- Replies and acknowledgements have no durable delivery-attempt lifecycle, so\n partial success and retry behaviour differ by adapter.\n- Production code has two competing channel-neutral message contracts.\n- The existing Archie session implementation lacks the required tool-call\n semantics and exact branch lineage and MUST be brought to the canonical\n contract.\n\n## Decisions still required\n\n- the cohesive owner and final location of messaging behaviour;\n- the cohesive owner and final location of work-intake admission and routing;\n- the canonical inbound message, conversation, and source-correlation model;\n- the boundary between an agent turn, direct capability use, and a proposed\n durable work request;\n- the exact relationship between an Agent, its acting IdentityID, its user, and\n its channel bindings;\n- outbound response, acknowledgement, delivery-attempt, retry, deduplication,\n and failure semantics;\n- the smallest accepted-work-request contract consumed by the Agent System;\n- migration of forge polling, replies, chat commands, email, webhook, Telegram,\n and Jira integrations.\n\n## Non-goal: semantic search over chat messages\n\nSemantic/vector search over chat messages is an **explicit non-goal** until usage\ndata shows a real need.\n\nMessage search is lexical only: session-scoped SQLite FTS5 queries paged via\n`MessageQuery{Query,Limit,Offset}` → `MessagePage`. Do not add embeddings\nconfiguration or vector columns to message records.\n\nThe rationale is cost without a demonstrated use case. Measured 2026-08-03 with\n`embeddinggemma:300m` — note this was never an infrastructure problem, the model\nwas already pulled and `ai-sdk/provider/ollama` already has `embed.go`:\n\n- 121 ms median warm per message on the write path, 12.5 s cold;\n- 768 dims = 3072 bytes per vector, roughly 15× a typical message payload;\n- chronological reads would carry vector data they do not use on the hot path\n of every turn;\n- similarity search requires an explicit indexing and retention design.\n\nIf this is revisited, vectors belong in a sibling table so chronological reads\nnever load them.\n" }, { "url": "/dev/architecture/migration-decisions/", @@ -88,7 +88,7 @@ "url": "/dev/architecture/plugins-and-extensions/", "title": "Plugins and Extensions", "section": "Architecture", - "body": "# Plugins and Extensions\n\n**Status:** Approved foundation \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Plugin framework\n\nThe core plugin domain lives under `internal/domain/plugin`. It owns:\n\n- plugin identity and metadata;\n- discovery and registration contracts;\n- lifecycle rules;\n- capability registration;\n- validation and compatibility rules.\n\nThe generic plugin contract remains metadata-only. Functionality is exposed\nthrough narrow typed capability contracts with an owning registry or manager.\n\nWorkflow plugins belong to the Workflow capability family. The Workflow domain\nat `internal/domain/workflow` owns their typed registration, validation,\nversioning, definition, and execution contracts. The generic plugin domain may\nprovide plugin identity, discovery, compatibility, and lifecycle mechanics, but\nit MUST NOT define Workflow semantics or expose untyped Workflow hooks.\n\n## Plugin implementations\n\nFirst-party plugins are self-contained vertical modules under\n`plugins/\u003cplugin\u003e`.\n\nA plugin:\n\n- owns its implementation, settings, templates, assets, and tests;\n- receives only explicitly supplied contracts and services;\n- MUST NOT receive daemon internals, a service locator, or unrestricted\n application access;\n- MUST NOT import arbitrary infrastructure or application composition code;\n- MAY use approved domain and shared application contracts;\n- MUST remain removable without changes to unrelated domains.\n\nPlugins may be internally cohesive like small services. They do not become\nnetwork services without a concrete security or operational reason.\n\n## Event sources and reactions\n\n`archie-core-7d5u.1` asked whether extensions should react to events\nin-process through typed capability families (this document's model) or\nout-of-process like a generic `DispatchEvent` dispatcher. Decided 2026-08-22\nin `docs/prds/event-sources-and-reactions.md`: in-process typed families,\nconfirming this document's model rather than amending it. The forge webhook\nreceiver (`internal/forge/webhook`, see `ARCHITECTURE.md`'s \"Webhook intake\"\nsection) and the EDA playbook engine's Module action position\n(`docs/prds/eda-playbook-engine.md`, `internal/domain/eda/module`) are the\nshipped instances of that decision -- each is a narrow typed contract with an\nowning registry, not a generic event hook. The playbook engine's Channel and\nForge action positions are designed but not yet implemented (open\ninvestigation `archie-core-t2db.19`); when they land they follow the same\nmodel, not a new one.\n\n## Curator surprisal sampling\n\n**Decided 2026-08-05, wave 1 landed 2026-09-05** — the curator engine family\n(epic [#435](https://github.com/samcharles93/archie-core/issues/435)) selects which memories deserve agentic attention via\nsurprisal-based sampling: score memories by how surprising they are and\nspend the sampled reasoner's agentic budget on the most surprising items.\n\nThe prerequisites are built: the embeddings capability\n([#436](https://github.com/samcharles93/archie-core/issues/436), `internal/domain/embedding` contract +\n`internal/infrastructure/embedding` implementation, config-driven via\n`models.embedding`) and the wave-1 `Sampler` strategy seam\n(`internal/domain/sampling` -- not `internal/curator`, see\n`docs/prds/curator-sampler-wave1.md` for the settled design and tracking)\nwith four\ncheap, embedding-free strategies (recency, random, all, staleness proxy).\nNo curator consumes a `Sampler` yet, and the embeddings client is likewise\ndeliberately inert: `setupEmbeddings` constructs it at boot and holds it on\n`b.embeddings`, but no production code reads that field yet, so a resolved\n`models.embedding` role currently produces a client whose only observable\neffect is a boot log line. Both are prerequisites waiting on #437, not\nunwired defects.\n\nStill deferred: [#437](https://github.com/samcharles93/archie-core/issues/437), the embedding-backed surprisal strategy, which\ndepends on the above and on [#407](https://github.com/samcharles93/archie-core/issues/407). Requirements for that deferred work,\nso it can be picked up without re-deriving the design:\n\n- A `Sampler` implementation behind the same interface as the cheap\n strategies; selection deterministic given fixed inputs.\n- Algorithm: sample up to a candidate cap; build a k-NN graph over embedding\n vectors; row-normalize the adjacency with self-loops; power-iterate to the\n stationary distribution; surprisal = −log(stationary probability of the\n nearest point); select the highest-surprisal items.\n- Degradation: content without an embedding is skipped; an embedding failure\n aborts sampling and falls back to free exploration — a sampling failure\n must never fail or crash a curator pass.\n- Cost bound: the graph build is quadratic in the candidate cap; the cap must\n be a named constant with a documented bound.\n" + "body": "# Plugins and Extensions\n\n**Status:** Approved foundation \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Plugin framework\n\nThe core plugin domain lives under `internal/domain/plugin`. It owns:\n\n- plugin identity and metadata;\n- discovery and registration contracts;\n- lifecycle rules;\n- capability registration;\n- validation and compatibility rules.\n\nThe generic plugin contract remains metadata-only. Functionality is exposed\nthrough narrow typed capability contracts with an owning registry or manager.\n\nWorkflow plugins belong to the Workflow capability family. The Workflow domain\nat `internal/domain/workflow` owns their typed registration, validation,\nversioning, definition, and execution contracts. The generic plugin domain may\nprovide plugin identity, discovery, compatibility, and lifecycle mechanics, but\nit MUST NOT define Workflow semantics or expose untyped Workflow hooks.\n\n## Plugin engine rule (strict)\n\n“Engine” is reserved for a typed, lifecycle-managed capability family. It is not\na synonym for a generic plugin hook. Every new plugin engine must satisfy all of\nthese invariants:\n\n1. **Typed domain contract.** The engine interface exposes real\n capability-specific operations in addition to identity metadata. Examples are\n resolving a secret, delivering a message, or running an agent. A plugin that\n only exposes `Name` and `Version` is metadata, not an engine.\n2. **Owning family manager.** The capability package owns a `Registry` or\n `Manager` that controls provider registration and discovery, initialization,\n or access. Providers do not register arbitrary callbacks directly on the\n daemon.\n3. **Lifecycle and health.** When an implementation owns resources or background\n work, its typed family contract must expose start, health, and stop\n semantics. The owning manager invokes them and defines failure isolation and\n shutdown ordering. Stateless providers may make these operations explicit\n no-ops.\n4. **Narrow host access.** Plugins receive the typed family registrar and only\n the host services their declared capability requires. They never receive\n `*daemon.Daemon`, a service locator, or an untyped hook map.\n5. **Metadata-only generic plugin contract.** `plugin.Plugin` remains limited to\n `Name()` and `Version()`. Domain hooks belong on typed engine interfaces;\n adding capability methods to the generic interface is forbidden.\n6. **Trust boundary stays explicit.** Explicitly trusted, operator-installed\n plugins—including third-party secret-engine plugins the operator chooses to\n trust—may run in-process and have daemon privileges. Repository-supplied code\n runs in the task container. Integrations that are not trusted as daemon code\n run out of process through MCP or a versioned protocol and, when a\n confidentiality or integrity boundary is required, inside a real container\n sandbox. A subprocess boundary alone is not a security sandbox, and calling\n Yaegi-loaded code a plugin does not isolate it.\n\n`internal/plugin/architecture_test.go` enforces the mechanically testable parts\nof this rule: the generic plugin method set, the shared agent-instruction\nsource, and the requirement that engine interfaces have domain behavior plus an\nowning registry or manager.\n\n### Memory engine family\n\nOne engine, addressed by four typed scopes -- `ScopeGlobal`, `ScopeAgent`,\n`ScopeUser`, `ScopeAgentUser` (`internal/domain/memory`, `Scope`/`ScopeKind`).\nA caller names exactly the scopes it may read or write; the engine applies no\naccess policy of its own, so isolation falls out mechanically because\n`ScopeAgent{a}` and `ScopeAgent{b}` are different storage keys. See\n`docs/prds/memory-engine-unification.md` for the decision this contract\nimplements; it superseded `internal/memory` (`MemoryProvider`/`Manager`),\ndeleted in that PRD's slice 5.\n\n`MemoryEngine` (`internal/domain/memory/contract.go`) is the typed family\ncontract: `Create`, `Get`, `Query`, `List`, `Update`, `Forget`, `Revisions` --\nCRUD with revisions retained, not just create/read/delete. `Update` supersedes\nrather than overwrites: the superseded state is appended to the scope's\n`HISTORY.md` before the live block is replaced, and `Update` carries no\n`Scope` field, so a record can never change scope. The family's content\nscanner (prompt-injection / sensitive-data patterns) is applied inside\n`Create` and `Update`, the single choke point every producer crosses.\n\n`internal/domain/memory.Registry` is the owning family registry: registration,\nstart/health/stop, shutdown ordering, and failure isolation, following the\nplugin engine rule above. `internal/infrastructure/memory.NewBuiltinEngine` is\nthe only registered engine today; it persists through\n`internal/infrastructure/memory/builtin`, a markdown store (`MEMORY.md`-shaped\nper-scope files) that owns its on-disk format end-to-end.\n\nTwo per-turn seams in `internal/gateway` connect a chat turn to the engine,\neach narrowed to exactly the operations it needs:\n\n- **Read** (`turn_memory.go`, `MemoryStore`): `prepareTurn` runs a\n scope-only `Query` over the resolved `Subject`'s readable scopes,\n synchronously and before the prompt is built. It never returns an error --\n a panicking or failing engine degrades to an empty `\u003cmemory\u003e` block rather\n than failing the turn -- and is bounded by a per-scope record limit and a\n byte cap on the rendered block.\n- **Write** (`turn_memory_tool.go`, `MemoryWriteStore`): four tools --\n `memory_create`, `memory_update`, `memory_delete`, `memory_list` -- built\n per turn onto the resolved `Subject`'s writable scopes, not registered once\n at boot. The model chooses the scope kind; it never chooses the agent or\n user id, which come from the turn's resolved `Subject` and have no schema\n field to override. `ScopeGlobal` is never model-writable.\n\n`boot.setupMemoryEngine` (`internal/app/archied/bootstrap.go`) registers and\nstarts the engine and must run before `setupGatewayChat`, which constructs the\nonly chat turn runner in production and captures the engine at construction\ntime (`internal/app/archied/composition_order_test.go` pins the ordering), and\nbefore `setupCurators`, whose `Registrar` captures the engine registry. The\nsession curator (`internal/infrastructure/sessioncurator`) writes derived\nrecords onto `ScopeAgentUser`, making it the engine's other live producer\nbesides the chat write path.\n\n## Plugin implementations\n\nFirst-party plugins are self-contained vertical modules under\n`plugins/\u003cplugin\u003e`.\n\nA plugin:\n\n- owns its implementation, settings, templates, assets, and tests;\n- receives only explicitly supplied contracts and services;\n- MUST NOT receive daemon internals, a service locator, or unrestricted\n application access;\n- MUST NOT import arbitrary infrastructure or application composition code;\n- MAY use approved domain and shared application contracts;\n- MUST remain removable without changes to unrelated domains.\n\nPlugins may be internally cohesive like small services. They do not become\nnetwork services without a concrete security or operational reason.\n\n## Event sources and reactions\n\n`archie-core-7d5u.1` asked whether extensions should react to events\nin-process through typed capability families (this document's model) or\nout-of-process like a generic `DispatchEvent` dispatcher. Decided 2026-08-22\nin `docs/prds/event-sources-and-reactions.md`: in-process typed families,\nconfirming this document's model rather than amending it. The forge webhook\nreceiver (`internal/forge/webhook`; see \"Intake provenance\" in\n`docs/architecture/messaging-and-work-intake.md`) and the EDA playbook engine's\nModule action position\n(`docs/prds/eda-playbook-engine.md`, `internal/domain/eda/module`) are the\nshipped instances of that decision -- each is a narrow typed contract with an\nowning registry, not a generic event hook. The playbook engine's Channel and\nForge action positions are designed but not yet implemented (open\ninvestigation `archie-core-t2db.19`); when they land they follow the same\nmodel, not a new one.\n\n## Curator surprisal sampling\n\n**Decided 2026-08-05, wave 1 landed 2026-09-05** — the curator engine family\n(epic [#435](https://github.com/samcharles93/archie-core/issues/435)) selects which memories deserve agentic attention via\nsurprisal-based sampling: score memories by how surprising they are and\nspend the sampled reasoner's agentic budget on the most surprising items.\n\nThe prerequisites are built: the embeddings capability\n([#436](https://github.com/samcharles93/archie-core/issues/436), `internal/domain/embedding` contract +\n`internal/infrastructure/embedding` implementation, config-driven via\n`models.embedding`) and the wave-1 `Sampler` strategy seam\n(`internal/domain/sampling` -- not `internal/curator`, see\n`docs/prds/curator-sampler-wave1.md` for the settled design and tracking)\nwith four\ncheap, embedding-free strategies (recency, random, all, staleness proxy).\nNo curator consumes a `Sampler` yet, and the embeddings client is likewise\ndeliberately inert: `setupEmbeddings` constructs it at boot and holds it on\n`b.embeddings`, but no production code reads that field yet, so a resolved\n`models.embedding` role currently produces a client whose only observable\neffect is a boot log line. Both are prerequisites waiting on #437, not\nunwired defects.\n\nStill deferred: [#437](https://github.com/samcharles93/archie-core/issues/437), the embedding-backed surprisal strategy, which\ndepends on the above and on [#407](https://github.com/samcharles93/archie-core/issues/407). Requirements for that deferred work,\nso it can be picked up without re-deriving the design:\n\n- A `Sampler` implementation behind the same interface as the cheap\n strategies; selection deterministic given fixed inputs.\n- Algorithm: sample up to a candidate cap; build a k-NN graph over embedding\n vectors; row-normalize the adjacency with self-loops; power-iterate to the\n stationary distribution; surprisal = −log(stationary probability of the\n nearest point); select the highest-surprisal items.\n- Degradation: content without an embedding is skipped; an embedding failure\n aborts sampling and falls back to free exploration — a sampling failure\n must never fail or crash a curator pass.\n- Cost bound: the graph build is quadratic in the candidate cap; the cap must\n be a named constant with a documented bound.\n" }, { "url": "/dev/architecture/policy/", @@ -202,7 +202,7 @@ "url": "/dev/prds/deep-research-capability/", "title": "Deep research capability — design", "section": "PRDs", - "body": "# Deep research capability — design\n\n**Status:** Draft\n**Date:** 2026-09-14\n**Scope:** A deep-research capability in archie-core: an orchestrator that\nspawns 1..N subagents to explore and research in parallel, collates their\nfindings, and returns one cited report. Grounded in web search and web fetch.\n**Beads issue:** none yet (this document is the input to it)\n**Depends on:** nothing shipped for slice 1; slice 2 depends on slice 1.\n\n## §0 Implementation status\n\nEvery claim in this document is mapped to what exists in the tree.\n\n| Claim | Status | Where it stands |\n|---|---|---|\n| `web_fetch` tool (URL → readable text) | **Implemented** | `internal/tools/webfetch/{tool.go,webfetch.go}`; registered in `internal/app/archied/bootstrap.go` `registerStandaloneTools`, disabled-by-config returns `nil` |\n| **Web search** (query → result list) | **Aspirational — zero code** | No search provider, no search tool, no config block anywhere in the tree or in `ai-sdk` v0.1.32 |\n| `agent.Subagent` — nested agent in a fresh context window | **Implemented** (dependency) | `github.com/samcharles93/ai-sdk@v0.1.32/agent/subagent.go`. Synchronous, non-streaming, own model/system/toolset, `MaxSteps` defaults to 10, nesting depth capped at 5, returns final text only |\n| Subagent used in production | **Implemented, one consumer** | `internal/app/agentworker/review.go` — the adversarial reviewer. That is the only non-test use of `Subagent` in the repo |\n| A **delegate/subagent tool** on the chat surface | **Aspirational — zero code** | No `subagent`/`delegate` tool name is registered anywhere |\n| **Fan-out orchestration** (N missions, one collator) | **Aspirational — zero code** | `agentloop`'s package doc states it runs *one* mission: \"orchestration across missions lives with the caller\". No caller does this yet |\n| Parallel-safe tool execution | **Aspirational — zero code** | The unreachable `internal/tools/dispatch.go` dispatch/gating engine (`ExecutionClass`, `DispatchSequential`/`DispatchConcurrent`/`ClassifyEntry`) was deleted as dead code (tools-core-1); live concurrency is ai-sdk's `GenerateOptions.MaxParallelToolCalls`, which archie-core never sets |\n| Per-result caps + spill-to-disk | **Implemented** | `internal/agentexec/toolset.go` `ToolLimits{MaxResultChars,SpillDir}`, applied in `toolExecute` via `tools.CapPayload` (`limit.go`); no **aggregate** turn cap by design (`toolset.go`). `config.ToolPolicy.SpillDir`; default cap 50k chars, spill dir deliberately not defaulted |\n| Tool guardrails (repeat-failure / no-progress) | **Implemented, not on the call path** | `internal/tools/guardrail.go` engine exists. Its only non-test caller is `dispatchOne` — which is itself unreachable — so **the chat path never consults it**, and the agent path only records per *stage* (`domain/workflow/agent.go`) |\n| Structured capture tools (model reports into a typed store, not prose) | **Implemented** | `internal/app/agentworker/review.go` `record_finding` / `record_checked` — `core.NewTypedTool` appending to a caller-owned slice. This is the pattern a research source ledger should reuse |\n| Curator engine (interval, declared tool set, lifecycle) | **Implemented, tools broken** | `internal/domain/curator`; a curator declaring tools is **refused** at registration (`registry.go`, no `ToolBuilder` implementation) and `curatorLLMRunner` (`internal/app/archied/main.go`) drops `ChatRequest.Tools`. Budgets are time+concurrency only (`runtime.go`) — **no token budget** |\n| Cron / scheduling + delivery router | **Implemented, wired into no binary** | `internal/domain/scheduling`, `internal/infrastructure/cronstore` (schema v2, `Kind` discriminator), `internal/infrastructure/crondelivery` — all present and green, but `go list -deps` for every `cmd/` contains none of them, and `cron`-expression firing arithmetic is unimplemented (`schedule.go`, \"slice 2\"). `ARCHITECTURE.md` still asserts \"**No daemon-internal cron.**\" |\n| Workflow engines (multi-stage, resumable) | **Implemented, worktree-bound** | `internal/domain/workflow` — every non-triage workflow opens with `StagePrepareWorktree()` and mounts `tc.Dir` into a container; the task persists after every stage (`workflow.go`). `triage.go` documents there is \"no cheaper directory-less classification primitive\" |\n| Long-running chat-initiated work with progress | **Partial** | Chat turns run **once, synchronously**, inline in the channel's turn runner (`internal/gateway/turn.go`). Chat can spawn a *workflow task* (`task_spawn`) which is asynchronous — but that path is worktree/PR-shaped |\n| **Operator-push delivery of an async result** | **Aspirational — zero code** | Nothing pushes a finished result to a chat. No channel subscribes to the event bus (Telegram only *publishes* `KindTurnCompleted`); `crondelivery.Courier` is the intended seam but has **no implementation and no caller**; `send_file` works only inside the live turn. The one out-of-band notification in the tree is feasibility's n8n webhook (`feasibility.go`) |\n| Citation storage / source ledger | **Aspirational — zero code** | No source, citation, or provenance store exists for research output |\n\n\u003e **Target state, not current state.** Everything marked aspirational below is\n\u003e design, not a description of the daemon as it runs.\n\n## Problem\n\nSam's requirement: *\"Deep research should spawn 1 or many subagents for\nexploration or research work. And they should collate all findings, they\nshould use web search/fetch features.\"*\n\nNone of the three primitives that sentence needs can be assembled from what\nships:\n\n1. **Web search does not exist.** `web_fetch` reads a URL you already have. A\n research loop's first act is to *discover* URLs from a query, and there is\n no tool, provider, or config for that — in archie-core or in the shared SDK.\n2. **Nothing can fan out.** `agent.Subagent` can nest *one* nested generation,\n synchronously, and only the PR reviewer calls it. There is no N-way fan-out,\n no bounded concurrency, no shared budget across children, no collation pass.\n `agentloop` explicitly refuses to orchestrate: it runs one mission and the\n caller owns orchestration — and no caller orchestrates.\n3. **A research run outlives a chat turn.** A chat turn is synchronous and\n inline: the operator waits, and the turn's own context window is what\n carries the result. A fan-out over N subagents with web search is minutes to\n tens of minutes of wall clock. It cannot be a turn.\n\nThe interesting part is not the loop. It is where the boundary sits: which\nexisting long-running mechanism this reuses, how citations survive collation\nwhen the thing being collated is model-written prose, and what stops a run\nfrom spending unbounded money.\n\n## What the run actually is\n\nSix stages, each of which must be a *stage* rather than a prompt instruction:\n\n1. **Frame.** One cheap model call turns the request into a research brief: the\n question, 3..N distinct sub-questions, and a per-sub-question budget. Output\n is structured (a capture tool), not prose — the fan-out width comes from\n this step, so it must be a number the code reads, not a sentence.\n2. **Fan out.** One subagent per sub-question, **in parallel**, with a bounded\n concurrency, each running a read-only research toolset (search + fetch +\n read) against its own fresh context window, each required to record its\n findings as *typed source records* before finishing, and each compressing\n what it returns.\n3. **Cross-check.** Deterministic, no model: group by canonical URL and by\n claim, flag single-sourced claims, mark conflicts. See D6.\n4. **Collate.** One model pass over the children's *conclusions and source\n records* — never their transcripts — that writes the report body against the\n pre-reconciled source set.\n5. **Citation pass.** Its own stage: validate every marker in the body resolves\n to a stored source, and every stored source was fetched successfully.\n6. **Render.** The citation list is rendered from the **source store**, not\n from the collator's prose, so a URL can only appear in a report if some\n subagent actually recorded it.\n\nStages 5 and 6 are the load-bearing decision, and they are what the commercial\nproducts' published failure modes point at: a synthesizer that is free to write\ncitations will invent them, and a citation that cannot be traced to a fetch is\nworse than no citation.\n\n## Prior art: what the disclosed designs and the open-source field say\n\nTwo external research passes inform this design — one on the commercial\nproducts (official engineering blogs, model docs, system cards, plus\npeer-reviewed benchmarks), one on open-source deep-research implementations\nread at pinned commits. The findings that changed decisions:\n\n**Only one fan-out unit buys capacity.** Four different units are in use:\nAnthropic fans out *agents with their own context windows*; Google fans out\n*queries*; Perplexity fans out *generated programs*; OpenAI uses pipeline-stage\nagents for query enrichment only. Query fan-out is cheap and buys breadth of\nretrieval; agent fan-out is what buys context capacity. Our step 2 is agent\nfan-out, deliberately, because context isolation is the thing a single GPU\ncannot otherwise buy.\n\n**Nobody collates section-by-section.** The open-source survey found one merge\nprompt over concatenated child summaries (`final_report_generation` in\n`langchain-ai/open_deep_research`), lead-agent synthesis (Anthropic), a\ncode-level reconcile (Perplexity), or single-model continuation. STORM is the\nonly demonstrated section-mapped merge. This confirms D3: one collation pass,\nnot a mapped merge.\n\n**Citations are lost at the merge in 8 of 10 surveyed repos.** Only `lajosdeme/mole`\nassigns citation markers **mechanically in code before the prompt is built** —\nits `internal/output/report.go` carries the comment *\"it cannot introduce a\ncitation, because the citation markers are assigned mechanically before the\nprompt is built\"* — and stores a verbatim `Quote` plus a `QuoteOffset` (byte\noffset into the fetched document) so a citation can be re-verified later.\nClaims whose quote is not found in the source are dropped by code\n(`internal/actors/mine.go`: `\"claim rejected: quote not found in source\"`).\nThis is the strongest available argument for D1, and `QuoteOffset` is worth\nadopting: it makes re-verification cheap and turns \"the model said so\" into\n\"the document says so, here\".\n\n**The failure is measurable and it is at the orchestrator.** An EMNLP 2026\nstudy localising errors across three top-ranked open-source deep-research\nsystems found **84.7% of final-report errors originate at the orchestrator,\nroughly 31% of them hallucinations and the rest citation mistakes**, with\ninformation passing \"through agents like a telephone game\"\n(https://arxiv.org/abs/2608.24306). Anthropic's own mitigations for this are a\nseparate **CitationAgent** pass after synthesis and a filesystem-artefact\npattern — subagents store work externally and pass back lightweight references\nrather than full outputs. Perplexity reached the same conclusion independently.\nBoth are adopted here: D1 (store, not prose) and a citation pass that is its\nown stage.\n\n**A cross-check stage that no vendor has.** Worker A and worker B never compare\nnotes in any disclosed design, and nothing in the published set *resolves*\nconflicts — Perplexity only \"notes\" them. Grouping findings by URL and by claim,\nflagging single-sourced claims, and marking unresolved conflicts is the\ncheapest correctness win available and is where a local implementation can beat\nthe frontier products rather than imitate them.\n\n**Fan-out is expensive in tokens, and on one GPU that is wall clock.** Anthropic\ndiscloses ~15x tokens for multi-agent versus ~4x for a single agent, and\nexplicit tier rules in the orchestrator prompt (1 agent / 3-10 tool calls;\n2-4 subagents / 10-15 calls each; \u003e10 subagents for complex research, 3-5\nspawned concurrently). On a single card, N concurrent workers contend for the\nsame weights and KV budget, so **wall clock scales roughly linearly with worker\ncount** — the isolation benefit survives, the throughput benefit does not. One\ninstrumented study puts **web search at 73% of wall-clock time on average (up to\n91%)** (https://cseweb.ucsd.edu/~yiying/2025_NIPS_ERW_Deep_Research_Perf_Study.pdf),\nwhich is the argument for fanning out *retrieval* aggressively and *generation*\nsparingly, and for overlapping generation with retrieval.\n\n**Frontier citation volume is not a target.** Frontier runs emit 41-113\ncitation URLs per query, while 3-13% of citation URLs are hallucinated and \"more\ncitations per query do not mean fewer errors per citation\"\n(https://arxiv.org/abs/2604.03173). Ten to twenty *verified* citations is a\ndefensible, evidence-backed target.\n\n**Do not port Perplexity's style.** Search-as-code-generation needs\nfrontier-class codegen against a bespoke SDK and an atomised self-owned search\nstack. Port the *intent* — deterministic fan-out, dedupe and filter in code\nbefore anything reaches a model — and leave the code generation to the Go\norchestrator.\n\n## Architecture: three candidate shapes\n\n| # | Shape | Where a run starts | Durability | New code | Cost |\n|---|---|---|---|---|---|\n| **A** | New capability family `internal/domain/research` + `infrastructure/research`, typed engine, owning registry, lifecycle, background run, completion delivered through an injected courier | A chat tool `research_start` returns a run id | Runs survive the turn; progress observable; resumable | Domain + infra + store + wiring + UI + config | Largest |\n| **B** | A fourth cron job kind (`KindResearch`) in `crondelivery`, reusing `scheduling` + `cronstore` + the delivery router | Chat enqueues a one-shot immediate job | Same store and delivery as cron; no new persistence | Runner + kind + blueprint only | Medium — but fakes \"on demand\" as a scheduled job, and a job store keyed on a schedule is the wrong shape for a request |\n| **C** | A `deep_research` **tool** that runs the whole fan-out synchronously inside one chat turn, using `agent.Subagent` for the children | Directly — the model calls the tool | None: the run dies with the turn | Search seam + tool + fan-out helper | Smallest |\n\n**Recommendation: ship C's machinery, then promote it to A.** Slice 2 (below)\nbuilds the orchestrator as a tool because that is the smallest thing that\ndelivers Sam's stated design and reuses the `Subagent` primitive that already\nworks. Slice 3 lifts the identical orchestrator behind a durable run once its\nruntime cost and shape are known from real use — which is exactly the order\nAGENTS.md's solo-project rule asks for (\"prefer the smallest workable change\"),\nand it avoids inventing persistence for a shape that has not been exercised.\n\nShape B is rejected as the primary path: `cronstore.JobSpec` is a *schedule*\n(cron expression, next-run, catch-up); an on-demand research request is not.\nReusing it would mean every ad-hoc research run leaves a permanent job row.\n\n### Where the pieces live (shape C, then A)\n\n| Piece | Home | Contract |\n|---|---|---|\n| Search provider seam | `internal/domain/research` (interface) + `internal/infrastructure/research/search/\u003cprovider\u003e` (impls) | `Searcher.Search(ctx, query, opts) ([]Result, error)`; `Result{URL, Title, Snippet, RetrievedAt}`. Consumer-owned and one-method, per the `crondelivery.Courier` precedent — nothing here talks to a network |\n| `web_search` tool | `internal/tools/search` | Mirrors `internal/tools/webfetch`: `Tool(cfg) *tools.ToolEntry`, returns `nil` when disabled, `ClassIdempotent`, `Toolset: \"web\"` |\n| Source store | `internal/domain/research` entity + store | `Source{ID, URL, CanonicalURL, Title, RetrievedAt, Hash, State(ok/failed/duplicate), CitedBy []subq}` |\n| Fan-out orchestrator | `internal/domain/research` (pure) with the model/tool access injected | Bounded worker pool over sub-questions; per-child `agent.Subagent`-shaped `Researcher` seam; shared step/token/wall-clock budget |\n| Collator | `internal/domain/research` | Model pass over `[]ChildConclusion` + source store → report body |\n| Report renderer | `internal/domain/research` | Markdown + numbered citation list rendered **from the store** |\n| Chat trigger | slice 2: `internal/tools/research`; slice 3: gateway tool + courier in `internal/app/archied` | — |\n| Durable run + delivery (slice 3) | `internal/domain/research` + `internal/infrastructure/research` + `internal/app/archied` wiring | Follows the strict plugin-engine rule in `ARCHITECTURE.md` (\"Plugin engine\nrule (strict)\"): typed contract, owning registry with start/health/stop, narrow typed registrar |\n\n### Why not the curator\n\nThe curator family already looks like the right home — a declared interval, a\ndeclared tool set, an owning registry, per-pass budgets and panic isolation.\nTwo facts rule it out as the host for *this*:\n\n- **Its tool path does not work.** A curator declaring tools is refused\n registration because `ToolBuilder` has no implementation, and even if it were\n bound, `curatorLLMRunner` drops `ChatRequest.Tools` (`main.go`). Fixing\n that is `archie-core-trnd.1`, an independent piece of work.\n- **Its contract is maintenance, not requests.** The isolation invariant\n (`docs/architecture/agent-system.md`, \"Curator isolation invariant\") defines\n curators as periodic peers reasoning over memory. A deep-research run is\n operator-initiated work with a deliverable, on the operator's clock.\n\nA research *curator* is a legitimate later idea — \"research this standing topic\nweekly\" — but it is a consumer of this capability, not its foundation.\n\n## Decisions this design takes\n\n### D1. Citations come from a store, never from prose\n\nEach subagent gets a `record_source` typed tool (the `record_finding` pattern,\n`review.go`) whose handler canonicalises the URL, deduplicates against the\nstore, marks which sub-question recorded it, and appends. The collator may\n*select and order* sources; it may not author them. Rendering the reference\nlist is then a pure function of the store.\n\nThe strongest prior art goes further, and one step of it is worth adopting:\nstore a **verbatim quote plus a byte offset into the fetched document** with\neach claim, and drop any claim whose quote cannot be found in the source\n(`lajosdeme/mole`: `FindQuote` + `Claim.QuoteOffset`, with the miner logging\n`\"claim rejected: quote not found in source\"`). That converts an unfalsifiable\n\"the model said so\" into a re-checkable \"the document says so, at offset N\", and\nit makes citation re-verification cheap rather than a second research run.\n\n*Rejected alternative:* require the child to end with a JSON block of sources\nand parse it. A parse failure loses every source in the child's run, and a\nmodel that reformats its output for readability silently drops citations.\n\n### D2. Fan-out is parallel, bounded, and shares one budget\n\n`agent.Subagent.Run` is synchronous, so N children means N goroutines behind a\nsemaphore — concurrency from config, sized from the VRAM budget rather than\nfrom the research question. Two reasons to keep it small: the shared model\nruntime also serves chat, and on a single GPU N concurrent workers contend for\nthe same weights and KV budget, so **wall clock scales roughly linearly with\nworker count**. The isolation benefit of fan-out survives that; the throughput\nbenefit does not.\n\nFan-out width and per-child `MaxSteps` are enforced by the orchestrator, not\nrequested in a prompt, against a whole-run wall clock and token ceiling.\n`agentloop.Budget` already carries `MaxSteps`, `MaxTokens`, `WallClock`, and\n`TokenBudgetIs(maxTokens)` exists as a stop condition (`agentloop.go`) —\nthe shape exists, the fan-out owner does not.\n\n**Fan out retrieval aggressively and generation sparingly.** Web search is 73%\nof wall-clock time on average (up to 91%) in the one instrumented study\navailable, so retrieval is where the parallel win lives; generation is the\nserialised, expensive half.\n\n### D3. The orchestrator never holds child transcripts\n\n`Subagent.Run` sends a prompt and returns a string; the parent's history is\nnever passed down and the child's is never returned. That is the property that\nmakes N children affordable at all — the parent's context grows by N\nconclusions, not N conversations. This is also why collation is a *separate\npass* rather than an accumulating agent loop: the collator is the only place\nthat must hold all N conclusions, and it holds nothing else.\n\nEach child compresses its own findings before returning, so what crosses the\nboundary is a summary, not a transcript. Anthropic names the same pattern\n(filesystem artefacts with lightweight references back) and Perplexity arrived\nat it independently; it is the one disclosed primitive that *increases*\neffective capacity without a bigger context window.\n\n### D4. Search is a provider seam with no silent default\n\n`web_search` is registered only when a provider resolves, following two\nexisting conventions: return `nil` for a disabled capability\n(`webfetch/tool.go` — \"an advertised tool that never works costs a round\ntrip\") and resolve the credential *before* registering\n(`registerMinimaxTool`, `bootstrap.go`). At least two implementations are\nwanted — a self-hosted meta-search endpoint (no per-query cost, no data egress)\nand one hosted API (quality fallback) — behind the same one-method interface,\nselected by config.\n\n### D5. A run reports what it could not establish\n\nThe collator is required to emit, alongside the report, the sub-questions that\nreturned nothing usable and the sources that failed to fetch. A research report\nthat omits its own gaps reads as authoritative over a hole, and the operator\ncannot tell a negative finding from a missing one.\n\n### D6. Two stages no surveyed design has: a mechanical cross-check and a separate citation pass\n\nChildren never compare notes in any disclosed design, and nothing in the\npublished set *resolves* conflicts — Perplexity only \"notes\" them. Because\n84.7% of final-report errors originate at the orchestrator (mostly citation\nerrors), both gaps are worth closing in code rather than in a prompt:\n\n- **Cross-check (deterministic, no model).** Group findings by canonical URL and\n by claim. Flag single-sourced claims, mark unresolved conflicts explicitly,\n and count sources per sub-question. This runs *before* the collator, so the\n collator receives a pre-reconciled set rather than raw prose to untangle.\n- **Citation pass (its own stage, after collation).** Anthropic runs synthesis\n and then a separate CitationAgent. Here the pass is cheaper for being partly\n mechanical: it validates that every marker in the body resolves to a stored\n source, and that every stored source was actually fetched successfully.\n\nBoth stages are small, both are testable without a model, and together they are\nwhere this implementation can be *better* than the frontier products rather\nthan an imitation of them. A run that is expected to emit 10-20 well-verified\ncitations rather than 41-113 partly-unverified ones is making a deliberate,\nevidence-supported trade.\n\n## Open decisions\n\n1. **Trigger surface.** A chat tool that blocks the turn (shape C) is\n immediately useful but the operator waits; a durable run (shape A) is the\n right end state. Does slice 2 ship the blocking tool, or does the capability\n go straight to durability?\n2. **Delivery for a durable run.** Which channel gets the finished report, and\n is the report an inline message, a file, or a served page? The\n `crondelivery.Courier` seam covers the chat case; a long cited report\n probably wants a document, since chat surfaces render markdown but not\n footnotes well.\n3. **Search provider choice and egress.** A hosted search API sends every query\n to a third party; a self-hosted meta-search keeps it local. This is an\n operator decision with a real privacy dimension, and the default matters.\n4. **Report persistence and reuse.** Does a report become a memory, a document,\n or neither? Reusing a prior report is the difference between \"research\" and\n \"research again next week\".\n5. **Cost ceiling.** What is the per-run token ceiling, and is it a config\n value, a per-request parameter, or derived from the fan-out width?\n\n## Failure and security paths\n\n- **SSRF.** Search results are attacker-influenceable (a page can rank for a\n query). `web_fetch` already vets the *resolved address* at dial time with\n `AllowPrivateNetworks` off, which is the property to preserve — a search\n tool must not become a way to reach the daemon's own control surfaces by\n returning an internal URL.\n- **Prompt injection.** Fetched pages are untrusted content and will contain\n instructions. Subagents fetch; the orchestrator and collator must never\n execute an instruction sourced from a page. Bounded by giving subagents a\n read-only toolset with no shell and no write.\n- **Runaway spend.** Unbounded fan-out × unbounded steps is the failure mode.\n Bounded concurrency, per-child step caps, a whole-run wall clock, and a\n token ceiling, all enforced in code.\n- **Degenerate fan-out.** A model asked for \"N sub-questions\" will produce N\n near-duplicates of the same question. Deduplicate sub-questions before\n dispatch, and require distinct search queries per child.\n- **Failed dependencies.** Search provider down: the tool is absent (D4), not\n broken. Fetch fails: recorded as a `failed` source so the report can say so.\n- **Collator degradation.** If the collator call fails or truncates, the report\n must still be rendered from the source store and the child conclusions —\n degraded, not empty. This mirrors the reviewer's rule that findings captured\n before a failed run are still real (`review.go`).\n\n## Slices\n\n1. **Search seam + `web_search` tool.** Interface, one implementation, config\n block, tool registration, tests. Independently useful to every chat turn.\n2. **Fan-out orchestrator + collation, as a tool.** `record_source` capture\n tool (with quote + offset), bounded parallel children with per-child\n compression, cross-check, collator pass, citation pass, store-rendered\n citations. Triggerable from chat; blocks the turn.\n3. **Durable run + delivery.** Promote the orchestrator behind the engine\n contract, add the run store, progress events, and completion delivery.\n4. **UI + reuse.** Dashboard surface for runs and past reports; optional\n research curator on a schedule.\n\n## Non-goals\n\n- Replacing `web_fetch`; research uses it.\n- A general multi-agent collaboration framework (`archie-core-1786637492113-89`\n owns that question).\n- Report publication to a static site or external destination.\n- Fine-tuning or a dedicated model: the capability is orchestration, and it\n must work on whatever model the operator has configured.\n" + "body": "# Deep research capability — design\n\n**Status:** Draft\n**Date:** 2026-09-14\n**Scope:** A deep-research capability in archie-core: an orchestrator that\nspawns 1..N subagents to explore and research in parallel, collates their\nfindings, and returns one cited report. Grounded in web search and web fetch.\n**Beads issue:** none yet (this document is the input to it)\n**Depends on:** nothing shipped for slice 1; slice 2 depends on slice 1.\n\n## §0 Implementation status\n\nEvery claim in this document is mapped to what exists in the tree.\n\n| Claim | Status | Where it stands |\n|---|---|---|\n| `web_fetch` tool (URL → readable text) | **Implemented** | `internal/tools/webfetch/{tool.go,webfetch.go}`; registered in `internal/app/archied/bootstrap.go` `registerStandaloneTools`, disabled-by-config returns `nil` |\n| **Web search** (query → result list) | **Aspirational — zero code** | No search provider, no search tool, no config block anywhere in the tree or in `ai-sdk` v0.1.32 |\n| `agent.Subagent` — nested agent in a fresh context window | **Implemented** (dependency) | `github.com/samcharles93/ai-sdk@v0.1.32/agent/subagent.go`. Synchronous, non-streaming, own model/system/toolset, `MaxSteps` defaults to 10, nesting depth capped at 5, returns final text only |\n| Subagent used in production | **Implemented, one consumer** | `internal/app/agentworker/review.go` — the adversarial reviewer. That is the only non-test use of `Subagent` in the repo |\n| A **delegate/subagent tool** on the chat surface | **Aspirational — zero code** | No `subagent`/`delegate` tool name is registered anywhere |\n| **Fan-out orchestration** (N missions, one collator) | **Aspirational — zero code** | `agentloop`'s package doc states it runs *one* mission: \"orchestration across missions lives with the caller\". No caller does this yet |\n| Parallel-safe tool execution | **Aspirational — zero code** | The unreachable `internal/tools/dispatch.go` dispatch/gating engine (`ExecutionClass`, `DispatchSequential`/`DispatchConcurrent`/`ClassifyEntry`) was deleted as dead code (tools-core-1); live concurrency is ai-sdk's `GenerateOptions.MaxParallelToolCalls`, which archie-core never sets |\n| Per-result caps + spill-to-disk | **Implemented** | `internal/agentexec/toolset.go` `ToolLimits{MaxResultChars,SpillDir}`, applied in `toolExecute` via `tools.CapPayload` (`limit.go`); no **aggregate** turn cap by design (`toolset.go`). `config.ToolPolicy.SpillDir`; default cap 50k chars, spill dir deliberately not defaulted |\n| Tool guardrails (repeat-failure / no-progress) | **Implemented, not on the call path** | `internal/tools/guardrail.go` engine exists. Its only non-test caller is `dispatchOne` — which is itself unreachable — so **the chat path never consults it**, and the agent path only records per *stage* (`domain/workflow/agent.go`) |\n| Structured capture tools (model reports into a typed store, not prose) | **Implemented** | `internal/app/agentworker/review.go` `record_finding` / `record_checked` — `core.NewTypedTool` appending to a caller-owned slice. This is the pattern a research source ledger should reuse |\n| Curator engine (interval, declared tool set, lifecycle) | **Implemented, tools broken** | `internal/domain/curator`; a curator declaring tools is **refused** at registration (`registry.go`, no `ToolBuilder` implementation) and `curatorLLMRunner` (`internal/app/archied/main.go`) drops `ChatRequest.Tools`. Budgets are time+concurrency only (`runtime.go`) — **no token budget** |\n| Cron / scheduling + delivery router | **Implemented, wired into the daemon** | `internal/domain/scheduling` (ticker engine), `internal/infrastructure/cronstore` (schema v2, `Kind` discriminator), and `internal/infrastructure/crondelivery`; `internal/app/archied/scheduling.go` composes them and `main.go` starts the engine. |\n| Workflow engines (multi-stage, resumable) | **Implemented, worktree-bound** | `internal/domain/workflow` — every non-triage workflow opens with `StagePrepareWorktree()` and mounts `tc.Dir` into a container; the task persists after every stage (`workflow.go`). `triage.go` documents there is \"no cheaper directory-less classification primitive\" |\n| Long-running chat-initiated work with progress | **Partial** | Chat turns run **once, synchronously**, inline in the channel's turn runner (`internal/gateway/turn.go`). Chat can spawn a *workflow task* (`task_spawn`) which is asynchronous — but that path is worktree/PR-shaped |\n| **Operator-push delivery of an async result** | **Aspirational — zero code** | Nothing pushes a finished result to a chat. No channel subscribes to the event bus (Telegram only *publishes* `KindTurnCompleted`); `crondelivery.Courier` is the intended seam but has **no implementation and no caller**; `send_file` works only inside the live turn. The one out-of-band notification in the tree is feasibility's n8n webhook (`feasibility.go`) |\n| Citation storage / source ledger | **Aspirational — zero code** | No source, citation, or provenance store exists for research output |\n\n\u003e **Target state, not current state.** Everything marked aspirational below is\n\u003e design, not a description of the daemon as it runs.\n\n## Problem\n\nSam's requirement: *\"Deep research should spawn 1 or many subagents for\nexploration or research work. And they should collate all findings, they\nshould use web search/fetch features.\"*\n\nNone of the three primitives that sentence needs can be assembled from what\nships:\n\n1. **Web search does not exist.** `web_fetch` reads a URL you already have. A\n research loop's first act is to *discover* URLs from a query, and there is\n no tool, provider, or config for that — in archie-core or in the shared SDK.\n2. **Nothing can fan out.** `agent.Subagent` can nest *one* nested generation,\n synchronously, and only the PR reviewer calls it. There is no N-way fan-out,\n no bounded concurrency, no shared budget across children, no collation pass.\n `agentloop` explicitly refuses to orchestrate: it runs one mission and the\n caller owns orchestration — and no caller orchestrates.\n3. **A research run outlives a chat turn.** A chat turn is synchronous and\n inline: the operator waits, and the turn's own context window is what\n carries the result. A fan-out over N subagents with web search is minutes to\n tens of minutes of wall clock. It cannot be a turn.\n\nThe interesting part is not the loop. It is where the boundary sits: which\nexisting long-running mechanism this reuses, how citations survive collation\nwhen the thing being collated is model-written prose, and what stops a run\nfrom spending unbounded money.\n\n## What the run actually is\n\nSix stages, each of which must be a *stage* rather than a prompt instruction:\n\n1. **Frame.** One cheap model call turns the request into a research brief: the\n question, 3..N distinct sub-questions, and a per-sub-question budget. Output\n is structured (a capture tool), not prose — the fan-out width comes from\n this step, so it must be a number the code reads, not a sentence.\n2. **Fan out.** One subagent per sub-question, **in parallel**, with a bounded\n concurrency, each running a read-only research toolset (search + fetch +\n read) against its own fresh context window, each required to record its\n findings as *typed source records* before finishing, and each compressing\n what it returns.\n3. **Cross-check.** Deterministic, no model: group by canonical URL and by\n claim, flag single-sourced claims, mark conflicts. See D6.\n4. **Collate.** One model pass over the children's *conclusions and source\n records* — never their transcripts — that writes the report body against the\n pre-reconciled source set.\n5. **Citation pass.** Its own stage: validate every marker in the body resolves\n to a stored source, and every stored source was fetched successfully.\n6. **Render.** The citation list is rendered from the **source store**, not\n from the collator's prose, so a URL can only appear in a report if some\n subagent actually recorded it.\n\nStages 5 and 6 are the load-bearing decision, and they are what the commercial\nproducts' published failure modes point at: a synthesizer that is free to write\ncitations will invent them, and a citation that cannot be traced to a fetch is\nworse than no citation.\n\n## Prior art: what the disclosed designs and the open-source field say\n\nTwo external research passes inform this design — one on the commercial\nproducts (official engineering blogs, model docs, system cards, plus\npeer-reviewed benchmarks), one on open-source deep-research implementations\nread at pinned commits. The findings that changed decisions:\n\n**Only one fan-out unit buys capacity.** Four different units are in use:\nAnthropic fans out *agents with their own context windows*; Google fans out\n*queries*; Perplexity fans out *generated programs*; OpenAI uses pipeline-stage\nagents for query enrichment only. Query fan-out is cheap and buys breadth of\nretrieval; agent fan-out is what buys context capacity. Our step 2 is agent\nfan-out, deliberately, because context isolation is the thing a single GPU\ncannot otherwise buy.\n\n**Nobody collates section-by-section.** The open-source survey found one merge\nprompt over concatenated child summaries (`final_report_generation` in\n`langchain-ai/open_deep_research`), lead-agent synthesis (Anthropic), a\ncode-level reconcile (Perplexity), or single-model continuation. STORM is the\nonly demonstrated section-mapped merge. This confirms D3: one collation pass,\nnot a mapped merge.\n\n**Citations are lost at the merge in 8 of 10 surveyed repos.** Only `lajosdeme/mole`\nassigns citation markers **mechanically in code before the prompt is built** —\nits `internal/output/report.go` carries the comment *\"it cannot introduce a\ncitation, because the citation markers are assigned mechanically before the\nprompt is built\"* — and stores a verbatim `Quote` plus a `QuoteOffset` (byte\noffset into the fetched document) so a citation can be re-verified later.\nClaims whose quote is not found in the source are dropped by code\n(`internal/actors/mine.go`: `\"claim rejected: quote not found in source\"`).\nThis is the strongest available argument for D1, and `QuoteOffset` is worth\nadopting: it makes re-verification cheap and turns \"the model said so\" into\n\"the document says so, here\".\n\n**The failure is measurable and it is at the orchestrator.** An EMNLP 2026\nstudy localising errors across three top-ranked open-source deep-research\nsystems found **84.7% of final-report errors originate at the orchestrator,\nroughly 31% of them hallucinations and the rest citation mistakes**, with\ninformation passing \"through agents like a telephone game\"\n(https://arxiv.org/abs/2608.24306). Anthropic's own mitigations for this are a\nseparate **CitationAgent** pass after synthesis and a filesystem-artefact\npattern — subagents store work externally and pass back lightweight references\nrather than full outputs. Perplexity reached the same conclusion independently.\nBoth are adopted here: D1 (store, not prose) and a citation pass that is its\nown stage.\n\n**A cross-check stage that no vendor has.** Worker A and worker B never compare\nnotes in any disclosed design, and nothing in the published set *resolves*\nconflicts — Perplexity only \"notes\" them. Grouping findings by URL and by claim,\nflagging single-sourced claims, and marking unresolved conflicts is the\ncheapest correctness win available and is where a local implementation can beat\nthe frontier products rather than imitate them.\n\n**Fan-out is expensive in tokens, and on one GPU that is wall clock.** Anthropic\ndiscloses ~15x tokens for multi-agent versus ~4x for a single agent, and\nexplicit tier rules in the orchestrator prompt (1 agent / 3-10 tool calls;\n2-4 subagents / 10-15 calls each; \u003e10 subagents for complex research, 3-5\nspawned concurrently). On a single card, N concurrent workers contend for the\nsame weights and KV budget, so **wall clock scales roughly linearly with worker\ncount** — the isolation benefit survives, the throughput benefit does not. One\ninstrumented study puts **web search at 73% of wall-clock time on average (up to\n91%)** (https://cseweb.ucsd.edu/~yiying/2025_NIPS_ERW_Deep_Research_Perf_Study.pdf),\nwhich is the argument for fanning out *retrieval* aggressively and *generation*\nsparingly, and for overlapping generation with retrieval.\n\n**Frontier citation volume is not a target.** Frontier runs emit 41-113\ncitation URLs per query, while 3-13% of citation URLs are hallucinated and \"more\ncitations per query do not mean fewer errors per citation\"\n(https://arxiv.org/abs/2604.03173). Ten to twenty *verified* citations is a\ndefensible, evidence-backed target.\n\n**Do not port Perplexity's style.** Search-as-code-generation needs\nfrontier-class codegen against a bespoke SDK and an atomised self-owned search\nstack. Port the *intent* — deterministic fan-out, dedupe and filter in code\nbefore anything reaches a model — and leave the code generation to the Go\norchestrator.\n\n## Architecture: three candidate shapes\n\n| # | Shape | Where a run starts | Durability | New code | Cost |\n|---|---|---|---|---|---|\n| **A** | New capability family `internal/domain/research` + `infrastructure/research`, typed engine, owning registry, lifecycle, background run, completion delivered through an injected courier | A chat tool `research_start` returns a run id | Runs survive the turn; progress observable; resumable | Domain + infra + store + wiring + UI + config | Largest |\n| **B** | A fourth cron job kind (`KindResearch`) in `crondelivery`, reusing `scheduling` + `cronstore` + the delivery router | Chat enqueues a one-shot immediate job | Same store and delivery as cron; no new persistence | Runner + kind + blueprint only | Medium — but fakes \"on demand\" as a scheduled job, and a job store keyed on a schedule is the wrong shape for a request |\n| **C** | A `deep_research` **tool** that runs the whole fan-out synchronously inside one chat turn, using `agent.Subagent` for the children | Directly — the model calls the tool | None: the run dies with the turn | Search seam + tool + fan-out helper | Smallest |\n\n**Recommendation: ship C's machinery, then promote it to A.** Slice 2 (below)\nbuilds the orchestrator as a tool because that is the smallest thing that\ndelivers Sam's stated design and reuses the `Subagent` primitive that already\nworks. Slice 3 lifts the identical orchestrator behind a durable run once its\nruntime cost and shape are known from real use — which is exactly the order\nAGENTS.md's solo-project rule asks for (\"prefer the smallest workable change\"),\nand it avoids inventing persistence for a shape that has not been exercised.\n\nShape B is rejected as the primary path: `cronstore.JobSpec` is a *schedule*\n(cron expression, next-run, catch-up); an on-demand research request is not.\nReusing it would mean every ad-hoc research run leaves a permanent job row.\n\n### Where the pieces live (shape C, then A)\n\n| Piece | Home | Contract |\n|---|---|---|\n| Search provider seam | `internal/domain/research` (interface) + `internal/infrastructure/research/search/\u003cprovider\u003e` (impls) | `Searcher.Search(ctx, query, opts) ([]Result, error)`; `Result{URL, Title, Snippet, RetrievedAt}`. Consumer-owned and one-method, per the `crondelivery.Courier` precedent — nothing here talks to a network |\n| `web_search` tool | `internal/tools/search` | Mirrors `internal/tools/webfetch`: `Tool(cfg) *tools.ToolEntry`, returns `nil` when disabled, `ClassIdempotent`, `Toolset: \"web\"` |\n| Source store | `internal/domain/research` entity + store | `Source{ID, URL, CanonicalURL, Title, RetrievedAt, Hash, State(ok/failed/duplicate), CitedBy []subq}` |\n| Fan-out orchestrator | `internal/domain/research` (pure) with the model/tool access injected | Bounded worker pool over sub-questions; per-child `agent.Subagent`-shaped `Researcher` seam; shared step/token/wall-clock budget |\n| Collator | `internal/domain/research` | Model pass over `[]ChildConclusion` + source store → report body |\n| Report renderer | `internal/domain/research` | Markdown + numbered citation list rendered **from the store** |\n| Chat trigger | slice 2: `internal/tools/research`; slice 3: gateway tool + courier in `internal/app/archied` | — |\n| Durable run + delivery (slice 3) | `internal/domain/research` + `internal/infrastructure/research` + `internal/app/archied` wiring | Follows the strict plugin-engine rule in `docs/architecture/plugins-and-extensions.md` (\"Plugin engine\nrule (strict)\"): typed contract, owning registry with start/health/stop, narrow typed registrar |\n\n### Why not the curator\n\nThe curator family already looks like the right home — a declared interval, a\ndeclared tool set, an owning registry, per-pass budgets and panic isolation.\nTwo facts rule it out as the host for *this*:\n\n- **Its tool path does not work.** A curator declaring tools is refused\n registration because `ToolBuilder` has no implementation, and even if it were\n bound, `curatorLLMRunner` drops `ChatRequest.Tools` (`main.go`). Fixing\n that is `archie-core-trnd.1`, an independent piece of work.\n- **Its contract is maintenance, not requests.** The isolation invariant\n (`docs/architecture/agent-system.md`, \"Curator isolation invariant\") defines\n curators as periodic peers reasoning over memory. A deep-research run is\n operator-initiated work with a deliverable, on the operator's clock.\n\nA research *curator* is a legitimate later idea — \"research this standing topic\nweekly\" — but it is a consumer of this capability, not its foundation.\n\n## Decisions this design takes\n\n### D1. Citations come from a store, never from prose\n\nEach subagent gets a `record_source` typed tool (the `record_finding` pattern,\n`review.go`) whose handler canonicalises the URL, deduplicates against the\nstore, marks which sub-question recorded it, and appends. The collator may\n*select and order* sources; it may not author them. Rendering the reference\nlist is then a pure function of the store.\n\nThe strongest prior art goes further, and one step of it is worth adopting:\nstore a **verbatim quote plus a byte offset into the fetched document** with\neach claim, and drop any claim whose quote cannot be found in the source\n(`lajosdeme/mole`: `FindQuote` + `Claim.QuoteOffset`, with the miner logging\n`\"claim rejected: quote not found in source\"`). That converts an unfalsifiable\n\"the model said so\" into a re-checkable \"the document says so, at offset N\", and\nit makes citation re-verification cheap rather than a second research run.\n\n*Rejected alternative:* require the child to end with a JSON block of sources\nand parse it. A parse failure loses every source in the child's run, and a\nmodel that reformats its output for readability silently drops citations.\n\n### D2. Fan-out is parallel, bounded, and shares one budget\n\n`agent.Subagent.Run` is synchronous, so N children means N goroutines behind a\nsemaphore — concurrency from config, sized from the VRAM budget rather than\nfrom the research question. Two reasons to keep it small: the shared model\nruntime also serves chat, and on a single GPU N concurrent workers contend for\nthe same weights and KV budget, so **wall clock scales roughly linearly with\nworker count**. The isolation benefit of fan-out survives that; the throughput\nbenefit does not.\n\nFan-out width and per-child `MaxSteps` are enforced by the orchestrator, not\nrequested in a prompt, against a whole-run wall clock and token ceiling.\n`agentloop.Budget` already carries `MaxSteps`, `MaxTokens`, `WallClock`, and\n`TokenBudgetIs(maxTokens)` exists as a stop condition (`agentloop.go`) —\nthe shape exists, the fan-out owner does not.\n\n**Fan out retrieval aggressively and generation sparingly.** Web search is 73%\nof wall-clock time on average (up to 91%) in the one instrumented study\navailable, so retrieval is where the parallel win lives; generation is the\nserialised, expensive half.\n\n### D3. The orchestrator never holds child transcripts\n\n`Subagent.Run` sends a prompt and returns a string; the parent's history is\nnever passed down and the child's is never returned. That is the property that\nmakes N children affordable at all — the parent's context grows by N\nconclusions, not N conversations. This is also why collation is a *separate\npass* rather than an accumulating agent loop: the collator is the only place\nthat must hold all N conclusions, and it holds nothing else.\n\nEach child compresses its own findings before returning, so what crosses the\nboundary is a summary, not a transcript. Anthropic names the same pattern\n(filesystem artefacts with lightweight references back) and Perplexity arrived\nat it independently; it is the one disclosed primitive that *increases*\neffective capacity without a bigger context window.\n\n### D4. Search is a provider seam with no silent default\n\n`web_search` is registered only when a provider resolves, following two\nexisting conventions: return `nil` for a disabled capability\n(`webfetch/tool.go` — \"an advertised tool that never works costs a round\ntrip\") and resolve the credential *before* registering\n(`registerMinimaxTool`, `bootstrap.go`). At least two implementations are\nwanted — a self-hosted meta-search endpoint (no per-query cost, no data egress)\nand one hosted API (quality fallback) — behind the same one-method interface,\nselected by config.\n\n### D5. A run reports what it could not establish\n\nThe collator is required to emit, alongside the report, the sub-questions that\nreturned nothing usable and the sources that failed to fetch. A research report\nthat omits its own gaps reads as authoritative over a hole, and the operator\ncannot tell a negative finding from a missing one.\n\n### D6. Two stages no surveyed design has: a mechanical cross-check and a separate citation pass\n\nChildren never compare notes in any disclosed design, and nothing in the\npublished set *resolves* conflicts — Perplexity only \"notes\" them. Because\n84.7% of final-report errors originate at the orchestrator (mostly citation\nerrors), both gaps are worth closing in code rather than in a prompt:\n\n- **Cross-check (deterministic, no model).** Group findings by canonical URL and\n by claim. Flag single-sourced claims, mark unresolved conflicts explicitly,\n and count sources per sub-question. This runs *before* the collator, so the\n collator receives a pre-reconciled set rather than raw prose to untangle.\n- **Citation pass (its own stage, after collation).** Anthropic runs synthesis\n and then a separate CitationAgent. Here the pass is cheaper for being partly\n mechanical: it validates that every marker in the body resolves to a stored\n source, and that every stored source was actually fetched successfully.\n\nBoth stages are small, both are testable without a model, and together they are\nwhere this implementation can be *better* than the frontier products rather\nthan an imitation of them. A run that is expected to emit 10-20 well-verified\ncitations rather than 41-113 partly-unverified ones is making a deliberate,\nevidence-supported trade.\n\n## Open decisions\n\n1. **Trigger surface.** A chat tool that blocks the turn (shape C) is\n immediately useful but the operator waits; a durable run (shape A) is the\n right end state. Does slice 2 ship the blocking tool, or does the capability\n go straight to durability?\n2. **Delivery for a durable run.** Which channel gets the finished report, and\n is the report an inline message, a file, or a served page? The\n `crondelivery.Courier` seam covers the chat case; a long cited report\n probably wants a document, since chat surfaces render markdown but not\n footnotes well.\n3. **Search provider choice and egress.** A hosted search API sends every query\n to a third party; a self-hosted meta-search keeps it local. This is an\n operator decision with a real privacy dimension, and the default matters.\n4. **Report persistence and reuse.** Does a report become a memory, a document,\n or neither? Reusing a prior report is the difference between \"research\" and\n \"research again next week\".\n5. **Cost ceiling.** What is the per-run token ceiling, and is it a config\n value, a per-request parameter, or derived from the fan-out width?\n\n## Failure and security paths\n\n- **SSRF.** Search results are attacker-influenceable (a page can rank for a\n query). `web_fetch` already vets the *resolved address* at dial time with\n `AllowPrivateNetworks` off, which is the property to preserve — a search\n tool must not become a way to reach the daemon's own control surfaces by\n returning an internal URL.\n- **Prompt injection.** Fetched pages are untrusted content and will contain\n instructions. Subagents fetch; the orchestrator and collator must never\n execute an instruction sourced from a page. Bounded by giving subagents a\n read-only toolset with no shell and no write.\n- **Runaway spend.** Unbounded fan-out × unbounded steps is the failure mode.\n Bounded concurrency, per-child step caps, a whole-run wall clock, and a\n token ceiling, all enforced in code.\n- **Degenerate fan-out.** A model asked for \"N sub-questions\" will produce N\n near-duplicates of the same question. Deduplicate sub-questions before\n dispatch, and require distinct search queries per child.\n- **Failed dependencies.** Search provider down: the tool is absent (D4), not\n broken. Fetch fails: recorded as a `failed` source so the report can say so.\n- **Collator degradation.** If the collator call fails or truncates, the report\n must still be rendered from the source store and the child conclusions —\n degraded, not empty. This mirrors the reviewer's rule that findings captured\n before a failed run are still real (`review.go`).\n\n## Slices\n\n1. **Search seam + `web_search` tool.** Interface, one implementation, config\n block, tool registration, tests. Independently useful to every chat turn.\n2. **Fan-out orchestrator + collation, as a tool.** `record_source` capture\n tool (with quote + offset), bounded parallel children with per-child\n compression, cross-check, collator pass, citation pass, store-rendered\n citations. Triggerable from chat; blocks the turn.\n3. **Durable run + delivery.** Promote the orchestrator behind the engine\n contract, add the run store, progress events, and completion delivery.\n4. **UI + reuse.** Dashboard surface for runs and past reports; optional\n research curator on a schedule.\n\n## Non-goals\n\n- Replacing `web_fetch`; research uses it.\n- A general multi-agent collaboration framework (`archie-core-1786637492113-89`\n owns that question).\n- Report publication to a static site or external destination.\n- Fine-tuning or a dedicated model: the capability is orchestration, and it\n must work on whatever model the operator has configured.\n" }, { "url": "/dev/prds/dynamic-curators/", @@ -220,7 +220,7 @@ "url": "/dev/prds/eda-playbook-engine/", "title": "EDA playbook engine -- design", "section": "PRDs", - "body": "# EDA playbook engine -- design\n\n**Status:** Draft\n**Date:** 2026-09-03\n**Beads epic:** `archie-core-t2db`\n\n## Problem\n\n\"Which workflow runs for which trigger\" is hardcoded Go: a literal\nlabel vocabulary (`workintake.labelKinds`) and a literal label-\u003eworkflow map\n(`workflow/routing.go`), extended only by adding a Go case or shipping a\nskill whose `metadata.archie.workflow` overrides one *named* built-in\nworkflow. Extending the system to a new trigger, a new action, or a new\nevent source means a feature branch, a code change in a hardcoded\npath, and new Go tests -- for every addition. That is the barrier this\ndocument exists to remove.\n\nThe goal: triggers, the actions they run, and the\nconditions that connect them should be **drop-in and load**, not **drop-in\nand touch application code**. Yaegi-evaluated Go is the unit of imperative\n\"do a thing\" logic; YAML playbooks are pure orchestration data -- no logic,\njust \"on this event, with this data, call these actions, in this order,\nunder these conditions.\" The daemon's event coordinator doesn't know what a\n`bugfix` or `image-gen` action does, it resolves the name against a\nregistry and pipes typed data through.\n\nThis is explicitly **not** scoped to code-generation workflows. A playbook\naction can be any registered capability -- run a codegen workflow, send a\nmessage on a channel, open something on a forge, generate an image -- so\nlong as it's implemented behind one of the typed engine families below.\n\n## Relationship to existing decisions\n\n- `docs/prds/event-sources-and-reactions.md` ruled\n that reactions in the two epics it covers are **producer-only** (no veto,\n no mutation of in-flight work) and explicitly rejected a single generic\n `Source` interface as \"the untyped-hook shape the plugin engine rule\n exists to prevent.\" **This document extends that scope**: playbooks can\n dispatch to multiple typed action kinds, not just \"produce a\n `TaskEnvelope`.\" It does not relitigate producer-only-ness (still no\n veto/mutation of in-flight work here) and it does not introduce one\n generic interface -- see \"Typed positions, not one generic hook\" below,\n which is how it stays compliant with the plugin engine rule instead of\n reopening that document's #4.\n- `ARCHITECTURE.md#plugin-engine-rule-strict` governs every extension point\n introduced here. Every position below is a typed domain contract with an\n owning registry, not a generic callback map. This is a hard constraint,\n confirmed against the existing rule text before writing this doc.\n- Yaegi is already a proven in-process extension mechanism\n (`internal/plugin`, `internal/domain/workflow/wfeval/yaegi.go`,\n `internal/gate/gateeval/yaegi.go`, `internal/skill/plugin.go`) -- this\n design generalizes its existing role, it does not introduce a new\n execution mechanism.\n- `internal/domain/workflow/skillbuild.BuildRegistry` already composes a\n `workflow.Registry` from a skills catalog, plugin-defined overriding\n built-in. The playbook loader described here supersedes label routing\n specifically; it does not replace `skillbuild`'s workflow-composition role,\n it becomes an additional input the daemon composes at startup.\n- Webhook event intake is a separate design. This\n document defines the router each webhook-sourced event should be able to\n reach; it does not redesign webhook receipt itself.\n\n## Shape: four logic positions, each a typed engine family\n\nEvery entrypoint kind (mechanism, action, loop, schedule, route, trigger) is itself a schema emitter: defining a new kind produces the\ngenerated contract downstream tooling consumes. The kinds slot into a fixed\nhierarchy of logic positions:\n\n```\nWorkflow -\u003e Module -\u003e Channel (telegram/whatsapp/email/...)\n -\u003e Forge (GitHub/Gitea/GitLab/...)\n```\n\nEach position is its own typed engine family per the plugin engine rule:\nidentity + capability-specific operations, an owning `Registry`/`Manager`,\nexplicit lifecycle where the implementation owns resources, narrow host\naccess. Concretely:\n\n- **Workflow** -- already exists (`internal/domain/workflow`). A playbook\n action of kind `workflow` invokes a registered `workflow.Registry` entry\n by name, unchanged from the existing execution path.\n- **Module** -- the new general-purpose action position. A `Module` is a\n Yaegi-evaluated implementation of a generated interface (args in, typed\n result out) for one declared action kind (e.g. `image-gen`, `notify`,\n `open-pr-comment`). This is the position that makes actions \"not\n restricted to code-generation.\" Full design (schema-per-kind, the\n registry shape, the trust boundary, the recommended first slice):\n `docs/prds/module-position.md`.\n- **Channel** -- already exists as `channels.Channel` (telegram, email,\n webhook). A playbook action of kind `channel` sends through a named,\n already-registered channel. No new mechanism; this document just makes it\n addressable from a playbook.\n- **Forge** -- forge clients (GitHub, Gitea, GitLab) become an addressable\n action position the same way Channel is, so a playbook can say \"comment\n on this forge issue\" without that being a special case wired only into\n the poller.\n\n**Typed positions, not one generic hook.** There is no single `Action`\ninterface with an `any` payload. Each position keeps its own typed\ncontract; the playbook YAML names a position and an implementation, and the\nevent coordinator resolves that pair against the matching typed registry.\nThis is what keeps the design compliant with the plugin engine rule and\nwith `event-sources-and-reactions.md`'s rejection of a generic `Source`/\nhook interface -- the generality lives in the YAML data model (which\npositions exist, which named implementations are registered), not in a\nruntime `any`-typed dispatch path.\n\n## `go:generate`-driven schemas, from day one\n\nSchemas are not a future nicety bolted onto an ad hoc\nYAML shape. They are the struct+interface contracts for each logic\nposition, and they drive code generation from the start.\n\n- Each logic position (Workflow/Module/Channel/Forge) has a schema source\n (Go-annotated types, following the existing pattern elsewhere in the\n repo) from which `go:generate` emits the interface and struct scaffolding\n an implementation must satisfy.\n- A Yaegi module implements the generated interface for its position. The\n generated contract is what makes an implementation valid, not hand-review.\n- The schema is also what the linter and LSP (below) validate a playbook\n YAML against -- one source of truth for codegen, startup loading, and\n authoring-time tooling, so the three can't drift from each other.\n- Schema versioning is explicit per position, independent of a playbook's\n own workflow version (see collision handling below).\n\nThe generator itself is not designed here. The constraint is\nschema-first, `go:generate`-driven, per logic position.\n\n## Playbook YAML: pure orchestration data\n\nA playbook declares no embedded logic -- only:\n\n- **trigger**: the event predicate (e.g. forge issue assigned + labelled\n `bugfix`, a schedule tick, a webhook payload shape).\n- **actions**: an ordered list, each naming a logic position and a\n registered implementation, with typed argument data and optional\n conditions.\n- **data flow between actions**: results/facts from one action available to\n later actions in the same playbook run -- exact interpolation syntax is\n an open point, not decided here (see Open questions).\n\nPlaybooks live in a configured directory (repo/directory-scoped, per Sam's\n\"define repos, directories, playbooks\" framing) and are loaded at archied\nstartup, alongside the existing skills-catalog composition.\n\n## Dedup: detect and log, no arbitration mechanism\n\nNo runtime auto-resolution rule. An automatic \"pick a winner\" algorithm\n(e.g. by version) is itself a mechanism that can fail and need debugging\nwhen a collision happens -- the wrong kind of complexity to introduce for\na case that should just be visible and fixed by a human. Instead:\n\n- **At daemon startup:** if independently-sourced playbook directories\n declare a collision on the same trigger, that is a definition failure.\n The colliding definitions are dropped (not loaded) and the failure is\n logged plainly (which sources, which trigger) and reported back to the\n caller that requested the load -- not swallowed as a background log line\n only. The schema is the definition; a schema conflict is a reported\n failure, not something the daemon silently arbitrates.\n- **Linter**: a CI/dev-time tool, reading the same generated schemas, that\n flags duplicate or conflicting trigger definitions *within* a single\n maintained tree before merge.\n\n **Resolved** (`archie-core-t2db.12`). A standalone\n gopls-shaped binary (`cmd/archie-playbooks`, lint subcommand) validates\n playbook directories against the exact loaders the daemon uses\n (`LoadPlaybookDirs`/`LoadKindWorkflowsYAML`/`LoadLabelWorkflowsYAML`),\n exiting non-zero on any collision / malformed file / invalid binding.\n Findings are file-granular, not line-granular: the loader decodes with\n `yaml.Unmarshal` into a plain map, which discards line numbers. A\n compiler-style file:line diagnostic needs a `yaml.Node` decoding\n upgrade, tracked separately -- the linter agrees with runtime\n validation by construction, which is the load-bearing property.\n Discoverable via `task lint:playbooks` or direct `go run\n ./cmd/archie-playbooks lint -dir ...`. The LSP/serve mode is a later\n entrypoint of the SAME binary, per the shared-package decision above.\n- **LSP**: same schema source again, feeding author-time hover/completion/\n inline errors while someone edits a playbook YAML in an editor. Kept\n explicitly in scope per Sam (not deferred), because it's the same schema\n artifact as the linter's, just a different consumer.\n\nThe startup log-and-refuse path exists for what the linter/LSP structurally\ncannot see -- sources composed together only at daemon startup, never\nchecked against each other by either tool. If the linter and LSP do the job\nthey're built for, a real collision reaching that startup path should be\nrare in practice, not a routine occurrence the design leans on.\n\n**Shared package, thin consumers -- not a bundled-vs-standalone choice.**\nThe validation logic (`LoadKindWorkflowsYAML`, `LoadLabelWorkflowsYAML`, the\ndirectory-merge from t2db.11) lives in `internal/domain/workflow`, a plain\nGo package with no knowledge of which binary calls it. This makes \"should\nthe linter/LSP be bundled into archied or a separate tool\" a non-question:\n- `archied` already imports the package directly and validates in-process\n at startup -- no subprocess, no external tool call.\n- An agent that needs to validate a playbook in-process can import the same\n package directly, for the same reason.\n- The standalone `cmd/` binary (t2db.12, gopls-shaped: one binary, a lint\n mode now, an LSP/serve mode later) exists only because CI and editors are\n out-of-process consumers by nature -- a CI step needs an exit code, an\n editor speaks LSP over a process boundary, neither can `import` a Go\n package. It is a thin wrapper per `organisation.md`'s `cmd/` rule, not a\n second implementation.\n\nFolding CLI/LSP protocol handling into `archied` itself would grow the\ndaemon into unrelated concerns, against the plugin-engine rule's spirit of\nnarrow capability families.\n\n## Execution-time gaps (both resolved)\n\nEverything above covers *loading* a playbook. Nothing above covers what\nhappens while one *runs*. Two gaps, both load-bearing enough to resolve\nbefore implementation, not defer:\n\n### 1. Mid-playbook action failure\n\n**Resolved** (`archie-core-t2db.18`). The default below\nis a committed decision, not a candidate list; changing it later would\nrequire its own decision. Gap 2 below is resolved by `archie-core-t2db.17`.\n\nA playbook is an ordered action list with data flowing between steps. If\naction 2 of 4 fails at runtime (an image-gen API times out, a forge call\n403s), the run needs a defined outcome. This\ncodebase already answers a structurally identical question for its\nexisting workflow engine, precisely enough to reuse rather than\nre-derive: `workflow.Run` (`internal/domain/workflow/workflow.go`)\ndraws a three-way distinction on a stage failure, and a playbook run\ndraws the same one on an action failure.\n\n**Decision: reuse `Run`'s three-way distinction directly.** The stated\ndefault is **stop-on-first-failure**: a real action error halts the\nplaybook run immediately and is reported, with no continue-to-next-action\nand no automatic retry or compensating follow-up. The three points below\nare the complete decision.\n\n1. **A real action error** (the action's own logic failed -- an API\n error, a validation failure inside the action) stops the run\n immediately. No later actions execute. The failure is reported the\n same way a load collision is reported -- dropped, logged plainly\n (which action and why), and returned to the caller, never swallowed\n as a background log line only (the Dedup section's drop-and-report\n rule). `Run` is the mechanical\n precedent for the stop itself: a stage error sets the park reason and\n returns without running further stages (`workflow.go`). This\n is stop-on-first-failure, matching the document's existing bias\n against clever automatic recovery, and it is now a stated decision\n rather than a nobody-chose default.\n2. **Interruption** (the daemon is shutting down mid-dispatch, `ctx.Err()\n != nil`) is explicitly **not** a failure. `Run` already treats this\n case specially -- \"Daemon shutdown is not a workflow failure. Leave\n the task running ...; parking here would publish a false failure and\n require manual intervention\" (`workflow.go`) -- and the same\n reasoning applies to a playbook run: an in-flight dispatch interrupted\n by restart must not be recorded as a broken playbook. What \"leave it\n alone for restart\" means for a playbook run (there is no `store.Task`\n row a playbook run inherently owns the way a workflow stage does) is\n an open implementation detail for whoever builds the coordinator, not\n decided here -- but the *semantic* (interruption ≠ failure) is decided.\n3. **A run that produces no outcome at all** (every action ran without\n error, but nothing was assigned to happen) is itself an error\n condition, matching `Run`'s own \"a workflow must end with an explicit\n outcome; not doing so is a definition bug, which still must not vanish\n silently\" (`workflow.go`). A playbook with actions that all ran\n but never reached a coherent terminal state is a definition bug in the\n playbook, reported as such.\n\n**Rollback:** explicitly **not attempted**, for exactly the reason the\ngap's own description named -- a posted forge comment is not revocable.\nThis is not a limitation to work around later; it is the correct\nsemantic. An operator who needs a corrective action after a failure\nwrites a *new* playbook/action for that, they do not get automatic\nundo.\n\n**This stays inside the producer-only rule.** Stop-on-first-failure is the\nrun deciding its own terminal state from its own action's error. Nothing\nalready in flight is blocked or altered; the run only declines to start\nactions that had not started yet, and it records the failure on its own\nunit the way `Run` sets a park reason and transitions the task to\n`StatusParked`. A future compensating action (\"if the notify action\nfailed, also log an incident\") would be producing new work, which the rule\nalready permits. No syntax for it is designed here.\n\n**J3 relationship, stated explicitly:** J3 (a CEL `when` condition\nerroring at eval time → treated as false → skip that action → run\ncontinues) is a **different case from an action failure** and is not\nrevisited by this resolution. A condition error means \"we couldn't\ndetermine whether to run this action,\" which this document already\nresolved as \"then don't, and move on.\" An action failure means \"we tried\nto run this action and it broke,\" which stops the run per point 1 above.\nThe two must not be conflated: a coordinator implementation should be\nable to point at one `when`-evaluation code path and one\naction-invocation code path and show they use different outcomes.\n\n### 2. Idempotency at execution time\n\n**Resolved** (`archie-core-t2db.17`, a\ndesign-investigation ticket: resolve, do not implement). The decision\nbelow is the answer; the keying scheme itself is separate follow-up work\nand is intentionally not built here. It **extends**\n`event-sources-and-reactions.md`'s existing contract -- it does not\ncontradict it (relationship argued at the end).\n\n**Problem (unchanged):** `event-sources-and-reactions.md` decided\nreactions ride `internal/eventbus`'s at-least-once delivery specifically\nso `PublishUnique`/idempotency keys absorb redelivery without duplicate\nside effects. This document inherits that for free only for\n`workflow`-kind actions, because `TaskEnvelope` already carries an\nidempotency key. It said nothing for `channel`, `forge`, or `module`\nactions with real side effects -- a posted comment, a sent message, a\nbilled image generation.\n\nOne correction to the problem statement, checked against the code rather\nthan assumed, because it changes what the answer must be:\n**`PublishUnique`'s dedup is not a consumer-redelivery guard at all.**\n`PublishUnique` only sets the JetStream `Nats-Msg-Id` header on the\noutgoing message (`internal/infrastructure/eventbus/nats/publisher.go`;\nkey constant `message.go`); JetStream suppresses *republished*\nmessages carrying a repeated key inside `Config.DedupWindow`\n(`client.go`, `Duplicates: cfg.DedupWindow`), default 2 minutes\n(`config.go`, `DefaultDedupWindow = 2 * time.Minute`). A consumer\nredelivery is the same *stored* message re-delivered on Nak or\nacknowledge-timeout (`internal/eventbus/eventbus.go`: \"a handler\nreturning an error causes redelivery\") -- not a republish; its\n`Nats-Msg-Id` is never re-firewalled, and even if it were, two minutes is\nnot an at-most-once guarantee for a side effect that must never repeat.\n**Conclusion: side-effecting action kinds need a consumer-side, durable\ndedup record; the bus cannot supply it.** That is precisely the situation\n`binding_dispatches` was built for -- so the decision below reuses its\nmechanism rather than designing a new one.\n\n#### Decision\n\n**Granularity: per-action-per-event -- one dedup record per\n(action, event) pair, not per playbook run.** Whole-run keying, the\nobvious alternative, has two crash windows: the run row is written when the run starts, so a crash mid-run\nleaves the run \"started\" and a redelivery skips *every remaining\naction* (work silently lost); a crash before the row write re-runs\n*every already-fired action* (duplicate side effects -- the exact harm\nthis gap exists to prevent). Per-action keys collapse both into one\nwindow and one behavior: an action's record is written immediately\nbefore that action's side effect is invoked, so an action fires iff its\nown record is absent, no fired action ever re-fires, and a redelivered\nevent resumes at the first unrecorded action. For the shipped\nsingle-action playbooks the two granularities are equivalent in effect;\nthe difference starts with multi-action runs -- where whole-run keying\nwould already be wrong.\n\n**Storage/lookup: a new durable ledger table `playbook_dispatches` in\n`internal/store`, copying `binding_dispatches`'s conventions exactly**\n(durable table, `INSERT OR IGNORE`, sentinel error, rows freed with\ntheir parent, survives daemon restarts -- `internal/store/bindings.go`):\n\n```sql\nCREATE TABLE playbook_dispatches (\n playbook_id TEXT NOT NULL,\n playbook_version TEXT NOT NULL,\n event_id TEXT NOT NULL,\n action_id TEXT NOT NULL,\n dispatched_at TEXT NOT NULL,\n PRIMARY KEY (playbook_id, playbook_version, event_id, action_id)\n);\n```\n\nWritten with the same `INSERT OR IGNORE` + sentinel-error convention\n(`ErrAlreadyDispatched`, `bindings.go`), so a duplicate is a no-op\nwrite rather than a constraint error, and the caller matches it with\n`errors.Is` the way the binding dispatch loop already does\n(`internal/daemon/daemon.go` -- \"another cycle already won this\nrace,\" not an error to surface). The record is consumed by the playbook\ncoordinator at dispatch time through a domain-side interface implemented\nby `internal/store` -- the same shape as `BindingDispatcher`\n(`internal/store/interface.go`) -- and the table lands in the\nstore's schema application (`internal/store/store.go`). No change\nto `internal/eventbus`; no per-message dedup state on the client.\n\n**Lifetime: no time-based expiry.** Rows live as long as their playbook\nexists; when a playbook is removed from the configured directories, the\ncoordinator deletes its rows -- mirroring binding-dispatches-rows-deleted-\nwith-binding (`bindings.go`, same transaction as the delete). The\nat-most-once guarantee must not silently decay, which is exactly the\nproperty that disqualifies `DedupWindow` above; table growth is bounded\nby distinct (playbook, event, action) pairs over the playbook's lifetime\nand is accepted, the same tradeoff `binding_dispatches` already makes.\n\n**Redelivery behavior: skip the recorded action and continue the run.**\nA detected duplicate is not re-invoked, is not reported to the caller as\nan error, and is logged at debug level rather than as a warning -- this\nis the expected, correct behavior of at-least-once delivery, not an\nanomaly (same treatment as `ErrAlreadyDispatched` at\n`daemon.go`).\nThe run continues with the next action (or ends, for a single-action\nplaybook whose one action is recorded), which is what makes a redelivery\na resume rather than a dead restart.\n\n**Key derivation: a fixed composition of four structural identities,\ncomputed by the coordinator -- not a CEL expression.** The key is the\n`(playbook_id, playbook_version, event_id, action_id)` tuple above, where:\n\n- `playbook_id` is `Playbook.ID` -- the playbook file's path relative to\n its configured directory root, slash-normalized, unique within the load\n composition by construction (`internal/domain/eda/playbook`).\n- `playbook_version` is `Playbook.Version` -- a SHA-256 of the loaded\n file, recomputed on every load (`internal/domain/eda/playbook`),\n mirroring `Binding.Version`'s provenance-pinning purpose: the dispatch\n is pinned to the exact definition that was active when it fired. A\n changed playbook definition is a different key and may fire again for\n the same event -- deliberate, not a bug.\n- `event_id` is `DispatchInput.TaskID` (`playbook.go`), which\n carries the `TaskEnvelope.IdempotencyKey()` value\n (`\"archie:\" + owner/repo/number`,\n `internal/domain/workintake/envelope.go`). It is chosen over\n `store.Task.ID` because the trigger vocabulary is issue-level\n workintake label/kind, dispatched at discovery time (pollNATS, forge\n webhook receiver), where no task row exists yet (`playbook.go`\n comment; `playbook_test.go`: \"the value available at the\n discovery/dispatch point... NOT a store.Task.ID int64\"). Keying on the\n issue identity rather than the delivery source is the load-bearing\n property the existing contract already established for `TaskEnvelope`\n (`event-sources-and-reactions.md` question 4: a webhook-sourced\n envelope for the same issue produces the *same* idempotency key as a\n poll-discovered one; `internal/forge/webhook/receiver.go` --\n \"Idempotency is not this package's job\"); this resolution inherits that\n property rather than re-deciding it. Consequence, stated explicitly:\n same-issue re-discoveries (a re-label that re-triggers the same kind)\n are the *same event* by design -- the workintake trigger vocabulary is\n issue-granular, exactly as `TaskEnvelope`'s own key is. A future\n capture/binding-originated trigger type keys on `captured_events.id`\n instead; derivation is per-trigger-type, not one formula for all\n triggers (rule carried over unchanged from the draft).\n- `action_id` is the action's declared `id` -- J1's named ids, \"a\n required field on every action that later actions reference\", unique\n and stable-identifier-shaped, validated at load (the CEL resolution's\n J1 and its context table above in this document). For an action declaring no `id`\n (permitted by J1 while unreferenced), the key uses the action's 1-based\n position in the playbook: deterministic and stable for the\n single-action playbooks, and a later reorder is a file edit whose\n content hash changes `playbook_version` with it, so a positional\n fallback cannot silently collide with a prior definition.\n\n**No CEL in the key -- stated explicitly because the CEL resolution\nflagged this as gap 2's \"first real exercise\"** (\"CEL's read-only result\naccess makes gap 2's keying derivation readable\" -- the CEL resolution's\nclosing note). Readability\nis not the binding constraint; **determinism under redelivery is**, and\nCEL key expressions lose three ways. First, they are not load-checkable:\n`event` stays `dyn` by the resolved J4 (schema-by-example), so\na key expression over event fields cannot be type-checked at load,\nviolating the document's reject-at-load philosophy. Second, they can\nfail at dispatch (a missing field is a CEL error), and a key that cannot\nbe computed is an unfireable state -- neither \"skip\" nor \"fire\" is safe\nwithout losing or duplicating work. Third, the contract needs one\nderivation, and two authors writing different key expressions would\nfragment the ledger. The key is therefore structural by construction:\nevery component above is an immutable identity of the run's inputs, so\ntwo deliveries of the same event derive the same key with no evaluation.\nThe exercise's actual outcome is the derivation rule above; CEL remains\na *condition* surface (`when`), never a key surface. An action kind that\ngenuinely needs finer granularity than (action, event), such as \"per\ncomment, not per issue\", arrives with its own trigger type and its own\nevent identity, per the per-trigger-type rule.\n\n**Record-before-invoke -- one deliberate divergence from\n`binding_dispatches`, justified by the harm profile.** The binding\ndispatch loop enqueues the task first and records the ledger row after\n(`daemon.go`), accepting that racing cycles may both enqueue\n(\"the duplicate task remains queued... delete-on-races is its own can of\nworms\"); the selection query excludes already-dispatched captures\n(`bindings.go`), but the race window remains. That ordering is\ncorrect there because the duplicate artifact is an internal task row --\nrecoverable, low-harm. The playbook ledger's ordering is reversed on\npurpose: **the row is written and committed before the action's side\neffect is invoked** -- the `INSERT OR IGNORE` is the atomic gate, and\nthe lost-write loser skips without invoking. The cost is stated, not\nhidden: a crash between the row commit and the invoke loses that one\naction's side effect, repairable by hand -- the reverse direction of the\nsame window that would otherwise duplicate a posted comment or\nre-bill an image generation. For side effects whose duplication is\nexternally visible and not revocable, at-most-once is the required\ndirection; the binding system's at-least-once-tolerant ordering is not\nthe precedent for this ledger.\n\n**Gap-1 interplay, stated to keep the two mechanisms separate:** the\nledger answers one question per action -- has this action already fired\nfor this event? It does not answer whether the run continues, stops, or\nis re-attempted; that is gap 1's coordinator concern (a run-outcome\nrecord; gap 1 explicitly leaves \"what leave it alone for restart means\nfor a playbook run\" to whoever builds the coordinator). One consequence\nof record-before: an action that FAILED (gap 1's real-action-error case)\nhas a ledger row, so a later redelivery of the same event skips it --\nconsistent with gap 1's \"no automatic retry\" resolve (a corrective\naction is a *new* playbook/action, per gap 1's rollback paragraph). A\nfailed action is never re-attempted by the bus; if a run must be\nrecovered, an operator edits the playbook -- new version, new key --\nrather than fighting ledger state.\n\n**Relationship to `event-sources-and-reactions.md` -- extension, not\ncontradiction.** Keyed to that document's sections:\n\n1. Its decision 2 commits reactions to at-least-once delivery via\n `PublishUnique`'s \"without duplicate enqueue\" guarantee -- which, with\n the code facts above, is accurate for the *publish* path only:\n `PublishUnique` absorbs duplicate publishes of the same event (poll\n vs. webhook), which is what that decision was actually about. This\n document's own problem statement, taken literally, overstates it\n (\"keys absorb redelivery without duplicate side effects\") -- that\n overstatement is exactly what this gap exists to correct. The\n existing contract never claimed consumer redeliveries of\n side-effecting reactions were absorbed, so nothing it says contradicts\n a consumer-side ledger; this resolution pins down the boundary the\n sentence left implicit (publish-path dedup vs. consumer-path dedup)\n rather than changing either.\n2. Its decision 3 is untouched: the ledger is the run's own bookkeeping\n about its own actions, suppressing a re-fire and nothing else.\n3. Its decision 4 (no generic `Source` interface; `TaskEnvelope` is the\n typed contract, keyed on issue identity, not delivery source): the\n event half of the playbook key IS that same identity\n (`TaskEnvelope.IdempotencyKey()`), and the per-trigger-type rule\n means any future trigger type brings its own identity rather than a\n new generic one -- the playbook ledger introduces no second identity\n vocabulary and no untyped payload. The only new type in this\n resolution is the ledger's own value object, consumed through a\n narrow interface like `BindingDispatcher` -- the same shape the\n document's \"typed families\" rule (decision 1, plugin engine rule)\n already blesses.\n\n**Scope: this is the `channel`/`forge`/`module` decision, not a\n`workflow`-position change.** Workflow-kind actions keep their existing\nkeying claim unchanged -- the `TaskEnvelope` path that this document\nalready inherits for free (`daemon.go`, `PublishTask` publishes\nwith `task.IdempotencyKey()`), with its cross-poll/webhook property\nuntouched. The ledger gate applies to the side-effecting positions\n(channel, forge, module), which is why the closing statement below\nstill holds: the first slice (Module + loader + workflow-kind dispatch)\nwas buildable without this resolution, and channel/forge/multi-action\nplaybooks were not. A coordinator MAY implement one position-uniform\ngate as a simplification; what this resolution requires is only that\nthe side-effecting positions are gated.\n\n## Resolved questions\n\n1. **Data-flow/condition syntax.** Does this project want a small existing\n Go expression evaluator (there may be one already in the codebase worth\n checking, e.g. anything backing gate conditions), or a bespoke minimal\n grammar?\n\n **Resolved** (`archie-core-t2db.14`). CEL (`cel.dev/cel-go`, pinned) is\n the one mechanism for both an action's `when` condition and its `args`\n values. The trust boundary, evaluation context, load-time validation and\n the five judgment calls are in\n `docs/prds/playbook-expression-syntax.md`.\n\n2. **Playbook directory config field.** Where this lives in\n `internal/config` / `configuration.Document`, and whether it's one\n directory or a list (mirroring `SkillsDir`'s shape) -- follow existing\n config precedent, not invented fresh.\n\n **Resolved** (`archie-core-t2db.11`). The directory\n shape landed as `playbook_dirs` (a LIST of directories of\n `*.yaml`/`*.yml` binding files), loaded at startup as an additional\n input to the two single-file fields, which remain supported unchanged.\n A single `playbook_dir` would contradict\n supporting multiple independently-maintained playbook sources, which\n the Dedup section already assumes ('independently-sourced playbook\n directories collide'), so the field is a list. Cross-source collision\n (same key in two directories, or in a single-file field and a\n directory) is reported like an in-directory collision -- nothing is\n arbitrated by source precedence. This is what finally exercises the\n design doc's drop-and-report collision rule, since a single file\n cannot collide with itself. Repo-scoping (tying directories to\n `Config.Repos` owner/name) is explicitly deferred to a separate,\n later decision; org/tenant keying is out of scope (no auth/identity\n model).\n\n Decisions recorded from the label-vocabulary slice:\n - Arbitrary labels bind to registered workflow names via\n `LabelWorkflows` + `LoadLabelWorkflowsYAML`/`SetLabelWorkflows`\n (`internal/domain/workflow/routing.go`).\n - The closed `Kind`/NATS-subject set is untouched; the label map only\n extends binding authority for labels the kind layer does not own.\n - Collision rule: a label already owned by the\n kind set (bug/feature/bootstrap), an empty label, an empty workflow\n name, or a duplicate binding is a reported load failure -- dropped\n and logged, the error returned to the caller. Nothing is silently\n arbitrated by schema/version. This is the \"schema defines the\n accepted message; anything else is the caller's fault\" rule applied\n to the label layer.\n - Precedence in `Route()`: explicit `t.Workflow` → arbitrary-label\n binding → kind binding → triage → implement → default.\n3. **Trust boundary for Module Yaegi code.** The plugin engine rule's\n invariant 6 draws a line between operator-trusted in-process code and\n repository-supplied code that must run in a container.\n\n **Resolved** in `docs/prds/module-position.md`, \"Trust boundary\": a\n Module is operator-installed, in-process and daemon-privileged, the same\n tier as `PluginDir` and `SecretEngineDir`, never repository-supplied\n task code.\n4. **First implementation slice.** Recommend: Module position + the\n playbook loader + trigger-to-workflow dispatch only (subsuming the existing\n label routing) first, proving the schema-gen -\u003e Yaegi -\u003e playbook path\n end to end on the smallest useful case, before adding Channel/Forge\n action kinds or the linter/LSP.\n\n **Resolved** (`archie-core-t2db.13`, `.14`, `.15`). The Module position\n (registry + log kind, t2db.13), the CEL expression environment\n (t2db.14), and the playbook document + event coordinator with\n single-action workflow-kind dispatch (t2db.15) are shipped.\n\n **Resolved** (`archie-core-t2db.23`): the coordinator selects a real\n task's workflow. The wiring point is the daemon's definition pin\n (`Daemon.resolveWorkflowID`, consumed by `pinWorkflowFromCollection` in\n `internal/daemon`), not `pollNATS`/`publishTask`. Routing belongs to the\n pin because a workflow definition is a\n database row pinned before dispatch, which makes the pin the one place a\n task's workflow is chosen. Precedence there is explicit `Task.Workflow`\n (the waiting_human -\u003e approved requeue) → matching playbook →\n `workflow.ResolveWorkflowID`'s label binding → kind binding → triage →\n implement. A playbook naming an undefined workflow is a reported failure\n that parks the task rather than a silent fall-through to the binding's\n choice.\n\n `Store.Dispatch` returns the selected workflow *name* and the matching\n playbook's id and version, not a compiled `workflow.Workflow`: the\n production caller decides against the active definition collection, and\n the id and version are the first two components of the gap-2 idempotency\n key. The playbook package does not import `internal/domain/workflow`.\n\n A `when` condition at this point reads what the intake knows about the\n originating issue: `kind`, `labels`, `owner`, `repo`, `number`, `title`,\n `body`, `source`, built by `playbookInput` in `internal/daemon`.\n\n The trigger shape reuses the existing workintake kind/label vocabulary.\n The single-action hard boundary is superseded by\n `docs/prds/multi-action-playbooks.md` (two-shape rule); Module, Channel,\n and Forge positions still have no production dispatch path.\n## Standing constraint\n\nThis may be commercialized, and is to be discernible from other open-source\nevent-driven-automation tooling. No design decision follows from that yet;\nit bears on licensing or feature-gating questions about playbook authoring\nand the LSP when they arise.\n\n## Packages this touches (first slice)\n\n- New: a schema-generation package/convention (`go:generate` source +\n emitted contracts) for the Module position.\n- New: playbook loader + event coordinator (name/location TBD -- likely\n `internal/domain/eda` or similar, following the domain-layer rule: no\n infrastructure imports, no untyped hooks).\n- `internal/domain/workflow`: dispatch target for `workflow`-kind actions,\n unchanged in its own contracts.\n- `internal/domain/workintake`: `labelKinds`/routing becomes one playbook\n among others once the loader subsumes it; existing Go fallback stays for\n backward compatibility until playbooks fully replace it (mirrors the\n `dynamic-workflow-triage.md` precedent of a two-line fallback change, not\n a rip-and-replace).\n" + "body": "# EDA playbook engine -- design\n\n**Status:** Draft\n**Date:** 2026-09-03\n**Beads epic:** `archie-core-t2db`\n\n## Problem\n\n\"Which workflow runs for which trigger\" is hardcoded Go: a literal\nlabel vocabulary (`workintake.labelKinds`) and a literal label-\u003eworkflow map\n(`workflow/routing.go`), extended only by adding a Go case or shipping a\nskill whose `metadata.archie.workflow` overrides one *named* built-in\nworkflow. Extending the system to a new trigger, a new action, or a new\nevent source means a feature branch, a code change in a hardcoded\npath, and new Go tests -- for every addition. That is the barrier this\ndocument exists to remove.\n\nThe goal: triggers, the actions they run, and the\nconditions that connect them should be **drop-in and load**, not **drop-in\nand touch application code**. Yaegi-evaluated Go is the unit of imperative\n\"do a thing\" logic; YAML playbooks are pure orchestration data -- no logic,\njust \"on this event, with this data, call these actions, in this order,\nunder these conditions.\" The daemon's event coordinator doesn't know what a\n`bugfix` or `image-gen` action does, it resolves the name against a\nregistry and pipes typed data through.\n\nThis is explicitly **not** scoped to code-generation workflows. A playbook\naction can be any registered capability -- run a codegen workflow, send a\nmessage on a channel, open something on a forge, generate an image -- so\nlong as it's implemented behind one of the typed engine families below.\n\n## Relationship to existing decisions\n\n- `docs/prds/event-sources-and-reactions.md` ruled\n that reactions in the two epics it covers are **producer-only** (no veto,\n no mutation of in-flight work) and explicitly rejected a single generic\n `Source` interface as \"the untyped-hook shape the plugin engine rule\n exists to prevent.\" **This document extends that scope**: playbooks can\n dispatch to multiple typed action kinds, not just \"produce a\n `TaskEnvelope`.\" It does not relitigate producer-only-ness (still no\n veto/mutation of in-flight work here) and it does not introduce one\n generic interface -- see \"Typed positions, not one generic hook\" below,\n which is how it stays compliant with the plugin engine rule instead of\n reopening that document's #4.\n- `docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`\n governs every extension point\n introduced here. Every position below is a typed domain contract with an\n owning registry, not a generic callback map. This is a hard constraint,\n confirmed against the existing rule text before writing this doc.\n- Yaegi is already a proven in-process extension mechanism\n (`internal/plugin`, `internal/domain/workflow/wfeval/yaegi.go`,\n `internal/gate/gateeval/yaegi.go`, `internal/skill/plugin.go`) -- this\n design generalizes its existing role, it does not introduce a new\n execution mechanism.\n- `internal/domain/workflow/skillbuild.BuildRegistry` already composes a\n `workflow.Registry` from a skills catalog, plugin-defined overriding\n built-in. The playbook loader described here supersedes label routing\n specifically; it does not replace `skillbuild`'s workflow-composition role,\n it becomes an additional input the daemon composes at startup.\n- Webhook event intake is a separate design. This\n document defines the router each webhook-sourced event should be able to\n reach; it does not redesign webhook receipt itself.\n\n## Shape: four logic positions, each a typed engine family\n\nEvery entrypoint kind (mechanism, action, loop, schedule, route, trigger) is itself a schema emitter: defining a new kind produces the\ngenerated contract downstream tooling consumes. The kinds slot into a fixed\nhierarchy of logic positions:\n\n```\nWorkflow -\u003e Module -\u003e Channel (telegram/whatsapp/email/...)\n -\u003e Forge (GitHub/Gitea/GitLab/...)\n```\n\nEach position is its own typed engine family per the plugin engine rule:\nidentity + capability-specific operations, an owning `Registry`/`Manager`,\nexplicit lifecycle where the implementation owns resources, narrow host\naccess. Concretely:\n\n- **Workflow** -- already exists (`internal/domain/workflow`). A playbook\n action of kind `workflow` invokes a registered `workflow.Registry` entry\n by name, unchanged from the existing execution path.\n- **Module** -- the new general-purpose action position. A `Module` is a\n Yaegi-evaluated implementation of a generated interface (args in, typed\n result out) for one declared action kind (e.g. `image-gen`, `notify`,\n `open-pr-comment`). This is the position that makes actions \"not\n restricted to code-generation.\" Full design (schema-per-kind, the\n registry shape, the trust boundary, the recommended first slice):\n `docs/prds/module-position.md`.\n- **Channel** -- already exists as `channels.Channel` (telegram, email,\n webhook). A playbook action of kind `channel` sends through a named,\n already-registered channel. No new mechanism; this document just makes it\n addressable from a playbook.\n- **Forge** -- forge clients (GitHub, Gitea, GitLab) become an addressable\n action position the same way Channel is, so a playbook can say \"comment\n on this forge issue\" without that being a special case wired only into\n the poller.\n\n**Typed positions, not one generic hook.** There is no single `Action`\ninterface with an `any` payload. Each position keeps its own typed\ncontract; the playbook YAML names a position and an implementation, and the\nevent coordinator resolves that pair against the matching typed registry.\nThis is what keeps the design compliant with the plugin engine rule and\nwith `event-sources-and-reactions.md`'s rejection of a generic `Source`/\nhook interface -- the generality lives in the YAML data model (which\npositions exist, which named implementations are registered), not in a\nruntime `any`-typed dispatch path.\n\n## `go:generate`-driven schemas, from day one\n\nSchemas are not a future nicety bolted onto an ad hoc\nYAML shape. They are the struct+interface contracts for each logic\nposition, and they drive code generation from the start.\n\n- Each logic position (Workflow/Module/Channel/Forge) has a schema source\n (Go-annotated types, following the existing pattern elsewhere in the\n repo) from which `go:generate` emits the interface and struct scaffolding\n an implementation must satisfy.\n- A Yaegi module implements the generated interface for its position. The\n generated contract is what makes an implementation valid, not hand-review.\n- The schema is also what the linter and LSP (below) validate a playbook\n YAML against -- one source of truth for codegen, startup loading, and\n authoring-time tooling, so the three can't drift from each other.\n- Schema versioning is explicit per position, independent of a playbook's\n own workflow version (see collision handling below).\n\nThe generator itself is not designed here. The constraint is\nschema-first, `go:generate`-driven, per logic position.\n\n## Playbook YAML: pure orchestration data\n\nA playbook declares no embedded logic -- only:\n\n- **trigger**: the event predicate (e.g. forge issue assigned + labelled\n `bugfix`, a schedule tick, a webhook payload shape).\n- **actions**: an ordered list, each naming a logic position and a\n registered implementation, with typed argument data and optional\n conditions.\n- **data flow between actions**: results/facts from one action available to\n later actions in the same playbook run -- exact interpolation syntax is\n an open point, not decided here (see Open questions).\n\nPlaybooks live in a configured directory (repo/directory-scoped, per Sam's\n\"define repos, directories, playbooks\" framing) and are loaded at archied\nstartup, alongside the existing skills-catalog composition.\n\n## Dedup: detect and log, no arbitration mechanism\n\nNo runtime auto-resolution rule. An automatic \"pick a winner\" algorithm\n(e.g. by version) is itself a mechanism that can fail and need debugging\nwhen a collision happens -- the wrong kind of complexity to introduce for\na case that should just be visible and fixed by a human. Instead:\n\n- **At daemon startup:** if independently-sourced playbook directories\n declare a collision on the same trigger, that is a definition failure.\n The colliding definitions are dropped (not loaded) and the failure is\n logged plainly (which sources, which trigger) and reported back to the\n caller that requested the load -- not swallowed as a background log line\n only. The schema is the definition; a schema conflict is a reported\n failure, not something the daemon silently arbitrates.\n- **Linter**: a CI/dev-time tool, reading the same generated schemas, that\n flags duplicate or conflicting trigger definitions *within* a single\n maintained tree before merge.\n\n **Resolved** (`archie-core-t2db.12`). A standalone\n gopls-shaped binary (`cmd/archie-playbooks`, lint subcommand) validates\n playbook directories against the exact loaders the daemon uses\n (`LoadPlaybookDirs`/`LoadKindWorkflowsYAML`/`LoadLabelWorkflowsYAML`),\n exiting non-zero on any collision / malformed file / invalid binding.\n Findings are file-granular, not line-granular: the loader decodes with\n `yaml.Unmarshal` into a plain map, which discards line numbers. A\n compiler-style file:line diagnostic needs a `yaml.Node` decoding\n upgrade, tracked separately -- the linter agrees with runtime\n validation by construction, which is the load-bearing property.\n Discoverable via `task lint:playbooks` or direct `go run\n ./cmd/archie-playbooks lint -dir ...`. The LSP/serve mode is a later\n entrypoint of the SAME binary, per the shared-package decision above.\n- **LSP**: same schema source again, feeding author-time hover/completion/\n inline errors while someone edits a playbook YAML in an editor. Kept\n explicitly in scope per Sam (not deferred), because it's the same schema\n artifact as the linter's, just a different consumer.\n\nThe startup log-and-refuse path exists for what the linter/LSP structurally\ncannot see -- sources composed together only at daemon startup, never\nchecked against each other by either tool. If the linter and LSP do the job\nthey're built for, a real collision reaching that startup path should be\nrare in practice, not a routine occurrence the design leans on.\n\n**Shared package, thin consumers -- not a bundled-vs-standalone choice.**\nThe validation logic (`LoadKindWorkflowsYAML`, `LoadLabelWorkflowsYAML`, the\ndirectory-merge from t2db.11) lives in `internal/domain/workflow`, a plain\nGo package with no knowledge of which binary calls it. This makes \"should\nthe linter/LSP be bundled into archied or a separate tool\" a non-question:\n- `archied` already imports the package directly and validates in-process\n at startup -- no subprocess, no external tool call.\n- An agent that needs to validate a playbook in-process can import the same\n package directly, for the same reason.\n- The standalone `cmd/` binary (t2db.12, gopls-shaped: one binary, a lint\n mode now, an LSP/serve mode later) exists only because CI and editors are\n out-of-process consumers by nature -- a CI step needs an exit code, an\n editor speaks LSP over a process boundary, neither can `import` a Go\n package. It is a thin wrapper per `organisation.md`'s `cmd/` rule, not a\n second implementation.\n\nFolding CLI/LSP protocol handling into `archied` itself would grow the\ndaemon into unrelated concerns, against the plugin-engine rule's spirit of\nnarrow capability families.\n\n## Execution-time gaps (both resolved)\n\nEverything above covers *loading* a playbook. Nothing above covers what\nhappens while one *runs*. Two gaps, both load-bearing enough to resolve\nbefore implementation, not defer:\n\n### 1. Mid-playbook action failure\n\n**Resolved** (`archie-core-t2db.18`). The default below\nis a committed decision, not a candidate list; changing it later would\nrequire its own decision. Gap 2 below is resolved by `archie-core-t2db.17`.\n\nA playbook is an ordered action list with data flowing between steps. If\naction 2 of 4 fails at runtime (an image-gen API times out, a forge call\n403s), the run needs a defined outcome. This\ncodebase already answers a structurally identical question for its\nexisting workflow engine, precisely enough to reuse rather than\nre-derive: `workflow.Run` (`internal/domain/workflow/workflow.go`)\ndraws a three-way distinction on a stage failure, and a playbook run\ndraws the same one on an action failure.\n\n**Decision: reuse `Run`'s three-way distinction directly.** The stated\ndefault is **stop-on-first-failure**: a real action error halts the\nplaybook run immediately and is reported, with no continue-to-next-action\nand no automatic retry or compensating follow-up. The three points below\nare the complete decision.\n\n1. **A real action error** (the action's own logic failed -- an API\n error, a validation failure inside the action) stops the run\n immediately. No later actions execute. The failure is reported the\n same way a load collision is reported -- dropped, logged plainly\n (which action and why), and returned to the caller, never swallowed\n as a background log line only (the Dedup section's drop-and-report\n rule). `Run` is the mechanical\n precedent for the stop itself: a stage error sets the park reason and\n returns without running further stages (`workflow.go`). This\n is stop-on-first-failure, matching the document's existing bias\n against clever automatic recovery, and it is now a stated decision\n rather than a nobody-chose default.\n2. **Interruption** (the daemon is shutting down mid-dispatch, `ctx.Err()\n != nil`) is explicitly **not** a failure. `Run` already treats this\n case specially -- \"Daemon shutdown is not a workflow failure. Leave\n the task running ...; parking here would publish a false failure and\n require manual intervention\" (`workflow.go`) -- and the same\n reasoning applies to a playbook run: an in-flight dispatch interrupted\n by restart must not be recorded as a broken playbook. What \"leave it\n alone for restart\" means for a playbook run (there is no `store.Task`\n row a playbook run inherently owns the way a workflow stage does) is\n an open implementation detail for whoever builds the coordinator, not\n decided here -- but the *semantic* (interruption ≠ failure) is decided.\n3. **A run that produces no outcome at all** (every action ran without\n error, but nothing was assigned to happen) is itself an error\n condition, matching `Run`'s own \"a workflow must end with an explicit\n outcome; not doing so is a definition bug, which still must not vanish\n silently\" (`workflow.go`). A playbook with actions that all ran\n but never reached a coherent terminal state is a definition bug in the\n playbook, reported as such.\n\n**Rollback:** explicitly **not attempted**, for exactly the reason the\ngap's own description named -- a posted forge comment is not revocable.\nThis is not a limitation to work around later; it is the correct\nsemantic. An operator who needs a corrective action after a failure\nwrites a *new* playbook/action for that, they do not get automatic\nundo.\n\n**This stays inside the producer-only rule.** Stop-on-first-failure is the\nrun deciding its own terminal state from its own action's error. Nothing\nalready in flight is blocked or altered; the run only declines to start\nactions that had not started yet, and it records the failure on its own\nunit the way `Run` sets a park reason and transitions the task to\n`StatusParked`. A future compensating action (\"if the notify action\nfailed, also log an incident\") would be producing new work, which the rule\nalready permits. No syntax for it is designed here.\n\n**J3 relationship, stated explicitly:** J3 (a CEL `when` condition\nerroring at eval time → treated as false → skip that action → run\ncontinues) is a **different case from an action failure** and is not\nrevisited by this resolution. A condition error means \"we couldn't\ndetermine whether to run this action,\" which this document already\nresolved as \"then don't, and move on.\" An action failure means \"we tried\nto run this action and it broke,\" which stops the run per point 1 above.\nThe two must not be conflated: a coordinator implementation should be\nable to point at one `when`-evaluation code path and one\naction-invocation code path and show they use different outcomes.\n\n### 2. Idempotency at execution time\n\n**Resolved** (`archie-core-t2db.17`, a\ndesign-investigation ticket: resolve, do not implement). The decision\nbelow is the answer; the keying scheme itself is separate follow-up work\nand is intentionally not built here. It **extends**\n`event-sources-and-reactions.md`'s existing contract -- it does not\ncontradict it (relationship argued at the end).\n\n**Problem (unchanged):** `event-sources-and-reactions.md` decided\nreactions ride `internal/eventbus`'s at-least-once delivery specifically\nso `PublishUnique`/idempotency keys absorb redelivery without duplicate\nside effects. This document inherits that for free only for\n`workflow`-kind actions, because `TaskEnvelope` already carries an\nidempotency key. It said nothing for `channel`, `forge`, or `module`\nactions with real side effects -- a posted comment, a sent message, a\nbilled image generation.\n\nOne correction to the problem statement, checked against the code rather\nthan assumed, because it changes what the answer must be:\n**`PublishUnique`'s dedup is not a consumer-redelivery guard at all.**\n`PublishUnique` only sets the JetStream `Nats-Msg-Id` header on the\noutgoing message (`internal/infrastructure/eventbus/nats/publisher.go`;\nkey constant `message.go`); JetStream suppresses *republished*\nmessages carrying a repeated key inside `Config.DedupWindow`\n(`client.go`, `Duplicates: cfg.DedupWindow`), default 2 minutes\n(`config.go`, `DefaultDedupWindow = 2 * time.Minute`). A consumer\nredelivery is the same *stored* message re-delivered on Nak or\nacknowledge-timeout (`internal/eventbus/eventbus.go`: \"a handler\nreturning an error causes redelivery\") -- not a republish; its\n`Nats-Msg-Id` is never re-firewalled, and even if it were, two minutes is\nnot an at-most-once guarantee for a side effect that must never repeat.\n**Conclusion: side-effecting action kinds need a consumer-side, durable\ndedup record; the bus cannot supply it.** That is precisely the situation\n`binding_dispatches` was built for -- so the decision below reuses its\nmechanism rather than designing a new one.\n\n#### Decision\n\n**Granularity: per-action-per-event -- one dedup record per\n(action, event) pair, not per playbook run.** Whole-run keying, the\nobvious alternative, has two crash windows: the run row is written when the run starts, so a crash mid-run\nleaves the run \"started\" and a redelivery skips *every remaining\naction* (work silently lost); a crash before the row write re-runs\n*every already-fired action* (duplicate side effects -- the exact harm\nthis gap exists to prevent). Per-action keys collapse both into one\nwindow and one behavior: an action's record is written immediately\nbefore that action's side effect is invoked, so an action fires iff its\nown record is absent, no fired action ever re-fires, and a redelivered\nevent resumes at the first unrecorded action. For the shipped\nsingle-action playbooks the two granularities are equivalent in effect;\nthe difference starts with multi-action runs -- where whole-run keying\nwould already be wrong.\n\n**Storage/lookup: a new durable ledger table `playbook_dispatches` in\n`internal/store`, copying `binding_dispatches`'s conventions exactly**\n(durable table, `INSERT OR IGNORE`, sentinel error, rows freed with\ntheir parent, survives daemon restarts -- `internal/store/bindings.go`):\n\n```sql\nCREATE TABLE playbook_dispatches (\n playbook_id TEXT NOT NULL,\n playbook_version TEXT NOT NULL,\n event_id TEXT NOT NULL,\n action_id TEXT NOT NULL,\n dispatched_at TEXT NOT NULL,\n PRIMARY KEY (playbook_id, playbook_version, event_id, action_id)\n);\n```\n\nWritten with the same `INSERT OR IGNORE` + sentinel-error convention\n(`ErrAlreadyDispatched`, `bindings.go`), so a duplicate is a no-op\nwrite rather than a constraint error, and the caller matches it with\n`errors.Is` the way the binding dispatch loop already does\n(`internal/daemon/daemon.go` -- \"another cycle already won this\nrace,\" not an error to surface). The record is consumed by the playbook\ncoordinator at dispatch time through a domain-side interface implemented\nby `internal/store` -- the same shape as `BindingDispatcher`\n(`internal/store/interface.go`) -- and the table lands in the\nstore's schema application (`internal/store/store.go`). No change\nto `internal/eventbus`; no per-message dedup state on the client.\n\n**Lifetime: no time-based expiry.** Rows live as long as their playbook\nexists; when a playbook is removed from the configured directories, the\ncoordinator deletes its rows -- mirroring binding-dispatches-rows-deleted-\nwith-binding (`bindings.go`, same transaction as the delete). The\nat-most-once guarantee must not silently decay, which is exactly the\nproperty that disqualifies `DedupWindow` above; table growth is bounded\nby distinct (playbook, event, action) pairs over the playbook's lifetime\nand is accepted, the same tradeoff `binding_dispatches` already makes.\n\n**Redelivery behavior: skip the recorded action and continue the run.**\nA detected duplicate is not re-invoked, is not reported to the caller as\nan error, and is logged at debug level rather than as a warning -- this\nis the expected, correct behavior of at-least-once delivery, not an\nanomaly (same treatment as `ErrAlreadyDispatched` at\n`daemon.go`).\nThe run continues with the next action (or ends, for a single-action\nplaybook whose one action is recorded), which is what makes a redelivery\na resume rather than a dead restart.\n\n**Key derivation: a fixed composition of four structural identities,\ncomputed by the coordinator -- not a CEL expression.** The key is the\n`(playbook_id, playbook_version, event_id, action_id)` tuple above, where:\n\n- `playbook_id` is `Playbook.ID` -- the playbook file's path relative to\n its configured directory root, slash-normalized, unique within the load\n composition by construction (`internal/domain/eda/playbook`).\n- `playbook_version` is `Playbook.Version` -- a SHA-256 of the loaded\n file, recomputed on every load (`internal/domain/eda/playbook`),\n mirroring `Binding.Version`'s provenance-pinning purpose: the dispatch\n is pinned to the exact definition that was active when it fired. A\n changed playbook definition is a different key and may fire again for\n the same event -- deliberate, not a bug.\n- `event_id` is `DispatchInput.TaskID` (`playbook.go`), which\n carries the `TaskEnvelope.IdempotencyKey()` value\n (`\"archie:\" + owner/repo/number`,\n `internal/domain/workintake/envelope.go`). It is chosen over\n `store.Task.ID` because the trigger vocabulary is issue-level\n workintake label/kind, dispatched at discovery time (pollNATS, forge\n webhook receiver), where no task row exists yet (`playbook.go`\n comment; `playbook_test.go`: \"the value available at the\n discovery/dispatch point... NOT a store.Task.ID int64\"). Keying on the\n issue identity rather than the delivery source is the load-bearing\n property the existing contract already established for `TaskEnvelope`\n (`event-sources-and-reactions.md` question 4: a webhook-sourced\n envelope for the same issue produces the *same* idempotency key as a\n poll-discovered one; `internal/forge/webhook/receiver.go` --\n \"Idempotency is not this package's job\"); this resolution inherits that\n property rather than re-deciding it. Consequence, stated explicitly:\n same-issue re-discoveries (a re-label that re-triggers the same kind)\n are the *same event* by design -- the workintake trigger vocabulary is\n issue-granular, exactly as `TaskEnvelope`'s own key is. A future\n capture/binding-originated trigger type keys on `captured_events.id`\n instead; derivation is per-trigger-type, not one formula for all\n triggers (rule carried over unchanged from the draft).\n- `action_id` is the action's declared `id` -- J1's named ids, \"a\n required field on every action that later actions reference\", unique\n and stable-identifier-shaped, validated at load (the CEL resolution's\n J1 and its context table above in this document). For an action declaring no `id`\n (permitted by J1 while unreferenced), the key uses the action's 1-based\n position in the playbook: deterministic and stable for the\n single-action playbooks, and a later reorder is a file edit whose\n content hash changes `playbook_version` with it, so a positional\n fallback cannot silently collide with a prior definition.\n\n**No CEL in the key -- stated explicitly because the CEL resolution\nflagged this as gap 2's \"first real exercise\"** (\"CEL's read-only result\naccess makes gap 2's keying derivation readable\" -- the CEL resolution's\nclosing note). Readability\nis not the binding constraint; **determinism under redelivery is**, and\nCEL key expressions lose three ways. First, they are not load-checkable:\n`event` stays `dyn` by the resolved J4 (schema-by-example), so\na key expression over event fields cannot be type-checked at load,\nviolating the document's reject-at-load philosophy. Second, they can\nfail at dispatch (a missing field is a CEL error), and a key that cannot\nbe computed is an unfireable state -- neither \"skip\" nor \"fire\" is safe\nwithout losing or duplicating work. Third, the contract needs one\nderivation, and two authors writing different key expressions would\nfragment the ledger. The key is therefore structural by construction:\nevery component above is an immutable identity of the run's inputs, so\ntwo deliveries of the same event derive the same key with no evaluation.\nThe exercise's actual outcome is the derivation rule above; CEL remains\na *condition* surface (`when`), never a key surface. An action kind that\ngenuinely needs finer granularity than (action, event), such as \"per\ncomment, not per issue\", arrives with its own trigger type and its own\nevent identity, per the per-trigger-type rule.\n\n**Record-before-invoke -- one deliberate divergence from\n`binding_dispatches`, justified by the harm profile.** The binding\ndispatch loop enqueues the task first and records the ledger row after\n(`daemon.go`), accepting that racing cycles may both enqueue\n(\"the duplicate task remains queued... delete-on-races is its own can of\nworms\"); the selection query excludes already-dispatched captures\n(`bindings.go`), but the race window remains. That ordering is\ncorrect there because the duplicate artifact is an internal task row --\nrecoverable, low-harm. The playbook ledger's ordering is reversed on\npurpose: **the row is written and committed before the action's side\neffect is invoked** -- the `INSERT OR IGNORE` is the atomic gate, and\nthe lost-write loser skips without invoking. The cost is stated, not\nhidden: a crash between the row commit and the invoke loses that one\naction's side effect, repairable by hand -- the reverse direction of the\nsame window that would otherwise duplicate a posted comment or\nre-bill an image generation. For side effects whose duplication is\nexternally visible and not revocable, at-most-once is the required\ndirection; the binding system's at-least-once-tolerant ordering is not\nthe precedent for this ledger.\n\n**Gap-1 interplay, stated to keep the two mechanisms separate:** the\nledger answers one question per action -- has this action already fired\nfor this event? It does not answer whether the run continues, stops, or\nis re-attempted; that is gap 1's coordinator concern (a run-outcome\nrecord; gap 1 explicitly leaves \"what leave it alone for restart means\nfor a playbook run\" to whoever builds the coordinator). One consequence\nof record-before: an action that FAILED (gap 1's real-action-error case)\nhas a ledger row, so a later redelivery of the same event skips it --\nconsistent with gap 1's \"no automatic retry\" resolve (a corrective\naction is a *new* playbook/action, per gap 1's rollback paragraph). A\nfailed action is never re-attempted by the bus; if a run must be\nrecovered, an operator edits the playbook -- new version, new key --\nrather than fighting ledger state.\n\n**Relationship to `event-sources-and-reactions.md` -- extension, not\ncontradiction.** Keyed to that document's sections:\n\n1. Its decision 2 commits reactions to at-least-once delivery via\n `PublishUnique`'s \"without duplicate enqueue\" guarantee -- which, with\n the code facts above, is accurate for the *publish* path only:\n `PublishUnique` absorbs duplicate publishes of the same event (poll\n vs. webhook), which is what that decision was actually about. This\n document's own problem statement, taken literally, overstates it\n (\"keys absorb redelivery without duplicate side effects\") -- that\n overstatement is exactly what this gap exists to correct. The\n existing contract never claimed consumer redeliveries of\n side-effecting reactions were absorbed, so nothing it says contradicts\n a consumer-side ledger; this resolution pins down the boundary the\n sentence left implicit (publish-path dedup vs. consumer-path dedup)\n rather than changing either.\n2. Its decision 3 is untouched: the ledger is the run's own bookkeeping\n about its own actions, suppressing a re-fire and nothing else.\n3. Its decision 4 (no generic `Source` interface; `TaskEnvelope` is the\n typed contract, keyed on issue identity, not delivery source): the\n event half of the playbook key IS that same identity\n (`TaskEnvelope.IdempotencyKey()`), and the per-trigger-type rule\n means any future trigger type brings its own identity rather than a\n new generic one -- the playbook ledger introduces no second identity\n vocabulary and no untyped payload. The only new type in this\n resolution is the ledger's own value object, consumed through a\n narrow interface like `BindingDispatcher` -- the same shape the\n document's \"typed families\" rule (decision 1, plugin engine rule)\n already blesses.\n\n**Scope: this is the `channel`/`forge`/`module` decision, not a\n`workflow`-position change.** Workflow-kind actions keep their existing\nkeying claim unchanged -- the `TaskEnvelope` path that this document\nalready inherits for free (`daemon.go`, `PublishTask` publishes\nwith `task.IdempotencyKey()`), with its cross-poll/webhook property\nuntouched. The ledger gate applies to the side-effecting positions\n(channel, forge, module), which is why the closing statement below\nstill holds: the first slice (Module + loader + workflow-kind dispatch)\nwas buildable without this resolution, and channel/forge/multi-action\nplaybooks were not. A coordinator MAY implement one position-uniform\ngate as a simplification; what this resolution requires is only that\nthe side-effecting positions are gated.\n\n## Resolved questions\n\n1. **Data-flow/condition syntax.** Does this project want a small existing\n Go expression evaluator (there may be one already in the codebase worth\n checking, e.g. anything backing gate conditions), or a bespoke minimal\n grammar?\n\n **Resolved** (`archie-core-t2db.14`). CEL (`cel.dev/cel-go`, pinned) is\n the one mechanism for both an action's `when` condition and its `args`\n values. The trust boundary, evaluation context, load-time validation and\n the five judgment calls are in\n `docs/prds/playbook-expression-syntax.md`.\n\n2. **Playbook directory config field.** Where this lives in\n `internal/config` / `configuration.Document`, and whether it's one\n directory or a list (mirroring `SkillsDir`'s shape) -- follow existing\n config precedent, not invented fresh.\n\n **Resolved** (`archie-core-t2db.11`). The directory\n shape landed as `playbook_dirs` (a LIST of directories of\n `*.yaml`/`*.yml` binding files), loaded at startup as an additional\n input to the two single-file fields, which remain supported unchanged.\n A single `playbook_dir` would contradict\n supporting multiple independently-maintained playbook sources, which\n the Dedup section already assumes ('independently-sourced playbook\n directories collide'), so the field is a list. Cross-source collision\n (same key in two directories, or in a single-file field and a\n directory) is reported like an in-directory collision -- nothing is\n arbitrated by source precedence. This is what finally exercises the\n design doc's drop-and-report collision rule, since a single file\n cannot collide with itself. Repo-scoping (tying directories to\n `Config.Repos` owner/name) is explicitly deferred to a separate,\n later decision; org/tenant keying is out of scope (no auth/identity\n model).\n\n Decisions recorded from the label-vocabulary slice:\n - Arbitrary labels bind to registered workflow names via\n `LabelWorkflows` + `LoadLabelWorkflowsYAML`/`SetLabelWorkflows`\n (`internal/domain/workflow/routing.go`).\n - The closed `Kind`/NATS-subject set is untouched; the label map only\n extends binding authority for labels the kind layer does not own.\n - Collision rule: a label already owned by the\n kind set (bug/feature/bootstrap), an empty label, an empty workflow\n name, or a duplicate binding is a reported load failure -- dropped\n and logged, the error returned to the caller. Nothing is silently\n arbitrated by schema/version. This is the \"schema defines the\n accepted message; anything else is the caller's fault\" rule applied\n to the label layer.\n - Precedence in `Route()`: explicit `t.Workflow` → arbitrary-label\n binding → kind binding → triage → implement → default.\n3. **Trust boundary for Module Yaegi code.** The plugin engine rule's\n invariant 6 draws a line between operator-trusted in-process code and\n repository-supplied code that must run in a container.\n\n **Resolved** in `docs/prds/module-position.md`, \"Trust boundary\": a\n Module is operator-installed, in-process and daemon-privileged, the same\n tier as `PluginDir` and `SecretEngineDir`, never repository-supplied\n task code.\n4. **First implementation slice.** Recommend: Module position + the\n playbook loader + trigger-to-workflow dispatch only (subsuming the existing\n label routing) first, proving the schema-gen -\u003e Yaegi -\u003e playbook path\n end to end on the smallest useful case, before adding Channel/Forge\n action kinds or the linter/LSP.\n\n **Resolved** (`archie-core-t2db.13`, `.14`, `.15`). The Module position\n (registry + log kind, t2db.13), the CEL expression environment\n (t2db.14), and the playbook document + event coordinator with\n single-action workflow-kind dispatch (t2db.15) are shipped.\n\n **Resolved** (`archie-core-t2db.23`): the coordinator selects a real\n task's workflow. The wiring point is the daemon's definition pin\n (`Daemon.resolveWorkflowID`, consumed by `pinWorkflowFromCollection` in\n `internal/daemon`), not `pollNATS`/`publishTask`. Routing belongs to the\n pin because a workflow definition is a\n database row pinned before dispatch, which makes the pin the one place a\n task's workflow is chosen. Precedence there is explicit `Task.Workflow`\n (the waiting_human -\u003e approved requeue) → matching playbook →\n `workflow.ResolveWorkflowID`'s label binding → kind binding → triage →\n implement. A playbook naming an undefined workflow is a reported failure\n that parks the task rather than a silent fall-through to the binding's\n choice.\n\n `Store.Dispatch` returns the selected workflow *name* and the matching\n playbook's id and version, not a compiled `workflow.Workflow`: the\n production caller decides against the active definition collection, and\n the id and version are the first two components of the gap-2 idempotency\n key. The playbook package does not import `internal/domain/workflow`.\n\n A `when` condition at this point reads what the intake knows about the\n originating issue: `kind`, `labels`, `owner`, `repo`, `number`, `title`,\n `body`, `source`, built by `playbookInput` in `internal/daemon`.\n\n The trigger shape reuses the existing workintake kind/label vocabulary.\n The single-action hard boundary is superseded by\n `docs/prds/multi-action-playbooks.md` (two-shape rule); Module, Channel,\n and Forge positions still have no production dispatch path.\n## Standing constraint\n\nThis may be commercialized, and is to be discernible from other open-source\nevent-driven-automation tooling. No design decision follows from that yet;\nit bears on licensing or feature-gating questions about playbook authoring\nand the LSP when they arise.\n\n## Packages this touches (first slice)\n\n- New: a schema-generation package/convention (`go:generate` source +\n emitted contracts) for the Module position.\n- New: playbook loader + event coordinator (name/location TBD -- likely\n `internal/domain/eda` or similar, following the domain-layer rule: no\n infrastructure imports, no untyped hooks).\n- `internal/domain/workflow`: dispatch target for `workflow`-kind actions,\n unchanged in its own contracts.\n- `internal/domain/workintake`: `labelKinds`/routing becomes one playbook\n among others once the loader subsumes it; existing Go fallback stays for\n backward compatibility until playbooks fully replace it (mirrors the\n `dynamic-workflow-triage.md` precedent of a two-line fallback change, not\n a rip-and-replace).\n" }, { "url": "/dev/prds/embedded-nats/", @@ -238,7 +238,7 @@ "url": "/dev/prds/event-sources-and-reactions/", "title": "Event sources and reactions -- decision", "section": "PRDs", - "body": "# Event sources and reactions -- decision\n\n**Status:** Approved\n**Date:** 2026-08-22\n**Beads issue:** `archie-core-7d5u.1`, blocks `7d5u.2/.3/.4/.5`\n\nAnswers the four questions in `7d5u.1`'s description. Investigated against\nthe actual tree, not designed from scratch: three mechanisms already in this\ncodebase answer most of this, and the job here is picking the right one for\neach question and saying which packages change.\n\n## 1. In-process typed families, not out-of-process dispatch\n\nArchie has no out-of-process extension mechanism. The plugin engine\nrule (`ARCHITECTURE.md#plugin-engine-rule-strict`, mechanically checked by\n`internal/plugin/architecture_test.go`) keeps every capability family\nin-process behind a narrow typed registrar; NATS RPC exists for\ndaemon\u003c-\u003eagentexec, a different actor with a different trust boundary (task\nexecution, not third-party extension code), not a precedent for generic\nplugin dispatch. Neither epic's near-term payoff -- react to a labelled\nissue, bind a captured webhook to a workflow -- needs extension code in\nanother language or process. Building out-of-process dispatch now would be\ndesigning for a requirement neither epic states.\n\n`docs/architecture/plugins-and-extensions.md` does **not** need amending.\nThe line that looked like a blanket ban on hooks (`archie-core-7d5u`'s notes\nrecord the correction) is scoped to Workflow semantics specifically, and\nnothing decided here touches Workflow's ownership of its own contracts.\n\n## 2. Reaction delivery: `internal/eventbus`'s contract, a NEW stream\n\n`internal/eventbus.Publisher`/`Consumer` already has everything a reaction\nneeds: `PublishUnique(subject, idempotencyKey, payload)` for at-least-once\nwithout duplicate enqueue, `Ack`/`Nak` with \"a handler returning an error\ncauses redelivery.\" Do not design new delivery semantics; use this contract.\n\n**Do not reuse the existing `nats.Client` as-is.** `Config.StreamName` and\n`Config.Subjects` are already composition-supplied (\"the bus must not know\nwhich subjects belong to which domain,\" `config.go`), but the\nretention policy is not parameterized: `New` hardcodes\n`Retention: jetstream.WorkQueuePolicy` (`client.go`), the competing-consumer\npolicy correct for `ARCHIE_TASKS` task distribution -- one message, claimed by\nexactly one consumer -- and wrong for reactions. Two independent reactions\nboth wanting to see the same event under `WorkQueuePolicy` hit exactly the\ntrap `CLAUDE.md` already documents: overlapping filter subjects on a\nwork-queue stream mean the second consumer silently gets nothing. Reactions\nneed fan-out, not claim-once, so the client needs retention made a `Config`\nfield (defaulting to `WorkQueuePolicy` so `ARCHIE_TASKS` is\nunaffected), and a second stream (`ARCHIE_REACTIONS` or similar) configured\nunder a fan-out-capable policy, alongside the existing task-distribution\nstream -- not instead of it.\n\nThis is where `7d5u.2`'s embedded-vs-external axis attaches: whichever NATS\ninstance backs `ARCHIE_TASKS` also backs the reaction stream. NATS is not\noptional as a technology -- the choice is only\nwhether it runs as an external server or embedded in-process\n(`github.com/nats-io/nats-server/v2` is already a direct, test-proven\ndependency, just never wired into production startup). `7d5u.2` wires that;\nthis document does not re-decide it.\n\n## 3. Reactions in these two epics are producer-only\n\nNo veto, no mutation of in-flight work. Two existing shapes cover everything\neither epic actually needs, and neither is a hook that can block something\nelse:\n\n- **Cheap wake, no payload.** `internal/domain/curator/wake.go`'s\n `WakeOnPrimaryInput` subscribes to the lossy `events.Bus` and nudges a\n runtime to check sooner; a dropped event only delays a check-in, never\n loses work, because the trigger decision is a deterministic state read at\n pass time, not the event payload. Model any \"check something sooner\"\n reaction on this, unchanged.\n- **Produce new work, real payload.** `daemon.go`'s existing `pollNATS` -\u003e\n `publishTask` path is already exactly a reaction that observes forge state\n and produces a `workintake.TaskEnvelope`. A webhook source is a second\n producer into the same path, not a new capability -- see question 4.\n\nA reaction that can veto or mutate an *in-flight* task (block a PR from\nopening, alter a running stage) is a different, harder problem with its own\nordering and failure semantics, and belongs to `archie-core-h019` (adversarial\nself-review) if and when that epic needs one -- not here. Do not build a\ngeneral block-capable hook mechanism for either epic; nothing in their stated\nscope requires it, and it would duplicate whatever h019 eventually needs.\n\n## 4. No single generic intake interface; each domain keeps its own\n\nA `Source` interface generic enough to cover a forge issue, a chat message,\nand a cron tick would need an `any` payload -- the untyped-hook shape the\nplugin engine rule exists to prevent, just moved into a new package instead\nof the old one.\n\n**What's genuinely shared, and belongs in a small cross-cutting package**\n(imports nothing per the cross-cutting rule): the mechanics from\n`archie-core-t2db.5` that apply to any inbound HTTP source regardless of what\nconsumes it -- HMAC verification (already a standalone function,\n`internal/channels/webhook/webhook.go`'s `validateHMAC`), per-source rate\nlimiting, capture/audit, and the draft/pending_approval/armed gate. This\npackage owns none of forge, chat, or workintake's vocabulary.\n\n**What stays domain-owned, per existing convention:**\n- Forge-issue intake needs no new type. `internal/domain/workintake.TaskEnvelope`\n already is the typed contract -- `Subject()`, `IdempotencyKey()` (keyed on\n `owner/repo/number`, not on delivery source), `Encode()`. A webhook-sourced\n envelope for the same issue produces the *same* idempotency key as a\n poll-discovered one, so `PublishUnique` dedups the two paths for free. This\n is `7d5u.5`'s answer, not a new mechanism: reuse `TaskEnvelope` and\n `publishTask`, don't invent a second envelope shape.\n- Chat-webhook intake stays exactly what it is:\n `internal/channels/webhook`'s `RouteConfig` routes into `gateway.Router`,\n documented as chat-scoped (\"External services push messages into\n archie-core's gateway for LLM processing\"). Do not generalize this package\n into a multi-domain receiver -- it already does one thing per \"a package\n owns its own format end to end.\"\n- A forge webhook receiver (`7d5u.4`) is new: it builds on the shared\n mechanics above, decodes a GitHub payload into the *same*\n `workintake.TaskEnvelope`, and calls the *same* `publishTask` the poller\n calls. It does not get its own dispatch predicate -- `7d5u.4`'s bead\n already requires reusing the poller's label/assignee predicate rather than\n reimplementing it, and this decision extends that to the envelope and\n publish path too.\n\n## Packages this touches\n\n- New: a cross-cutting intake-mechanics package (HMAC, rate limit, capture,\n approval state machine) -- name and exact location decided in `t2db.1`/`t2db.5`.\n- `internal/infrastructure/eventbus/nats`: a second stream/consumer\n configuration for reactions, distinct retention policy from `ARCHIE_TASKS`.\n- `internal/domain/workintake`: unchanged in shape; `TaskEnvelope` is reused,\n not extended.\n- `internal/daemon`: gains a webhook-sourced call path into the existing\n `publishTask`, alongside the existing poll-sourced one.\n- `internal/channels/webhook`: unchanged. Not the forge receiver.\n" + "body": "# Event sources and reactions -- decision\n\n**Status:** Approved\n**Date:** 2026-08-22\n**Beads issue:** `archie-core-7d5u.1`, blocks `7d5u.2/.3/.4/.5`\n\nAnswers the four questions in `7d5u.1`'s description. Investigated against\nthe actual tree, not designed from scratch: three mechanisms already in this\ncodebase answer most of this, and the job here is picking the right one for\neach question and saying which packages change.\n\n## 1. In-process typed families, not out-of-process dispatch\n\nArchie has no out-of-process extension mechanism. The plugin engine\nrule (`docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`,\nmechanically checked by\n`internal/plugin/architecture_test.go`) keeps every capability family\nin-process behind a narrow typed registrar; NATS RPC exists for\ndaemon\u003c-\u003eagentexec, a different actor with a different trust boundary (task\nexecution, not third-party extension code), not a precedent for generic\nplugin dispatch. Neither epic's near-term payoff -- react to a labelled\nissue, bind a captured webhook to a workflow -- needs extension code in\nanother language or process. Building out-of-process dispatch now would be\ndesigning for a requirement neither epic states.\n\n`docs/architecture/plugins-and-extensions.md` does **not** need amending.\nThe line that looked like a blanket ban on hooks (`archie-core-7d5u`'s notes\nrecord the correction) is scoped to Workflow semantics specifically, and\nnothing decided here touches Workflow's ownership of its own contracts.\n\n## 2. Reaction delivery: `internal/eventbus`'s contract, a NEW stream\n\n`internal/eventbus.Publisher`/`Consumer` already has everything a reaction\nneeds: `PublishUnique(subject, idempotencyKey, payload)` for at-least-once\nwithout duplicate enqueue, `Ack`/`Nak` with \"a handler returning an error\ncauses redelivery.\" Do not design new delivery semantics; use this contract.\n\n**Do not reuse the existing `nats.Client` as-is.** `Config.StreamName` and\n`Config.Subjects` are already composition-supplied (\"the bus must not know\nwhich subjects belong to which domain,\" `config.go`), but the\nretention policy is not parameterized: `New` hardcodes\n`Retention: jetstream.WorkQueuePolicy` (`client.go`), the competing-consumer\npolicy correct for `ARCHIE_TASKS` task distribution -- one message, claimed by\nexactly one consumer -- and wrong for reactions. Two independent reactions\nboth wanting to see the same event under `WorkQueuePolicy` hit exactly the\ntrap `CLAUDE.md` already documents: overlapping filter subjects on a\nwork-queue stream mean the second consumer silently gets nothing. Reactions\nneed fan-out, not claim-once, so the client needs retention made a `Config`\nfield (defaulting to `WorkQueuePolicy` so `ARCHIE_TASKS` is\nunaffected), and a second stream (`ARCHIE_REACTIONS` or similar) configured\nunder a fan-out-capable policy, alongside the existing task-distribution\nstream -- not instead of it.\n\nThis is where `7d5u.2`'s embedded-vs-external axis attaches: whichever NATS\ninstance backs `ARCHIE_TASKS` also backs the reaction stream. NATS is not\noptional as a technology -- the choice is only\nwhether it runs as an external server or embedded in-process\n(`github.com/nats-io/nats-server/v2` is already a direct, test-proven\ndependency, just never wired into production startup). `7d5u.2` wires that;\nthis document does not re-decide it.\n\n## 3. Reactions in these two epics are producer-only\n\nNo veto, no mutation of in-flight work. Two existing shapes cover everything\neither epic actually needs, and neither is a hook that can block something\nelse:\n\n- **Cheap wake, no payload.** `internal/domain/curator/wake.go`'s\n `WakeOnPrimaryInput` subscribes to the lossy `events.Bus` and nudges a\n runtime to check sooner; a dropped event only delays a check-in, never\n loses work, because the trigger decision is a deterministic state read at\n pass time, not the event payload. Model any \"check something sooner\"\n reaction on this, unchanged.\n- **Produce new work, real payload.** `daemon.go`'s existing `pollNATS` -\u003e\n `publishTask` path is already exactly a reaction that observes forge state\n and produces a `workintake.TaskEnvelope`. A webhook source is a second\n producer into the same path, not a new capability -- see question 4.\n\nA reaction that can veto or mutate an *in-flight* task (block a PR from\nopening, alter a running stage) is a different, harder problem with its own\nordering and failure semantics, and belongs to `archie-core-h019` (adversarial\nself-review) if and when that epic needs one -- not here. Do not build a\ngeneral block-capable hook mechanism for either epic; nothing in their stated\nscope requires it, and it would duplicate whatever h019 eventually needs.\n\n## 4. No single generic intake interface; each domain keeps its own\n\nA `Source` interface generic enough to cover a forge issue, a chat message,\nand a cron tick would need an `any` payload -- the untyped-hook shape the\nplugin engine rule exists to prevent, just moved into a new package instead\nof the old one.\n\n**What's genuinely shared, and belongs in a small cross-cutting package**\n(imports nothing per the cross-cutting rule): the mechanics from\n`archie-core-t2db.5` that apply to any inbound HTTP source regardless of what\nconsumes it -- HMAC verification (already a standalone function,\n`internal/channels/webhook/webhook.go`'s `validateHMAC`), per-source rate\nlimiting, capture/audit, and the draft/pending_approval/armed gate. This\npackage owns none of forge, chat, or workintake's vocabulary.\n\n**What stays domain-owned, per existing convention:**\n- Forge-issue intake needs no new type. `internal/domain/workintake.TaskEnvelope`\n already is the typed contract -- `Subject()`, `IdempotencyKey()` (keyed on\n `owner/repo/number`, not on delivery source), `Encode()`. A webhook-sourced\n envelope for the same issue produces the *same* idempotency key as a\n poll-discovered one, so `PublishUnique` dedups the two paths for free. This\n is `7d5u.5`'s answer, not a new mechanism: reuse `TaskEnvelope` and\n `publishTask`, don't invent a second envelope shape.\n- Chat-webhook intake stays exactly what it is:\n `internal/channels/webhook`'s `RouteConfig` routes into `gateway.Router`,\n documented as chat-scoped (\"External services push messages into\n archie-core's gateway for LLM processing\"). Do not generalize this package\n into a multi-domain receiver -- it already does one thing per \"a package\n owns its own format end to end.\"\n- A forge webhook receiver (`7d5u.4`) is new: it builds on the shared\n mechanics above, decodes a GitHub payload into the *same*\n `workintake.TaskEnvelope`, and calls the *same* `publishTask` the poller\n calls. It does not get its own dispatch predicate -- `7d5u.4`'s bead\n already requires reusing the poller's label/assignee predicate rather than\n reimplementing it, and this decision extends that to the envelope and\n publish path too.\n\n## Packages this touches\n\n- New: a cross-cutting intake-mechanics package (HMAC, rate limit, capture,\n approval state machine) -- name and exact location decided in `t2db.1`/`t2db.5`.\n- `internal/infrastructure/eventbus/nats`: a second stream/consumer\n configuration for reactions, distinct retention policy from `ARCHIE_TASKS`.\n- `internal/domain/workintake`: unchanged in shape; `TaskEnvelope` is reused,\n not extended.\n- `internal/daemon`: gains a webhook-sourced call path into the existing\n `publishTask`, alongside the existing poll-sourced one.\n- `internal/channels/webhook`: unchanged. Not the forge receiver.\n" }, { "url": "/dev/prds/extensions-center/", @@ -256,7 +256,7 @@ "url": "/dev/prds/image-capability-contract/", "title": "Image capability contract and configuration", "section": "PRDs", - "body": "# Image capability contract and configuration\n\n**Status:** Approved\n\nEpic: archie-core-1786748942120-1-6636e629 / GitHub #496 (\"Image generation\ncapability: hosted API baseline plus local GPU workflows\").\n\nThe contract and configuration (#497) shipped first and is settled — see\n\"Package: `internal/domain/image`\" below. This document now covers the full\nfeature set: hosted provider (#498), local GPU provider (#499), `/image`\nrouting and clarification (#500), delivery through channels and web (#501).\nDocumentation (#502) and the cross-cutting test matrix (#503) are execution\nwork against the design below, not open design questions, so they are not\ngiven their own sections — each preceding section states its own test\nobligations inline.\n\n## Problem\n\nArchie needs a provider-neutral boundary for image generation/editing so a\nhosted API (gpt-image-2-class) and a future local GPU backend can both\nimplement it without either one leaking into the domain contract. Nothing\nin the repo defines this (`grep -ri image` over `internal/domain` and\n`internal/config` returns nothing image-specific).\n\n## Design\n\nFollows the plugin engine family shape already established by\n`internal/domain/curator` and `internal/domain/memory` (ARCHITECTURE.md\n\"plugin engine rule strict\"): a typed contract with real domain operations,\nan owning `Registry` (register/get/names, no start/stop lifecycle — see\nbelow), and narrow host access via a `Registrar`.\n\n### Package: `internal/domain/image`\n\n**Why no `Lifecycle` (`Start/Health/Stop`)**: curator and memory engines are\nlong-running — they hold a background cadence or a persistent connection\nworth health-checking. An image provider is call-scoped: it either serves a\n`Generate`/`Edit` request or it does not. Health here means \"is this\nprovider usable at call time\", which is exactly what `Generate`/`Edit`\nreturning a typed `ErrUnavailable` already communicates — a separate\n`Health()` poll would just duplicate that. `Registry` still owns\nregistration and lookup, so provider selection is testable with fakes per\nthe acceptance criteria, but there is no `Start`/`Stop` to orchestrate.\n\n```go\n// Capability declares what one provider supports, discoverable before a\n// request is built so a caller can reject an unsupported combination (e.g.\n// edit-with-mask on a generate-only provider) instead of sending it and\n// parsing a provider-specific error back out.\ntype Capability struct {\n Generate bool\n Edit bool // edit requires at least one input image\n Mask bool // edit supports a mask restricting the edited region\n MaxInputs int // 0 = Edit unsupported\n Sizes []string // provider-declared output sizes, e.g. \"1024x1024\"\n}\n\n// Provider is the typed contract every image backend implements — hosted\n// (gpt-image-2-class) or local (GPU recipe). No Lifecycle: a provider is\n// call-scoped, not long-running (see design doc rationale).\ntype Provider interface {\n Name() string\n // Class distinguishes hosted (spends money/quota per call) from local\n // (spends host GPU/time). Selection logic and config validation key off\n // this, never off Name, so a new provider cannot silently slip through\n // the hosted-cost gate under an unrecognized name.\n Class() ProviderClass\n Capability() Capability\n Generate(ctx context.Context, req GenerateRequest) (Result, error)\n Edit(ctx context.Context, req EditRequest) (Result, error)\n}\n\ntype ProviderClass string\n\nconst (\n ClassHosted ProviderClass = \"hosted\"\n ClassLocal ProviderClass = \"local\"\n)\n```\n\nRequest/result types (provider-neutral, no OpenAI types):\n\n```go\ntype GenerateRequest struct {\n Prompt string\n Size string // \"\" = provider default\n N int // number of images; 0 = 1\n Options map[string]string // provider-specific overflow (e.g. \"quality\"),\n // deliberately untyped so the contract does not\n // grow a field per provider\n}\n\ntype EditRequest struct {\n Prompt string\n Inputs []ImageData // 1..Capability.MaxInputs\n Mask *ImageData // nil unless Capability.Mask\n Size string\n N int\n Options map[string]string\n}\n\n// ImageData carries image bytes in-process, never a path or URL: the\n// domain layer must not assume a filesystem or a fetchable location (mirrors\n// why curator's Registrar never receives *daemon.Daemon — no ambient host\n// access smuggled in through a \"just a string\" field).\ntype ImageData struct {\n Bytes []byte\n MIMEType string\n}\n\ntype Result struct {\n Images []ImageData\n Provider string\n Model string\n CreatedAt time.Time\n}\n```\n\n### Error taxonomy\n\n```go\nvar (\n ErrUnavailable = errors.New(\"image provider unavailable\") // missing credential, backend down\n ErrUnsupported = errors.New(\"image capability unsupported by provider\") // e.g. Edit on generate-only\n ErrInputTooLarge = errors.New(\"image input exceeds configured limit\")\n ErrTimeout = errors.New(\"image request timed out\")\n ErrCanceled = errors.New(\"image request canceled\")\n)\n```\n\nEvery provider wraps these with `fmt.Errorf(\"%w: %s\", ErrX, detail)` so\ncallers can `errors.Is` regardless of backend. `ErrUnavailable` is the\nexplicit instruction from the epic: \"a missing credential or unavailable\nbackend must make the capability unavailable with a useful user-facing\nerror, not prevent Archie from starting or break text chat\" — Register\nnever rejects the daemon's boot, only marks that one provider absent from\n`Registry.Names()`.\n\n### Registry\n\n```go\ntype Registry struct { /* map[string]Provider, mutex, sorted Names() */ }\n\nfunc NewRegistry() *Registry\nfunc (r *Registry) Register(p Provider) error // ErrDuplicate on name clash\nfunc (r *Registry) Get(name string) (Provider, bool)\nfunc (r *Registry) Names() []string\nfunc (r *Registry) ByClass(class ProviderClass) []string\n```\n\nNo `Registrar`/host-injection type: unlike curator (needs `LLM`, `Skills`,\n`Conversations`) and memory (needs `Clock`, `Events`), an image provider's\ndependencies (HTTP client, API key, local backend handle) are entirely its\nown construction-time concern in `internal/infrastructure/image/*`, not\ndaemon services it needs injected. Introducing an empty `Registrar` now\nwould be exactly the \"field nobody sets\" pattern to avoid.\n\n### Configuration\n\nFollows `MinimaxConfig`'s hosted-cost-guard convention exactly: absent\nconfig or absent explicit `enabled = true` per provider means that\nprovider does not register, not \"registers but errors on first call\".\n\n```go\n// ImageConfig configures the image generation capability.\ntype ImageConfig struct {\n // Default selects which registered provider name /image uses when the\n // request/session has not already picked one. Empty means /image must\n // always ask (the epic's stated default UX) — an empty Default is valid\n // and is not a misconfiguration.\n Default string `toml:\"default\"`\n Hosted map[string]ImageHostedProvider `toml:\"hosted\"` // keyed by provider name\n Local map[string]ImageLocalProvider `toml:\"local\"` // keyed by recipe name\n}\n\n// ImageHostedProvider configures one hosted (paid) provider. Mirrors\n// MinimaxConfig: Enabled defaults false, spending real credits requires an\n// explicit opt-in, never an inferred one from the presence of a key alone.\ntype ImageHostedProvider struct {\n Enabled bool `toml:\"enabled\"`\n Class string `toml:\"class\"` // \"openai\" today; provider-neutral field name\n APIKeyEnv string `toml:\"api_key_env\"`\n APIKey secret.SecretRef `toml:\"api_key\" json:\"-\"`\n BaseURL string `toml:\"base_url\"`\n}\n\n// ImageLocalProvider configures one local GPU recipe. Enabled defaults\n// false too: a local backend can still saturate the host GPU/CPU, which is\n// its own kind of unwanted cost, so it gets the same explicit gate as a\n// hosted provider rather than an on-by-default WebFetch-style treatment.\ntype ImageLocalProvider struct {\n Enabled bool `toml:\"enabled\"`\n Backend string `toml:\"backend\"` // recipe identifier, defined by the local-backend bead\n}\n```\n\nValidation (`internal/infrastructure/configuration/validate.go`, alongside\n`validateMemory`):\n\n- `Default`, if set, must name a provider present (enabled) in `Hosted` or\n `Local` — rejects a config that silently falls back to \"always ask\" when\n the operator thought they'd pinned one.\n- A `Hosted` entry with `Enabled = true` must have `Class` non-empty and\n either `APIKeyEnv` or a resolvable `APIKey` — the same shape check\n `Provider` (chat) already gets, not a new invention.\n- No cross-field surprise: an *absent* `[image]` section is valid and\n produces zero registered providers, matching the epic's non-goal\n (\"silently selecting a paid hosted provider\" must never happen by\n omission either).\n\n### Capability discovery\n\n`Registry.Names()` / `ByClass()` plus `Provider.Capability()` give a caller\n(the future `/image` routing bead) everything needed to ask \"what can I\noffer this user\" without a type switch on provider identity — this is the\nwhole point of the contract being provider-neutral.\n\n### Timeouts, size limits, cancellation\n\n- Every `Provider.Generate`/`Edit` call takes the caller's `ctx`; a\n provider that ignores cancellation is a provider bug, not a contract gap\n — mirrors every other Lifecycle-less call in the codebase (chat's\n `runtime.Chat`).\n- `ImageData` size limits are enforced by the *caller* (the future routing\n bead) before constructing a request, using a configured max — not by this\n contract, which has no config surface for \"current caller's limit\". This\n package only defines `ErrInputTooLarge` so a provider that discovers an\n oversized input itself (rather than trusting the caller) has a typed way\n to say so.\n\n## Testing\n\n- `internal/domain/image/{contract,registry}_test.go` mirror memory's test\n suite shape: registration (duplicate, nil provider), `Names`/`ByClass`\n filtering, `Get` miss, table-driven `Capability` checks via a fake\n provider.\n- `internal/infrastructure/configuration/validate_test.go` gains cases for\n `ImageConfig`: absent section, valid hosted+local, missing `Class`,\n missing credential, `Default` naming an unregistered/disabled provider.\n- No provider implementation exists yet, so no live-call tests — those\n belong to the hosted/local provider beads.\n\n## Hosted provider (#498)\n\n`github.com/samcharles93/ai-sdk` (already a repo dependency) has its own\nprovider-agnostic image facility: `image.Provider` (`Name`,\n`GenerateImage(ctx, GenerateImageRequest) (GenerateImageResponse, error)`),\n`image.Client`, and `registry.Registry.RegisterImage`/`.Image(name)` — the\nsame shape chat model registration already uses. Three backends implement it\n(azure, togetherai, xai); OpenAI does not.\n\n`internal/infrastructure/image.\u003cname\u003eProvider` wraps an `ai-sdk/image.Provider`\nbehind this package's `Provider` interface, the same way `curatorLLMRunner`\n(`internal/app/archied/main.go`) adapts `runtime.Runtime` for chat rather\nthan the domain layer talking to ai-sdk directly.\n\n**Decision: generation goes through ai-sdk, edit does not.** ai-sdk's\n`image.GenerateImageRequest` is text-to-image only — no input-image or mask\nfields — so there is nothing to adapt `EditRequest`/`Capability.Edit` onto.\nBlocking the entire hosted provider on an upstream ai-sdk change is not\nthis epic's call to make on someone else's project's timeline. Instead:\n\n- `Generate` wraps `ai-sdk/image.Client.GenerateImage` against whichever\n backend `ImageHostedProvider.Class` names (azure/togetherai/xai).\n- `Edit` is a direct HTTP client hitting the OpenAI-compatible\n `/images/edits` multipart endpoint, built the same way `minimax.Client`\n (`internal/tools/minimax/minimax.go`) is a direct client for an API\n ai-sdk doesn't cover — not a second abstraction layer, one `http.Client`\n call with request/response mapping into this contract's `EditRequest`/\n `Result`.\n- A provider that only wraps ai-sdk (no edit client configured) declares\n `Capability{Generate: true, Edit: false}` truthfully rather than\n returning `ErrUnsupported` at call time for something `Capability`\n already said it couldn't do.\n- If ai-sdk gains edit support upstream later, the direct client is deleted\n and `Edit` moves onto it — recorded here so that's a known, not a\n surprise, follow-up.\n\n**Provider error mapping** (every ai-sdk/HTTP error translated at the\nadapter boundary, never leaked as-is):\n\n| ai-sdk / HTTP condition | contract error |\n|---|---|\n| `image.ErrNoProvider`, `image.ErrAuthFailed`, missing/invalid credential, backend unreachable | `ErrUnavailable` |\n| `image.ErrEditNotSupported`, request needs a `Capability` the provider didn't declare | `ErrUnsupported` |\n| input image over configured byte limit (checked before the call, per \"Timeouts, size limits, cancellation\" above) | `ErrInputTooLarge` |\n| `ctx.Err() == context.DeadlineExceeded` | `ErrTimeout` |\n| `ctx.Err() == context.Canceled` | `ErrCanceled` |\n| `image.ErrRateLimited`, `image.ErrContentFiltered`, any other 4xx/5xx | wrapped verbatim under `ErrUnavailable` with the provider's own message as detail — a rate limit or content filter is \"unavailable for this request\", not a new sentinel the caller needs to branch on differently |\n\n**Testing**: `httptest.Server` fixtures for the edit client (request\nmultipart shape, response decoding, 4xx/5xx mapping); a fake `ai-sdk/image.Provider`\nfor the generate path so the adapter's translation is tested without a live\nbackend, matching #498's \"no live API key needed for tests\" criterion.\n\n## Local GPU provider (#499)\n\n**Decision: ComfyUI is the baseline recipe**, not a name this doc picked\narbitrarily — it is the de facto scriptable HTTP API for Stable-Diffusion-class\nlocal generation, already shaped like the async submit/poll pattern\n`minimax.Client.GenerateAndWait` established in this codebase (submit a\nworkflow, poll `/history/{id}` until the output node has files, download).\nNo other local backend is referenced anywhere in this repo or its docs, so\nthis decision has no competing prior art to reconcile — it is a fresh\nchoice, recorded here so the implementing bead does not re-litigate it.\n\n```go\n// internal/infrastructure/image/comfyui\ntype Config struct {\n Enabled bool\n BaseURL string // e.g. http://127.0.0.1:8188\n Workflow string // path to a saved ComfyUI workflow JSON template\n // (prompt/size/seed substituted at call time)\n Timeout time.Duration\n PollInterval time.Duration\n MaxWait time.Duration\n}\n```\n\n- `Generate`/`Edit` load `Workflow`, substitute the templated fields (prompt,\n size, seed, and for `Edit` the input image as a base64 node input — ComfyUI\n supports img2img/inpaint workflows natively, so `Capability.Edit`/`Mask`\n can both be true for a correctly authored workflow, unlike the hosted\n provider), POST to `/prompt`, then poll `/history/{prompt_id}` the same\n shape MiniMax's poll loop already proves out in this codebase.\n- `Capability.Sizes` is empty (ComfyUI workflows declare their own\n resolution in the template, not a fixed enum) — the contract already\n allows an empty `Sizes` to mean \"provider does not constrain size\".\n- Backend-down (`BaseURL` unreachable) and workflow-missing are both\n `ErrUnavailable`, not a startup failure — same \"degrade the capability,\n never break text chat\" rule the contract already states.\n- Per #499's own acceptance criteria, at least one recipe must be exercised\n end-to-end on real hardware or a reproducible harness, and no model\n weights are ever committed or fetched by Archie itself — the operator\n points `Workflow`/`BaseURL` at their own running ComfyUI instance and\n already-downloaded checkpoint, exactly like `[providers.ollama]` points at\n an already-running Ollama rather than Archie managing the model.\n\n**Testing**: a fake HTTP server standing in for ComfyUI's `/prompt` and\n`/history` endpoints (submit, poll, malformed history, timeout, job\nnot-found); a real-hardware smoke test documented as a manual `task`\ntarget per #503, not part of `task check` — the same reasoning `task vuln`\nand `task deadcode` are excluded from the gate.\n\n## `/image` routing and clarification flow (#500)\n\n**Call site: `internal/gateway/gateway.go`'s `Router`**, following the\n`/model` shape exactly — `/model`, `/spawn`, `/approve` etc. are already\n\"commands handled without the LLM\" (`Router.handleModel`, dispatched via\n`isLocalCommand`/`parseCmd` against `messaging.LocalCommandSpecs()`).\n`/image` is a new entry in `internal/domain/messaging/commands.go`'s\n`localCommandSpecs`, and a new `Router.handleImage(ctx, arg)` beside\n`handleModel`, gated on a new `Router.Images *image.Registry` field (nil =\n\"/image not configured\", the same pattern `Models`/`Tasks`/`Controller`\nalready use — not a new gating convention).\n\n**Decision: mechanism selection is per-request, not session-persisted.**\nThe epic allows \"a remembered preference only where existing session/config\nconventions support it\" — and checking `Router`'s actual fields\n(`ModelPersist`, `Identity`, the rest) shows there is no generic per-session\nkey/value store the neutral command path can lean on; `/model`'s own\n`--session`/`--global` scoping lives entirely in\n`internal/channels/telegram/model.go`, a Telegram-specific enrichment, not\nsomething `Router` provides generically. Inventing a new generic\nsession-preference store just for `/image` would be exactly the kind of\nscaffolding CLAUDE.md's Architectural Simplicity rule warns against for one\ncaller. So: every `/image` invocation with no explicit mechanism produces\nthe clarification prompt; an explicit `--hosted`/`--local` flag (mirroring\n`parseModelRest`'s `--provider` flag shape) skips it for that one call.\nRevisit if a second command wants session-scoped preference too — that's\nthe point at which shared infra earns its cost.\n\n**Flow**, matching `handleModel`'s reply-string shape (no separate state\nmachine object, since the whole exchange is \"ask once, use the answer\ninline in the next message\" — the model-selector's own worked example):\n\n1. `/image \u003cprompt\u003e` with no `--hosted`/`--local` → reply lists the\n registered classes (from `Registry.ByClass`) and asks the user to repeat\n the command with `--hosted` or `--local`. No provider is called.\n2. `/image --hosted \u003cprompt\u003e` or `--local` → resolves to\n `Registry.ByClass(class)`, picks `ImageConfig.Default` if it names a\n provider of that class, else the sole registered provider of that class,\n else (more than one, no default) asks which named provider.\n3. Edit requests (`/image --edit \u003cprompt\u003e` with an attached image on the\n inbound message) validate against `Capability.Edit`/`MaxInputs`/`Mask`\n before calling — an edit request to a generate-only provider is a\n clarification reply (\"this provider doesn't support editing\"), not a\n provider call that returns `ErrUnsupported`.\n4. A successful call returns a `Result`; delivery is the next section's\n concern, not this one's — `handleImage` returns the same\n `(string, error)` shape `handleModel` does for the text portion, and a\n separate mechanism (below) carries the actual image bytes.\n\n**Testing**: channel-neutral state-machine tests for `handleImage` against\na fake `*image.Registry` (mirrors `#500`'s own acceptance criterion); Telegram\nadapter tests only for wiring, not for re-testing the state machine.\n\n## Delivery through channels and web (#501)\n\nThis is the smallest section because most of the delivery path **already\nexists and is already proven** — by `generate_video`\n(`internal/tools/minimax`), which ships. Do not rebuild it.\n\nThe existing pipeline: a result carrying media returns\n`tools.MultimodalResult{IsMultimodal: true, URLs: []tools.MediaRef{{Type: \"image\", ...}}}`\n→ `multimodalMediaRefs` (`internal/app/archied/telegram_setup.go`) decodes\nit → `TurnStream.Media(gateway.MediaEvent)` → each channel's own\n`Media`/`MediaAttachment` handling. `MediaRef.Type` already documents\n`\"image\"` as a valid value alongside `\"video\"`; nothing in the pipeline is\nvideo-specific.\n\n**Per-channel state, checked directly rather than assumed:**\n\n| channel | image delivery | evidence |\n|---|---|---|\n| Telegram | **already works** | `internal/channels/telegram/media.go`'s `telegramMediaSender.dispatch` already branches on `att.Type` and calls `bot.SendPhoto` for `\"image\"` (`SendVideo` for `\"video\"` is the same function) |\n| Dashboard (webui) | **tracked gap, not new** | `internal/webui/api_chat.go`'s `chatStreamSink.Media` has no inline rendering path yet and degrades to a link/error string — already tracked as archie-core-1786748942243-6-f109697e. This epic does not need to fix it, but #501's acceptance criterion (\"unsupported paths report a clear limitation\") is already met by the existing degrade-to-text behavior; closing the dashboard gap is that bead's job, not this one's |\n| Email | **no attachment sender exists** | `grep MediaAttachment internal/channels/email` returns nothing. Out of scope for this epic per #496's non-goals (no channel-abstraction rework) — email delivery of a generated image degrades the same way the dashboard does which is acceptable under #501's \"or unsupported paths report a clear limitation\" clause |\n| Webhook | not applicable | webhook is an inbound intake channel (`internal/channels/webhook`), not an outbound chat reply target — no `/image` command can originate there in the first place |\n\n**The only new work `internal/tools/minimax` didn't already need**: the\nautonomous-tool call site. `/image` (previous section) calls a provider\ndirectly and does not go through the model's tool-calling loop at all.\nSeparately, exposing `generate_image`/`edit_image` as an ordinary tool\n(so an agent task can generate a diagram or illustration mid-task, not only\na human typing `/image`) is a thin wrapper — `internal/tools/image/tool.go`,\nsame shape as `internal/tools/minimax/tool.go`, calling\n`image.Registry.Get(name).Generate/Edit` and returning the identical\n`MultimodalResult` shape. Wired in `bootstrap.go` next to\n`registerMinimaxTool`, gated the same way (registered only when a provider\nresolved a working credential/backend, never advertised broken). This\nreuses the entire delivery pipeline above with zero new plumbing — the tool\ncall site is what's new, not the delivery.\n\n**Testing**: `multimodalMediaRefs` already has coverage for the shape;\nadd a table case for `Type: \"image\"` if one doesn't already exist. The tool\nwrapper gets the same fake-provider test shape as the routing flow above.\nNo new delivery-path tests are needed — that pipeline's tests belong to\nwhichever change originally proved it out.\n\n## Call site inventory\n\nEvery file a full implementation touches, so no bead re-derives this by\ngrep:\n\n| concern | file | change |\n|---|---|---|\n| domain contract | `internal/domain/image/{contract,registry}.go` | done (#497) |\n| config schema + validation | `internal/infrastructure/configuration/{types,validate}.go` | done (#497); extended here only if ComfyUI's `Workflow` field needs a new validation rule (path exists, non-empty when `Local.Enabled`) |\n| hosted provider | `internal/infrastructure/image/openai/*.go` (or per `Class`) | new (#498) |\n| local provider | `internal/infrastructure/image/comfyui/*.go` | new (#499) |\n| `/image` command spec | `internal/domain/messaging/commands.go` | new entry in `localCommandSpecs` |\n| `/image` routing | `internal/gateway/gateway.go` (`Router.Images`, `handleImage`) | new (#500) |\n| autonomous tool | `internal/tools/image/tool.go` | new (#501) |\n| tool registration | `internal/app/archied/bootstrap.go` (`registerImageTool`, beside `registerMinimaxTool`) | new (#501) |\n| `Registry` wiring | `internal/app/archied/bootstrap.go` (construct `image.Registry`, register configured providers, hand to `Router.Images`) | new — this is the \"nothing to wire until a real provider exists\" step the original contract doc deferred |\n| Telegram delivery | `internal/channels/telegram/media.go` | none — already handles `\"image\"` |\n| Dashboard delivery | `internal/webui/api_chat.go` | none in this epic — tracked separately (archie-core-1786748942243-6-f109697e) |\n| docs | `docs/` user-facing guide (#502) | new, against this design |\n\n## Execution: multi-agent team breakdown\n\nEach row is independently implementable from its own section above plus the\ncall-site inventory — an implementer does not need this whole document\nre-explained, only its row. Lenses are picked from\n`.claude/workflows/council.js` (`boundary`, `contract`, `deletionist`,\n`operator`, `maintainer`) by what each slice actually risks getting wrong,\nnot all five by default.\n\n| sub-feature | issue | implementer scope (call-site rows) | suggested lenses | why |\n|---|---|---|---|---|\n| Hosted provider | #498 | `internal/infrastructure/image/openai/*.go` (generate via ai-sdk, edit via direct HTTP client) | `lens-contract`, `lens-operator` | contract: the error-mapping table must actually produce the sentinels it claims, not leak ai-sdk/HTTP errors raw; operator: a live paid API's rate-limit/timeout/auth-failure paths are exactly the 3am-unattended-host case this lens exists for |\n| Local GPU provider | #499 | `internal/infrastructure/image/comfyui/*.go` (submit/poll against ComfyUI) | `lens-operator`, `lens-deletionist` | operator: backend-down, workflow-missing, and stuck-job failure modes on a host the operator's GPU actually runs; deletionist: a second, unevidenced \"recipe\" added later without real-hardware proof is exactly the speculative scaffolding this lens rejects |\n| `/image` routing | #500 | `internal/domain/messaging/commands.go`, `internal/gateway/gateway.go` (`Router.Images`, `handleImage`) | `lens-boundary`, `lens-maintainer` | boundary: a new `Router` field and its dependency on `image.Registry` must respect the same layering `Models`/`Tasks`/`Controller` already do; maintainer: this is the first local command with an explicit ask-once clarification exchange, not a single-shot reply — legibility for the next person who adds a command matters here |\n| Delivery + autonomous tool | #501 | `internal/tools/image/tool.go`, `internal/app/archied/bootstrap.go` (`registerImageTool`) | `lens-deletionist`, `lens-contract` | deletionist: the whole point of this section is \"reuse `generate_video`'s pipeline, do not rebuild it\" — this lens is the check that the implementation actually did that; contract: the new tool's `MultimodalResult`/`MediaRef{Type: \"image\"}` output must match the shape `multimodalMediaRefs` already decodes, not a near-miss |\n\nRun `/council --lenses \u003cpicked\u003e` once a row's implementation is gate-clean\n(`task check` passing), before opening its PR — same timing as any other\nopen-decision residue at the end of a session, not a standing per-commit\ngate.\n\n## File and link the beads\n\n`#498`–`#501` already exist (bd `archie-core-1786748942169-3-ae7cfaea`,\n`…-219-5-9844a72e`, `…-193-4-5072b564`, `…-243-6-f109697e`), filed under\nepic #496 before this document's decisions were made. Each one's\ndescription now links back to its matching section above instead of\nrestating it, and carries its row from the Execution table above so picking\nit up needs no round-trip back to this document.\n" + "body": "# Image capability contract and configuration\n\n**Status:** Approved\n\nEpic: archie-core-1786748942120-1-6636e629 / GitHub #496 (\"Image generation\ncapability: hosted API baseline plus local GPU workflows\").\n\nThe contract and configuration (#497) shipped first and is settled — see\n\"Package: `internal/domain/image`\" below. This document now covers the full\nfeature set: hosted provider (#498), local GPU provider (#499), `/image`\nrouting and clarification (#500), delivery through channels and web (#501).\nDocumentation (#502) and the cross-cutting test matrix (#503) are execution\nwork against the design below, not open design questions, so they are not\ngiven their own sections — each preceding section states its own test\nobligations inline.\n\n## Problem\n\nArchie needs a provider-neutral boundary for image generation/editing so a\nhosted API (gpt-image-2-class) and a future local GPU backend can both\nimplement it without either one leaking into the domain contract. Nothing\nin the repo defines this (`grep -ri image` over `internal/domain` and\n`internal/config` returns nothing image-specific).\n\n## Design\n\nFollows the plugin engine family shape already established by\n`internal/domain/curator` and `internal/domain/memory`\n(`docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`): a\ntyped contract with real domain operations,\nan owning `Registry` (register/get/names, no start/stop lifecycle — see\nbelow), and narrow host access via a `Registrar`.\n\n### Package: `internal/domain/image`\n\n**Why no `Lifecycle` (`Start/Health/Stop`)**: curator and memory engines are\nlong-running — they hold a background cadence or a persistent connection\nworth health-checking. An image provider is call-scoped: it either serves a\n`Generate`/`Edit` request or it does not. Health here means \"is this\nprovider usable at call time\", which is exactly what `Generate`/`Edit`\nreturning a typed `ErrUnavailable` already communicates — a separate\n`Health()` poll would just duplicate that. `Registry` still owns\nregistration and lookup, so provider selection is testable with fakes per\nthe acceptance criteria, but there is no `Start`/`Stop` to orchestrate.\n\n```go\n// Capability declares what one provider supports, discoverable before a\n// request is built so a caller can reject an unsupported combination (e.g.\n// edit-with-mask on a generate-only provider) instead of sending it and\n// parsing a provider-specific error back out.\ntype Capability struct {\n Generate bool\n Edit bool // edit requires at least one input image\n Mask bool // edit supports a mask restricting the edited region\n MaxInputs int // 0 = Edit unsupported\n Sizes []string // provider-declared output sizes, e.g. \"1024x1024\"\n}\n\n// Provider is the typed contract every image backend implements — hosted\n// (gpt-image-2-class) or local (GPU recipe). No Lifecycle: a provider is\n// call-scoped, not long-running (see design doc rationale).\ntype Provider interface {\n Name() string\n // Class distinguishes hosted (spends money/quota per call) from local\n // (spends host GPU/time). Selection logic and config validation key off\n // this, never off Name, so a new provider cannot silently slip through\n // the hosted-cost gate under an unrecognized name.\n Class() ProviderClass\n Capability() Capability\n Generate(ctx context.Context, req GenerateRequest) (Result, error)\n Edit(ctx context.Context, req EditRequest) (Result, error)\n}\n\ntype ProviderClass string\n\nconst (\n ClassHosted ProviderClass = \"hosted\"\n ClassLocal ProviderClass = \"local\"\n)\n```\n\nRequest/result types (provider-neutral, no OpenAI types):\n\n```go\ntype GenerateRequest struct {\n Prompt string\n Size string // \"\" = provider default\n N int // number of images; 0 = 1\n Options map[string]string // provider-specific overflow (e.g. \"quality\"),\n // deliberately untyped so the contract does not\n // grow a field per provider\n}\n\ntype EditRequest struct {\n Prompt string\n Inputs []ImageData // 1..Capability.MaxInputs\n Mask *ImageData // nil unless Capability.Mask\n Size string\n N int\n Options map[string]string\n}\n\n// ImageData carries image bytes in-process, never a path or URL: the\n// domain layer must not assume a filesystem or a fetchable location (mirrors\n// why curator's Registrar never receives *daemon.Daemon — no ambient host\n// access smuggled in through a \"just a string\" field).\ntype ImageData struct {\n Bytes []byte\n MIMEType string\n}\n\ntype Result struct {\n Images []ImageData\n Provider string\n Model string\n CreatedAt time.Time\n}\n```\n\n### Error taxonomy\n\n```go\nvar (\n ErrUnavailable = errors.New(\"image provider unavailable\") // missing credential, backend down\n ErrUnsupported = errors.New(\"image capability unsupported by provider\") // e.g. Edit on generate-only\n ErrInputTooLarge = errors.New(\"image input exceeds configured limit\")\n ErrTimeout = errors.New(\"image request timed out\")\n ErrCanceled = errors.New(\"image request canceled\")\n)\n```\n\nEvery provider wraps these with `fmt.Errorf(\"%w: %s\", ErrX, detail)` so\ncallers can `errors.Is` regardless of backend. `ErrUnavailable` is the\nexplicit instruction from the epic: \"a missing credential or unavailable\nbackend must make the capability unavailable with a useful user-facing\nerror, not prevent Archie from starting or break text chat\" — Register\nnever rejects the daemon's boot, only marks that one provider absent from\n`Registry.Names()`.\n\n### Registry\n\n```go\ntype Registry struct { /* map[string]Provider, mutex, sorted Names() */ }\n\nfunc NewRegistry() *Registry\nfunc (r *Registry) Register(p Provider) error // ErrDuplicate on name clash\nfunc (r *Registry) Get(name string) (Provider, bool)\nfunc (r *Registry) Names() []string\nfunc (r *Registry) ByClass(class ProviderClass) []string\n```\n\nNo `Registrar`/host-injection type: unlike curator (needs `LLM`, `Skills`,\n`Conversations`) and memory (needs `Clock`, `Events`), an image provider's\ndependencies (HTTP client, API key, local backend handle) are entirely its\nown construction-time concern in `internal/infrastructure/image/*`, not\ndaemon services it needs injected. Introducing an empty `Registrar` now\nwould be exactly the \"field nobody sets\" pattern to avoid.\n\n### Configuration\n\nFollows `MinimaxConfig`'s hosted-cost-guard convention exactly: absent\nconfig or absent explicit `enabled = true` per provider means that\nprovider does not register, not \"registers but errors on first call\".\n\n```go\n// ImageConfig configures the image generation capability.\ntype ImageConfig struct {\n // Default selects which registered provider name /image uses when the\n // request/session has not already picked one. Empty means /image must\n // always ask (the epic's stated default UX) — an empty Default is valid\n // and is not a misconfiguration.\n Default string `toml:\"default\"`\n Hosted map[string]ImageHostedProvider `toml:\"hosted\"` // keyed by provider name\n Local map[string]ImageLocalProvider `toml:\"local\"` // keyed by recipe name\n}\n\n// ImageHostedProvider configures one hosted (paid) provider. Mirrors\n// MinimaxConfig: Enabled defaults false, spending real credits requires an\n// explicit opt-in, never an inferred one from the presence of a key alone.\ntype ImageHostedProvider struct {\n Enabled bool `toml:\"enabled\"`\n Class string `toml:\"class\"` // \"openai\" today; provider-neutral field name\n APIKeyEnv string `toml:\"api_key_env\"`\n APIKey secret.SecretRef `toml:\"api_key\" json:\"-\"`\n BaseURL string `toml:\"base_url\"`\n}\n\n// ImageLocalProvider configures one local GPU recipe. Enabled defaults\n// false too: a local backend can still saturate the host GPU/CPU, which is\n// its own kind of unwanted cost, so it gets the same explicit gate as a\n// hosted provider rather than an on-by-default WebFetch-style treatment.\ntype ImageLocalProvider struct {\n Enabled bool `toml:\"enabled\"`\n Backend string `toml:\"backend\"` // recipe identifier, defined by the local-backend bead\n}\n```\n\nValidation (`internal/infrastructure/configuration/validate.go`, alongside\n`validateMemory`):\n\n- `Default`, if set, must name a provider present (enabled) in `Hosted` or\n `Local` — rejects a config that silently falls back to \"always ask\" when\n the operator thought they'd pinned one.\n- A `Hosted` entry with `Enabled = true` must have `Class` non-empty and\n either `APIKeyEnv` or a resolvable `APIKey` — the same shape check\n `Provider` (chat) already gets, not a new invention.\n- No cross-field surprise: an *absent* `[image]` section is valid and\n produces zero registered providers, matching the epic's non-goal\n (\"silently selecting a paid hosted provider\" must never happen by\n omission either).\n\n### Capability discovery\n\n`Registry.Names()` / `ByClass()` plus `Provider.Capability()` give a caller\n(the future `/image` routing bead) everything needed to ask \"what can I\noffer this user\" without a type switch on provider identity — this is the\nwhole point of the contract being provider-neutral.\n\n### Timeouts, size limits, cancellation\n\n- Every `Provider.Generate`/`Edit` call takes the caller's `ctx`; a\n provider that ignores cancellation is a provider bug, not a contract gap\n — mirrors every other Lifecycle-less call in the codebase (chat's\n `runtime.Chat`).\n- `ImageData` size limits are enforced by the *caller* (the future routing\n bead) before constructing a request, using a configured max — not by this\n contract, which has no config surface for \"current caller's limit\". This\n package only defines `ErrInputTooLarge` so a provider that discovers an\n oversized input itself (rather than trusting the caller) has a typed way\n to say so.\n\n## Testing\n\n- `internal/domain/image/{contract,registry}_test.go` mirror memory's test\n suite shape: registration (duplicate, nil provider), `Names`/`ByClass`\n filtering, `Get` miss, table-driven `Capability` checks via a fake\n provider.\n- `internal/infrastructure/configuration/validate_test.go` gains cases for\n `ImageConfig`: absent section, valid hosted+local, missing `Class`,\n missing credential, `Default` naming an unregistered/disabled provider.\n- No provider implementation exists yet, so no live-call tests — those\n belong to the hosted/local provider beads.\n\n## Hosted provider (#498)\n\n`github.com/samcharles93/ai-sdk` (already a repo dependency) has its own\nprovider-agnostic image facility: `image.Provider` (`Name`,\n`GenerateImage(ctx, GenerateImageRequest) (GenerateImageResponse, error)`),\n`image.Client`, and `registry.Registry.RegisterImage`/`.Image(name)` — the\nsame shape chat model registration already uses. Three backends implement it\n(azure, togetherai, xai); OpenAI does not.\n\n`internal/infrastructure/image.\u003cname\u003eProvider` wraps an `ai-sdk/image.Provider`\nbehind this package's `Provider` interface, the same way `curatorLLMRunner`\n(`internal/app/archied/main.go`) adapts `runtime.Runtime` for chat rather\nthan the domain layer talking to ai-sdk directly.\n\n**Decision: generation goes through ai-sdk, edit does not.** ai-sdk's\n`image.GenerateImageRequest` is text-to-image only — no input-image or mask\nfields — so there is nothing to adapt `EditRequest`/`Capability.Edit` onto.\nBlocking the entire hosted provider on an upstream ai-sdk change is not\nthis epic's call to make on someone else's project's timeline. Instead:\n\n- `Generate` wraps `ai-sdk/image.Client.GenerateImage` against whichever\n backend `ImageHostedProvider.Class` names (azure/togetherai/xai).\n- `Edit` is a direct HTTP client hitting the OpenAI-compatible\n `/images/edits` multipart endpoint, built the same way `minimax.Client`\n (`internal/tools/minimax/minimax.go`) is a direct client for an API\n ai-sdk doesn't cover — not a second abstraction layer, one `http.Client`\n call with request/response mapping into this contract's `EditRequest`/\n `Result`.\n- A provider that only wraps ai-sdk (no edit client configured) declares\n `Capability{Generate: true, Edit: false}` truthfully rather than\n returning `ErrUnsupported` at call time for something `Capability`\n already said it couldn't do.\n- If ai-sdk gains edit support upstream later, the direct client is deleted\n and `Edit` moves onto it — recorded here so that's a known, not a\n surprise, follow-up.\n\n**Provider error mapping** (every ai-sdk/HTTP error translated at the\nadapter boundary, never leaked as-is):\n\n| ai-sdk / HTTP condition | contract error |\n|---|---|\n| `image.ErrNoProvider`, `image.ErrAuthFailed`, missing/invalid credential, backend unreachable | `ErrUnavailable` |\n| `image.ErrEditNotSupported`, request needs a `Capability` the provider didn't declare | `ErrUnsupported` |\n| input image over configured byte limit (checked before the call, per \"Timeouts, size limits, cancellation\" above) | `ErrInputTooLarge` |\n| `ctx.Err() == context.DeadlineExceeded` | `ErrTimeout` |\n| `ctx.Err() == context.Canceled` | `ErrCanceled` |\n| `image.ErrRateLimited`, `image.ErrContentFiltered`, any other 4xx/5xx | wrapped verbatim under `ErrUnavailable` with the provider's own message as detail — a rate limit or content filter is \"unavailable for this request\", not a new sentinel the caller needs to branch on differently |\n\n**Testing**: `httptest.Server` fixtures for the edit client (request\nmultipart shape, response decoding, 4xx/5xx mapping); a fake `ai-sdk/image.Provider`\nfor the generate path so the adapter's translation is tested without a live\nbackend, matching #498's \"no live API key needed for tests\" criterion.\n\n## Local GPU provider (#499)\n\n**Decision: ComfyUI is the baseline recipe**, not a name this doc picked\narbitrarily — it is the de facto scriptable HTTP API for Stable-Diffusion-class\nlocal generation, already shaped like the async submit/poll pattern\n`minimax.Client.GenerateAndWait` established in this codebase (submit a\nworkflow, poll `/history/{id}` until the output node has files, download).\nNo other local backend is referenced anywhere in this repo or its docs, so\nthis decision has no competing prior art to reconcile — it is a fresh\nchoice, recorded here so the implementing bead does not re-litigate it.\n\n```go\n// internal/infrastructure/image/comfyui\ntype Config struct {\n Enabled bool\n BaseURL string // e.g. http://127.0.0.1:8188\n Workflow string // path to a saved ComfyUI workflow JSON template\n // (prompt/size/seed substituted at call time)\n Timeout time.Duration\n PollInterval time.Duration\n MaxWait time.Duration\n}\n```\n\n- `Generate`/`Edit` load `Workflow`, substitute the templated fields (prompt,\n size, seed, and for `Edit` the input image as a base64 node input — ComfyUI\n supports img2img/inpaint workflows natively, so `Capability.Edit`/`Mask`\n can both be true for a correctly authored workflow, unlike the hosted\n provider), POST to `/prompt`, then poll `/history/{prompt_id}` the same\n shape MiniMax's poll loop already proves out in this codebase.\n- `Capability.Sizes` is empty (ComfyUI workflows declare their own\n resolution in the template, not a fixed enum) — the contract already\n allows an empty `Sizes` to mean \"provider does not constrain size\".\n- Backend-down (`BaseURL` unreachable) and workflow-missing are both\n `ErrUnavailable`, not a startup failure — same \"degrade the capability,\n never break text chat\" rule the contract already states.\n- Per #499's own acceptance criteria, at least one recipe must be exercised\n end-to-end on real hardware or a reproducible harness, and no model\n weights are ever committed or fetched by Archie itself — the operator\n points `Workflow`/`BaseURL` at their own running ComfyUI instance and\n already-downloaded checkpoint, exactly like `[providers.ollama]` points at\n an already-running Ollama rather than Archie managing the model.\n\n**Testing**: a fake HTTP server standing in for ComfyUI's `/prompt` and\n`/history` endpoints (submit, poll, malformed history, timeout, job\nnot-found); a real-hardware smoke test documented as a manual `task`\ntarget per #503, not part of `task check` — the same reasoning `task vuln`\nand `task deadcode` are excluded from the gate.\n\n## `/image` routing and clarification flow (#500)\n\n**Call site: `internal/gateway/gateway.go`'s `Router`**, following the\n`/model` shape exactly — `/model`, `/spawn`, `/approve` etc. are already\n\"commands handled without the LLM\" (`Router.handleModel`, dispatched via\n`isLocalCommand`/`parseCmd` against `messaging.LocalCommandSpecs()`).\n`/image` is a new entry in `internal/domain/messaging/commands.go`'s\n`localCommandSpecs`, and a new `Router.handleImage(ctx, arg)` beside\n`handleModel`, gated on a new `Router.Images *image.Registry` field (nil =\n\"/image not configured\", the same pattern `Models`/`Tasks`/`Controller`\nalready use — not a new gating convention).\n\n**Decision: mechanism selection is per-request, not session-persisted.**\nThe epic allows \"a remembered preference only where existing session/config\nconventions support it\" — and checking `Router`'s actual fields\n(`ModelPersist`, `Identity`, the rest) shows there is no generic per-session\nkey/value store the neutral command path can lean on; `/model`'s own\n`--session`/`--global` scoping lives entirely in\n`internal/channels/telegram/model.go`, a Telegram-specific enrichment, not\nsomething `Router` provides generically. Inventing a new generic\nsession-preference store just for `/image` would be exactly the kind of\nscaffolding CLAUDE.md's Architectural Simplicity rule warns against for one\ncaller. So: every `/image` invocation with no explicit mechanism produces\nthe clarification prompt; an explicit `--hosted`/`--local` flag (mirroring\n`parseModelRest`'s `--provider` flag shape) skips it for that one call.\nRevisit if a second command wants session-scoped preference too — that's\nthe point at which shared infra earns its cost.\n\n**Flow**, matching `handleModel`'s reply-string shape (no separate state\nmachine object, since the whole exchange is \"ask once, use the answer\ninline in the next message\" — the model-selector's own worked example):\n\n1. `/image \u003cprompt\u003e` with no `--hosted`/`--local` → reply lists the\n registered classes (from `Registry.ByClass`) and asks the user to repeat\n the command with `--hosted` or `--local`. No provider is called.\n2. `/image --hosted \u003cprompt\u003e` or `--local` → resolves to\n `Registry.ByClass(class)`, picks `ImageConfig.Default` if it names a\n provider of that class, else the sole registered provider of that class,\n else (more than one, no default) asks which named provider.\n3. Edit requests (`/image --edit \u003cprompt\u003e` with an attached image on the\n inbound message) validate against `Capability.Edit`/`MaxInputs`/`Mask`\n before calling — an edit request to a generate-only provider is a\n clarification reply (\"this provider doesn't support editing\"), not a\n provider call that returns `ErrUnsupported`.\n4. A successful call returns a `Result`; delivery is the next section's\n concern, not this one's — `handleImage` returns the same\n `(string, error)` shape `handleModel` does for the text portion, and a\n separate mechanism (below) carries the actual image bytes.\n\n**Testing**: channel-neutral state-machine tests for `handleImage` against\na fake `*image.Registry` (mirrors `#500`'s own acceptance criterion); Telegram\nadapter tests only for wiring, not for re-testing the state machine.\n\n## Delivery through channels and web (#501)\n\nThis is the smallest section because most of the delivery path **already\nexists and is already proven** — by `generate_video`\n(`internal/tools/minimax`), which ships. Do not rebuild it.\n\nThe existing pipeline: a result carrying media returns\n`tools.MultimodalResult{IsMultimodal: true, URLs: []tools.MediaRef{{Type: \"image\", ...}}}`\n→ `multimodalMediaRefs` (`internal/app/archied/telegram_setup.go`) decodes\nit → `TurnStream.Media(gateway.MediaEvent)` → each channel's own\n`Media`/`MediaAttachment` handling. `MediaRef.Type` already documents\n`\"image\"` as a valid value alongside `\"video\"`; nothing in the pipeline is\nvideo-specific.\n\n**Per-channel state, checked directly rather than assumed:**\n\n| channel | image delivery | evidence |\n|---|---|---|\n| Telegram | **already works** | `internal/channels/telegram/media.go`'s `telegramMediaSender.dispatch` already branches on `att.Type` and calls `bot.SendPhoto` for `\"image\"` (`SendVideo` for `\"video\"` is the same function) |\n| Dashboard (webui) | **tracked gap, not new** | `internal/webui/api_chat.go`'s `chatStreamSink.Media` has no inline rendering path yet and degrades to a link/error string — already tracked as archie-core-1786748942243-6-f109697e. This epic does not need to fix it, but #501's acceptance criterion (\"unsupported paths report a clear limitation\") is already met by the existing degrade-to-text behavior; closing the dashboard gap is that bead's job, not this one's |\n| Email | **no attachment sender exists** | `grep MediaAttachment internal/channels/email` returns nothing. Out of scope for this epic per #496's non-goals (no channel-abstraction rework) — email delivery of a generated image degrades the same way the dashboard does which is acceptable under #501's \"or unsupported paths report a clear limitation\" clause |\n| Webhook | not applicable | webhook is an inbound intake channel (`internal/channels/webhook`), not an outbound chat reply target — no `/image` command can originate there in the first place |\n\n**The only new work `internal/tools/minimax` didn't already need**: the\nautonomous-tool call site. `/image` (previous section) calls a provider\ndirectly and does not go through the model's tool-calling loop at all.\nSeparately, exposing `generate_image`/`edit_image` as an ordinary tool\n(so an agent task can generate a diagram or illustration mid-task, not only\na human typing `/image`) is a thin wrapper — `internal/tools/image/tool.go`,\nsame shape as `internal/tools/minimax/tool.go`, calling\n`image.Registry.Get(name).Generate/Edit` and returning the identical\n`MultimodalResult` shape. Wired in `bootstrap.go` next to\n`registerMinimaxTool`, gated the same way (registered only when a provider\nresolved a working credential/backend, never advertised broken). This\nreuses the entire delivery pipeline above with zero new plumbing — the tool\ncall site is what's new, not the delivery.\n\n**Testing**: `multimodalMediaRefs` already has coverage for the shape;\nadd a table case for `Type: \"image\"` if one doesn't already exist. The tool\nwrapper gets the same fake-provider test shape as the routing flow above.\nNo new delivery-path tests are needed — that pipeline's tests belong to\nwhichever change originally proved it out.\n\n## Call site inventory\n\nEvery file a full implementation touches, so no bead re-derives this by\ngrep:\n\n| concern | file | change |\n|---|---|---|\n| domain contract | `internal/domain/image/{contract,registry}.go` | done (#497) |\n| config schema + validation | `internal/infrastructure/configuration/{types,validate}.go` | done (#497); extended here only if ComfyUI's `Workflow` field needs a new validation rule (path exists, non-empty when `Local.Enabled`) |\n| hosted provider | `internal/infrastructure/image/openai/*.go` (or per `Class`) | new (#498) |\n| local provider | `internal/infrastructure/image/comfyui/*.go` | new (#499) |\n| `/image` command spec | `internal/domain/messaging/commands.go` | new entry in `localCommandSpecs` |\n| `/image` routing | `internal/gateway/gateway.go` (`Router.Images`, `handleImage`) | new (#500) |\n| autonomous tool | `internal/tools/image/tool.go` | new (#501) |\n| tool registration | `internal/app/archied/bootstrap.go` (`registerImageTool`, beside `registerMinimaxTool`) | new (#501) |\n| `Registry` wiring | `internal/app/archied/bootstrap.go` (construct `image.Registry`, register configured providers, hand to `Router.Images`) | new — this is the \"nothing to wire until a real provider exists\" step the original contract doc deferred |\n| Telegram delivery | `internal/channels/telegram/media.go` | none — already handles `\"image\"` |\n| Dashboard delivery | `internal/webui/api_chat.go` | none in this epic — tracked separately (archie-core-1786748942243-6-f109697e) |\n| docs | `docs/` user-facing guide (#502) | new, against this design |\n\n## Execution: multi-agent team breakdown\n\nEach row is independently implementable from its own section above plus the\ncall-site inventory — an implementer does not need this whole document\nre-explained, only its row. Lenses are picked from\n`.claude/workflows/council.js` (`boundary`, `contract`, `deletionist`,\n`operator`, `maintainer`) by what each slice actually risks getting wrong,\nnot all five by default.\n\n| sub-feature | issue | implementer scope (call-site rows) | suggested lenses | why |\n|---|---|---|---|---|\n| Hosted provider | #498 | `internal/infrastructure/image/openai/*.go` (generate via ai-sdk, edit via direct HTTP client) | `lens-contract`, `lens-operator` | contract: the error-mapping table must actually produce the sentinels it claims, not leak ai-sdk/HTTP errors raw; operator: a live paid API's rate-limit/timeout/auth-failure paths are exactly the 3am-unattended-host case this lens exists for |\n| Local GPU provider | #499 | `internal/infrastructure/image/comfyui/*.go` (submit/poll against ComfyUI) | `lens-operator`, `lens-deletionist` | operator: backend-down, workflow-missing, and stuck-job failure modes on a host the operator's GPU actually runs; deletionist: a second, unevidenced \"recipe\" added later without real-hardware proof is exactly the speculative scaffolding this lens rejects |\n| `/image` routing | #500 | `internal/domain/messaging/commands.go`, `internal/gateway/gateway.go` (`Router.Images`, `handleImage`) | `lens-boundary`, `lens-maintainer` | boundary: a new `Router` field and its dependency on `image.Registry` must respect the same layering `Models`/`Tasks`/`Controller` already do; maintainer: this is the first local command with an explicit ask-once clarification exchange, not a single-shot reply — legibility for the next person who adds a command matters here |\n| Delivery + autonomous tool | #501 | `internal/tools/image/tool.go`, `internal/app/archied/bootstrap.go` (`registerImageTool`) | `lens-deletionist`, `lens-contract` | deletionist: the whole point of this section is \"reuse `generate_video`'s pipeline, do not rebuild it\" — this lens is the check that the implementation actually did that; contract: the new tool's `MultimodalResult`/`MediaRef{Type: \"image\"}` output must match the shape `multimodalMediaRefs` already decodes, not a near-miss |\n\nRun `/council --lenses \u003cpicked\u003e` once a row's implementation is gate-clean\n(`task check` passing), before opening its PR — same timing as any other\nopen-decision residue at the end of a session, not a standing per-commit\ngate.\n\n## File and link the beads\n\n`#498`–`#501` already exist (bd `archie-core-1786748942169-3-ae7cfaea`,\n`…-219-5-9844a72e`, `…-193-4-5072b564`, `…-243-6-f109697e`), filed under\nepic #496 before this document's decisions were made. Each one's\ndescription now links back to its matching section above instead of\nrestating it, and carries its row from the Execution table above so picking\nit up needs no round-trip back to this document.\n" }, { "url": "/dev/prds/inline-review/", @@ -274,7 +274,7 @@ "url": "/dev/prds/memory-engine-unification/", "title": "Memory engine unification: four scopes, CRUD, and one write path -- decision", "section": "PRDs", - "body": "# Memory engine unification: four scopes, CRUD, and one write path -- decision\n\n**Status:** Approved\n**Date:** 2026-09-16\n**Findings:** `memory-4` and its dependants in the adversarial sweep\n(`.local/issue-tracker`); the real defect is the split described below, for\nwhich `memory-4` is a symptom\n**Supersedes:** the draft `memory-turn-context.md` (its synchronous-read\nargument survives and is reused; its `Manager` method and consumer-owned port do\nnot)\n**Settles:** four open items in `docs/architecture/migration-decisions.md#5`\n(listed at the end)\n\n## Problem\n\nThere are **two live memory stores for one assistant**, split by which component\nwrites them, and the chat turn reads neither.\n\n| Family | Written by | Root | Format |\n|---|---|---|---|\n| `internal/memory` (`Manager`, `builtin.Provider`) | the chat agent's `memory_edit` tool via `memorytoolprovider` (`bootstrap.go`) | `\u003cworkDir\u003e/memory` | `MEMORY.md` / `USER.md` |\n| `internal/domain/memory` + `internal/infrastructure/memory` | the session curator (`sessioncurator.go`) | `\u003cworkDir\u003e/memory-engine` | identity-keyed blocks |\n\nThey share no code, no format and no scope vocabulary. Neither reaches a prompt:\n`Manager.SystemPromptBlock` (`internal/memory/manager.go`) has no production\ncaller anywhere, so the agent can write memory through a tool and never read it\nback.\n\nThree further defects fall out of the same split:\n\n1. **The engine's contract is not CRUD.** `Store` is `Write`/`Query`/`List`/\n `Forget` — create, read, delete, and **no Update**. The retired file provider\n *does* have `handleAdd`/`handleReplace`/`handleRemove`, so the family being\n deleted is operationally richer than its replacement.\n2. **There is no scope model.** The contract carries one opaque `Identity\n string`, and the curator fills it with a **session id** (`sessioncurator.go`)\n — which is none of the four scopes `docs/architecture/agent-system.md`\n requires, and stops meaning anything when the session ends.\n3. **The chat turn has no user identity on half the channels.** Telegram and\n email populate `SenderID` with a person; the dashboard populates it with\n nothing, and a webhook populates it with nothing either — a route path names\n a source, not a person, so `webhook.go` leaves `SenderID` empty and carries\n the route in `Inbound.BudgetKey` for rate limiting. `sender_id` is persisted\n (`session_store_sqlite.go`), so whatever a channel puts there reaches the\n consumers that read it.\n\n`agent-system.md` says the write-scope selection and cross-scope sharing\nrules \"require a focused memory design decision\". This is that decision.\n\n## Decision\n\n### 1. One engine, addressed by four typed scopes\n\n`internal/domain/memory` is the only memory engine. `Observation.Identity`,\n`Query.Identity`, `Record.Identity` and `List(identity)` are deleted, not\nextended: a field whose meaning depends on the caller cannot carry a scope.\n\n```go\ntype ScopeKind string\n\nconst (\n ScopeGlobal ScopeKind = \"global\" // shared across the application\n ScopeAgent ScopeKind = \"agent\" // private to one Agent\n ScopeUser ScopeKind = \"user\" // one user, across their Agents\n ScopeAgentUser ScopeKind = \"agent-user\" // one Agent's relationship with one user\n)\n\ntype Scope struct {\n Kind ScopeKind\n Agent AgentID // required for ScopeAgent, ScopeAgentUser\n User IdentityID // required for ScopeUser, ScopeAgentUser\n}\n\nfunc (s Scope) Validate() error // rejects a component present where it is not required\nfunc (s Scope) Key() string // canonical, collision-free storage key\n```\n\n`AgentID` and `IdentityID` are opaque. This package never resolves or\ncanonicalises them.\n\nA turn's resolved identity is one value:\n\n```go\ntype Subject struct {\n AgentID AgentID\n UserID IdentityID // \"\" when the channel carries no per-person identity\n}\n\nfunc (s Subject) Scopes() []Scope // readable, most specific first\nfunc (s Subject) WritableScopes() []Scope // never includes ScopeGlobal\n```\n\n**Access control is naming a scope, and nothing else.** The read path names\nexactly the scopes of the resolved `Subject`; the engine never unions beyond what\nit is handed and applies no policy of its own. An engine that enforced policy\nwould need to know about channels, sessions and bindings, and every backend would\nreimplement it. \"Agent-wide is private to its owning Agent by default\"\n(`agent-system.md`) falls out mechanically, because `ScopeAgent{a}` and\n`ScopeAgent{b}` are different storage keys.\n\n`ScopeGlobal` is never model-writable. Writing it is an operator action; nothing\nproduces one yet, but the contract models it because the read set includes it.\n\n### 2. CRUD, with revisions retained\n\n```go\ntype Store interface {\n Create(ctx context.Context, in NewRecord) (Record, error)\n Get(ctx context.Context, scope Scope, id RecordID) (Record, error) // ErrNotFound\n Query(ctx context.Context, q Query) ([]Record, error) // heads only\n List(ctx context.Context, scope Scope) ([]Record, error) // heads only\n Update(ctx context.Context, in RecordUpdate) (Record, error) // ErrNotFound, ErrStaleRevision\n Forget(ctx context.Context, scope Scope, id RecordID) error // idempotent\n Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)\n}\n```\n\n`Get` and `Forget` take a scope. The current `Forget` scans every identity's\nstore to find a marker (`builtin_engine.go`) purely because the contract\ngave it nothing better.\n\n`Record` carries what `agent-system.md` requires be retained: `Scope`,\n`Kind`, `Content`, `Revision`, `Author`, `OriginUser`, `Source`, `CreatedAt`,\n`UpdatedAt`. `Metadata` is dropped — nothing sets it and the parser never\nrestores it, so it is a field that silently does not round-trip.\n\n`Update` supersedes rather than overwrites. The superseded state is appended to\nthe scope's `HISTORY.md` **before** the live block is replaced, so a crash\nbetween the two leaves an extra history entry and an unchanged record —\nrecoverable — never a record whose previous state was lost. `Update` carries no\n`Scope`: a record may never move between scopes, and the API makes that\nunrepresentable rather than validating it. `Expected` optionally names the\nrevision being replaced and returns `ErrStaleRevision` on mismatch, which matters\nbecause two processes (`archied` and `archie-gateway`) can hold the same scope's\nfile.\n\n`Forget` records a deleted revision before removing the live block, so a record's\nprovenance outlives its content.\n\n### 3. The caller resolves identity; the engine never does\n\nResolution happens in the gateway turn path, where the native sender id and the\nsession are both in hand, and it is supplied **per channel by the composition**:\n\n```go\n// TurnRunnerConfig\n// UserIdentity resolves the initiating user's identity for one inbound message.\n// false means the channel carries no per-person identity, and the turn gets\n// global and agent scopes only. Nil means the same.\nUserIdentity func(msg messaging.Message) (IdentityID, bool)\n```\n\nThis must be injectable rather than a blanket `channel + \":\" + SenderID`\nconcatenation, because a channel's sender id is not necessarily a person: a\nwebhook's route path names a source, and treating it as an identity would give\na URL one -- which is why `webhook.go` leaves `SenderID` empty and carries the\nroute in `Inbound.BudgetKey` instead. Telegram and email supply their sender\nid; the dashboard and webhook resolve nothing.\n\n**Fail closed.** When `UserID` is empty the turn proceeds with global and agent\nscopes only, and a write targeting `user` or `agent-user` fails with a clear\nerror. `agent-system.md` forbids showing one user's memory to another, so\n\"we do not know who this is\" must mean \"show none of it\", never \"fall back to a\nwider scope\".\n\n### 4. The chat read path\n\nSynchronous and total. It runs inside `prepareTurn` (`turn.go`) before the\nprompt is built, never returns an error to the caller, and degrades to an empty\nblock on any engine failure or provider panic. A fire-and-forget hook would race\nthe prompt build in the same turn, which is why this is not a lifecycle hook.\n\nThe read is a scope-only `Query` over the subject's scopes — **not** ranked\nsearch, which remains open in `migration-decisions.md#5`. A ranked read on the\nhot path with no ranking implementation would be a lie.\n\n`SystemPromptConfig` gains a `Memory` field, rendered into\n`templates/archie.md.tpl` between the `\u003ctools\u003e` and `\u003cenv\u003e` blocks:\n\n```\n\u003cmemory purpose=\"durable_context\" trust=\"data\"\u003e\n```\n\n`trust=\"data\"` and inclusion in the precedence clause follow the `\u003ctools\u003e`/`\u003cenv\u003e`\nprecedent exactly: memory content is model-written from untrusted input and must\nbe framed as data, never instructions. The rendered block carries each record's\nshort id, because the write path addresses records by id — one source of truth\nfor both directions.\n\nBoth a per-scope record limit and a byte cap on the rendered block are named\nconstants. The cap is about size, not latency; a local read needs no timeout.\n\n**Prerequisite:** `setupGateways` runs before `setupMemoryAll`\n(`main.go` vs `:428`), and `startGatewayRuntime` before `setupMemory`\n(`gateway.go` vs `:115`), so the engine is nil at every turn-runner\nconstruction site. The ordering is the root-cause fix and nothing in either\nmemory step reads what those later steps produce.\n\n### 5. The chat write path\n\nThe memory tool is built **per turn**, not registered once at boot, because it\nmust write into a scope derived from the turn's resolved subject:\n\n| action | engine call |\n|---|---|\n| `create` | `Create` |\n| `update` | `Update` (by id) |\n| `delete` | `Forget` (by id) |\n| `list` | `List` |\n\n**The model chooses the scope kind; it never chooses the ids.** Agent and user\nids come from the turn's resolution and there is no schema field to put anything\nelse in — even a model that emits another user's id has nowhere to write it. That\nis the root-cause form of `agent-system.md`, matching this system's other\nnon-waivable boundaries (the model never runs git; gates cannot be waived by the\nagent). `global` is not offered at all.\n\nThe tool takes a record **id** where the retired provider took a content\nsubstring (`handleReplace`/`handleRemove`). The substring addressing existed\nbecause a markdown file was hand-edited and the model had no stable handle on a\nblock; with ids rendered into the prompt it disappears, and one addressing scheme\nreplaces two.\n\n### 6. The curator's write\n\n`sessioncurator.reviewOne` stops writing `Identity: sessionID` and writes\n`ScopeAgentUser{AgentID: \u003csession's agent\u003e, UserID: \u003cparticipant\u003e}`. Relationship\nscope is the only defensible home: agent-wide is explicitly unsafe for user facts\nand would leak user A's extracted facts to user B through the same agent;\nuser-wide would pool facts across agents; and a session id cannot survive the\nsession ending, which is the whole requirement.\n\nThis needs data the tree does not persist: the SQLite `messages` table\ngains `sender_id`, and `curator.SessionSummary` / `ConversationMessage` gain the\nagent and participant. The participant is derived from the session's own\nuser-role messages: exactly one distinct non-empty sender writes; zero (a\ndashboard or webhook session) or several (a group chat) writes nothing and\nrecords why. Ambiguity fails closed rather than guessing.\n\nThe payoff is the point of the exercise: chat-written and curator-derived memory\nnow occupy the same scopes, so a future turn's prompt contains both, and a\ncurator's bad note becomes id-addressable and therefore correctable by the agent.\n\n### 7. The content scanner moves to the engine\n\n`internal/memory/scanner.go` and its installation at `bootstrap.go` are what\nmake `configuration.md`'s \"Memory safety scanner\" row describe a control that\nactually runs. Deleting the package would silently remove prompt-injection\nscanning on memory writes. The scanner moves into the engine and is applied in\n`Create` and `Update`, which is the single choke point every producer crosses —\nso the curator's model-extracted content is covered too, not just the chat tool's.\n\n### 8. `internal/memory` is deleted\n\nNot shimmed, not partially, not behind a flag: `manager.go`, `provider.go`,\n`lifecycle.go`, `sync.go`, `scanner.go` (moved), `schema.go`, `backup.go`,\n`config.go`, `builtin/provider.go`, and their tests. Its markdown store\n(`builtin/store.go`) is the one survivor, relocated to\n`internal/infrastructure/memory/builtin/` — `organisation.md`'s \"a package owns\nits on-disk format end-to-end\" makes infrastructure its correct home, and the\nengine already parses that format.\n\nAlso deleted: `internal/tools/provider/memory`'s manager wrapping (rewired to the\nengine), `config.Memory.{Provider,ProviderConfig}` and the\n`validateMemory` rejection that exists only to complain about `Provider`\n(`validate.go`), and `webui.MemoryStatus`/`MemoryProviderHandle` — never\nimplemented in production, so `/api/memory` reports empty forever unless it is\nrepointed at the engine registry.\n\nExisting data at `\u003cworkDir\u003e/memory` is left on disk, unreferenced. No migration.\n`\u003cworkDir\u003e/memory-engine` is likewise orphaned: its directories were keyed by\n`sha256(sessionID)` and no scope key can hash to one.\n\n## What we deliberately do NOT do\n\n- **No cross-scope sharing or grants in this wave.** Promoting a record between\n scopes is a future explicit command, and `Update` cannot change scope, so it\n cannot happen by accident.\n- **No canonical user identity.** `SenderID` is channel-native, so one person on\n Telegram and email is two identities. Unifying needs an Identity aggregate that\n does not exist (`internal/domain` has no identity package and `IdentityID`\n appears nowhere in Go). The contract does not change when it lands.\n- **No ranked retrieval.** `Query.Text` stays, unused by the prompt path.\n- **No data migration**, per the decision above.\n- **No prompt-memory timeout mechanism.** The read is bounded by construction.\n- **No `Metadata` field.** It is write-only; a caller needing it later\n needs one field, not a speculative contract.\n\n## Acceptance criteria\n\n1. A record created through the memory tool in turn 1 appears in turn 2's prompt,\n asserted on content the test planted, with its id.\n2. The same record is **absent** from a different user's prompt on the same\n channel and agent. Isolation asserted end to end, not at the engine boundary.\n3. An agent-scope record is visible to both of that agent's users; a global\n record is visible to a second agent; an agent-user record is not.\n4. A curator pass over a session writes a record a later turn reads, and which\n survives that session being deleted.\n5. `create` → prompt shows it; `update` → prompt shows new content and\n `Revisions` returns the prior content with author and time; `delete` → gone\n from prompt and `List`. Round-trip is asserted as `parse(render(r)) == r`,\n never against a fixture string.\n6. A panicking engine and a failing engine both complete the turn with no memory\n block.\n7. A channel with no resolvable user renders only global and agent scopes, and\n the tool refuses `user`/`agent-user` with a reason. A webhook route path never\n becomes a user identity.\n8. A write that would exceed the file cap fails loudly rather than silently\n dropping content.\n9. `internal/memory` is gone, `task check` is green, and no document describes a\n manager that no longer exists.\n\nEvery criterion asserts behaviour or planted content. None asserts on the text of\na hand-written file, and none pins that a method was called. In particular none\nasserts the composition calls setup in a given order: if the ordering were\nswapped back, criteria 1 and 4 would fail, and that is the assertion.\n\n## Sequencing\n\nEach step is independently verifiable and lands green on its own.\n\n1. **Persist the participant.** `sender_id` column on `messages`,\n insert/select/migration, `curator.SessionSummary.AgentID` and\n `ConversationMessage.SenderID`. Touches no memory contract and is shippable\n alone — nothing consumes it yet.\n2. **Scope, CRUD and revisions.** The `internal/domain/memory` contract, the\n builtin engine with `HISTORY.md`, the store relocation, the scanner move, and\n the curator rewritten onto agent-user scope (it is the engine's only live\n consumer, so it belongs in this slice). *Depends on 1 for the curator half.*\n3. **Chat read path.** The `Memory` seam, the template block, subject resolution,\n and the composition-ordering fix. *Depends on 2.*\n4. **Chat write path.** The tool adapter onto CRUD and the turn's subject on the\n context. *Depends on 3, because a write is only observable through the read.*\n5. **Delete `internal/memory` and the dead config/dashboard surface**, and\n rewrite `ARCHITECTURE.md`'s \"Memory engine family\" section to describe the\n engine family rather than a manager it deleted. *Depends on 3 and 4.*\n\nSteps 3 and 4 are separable from each other; step 1 is separable from everything.\n\n## Settles these open items in `migration-decisions.md` §5\n\n| §5 open item | Settled by |\n|---|---|\n| authoritative memory record and revision model | §1, §2 — the live marker becomes JSON, since the ratified `\u003c!--mem:\u003culid\u003e--\u003e` cannot carry provenance; superseded states live in a per-scope `HISTORY.md` |\n| provider and infrastructure boundaries | §2 — contract in `internal/domain/memory`, builtin format in `internal/infrastructure/memory/builtin` |\n| retrieval and access enforcement | §1 — caller-named scope sets, no engine-side policy |\n| provenance representation | §2 |\n\n## Not determined\n\n- **Group chats produce no derived memory**, because the curator fails closed on\n mixed-sender sessions. Whether extraction should instead become\n per-participant is a product call.\n- **Two processes, one scope file.** After step 3 both `archied` and\n `archie-gateway` hold an engine over the same root, and the engine caches a\n store per scope and rewrites the whole file, so concurrent writers on one scope\n can lose a block. `Update.Expected` makes that detectable for one record; it\n does not make the file safe. A reload-before-mutate or advisory lock is the fix\n and is not designed here.\n- **Unbounded history.** `HISTORY.md` grows without limit; no compaction policy.\n- **Size bounds.** `defaultMaxFileBytes` is now per scope rather than per\n identity; whether that is right is unexamined.\n- **Whether the dashboard should carry a user identity.** It has one bearer\n token, not users. Until it has users, web turns are correctly limited to global\n and agent scope.\n" + "body": "# Memory engine unification: four scopes, CRUD, and one write path -- decision\n\n**Status:** Approved\n**Date:** 2026-09-16\n**Findings:** `memory-4` and its dependants in the adversarial sweep\n(`.local/issue-tracker`); the real defect is the split described below, for\nwhich `memory-4` is a symptom\n**Supersedes:** the draft `memory-turn-context.md` (its synchronous-read\nargument survives and is reused; its `Manager` method and consumer-owned port do\nnot)\n**Settles:** four open items in `docs/architecture/migration-decisions.md#5`\n(listed at the end)\n\n## Problem\n\nThere are **two live memory stores for one assistant**, split by which component\nwrites them, and the chat turn reads neither.\n\n| Family | Written by | Root | Format |\n|---|---|---|---|\n| `internal/memory` (`Manager`, `builtin.Provider`) | the chat agent's `memory_edit` tool via `memorytoolprovider` (`bootstrap.go`) | `\u003cworkDir\u003e/memory` | `MEMORY.md` / `USER.md` |\n| `internal/domain/memory` + `internal/infrastructure/memory` | the session curator (`sessioncurator.go`) | `\u003cworkDir\u003e/memory-engine` | identity-keyed blocks |\n\nThey share no code, no format and no scope vocabulary. Neither reaches a prompt:\n`Manager.SystemPromptBlock` (`internal/memory/manager.go`) has no production\ncaller anywhere, so the agent can write memory through a tool and never read it\nback.\n\nThree further defects fall out of the same split:\n\n1. **The engine's contract is not CRUD.** `Store` is `Write`/`Query`/`List`/\n `Forget` — create, read, delete, and **no Update**. The retired file provider\n *does* have `handleAdd`/`handleReplace`/`handleRemove`, so the family being\n deleted is operationally richer than its replacement.\n2. **There is no scope model.** The contract carries one opaque `Identity\n string`, and the curator fills it with a **session id** (`sessioncurator.go`)\n — which is none of the four scopes `docs/architecture/agent-system.md`\n requires, and stops meaning anything when the session ends.\n3. **The chat turn has no user identity on half the channels.** Telegram and\n email populate `SenderID` with a person; the dashboard populates it with\n nothing, and a webhook populates it with nothing either — a route path names\n a source, not a person, so `webhook.go` leaves `SenderID` empty and carries\n the route in `Inbound.BudgetKey` for rate limiting. `sender_id` is persisted\n (`session_store_sqlite.go`), so whatever a channel puts there reaches the\n consumers that read it.\n\n`agent-system.md` says the write-scope selection and cross-scope sharing\nrules \"require a focused memory design decision\". This is that decision.\n\n## Decision\n\n### 1. One engine, addressed by four typed scopes\n\n`internal/domain/memory` is the only memory engine. `Observation.Identity`,\n`Query.Identity`, `Record.Identity` and `List(identity)` are deleted, not\nextended: a field whose meaning depends on the caller cannot carry a scope.\n\n```go\ntype ScopeKind string\n\nconst (\n ScopeGlobal ScopeKind = \"global\" // shared across the application\n ScopeAgent ScopeKind = \"agent\" // private to one Agent\n ScopeUser ScopeKind = \"user\" // one user, across their Agents\n ScopeAgentUser ScopeKind = \"agent-user\" // one Agent's relationship with one user\n)\n\ntype Scope struct {\n Kind ScopeKind\n Agent AgentID // required for ScopeAgent, ScopeAgentUser\n User IdentityID // required for ScopeUser, ScopeAgentUser\n}\n\nfunc (s Scope) Validate() error // rejects a component present where it is not required\nfunc (s Scope) Key() string // canonical, collision-free storage key\n```\n\n`AgentID` and `IdentityID` are opaque. This package never resolves or\ncanonicalises them.\n\nA turn's resolved identity is one value:\n\n```go\ntype Subject struct {\n AgentID AgentID\n UserID IdentityID // \"\" when the channel carries no per-person identity\n}\n\nfunc (s Subject) Scopes() []Scope // readable, most specific first\nfunc (s Subject) WritableScopes() []Scope // never includes ScopeGlobal\n```\n\n**Access control is naming a scope, and nothing else.** The read path names\nexactly the scopes of the resolved `Subject`; the engine never unions beyond what\nit is handed and applies no policy of its own. An engine that enforced policy\nwould need to know about channels, sessions and bindings, and every backend would\nreimplement it. \"Agent-wide is private to its owning Agent by default\"\n(`agent-system.md`) falls out mechanically, because `ScopeAgent{a}` and\n`ScopeAgent{b}` are different storage keys.\n\n`ScopeGlobal` is never model-writable. Writing it is an operator action; nothing\nproduces one yet, but the contract models it because the read set includes it.\n\n### 2. CRUD, with revisions retained\n\n```go\ntype Store interface {\n Create(ctx context.Context, in NewRecord) (Record, error)\n Get(ctx context.Context, scope Scope, id RecordID) (Record, error) // ErrNotFound\n Query(ctx context.Context, q Query) ([]Record, error) // heads only\n List(ctx context.Context, scope Scope) ([]Record, error) // heads only\n Update(ctx context.Context, in RecordUpdate) (Record, error) // ErrNotFound, ErrStaleRevision\n Forget(ctx context.Context, scope Scope, id RecordID) error // idempotent\n Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)\n}\n```\n\n`Get` and `Forget` take a scope. The current `Forget` scans every identity's\nstore to find a marker (`builtin_engine.go`) purely because the contract\ngave it nothing better.\n\n`Record` carries what `agent-system.md` requires be retained: `Scope`,\n`Kind`, `Content`, `Revision`, `Author`, `OriginUser`, `Source`, `CreatedAt`,\n`UpdatedAt`. `Metadata` is dropped — nothing sets it and the parser never\nrestores it, so it is a field that silently does not round-trip.\n\n`Update` supersedes rather than overwrites. The superseded state is appended to\nthe scope's `HISTORY.md` **before** the live block is replaced, so a crash\nbetween the two leaves an extra history entry and an unchanged record —\nrecoverable — never a record whose previous state was lost. `Update` carries no\n`Scope`: a record may never move between scopes, and the API makes that\nunrepresentable rather than validating it. `Expected` optionally names the\nrevision being replaced and returns `ErrStaleRevision` on mismatch, which matters\nbecause two processes (`archied` and `archie-gateway`) can hold the same scope's\nfile.\n\n`Forget` records a deleted revision before removing the live block, so a record's\nprovenance outlives its content.\n\n### 3. The caller resolves identity; the engine never does\n\nResolution happens in the gateway turn path, where the native sender id and the\nsession are both in hand, and it is supplied **per channel by the composition**:\n\n```go\n// TurnRunnerConfig\n// UserIdentity resolves the initiating user's identity for one inbound message.\n// false means the channel carries no per-person identity, and the turn gets\n// global and agent scopes only. Nil means the same.\nUserIdentity func(msg messaging.Message) (IdentityID, bool)\n```\n\nThis must be injectable rather than a blanket `channel + \":\" + SenderID`\nconcatenation, because a channel's sender id is not necessarily a person: a\nwebhook's route path names a source, and treating it as an identity would give\na URL one -- which is why `webhook.go` leaves `SenderID` empty and carries the\nroute in `Inbound.BudgetKey` instead. Telegram and email supply their sender\nid; the dashboard and webhook resolve nothing.\n\n**Fail closed.** When `UserID` is empty the turn proceeds with global and agent\nscopes only, and a write targeting `user` or `agent-user` fails with a clear\nerror. `agent-system.md` forbids showing one user's memory to another, so\n\"we do not know who this is\" must mean \"show none of it\", never \"fall back to a\nwider scope\".\n\n### 4. The chat read path\n\nSynchronous and total. It runs inside `prepareTurn` (`turn.go`) before the\nprompt is built, never returns an error to the caller, and degrades to an empty\nblock on any engine failure or provider panic. A fire-and-forget hook would race\nthe prompt build in the same turn, which is why this is not a lifecycle hook.\n\nThe read is a scope-only `Query` over the subject's scopes — **not** ranked\nsearch, which remains open in `migration-decisions.md#5`. A ranked read on the\nhot path with no ranking implementation would be a lie.\n\n`SystemPromptConfig` gains a `Memory` field, rendered into\n`templates/archie.md.tpl` between the `\u003ctools\u003e` and `\u003cenv\u003e` blocks:\n\n```\n\u003cmemory purpose=\"durable_context\" trust=\"data\"\u003e\n```\n\n`trust=\"data\"` and inclusion in the precedence clause follow the `\u003ctools\u003e`/`\u003cenv\u003e`\nprecedent exactly: memory content is model-written from untrusted input and must\nbe framed as data, never instructions. The rendered block carries each record's\nshort id, because the write path addresses records by id — one source of truth\nfor both directions.\n\nBoth a per-scope record limit and a byte cap on the rendered block are named\nconstants. The cap is about size, not latency; a local read needs no timeout.\n\n**Prerequisite:** `setupGateways` runs before `setupMemoryAll`\n(`main.go` vs `:428`), and `startGatewayRuntime` before `setupMemory`\n(`gateway.go` vs `:115`), so the engine is nil at every turn-runner\nconstruction site. The ordering is the root-cause fix and nothing in either\nmemory step reads what those later steps produce.\n\n### 5. The chat write path\n\nThe memory tool is built **per turn**, not registered once at boot, because it\nmust write into a scope derived from the turn's resolved subject:\n\n| action | engine call |\n|---|---|\n| `create` | `Create` |\n| `update` | `Update` (by id) |\n| `delete` | `Forget` (by id) |\n| `list` | `List` |\n\n**The model chooses the scope kind; it never chooses the ids.** Agent and user\nids come from the turn's resolution and there is no schema field to put anything\nelse in — even a model that emits another user's id has nowhere to write it. That\nis the root-cause form of `agent-system.md`, matching this system's other\nnon-waivable boundaries (the model never runs git; gates cannot be waived by the\nagent). `global` is not offered at all.\n\nThe tool takes a record **id** where the retired provider took a content\nsubstring (`handleReplace`/`handleRemove`). The substring addressing existed\nbecause a markdown file was hand-edited and the model had no stable handle on a\nblock; with ids rendered into the prompt it disappears, and one addressing scheme\nreplaces two.\n\n### 6. The curator's write\n\n`sessioncurator.reviewOne` stops writing `Identity: sessionID` and writes\n`ScopeAgentUser{AgentID: \u003csession's agent\u003e, UserID: \u003cparticipant\u003e}`. Relationship\nscope is the only defensible home: agent-wide is explicitly unsafe for user facts\nand would leak user A's extracted facts to user B through the same agent;\nuser-wide would pool facts across agents; and a session id cannot survive the\nsession ending, which is the whole requirement.\n\nThis needs data the tree does not persist: the SQLite `messages` table\ngains `sender_id`, and `curator.SessionSummary` / `ConversationMessage` gain the\nagent and participant. The participant is derived from the session's own\nuser-role messages: exactly one distinct non-empty sender writes; zero (a\ndashboard or webhook session) or several (a group chat) writes nothing and\nrecords why. Ambiguity fails closed rather than guessing.\n\nThe payoff is the point of the exercise: chat-written and curator-derived memory\nnow occupy the same scopes, so a future turn's prompt contains both, and a\ncurator's bad note becomes id-addressable and therefore correctable by the agent.\n\n### 7. The content scanner moves to the engine\n\n`internal/memory/scanner.go` and its installation at `bootstrap.go` are what\nmake `configuration.md`'s \"Memory safety scanner\" row describe a control that\nactually runs. Deleting the package would silently remove prompt-injection\nscanning on memory writes. The scanner moves into the engine and is applied in\n`Create` and `Update`, which is the single choke point every producer crosses —\nso the curator's model-extracted content is covered too, not just the chat tool's.\n\n### 8. `internal/memory` is deleted\n\nNot shimmed, not partially, not behind a flag: `manager.go`, `provider.go`,\n`lifecycle.go`, `sync.go`, `scanner.go` (moved), `schema.go`, `backup.go`,\n`config.go`, `builtin/provider.go`, and their tests. Its markdown store\n(`builtin/store.go`) is the one survivor, relocated to\n`internal/infrastructure/memory/builtin/` — `organisation.md`'s \"a package owns\nits on-disk format end-to-end\" makes infrastructure its correct home, and the\nengine already parses that format.\n\nAlso deleted: `internal/tools/provider/memory`'s manager wrapping (rewired to the\nengine), `config.Memory.{Provider,ProviderConfig}` and the\n`validateMemory` rejection that exists only to complain about `Provider`\n(`validate.go`), and `webui.MemoryStatus`/`MemoryProviderHandle` — never\nimplemented in production, so `/api/memory` reports empty forever unless it is\nrepointed at the engine registry.\n\nExisting data at `\u003cworkDir\u003e/memory` is left on disk, unreferenced. No migration.\n`\u003cworkDir\u003e/memory-engine` is likewise orphaned: its directories were keyed by\n`sha256(sessionID)` and no scope key can hash to one.\n\n## What we deliberately do NOT do\n\n- **No cross-scope sharing or grants in this wave.** Promoting a record between\n scopes is a future explicit command, and `Update` cannot change scope, so it\n cannot happen by accident.\n- **No canonical user identity.** `SenderID` is channel-native, so one person on\n Telegram and email is two identities. Unifying needs an Identity aggregate that\n does not exist (`internal/domain` has no identity package and `IdentityID`\n appears nowhere in Go). The contract does not change when it lands.\n- **No ranked retrieval.** `Query.Text` stays, unused by the prompt path.\n- **No data migration**, per the decision above.\n- **No prompt-memory timeout mechanism.** The read is bounded by construction.\n- **No `Metadata` field.** It is write-only; a caller needing it later\n needs one field, not a speculative contract.\n\n## Acceptance criteria\n\n1. A record created through the memory tool in turn 1 appears in turn 2's prompt,\n asserted on content the test planted, with its id.\n2. The same record is **absent** from a different user's prompt on the same\n channel and agent. Isolation asserted end to end, not at the engine boundary.\n3. An agent-scope record is visible to both of that agent's users; a global\n record is visible to a second agent; an agent-user record is not.\n4. A curator pass over a session writes a record a later turn reads, and which\n survives that session being deleted.\n5. `create` → prompt shows it; `update` → prompt shows new content and\n `Revisions` returns the prior content with author and time; `delete` → gone\n from prompt and `List`. Round-trip is asserted as `parse(render(r)) == r`,\n never against a fixture string.\n6. A panicking engine and a failing engine both complete the turn with no memory\n block.\n7. A channel with no resolvable user renders only global and agent scopes, and\n the tool refuses `user`/`agent-user` with a reason. A webhook route path never\n becomes a user identity.\n8. A write that would exceed the file cap fails loudly rather than silently\n dropping content.\n9. `internal/memory` is gone, `task check` is green, and no document describes a\n manager that no longer exists.\n\nEvery criterion asserts behaviour or planted content. None asserts on the text of\na hand-written file, and none pins that a method was called. In particular none\nasserts the composition calls setup in a given order: if the ordering were\nswapped back, criteria 1 and 4 would fail, and that is the assertion.\n\n## Sequencing\n\nEach step is independently verifiable and lands green on its own.\n\n1. **Persist the participant.** `sender_id` column on `messages`,\n insert/select/migration, `curator.SessionSummary.AgentID` and\n `ConversationMessage.SenderID`. Touches no memory contract and is shippable\n alone — nothing consumes it yet.\n2. **Scope, CRUD and revisions.** The `internal/domain/memory` contract, the\n builtin engine with `HISTORY.md`, the store relocation, the scanner move, and\n the curator rewritten onto agent-user scope (it is the engine's only live\n consumer, so it belongs in this slice). *Depends on 1 for the curator half.*\n3. **Chat read path.** The `Memory` seam, the template block, subject resolution,\n and the composition-ordering fix. *Depends on 2.*\n4. **Chat write path.** The tool adapter onto CRUD and the turn's subject on the\n context. *Depends on 3, because a write is only observable through the read.*\n5. **Delete `internal/memory` and the dead config/dashboard surface**, and\n rewrite the \"Memory engine family\" section of\n `docs/architecture/plugins-and-extensions.md` to describe the\n engine family rather than a manager it deleted. *Depends on 3 and 4.*\n\nSteps 3 and 4 are separable from each other; step 1 is separable from everything.\n\n## Settles these open items in `migration-decisions.md` §5\n\n| §5 open item | Settled by |\n|---|---|\n| authoritative memory record and revision model | §1, §2 — the live marker becomes JSON, since the ratified `\u003c!--mem:\u003culid\u003e--\u003e` cannot carry provenance; superseded states live in a per-scope `HISTORY.md` |\n| provider and infrastructure boundaries | §2 — contract in `internal/domain/memory`, builtin format in `internal/infrastructure/memory/builtin` |\n| retrieval and access enforcement | §1 — caller-named scope sets, no engine-side policy |\n| provenance representation | §2 |\n\n## Not determined\n\n- **Group chats produce no derived memory**, because the curator fails closed on\n mixed-sender sessions. Whether extraction should instead become\n per-participant is a product call.\n- **Two processes, one scope file.** After step 3 both `archied` and\n `archie-gateway` hold an engine over the same root, and the engine caches a\n store per scope and rewrites the whole file, so concurrent writers on one scope\n can lose a block. `Update.Expected` makes that detectable for one record; it\n does not make the file safe. A reload-before-mutate or advisory lock is the fix\n and is not designed here.\n- **Unbounded history.** `HISTORY.md` grows without limit; no compaction policy.\n- **Size bounds.** `defaultMaxFileBytes` is now per scope rather than per\n identity; whether that is right is unexamined.\n- **Whether the dashboard should carry a user identity.** It has one bearer\n token, not users. Until it has users, web turns are correctly limited to global\n and agent scope.\n" }, { "url": "/dev/prds/messaging-service-boundary/", @@ -364,7 +364,7 @@ "url": "/dev/prds/service-decomposition/", "title": "Service decomposition -- decision", "section": "PRDs", - "body": "# Service decomposition -- decision\n\n**Status:** Draft\n**Date:** 2026-09-05\n**Beads epic:** archie-core-8cda\n**Prior art:** `docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md`,\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`,\n`docs/inspiration/service-discovery-research-2026-09-05.md`,\n`/work/artefacts/archie-core-current-state-archify.html`,\n`/work/artefacts/archie-core-ideal-state-archify.html`\n\nThe Phase 3 UI boundary and configuration decision is ratified in\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/). That focused document is\nthe implementation authority for `archie-core-8cda.5.2`; this parent remains\nthe broader service-distribution proposal.\n\n## Decision\n\narchie-core moves from one Go binary (`archied`) hosting every domain\nin-process to a set of independently deployable services, each exposing\nexactly one typed contract. Consumers depend on the contract; they never\nreach into another service's internal types, session state, or config.\ngRPC is the synchronous transport; NATS remains for asynchronous fan-out\n(status, telemetry) rather than being replaced.\n\nThis is a **distribution decision**, not a rewrite of Archie's domain\nmodel. The six-domain split already approved in\n`docs/architecture/index.md` and `docs/architecture/organisation.md`\n(Identity, Agent, Messaging, Workflow, Work Intake, Plugin) is unchanged;\nthis PRD is about how the *processes* running that domain logic get\npackaged, contracted, and discovered -- not about redrawing domain\nboundaries.\n\n### Target services\n\n| Service | Contract it owns | Existing code it replaces/wraps |\n|---|---|---|\n| UI Service | thin client only; talks to Gateway and State Store contracts | `internal/webui` (SPA hosting stays; API surface becomes typed contract clients, not a peer with direct access) |\n| Gateway Service | routing / turn / session contract | `internal/gateway` |\n| Messaging Service | channel-plugin contract (Telegram, email, Discord, Teams, ...) | `internal/channels` |\n| Curator Service | curator contract; **optional**, installs as a Gateway plugin | `internal/domain/curator` |\n| Scheduler Service | cron/ticker contract | `internal/domain/scheduling` |\n| Execution Environment | execution-dispatch contract | `internal/domain/workflow`, task dispatch paths |\n| Runner | isolated worktree/container execution | `internal/container`, `internal/agentexec`, `internal/worktree` |\n| State Store | state-access contract | `internal/store` |\n\n### Discovery\n\nPer `docs/inspiration/service-discovery-research-2026-09-05.md`:\n\n- **Primary:** Kubernetes-native discovery (Service + headless-Service/DNS\n + EndpointSlice, served by CoreDNS), delivered as a Helm chart + Operator,\n on k3s for single-host installs and a real cluster for multi-host or a\n hosted offering.\n- **Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV\n registry) for no-K8s single-host installs, with a custom gRPC resolver\n reading `host:port` from the KV bucket.\n- Both implementations sit behind one internal interface:\n\n ```go\n type ServiceRegistry interface {\n Resolve(service string) []Endpoint\n Watch(service string) \u003c-chan Event // join/leave\n }\n ```\n\n- An optional service (Curator; others later) that is not installed\n must resolve to a distinct `NotInstalled` outcome, never a `Down`/\n `Unhealthy` one. Under K8s this falls out of DNS returning no record;\n under the NATS fallback it requires an explicit \"installed\" marker in the\n KV registry, not just absence-implies-down.\n\n### Distribution model this supports\n\nBoth self-hosted (customer runs the Helm chart or the NATS-fallback\nbinaries themselves) and a managed/hosted offering (we run the same Helm\nchart) are in scope -- this is the RHEL / Ansible Automation Platform\nmodel: one architecture, sold both ways. This PRD does not decide the\nbusiness model; it decides that the architecture must not foreclose\neither shape.\n\n## Why this shape\n\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`\nestablished that Archie's existing domain split already does what AWX's\nservice-decomposition post argues for -- boundaries exist. What didn't\nexist was deployability: `internal/webui` was not a passive frontend. The\nre-baselined coupling inventory is below, and the ratified boundary is\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\n\n- Telegram task actions now call Gateway's `ChatContract` directly through\n `chatTaskActorAdapter`; the old Telegram-to-webui HTTP detour is historical.\n- `internal/webui/api_chat.go` now holds `gateway.ChatContract`; direct\n `SessionStore`, `Router`, and `TurnHistory` access is historical.\n- The third coupling is closed. UI extraction shipped (`archie-core-8cda.5`,\n closed 2026-09-11): `cmd/archie-ui` serves the dashboard against remote\n Gateway and State Store contracts, holds no `config.Holder`, and relinks no\n daemon runtime (`cmd/archie-ui/architecture_test.go` enforces that on every\n `task check`). The `b.web.Cfg = b.d.Cfg` alias left in `buildDaemon` is now\n inside the daemon's own process -- it renders the published `ConfigView`\n snapshot and holds the readiness probes -- so it no longer crosses a service\n boundary. That cleanup is `archie-core-ml30` (P2); the UI boundary is\n ratified in [`ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\n\n## Non-goals\n\n- Redesigning the domain model (Identity/Agent/Messaging/Workflow/Work\n Intake/Plugin boundaries are settled elsewhere and unchanged).\n- Replacing NATS. It remains the async transport; this PRD only adds gRPC\n for synchronous request/response and (optionally) K8s-native discovery.\n- Deciding the business model (self-hosted vs. managed vs. both). Kept\n open deliberately; the architecture must serve either.\n- Config relocation (`config.toml` -\u003e ConfigMaps/Secrets) is real work\n implied by a K8s-native deployment but is its own decision, not decided\n here. Flagged in the discovery research as a phased concern.\n- Any concrete adoption of Pi (or another external agent harness) as an\n execution runner. Raised in conversation alongside this decomposition,\n but it is a separate decision (agent execution as a strict data\n boundary, `ARCHITECTURE.md`) and does not gate or get gated by this PRD.\n\n## Open questions\n\nAll ten are resolved; each carries its resolution inline. Three phases have\nsince shipped against them (Gateway, State Store, UI), so the framing here is\nhistorical record rather than an outstanding gate.\n\n1. **Extraction order.** Which service gets pulled out of the monolith\n first? The webui entanglement findings above suggest Gateway Service's\n `ChatContract` should exist before UI Service is extracted, since UI is\n the most entangled consumer. Not yet sequenced.\n\n **RESOLVED.** Gateway Service first, then State Store,\n then UI, then Messaging, then Execution/Runner/Scheduler, then Curator\n (optional) — preceded by an in-process \"contract seams\" (modular\n monolith) phase. Confirms the hypothesis and strengthens it: the\n Gateway is already store-decoupled (it uses narrow contract adapters,\n not `internal/store`) and already owns an isolated session SQLite, so it\n is both the stickiest capability *and* the one with a ready seam and\n isolated data — extractable alone, first. Full phase list and reasoning:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#1`.\n2. **Contract definition mechanism.** Protobuf/gRPC service definitions\n need a home (a new `proto/` or `contracts/` tree, versioning scheme,\n codegen wiring into `Taskfile.yml`). Not yet designed.\n\n **RESOLVED.** buf v2 driving `protoc-gen-go` +\n `protoc-gen-go-grpc` (grpc-go, not connect-go — connect-go's net/http\n client breaks the discovery research's gRPC resolver plumbing for K8s\n DNS and NATS-KV). One committed `proto/\u003cservice\u003e/v1/` tree, generated\n Go committed to `internal/contracts/\u003cservice\u003e/v1/`, directory\n versioning (breaking change -\u003e new `vN`, enforced by `buf breaking` in\n the gate), and `task proto:generate`/`proto:lint`/`proto:check` wired\n into `task check` and mirrored into `.github/workflows/deploy.yml`\n (CI does not run `task check`). Full toolchain comparison, layout\n rationale, and scratch-verified `buf`/`buf breaking`/`connect-go`\n results:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#2`.\n3. **In-process vs. out-of-process during migration.** Whether services\n can run multiplexed inside one binary behind the same `ServiceRegistry`\n interface during a transition period (so extraction is incremental and\n testable) or whether each service must be a separate binary from day\n one. Strongly recommend the former for a solo maintainer, but not\n decided here.\n\n **RESOLVED.** Multiplexed in-process is the migration\n default -- this is `docs/architecture/organisation.md`'s standing\n \"Process boundaries\" rule, not a new lean. Mechanism: one wire-safe Go\n contract interface per service; a local adapter and a generated gRPC\n client adapter, both satisfying the same interface via a compile-time\n assertion (the `var _ workflow.Store = (*Client)(nil)` idiom;\n `store.WorkflowStore` is superseded by the consumer-owned\n `workflow.Store`); composition (not `ServiceRegistry`, which stays\n network-only) picks the adapter by the presence of a\n `[services.\u003cname\u003e].target` address -- empty = local (in-process) adapter,\n set = dial the remote gRPC client. This replaced the earlier\n `mode = \"inproc\" | \"remote\"` enum, which was dropped (see the gateway's\n `services.gateway.target`). The adapter is chosen by presence of the\n target; Phase 2 then deleted every local fallback, so remote is now the\n only mode and an empty `[services.state].target` is a startup error\n (docs/prds/state-store-contract.md §12 step 7). A\n per-contract conformance suite runs against both adapters over\n `bufconn`; the flip to a real process is a same-commit deletion of the\n in-process path, no dual-live window. Consequence: Gateway's Phase 0\n (`ChatContract` seam) ships in-process first, with the network protocol\n staying out of the production path until Phase 1 -- no amendment to Q1's\n phase list needed. Full mechanism and evidence:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#3`\n (the State Store's concrete ratified config shape is\n `docs/prds/state-store-contract.md` §10).\n4. **State Store's contract shape.** Whether every service gets its own\n storage (true microservice isolation) or several services share the\n State Store service's contract (less isolation, less migration risk).\n Not yet decided.\n\n **RESOLVED.** One State Store service owns the existing\n single SQLite file behind narrow typed contracts (`TaskStore`,\n `CaptureStore`, `MappingStore`, `BindingStore`, `WorkflowStore`) over\n gRPC. *(Ratified 2026-09-06: the workflow contract is now the\n domain-owned `workflow.Store`, which supersedes\n `internal/store`'s `WorkflowStore`. See note below and\n `docs/prds/state-store-contract.md`.)*\n No physical database-per-service: the solo-maintainer / team-\n topology rule of thumb plus the low value of per-service autonomy for\n one person rule it out, and one store means no distributed-transactions\n problem. The Gateway keeps its own already-separate session SQLite.\n Concrete migration path (generalise `storerpc`, swap consumers one\n contract at a time, stand up the store service, split the file only on\n evidence):\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#4`.\n\n **RATIFIED (2026-09-06, beads archie-core-8cda.4.1).** The authoritative\n contract boundary \u0026 transport design for this is fixed in\n `docs/prds/state-store-contract.md` (rev. 2c): one `StateStore` gRPC service,\n 42 RPCs; **ownership split** — consumer-owned `workflow.Store` +\n `workflow.Task`/`Status`/`Source` in the workflow domain (dependency rules\n #2/#7), producer-owned daemon/webui store surfaces in `internal/store`;\n the `Task`/`Status`/`Source` + interface relocation is a Phase 2\n prerequisite pulled from migration-decisions §4 (`store.Task →\n WorkflowExecution`); `RecordDispatch` drops `*sql.Tx`; `Close` off-wire;\n structured gRPC error codes with sanitised details; explicit agent State\n Store gRPC handoff (`STATE_STORE_URL`/`STATE_STORE_TOKEN`); loopback+bearer\n transport-security boundary; presence-based `[services.state].target`\n defaulting local; migration order `workflow.Store -\u003e archie-state-store -\u003e\n Capture -\u003e Mapping -\u003e Binding -\u003e TaskStore`. `.4.2` and `.4.3` implement\n against that doc.\n5. **Helm chart / Operator ownership.** Net-new work with no existing\n analog in this repository (`deployments/` holds TOML profiles\n and a `docker-compose.yml`, not Kubernetes manifests). Scoping not yet\n started.\n\n **RESOLVED.** Three charts: an `archie-library` chart\n (shared Deployment/Service templates, named ports), an `archie` chart\n for the 7 core services, and an `archie-curator` chart + Curator\n Operator (CRD, controller, RBAC). The Curator reconciler is a\n level-triggered loop that fetches the CR, builds Deployment + Service,\n sets owner refs, applies idempotently (`CreateOrUpdate`), watches via\n `Owns()`, and reports a `Ready` status condition. Smallest real\n milestone (M1): install a Curator CR → Operator creates Deployment +\n Service → CoreDNS serves `curator` → Gateway's resolver flips it from\n `NotInstalled` to Installed with no redeploy; uninstall flips it back.\n Full scoping and first bead-issue list:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#5`.\n\n## Completion criteria\n\nNot applicable yet -- this PRD records the target shape and the discovery\ndecision. It does not authorize implementation. Before work begins: file\na beads epic, resolve the open questions above (at minimum #1-#3), and\nget explicit go-ahead given the scale of this change relative to\n`AGENTS.md`'s \"smallest workable change\" default for this solo project.\n" + "body": "# Service decomposition -- decision\n\n**Status:** Draft\n**Date:** 2026-09-05\n**Beads epic:** archie-core-8cda\n**Prior art:** `docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md`,\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`,\n`docs/inspiration/service-discovery-research-2026-09-05.md`,\n`/work/artefacts/archie-core-current-state-archify.html`,\n`/work/artefacts/archie-core-ideal-state-archify.html`\n\nThe Phase 3 UI boundary and configuration decision is ratified in\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/). That focused document is\nthe implementation authority for `archie-core-8cda.5.2`; this parent remains\nthe broader service-distribution proposal.\n\n## Decision\n\narchie-core moves from one Go binary (`archied`) hosting every domain\nin-process to a set of independently deployable services, each exposing\nexactly one typed contract. Consumers depend on the contract; they never\nreach into another service's internal types, session state, or config.\ngRPC is the synchronous transport; NATS remains for asynchronous fan-out\n(status, telemetry) rather than being replaced.\n\nThis is a **distribution decision**, not a rewrite of Archie's domain\nmodel. The six-domain split already approved in\n`docs/architecture/index.md` and `docs/architecture/organisation.md`\n(Identity, Agent, Messaging, Workflow, Work Intake, Plugin) is unchanged;\nthis PRD is about how the *processes* running that domain logic get\npackaged, contracted, and discovered -- not about redrawing domain\nboundaries.\n\n### Target services\n\n| Service | Contract it owns | Existing code it replaces/wraps |\n|---|---|---|\n| UI Service | thin client only; talks to Gateway and State Store contracts | `internal/webui` (SPA hosting stays; API surface becomes typed contract clients, not a peer with direct access) |\n| Gateway Service | routing / turn / session contract | `internal/gateway` |\n| Messaging Service | channel-plugin contract (Telegram, email, Discord, Teams, ...) | `internal/channels` |\n| Curator Service | curator contract; **optional**, installs as a Gateway plugin | `internal/domain/curator` |\n| Scheduler Service | cron/ticker contract | `internal/domain/scheduling` |\n| Execution Environment | execution-dispatch contract | `internal/domain/workflow`, task dispatch paths |\n| Runner | isolated worktree/container execution | `internal/container`, `internal/agentexec`, `internal/worktree` |\n| State Store | state-access contract | `internal/store` |\n\n### Discovery\n\nPer `docs/inspiration/service-discovery-research-2026-09-05.md`:\n\n- **Primary:** Kubernetes-native discovery (Service + headless-Service/DNS\n + EndpointSlice, served by CoreDNS), delivered as a Helm chart + Operator,\n on k3s for single-host installs and a real cluster for multi-host or a\n hosted offering.\n- **Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV\n registry) for no-K8s single-host installs, with a custom gRPC resolver\n reading `host:port` from the KV bucket.\n- Both implementations sit behind one internal interface:\n\n ```go\n type ServiceRegistry interface {\n Resolve(service string) []Endpoint\n Watch(service string) \u003c-chan Event // join/leave\n }\n ```\n\n- An optional service (Curator; others later) that is not installed\n must resolve to a distinct `NotInstalled` outcome, never a `Down`/\n `Unhealthy` one. Under K8s this falls out of DNS returning no record;\n under the NATS fallback it requires an explicit \"installed\" marker in the\n KV registry, not just absence-implies-down.\n\n### Distribution model this supports\n\nBoth self-hosted (customer runs the Helm chart or the NATS-fallback\nbinaries themselves) and a managed/hosted offering (we run the same Helm\nchart) are in scope -- this is the RHEL / Ansible Automation Platform\nmodel: one architecture, sold both ways. This PRD does not decide the\nbusiness model; it decides that the architecture must not foreclose\neither shape.\n\n## Why this shape\n\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`\nestablished that Archie's existing domain split already does what AWX's\nservice-decomposition post argues for -- boundaries exist. What didn't\nexist was deployability: `internal/webui` was not a passive frontend. The\nre-baselined coupling inventory is below, and the ratified boundary is\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\n\n- Telegram task actions now call Gateway's `ChatContract` directly through\n `chatTaskActorAdapter`; the old Telegram-to-webui HTTP detour is historical.\n- `internal/webui/api_chat.go` now holds `gateway.ChatContract`; direct\n `SessionStore`, `Router`, and `TurnHistory` access is historical.\n- The third coupling is closed. UI extraction shipped (`archie-core-8cda.5`,\n closed 2026-09-11): `cmd/archie-ui` serves the dashboard against remote\n Gateway and State Store contracts, holds no `config.Holder`, and relinks no\n daemon runtime (`cmd/archie-ui/architecture_test.go` enforces that on every\n `task check`). The `b.web.Cfg = b.d.Cfg` alias left in `buildDaemon` is now\n inside the daemon's own process -- it renders the published `ConfigView`\n snapshot and holds the readiness probes -- so it no longer crosses a service\n boundary. That cleanup is `archie-core-ml30` (P2); the UI boundary is\n ratified in [`ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\n\n## Non-goals\n\n- Redesigning the domain model (Identity/Agent/Messaging/Workflow/Work\n Intake/Plugin boundaries are settled elsewhere and unchanged).\n- Replacing NATS. It remains the async transport; this PRD only adds gRPC\n for synchronous request/response and (optionally) K8s-native discovery.\n- Deciding the business model (self-hosted vs. managed vs. both). Kept\n open deliberately; the architecture must serve either.\n- Config relocation (`config.toml` -\u003e ConfigMaps/Secrets) is real work\n implied by a K8s-native deployment but is its own decision, not decided\n here. Flagged in the discovery research as a phased concern.\n- Any concrete adoption of Pi (or another external agent harness) as an\n execution runner. Raised in conversation alongside this decomposition,\n but it is a separate decision (agent execution as a strict data\n boundary, `docs/architecture/agent-system.md`) and does not gate or get gated by this PRD.\n\n## Open questions\n\nAll ten are resolved; each carries its resolution inline. Three phases have\nsince shipped against them (Gateway, State Store, UI), so the framing here is\nhistorical record rather than an outstanding gate.\n\n1. **Extraction order.** Which service gets pulled out of the monolith\n first? The webui entanglement findings above suggest Gateway Service's\n `ChatContract` should exist before UI Service is extracted, since UI is\n the most entangled consumer. Not yet sequenced.\n\n **RESOLVED.** Gateway Service first, then State Store,\n then UI, then Messaging, then Execution/Runner/Scheduler, then Curator\n (optional) — preceded by an in-process \"contract seams\" (modular\n monolith) phase. Confirms the hypothesis and strengthens it: the\n Gateway is already store-decoupled (it uses narrow contract adapters,\n not `internal/store`) and already owns an isolated session SQLite, so it\n is both the stickiest capability *and* the one with a ready seam and\n isolated data — extractable alone, first. Full phase list and reasoning:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#1`.\n2. **Contract definition mechanism.** Protobuf/gRPC service definitions\n need a home (a new `proto/` or `contracts/` tree, versioning scheme,\n codegen wiring into `Taskfile.yml`). Not yet designed.\n\n **RESOLVED.** buf v2 driving `protoc-gen-go` +\n `protoc-gen-go-grpc` (grpc-go, not connect-go — connect-go's net/http\n client breaks the discovery research's gRPC resolver plumbing for K8s\n DNS and NATS-KV). One committed `proto/\u003cservice\u003e/v1/` tree, generated\n Go committed to `internal/contracts/\u003cservice\u003e/v1/`, directory\n versioning (breaking change -\u003e new `vN`, enforced by `buf breaking` in\n the gate), and `task proto:generate`/`proto:lint`/`proto:check` wired\n into `task check` and mirrored into `.github/workflows/deploy.yml`\n (CI does not run `task check`). Full toolchain comparison, layout\n rationale, and scratch-verified `buf`/`buf breaking`/`connect-go`\n results:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#2`.\n3. **In-process vs. out-of-process during migration.** Whether services\n can run multiplexed inside one binary behind the same `ServiceRegistry`\n interface during a transition period (so extraction is incremental and\n testable) or whether each service must be a separate binary from day\n one. Strongly recommend the former for a solo maintainer, but not\n decided here.\n\n **RESOLVED.** Multiplexed in-process is the migration\n default -- this is `docs/architecture/organisation.md`'s standing\n \"Process boundaries\" rule, not a new lean. Mechanism: one wire-safe Go\n contract interface per service; a local adapter and a generated gRPC\n client adapter, both satisfying the same interface via a compile-time\n assertion (the `var _ workflow.Store = (*Client)(nil)` idiom;\n `store.WorkflowStore` is superseded by the consumer-owned\n `workflow.Store`); composition (not `ServiceRegistry`, which stays\n network-only) picks the adapter by the presence of a\n `[services.\u003cname\u003e].target` address -- empty = local (in-process) adapter,\n set = dial the remote gRPC client. This replaced the earlier\n `mode = \"inproc\" | \"remote\"` enum, which was dropped (see the gateway's\n `services.gateway.target`). The adapter is chosen by presence of the\n target; Phase 2 then deleted every local fallback, so remote is now the\n only mode and an empty `[services.state].target` is a startup error\n (docs/prds/state-store-contract.md §12 step 7). A\n per-contract conformance suite runs against both adapters over\n `bufconn`; the flip to a real process is a same-commit deletion of the\n in-process path, no dual-live window. Consequence: Gateway's Phase 0\n (`ChatContract` seam) ships in-process first, with the network protocol\n staying out of the production path until Phase 1 -- no amendment to Q1's\n phase list needed. Full mechanism and evidence:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#3`\n (the State Store's concrete ratified config shape is\n `docs/prds/state-store-contract.md` §10).\n4. **State Store's contract shape.** Whether every service gets its own\n storage (true microservice isolation) or several services share the\n State Store service's contract (less isolation, less migration risk).\n Not yet decided.\n\n **RESOLVED.** One State Store service owns the existing\n single SQLite file behind narrow typed contracts (`TaskStore`,\n `CaptureStore`, `MappingStore`, `BindingStore`, `WorkflowStore`) over\n gRPC. *(Ratified 2026-09-06: the workflow contract is now the\n domain-owned `workflow.Store`, which supersedes\n `internal/store`'s `WorkflowStore`. See note below and\n `docs/prds/state-store-contract.md`.)*\n No physical database-per-service: the solo-maintainer / team-\n topology rule of thumb plus the low value of per-service autonomy for\n one person rule it out, and one store means no distributed-transactions\n problem. The Gateway keeps its own already-separate session SQLite.\n Concrete migration path (generalise `storerpc`, swap consumers one\n contract at a time, stand up the store service, split the file only on\n evidence):\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#4`.\n\n **RATIFIED (2026-09-06, beads archie-core-8cda.4.1).** The authoritative\n contract boundary \u0026 transport design for this is fixed in\n `docs/prds/state-store-contract.md` (rev. 2c): one `StateStore` gRPC service,\n 42 RPCs; **ownership split** — consumer-owned `workflow.Store` +\n `workflow.Task`/`Status`/`Source` in the workflow domain (dependency rules\n #2/#7), producer-owned daemon/webui store surfaces in `internal/store`;\n the `Task`/`Status`/`Source` + interface relocation is a Phase 2\n prerequisite pulled from migration-decisions §4 (`store.Task →\n WorkflowExecution`); `RecordDispatch` drops `*sql.Tx`; `Close` off-wire;\n structured gRPC error codes with sanitised details; explicit agent State\n Store gRPC handoff (`STATE_STORE_URL`/`STATE_STORE_TOKEN`); loopback+bearer\n transport-security boundary; presence-based `[services.state].target`\n defaulting local; migration order `workflow.Store -\u003e archie-state-store -\u003e\n Capture -\u003e Mapping -\u003e Binding -\u003e TaskStore`. `.4.2` and `.4.3` implement\n against that doc.\n5. **Helm chart / Operator ownership.** Net-new work with no existing\n analog in this repository (`deployments/` holds TOML profiles\n and a `docker-compose.yml`, not Kubernetes manifests). Scoping not yet\n started.\n\n **RESOLVED.** Three charts: an `archie-library` chart\n (shared Deployment/Service templates, named ports), an `archie` chart\n for the 7 core services, and an `archie-curator` chart + Curator\n Operator (CRD, controller, RBAC). The Curator reconciler is a\n level-triggered loop that fetches the CR, builds Deployment + Service,\n sets owner refs, applies idempotently (`CreateOrUpdate`), watches via\n `Owns()`, and reports a `Ready` status condition. Smallest real\n milestone (M1): install a Curator CR → Operator creates Deployment +\n Service → CoreDNS serves `curator` → Gateway's resolver flips it from\n `NotInstalled` to Installed with no redeploy; uninstall flips it back.\n Full scoping and first bead-issue list:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#5`.\n\n## Completion criteria\n\nNot applicable yet -- this PRD records the target shape and the discovery\ndecision. It does not authorize implementation. Before work begins: file\na beads epic, resolve the open questions above (at minimum #1-#3), and\nget explicit go-ahead given the scale of this change relative to\n`AGENTS.md`'s \"smallest workable change\" default for this solo project.\n" }, { "url": "/dev/prds/service-registry/", @@ -424,7 +424,7 @@ "url": "/dev/prds/webhook-intake-security/", "title": "Webhook intake security -- decision", "section": "PRDs", - "body": "# Webhook intake security -- decision\n\n**Status:** Approved\n**Date:** 2026-08-22\n**Beads issue:** `archie-core-t2db.5`, blocks `archie-core-t2db.4`\n\n## Why this is decided ahead of the code it constrains\n\n`archie-core-t2db` (no-code playbooks) and `archie-core-7d5u` (typed event\nsources) both land a public intake surface whose eventual output is an agent\nwith tool access and a repo. Neither capture (`t2db.1`) nor the intake family\n(`7d5u.3`) has shipped yet, so this decision exists to be a contract the two\nepics build against, not a retrofit once the surface exists. A webhook URL\nthat can start an agent is remote code execution with extra steps if any of\nthe five points below is missing on day one.\n\n## 1. Per-source authentication\n\nReuse the per-route HMAC-SHA256 scheme already in\n`internal/channels/webhook/webhook.go` (`validateHMAC`), unchanged: a source\nregisters with a shared secret; the signature is checked against the raw\nbody. Do not invent a second scheme for intake.\n\n**Capture and triggering split at the signature, not at a separate flag.** An\nevent with a missing or invalid signature is still captured (visible in the\ninspector, marked unauthenticated) so an operator can diagnose a\nmisconfigured sender -- but an unauthenticated event can never be matched\nagainst a binding. Authentication is what promotes an event from \"something\narrived\" to \"something that can run a workflow.\"\n\n## 2. Human approval before a binding arms\n\nA binding has three states: `draft -\u003e pending_approval -\u003e armed`. Only\n`armed` bindings are evaluated against incoming events. Creating or editing a\nbinding's matcher, mapping, or target workflow always sets it to\n`pending_approval`, including edits to an already-armed binding -- an edit is\nnot silently re-armed, or the approval gate is decorative. Approval requires\nan authenticated dashboard session; there is no unattended approval path.\n\nModel: `internal/channels/telegram`'s dangerous-command approval flow\n(`dangerousAction` / `pendingApproval` in `dangerous.go` / `approval.go`) is\nthe existing precedent for \"a human decision gates code from running\" in this\ncodebase -- follow its shape rather than designing a new approval primitive.\n\n## 3. Rate and volume limits, per source\n\nA token-bucket limiter keyed by source ID, not global -- one noisy or hostile\nsender must not exhaust another source's capture budget. Precedent for\nper-key volume capping: `internal/pairing/store.go`'s `MaxPendingPerPlatform`.\n\nOn limit exceeded, the request is rejected (HTTP 429), not silently dropped.\nA silently dropped event is indistinguishable from a sender that never fired\n-- the same invisible-failure shape the update-verification work\n(`archie-core-522`) exists to prevent. The rejection itself is observable in\nthe inspector as a distinct outcome from \"not received.\"\n\n## 4. Blast radius bounded by code, not by prompt wording\n\nA playbook-triggered task runs through exactly the same enforcement as every\nother task: the `[[repos.gate]]` quality gate, `diff_cap_lines`, the\nsandboxed per-task container (`internal/container`), worktree isolation, and\n\"the model never runs git.\" No new relaxation is introduced for\nplaybook-triggered work, and none is needed -- the existing constraints are\nthe control.\n\nMapped payload fields are treated as ordinary user-controlled text inside an\nagent prompt, exactly like an inbound chat message never as trusted\ninstructions. \"The agent should ignore instructions in the payload\" is not a\ncontrol and must not appear as one anywhere in this feature -- per\n`ARCHITECTURE.md`'s environmental-enforcement-over-prompt-rules principle,\nthe boundary is the gate and the sandbox, not a sentence in the system\nprompt.\n\n## 5. Retention and redaction of captured payloads\n\n**Retention:** captured events (bound or not) are purged after 7 days,\nconfigurable per deployment. Unbound events exist to be mapped against; nothing\njustifies keeping them indefinitely once that window has passed, and an\nunbounded capture store is the same disk-exhaustion risk item 3 addresses at\nthe request-rate level.\n\n**Redaction:** before a captured payload is persisted, apply the same\nkey-name heuristic `internal/gateway/stream.go`'s `sensitiveParameterKey`\nalready uses for tool-call parameters (matches `token`, `secret`, `password`,\n`passwd`, `api_key`, `apikey`, `authorization`, `credential`, `cookie`,\n`private_key` against normalized JSON keys) and replace matching values with\nthe same `[redacted]` marker. This is best-effort, not a guarantee -- a\nsender that names a secret field something the heuristic misses will still\nhave it captured. Document that limitation where captured payloads are\ndisplayed, rather than implying the redaction is complete.\n\n**Visibility:** captured payloads, redacted or not, are visible only to an\nauthenticated dashboard operator -- the same authorization boundary as every\nother webui surface, not a separate or lower bar.\n\n## What this does not decide\n\nThe intake family's registration interface (`7d5u.3`), the capture storage\nmechanism (`7d5u.2`, embedded NATS), and the binding data model's persistence\nand versioning (`t2db.4`) are separate decisions. This document constrains\nall three: whatever they build must satisfy points 1-5 above.\n" + "body": "# Webhook intake security -- decision\n\n**Status:** Approved\n**Date:** 2026-08-22\n**Beads issue:** `archie-core-t2db.5`, blocks `archie-core-t2db.4`\n\n## Why this is decided ahead of the code it constrains\n\n`archie-core-t2db` (no-code playbooks) and `archie-core-7d5u` (typed event\nsources) both land a public intake surface whose eventual output is an agent\nwith tool access and a repo. Neither capture (`t2db.1`) nor the intake family\n(`7d5u.3`) has shipped yet, so this decision exists to be a contract the two\nepics build against, not a retrofit once the surface exists. A webhook URL\nthat can start an agent is remote code execution with extra steps if any of\nthe five points below is missing on day one.\n\n## 1. Per-source authentication\n\nReuse the per-route HMAC-SHA256 scheme already in\n`internal/channels/webhook/webhook.go` (`validateHMAC`), unchanged: a source\nregisters with a shared secret; the signature is checked against the raw\nbody. Do not invent a second scheme for intake.\n\n**Capture and triggering split at the signature, not at a separate flag.** An\nevent with a missing or invalid signature is still captured (visible in the\ninspector, marked unauthenticated) so an operator can diagnose a\nmisconfigured sender -- but an unauthenticated event can never be matched\nagainst a binding. Authentication is what promotes an event from \"something\narrived\" to \"something that can run a workflow.\"\n\n## 2. Human approval before a binding arms\n\nA binding has three states: `draft -\u003e pending_approval -\u003e armed`. Only\n`armed` bindings are evaluated against incoming events. Creating or editing a\nbinding's matcher, mapping, or target workflow always sets it to\n`pending_approval`, including edits to an already-armed binding -- an edit is\nnot silently re-armed, or the approval gate is decorative. Approval requires\nan authenticated dashboard session; there is no unattended approval path.\n\nModel: `internal/channels/telegram`'s dangerous-command approval flow\n(`dangerousAction` / `pendingApproval` in `dangerous.go` / `approval.go`) is\nthe existing precedent for \"a human decision gates code from running\" in this\ncodebase -- follow its shape rather than designing a new approval primitive.\n\n## 3. Rate and volume limits, per source\n\nA token-bucket limiter keyed by source ID, not global -- one noisy or hostile\nsender must not exhaust another source's capture budget. Precedent for\nper-key volume capping: `internal/pairing/store.go`'s `MaxPendingPerPlatform`.\n\nOn limit exceeded, the request is rejected (HTTP 429), not silently dropped.\nA silently dropped event is indistinguishable from a sender that never fired\n-- the same invisible-failure shape the update-verification work\n(`archie-core-522`) exists to prevent. The rejection itself is observable in\nthe inspector as a distinct outcome from \"not received.\"\n\n## 4. Blast radius bounded by code, not by prompt wording\n\nA playbook-triggered task runs through exactly the same enforcement as every\nother task: the `[[repos.gate]]` quality gate, `diff_cap_lines`, the\nsandboxed per-task container (`internal/container`), worktree isolation, and\n\"the model never runs git.\" No new relaxation is introduced for\nplaybook-triggered work, and none is needed -- the existing constraints are\nthe control.\n\nMapped payload fields are treated as ordinary user-controlled text inside an\nagent prompt, exactly like an inbound chat message never as trusted\ninstructions. \"The agent should ignore instructions in the payload\" is not a\ncontrol and must not appear as one anywhere in this feature -- per the\nenvironmental-enforcement principle (`docs/architecture/agent-system.md`,\n\"Existing invariants to preserve\"), the boundary is the gate and the sandbox,\nnot a sentence in the system prompt.\n\n## 5. Retention and redaction of captured payloads\n\n**Retention:** captured events (bound or not) are purged after 7 days,\nconfigurable per deployment. Unbound events exist to be mapped against; nothing\njustifies keeping them indefinitely once that window has passed, and an\nunbounded capture store is the same disk-exhaustion risk item 3 addresses at\nthe request-rate level.\n\n**Redaction:** before a captured payload is persisted, apply the same\nkey-name heuristic `internal/gateway/stream.go`'s `sensitiveParameterKey`\nalready uses for tool-call parameters (matches `token`, `secret`, `password`,\n`passwd`, `api_key`, `apikey`, `authorization`, `credential`, `cookie`,\n`private_key` against normalized JSON keys) and replace matching values with\nthe same `[redacted]` marker. This is best-effort, not a guarantee -- a\nsender that names a secret field something the heuristic misses will still\nhave it captured. Document that limitation where captured payloads are\ndisplayed, rather than implying the redaction is complete.\n\n**Visibility:** captured payloads, redacted or not, are visible only to an\nauthenticated dashboard operator -- the same authorization boundary as every\nother webui surface, not a separate or lower bar.\n\n## What this does not decide\n\nThe intake family's registration interface (`7d5u.3`), the capture storage\nmechanism (`7d5u.2`, embedded NATS), and the binding data model's persistence\nand versioning (`t2db.4`) are separate decisions. This document constrains\nall three: whatever they build must satisfy points 1-5 above.\n" } ], "nav": [ diff --git a/docs/data/generated/docs.json b/docs/data/generated/docs.json index b56e5b667..809f54f47 100644 --- a/docs/data/generated/docs.json +++ b/docs/data/generated/docs.json @@ -10,7 +10,7 @@ "url": "/docs/guides/first-playbook/", "title": "Building your first playbook binding", "section": "Guides", - "body": "# Building your first playbook binding\n\nThis walks through the whole no-code path the event-capture epic promises:\npoint an external app's webhook at archie, see the real payload it sends,\nmap the fields you care about, bind them to a workflow, and watch the next\nmatching event turn into a task -- no code, no redeploy, no restart. See\n`docs/architecture/bindings.md` for the model and threat model behind each\nstep below.\n\n## 1. Send a test event\n\nPoint whatever external app you want archie to react to at:\n\n```\nPOST http://\u003cyour-archied-host\u003e:8484/webhooks/capture/\u003csource\u003e\n```\n\n`\u003csource\u003e` is a name you choose for this sender (e.g. `sentry`, `github-ci`,\n`stripe`) -- it's just a URL path segment, not a preset list. Send one real\nevent however is easiest for a first test, e.g.:\n\n```bash\ncurl -X POST http://localhost:8484/webhooks/capture/sentry \\\n -H \"Content-Type: application/json\" \\\n -d '{\"issue\": {\"title\": \"NullPointerException in checkout\"}, \"level\": \"error\"}'\n```\n\nThis event isn't signed yet, so it's captured but marked **unauthenticated**\n-- visible for you to design against, but incapable of triggering anything\n(`docs/architecture/bindings.md`'s threat-model section explains why: an\nunauthenticated event can never match a binding, no matter how it looks).\n\n## 2. Inspect the captured payload\n\nOpen the dashboard's **Event inspector** (`/captures`). Your test event\nappears there with its real body -- this is schema-by-example: you're\nlooking at an actual payload from the actual sender, not guessing at a\nschema in advance.\n\n## 3. Map the fields you care about\n\nOpen **Field mappings** (`/mappings`) → **New mapping**. Pick your captured\nevent from the dropdown, then click through its payload tree on the right --\nevery key and value is clickable. Clicking a value binds it as a field\n(name, JSON path, and an inferred type you can change). Add whichever fields\nyour workflow needs (e.g. the issue title), mark the ones that must be\npresent as **Required**, hit **Preview** to confirm they resolve correctly\nagainst the real payload, then **Save**.\n\n## 4. Create the binding\n\nOpen **Playbook bindings** (`/bindings`) → **New binding**:\n\n- **Name** -- anything descriptive.\n- **Source** -- the same `\u003csource\u003e` segment from step 1 (e.g. `sentry`).\n- **Field mapping** -- the one you just saved.\n- **Workflow** -- which registered workflow should run on a match.\n- **Repo pin** (optional) -- leave both blank to use the single configured\n repo; set owner/repo explicitly if you run a multi-repo deployment.\n- **Shared secret** -- a real secret (16+ bytes), never `test1234`. The\n sender signs every request body with this via HMAC-SHA256 so archie can\n tell a genuine event from anyone who finds the URL.\n\nSave. The binding starts in **draft** -- nothing fires yet.\n\n## 5. Approve it\n\nA **draft** or freshly-**edited** binding needs an explicit human approval\nbefore it can ever match a live event -- this is deliberate, not a missing\nstep: an operator authoring a binding is a considered act, an event arriving\nis not, and nothing should self-arm (`docs/architecture/bindings.md`'s\nthreat-model section, point 2). Click **Approve**. Status becomes **armed**.\n\n## 6. Fire it for real\n\nSend the same kind of event again, this time **signed** with the binding's\nsecret:\n\n```bash\nBODY='{\"issue\": {\"title\": \"NullPointerException in checkout\"}, \"level\": \"error\"}'\nSECRET=\"the-secret-you-set-in-step-4\"\nSIG=$(printf '%s' \"$BODY\" | openssl dgst -sha256 -hmac \"$SECRET\" | sed 's/^.* //')\n\ncurl -X POST http://localhost:8484/webhooks/capture/sentry \\\n -H \"Content-Type: application/json\" \\\n -H \"X-Hub-Signature-256: sha256=$SIG\" \\\n -d \"$BODY\"\n```\n\nThis event is captured as **authenticated**, matched against your armed\nbinding, and dispatched into a new task through the exact same\ngate/worktree/sandbox pipeline every other task goes through -- see\n`ARCHITECTURE.md`'s Task Lifecycle section, \"Provenance by origin.\" Check\nthe **Tasks** page; a new task should appear, its body built from the fields\nyour mapping resolved.\n\n## Editing a live binding\n\nAny edit to an armed binding -- even a typo fix -- drops it back to\n**pending_approval** automatically. This can't be bypassed by editing; it\nhas to be re-approved. When editing, leave the **Shared secret** field blank\nto keep the existing secret; only fill it in if you're deliberately\nrotating it.\n" + "body": "# Building your first playbook binding\n\nThis walks through the whole no-code path the event-capture epic promises:\npoint an external app's webhook at archie, see the real payload it sends,\nmap the fields you care about, bind them to a workflow, and watch the next\nmatching event turn into a task -- no code, no redeploy, no restart. See\n`docs/architecture/bindings.md` for the model and threat model behind each\nstep below.\n\n## 1. Send a test event\n\nPoint whatever external app you want archie to react to at:\n\n```\nPOST http://\u003cyour-archied-host\u003e:8484/webhooks/capture/\u003csource\u003e\n```\n\n`\u003csource\u003e` is a name you choose for this sender (e.g. `sentry`, `github-ci`,\n`stripe`) -- it's just a URL path segment, not a preset list. Send one real\nevent however is easiest for a first test, e.g.:\n\n```bash\ncurl -X POST http://localhost:8484/webhooks/capture/sentry \\\n -H \"Content-Type: application/json\" \\\n -d '{\"issue\": {\"title\": \"NullPointerException in checkout\"}, \"level\": \"error\"}'\n```\n\nThis event isn't signed yet, so it's captured but marked **unauthenticated**\n-- visible for you to design against, but incapable of triggering anything\n(`docs/architecture/bindings.md`'s threat-model section explains why: an\nunauthenticated event can never match a binding, no matter how it looks).\n\n## 2. Inspect the captured payload\n\nOpen the dashboard's **Event inspector** (`/captures`). Your test event\nappears there with its real body -- this is schema-by-example: you're\nlooking at an actual payload from the actual sender, not guessing at a\nschema in advance.\n\n## 3. Map the fields you care about\n\nOpen **Field mappings** (`/mappings`) → **New mapping**. Pick your captured\nevent from the dropdown, then click through its payload tree on the right --\nevery key and value is clickable. Clicking a value binds it as a field\n(name, JSON path, and an inferred type you can change). Add whichever fields\nyour workflow needs (e.g. the issue title), mark the ones that must be\npresent as **Required**, hit **Preview** to confirm they resolve correctly\nagainst the real payload, then **Save**.\n\n## 4. Create the binding\n\nOpen **Playbook bindings** (`/bindings`) → **New binding**:\n\n- **Name** -- anything descriptive.\n- **Source** -- the same `\u003csource\u003e` segment from step 1 (e.g. `sentry`).\n- **Field mapping** -- the one you just saved.\n- **Workflow** -- which registered workflow should run on a match.\n- **Repo pin** (optional) -- leave both blank to use the single configured\n repo; set owner/repo explicitly if you run a multi-repo deployment.\n- **Shared secret** -- a real secret (16+ bytes), never `test1234`. The\n sender signs every request body with this via HMAC-SHA256 so archie can\n tell a genuine event from anyone who finds the URL.\n\nSave. The binding starts in **draft** -- nothing fires yet.\n\n## 5. Approve it\n\nA **draft** or freshly-**edited** binding needs an explicit human approval\nbefore it can ever match a live event -- this is deliberate, not a missing\nstep: an operator authoring a binding is a considered act, an event arriving\nis not, and nothing should self-arm (`docs/architecture/bindings.md`'s\nthreat-model section, point 2). Click **Approve**. Status becomes **armed**.\n\n## 6. Fire it for real\n\nSend the same kind of event again, this time **signed** with the binding's\nsecret:\n\n```bash\nBODY='{\"issue\": {\"title\": \"NullPointerException in checkout\"}, \"level\": \"error\"}'\nSECRET=\"the-secret-you-set-in-step-4\"\nSIG=$(printf '%s' \"$BODY\" | openssl dgst -sha256 -hmac \"$SECRET\" | sed 's/^.* //')\n\ncurl -X POST http://localhost:8484/webhooks/capture/sentry \\\n -H \"Content-Type: application/json\" \\\n -H \"X-Hub-Signature-256: sha256=$SIG\" \\\n -d \"$BODY\"\n```\n\nThis event is captured as **authenticated**, matched against your armed\nbinding, and dispatched into a new task through the exact same\ngate/worktree/sandbox pipeline every other task goes through -- see\n`docs/architecture/messaging-and-work-intake.md`'s \"Intake provenance\". Check\nthe **Tasks** page; a new task should appear, its body built from the fields\nyour mapping resolved.\n\n## Editing a live binding\n\nAny edit to an armed binding -- even a typo fix -- drops it back to\n**pending_approval** automatically. This can't be bypassed by editing; it\nhas to be re-approved. When editing, leave the **Shared secret** field blank\nto keep the existing secret; only fill it in if you're deliberately\nrotating it.\n" } ], "nav": [ diff --git a/docs/guides/first-playbook.md b/docs/guides/first-playbook.md index 27e259006..022209b2c 100644 --- a/docs/guides/first-playbook.md +++ b/docs/guides/first-playbook.md @@ -90,7 +90,7 @@ curl -X POST http://localhost:8484/webhooks/capture/sentry \ This event is captured as **authenticated**, matched against your armed binding, and dispatched into a new task through the exact same gate/worktree/sandbox pipeline every other task goes through -- see -`ARCHITECTURE.md`'s Task Lifecycle section, "Provenance by origin." Check +`docs/architecture/messaging-and-work-intake.md`'s "Intake provenance". Check the **Tasks** page; a new task should appear, its body built from the fields your mapping resolved. diff --git a/docs/prds/deep-research-capability.md b/docs/prds/deep-research-capability.md index ac7a383a8..fecb27e57 100644 --- a/docs/prds/deep-research-capability.md +++ b/docs/prds/deep-research-capability.md @@ -25,7 +25,7 @@ Every claim in this document is mapped to what exists in the tree. | Tool guardrails (repeat-failure / no-progress) | **Implemented, not on the call path** | `internal/tools/guardrail.go` engine exists. Its only non-test caller is `dispatchOne` — which is itself unreachable — so **the chat path never consults it**, and the agent path only records per *stage* (`domain/workflow/agent.go`) | | Structured capture tools (model reports into a typed store, not prose) | **Implemented** | `internal/app/agentworker/review.go` `record_finding` / `record_checked` — `core.NewTypedTool` appending to a caller-owned slice. This is the pattern a research source ledger should reuse | | Curator engine (interval, declared tool set, lifecycle) | **Implemented, tools broken** | `internal/domain/curator`; a curator declaring tools is **refused** at registration (`registry.go`, no `ToolBuilder` implementation) and `curatorLLMRunner` (`internal/app/archied/main.go`) drops `ChatRequest.Tools`. Budgets are time+concurrency only (`runtime.go`) — **no token budget** | -| Cron / scheduling + delivery router | **Implemented, wired into no binary** | `internal/domain/scheduling`, `internal/infrastructure/cronstore` (schema v2, `Kind` discriminator), `internal/infrastructure/crondelivery` — all present and green, but `go list -deps` for every `cmd/` contains none of them, and `cron`-expression firing arithmetic is unimplemented (`schedule.go`, "slice 2"). `ARCHITECTURE.md` still asserts "**No daemon-internal cron.**" | +| Cron / scheduling + delivery router | **Implemented, wired into the daemon** | `internal/domain/scheduling` (ticker engine), `internal/infrastructure/cronstore` (schema v2, `Kind` discriminator), and `internal/infrastructure/crondelivery`; `internal/app/archied/scheduling.go` composes them and `main.go` starts the engine. | | Workflow engines (multi-stage, resumable) | **Implemented, worktree-bound** | `internal/domain/workflow` — every non-triage workflow opens with `StagePrepareWorktree()` and mounts `tc.Dir` into a container; the task persists after every stage (`workflow.go`). `triage.go` documents there is "no cheaper directory-less classification primitive" | | Long-running chat-initiated work with progress | **Partial** | Chat turns run **once, synchronously**, inline in the channel's turn runner (`internal/gateway/turn.go`). Chat can spawn a *workflow task* (`task_spawn`) which is asynchronous — but that path is worktree/PR-shaped | | **Operator-push delivery of an async result** | **Aspirational — zero code** | Nothing pushes a finished result to a chat. No channel subscribes to the event bus (Telegram only *publishes* `KindTurnCompleted`); `crondelivery.Courier` is the intended seam but has **no implementation and no caller**; `send_file` works only inside the live turn. The one out-of-band notification in the tree is feasibility's n8n webhook (`feasibility.go`) | @@ -198,7 +198,7 @@ Reusing it would mean every ad-hoc research run leaves a permanent job row. | Collator | `internal/domain/research` | Model pass over `[]ChildConclusion` + source store → report body | | Report renderer | `internal/domain/research` | Markdown + numbered citation list rendered **from the store** | | Chat trigger | slice 2: `internal/tools/research`; slice 3: gateway tool + courier in `internal/app/archied` | — | -| Durable run + delivery (slice 3) | `internal/domain/research` + `internal/infrastructure/research` + `internal/app/archied` wiring | Follows the strict plugin-engine rule in `ARCHITECTURE.md` ("Plugin engine +| Durable run + delivery (slice 3) | `internal/domain/research` + `internal/infrastructure/research` + `internal/app/archied` wiring | Follows the strict plugin-engine rule in `docs/architecture/plugins-and-extensions.md` ("Plugin engine rule (strict)"): typed contract, owning registry with start/health/stop, narrow typed registrar | ### Why not the curator diff --git a/docs/prds/eda-playbook-engine.md b/docs/prds/eda-playbook-engine.md index f7a55124f..7e515c8c8 100644 --- a/docs/prds/eda-playbook-engine.md +++ b/docs/prds/eda-playbook-engine.md @@ -42,7 +42,8 @@ long as it's implemented behind one of the typed engine families below. generic interface -- see "Typed positions, not one generic hook" below, which is how it stays compliant with the plugin engine rule instead of reopening that document's #4. -- `ARCHITECTURE.md#plugin-engine-rule-strict` governs every extension point +- `docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict` + governs every extension point introduced here. Every position below is a typed domain contract with an owning registry, not a generic callback map. This is a hard constraint, confirmed against the existing rule text before writing this doc. diff --git a/docs/prds/event-sources-and-reactions.md b/docs/prds/event-sources-and-reactions.md index 71ba40d0f..0f9b93f71 100644 --- a/docs/prds/event-sources-and-reactions.md +++ b/docs/prds/event-sources-and-reactions.md @@ -12,7 +12,8 @@ each question and saying which packages change. ## 1. In-process typed families, not out-of-process dispatch Archie has no out-of-process extension mechanism. The plugin engine -rule (`ARCHITECTURE.md#plugin-engine-rule-strict`, mechanically checked by +rule (`docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`, +mechanically checked by `internal/plugin/architecture_test.go`) keeps every capability family in-process behind a narrow typed registrar; NATS RPC exists for daemon<->agentexec, a different actor with a different trust boundary (task diff --git a/docs/prds/image-capability-contract.md b/docs/prds/image-capability-contract.md index c9399e445..1950b2327 100644 --- a/docs/prds/image-capability-contract.md +++ b/docs/prds/image-capability-contract.md @@ -25,8 +25,9 @@ in the repo defines this (`grep -ri image` over `internal/domain` and ## Design Follows the plugin engine family shape already established by -`internal/domain/curator` and `internal/domain/memory` (ARCHITECTURE.md -"plugin engine rule strict"): a typed contract with real domain operations, +`internal/domain/curator` and `internal/domain/memory` +(`docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`): a +typed contract with real domain operations, an owning `Registry` (register/get/names, no start/stop lifecycle — see below), and narrow host access via a `Registrar`. diff --git a/docs/prds/memory-engine-unification.md b/docs/prds/memory-engine-unification.md index 89ecaee07..e5d5deb66 100644 --- a/docs/prds/memory-engine-unification.md +++ b/docs/prds/memory-engine-unification.md @@ -332,7 +332,8 @@ Each step is independently verifiable and lands green on its own. 4. **Chat write path.** The tool adapter onto CRUD and the turn's subject on the context. *Depends on 3, because a write is only observable through the read.* 5. **Delete `internal/memory` and the dead config/dashboard surface**, and - rewrite `ARCHITECTURE.md`'s "Memory engine family" section to describe the + rewrite the "Memory engine family" section of + `docs/architecture/plugins-and-extensions.md` to describe the engine family rather than a manager it deleted. *Depends on 3 and 4.* Steps 3 and 4 are separable from each other; step 1 is separable from everything. diff --git a/docs/prds/service-decomposition.md b/docs/prds/service-decomposition.md index 5468e16c0..eee35bad0 100644 --- a/docs/prds/service-decomposition.md +++ b/docs/prds/service-decomposition.md @@ -116,7 +116,7 @@ re-baselined coupling inventory is below, and the ratified boundary is - Any concrete adoption of Pi (or another external agent harness) as an execution runner. Raised in conversation alongside this decomposition, but it is a separate decision (agent execution as a strict data - boundary, `ARCHITECTURE.md`) and does not gate or get gated by this PRD. + boundary, `docs/architecture/agent-system.md`) and does not gate or get gated by this PRD. ## Open questions diff --git a/docs/prds/webhook-intake-security.md b/docs/prds/webhook-intake-security.md index c2e36ded3..0f8604e15 100644 --- a/docs/prds/webhook-intake-security.md +++ b/docs/prds/webhook-intake-security.md @@ -66,10 +66,10 @@ the control. Mapped payload fields are treated as ordinary user-controlled text inside an agent prompt, exactly like an inbound chat message never as trusted instructions. "The agent should ignore instructions in the payload" is not a -control and must not appear as one anywhere in this feature -- per -`ARCHITECTURE.md`'s environmental-enforcement-over-prompt-rules principle, -the boundary is the gate and the sandbox, not a sentence in the system -prompt. +control and must not appear as one anywhere in this feature -- per the +environmental-enforcement principle (`docs/architecture/agent-system.md`, +"Existing invariants to preserve"), the boundary is the gate and the sandbox, +not a sentence in the system prompt. ## 5. Retention and redaction of captured payloads