diff --git a/.agents/skills/archie-agent-fleet/SKILL.md b/.agents/skills/archie-agent-fleet/SKILL.md index 9553b1003..6817655be 100644 --- a/.agents/skills/archie-agent-fleet/SKILL.md +++ b/.agents/skills/archie-agent-fleet/SKILL.md @@ -40,7 +40,7 @@ State these as facts to each lane, not as advice. Each one has cost a real run. |---|---| | **The control plane fails closed.** A stored resource that will not validate stops `archied` starting, and nothing bypasses the control plane at boot. Resource history is the only in-band remedy, and it needs the State Store up. | A lane touching the boot path, a stored resource kind, or the recovery commands is a risk-routed lane for the operator's eyes, not a gate-and-merge lane. Evidence: `docs/architecture/safe-change-and-recovery.md` ("Database settings: degrade paths and recovery"). | | **`internal/webui` must not reach `internal/domain/workflow`.** Go imports are atomic: one reference relinks the whole daemon runtime into the UI binary. | `cmd/archie-ui/architecture_test.go` measures the linked binary (`go list -deps .`), not the imports, and fails on anything under `internal/domain/workflow`. `internal/domain/workflow/task` is a named `gateExceptions` entry that must stay linked — it carries only contract and view types, and a stale exception fails the test in both directions. The same test's gate also excludes `internal/app/archied` from the UI binary, so a helper needed by both the daemon and the `cmd/` binaries belongs in `internal/infrastructure/configuration` (which already owns path resolution), never in `archied`. | -| **The task DB has one owner.** `-tasks.sqlite` is served by the standalone `cmd/archie-state-store` process alone. No second listener, no in-process serving path in the daemon or Gateway. | `boot.openStateStore` (`internal/app/archied/state_store.go`) is the only production caller of `openProductionTaskStore`, and it takes an exclusive flock on `.lock` for the process's life — a second State Store on the same file fails there rather than merely being discouraged. Offline maintenance belongs in subcommands of that binary, never a new `cmd/` entry (`docs/architecture/organisation.md#process-boundaries`). | +| **The task DB has one owner.** The task tables are served by the standalone `cmd/archie-state-store` process alone. No second listener, no in-process serving path in the daemon or Gateway. | `boot.openStateStore` (`internal/app/archied/state_store.go`) is the only place the task store is built, and it holds the `state-store` Postgres ownership lock for the process's life — a second State Store on the same database fails there rather than merely being discouraged. Offline maintenance belongs in subcommands of that binary, never a new `cmd/` entry (`docs/architecture/organisation.md#process-boundaries`). | | **Never commit `.beads/`.** Bead state moves through Dolt; `issues.jsonl` and `interactions.jsonl` are local-only. | A lane that stages `.beads/` pollutes its diff. Closing a bead is not commit-worthy on its own. | | **Generated assets are LAW.** `ui/dist/` and the schema outputs (`docs/data/generated/contracts.json`). | Incorporate them into the commit that shifted them. Never revert, regenerate away, or leave them out. | | **Scratch goes to `/tmp` only.** Never the working tree. | `worktree.CommitAll` stages with go-git's `AddWithOptions{All: true}` (`internal/worktree/worktree.go:502`), which does **not** honour `.gitignore`. An untracked scratch file in a lane worktree is swept onto the task branch. The task brief lives in `.git/task.json` for exactly this reason. | diff --git a/.agents/skills/archie-architecture-contract/SKILL.md b/.agents/skills/archie-architecture-contract/SKILL.md index 40410e40f..4aefe05e5 100644 --- a/.agents/skills/archie-architecture-contract/SKILL.md +++ b/.agents/skills/archie-architecture-contract/SKILL.md @@ -88,7 +88,7 @@ reconciliation, parking/retry/exhaustion, and crash recovery. Read `Store.Transition(ctx, taskID, from, to, detail)` now guards on `from` (`WHERE id=? AND status=?`, returning `store.ErrStaleTransition` on mismatch) and runs the status update plus the transition-history insert in one -transaction (`internal/store/store.go`). The sentinel crosses the State Store +transaction (`internal/infrastructure/postgres/store.go`). The sentinel crosses the State Store gRPC boundary via `mapError`/`unmapError` in `internal/infrastructure/staterpc/values.go`; its canonical message string is part of the wire contract. Earlier non-atomic behavior is the settled fix, @@ -101,7 +101,7 @@ not an open design. | The model never runs git | Keep clone, branch, commit, push, diff, and cleanup in deterministic workspace/worktree steps. | CURRENT: `docs/architecture/agent-system.md`; `internal/worktree/` | | Deterministic constraints outrank prompts | Enforce gates, protected paths, read-only stages, test protection, and change caps outside model instructions. | CURRENT: `docs/architecture/agent-system.md`; `internal/gate/`; `internal/domain/workflow/` | | Agent execution is a data boundary | Send bounded, versioned input; validate correlated output before applying it. | CURRENT foundation: `internal/agentexec/protocol.go`; `docs/architecture/agent-system.md` | -| Every workflow ends explicitly | Preserve terminal outcome or visible park. Preserve crash recovery of interrupted `running` work. | CURRENT: `internal/domain/workflow/workflow.go`; `internal/store/store.go` | +| Every workflow ends explicitly | Preserve terminal outcome or visible park. Preserve crash recovery of interrupted `running` work. | CURRENT: `internal/domain/workflow/workflow.go`; `internal/infrastructure/postgres/store.go` | | Generic plugins stay metadata-only | Keep `plugin.Plugin` at `Name()`/`Version()`. Behavior on typed capability interface with owning `Registry`/`Manager`. | CURRENT-enforced rule: `internal/plugin/architecture_test.go` | | Capability engines own lifecycle | Give resource-owning engines explicit start/health/stop semantics and failure isolation. | APPROVED rule: `docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict` | | Trust is explicit | Treat trusted operator-installed in-process code, repository code, out-of-process integrations, and container-isolated code as different trust classes. | APPROVED rule: `docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict` | @@ -202,7 +202,7 @@ Source: `docs/architecture/configuration.md`. | Capability-specific plugin behavior | Owning domain's typed registry/manager | Never add behavior to `plugin.Plugin`. | | Broker-neutral publication/subscription mechanics | `internal/eventbus` | Keep message meaning and schemas with their domain. | | Shared policy evaluation/evidence mechanics | No current package. `internal/policy` was the planned location; as of 2026-09-18 policy lives inside playbook bindings (`playbooks/`, `internal/domain/eda/playbook`) and `internal/gate/`. | Keep policy vocabulary, evaluators, and consequences with consuming domain. | -| NATS, SQLite, forge, config decoding, external SDK adapters | `internal/infrastructure/` | Implement a narrow owner-defined contract; translate at boundary. | +| NATS, PostgreSQL, forge, config decoding, external SDK adapters | `internal/infrastructure/` | Implement a narrow owner-defined contract; translate at boundary. | | Construction, cross-domain connection, startup/shutdown | `internal/app/archied` or `internal/app/agentworker` | Keep `cmd/*` thin; define health and shutdown order. | | Deploy assembly | `deployments/` | Do not move application behavior into deployment files. | | Unreviewed memory/tools/workspace/gate/storage/web/RPC/scheduling | **OPEN** | Run `archie-architecture-planning-campaign`. | @@ -213,7 +213,7 @@ Source: `docs/architecture/configuration.md`. | --- | --- | | `TaskContext` is a mutable service bag with config, services, policy, scratch state. | Design narrow step inputs and staged removal. | | Workflow lifecycle split across store, daemon, workflow, gateway, worker. | Trace every writer and reader before changing status. | -| State changes and transition history were non-atomic and `from` was advisory — **settled** in `internal/store`: `Transition` guards on `status=?` and writes status plus history in one transaction. | Keep every status writer behind `Transition`; do not reintroduce a raw `UPDATE tasks SET status`. | +| State changes and transition history were non-atomic and `from` was advisory — **settled** in `internal/infrastructure/postgres`: `Transition` guards on the current status and writes status plus history in one transaction. | Keep every status writer behind `Transition`; do not reintroduce a raw `UPDATE tasks SET status`. | | In-process and container paths put orchestration on different sides of process boundary. | Decide which path survives before extending both. | | Identity uses strings; task uniqueness/NATS dedup omit identity. | Don't promise multi-identity ownership until canonical IDs migrate. | | Container RPC uses root forge/worktree, not task owner's identity. | Known trust/correctness defect — not identity-isolated. | diff --git a/.agents/skills/archie-codebase-discovery/SKILL.md b/.agents/skills/archie-codebase-discovery/SKILL.md index 545638fcd..6ef5af266 100644 --- a/.agents/skills/archie-codebase-discovery/SKILL.md +++ b/.agents/skills/archie-codebase-discovery/SKILL.md @@ -68,8 +68,8 @@ Start at these verified anchors: | Resident daemon | `cmd/archied/main.go` → `internal/app/archied.Run()` (`internal/app/archied/main.go`) → `daemon.Daemon` | | Sandboxed worker | `cmd/archie-agent/main.go` → `internal/app/agentworker/`; NATS boundary in `internal/infrastructure/agenttransport/nats/` | | Task orchestration | `internal/daemon/daemon.go`, `internal/domain/workflow/` | -| Task contracts | `internal/domain/storecontract/storecontract.go` (`TaskStore`); `internal/store/interface.go` holds compatibility aliases only | -| Production task persistence | `internal/store/`; `openProductionTaskStore` reached via `internal/app/archied/state_store.go` | +| Task contracts | `internal/domain/storecontract/storecontract.go` (`TaskStore`) | +| Production task persistence | `internal/infrastructure/postgres/`; built in `internal/app/archied/state_store.go` | | RPC split | `internal/{taskrun,natsrpc,forgerpc,worktreerpc}/` plus `internal/infrastructure/staterpc/` (gRPC State Store) | | Chat and channels | `internal/gateway/`, `internal/channels/` | | Configuration | `internal/config/`, `internal/infrastructure/configuration/`, `config.example.toml`, `deployments/*.toml` | @@ -80,8 +80,8 @@ Start at these verified anchors: ### Map packages before reading files ```sh -go list -f '{{.ImportPath}}|go={{join .GoFiles ","}}|tests={{join .TestGoFiles ","}}' ./internal/store -go list -json ./internal/store +go list -f '{{.ImportPath}}|go={{join .GoFiles ","}}|tests={{join .TestGoFiles ","}}' ./internal/infrastructure/postgres +go list -json ./internal/infrastructure/postgres .agents/skills/archie-codebase-discovery/scripts/package-edges.sh ./internal/... ``` @@ -96,7 +96,7 @@ gopls references -d internal/domain/storecontract/storecontract.go:33:2 gopls implementation internal/domain/storecontract/storecontract.go:33:2 gopls call_hierarchy internal/domain/storecontract/storecontract.go:33:2 go doc ./internal/domain/storecontract.TaskStore -go test ./internal/store -list '.' +go test ./internal/infrastructure/postgres -list '.' ``` ### Generate AST candidates for construction @@ -189,14 +189,16 @@ and `Loader.Dir` for overlay/`conf.d` inputs). There is no `config.Load` or 6. Read contract/architecture tests. 7. Confirm deploy inputs in Compose and CI. -### Trace SQLite persistence +### Trace persistence Treat `internal/domain/storecontract` as the application contract and -`internal/store` as the production SQLite adapter behind it. +`internal/infrastructure/postgres` as the production adapter behind it; the +schema is `internal/infrastructure/postgres/migrations/` and the queries are +sqlc-generated from `queries/`. ```sh gopls implementation internal/domain/storecontract/storecontract.go:33:2 -rg -n 'CREATE TABLE|CREATE INDEX|QueryContext|ExecContext|BeginTx' internal/store +rg -n 'CREATE TABLE|CREATE INDEX' internal/infrastructure/postgres/migrations ``` ## Reject false confidence diff --git a/.agents/skills/archie-config-and-flags/SKILL.md b/.agents/skills/archie-config-and-flags/SKILL.md index 5dce85191..cd1ab837d 100644 --- a/.agents/skills/archie-config-and-flags/SKILL.md +++ b/.agents/skills/archie-config-and-flags/SKILL.md @@ -106,7 +106,7 @@ Do not expose these helper flags as daemon settings. | Keys | Default, validation, and consumer | Status | | --- | --- | --- | -| `work_dir`, `db_path` | Derive from `$XDG_DATA_HOME/archie`, else `$HOME/.local/share/archie`; explicit `~` paths are home-expanded. `db_path` is a prefix: tasks/events use `-tasks.sqlite` and conversations use `-conversations.sqlite`. | production-wired | +| `work_dir`, `state_dir`, `db_path` | Derive from `$XDG_DATA_HOME/archie`, else `$HOME/.local/share/archie`; explicit `~` paths are home-expanded. `state_dir` holds the embedded NATS store and endpoint file, task logs and the readiness disk target. `db_path` only locates the legacy `-tasks.sqlite`, `-eda.sqlite` and `-conversations.sqlite` files for `archie-state-store import` and the boot gate. | production-wired | | `skills_dir` | Empty falls back to `work_dir` for startup workflow registry. | partially-wired | | `plugin_dir` | Empty disables daemon plugin loading; configured directory loaded in `internal/app/archied`. | production-wired | | `poll_interval` | Default `60s`; root poll loop uses it, identities may override only interval. | production-wired | @@ -125,7 +125,7 @@ Do not expose these helper flags as daemon settings. | `diff_cap_lines` | Zero replaced by `400` in `finalize`; `StageDiffCap` treats `<= 0` as disabled. TOML zero cannot disable; negative value does. | production-wired | | `web.listen` | Empty becomes `127.0.0.1:8484`; literal `off` disables. No authentication. | production-wired | | `notify.webhook` | Empty disables; feasibility workflow POSTs human-decision notifications when set. | production-wired | -| `nats.url`, `token_env` | Empty URL keeps SQLite path. Configured token variable must be non-empty. | production-wired | +| `nats.url`, `token_env` | Empty URL keeps the in-process path. Configured token variable must be non-empty. | production-wired | | `containers.*` | See dedicated table below. | production-wired | | `chat.*` | See channel table below. | production-wired | | `memory.provider`, `provider_config` | Feature YAML decodes these; `internal/app/archied` always starts built-in file provider at `work_dir/memory`. | decoded-but-unwired | diff --git a/.agents/skills/archie-debugging-playbook/SKILL.md b/.agents/skills/archie-debugging-playbook/SKILL.md index a90c3bda8..3c4150285 100644 --- a/.agents/skills/archie-debugging-playbook/SKILL.md +++ b/.agents/skills/archie-debugging-playbook/SKILL.md @@ -1,6 +1,6 @@ --- name: archie-debugging-playbook -description: Project-specific symptom-to-evidence triage for archie-core. Load this skill when Archied hangs, parks a task, loses or duplicates work, cannot reach NATS or an agent container, rejects configuration or secrets, shows stale Telegram or dashboard state, loses streamed text, fails MCP startup, reports SQLite state surprises, degrades an optional feature, or when Go tests disagree between a sandbox, editor, CI, and the host. Use it to separate environment failures from code regressions and to choose the next exact diagnostic. +description: Project-specific symptom-to-evidence triage for archie-core. Load this skill when Archied hangs, parks a task, loses or duplicates work, cannot reach NATS or an agent container, rejects configuration or secrets, shows stale Telegram or dashboard state, loses streamed text, fails MCP startup, reports task-state surprises, degrades an optional feature, or when Go tests disagree between a sandbox, editor, CI, and the host. Use it to separate environment failures from code regressions and to choose the next exact diagnostic. --- # Debug Archie from evidence @@ -42,7 +42,7 @@ Use this symptom index: | Telegram reply hangs on the first token | Stream consumer | Streaming | | MCP unit tests pass but a real server hangs | Framing mirror | MCP | | Telegram shows old commands | Token-scoped Telegram state | Telegram | -| Transition succeeds from the wrong state | Store semantics | SQLite/state | +| Transition succeeds from the wrong state | Store semantics | Database/state | | Dashboard skips events but task state is correct | Bounded event buffers | Events | | Search works but never becomes indexed | Missing production wiring | Optional features | | Agent container cannot resolve `nats` | Docker network | Containers | @@ -199,16 +199,16 @@ env GOTMPDIR=/tmp GOCACHE=/tmp/archie-telegram-gocache \ ## Diagnose state, events, containers, and optional features -### SQLite and transitions -`internal/store.Store.Transition` owns task state changes and transition -history: it guards on `from` (`WHERE id=? AND status=?`, returning -`store.ErrStaleTransition`) and writes status plus history in one transaction. +### Database and transitions +`internal/infrastructure/postgres.Store.Transition` owns task state changes and +transition history: it guards on `from`, returning +`storecontract.ErrStaleTransition`, and writes status plus history in one transaction. When state is surprising: read `TaskByID` before/after caller; trace every `Transition`, `Requeue`, `RecoverStale`, `Update` — and remember the daemon exposes these over the gRPC State Store, not NATS. ```bash -env GOTMPDIR=/tmp GOCACHE=/tmp/archie-state-gocache go test ./internal/store -count=1 +env GOTMPDIR=/tmp GOCACHE=/tmp/archie-state-gocache go test ./internal/infrastructure/postgres -count=1 # needs Docker ``` ### Events diff --git a/.agents/skills/archie-diagnostics-and-tooling/SKILL.md b/.agents/skills/archie-diagnostics-and-tooling/SKILL.md index 03697c9ce..a7c16d51a 100644 --- a/.agents/skills/archie-diagnostics-and-tooling/SKILL.md +++ b/.agents/skills/archie-diagnostics-and-tooling/SKILL.md @@ -189,9 +189,9 @@ no dead-code analyzer. ## Collect test, race, coverage, and runtime evidence ```sh -go test ./internal/store -count=1 -coverprofile=/tmp/archie-store.cover +go test ./internal/infrastructure/postgres -count=1 -coverprofile=/tmp/archie-store.cover go tool cover -func=/tmp/archie-store.cover -go test -race ./internal/store -count=1 +go test -race ./internal/infrastructure/postgres -count=1 ``` As of 2026-09-18, the package reported 78.6% statement coverage and passed its diff --git a/.agents/skills/archie-diagnostics-and-tooling/scripts/delivery-snapshot.sh b/.agents/skills/archie-diagnostics-and-tooling/scripts/delivery-snapshot.sh index ea2f40573..97837906c 100755 --- a/.agents/skills/archie-diagnostics-and-tooling/scripts/delivery-snapshot.sh +++ b/.agents/skills/archie-diagnostics-and-tooling/scripts/delivery-snapshot.sh @@ -66,7 +66,7 @@ scan_literal github_container_publish github-workflows 'docker build|docker push # file". The wiring lives in internal/app/archied. scan_literal production_config_resolve composition-root 'configuration\.New\(' internal/app/archied scan_literal production_config_load composition-root 'loadConfig\(|loadRuntimeConfig\(' internal/app/archied -scan_literal production_sqlite_store composition-root 'openProductionTaskStore\(|openStateStore\(' internal/app/archied +scan_literal production_task_store composition-root 'openStateStore\(' internal/app/archied scan_literal production_daemon composition-root '&daemon\.Daemon\{' internal/app/archied scan_literal production_rpc_servers composition-root 'registerTaskRPCServers\(' internal/app/archied diff --git a/.agents/skills/archie-domain-reference/SKILL.md b/.agents/skills/archie-domain-reference/SKILL.md index 39700c12b..8d5bf9f1a 100644 --- a/.agents/skills/archie-domain-reference/SKILL.md +++ b/.agents/skills/archie-domain-reference/SKILL.md @@ -54,7 +54,7 @@ never authorize because two mutable strings match; never rename Agent when only model/provider/display name changes; never pool two Agents' memory. ```bash -rg -n 'type IdentityConfig|type IdentityRunner|Identity string|BotUser string' internal/config internal/daemon internal/store internal/gateway +rg -n 'type IdentityConfig|type IdentityRunner|Identity string|BotUser string' internal/config internal/daemon internal/infrastructure/postgres internal/gateway ``` ## Use the canonical interaction vocabulary @@ -87,7 +87,7 @@ Do not describe current `/spawn` as this target. It directly creates a `task.Task` with synthetic forge issue number. ```bash -rg -n 'type Message struct|type MessageEvent struct|ToLegacy|type Conversation struct|CreateTask|EnqueueChatTask|synthetic' internal/domain/messaging internal/gateway internal/store internal/app/archied +rg -n 'type Message struct|type MessageEvent struct|ToLegacy|type Conversation struct|CreateTask|EnqueueChatTask|synthetic' internal/domain/messaging internal/gateway internal/infrastructure/postgres internal/app/archied ``` ## Use the canonical execution vocabulary @@ -119,12 +119,12 @@ running after crash -> queued ``` Do not claim enforced. `Store.Transition` now guards on `from` -(`WHERE id=? AND status=?` → `store.ErrStaleTransition`) and writes status plus +(→ `storecontract.ErrStaleTransition`) and writes status plus history in one transaction; the sentinel crosses the State Store gRPC boundary. -See `internal/store/store.go`. +See `internal/infrastructure/postgres/store.go`. ```bash -rg -n 'Status[A-Za-z]+|func \(s \*Store\) Transition|func \(s \*Store\) Requeue|RecoverStale' internal/store internal/domain/workflow internal/gateway +rg -n 'Status[A-Za-z]+|func \(s \*Store\) Transition|func \(s \*Store\) Requeue|RecoverStale' internal/infrastructure/postgres internal/domain/workflow internal/gateway ``` ## Keep memory scope separate from conversation scope diff --git a/.agents/skills/archie-run-and-operate/SKILL.md b/.agents/skills/archie-run-and-operate/SKILL.md index 79c200cbd..78e440324 100644 --- a/.agents/skills/archie-run-and-operate/SKILL.md +++ b/.agents/skills/archie-run-and-operate/SKILL.md @@ -13,12 +13,12 @@ history. | Term | Meaning here | |---|---| -| `archied` | Resident orchestrator. Owns config, forge credentials, SQLite stores, worktrees, gateways, dashboard, optional NATS/Docker orchestration. | +| `archied` | Resident orchestrator. Owns config, forge credentials, worktrees, gateways, dashboard, optional NATS/Docker orchestration. | | `archie-agent` | Long-running NATS worker. Consumes per-stage `archie.agent.>` requests and full-task `archie.taskrun.>` requests. Not a stdin/stdout worker. | | Stage request | One autonomous workflow stage sent on `archie.agent..request`. | | Full-task handoff | A whole workflow sent on core NATS subject `archie.taskrun.` to a task container. | | Work directory | `work_dir`: task clones plus memory and candidate index artifacts. | -| State path prefix | `db_path`: the configured prefix. Task/event state uses `-tasks.sqlite`; conversation state uses `-conversations.sqlite`. | +| State directory | `state_dir` (default `~/.local/share/archie`): embedded NATS store, task logs. Task, event, capture and conversation data live in the PostgreSQL database `database_url` names. `db_path` only locates pre-PostgreSQL files for `archie-state-store import`. | ## Apply the operational safety boundary @@ -65,7 +65,7 @@ Both linked binaries may also display Go's `-quickchecks` flag from | Config shape | What runs | Operational status on 2026-09-18 | |---|---|---| -| `agent.mode = "inprocess"` and no `[nats]` | `archied` polls and claims from SQLite, runs workflow stages and model tools in-process. | **Production-wired candidate.** No process or OS isolation. | +| `agent.mode = "inprocess"` and no `[nats]` | `archied` polls and claims through the State Store, runs workflow stages and model tools in-process. | **Production-wired candidate.** No process or OS isolation. | | `agent.mode = "subprocess"` | `archied` starts `agent.command` per stage and expects one JSON invocation on stdin, one response on stdout. | **Open/broken with default command.** `cmd/archie-agent` is a long-running NATS worker and never calls `agentexec.ServeOne`. | | `agent.mode = "nats"`, `[nats]`, containers disabled | `archied` runs the workflow; each autonomous stage goes to a separately started `archie-agent`. | **Implemented but operator-assembled.** Launch and supervise a worker separately. | | `agent.mode = "nats"`, `[nats]`, `containers.enabled = true` | `archied` publishes task discovery through JetStream, prepares a worktree, spawns one agent container, sends the whole workflow on `archie.taskrun.`. | **Host daemon + Compose NATS path.** `deployments/docker-nats-stack.toml` demonstrates it. | @@ -95,7 +95,7 @@ startup did not complete. The daemon has no HTTP health endpoint. Stop with `Ctrl-C`. The signal cancels active work, waits for dispatch goroutines, stops capability and memory managers, removes Archie-labelled -containers, closes NATS, and closes the SQLite stores. +containers, closes NATS, and closes its store connections. ## Run a standalone NATS worker @@ -154,8 +154,8 @@ are spawned by the host `archied` process through `/var/run/docker.sock`. |---|---| | Daemon logs | JSON on host stderr; the configured host supervisor captures them. | | Agent logs | JSON on worker/container stderr. Task containers use `AutoRemove`. | -| Tasks and events | SQLite database at `-tasks.sqlite`; default `~/.local/share/archie/archie.db-tasks.sqlite`. | -| Chat sessions and messages | SQLite database at `-conversations.sqlite`. | +| Tasks and events | PostgreSQL (`database_url`), served by `archie-state-store`. | +| Chat sessions and messages | PostgreSQL (`database_url`), served by `archie-gateway`. | | Default task worktree | `/-/issue-`; default root `~/.local/share/archie/work`. | | Identity worktree | `/identity-/-/issue-`. | | Container worktree | Host worktree bind-mounted read/write at `/data/worktree`; `task.json` written under `.git/` so the agent's commit cannot sweep it onto the task branch. | @@ -217,7 +217,7 @@ Docker builds. The Gitea workflow that used to be the production build path was dropped (`dd9bddc2`); `deployments/*.toml` are the supported profiles. Before promotion: record running image IDs/digests; config checksums (no -secrets); task counts via safe GET; establish a SQLite backup procedure. +secrets); task counts via safe GET; take backups with `archie-state-store backup`. No complete rollback command exists, no versioned image tags. Restoring a binary does not undo forge changes, task transitions, or data-format changes. diff --git a/CLAUDE.md b/CLAUDE.md index e11f2c3b3..f7ed0376a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -222,10 +222,10 @@ structures found in legacy packages. breaks `errors.Is` on the client without changing behavior visibly -- treat those message constants as part of the wire contract. - The State Store is a standalone process (`cmd/archie-state-store`, run via - `archied.RunStateStore`) -- the daemon and Gateway never own `archie.db` or - serve this service in-process; `boot.openStateStore` in - `internal/app/archied/state_store.go` is the only caller of - `openProductionTaskStore`. Never add a second listener for the same + `archied.RunStateStore`) -- the daemon and Gateway never own the task + tables or serve this service in-process; `boot.openStateStore` in + `internal/app/archied/state_store.go` is the only place the task store + is built. Never add a second listener for the same service, and never reintroduce an in-process serving path in the daemon/Gateway (`TestOpenStoresNeverOwnsTaskDB` guards this). - Per-task credentials are scoped, not just authenticated: `daemon. diff --git a/config.example.toml b/config.example.toml index 3267ec6b8..9323faac5 100644 --- a/config.example.toml +++ b/config.example.toml @@ -13,10 +13,11 @@ # secret_engine_dir = "~/.config/archie/secret-engines" # work_dir = "~/.local/share/archie/work" # default -# db_path = "~/.local/share/archie/archie.db" # prefix for independent task and conversation SQLite files +# state_dir = "~/.local/share/archie" # default; embedded NATS store, task logs, disk probe +# db_path = "~/.local/share/archie/archie.db" # default; locates pre-PostgreSQL SQLite files for the import only # The PostgreSQL connection archie-state-store, archied and archie-gateway -# serve from. Required: each fails closed rather than fall back to SQLite. +# serve from. Required: each fails closed without it. # An install with SQLite stores under db_path runs `archie-state-store import` # once, with archie stopped, before it starts. `docker compose up -d postgres` starts # a matching local postgres:18 with this default URL. @@ -349,9 +350,9 @@ target = "127.0.0.1:8585" # archie-gateway's gRPC address (always remot [services.state] target = "127.0.0.1:9090" # REQUIRED. The State Store gRPC address archied and -# # archie-gateway dial. They no longer own archie.db +# # archie-gateway dial. They own no task data # # in-process: the standalone archie-state-store process -# # owns the single SQLite file (docs/prds/state-store-contract.md +# # owns it in PostgreSQL (docs/prds/state-store-contract.md # # §10/§12 step 7). An empty target is a startup error. # listen = "127.0.0.1:9090" # The address archie-state-store BINDS. Read only by that # # process; every client ignores it. Defaults to the target @@ -379,10 +380,9 @@ target = "127.0.0.1:9090" # REQUIRED. The State Store gRPC address arch # Unbound webhook capture: POST /webhooks/capture/{source} accepts and # stores any inbound payload archie doesn't yet have a binding for, so it can # be inspected and mapped later (see docs/prds/event-capture-storage.md). -# Stored in the same SQLite database as tasks, not NATS -- capture has one +# Stored in the same PostgreSQL database as tasks, not NATS -- capture has one # writer and one (human, dashboard) reader, so it doesn't need JetStream's -# fan-out/replay machinery, and SQLite is available in every deployment -# shape unconditionally. All fields default when the section is absent; +# fan-out/replay machinery. All fields default when the section is absent; # capture is on by default. # # From the UI cutover (archie-core-8cda.5.4) the receiver is mounted by the @@ -435,7 +435,7 @@ target = "127.0.0.1:9090" # REQUIRED. The State Store gRPC address arch # Playbook-binding at-rest encryption: [bindings].encryption_key resolves a # 32-byte high-entropy key through the secret registry (engine + key) and seals -# each binding's HMAC secret in the SQLite bindings.secret column with AES-256- +# each binding's HMAC secret in the bindings.secret column with AES-256- # GCM (see docs/prds/binding-secret-encryption.md). When the key is absent, # binding secrets are stored as plaintext (the pre-t2db.7 behaviour) -- enabling # encryption does not retroactively encrypt existing rows, which stay plaintext diff --git a/deployments/INSTRUCTIONS.md b/deployments/INSTRUCTIONS.md index 6b78be68a..b4c97fbb4 100644 --- a/deployments/INSTRUCTIONS.md +++ b/deployments/INSTRUCTIONS.md @@ -18,7 +18,7 @@ This archive contains six linux/amd64 binaries: this process they are simply absent — and nothing logs an error (`archie-core-1c01`), which is why it has a unit of its own in `deployments/systemd-user-service.md`. -- **`archie-state-store`** — owns the single `archie.db` SQLite file behind +- **`archie-state-store`** — owns the task data in PostgreSQL behind a gRPC contract. `archied` and `archie-gateway` are gRPC clients of it, not owners of the database. - **`archie-playbooks`** — standalone CLI to lint playbook binding YAML diff --git a/deployments/README.md b/deployments/README.md index 24c9c4d20..27448a581 100644 --- a/deployments/README.md +++ b/deployments/README.md @@ -43,13 +43,13 @@ systemctl --user start archied ``` The templates use the extracted Gateway Service by default. Start -`archie-gateway` before `archied`; it owns the conversation SQLite file and +`archie-gateway` before `archied`; it owns the conversation tables and serves the `ChatContract` gRPC API on the configured target. -The State Store service (`archie-state-store`) owns the single `archie.db` task -SQLite file and serves the `StateStore` gRPC contract. After the in-process -store path was deleted, BOTH `archied` and `archie-gateway` dial it via -`[services.state].target` and never open `archie.db` themselves +The State Store service (`archie-state-store`) owns the task data in the +PostgreSQL database `database_url` names and serves the `StateStore` gRPC +contract. BOTH `archied` and `archie-gateway` dial it via +`[services.state].target` and never open the task tables themselves (`docs/prds/state-store-contract.md` §12 step 7). The templates set `[services.state]` accordingly: a host-only agent reaches a loopback bind (`127.0.0.1:9090`) with no token, while a container-mode agent needs the @@ -117,10 +117,10 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:9091/healthz # curl -s http://127.0.0.1:9091/health/detailed # state_db component -> 200, or 503 when degraded ``` -A running State Store owns exactly one task SQLite file (`-tasks.sqlite`); -`archied` and `archie-gateway` never open `archie.db` directly, so a second -store process on the same `db_path`, or a consumer that opens the task file, -indicates dual-store ownership (`docs/prds/state-store-contract.md` §12 step 7/8). +A running State Store holds the `state-store` ownership lock on its database; +`archied` and `archie-gateway` never open the task tables directly, so a second +store process on the same `database_url` refuses to start, and a consumer that +opens the task tables indicates dual-store ownership (`docs/prds/state-store-contract.md` §12 step 7/8). The readiness probe is the single-writer check the runbook has against a degraded store: a `503` from `/health/detailed` means `archied` should be started only after the store is healthy. diff --git a/deployments/docker-nats-stack.toml b/deployments/docker-nats-stack.toml index fa22f1614..3554396c3 100644 --- a/deployments/docker-nats-stack.toml +++ b/deployments/docker-nats-stack.toml @@ -15,7 +15,9 @@ poll_interval = "60s" bot_user = "archie" diff_cap_lines = 0 work_dir = "~/.local/share/archie/work" -db_path = "~/.local/share/archie/archie.db" +state_dir = "~/.local/share/archie" # NATS store, task logs +# db_path (default ~/.local/share/archie/archie.db) only locates pre-PostgreSQL +# files for `archie-state-store import`. # The State Store's PostgreSQL connection (required; Postgres-only epic). # `docker compose up -d postgres` provides this local default; point it at # your own Postgres 18 server otherwise. @@ -24,8 +26,8 @@ database_url = "postgres://archie:archie@127.0.0.1:5432/archie?sslmode=disable" [services.gateway] target = "127.0.0.1:8585" -# The State Store gRPC service owns the single archie.db SQLite file; archied -# and archie-gateway dial it and never open archie.db themselves (§12 step 7). +# The State Store gRPC service owns the PostgreSQL task data; archied and +# archie-gateway dial it and never open those tables themselves (§12 step 7). # This deployment runs the agent in containers, so target is the Docker bridge # gateway (172.17.0.1) the container can reach, with a matching target_token # that archie-state-store validates. Host-only agent: use 127.0.0.1:9090 with diff --git a/deployments/local-ollama-standalone.toml b/deployments/local-ollama-standalone.toml index 3c202d039..2d3804d0e 100644 --- a/deployments/local-ollama-standalone.toml +++ b/deployments/local-ollama-standalone.toml @@ -6,7 +6,9 @@ # ============================================================================== work_dir = "~/.local/share/archie/work" -db_path = "~/.local/share/archie/archie.db" +state_dir = "~/.local/share/archie" # NATS store, task logs +# db_path (default ~/.local/share/archie/archie.db) only locates pre-PostgreSQL +# files for `archie-state-store import`. # The State Store's PostgreSQL connection (required; Postgres-only epic). # `docker compose up -d postgres` provides this local default; point it at # your own Postgres 18 server otherwise. @@ -15,8 +17,8 @@ database_url = "postgres://archie:archie@127.0.0.1:5432/archie?sslmode=disable" [services.gateway] target = "127.0.0.1:8585" -# The State Store gRPC service owns the single archie.db SQLite file; archied -# and archie-gateway dial it and never open archie.db themselves (§12 step 7). +# The State Store gRPC service owns the PostgreSQL task data; archied and +# archie-gateway dial it and never open those tables themselves (§12 step 7). # Host-only deployment: the agent runs on the host, so the loopback target # 127.0.0.1:9090 needs no token. Start archie-state-store with the matching # `-listen 127.0.0.1:9090`. diff --git a/deployments/multi-forge-github-gitea.toml b/deployments/multi-forge-github-gitea.toml index f5727333b..09ea48e2d 100644 --- a/deployments/multi-forge-github-gitea.toml +++ b/deployments/multi-forge-github-gitea.toml @@ -7,7 +7,9 @@ # ============================================================================== work_dir = "~/.local/share/archie/work" -db_path = "~/.local/share/archie/archie.db" +state_dir = "~/.local/share/archie" # NATS store, task logs +# db_path (default ~/.local/share/archie/archie.db) only locates pre-PostgreSQL +# files for `archie-state-store import`. # The State Store's PostgreSQL connection (required; Postgres-only epic). # `docker compose up -d postgres` provides this local default; point it at # your own Postgres 18 server otherwise. @@ -16,8 +18,8 @@ database_url = "postgres://archie:archie@127.0.0.1:5432/archie?sslmode=disable" [services.gateway] target = "127.0.0.1:8585" -# The State Store gRPC service owns the single archie.db SQLite file; archied -# and archie-gateway dial it and never open archie.db themselves (§12 step 7). +# The State Store gRPC service owns the PostgreSQL task data; archied and +# archie-gateway dial it and never open those tables themselves (§12 step 7). # This deployment runs the agent in containers, so target is the Docker bridge # gateway (172.17.0.1) the container can reach, with a matching target_token # that archie-state-store validates. Host-only agent: use 127.0.0.1:9090 with diff --git a/deployments/single-forge-github.toml b/deployments/single-forge-github.toml index 88ea68fb5..0d9823dd0 100644 --- a/deployments/single-forge-github.toml +++ b/deployments/single-forge-github.toml @@ -6,7 +6,9 @@ # ============================================================================== work_dir = "~/.local/share/archie/work" -db_path = "~/.local/share/archie/archie.db" +state_dir = "~/.local/share/archie" # NATS store, task logs +# db_path (default ~/.local/share/archie/archie.db) only locates pre-PostgreSQL +# files for `archie-state-store import`. # The State Store's PostgreSQL connection (required; Postgres-only epic). # `docker compose up -d postgres` provides this local default; point it at # your own Postgres 18 server otherwise. @@ -23,8 +25,8 @@ diff_cap_lines = 400 [services.gateway] target = "127.0.0.1:8585" -# The State Store gRPC service owns the single archie.db SQLite file. archied -# and archie-gateway dial it here and never open archie.db themselves +# The State Store gRPC service owns the PostgreSQL task data. archied and +# archie-gateway dial it here and never open those tables themselves # (docs/prds/state-store-contract.md §12 step 7). This deployment runs the # agent in containers, so the target must be the Docker bridge gateway address # (172.17.0.1) that the container can reach, and a non-loopback target requires diff --git a/deployments/systemd-user-service.md b/deployments/systemd-user-service.md index 96832982d..687eb577d 100644 --- a/deployments/systemd-user-service.md +++ b/deployments/systemd-user-service.md @@ -16,11 +16,10 @@ here, where it would drift from what is written. The Gateway unit is `archie-gateway.service`, written by `install.sh` beside `archied.service`. -The State Store unit is `archie-state-store.service`. It owns the single `archie.db` SQLite -file and serves the `StateStore` gRPC contract. After the in-process store path -was deleted, BOTH `archied` and `archie-gateway` dial it via -`[services.state].target` (see the `archied` config below) and never open -`archie.db` themselves (`docs/prds/state-store-contract.md` §12 step 7). A +The State Store unit is `archie-state-store.service`. It owns the task data in +PostgreSQL and serves the `StateStore` gRPC contract. BOTH `archied` and +`archie-gateway` dial it via `[services.state].target` (see the `archied` +config below) and never open the task tables themselves (`docs/prds/state-store-contract.md` §12 step 7). A host-only agent uses the loopback bind with no token below; a container-mode agent needs the Docker bridge gateway address plus a bearer token via `--token`/`STATE_STORE_TOKEN` or `[services.state].target_token`, and a @@ -254,6 +253,6 @@ with the operator's Docker credentials and retain `pull_policy = "missing"`: docker compose pull agent ``` -`db_path` and other paths are native host paths. Point them at the intended -state directory; changing from an old containerized daemon path can otherwise -start Archie against empty state files. +`state_dir` and other paths are native host paths. Point `state_dir` at the +intended state directory (embedded NATS store, task logs); `db_path` only +locates pre-PostgreSQL files for `archie-state-store import`. diff --git a/docs/architecture/agent-system.md b/docs/architecture/agent-system.md index bcf7d48da..daafd0942 100644 --- a/docs/architecture/agent-system.md +++ b/docs/architecture/agent-system.md @@ -194,8 +194,9 @@ Every WorkflowExecution is handed wholesale over core NATS to a task-scoped The behaviour is divided across technical packages: -- `internal/store` defines the current `Task` execution record and status - strings and persists mutable state and transition history. +- `internal/domain/workflow` defines the current `Task` execution record and + status strings; `internal/infrastructure/postgres` persists mutable state + and transition history behind `internal/domain/storecontract`. - `internal/daemon` discovers, claims, schedules, retries, recovers, processes, waits for human input, and reconciles pull requests. - `internal/domain/workflow` routes work, executes stages, mutates tasks, applies @@ -205,7 +206,7 @@ The behaviour is divided across technical packages: - `internal/gate` and repository configuration define execution constraints. - `internal/gateway` duplicates task status strings and applies task-control rules. -- forge, worktree, NATS, SQLite, containers, and agent runtimes are concrete +- forge, worktree, NATS, PostgreSQL, containers, and agent runtimes are concrete dependencies mixed into orchestration. ## Current `task` meanings and required replacements @@ -217,7 +218,7 @@ each meaning deliberately; a global text replacement is prohibited. | -------------------------------------------------- | --------------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Forge issue or external task | `internal/forge`, daemon polling | Channel message | Preserve the channel's native reference and payload. Translate it into an Agent System request without importing forge or issue semantics into Workflow. | | Jira issue or other external work item | Messaging infrastructure adapter | Channel message or external artifact | The external system retains its own vocabulary inside its integration. Its record may lead to an accepted work request but is not a Workflow or WorkflowExecution. | -| Persisted `store.Task` row | `internal/store` | WorkflowExecution | Own identity, Workflow ID and version, lifecycle, source-message reference, outputs, attempts, and outcome. Persistence implements the Agent System repository contract. | +| Persisted `workflow.Task` row | `internal/infrastructure/postgres` | WorkflowExecution | Own identity, Workflow ID and version, lifecycle, source-message reference, outputs, attempts, and outcome. Persistence implements the Agent System repository contract. | | Task status and transition history | `store.Status*`, `transitions` | WorkflowExecution lifecycle and domain events | Enforce legal transitions atomically. Domain events and state changes cannot diverge. Remove duplicated status strings from consumers. | | Claimed or queued task | `ClaimNext`, `ClaimByIssue`, `RecoverStale` | Scheduled WorkflowExecution | Claim by WorkflowExecution ID and expected version/state. Crash recovery and leases must not silently create a new execution. | | `workflow.TaskContext` | `internal/domain/workflow` | WorkflowExecution context | Replace the mutable service bag with execution state plus narrow step-required contracts. Complete application configuration and unrelated services are forbidden. | diff --git a/docs/architecture/bindings.md b/docs/architecture/bindings.md index 33b759599..ece2b5174 100644 --- a/docs/architecture/bindings.md +++ b/docs/architecture/bindings.md @@ -29,7 +29,7 @@ type Binding struct { } ``` -Bindings live in the store (SQLite), not `config.toml` -- operators author +Bindings live in the store (PostgreSQL), not `config.toml` -- operators author them from the dashboard while the daemon runs, without a restart. ### Matching @@ -96,8 +96,9 @@ unauthenticated event can never trigger a binding, no matter how well it would otherwise match." Enforced structurally at two points, not just convention: -- `captured_events.authenticated` is a real column (`internal/store/ - captures.go`), set at capture time. +- `captures.authenticated` is a real column + (`internal/infrastructure/postgres/migrations/0002_edastore.sql`), set at + capture time. - `dispatchBindings` (`internal/daemon/daemon.go:479`) walks **authenticated captures whose source has at least one armed binding** -- an unauthenticated capture is never even considered for dispatch. @@ -112,7 +113,7 @@ binding reverts it. Capture writes are bounded independently of the approval gate, so a flood of unauthenticated traffic cannot exhaust disk or drown the inspector: -`InsertCapture` (`internal/store/captures.go`) prunes by both **retention +`InsertCapture` (`internal/infrastructure/postgres/edastore.go`) prunes by both **retention window** (`WHERE received_at < cutoff`) and **max event count** in the same insert transaction. @@ -121,14 +122,14 @@ insert transaction. A playbook-originated task is dispatched into the **same** gate, diff-cap, sandboxed-container, worktree-isolation pipeline every task goes through regardless of origin. Verified precisely, not assumed: `dispatchOneBinding` -calls `EnqueueBindingTask` (`internal/store/store.go`), which wraps +calls `EnqueueBindingTask` (`internal/infrastructure/postgres/store.go`), which wraps `EnqueueChatTask` -- the same direct-to-`tasks`-table enqueue path chat-spawned tasks already use, then stamps `binding_id`/`binding_version` for provenance in a second statement. This is a **different** enqueue mechanism from forge-issue polling's `workintake.TaskEnvelope`/NATS path (`docs/prds/event-sources-and-reactions.md`) -- binding and forge-issue tasks arrive by different producers, but both become an ordinary -`store.Task` row that `ClaimNext` picks up identically, so both get the +`workflow.Task` row that `ClaimNext` picks up identically, so both get the same gate/worktree pipeline downstream regardless of which producer created the row. Payload content is ordinary untrusted prompt input; the sender's text is never treated as a @@ -151,8 +152,8 @@ Two independent protections, not one: ### Secrets at rest `Binding.Secret` (the HMAC shared secret a sender signs with) is encrypted -at rest via AES-GCM (`internal/store/binding_cipher.go`, `archie-core- -t2db.7`) -- a leaked SQLite file does not leak sender-side shared secrets in +at rest via AES-GCM (`internal/infrastructure/bindingcipher`, `archie-core- +t2db.7`) -- a leaked database dump does not leak sender-side shared secrets in plaintext. The envelope is versioned (`bindingEnvelopeVersion`) so a future cipher or KDF change can roll forward without breaking already-encrypted rows, and the AAD binds ciphertext to the binding-secret context so it @@ -164,8 +165,8 @@ documented legacy behaviour, not silent degradation. `dispatchBindings` enqueues at most one task per `(binding, capture)` pair, enforced by a real ledger, not best-effort application logic: -`RecordDispatch` (`internal/store/bindings.go`) does `INSERT OR IGNORE` into -`binding_dispatches`, unique on `(binding_id, capture_id)`; a duplicate +`RecordDispatch` (`internal/infrastructure/postgres/edastore.go`) inserts into +`binding_dispatches`, unique on `(binding, capture)`; a duplicate insert returns the sentinel `ErrAlreadyDispatched`, which the dispatch loop treats as "another cycle raced us," not an error. This guarantee holds **across daemon restarts**, not just within one process's lifetime, because @@ -173,8 +174,8 @@ the ledger is a durable table, not in-memory state. This ledger is also the direct precedent for the EDA playbook engine's execution-time idempotency, now landed (`archie-core-t2db.17` closed): the -`playbook_dispatches` table in `internal/store/playbook_dispatches.go` copies -`binding_dispatches`'s `INSERT OR IGNORE` conventions with a +`playbook_dispatches` table (`RecordPlaybookDispatch`) copies +`binding_dispatches`'s unique-ledger conventions with a `(playbook_id, playbook_version, event_id, action_id)` key (`docs/prds/eda-playbook-engine.md` gap 2). diff --git a/docs/architecture/configuration.md b/docs/architecture/configuration.md index a0e832bec..e66a6ed33 100644 --- a/docs/architecture/configuration.md +++ b/docs/architecture/configuration.md @@ -141,7 +141,7 @@ existing structure combines different owners. | Current values | Current location | Target owner and requirement | | ----------------------------------------------------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `WorkDir`, `DBPath` | root `Config` | Workspace/storage infrastructure settings. `DBPath` is the prefix for independent task and conversation SQLite files. Defaults derive from the platform data directory. | +| `WorkDir`, `StateDir`, `DBPath` | root `Config` | Workspace/storage infrastructure settings, bootstrap-only. `StateDir` holds the embedded NATS store and endpoint file, task logs and the readiness disk target. `DBPath` only locates the legacy SQLite files (`-tasks.sqlite`, `-eda.sqlite`, `-conversations.sqlite`) for `archie-state-store import` and the boot gate that refuses to serve until it has run; it goes when the importer does. Defaults derive from the platform data directory. | | `SkillsDir`, `PluginDir` | root `Config` | Skill and plugin infrastructure settings, independently typed. | | `PollInterval` | root `Config` | Work-intake scheduling setting; not a daemon-wide constant. | | `Label`, `BotUser`, `BotEmail` | root `Config` | Identity and dispatch associations. `BotEmail` may be derived from the selected forge account but remains overrideable. | diff --git a/docs/architecture/identity.md b/docs/architecture/identity.md index c689cc7f0..cba27b4aa 100644 --- a/docs/architecture/identity.md +++ b/docs/architecture/identity.md @@ -293,7 +293,7 @@ Focused tests verify that: ## Current hazards -- SQLite task uniqueness is `(owner, repo, issue_number)` and excludes identity. +- Task uniqueness is `(owner, repo, issue_number)` and excludes identity. Two identities discovering the same forge issue cannot own distinct task records. - NATS deduplication uses `owner/repo/issue` and excludes identity, so discovery diff --git a/docs/architecture/messaging-and-work-intake.md b/docs/architecture/messaging-and-work-intake.md index d7f6b8c93..843102764 100644 --- a/docs/architecture/messaging-and-work-intake.md +++ b/docs/architecture/messaging-and-work-intake.md @@ -104,7 +104,7 @@ The canonical session and message implementation must supply: metadata; - session lifecycle and active-request state; - session-to-agent-instance and parent-session lineage; -- durable SQLite round trips and timestamp validation. +- durable database round trips and timestamp validation. These semantics must use the confirmed Agent, user, channel-binding, and conversation ownership model. @@ -264,7 +264,7 @@ queued. delivery source, so a webhook-delivered issue and a later poll of the same issue dedup through `PublishUnique` instead of double-enqueuing. - Chat-spawned and playbook-binding-dispatched work enqueues through - `EnqueueChatTask`/`EnqueueBindingTask` (`internal/store`), stamped with + `EnqueueChatTask`/`EnqueueBindingTask` (`internal/infrastructure/postgres`), stamped with `binding_id`/`binding_version` for provenance. Webhook intake is opt-in per deployment (`Forge.Intake` = `poll` (default), @@ -287,7 +287,7 @@ model and its threat model are `docs/architecture/bindings.md`. the Agent System. - Source identity is incomplete: sessions use configured bot usernames, tasks use identity strings, and external message IDs are generally absent. -- SQLite and NATS deduplication are based on forge coordinates rather than a +- Database and NATS deduplication are based on forge coordinates rather than a channel-neutral source-message reference. - Replies and acknowledgements have no durable delivery-attempt lifecycle, so partial success and retry behaviour differ by adapter. @@ -316,7 +316,7 @@ model and its threat model are `docs/architecture/bindings.md`. Semantic/vector search over chat messages is an **explicit non-goal** until usage data shows a real need. -Message search is lexical only: session-scoped SQLite FTS5 queries paged via +Message search is lexical only: session-scoped PostgreSQL full-text queries paged via `MessageQuery{Query,Limit,Offset}` → `MessagePage`. Do not add embeddings configuration or vector columns to message records. diff --git a/docs/architecture/migration-decisions.md b/docs/architecture/migration-decisions.md index 326c1f348..49ceb8a70 100644 --- a/docs/architecture/migration-decisions.md +++ b/docs/architecture/migration-decisions.md @@ -182,8 +182,8 @@ the chat service's session-store RPCs and is proven by `TestStoreClientPreservesCanonicalRecords`. It is deliberately unwired. Production `archied` dials the standalone gateway for `ChatContract` only (`composeChatContract` → `gatewayrpc.Dial` returns a `*Client`, not a -`*StoreClient`), and the daemon's session store is in-process SQLite via -`makeTelegramSessionStore`. The test comment that says it mirrors production +`*StoreClient`), and the Gateway's session store is PostgreSQL via +`gateway.NewPostgresSessionStore`. The test comment that says it mirrors production describes the target, not today's tree. Kept rather than deleted because the migration it serves is deferred by this very section, and re-deriving a tested wire adapter once that lands is pure cost. @@ -221,7 +221,10 @@ prohibition list is untouched. not cover (2026-09-22, `archie-core-1ng1`).** The PRD's deletion gate cannot pass today. `cmd/archie-messaging/architecture_test.go` does not exist, and the check that section specifies -- `go list -deps ./cmd/archie-messaging` links -zero banned runtime packages -- reports four: +zero banned runtime packages -- reported four at the time (the task store was +then `internal/store` over `modernc.org/sqlite`; the gate now bans +`modernc.org/sqlite`, `github.com/jackc/pgx`, `internal/infrastructure/postgres` +and `internal/infrastructure/legacyread` in its place): | banned package | PRD category | | --- | --- | @@ -577,9 +580,9 @@ viewer count, and `Broadcast` is a no-op when nobody is watching. Three properties make the cursor sufficient, and a push hub unnecessary: -- **The cursor cannot skip.** `internal/store/store.go` opens the database with - `SetMaxOpenConns(1)`, so writes serialise and a row's `id` is assigned in - commit order. There is no window in which a lower id becomes visible after a +- **The cursor cannot skip.** `internal/infrastructure/postgres` takes an + advisory event-append lock around each insert (`TestEventInsertsCommitInIDOrder`), + so a row's `id` is assigned in commit order. There is no window in which a lower id becomes visible after a higher one, which is the failure mode that normally rules out polling an autoincrement cursor. SSE catch-up already depends on this same property. - **A broadcast is a wakeup, not the payload.** `sseStream.drain` treats each @@ -592,7 +595,7 @@ Three properties make the cursor sufficient, and a push hub unnecessary: - **The table is the only complete fan-out point.** Events reach the store through several paths that never touch the daemon's bus, including the audit event `store.ArchiveTask` writes inside its transaction - (`internal/store/store.go`) and the binding-dispatch failure + (`internal/infrastructure/postgres/store.go`) and the binding-dispatch failure `internal/daemon/daemon.go` inserts directly. A push hub would have to be hooked at every write site and at post-commit, and would silently miss any site added later that forgets to notify. Reading the table misses nothing. diff --git a/docs/architecture/observability.md b/docs/architecture/observability.md index 060716257..86747bf4d 100644 --- a/docs/architecture/observability.md +++ b/docs/architecture/observability.md @@ -165,11 +165,9 @@ would have to guess which side of a boundary an event belongs to across a daemon restart, a stale-run recovery, or two interleaved attempts, and a guess rendered as structure is exactly the false fidelity this surface exists to avoid. -The column reaches an existing database through the presence-gated arm in -`internal/store`'s `migrateTasks` — `eventsSchema` is `CREATE TABLE IF NOT -EXISTS`, so on an existing database the migrator is the *only* thing that adds -it. `PRAGMA user_version` is unchanged because the loop is presence-gated -rather than version-gated, and nothing is backfilled. +The column is part of the events table in +`internal/infrastructure/postgres/migrations/0001_state_store.sql`, and nothing +is backfilled. ### Read surface @@ -310,8 +308,8 @@ attempt". Every part is additive. A capture or a configuration emit that cannot happen leaves the run alone — reporting must never park a task. Reverting the change set removes the endpoints and the page; the extra column and the extra event -kinds are inert to the previous revision (SQLite ignores a column no statement -names, and an unfamiliar event kind already renders as unfamiliar), and there is +kinds are inert to the previous revision (a column no statement names is +ignored, and an unfamiliar event kind already renders as unfamiliar), and there is no data transformation to undo. ## Dependency direction diff --git a/docs/architecture/safe-change-and-recovery.md b/docs/architecture/safe-change-and-recovery.md index 3be17b1d2..65af755ce 100644 --- a/docs/architecture/safe-change-and-recovery.md +++ b/docs/architecture/safe-change-and-recovery.md @@ -102,7 +102,8 @@ silently oscillate between versions or retry forever. ## Database settings: degrade paths and recovery (2026-09) -The control plane owns runtime-tunable settings and stores them in `archie.db`. +The control plane owns runtime-tunable settings and stores them in the State +Store's PostgreSQL database. This replaced a dashboard overlay store that degraded to file config on any failure and had three documented ways back. The replacement has none, and the posture is the opposite: **the daemon fails closed.** @@ -133,6 +134,12 @@ steps: are read, never written. 3. Start archie. +`db_path` exists only to locate these files for the import and the boot gate +below, and goes with the importer. The embedded NATS store, its endpoint file +and task logs live under `state_dir` (default: the data home's `archie` +directory). An install whose `db_path` sat outside the data home sets +`state_dir` to that directory to keep them where they were. + Each serving process checks this at boot, after migrating the schema. With no completion record and a non-empty legacy file it refuses to start and names the files and the import command, rather than serve empty tables over existing @@ -149,18 +156,20 @@ would fail the first one's in-flight turns). The State Store holds its own. Nothing skips the control plane at boot, and the only writer for a resource is `ControlPlaneService.Command` on a running State Store -- which is exactly what the daemon fails closed without. `archie-state-store` therefore carries the -offline recovery commands (`archie-core-6zw0`). They work on the task database -file directly -- the configured `db_path` with `-tasks.sqlite` appended, not -`db_path` itself -- and need neither `archied` nor the Web UI running: `validate` -reads the configuration file the daemon boots with, which is what makes its -verdict the daemon's verdict. +offline recovery commands (`archie-core-6zw0`). They work on the PostgreSQL +database the configuration's `database_url` names +(`internal/app/archied/state_store_recovery_postgres.go`, +`internal/infrastructure/postgres/maintenance.go`) and need neither `archied` +nor the Web UI running: `validate` reads the configuration file the daemon +boots with, which is what makes its verdict the daemon's verdict. One database +backs every service, so the scope is always the whole database. -| Command | What it does | State Store | +| Command | What it does | Serving processes | | --- | --- | --- | -| `backup -db F -out F` | writes a transactionally consistent snapshot (`VACUUM INTO`), verified as a usable store before it is published | may be running | -| `restore -db F -from F` | replaces the database with a snapshot and drops the replaced file's `-wal`/`-shm` | must be stopped | -| `validate -db F [-config C]` | runs the checks that stop archied starting: the file is readable, uncorrupted and no newer than the binary; every stored resource decodes under the definition that owns it; and `configuration.Validate` accepts the settings the daemon would read -- the stored ones, or the seed the State Store writes for a kind the file does not hold yet -- layered onto `C`, the check the daemon reports as `validate database settings` | may be running | -| `rollback -db F -kind K [-revision N]` | replays an earlier revision of a stored resource through the ordinary replace, recording the rollback as a new revision | must be stopped | +| `backup -out F` | runs `pg_dump --format=custom`, reads the archive back with `pg_restore --list` and only then replaces `F` | may be running | +| `restore -from F` | drops tables the snapshot does not hold and runs `pg_restore --clean --single-transaction`; every write made after the snapshot is lost | every one must be stopped | +| `validate [-config C]` | refuses a schema newer than the binary's highest migration; every stored resource decodes under the definition that owns it; and `configuration.Validate` accepts the settings the daemon would read -- the stored ones, or the seed the State Store writes for a kind the database does not hold yet -- layered onto `C`, the check the daemon reports as `validate database settings` | may be running | +| `rollback -kind K [-revision N]` | replays an earlier revision of a stored resource through the ordinary replace, recording the rollback as a new revision | the State Store must be stopped | The two validation layers `validate` reports are not the same check, and it reports both rather than merging them: the write path's own decode is stricter @@ -173,41 +182,21 @@ restoring one is the same replace the dashboard and the channels already use. The offline command is that operation with the store stopped -- the one state in which the daemon's own API cannot serve it. -"Must be stopped" is checked rather than assumed. The serving process takes an -exclusive flock on `.lock` for its whole life, and `restore` and `rollback` -refuse with `task database is owned by another process` while another process -holds it. The read-only commands -- `backup` and `validate` -- take no lock and -are meant to run against a serving store: `backup` because the update installer -runs inside the process it is replacing and cannot stop the State Store, and -`validate` because an operator checks a running deployment with it. That -`backup` exemption is only sound while backup never replaces the database, so -naming the database as its own `-out` is refused. `scripts/archie-update-install` -takes its pre-flight snapshot through this subcommand rather than through its -own `sqlite3` shell, so the installer and an operator run one implementation. +"Must be stopped" is checked rather than assumed. Each serving process holds a +Postgres ownership lock for its whole life. `rollback` holds the `state-store` +claim for the replay. `restore` claims every service role's lock (`state-store` +and `gateway`) without waiting and refuses if any is held or the check cannot +complete. The read-only commands -- `backup` and `validate` -- take no claim and +are meant to run against a serving database: `backup` because the update +installer runs inside the process it is replacing and cannot stop the State +Store, and `validate` because an operator checks a running deployment with it. +`scripts/archie-update-install` takes its pre-flight snapshot through this +subcommand, so the installer and an operator run one implementation. `rollback` is the surgical route back from a bad stored value; `restore` is the blunt one, and is also the way back from a schema migration a release cannot grow out of. -### The same commands on PostgreSQL - -Without `-db`, each command works on the PostgreSQL database the configuration's -`database_url` names (`internal/app/archied/state_store_recovery_postgres.go`, -`internal/infrastructure/postgres/maintenance.go`). One database backs every -service, so the scope is always the whole database. - -- `backup -out F` runs `pg_dump --format=custom`, reads the archive back with - `pg_restore --list` and only then replaces `F`. It takes no claim and runs - against a serving database. -- `restore -from F` is destructive and offline. It claims every service role's - ownership lock (`state-store` and `gateway`) without waiting and refuses if - any is held or the check cannot complete, then drops tables the snapshot does - not hold and runs `pg_restore --clean --single-transaction`. Every write made - after the snapshot is lost. -- `validate` refuses a schema newer than the binary's highest migration, then - runs the same two validation layers. -- `rollback` holds the `state-store` claim for the replay. - There is no automatic schema rollback. goose `Down` drops tables, so it is data loss, not a rollback. `scripts/archie-update-install` refuses a daemon update it cannot snapshot first, and on a failed update `scripts/archie-update-watchdog` diff --git a/docs/data/generated/dev-docs.json b/docs/data/generated/dev-docs.json index 4736e3ce6..f33bfe1f8 100644 --- a/docs/data/generated/dev-docs.json +++ b/docs/data/generated/dev-docs.json @@ -16,19 +16,19 @@ "url": "/dev/architecture/agent-system/", "title": "Agent System", "section": "Architecture", - "body": "# Agent System\n\n**Status:** Current-state investigation in progress \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Domain framing\n\nThe domain is the Agent System.\n\n`Task` is not a core Agent System concept. Forge tasks, forge issues, Jira\nissues, email, Telegram messages, webhooks, and future external inputs are\nchannel-specific messages. Channels translate those messages into requests to\nthe Agent System; external issue or task models MUST NOT shape execution state.\n\nThe current `store.Task` is a persisted workflow execution. Treating that row as\na task-domain aggregate would incorrectly turn Archie into an agentic issue\nboard.\n\n## Canonical agent interaction model\n\nAn **Agent** is a persistent assistant selected for an interaction. It is not a\nmodel invocation and is not a WorkflowExecution.\n\nEach Agent has its own:\n\n- immutable acting `IdentityID` for ownership and audit;\n- durable memory and learned information about the user;\n- perspective, specialisation, instructions, and communication behaviour;\n- conversation history and channel bindings;\n- available tools, capabilities, and policy;\n- continuity across model or provider changes.\n\nThe underlying model is a replaceable runtime dependency. Changing a model,\nprovider, or model version MUST NOT create a new Agent, discard its memory, or\nchange ownership of its history.\n\nA user MAY create multiple Agents. Each Agent develops its own memory and view\nof the user from the interactions and responsibilities assigned to it. Memories\nMUST NOT be silently pooled merely because Agents belong to the same user or\napplication.\n\nAgent and Identity are separate aggregates. Every Agent references exactly one\nacting IdentityID, and no two Agents share an agent identity. User, system, and\nservice-account identities may exist without an Agent.\n\nAn action initiated through an Agent records the Agent's acting IdentityID and,\nwhen applicable, the initiating user's IdentityID. A future fully automated\nWorkflowExecution MAY use an explicitly selected service-account identity for a\nconcrete unattended use case; this is not the default execution identity.\n\n## Workflow execution identity\n\nA Workflow is owned by an IdentityID. By default, every WorkflowExecution acts\nas the Workflow owner's identity.\n\nFuture delegated execution MAY add an explicit `RunAsIdentityID` assignment to a\nWorkflow. This allows a Workflow owner to select another user or service-account\nidentity for execution without changing Workflow ownership. Until that\ncapability is deliberately introduced, owner identity and execution identity are\nthe same.\n\nWorkflowExecution audit records MUST preserve the Workflow owner, effective\nexecution identity, initiating user when one exists, and any explicit delegation\nthat selected a different run-as identity.\n\n## Memory scopes\n\nThe Agent System requires four distinct durable memory scopes:\n\n- **global shared memory**: information intentionally available across the\n Archie application;\n- **agent-wide memory**: general expertise, operating knowledge, and perspective\n owned by one Agent and safe for that Agent to use across users;\n- **user-wide memory**: information associated with one user and available to\n authorized Agents serving that user;\n- **Agent-user relationship memory**: private history and learned context from\n one particular Agent's relationship with one particular user.\n\nAgent-wide memory is private to its owning Agent by default. Another Agent MUST\nNOT read, inherit, search, or mutate it implicitly.\n\nAgent-user relationship memory is addressed by both AgentID and User IdentityID.\nNeither another Agent serving the same user nor another user speaking to the\nsame Agent may access it implicitly.\n\nArchie is a multi-user system even when initially deployed for one person.\nMultiple users, including family members, may interact with the same configured\nAgents. Per-user memory is therefore an isolation and personalization boundary,\nnot merely an attribution tag on the primary user's records.\n\nAn Agent MUST resolve the initiating user before assembling memory context.\nMemory belonging to one user MUST NOT become visible to another user merely\nbecause both users interact with the same Agent.\n\nShared memory does not erase provenance. Every memory record retains its scope,\nauthor or acting identity, originating user when applicable, source, and\nrevision history. The rules for selecting a write scope, granting access\nto global and user-wide memory, and authorizing explicit cross-scope sharing are\nsettled by [`docs/prds/memory-engine-unification.md`](/dev/prds/memory-engine-unification/)\nand recorded in [`migration-decisions.md`](/dev/architecture/migration-decisions/#5-memory-placement-and-storage)\nsection 5.\n\nAn Agent interaction may answer conversationally, use a tool or capability, read\nor send messages, retain information, or propose durable workflow-backed work.\nNone of these operations inherently requires a WorkflowExecution.\n\n## Agent and Workflow domain boundary\n\nAgent and Workflow are separate domains:\n\n```text\ninternal/domain/agent\ninternal/domain/workflow\n```\n\nThe Agent domain owns persistent assistants, their specialisation, instructions,\nmodel-independent continuity, capability context, and associations with identity\nand memory.\n\nThe Workflow domain owns reusable, user- or Agent-defined behaviour and every\ndurable execution of that behaviour. An Agent may invoke or define a Workflow,\nbut the Workflow is not part of the Agent aggregate and does not require a\nconversational Agent interaction to execute.\n\n## Canonical workflow model\n\nThe Workflow feature uses these canonical terms:\n\n- **Workflow**: a reusable, versioned definition of how an agent behaves and\n performs a kind of work.\n- **WorkflowExecution**: one durable execution of a specific Workflow version.\n- **WorkflowStep**: one defined operation in a Workflow.\n- **StepExecution**: the recorded execution and outcome of one WorkflowStep.\n- **Attempt**: a retry of the same WorkflowExecution, not a new execution.\n\nA Workflow MAY structure any repeatable agent behaviour, including personal\nassistance and scheduled routines. For example, a morning email review may\ncombine a schedule event, message retrieval tool, LLM classification and\nsummarisation, and delivery of the resulting rundown. Workflow is therefore not\nrestricted to forge work, source changes, or requests submitted by a user at\nexecution time.\n\nWorkflows are extensible through Workflow-owned plugins. The Workflow domain\ndefines the typed plugin registration, definition, validation, versioning, and\nexecution contracts. Users and Agents may define reusable Workflows through\nthose contracts.\n\nThe generic plugin contract remains metadata-only. It MUST NOT own Workflow\nsemantics or replace the Workflow registry with generic hooks.\n\n`Workflow` and `WorkflowExecution` are the names used in architecture, generated\nreference, configuration, commands, events, and external contracts. The\nimplementation MAY use package qualification to remain concise:\n\n```go\nworkflow.Definition\nworkflow.Execution\nworkflow.Step\nworkflow.StepExecution\n```\n\nThe unqualified name `Execution` MUST NOT be used in cross-domain contracts or\ndocumentation because it is ambiguous among workflow, tool, command, policy, and\nservice execution.\n\n## Current workflow-execution feature\n\nThe current feature takes a channel request through agent execution and a\ndurable outcome.\n\nCurrent execution state includes source-message references, owning identity,\nrepository, selected workflow and stage, agent output, branch and pull-request\nreferences, token and iteration counts, retry information, human-wait\ninformation, and failure details.\n\nThe current `store.Task` lifecycle uses:\n\n```text\nqueued -\u003e running\nrunning -\u003e waiting_human | pr_open | merged | parked | closed_wont_do\nwaiting_human -\u003e queued | closed_wont_do\nparked -\u003e queued | dead\npr_open -\u003e merged | rejected\nrunning after crash -\u003e queued\n```\n\nWorkflows select and sequence deterministic and agent-run stages. Implement,\nTDD, feasibility, and bootstrap are current Workflow definitions.\nEvery WorkflowExecution is handed wholesale over core NATS to a task-scoped\n`archie-agent` container, which owns the workflow stage loop.\n\n## Current workflow-feature placement\n\nThe behaviour is divided across technical packages:\n\n- `internal/store` defines the current `Task` execution record and status\n strings and persists mutable state and transition history.\n- `internal/daemon` discovers, claims, schedules, retries, recovers, processes,\n waits for human input, and reconciles pull requests.\n- `internal/domain/workflow` routes work, executes stages, mutates tasks, applies\n outcomes, parks failures, and performs forge/worktree operations.\n- `internal/agentexec` defines and runs individual autonomous-stage protocols.\n- `internal/taskrun` hands a complete WorkflowExecution to a container worker.\n- `internal/gate` and repository configuration define execution constraints.\n- `internal/gateway` duplicates task status strings and applies task-control\n rules.\n- forge, worktree, NATS, SQLite, containers, and agent runtimes are concrete\n dependencies mixed into orchestration.\n\n## Current `task` meanings and required replacements\n\nThe existing code uses `task` for unrelated concepts. Migration MUST resolve\neach meaning deliberately; a global text replacement is prohibited.\n\n| Current meaning | Representative locations | Target concept | Requirements |\n| -------------------------------------------------- | --------------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Forge issue or external task | `internal/forge`, daemon polling | Channel message | Preserve the channel's native reference and payload. Translate it into an Agent System request without importing forge or issue semantics into Workflow. |\n| Jira issue or other external work item | Messaging infrastructure adapter | Channel message or external artifact | The external system retains its own vocabulary inside its integration. Its record may lead to an accepted work request but is not a Workflow or WorkflowExecution. |\n| Persisted `store.Task` row | `internal/store` | WorkflowExecution | Own identity, Workflow ID and version, lifecycle, source-message reference, outputs, attempts, and outcome. Persistence implements the Agent System repository contract. |\n| Task status and transition history | `store.Status*`, `transitions` | WorkflowExecution lifecycle and domain events | Enforce legal transitions atomically. Domain events and state changes cannot diverge. Remove duplicated status strings from consumers. |\n| Claimed or queued task | `ClaimNext`, `ClaimByIssue`, `RecoverStale` | Scheduled WorkflowExecution | Claim by WorkflowExecution ID and expected version/state. Crash recovery and leases must not silently create a new execution. |\n| `workflow.TaskContext` | `internal/domain/workflow` | WorkflowExecution context | Replace the mutable service bag with execution state plus narrow step-required contracts. Complete application configuration and unrelated services are forbidden. |\n| `taskrun.Request` and `taskrun.Response` | `internal/taskrun`; consumed by `internal/app/agentworker`, decoded by `internal/infrastructure/agenttransport/nats` | WorkflowExecution handoff | This is the versioned worker contract for executing a WorkflowExecution. It carries domain-owned execution data, not store rows or global configuration DTOs. |\n| Per-stage `agentexec.Request.TaskID` | `internal/agentexec` | WorkflowExecutionID correlation | An agent invocation is a bounded WorkflowStep execution. Correlate request/result with WorkflowExecutionID, StepExecutionID, attempt, and protocol version. |\n| NATS `TaskMessage` and `archie.task.*` | `internal/domain/workintake` (subjects, `TaskEnvelope`); `internal/infrastructure/agenttransport/nats` (transport) -- `internal/nats` no longer exists; whether this row's \"current meaning\" itself is now superseded by `TaskEnvelope` needs a closer look, not asserted here | Channel request or WorkflowExecution transport | Discovery messages and execution dispatch are different contracts and subjects. Domain meaning stays outside NATS infrastructure. Identity and source-message idempotency must be explicit. |\n| `taskDispatcher` work item | `internal/daemon` | WorkflowExecution scheduling | Scheduling operates on execution references and declared concurrency policy. Repository serialization is policy, not identity derived from an incomplete task struct. |\n| `container.TaskPayload` and `task.json` | `internal/container`; read by `internal/infrastructure/agentboot` | WorkflowExecution brief | Rename the boot artifact and schema. It identifies the mounted execution and required workspace only; it is not an alternative execution definition. Missing or malformed input must fail startup before the worker subscribes. |\n| `storage.TaskRef` | `internal/storage` | WorkflowExecution workspace lease | Storage receives the minimal execution/workspace reference and policy required to provision and release resources. It does not receive the WorkflowExecution aggregate. |\n| Gateway `TaskCreator` | `internal/gateway` | Accepted work request handoff | A channel may carry a request for durable workflow-backed work, but it does not request or create a WorkflowExecution directly. Work intake admits and routes the request; the Agent System consumes the accepted work request and is the sole owner of selecting a Workflow version and creating the WorkflowExecution. Ordinary agent interactions do not require this handoff. |\n| Gateway `TaskController` and `ChatTaskStatus` | `internal/gateway/tasks.go` | WorkflowExecution commands and query view | Approve, cancel, and status use Agent System contracts. Channel code does not duplicate lifecycle constants or decide transitions. |\n| Gateway `TaskProfile` | `internal/gateway/tasks.go`, `cmd/archied` | Channel/identity execution defaults | This is configuration selecting allowed source targets and default Workflow inputs, not task state. Split it into settings owned by the relevant channel and Agent System features. |\n| `config.TaskConfig` and `TaskForge` | `internal/config` | Worker execution settings and channel/service references | Dissolve with `internal/config`. Each field moves to its owning runtime settings or versioned WorkflowExecution handoff contract. Secrets and external configuration forms do not enter the domain. |\n| `events.Event.TaskID`, task event timelines | `internal/events`, stores, web UI | WorkflowExecutionID and WorkflowExecution events | Authoritative events are Agent System events. Observability and timelines project those events rather than inventing parallel generic task events. |\n| Web UI task lists and counts | `internal/webui`, store query APIs | WorkflowExecution read model | Dashboard status, history, and counts are projections of WorkflowExecution state and events. The UI does not depend on persistence records. |\n| Task-scoped forge/store/worktree RPC | `forgerpc`, `storerpc`, `worktreerpc`, `registerTaskRPCServers` | Execution-scoped capability ports | Worker calls use narrow domain-required service contracts, carry WorkflowExecutionID and IdentityID, and resolve the owning capability binding. Root-client fallback is forbidden. |\n| Per-task worktree, branch, volume, and cleanup | daemon, worktree, storage, container | WorkflowExecution workspace | Every WorkflowExecution receives isolated workspace state. Workspace lifecycle is explicit, identity-aware, recoverable, and controlled outside the model. |\n| Task retry, dead, waiting-human, PR reconciliation | daemon, workflow, gateway | WorkflowExecution lifecycle policies | Retry, waiting, cancellation, reconciliation, and terminal outcomes are Agent System behaviour. Channels and infrastructure report facts or submit commands; they do not mutate state. |\n| Task-run readiness and retry timeouts | daemon taskrun request loop | Worker-dispatch settings and policy | These settings govern WorkflowExecution handoff availability. They are not Workflow data and must be owned by the worker-dispatch capability. |\n\n### Terms that do not migrate\n\n- The `task` command in `Taskfile.yml` is the external Task build tool and keeps\n its name.\n- A forge may call its own records issues or tasks inside its channel adapter.\n- General prose may use “task” informally only when it clearly refers to an\n external system. Agent System contracts and generated reference MUST use the\n canonical Workflow vocabulary.\n\n## Current WorkflowExecution data inventory\n\nThis inventory is the migration ledger for data currently spread across\n`store.Task`, database-only columns, `workflow.TaskContext`, worker messages,\nand surrounding configuration. Implementation SHOULD NOT require another\nrepository-wide search to determine what existing execution data must be\npreserved or deliberately removed.\n\n| Current data | Current representation | Required target |\n| ---------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |\n| Execution identity | `Task.ID`, `task_id`, NATS subjects, logs | Immutable WorkflowExecutionID used consistently by state, events, workers, projections, and capability calls |\n| Owning actor | `Task.Identity` string | Canonical IdentityID |\n| Source type | `Task.Source` (`forge` or `chat`) | Typed channel and SourceMessageReference; no closed source enum in WorkflowExecution |\n| Source address | `Owner`, `Repo`, `IssueNumber`; synthetic chat issue number | Channel-owned source reference with native message/thread/object identifiers; synthetic identifiers removed |\n| Requested work | `Title`, `Body`, `Labels` | Immutable execution input captured from the source request; channel-specific metadata retained in the source envelope |\n| Workflow selection | `Workflow` string | Immutable WorkflowID plus selected WorkflowVersion |\n| Lifecycle | `Status` string | Enforced WorkflowExecution state |\n| Current position | mutable `Stage` string | Derived current WorkflowStep reference plus durable StepExecution history |\n| Attempts | `Attempt`, `RetryCount` | Execution attempt history and retry-policy evidence; distinguish worker delivery retry from workflow retry |\n| Failure | `ParkReason`, transition detail, response errors | Typed failure classification, evidence, retryability, and resulting lifecycle event |\n| Human wait | `WatchCommentID`, `waiting_human` | Channel-neutral wait condition and correlation reference; forge comment IDs stay in the forge channel |\n| Workspace | `Dir`, `Branch`, worktree path, `task.json` | Execution workspace reference and independently managed workspace lifecycle |\n| Change outcome | `BuildSummary`, `BuildNoChanges`, changed files | Step outputs and final WorkflowExecution outcome |\n| Reproduction evidence | `ReproProof` | Typed TDD step evidence retained with the StepExecution and final outcome |\n| Planning output | `Plan` | Typed step output referenced by later steps rather than a special aggregate field |\n| Notes | `Notes`, appended agent notes | Ordered execution annotations with actor, source step, and timestamp |\n| Pull request | `PRNumber`, forge host/repository assumptions | Forge-channel delivery reference and reconciliation state |\n| Agent usage | `TokensUsed`, `Iterations` | StepExecution usage measurements aggregated into execution projections |\n| Decision capture | private feasibility `decision`, capture-tool output | Typed WorkflowStep output defined by that Workflow version |\n| Terminal outcome | mutable `Outcome{Status, Detail}` | Explicit WorkflowExecution outcome command/event validated by the state machine |\n| Skill context | `SkillBody`, `SkillPlugins` | Resolved Workflow version dependencies or execution inputs, with provenance |\n| Agent prompt context | `SystemPrompt`, `ExtraRules`, mission strings | Versioned WorkflowStep definition plus resolved execution context; mutable global prompt callbacks removed |\n| Gates and protection | `Gate`, `Preflight`, `Protection`, repository config | Agent System policy definitions resolved for the WorkflowExecution and enforced outside the agent |\n| Budgets | steps, tokens, wall clock, gate failures | Execution and StepExecution policy values with measured usage and evidence |\n| Tool access | capture tools, registry, guardrails, skill plugins | Resolved step capability set and policy; global registry pointers do not enter execution state |\n| Persistence timestamps | SQL `created_at` and `updated_at` | Domain timestamps and version used for optimistic concurrency and audit |\n| Transition history | SQL `transitions` and generic events | Authoritative WorkflowExecution domain events stored atomically with state |\n\n## Current runtime responsibilities to unify\n\nThe following behaviours currently act on the same WorkflowExecution but are\nimplemented in different packages:\n\n- accept a request from a channel and enforce idempotency;\n- select and version a Workflow;\n- create, queue, claim, schedule, and execute a WorkflowExecution;\n- serialize same-repository execution where policy requires it;\n- resolve identity-specific capabilities;\n- prepare, retain, recover, and clean up the workspace;\n- run deterministic and agent-backed WorkflowSteps;\n- enforce gates, budgets, protected paths, tool constraints, and result review;\n- persist state and StepExecution outputs;\n- publish authoritative lifecycle events;\n- wait for channel input and resume;\n- classify failure, retry, park, mark exhausted, cancel, or recover after crash;\n- deliver and reconcile forge changes;\n- project status, history, counts, and audit for channels and the web UI;\n- hand execution to a worker without moving domain ownership to the worker.\n\n## Current workflow-feature hazards\n\n- There is no enforced WorkflowExecution state machine. `Store.Transition`\n ignores the supplied `from` state when updating the row.\n- State update and transition-history insertion are separate operations, so\n state and audit can diverge.\n- Callers frequently discard persistence and transition errors.\n- Gateway task control duplicates lifecycle status constants by hand.\n- `workflow.TaskContext` is a mutable service bag containing complete\n configuration plus forge, store, worktree, agent, event bus, logging, memory,\n guardrails, skills, and scratch state.\n- Lifecycle changes occur in both daemon and workflow code.\n- Workflow orchestration runs inside task-scoped agent containers while the\n daemon retains lifecycle, admission, and authority-bearing RPC ownership.\n- Forge-backed and chat-created work share one record through source flags and\n synthetic issue numbers.\n- Repository labels currently select workflows and mirror task state, coupling\n domain transitions to forge presentation.\n- Failure generally becomes `parked`, but failure classification and\n retryability are not represented explicitly.\n- Generic observability events are emitted alongside state changes rather than\n being the authoritative domain events.\n\n## Existing invariants to preserve\n\n- The model never performs git operations directly.\n- Deterministic policy and gates cannot be waived by the agent.\n- TDD captures a failing reproduction before implementation and protects the\n test during the fix.\n- Every run ends in an explicit outcome; definition errors and stage errors do\n not disappear silently.\n- Crash recovery requeues interrupted work.\n- Agent execution crosses a versioned data boundary.\n- Forge, persistence, workspace, transport, container, and model runtimes remain\n replaceable infrastructure.\n- Work from the same repository is serialized by default.\n\n## Required migration\n\n- Rename the current persisted `store.Task` concept to WorkflowExecution.\n- Replace task lifecycle terminology in core commands, events, persistence,\n worker contracts, logs, and generated reference.\n- Preserve external source terminology only inside its channel adapter.\n- Replace synthetic issue numbers with explicit source-message references.\n- Associate every WorkflowExecution with an immutable Workflow identifier and\n version.\n- Record StepExecutions explicitly rather than treating one mutable `stage`\n field as complete execution history.\n- Keep Attempt as retry state within the same WorkflowExecution.\n\n## Curator isolation invariant\n\nDecided 2026-08-05\n([#435](https://github.com/samcharles93/archie-core/issues/435) epic,\n[#98](https://github.com/samcharles93/archie-core/issues/98) runtime).\nRecovered 2026-08-09 from the pre-migration issue tracker.\n\n**Curators are peers, not dependencies.** They must never block, delay, or take\ndown chat turns, agent runs, or the daemon.\n\n1. Registry-owned goroutines, one per curator, with per-pass `recover()`,\n per-pass budgets (max steps, tokens, timeout), and a configurable maximum of\n concurrent passes (default 1) so curators cannot saturate the shared LLM\n runtime against chat.\n2. Wake events ride the in-process `events.Bus`, which is mutex fan-out with\n bounded per-subscriber buffers that **drop** on overflow\n (`internal/events/events.go`). A slow curator can never backpressure the chat\n publisher.\n3. Wake is a best-effort nudge. The trigger is a deterministic read of persisted\n primary-input state at pass time, so a dropped wake only delays a run to the\n next check-in and never corrupts trigger accounting\n ([#22](https://github.com/samcharles93/archie-core/issues/22)).\n4. The curator registry lifecycle is independent of the agent loop. Daemon\n shutdown cancels in-flight passes and waits bounded.\n\nChat turns keep their per-session lanes; agent runs keep their workflow\ngoroutines and containers. Rule of thumb: **no code path a curator executes may\never appear on a chat turn's critical path.**\n" + "body": "# Agent System\n\n**Status:** Current-state investigation in progress \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Domain framing\n\nThe domain is the Agent System.\n\n`Task` is not a core Agent System concept. Forge tasks, forge issues, Jira\nissues, email, Telegram messages, webhooks, and future external inputs are\nchannel-specific messages. Channels translate those messages into requests to\nthe Agent System; external issue or task models MUST NOT shape execution state.\n\nThe current `store.Task` is a persisted workflow execution. Treating that row as\na task-domain aggregate would incorrectly turn Archie into an agentic issue\nboard.\n\n## Canonical agent interaction model\n\nAn **Agent** is a persistent assistant selected for an interaction. It is not a\nmodel invocation and is not a WorkflowExecution.\n\nEach Agent has its own:\n\n- immutable acting `IdentityID` for ownership and audit;\n- durable memory and learned information about the user;\n- perspective, specialisation, instructions, and communication behaviour;\n- conversation history and channel bindings;\n- available tools, capabilities, and policy;\n- continuity across model or provider changes.\n\nThe underlying model is a replaceable runtime dependency. Changing a model,\nprovider, or model version MUST NOT create a new Agent, discard its memory, or\nchange ownership of its history.\n\nA user MAY create multiple Agents. Each Agent develops its own memory and view\nof the user from the interactions and responsibilities assigned to it. Memories\nMUST NOT be silently pooled merely because Agents belong to the same user or\napplication.\n\nAgent and Identity are separate aggregates. Every Agent references exactly one\nacting IdentityID, and no two Agents share an agent identity. User, system, and\nservice-account identities may exist without an Agent.\n\nAn action initiated through an Agent records the Agent's acting IdentityID and,\nwhen applicable, the initiating user's IdentityID. A future fully automated\nWorkflowExecution MAY use an explicitly selected service-account identity for a\nconcrete unattended use case; this is not the default execution identity.\n\n## Workflow execution identity\n\nA Workflow is owned by an IdentityID. By default, every WorkflowExecution acts\nas the Workflow owner's identity.\n\nFuture delegated execution MAY add an explicit `RunAsIdentityID` assignment to a\nWorkflow. This allows a Workflow owner to select another user or service-account\nidentity for execution without changing Workflow ownership. Until that\ncapability is deliberately introduced, owner identity and execution identity are\nthe same.\n\nWorkflowExecution audit records MUST preserve the Workflow owner, effective\nexecution identity, initiating user when one exists, and any explicit delegation\nthat selected a different run-as identity.\n\n## Memory scopes\n\nThe Agent System requires four distinct durable memory scopes:\n\n- **global shared memory**: information intentionally available across the\n Archie application;\n- **agent-wide memory**: general expertise, operating knowledge, and perspective\n owned by one Agent and safe for that Agent to use across users;\n- **user-wide memory**: information associated with one user and available to\n authorized Agents serving that user;\n- **Agent-user relationship memory**: private history and learned context from\n one particular Agent's relationship with one particular user.\n\nAgent-wide memory is private to its owning Agent by default. Another Agent MUST\nNOT read, inherit, search, or mutate it implicitly.\n\nAgent-user relationship memory is addressed by both AgentID and User IdentityID.\nNeither another Agent serving the same user nor another user speaking to the\nsame Agent may access it implicitly.\n\nArchie is a multi-user system even when initially deployed for one person.\nMultiple users, including family members, may interact with the same configured\nAgents. Per-user memory is therefore an isolation and personalization boundary,\nnot merely an attribution tag on the primary user's records.\n\nAn Agent MUST resolve the initiating user before assembling memory context.\nMemory belonging to one user MUST NOT become visible to another user merely\nbecause both users interact with the same Agent.\n\nShared memory does not erase provenance. Every memory record retains its scope,\nauthor or acting identity, originating user when applicable, source, and\nrevision history. The rules for selecting a write scope, granting access\nto global and user-wide memory, and authorizing explicit cross-scope sharing are\nsettled by [`docs/prds/memory-engine-unification.md`](/dev/prds/memory-engine-unification/)\nand recorded in [`migration-decisions.md`](/dev/architecture/migration-decisions/#5-memory-placement-and-storage)\nsection 5.\n\nAn Agent interaction may answer conversationally, use a tool or capability, read\nor send messages, retain information, or propose durable workflow-backed work.\nNone of these operations inherently requires a WorkflowExecution.\n\n## Agent and Workflow domain boundary\n\nAgent and Workflow are separate domains:\n\n```text\ninternal/domain/agent\ninternal/domain/workflow\n```\n\nThe Agent domain owns persistent assistants, their specialisation, instructions,\nmodel-independent continuity, capability context, and associations with identity\nand memory.\n\nThe Workflow domain owns reusable, user- or Agent-defined behaviour and every\ndurable execution of that behaviour. An Agent may invoke or define a Workflow,\nbut the Workflow is not part of the Agent aggregate and does not require a\nconversational Agent interaction to execute.\n\n## Canonical workflow model\n\nThe Workflow feature uses these canonical terms:\n\n- **Workflow**: a reusable, versioned definition of how an agent behaves and\n performs a kind of work.\n- **WorkflowExecution**: one durable execution of a specific Workflow version.\n- **WorkflowStep**: one defined operation in a Workflow.\n- **StepExecution**: the recorded execution and outcome of one WorkflowStep.\n- **Attempt**: a retry of the same WorkflowExecution, not a new execution.\n\nA Workflow MAY structure any repeatable agent behaviour, including personal\nassistance and scheduled routines. For example, a morning email review may\ncombine a schedule event, message retrieval tool, LLM classification and\nsummarisation, and delivery of the resulting rundown. Workflow is therefore not\nrestricted to forge work, source changes, or requests submitted by a user at\nexecution time.\n\nWorkflows are extensible through Workflow-owned plugins. The Workflow domain\ndefines the typed plugin registration, definition, validation, versioning, and\nexecution contracts. Users and Agents may define reusable Workflows through\nthose contracts.\n\nThe generic plugin contract remains metadata-only. It MUST NOT own Workflow\nsemantics or replace the Workflow registry with generic hooks.\n\n`Workflow` and `WorkflowExecution` are the names used in architecture, generated\nreference, configuration, commands, events, and external contracts. The\nimplementation MAY use package qualification to remain concise:\n\n```go\nworkflow.Definition\nworkflow.Execution\nworkflow.Step\nworkflow.StepExecution\n```\n\nThe unqualified name `Execution` MUST NOT be used in cross-domain contracts or\ndocumentation because it is ambiguous among workflow, tool, command, policy, and\nservice execution.\n\n## Current workflow-execution feature\n\nThe current feature takes a channel request through agent execution and a\ndurable outcome.\n\nCurrent execution state includes source-message references, owning identity,\nrepository, selected workflow and stage, agent output, branch and pull-request\nreferences, token and iteration counts, retry information, human-wait\ninformation, and failure details.\n\nThe current `store.Task` lifecycle uses:\n\n```text\nqueued -\u003e running\nrunning -\u003e waiting_human | pr_open | merged | parked | closed_wont_do\nwaiting_human -\u003e queued | closed_wont_do\nparked -\u003e queued | dead\npr_open -\u003e merged | rejected\nrunning after crash -\u003e queued\n```\n\nWorkflows select and sequence deterministic and agent-run stages. Implement,\nTDD, feasibility, and bootstrap are current Workflow definitions.\nEvery WorkflowExecution is handed wholesale over core NATS to a task-scoped\n`archie-agent` container, which owns the workflow stage loop.\n\n## Current workflow-feature placement\n\nThe behaviour is divided across technical packages:\n\n- `internal/domain/workflow` defines the current `Task` execution record and\n status strings; `internal/infrastructure/postgres` persists mutable state\n and transition history behind `internal/domain/storecontract`.\n- `internal/daemon` discovers, claims, schedules, retries, recovers, processes,\n waits for human input, and reconciles pull requests.\n- `internal/domain/workflow` routes work, executes stages, mutates tasks, applies\n outcomes, parks failures, and performs forge/worktree operations.\n- `internal/agentexec` defines and runs individual autonomous-stage protocols.\n- `internal/taskrun` hands a complete WorkflowExecution to a container worker.\n- `internal/gate` and repository configuration define execution constraints.\n- `internal/gateway` duplicates task status strings and applies task-control\n rules.\n- forge, worktree, NATS, PostgreSQL, containers, and agent runtimes are concrete\n dependencies mixed into orchestration.\n\n## Current `task` meanings and required replacements\n\nThe existing code uses `task` for unrelated concepts. Migration MUST resolve\neach meaning deliberately; a global text replacement is prohibited.\n\n| Current meaning | Representative locations | Target concept | Requirements |\n| -------------------------------------------------- | --------------------------------------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Forge issue or external task | `internal/forge`, daemon polling | Channel message | Preserve the channel's native reference and payload. Translate it into an Agent System request without importing forge or issue semantics into Workflow. |\n| Jira issue or other external work item | Messaging infrastructure adapter | Channel message or external artifact | The external system retains its own vocabulary inside its integration. Its record may lead to an accepted work request but is not a Workflow or WorkflowExecution. |\n| Persisted `workflow.Task` row | `internal/infrastructure/postgres` | WorkflowExecution | Own identity, Workflow ID and version, lifecycle, source-message reference, outputs, attempts, and outcome. Persistence implements the Agent System repository contract. |\n| Task status and transition history | `store.Status*`, `transitions` | WorkflowExecution lifecycle and domain events | Enforce legal transitions atomically. Domain events and state changes cannot diverge. Remove duplicated status strings from consumers. |\n| Claimed or queued task | `ClaimNext`, `ClaimByIssue`, `RecoverStale` | Scheduled WorkflowExecution | Claim by WorkflowExecution ID and expected version/state. Crash recovery and leases must not silently create a new execution. |\n| `workflow.TaskContext` | `internal/domain/workflow` | WorkflowExecution context | Replace the mutable service bag with execution state plus narrow step-required contracts. Complete application configuration and unrelated services are forbidden. |\n| `taskrun.Request` and `taskrun.Response` | `internal/taskrun`; consumed by `internal/app/agentworker`, decoded by `internal/infrastructure/agenttransport/nats` | WorkflowExecution handoff | This is the versioned worker contract for executing a WorkflowExecution. It carries domain-owned execution data, not store rows or global configuration DTOs. |\n| Per-stage `agentexec.Request.TaskID` | `internal/agentexec` | WorkflowExecutionID correlation | An agent invocation is a bounded WorkflowStep execution. Correlate request/result with WorkflowExecutionID, StepExecutionID, attempt, and protocol version. |\n| NATS `TaskMessage` and `archie.task.*` | `internal/domain/workintake` (subjects, `TaskEnvelope`); `internal/infrastructure/agenttransport/nats` (transport) -- `internal/nats` no longer exists; whether this row's \"current meaning\" itself is now superseded by `TaskEnvelope` needs a closer look, not asserted here | Channel request or WorkflowExecution transport | Discovery messages and execution dispatch are different contracts and subjects. Domain meaning stays outside NATS infrastructure. Identity and source-message idempotency must be explicit. |\n| `taskDispatcher` work item | `internal/daemon` | WorkflowExecution scheduling | Scheduling operates on execution references and declared concurrency policy. Repository serialization is policy, not identity derived from an incomplete task struct. |\n| `container.TaskPayload` and `task.json` | `internal/container`; read by `internal/infrastructure/agentboot` | WorkflowExecution brief | Rename the boot artifact and schema. It identifies the mounted execution and required workspace only; it is not an alternative execution definition. Missing or malformed input must fail startup before the worker subscribes. |\n| `storage.TaskRef` | `internal/storage` | WorkflowExecution workspace lease | Storage receives the minimal execution/workspace reference and policy required to provision and release resources. It does not receive the WorkflowExecution aggregate. |\n| Gateway `TaskCreator` | `internal/gateway` | Accepted work request handoff | A channel may carry a request for durable workflow-backed work, but it does not request or create a WorkflowExecution directly. Work intake admits and routes the request; the Agent System consumes the accepted work request and is the sole owner of selecting a Workflow version and creating the WorkflowExecution. Ordinary agent interactions do not require this handoff. |\n| Gateway `TaskController` and `ChatTaskStatus` | `internal/gateway/tasks.go` | WorkflowExecution commands and query view | Approve, cancel, and status use Agent System contracts. Channel code does not duplicate lifecycle constants or decide transitions. |\n| Gateway `TaskProfile` | `internal/gateway/tasks.go`, `cmd/archied` | Channel/identity execution defaults | This is configuration selecting allowed source targets and default Workflow inputs, not task state. Split it into settings owned by the relevant channel and Agent System features. |\n| `config.TaskConfig` and `TaskForge` | `internal/config` | Worker execution settings and channel/service references | Dissolve with `internal/config`. Each field moves to its owning runtime settings or versioned WorkflowExecution handoff contract. Secrets and external configuration forms do not enter the domain. |\n| `events.Event.TaskID`, task event timelines | `internal/events`, stores, web UI | WorkflowExecutionID and WorkflowExecution events | Authoritative events are Agent System events. Observability and timelines project those events rather than inventing parallel generic task events. |\n| Web UI task lists and counts | `internal/webui`, store query APIs | WorkflowExecution read model | Dashboard status, history, and counts are projections of WorkflowExecution state and events. The UI does not depend on persistence records. |\n| Task-scoped forge/store/worktree RPC | `forgerpc`, `storerpc`, `worktreerpc`, `registerTaskRPCServers` | Execution-scoped capability ports | Worker calls use narrow domain-required service contracts, carry WorkflowExecutionID and IdentityID, and resolve the owning capability binding. Root-client fallback is forbidden. |\n| Per-task worktree, branch, volume, and cleanup | daemon, worktree, storage, container | WorkflowExecution workspace | Every WorkflowExecution receives isolated workspace state. Workspace lifecycle is explicit, identity-aware, recoverable, and controlled outside the model. |\n| Task retry, dead, waiting-human, PR reconciliation | daemon, workflow, gateway | WorkflowExecution lifecycle policies | Retry, waiting, cancellation, reconciliation, and terminal outcomes are Agent System behaviour. Channels and infrastructure report facts or submit commands; they do not mutate state. |\n| Task-run readiness and retry timeouts | daemon taskrun request loop | Worker-dispatch settings and policy | These settings govern WorkflowExecution handoff availability. They are not Workflow data and must be owned by the worker-dispatch capability. |\n\n### Terms that do not migrate\n\n- The `task` command in `Taskfile.yml` is the external Task build tool and keeps\n its name.\n- A forge may call its own records issues or tasks inside its channel adapter.\n- General prose may use “task” informally only when it clearly refers to an\n external system. Agent System contracts and generated reference MUST use the\n canonical Workflow vocabulary.\n\n## Current WorkflowExecution data inventory\n\nThis inventory is the migration ledger for data currently spread across\n`store.Task`, database-only columns, `workflow.TaskContext`, worker messages,\nand surrounding configuration. Implementation SHOULD NOT require another\nrepository-wide search to determine what existing execution data must be\npreserved or deliberately removed.\n\n| Current data | Current representation | Required target |\n| ---------------------- | ----------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |\n| Execution identity | `Task.ID`, `task_id`, NATS subjects, logs | Immutable WorkflowExecutionID used consistently by state, events, workers, projections, and capability calls |\n| Owning actor | `Task.Identity` string | Canonical IdentityID |\n| Source type | `Task.Source` (`forge` or `chat`) | Typed channel and SourceMessageReference; no closed source enum in WorkflowExecution |\n| Source address | `Owner`, `Repo`, `IssueNumber`; synthetic chat issue number | Channel-owned source reference with native message/thread/object identifiers; synthetic identifiers removed |\n| Requested work | `Title`, `Body`, `Labels` | Immutable execution input captured from the source request; channel-specific metadata retained in the source envelope |\n| Workflow selection | `Workflow` string | Immutable WorkflowID plus selected WorkflowVersion |\n| Lifecycle | `Status` string | Enforced WorkflowExecution state |\n| Current position | mutable `Stage` string | Derived current WorkflowStep reference plus durable StepExecution history |\n| Attempts | `Attempt`, `RetryCount` | Execution attempt history and retry-policy evidence; distinguish worker delivery retry from workflow retry |\n| Failure | `ParkReason`, transition detail, response errors | Typed failure classification, evidence, retryability, and resulting lifecycle event |\n| Human wait | `WatchCommentID`, `waiting_human` | Channel-neutral wait condition and correlation reference; forge comment IDs stay in the forge channel |\n| Workspace | `Dir`, `Branch`, worktree path, `task.json` | Execution workspace reference and independently managed workspace lifecycle |\n| Change outcome | `BuildSummary`, `BuildNoChanges`, changed files | Step outputs and final WorkflowExecution outcome |\n| Reproduction evidence | `ReproProof` | Typed TDD step evidence retained with the StepExecution and final outcome |\n| Planning output | `Plan` | Typed step output referenced by later steps rather than a special aggregate field |\n| Notes | `Notes`, appended agent notes | Ordered execution annotations with actor, source step, and timestamp |\n| Pull request | `PRNumber`, forge host/repository assumptions | Forge-channel delivery reference and reconciliation state |\n| Agent usage | `TokensUsed`, `Iterations` | StepExecution usage measurements aggregated into execution projections |\n| Decision capture | private feasibility `decision`, capture-tool output | Typed WorkflowStep output defined by that Workflow version |\n| Terminal outcome | mutable `Outcome{Status, Detail}` | Explicit WorkflowExecution outcome command/event validated by the state machine |\n| Skill context | `SkillBody`, `SkillPlugins` | Resolved Workflow version dependencies or execution inputs, with provenance |\n| Agent prompt context | `SystemPrompt`, `ExtraRules`, mission strings | Versioned WorkflowStep definition plus resolved execution context; mutable global prompt callbacks removed |\n| Gates and protection | `Gate`, `Preflight`, `Protection`, repository config | Agent System policy definitions resolved for the WorkflowExecution and enforced outside the agent |\n| Budgets | steps, tokens, wall clock, gate failures | Execution and StepExecution policy values with measured usage and evidence |\n| Tool access | capture tools, registry, guardrails, skill plugins | Resolved step capability set and policy; global registry pointers do not enter execution state |\n| Persistence timestamps | SQL `created_at` and `updated_at` | Domain timestamps and version used for optimistic concurrency and audit |\n| Transition history | SQL `transitions` and generic events | Authoritative WorkflowExecution domain events stored atomically with state |\n\n## Current runtime responsibilities to unify\n\nThe following behaviours currently act on the same WorkflowExecution but are\nimplemented in different packages:\n\n- accept a request from a channel and enforce idempotency;\n- select and version a Workflow;\n- create, queue, claim, schedule, and execute a WorkflowExecution;\n- serialize same-repository execution where policy requires it;\n- resolve identity-specific capabilities;\n- prepare, retain, recover, and clean up the workspace;\n- run deterministic and agent-backed WorkflowSteps;\n- enforce gates, budgets, protected paths, tool constraints, and result review;\n- persist state and StepExecution outputs;\n- publish authoritative lifecycle events;\n- wait for channel input and resume;\n- classify failure, retry, park, mark exhausted, cancel, or recover after crash;\n- deliver and reconcile forge changes;\n- project status, history, counts, and audit for channels and the web UI;\n- hand execution to a worker without moving domain ownership to the worker.\n\n## Current workflow-feature hazards\n\n- There is no enforced WorkflowExecution state machine. `Store.Transition`\n ignores the supplied `from` state when updating the row.\n- State update and transition-history insertion are separate operations, so\n state and audit can diverge.\n- Callers frequently discard persistence and transition errors.\n- Gateway task control duplicates lifecycle status constants by hand.\n- `workflow.TaskContext` is a mutable service bag containing complete\n configuration plus forge, store, worktree, agent, event bus, logging, memory,\n guardrails, skills, and scratch state.\n- Lifecycle changes occur in both daemon and workflow code.\n- Workflow orchestration runs inside task-scoped agent containers while the\n daemon retains lifecycle, admission, and authority-bearing RPC ownership.\n- Forge-backed and chat-created work share one record through source flags and\n synthetic issue numbers.\n- Repository labels currently select workflows and mirror task state, coupling\n domain transitions to forge presentation.\n- Failure generally becomes `parked`, but failure classification and\n retryability are not represented explicitly.\n- Generic observability events are emitted alongside state changes rather than\n being the authoritative domain events.\n\n## Existing invariants to preserve\n\n- The model never performs git operations directly.\n- Deterministic policy and gates cannot be waived by the agent.\n- TDD captures a failing reproduction before implementation and protects the\n test during the fix.\n- Every run ends in an explicit outcome; definition errors and stage errors do\n not disappear silently.\n- Crash recovery requeues interrupted work.\n- Agent execution crosses a versioned data boundary.\n- Forge, persistence, workspace, transport, container, and model runtimes remain\n replaceable infrastructure.\n- Work from the same repository is serialized by default.\n\n## Required migration\n\n- Rename the current persisted `store.Task` concept to WorkflowExecution.\n- Replace task lifecycle terminology in core commands, events, persistence,\n worker contracts, logs, and generated reference.\n- Preserve external source terminology only inside its channel adapter.\n- Replace synthetic issue numbers with explicit source-message references.\n- Associate every WorkflowExecution with an immutable Workflow identifier and\n version.\n- Record StepExecutions explicitly rather than treating one mutable `stage`\n field as complete execution history.\n- Keep Attempt as retry state within the same WorkflowExecution.\n\n## Curator isolation invariant\n\nDecided 2026-08-05\n([#435](https://github.com/samcharles93/archie-core/issues/435) epic,\n[#98](https://github.com/samcharles93/archie-core/issues/98) runtime).\nRecovered 2026-08-09 from the pre-migration issue tracker.\n\n**Curators are peers, not dependencies.** They must never block, delay, or take\ndown chat turns, agent runs, or the daemon.\n\n1. Registry-owned goroutines, one per curator, with per-pass `recover()`,\n per-pass budgets (max steps, tokens, timeout), and a configurable maximum of\n concurrent passes (default 1) so curators cannot saturate the shared LLM\n runtime against chat.\n2. Wake events ride the in-process `events.Bus`, which is mutex fan-out with\n bounded per-subscriber buffers that **drop** on overflow\n (`internal/events/events.go`). A slow curator can never backpressure the chat\n publisher.\n3. Wake is a best-effort nudge. The trigger is a deterministic read of persisted\n primary-input state at pass time, so a dropped wake only delays a run to the\n next check-in and never corrupts trigger accounting\n ([#22](https://github.com/samcharles93/archie-core/issues/22)).\n4. The curator registry lifecycle is independent of the agent loop. Daemon\n shutdown cancels in-flight passes and waits bounded.\n\nChat turns keep their per-session lanes; agent runs keep their workflow\ngoroutines and containers. Rule of thumb: **no code path a curator executes may\never appear on a chat turn's critical path.**\n" }, { "url": "/dev/architecture/bindings/", "title": "Playbook bindings: model and threat model", "section": "Architecture", - "body": "# Playbook bindings: model and threat model\n\n**Status:** Approved foundation (documents shipped code: `archie-core-t2db.1`\nthrough `.5`, `.7`, `.9`-`.15`)\n**Date:** 2026-09-03\n\nThis document covers the binding model and the intake threat model portion\nof `archie-core-t2db.6`. It does not cover the task-lifecycle provenance rule\n(`messaging-and-work-intake.md`'s \"Intake provenance\") or the operator\nwalkthrough -- that walkthrough remains an open piece of the same ticket.\n\n## What a binding is\n\nA `binding` (`internal/domain/binding.Binding`) ties four things together:\na **matcher** (which captured events it applies to), a **mapping**\n(`internal/domain/mapping.Mapping`, how fields are pulled out of a payload),\na **workflow** (which registered workflow runs on a match), and a **state\nmachine** governing whether it is live.\n\n```go\ntype Binding struct {\n ID, Name // identity\n Matcher // { Source string } -- the webhook path segment, e.g. \"sentry\"\n MappingID // which Mapping resolves payload fields\n Workflow // registered workflow name to dispatch to\n Version // bumped on every edit\n Status // pending_approval | armed\n Secret // HMAC-SHA256 shared secret, encrypted at rest, never returned by GET\n}\n```\n\nBindings live in the store (SQLite), not `config.toml` -- operators author\nthem from the dashboard while the daemon runs, without a restart.\n\n### Matching\n\n`Matcher.Source` is the only predicate today: the path segment a sender\nPOSTs to (e.g. a webhook hitting `.../sentry` matches bindings with\n`Matcher.Source == \"sentry\"`). The storage column is plain `TEXT`, so adding\npredicates later is non-breaking -- extend `Matcher`, don't invent a second\nmatcher shape.\n\n### Mapping\n\n`mapping.Resolve(fields, payload)` walks a captured JSON payload against a\nlist of typed `Field`s (name, JSON path, expected type), returning resolved\nvalues plus a `[]Failure` for anything that didn't parse or type-match. This\nis schema-by-example in practice: a mapping is authored by pointing at a\nreal captured payload in the dashboard, not by guessing a schema in advance.\n\n### Versioning\n\n`Version` increments on every `UpdateBinding` call. This exists so an\neditor's later change can never silently rewrite the historical provenance\nof a task that already fired -- a dispatched task's binding reference is\npinned to the version that was armed when it fired, not \"whatever the\nbinding currently says.\"\n\n### State machine\n\n```\ncreate ---\u003e pending_approval ---\u003e armed\n ^ |\n +----------------+\n (any edit to an armed binding drops it back to pending_approval)\n```\n\nA binding is created `pending_approval`, so the operator who creates one can\napprove it immediately. Bindings stored as `draft` by an earlier version are\nmoved to `pending_approval` when the EDA store opens.\n\nModeled directly on Telegram's existing `dangerousAction`/`pendingApproval`\nflow (`internal/channels/telegram/dangerous.go`, `approval.go`) -- not a new\napproval mechanism invented for this feature. **Only `armed` evaluates\nagainst incoming events** (`binding.Matches`, `internal/domain/binding/\nbinding.go`); a `pending_approval` binding is inert. An edit to\nan already-armed binding cannot bypass approval by silent mutation -- it\ndrops back to `pending_approval` and requires an explicit `Approve` call to\nre-arm.\n\n## The threat model\n\nThis is a public, unauthenticated-by-default intake surface whose output is\nan agent with a repo, a worktree, and tool access. The design brief\n(`archie-core-t2db.5`, `docs/prds/webhook-intake-security.md`) named five\nnon-negotiables; each is verified below against the actual shipped code,\nnot the design intent.\n\n### 1. Per-source authentication\n\nReuses `internal/channels/webhook`'s existing HMAC-SHA256 scheme --\ncapture accepts unsigned events (visible in the dashboard, marked\nunauthenticated), but **only authenticated events can ever match a\nbinding**. `binding.Matches`' doc comment states this plainly: \"an\nunauthenticated event can never trigger a binding, no matter how well it\nwould otherwise match.\" Enforced structurally at two points, not just\nconvention:\n\n- `captured_events.authenticated` is a real column (`internal/store/\n captures.go`), set at capture time.\n- `dispatchBindings` (`internal/daemon/daemon.go:479`) walks **authenticated\n captures whose source has at least one armed binding** -- an\n unauthenticated capture is never even considered for dispatch.\n\n### 2. Human approval before a binding goes live\n\nCovered by the state machine above. Nothing self-arms: `pending_approval -\u003e\narmed` requires an explicit `Approve` call, and any edit to an armed\nbinding reverts it.\n\n### 3. Rate and volume limits per source\n\nCapture writes are bounded independently of the approval gate, so a flood\nof unauthenticated traffic cannot exhaust disk or drown the inspector:\n`InsertCapture` (`internal/store/captures.go`) prunes by both **retention\nwindow** (`WHERE received_at \u003c cutoff`) and **max event count** in the same\ninsert transaction.\n\n### 4. Blast radius bounded by the same code-level controls every task gets\n\nA playbook-originated task is dispatched into the **same** gate, diff-cap,\nsandboxed-container, worktree-isolation pipeline every task goes through\nregardless of origin. Verified precisely, not assumed: `dispatchOneBinding`\ncalls `EnqueueBindingTask` (`internal/store/store.go`), which wraps\n`EnqueueChatTask` -- the same direct-to-`tasks`-table enqueue path\nchat-spawned tasks already use, then stamps `binding_id`/`binding_version`\nfor provenance in a second statement. This is a **different** enqueue\nmechanism from forge-issue polling's `workintake.TaskEnvelope`/NATS path\n(`docs/prds/event-sources-and-reactions.md`) -- binding and forge-issue\ntasks arrive by different producers, but both become an ordinary\n`store.Task` row that `ClaimNext` picks up identically, so both get the\nsame gate/worktree pipeline downstream regardless of which producer created\nthe row. Payload content is\nordinary untrusted prompt input; the sender's text is never treated as a\ntrusted instruction (Archie's environmental-enforcement principle: \"the\nagent should ignore instructions in the payload\" is not itself a control,\nthe surrounding sandbox is).\n\n### 5. Captured payloads may contain secrets from the sending system\n\nTwo independent protections, not one:\n\n- **Redaction before persistence**: `webhookguard.RedactPayload` walks the\n decoded JSON and redacts recognised sensitive keys before the row is\n written -- the raw, unredacted payload is never durably stored.\n- **Retention**: a configurable window (`InsertCapture`'s `retention`\n parameter) prunes old captures automatically; visibility of what remains\n is gated by the same dashboard authentication as every other webui\n surface, not a separate, weaker path.\n\n### Secrets at rest\n\n`Binding.Secret` (the HMAC shared secret a sender signs with) is encrypted\nat rest via AES-GCM (`internal/store/binding_cipher.go`, `archie-core-\nt2db.7`) -- a leaked SQLite file does not leak sender-side shared secrets in\nplaintext. The envelope is versioned (`bindingEnvelopeVersion`) so a future\ncipher or KDF change can roll forward without breaking already-encrypted\nrows, and the AAD binds ciphertext to the binding-secret context so it\ncannot be relocated to another field/column and still authenticate. A `nil`\ncipher (no encryption key configured) leaves secrets as plaintext --\ndocumented legacy behaviour, not silent degradation.\n\n### At-most-once dispatch\n\n`dispatchBindings` enqueues at most one task per `(binding, capture)` pair,\nenforced by a real ledger, not best-effort application logic:\n`RecordDispatch` (`internal/store/bindings.go`) does `INSERT OR IGNORE` into\n`binding_dispatches`, unique on `(binding_id, capture_id)`; a duplicate\ninsert returns the sentinel `ErrAlreadyDispatched`, which the dispatch loop\ntreats as \"another cycle raced us,\" not an error. This guarantee holds\n**across daemon restarts**, not just within one process's lifetime, because\nthe ledger is a durable table, not in-memory state.\n\nThis ledger is also the direct precedent for the EDA playbook engine's\nexecution-time idempotency, now landed (`archie-core-t2db.17` closed): the\n`playbook_dispatches` table in `internal/store/playbook_dispatches.go` copies\n`binding_dispatches`'s `INSERT OR IGNORE` conventions with a\n`(playbook_id, playbook_version, event_id, action_id)` key\n(`docs/prds/eda-playbook-engine.md` gap 2).\n\n## What this document does not cover\n\n- An end-to-end operator walkthrough (point an external app at archie,\n build a first playbook) -- open, part of `archie-core-t2db.6`.\n- The playbook YAML / CEL / Module/Channel/Forge action-position work\n (`docs/prds/eda-playbook-engine.md`, `module-position.md`,\n `channel-forge-positions.md`) -- a different, newer mechanism than the\n binding model described here; the two are related but not the same\n system, and this document does not attempt to unify them.\n" + "body": "# Playbook bindings: model and threat model\n\n**Status:** Approved foundation (documents shipped code: `archie-core-t2db.1`\nthrough `.5`, `.7`, `.9`-`.15`)\n**Date:** 2026-09-03\n\nThis document covers the binding model and the intake threat model portion\nof `archie-core-t2db.6`. It does not cover the task-lifecycle provenance rule\n(`messaging-and-work-intake.md`'s \"Intake provenance\") or the operator\nwalkthrough -- that walkthrough remains an open piece of the same ticket.\n\n## What a binding is\n\nA `binding` (`internal/domain/binding.Binding`) ties four things together:\na **matcher** (which captured events it applies to), a **mapping**\n(`internal/domain/mapping.Mapping`, how fields are pulled out of a payload),\na **workflow** (which registered workflow runs on a match), and a **state\nmachine** governing whether it is live.\n\n```go\ntype Binding struct {\n ID, Name // identity\n Matcher // { Source string } -- the webhook path segment, e.g. \"sentry\"\n MappingID // which Mapping resolves payload fields\n Workflow // registered workflow name to dispatch to\n Version // bumped on every edit\n Status // pending_approval | armed\n Secret // HMAC-SHA256 shared secret, encrypted at rest, never returned by GET\n}\n```\n\nBindings live in the store (PostgreSQL), not `config.toml` -- operators author\nthem from the dashboard while the daemon runs, without a restart.\n\n### Matching\n\n`Matcher.Source` is the only predicate today: the path segment a sender\nPOSTs to (e.g. a webhook hitting `.../sentry` matches bindings with\n`Matcher.Source == \"sentry\"`). The storage column is plain `TEXT`, so adding\npredicates later is non-breaking -- extend `Matcher`, don't invent a second\nmatcher shape.\n\n### Mapping\n\n`mapping.Resolve(fields, payload)` walks a captured JSON payload against a\nlist of typed `Field`s (name, JSON path, expected type), returning resolved\nvalues plus a `[]Failure` for anything that didn't parse or type-match. This\nis schema-by-example in practice: a mapping is authored by pointing at a\nreal captured payload in the dashboard, not by guessing a schema in advance.\n\n### Versioning\n\n`Version` increments on every `UpdateBinding` call. This exists so an\neditor's later change can never silently rewrite the historical provenance\nof a task that already fired -- a dispatched task's binding reference is\npinned to the version that was armed when it fired, not \"whatever the\nbinding currently says.\"\n\n### State machine\n\n```\ncreate ---\u003e pending_approval ---\u003e armed\n ^ |\n +----------------+\n (any edit to an armed binding drops it back to pending_approval)\n```\n\nA binding is created `pending_approval`, so the operator who creates one can\napprove it immediately. Bindings stored as `draft` by an earlier version are\nmoved to `pending_approval` when the EDA store opens.\n\nModeled directly on Telegram's existing `dangerousAction`/`pendingApproval`\nflow (`internal/channels/telegram/dangerous.go`, `approval.go`) -- not a new\napproval mechanism invented for this feature. **Only `armed` evaluates\nagainst incoming events** (`binding.Matches`, `internal/domain/binding/\nbinding.go`); a `pending_approval` binding is inert. An edit to\nan already-armed binding cannot bypass approval by silent mutation -- it\ndrops back to `pending_approval` and requires an explicit `Approve` call to\nre-arm.\n\n## The threat model\n\nThis is a public, unauthenticated-by-default intake surface whose output is\nan agent with a repo, a worktree, and tool access. The design brief\n(`archie-core-t2db.5`, `docs/prds/webhook-intake-security.md`) named five\nnon-negotiables; each is verified below against the actual shipped code,\nnot the design intent.\n\n### 1. Per-source authentication\n\nReuses `internal/channels/webhook`'s existing HMAC-SHA256 scheme --\ncapture accepts unsigned events (visible in the dashboard, marked\nunauthenticated), but **only authenticated events can ever match a\nbinding**. `binding.Matches`' doc comment states this plainly: \"an\nunauthenticated event can never trigger a binding, no matter how well it\nwould otherwise match.\" Enforced structurally at two points, not just\nconvention:\n\n- `captures.authenticated` is a real column\n (`internal/infrastructure/postgres/migrations/0002_edastore.sql`), set at\n capture time.\n- `dispatchBindings` (`internal/daemon/daemon.go:479`) walks **authenticated\n captures whose source has at least one armed binding** -- an\n unauthenticated capture is never even considered for dispatch.\n\n### 2. Human approval before a binding goes live\n\nCovered by the state machine above. Nothing self-arms: `pending_approval -\u003e\narmed` requires an explicit `Approve` call, and any edit to an armed\nbinding reverts it.\n\n### 3. Rate and volume limits per source\n\nCapture writes are bounded independently of the approval gate, so a flood\nof unauthenticated traffic cannot exhaust disk or drown the inspector:\n`InsertCapture` (`internal/infrastructure/postgres/edastore.go`) prunes by both **retention\nwindow** (`WHERE received_at \u003c cutoff`) and **max event count** in the same\ninsert transaction.\n\n### 4. Blast radius bounded by the same code-level controls every task gets\n\nA playbook-originated task is dispatched into the **same** gate, diff-cap,\nsandboxed-container, worktree-isolation pipeline every task goes through\nregardless of origin. Verified precisely, not assumed: `dispatchOneBinding`\ncalls `EnqueueBindingTask` (`internal/infrastructure/postgres/store.go`), which wraps\n`EnqueueChatTask` -- the same direct-to-`tasks`-table enqueue path\nchat-spawned tasks already use, then stamps `binding_id`/`binding_version`\nfor provenance in a second statement. This is a **different** enqueue\nmechanism from forge-issue polling's `workintake.TaskEnvelope`/NATS path\n(`docs/prds/event-sources-and-reactions.md`) -- binding and forge-issue\ntasks arrive by different producers, but both become an ordinary\n`workflow.Task` row that `ClaimNext` picks up identically, so both get the\nsame gate/worktree pipeline downstream regardless of which producer created\nthe row. Payload content is\nordinary untrusted prompt input; the sender's text is never treated as a\ntrusted instruction (Archie's environmental-enforcement principle: \"the\nagent should ignore instructions in the payload\" is not itself a control,\nthe surrounding sandbox is).\n\n### 5. Captured payloads may contain secrets from the sending system\n\nTwo independent protections, not one:\n\n- **Redaction before persistence**: `webhookguard.RedactPayload` walks the\n decoded JSON and redacts recognised sensitive keys before the row is\n written -- the raw, unredacted payload is never durably stored.\n- **Retention**: a configurable window (`InsertCapture`'s `retention`\n parameter) prunes old captures automatically; visibility of what remains\n is gated by the same dashboard authentication as every other webui\n surface, not a separate, weaker path.\n\n### Secrets at rest\n\n`Binding.Secret` (the HMAC shared secret a sender signs with) is encrypted\nat rest via AES-GCM (`internal/infrastructure/bindingcipher`, `archie-core-\nt2db.7`) -- a leaked database dump does not leak sender-side shared secrets in\nplaintext. The envelope is versioned (`bindingEnvelopeVersion`) so a future\ncipher or KDF change can roll forward without breaking already-encrypted\nrows, and the AAD binds ciphertext to the binding-secret context so it\ncannot be relocated to another field/column and still authenticate. A `nil`\ncipher (no encryption key configured) leaves secrets as plaintext --\ndocumented legacy behaviour, not silent degradation.\n\n### At-most-once dispatch\n\n`dispatchBindings` enqueues at most one task per `(binding, capture)` pair,\nenforced by a real ledger, not best-effort application logic:\n`RecordDispatch` (`internal/infrastructure/postgres/edastore.go`) inserts into\n`binding_dispatches`, unique on `(binding, capture)`; a duplicate\ninsert returns the sentinel `ErrAlreadyDispatched`, which the dispatch loop\ntreats as \"another cycle raced us,\" not an error. This guarantee holds\n**across daemon restarts**, not just within one process's lifetime, because\nthe ledger is a durable table, not in-memory state.\n\nThis ledger is also the direct precedent for the EDA playbook engine's\nexecution-time idempotency, now landed (`archie-core-t2db.17` closed): the\n`playbook_dispatches` table (`RecordPlaybookDispatch`) copies\n`binding_dispatches`'s unique-ledger conventions with a\n`(playbook_id, playbook_version, event_id, action_id)` key\n(`docs/prds/eda-playbook-engine.md` gap 2).\n\n## What this document does not cover\n\n- An end-to-end operator walkthrough (point an external app at archie,\n build a first playbook) -- open, part of `archie-core-t2db.6`.\n- The playbook YAML / CEL / Module/Channel/Forge action-position work\n (`docs/prds/eda-playbook-engine.md`, `module-position.md`,\n `channel-forge-positions.md`) -- a different, newer mechanism than the\n binding model described here; the two are related but not the same\n system, and this document does not attempt to unify them.\n" }, { "url": "/dev/architecture/configuration/", "title": "Configuration", "section": "Architecture", - "body": "# Configuration\n\n**Status:** Approved; existing-type assignment is ongoing \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Ownership\n\nConfiguration has two separate responsibilities:\n\n1. `internal/infrastructure/configuration` loads files, environment variables,\n secret references, overlays, and compatibility representations.\n2. Each domain and plugin owns the typed runtime settings required for its\n behaviour.\n\n`internal/app` translates external input into those settings, validates\ncross-capability combinations, and supplies each component only its own\nsettings.\n\nThe following are prohibited:\n\n- passing complete application configuration into a domain;\n- making a domain depend on TOML, YAML, environment-variable names, secret\n references, or a configuration library;\n- defining a central runtime configuration type imported across the system;\n- placing plugin options in an unrelated global configuration model.\n\n## Private input document\n\n`internal/infrastructure/configuration` MAY use a complete decoded document\nwhile discovering files, decoding sections, and applying overlays. It is an\ninput-boundary DTO only.\n\nThe document:\n\n- remains private to the configuration infrastructure;\n- is composed from separately defined feature sections;\n- is immediately translated by application composition;\n- is never returned as the application's runtime model;\n- is never passed through runtime components.\n\nExternal DTOs and domain runtime settings are separate types.\n\nPlugins supply external configuration definitions through a narrow extension\ncontract. The core loader MUST NOT gain a field or recognized filename for every\nplugin.\n\nInvalid configuration input MUST NOT replace active valid settings or terminate\nunaffected capabilities. Configuration changes use the candidate validation,\npromotion, audit, and recovery requirements in\n[safe-change-and-recovery.md](/dev/architecture/safe-change-and-recovery/).\n\n## Dissolution of `internal/config`\n\n`internal/config` has no target location. It MUST be dissolved and MUST NOT\nsurvive as a shared package, configuration domain, or renamed global model.\n\nIts loading behaviour moves to `internal/infrastructure/configuration`. Existing\ntypes and methods are reassigned by behaviour. Repository execution policy,\ndispatch policy, retry policy, ecosystem defaults, tool policy, memory settings,\nprovider settings, and channel settings move to their\nrespective owners.\n\n## Repository settings\n\nThe existing `config.Repo` MUST NOT move intact into a repository domain or\nshared settings package. A managed repository is an application association:\n\n- work intake owns which repository references an identity monitors;\n- task execution owns gates, preflight, test protection, ecosystem behaviour,\n and change limits;\n- task lifecycle owns retry behaviour;\n- scheduling owns concurrency policy;\n- workspace management owns base branch and persistent-storage requirements.\n\nApplication composition correlates these settings by repository reference. No\ndomain receives a universal repository profile.\n\n## Documentation\n\nGenerated configuration reference describes external sections and their owning\ncapabilities. It MUST NOT establish a global Go configuration structure as the\npublic contract.\n\n## Value classification rule\n\nEvery value that controls behaviour MUST have an explicit owner and MUST be\nclassified as exactly one of:\n\n1. **Invariant** — changing it changes the meaning of the program, a wire\n protocol, or a domain concept. It may be a Go constant.\n2. **Runtime setting** — an operator, identity, domain, plugin, or deployment\n may legitimately choose a different value. It is typed configuration owned by\n the affected capability.\n3. **Policy value** — it expresses what is allowed, limited, routed, retried,\n protected, or gated. Its typed definition belongs to the owning domain and it\n is evaluated through the policy system.\n4. **Derived value** — it is computed from other settings and is not\n independently mutable.\n5. **Runtime state** — it changes as the application operates, such as the\n active model for a conversation. It is changed by a domain command and is not\n configuration merely because it can be changed.\n\nAn unclassified behavioural literal is a defect.\n\n`const` MUST NOT be used merely because a value currently has one default.\nConstants are reserved for actual invariants: enum discriminants, protocol\nversions, fixed compatibility tokens, and values whose alteration would change\nthe program's semantics rather than configure an installation.\n\nIn Go, a constant is a compile-time value and does not represent mutable runtime\nstorage. It generally has no address that can be observed or changed.\nArchitecturally, a constant therefore asserts that the value cannot change for\nthe lifetime of the application. Any value that may legitimately change while\nthe application is running, between application instances, or between\ndeployments is not a constant.\n\nA runtime setting may have a default, but the default is not its only access\npath. The owner exposes a typed settings structure, validation, and a\n`DefaultSettings` constructor or equivalent. Private constants used to implement\ndefaults do not turn those defaults into invariants and MUST NOT be consumed\ndirectly by runtime code.\n\nEach runtime setting and policy value MUST declare:\n\n- its owner;\n- type, units, validation constraints, and safe bounds;\n- default and derivation, if any;\n- whether it is startup-only, live-reloadable, or restart-required;\n- whether it may differ by installation, identity, repository, workflow,\n channel, or plugin instance;\n- whether it is secret, sensitive, or safe to report;\n- its current source, effective value, version, and last-known-good version;\n- the command and identity required to change it.\n\n## Current configuration migration ledger\n\nThis ledger records the current meanings hidden inside `internal/config`. It is\nthe source inventory for dissolving that package; rows are split where the\nexisting structure combines different owners.\n\n| Current values | Current location | Target owner and requirement |\n| ----------------------------------------------------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `WorkDir`, `DBPath` | root `Config` | Workspace/storage infrastructure settings. `DBPath` is the prefix for independent task and conversation SQLite files. Defaults derive from the platform data directory. |\n| `SkillsDir`, `PluginDir` | root `Config` | Skill and plugin infrastructure settings, independently typed. |\n| `PollInterval` | root `Config` | Work-intake scheduling setting; not a daemon-wide constant. |\n| `Label`, `BotUser`, `BotEmail` | root `Config` | Identity and dispatch associations. `BotEmail` may be derived from the selected forge account but remains overrideable. |\n| `MaxRetries` | root `Config`, `Repo` | Retry policy owned by the operation being retried. The duplicate global/repository representation must disappear. |\n| `DiffCapLines` | root `Config` | Agent-change policy value. |\n| `MaxSteps`, `WallClock`, `GateMaxFailures` | `Budgets` | Workflow-execution policy, with units and zero/unlimited semantics made explicit. |\n| `Class`, `APIKeyEnv`, `BaseURL` | `Provider` | Provider instance settings. Secret lookup becomes a `SecretRef`, not an environment variable name leaking into the provider domain. |\n| `Mode`, `Command`, `Env` | `LegacyAgent` | Decode-only compatibility input for the removed executor selector. Delete after the config migration window; it has no runtime settings owner. |\n| `Type`, `Host`, `Token` / `TokenEnv` (older form) | `Forge` | Forge adapter settings. The older `TokenEnv` input is accepted only at the loader boundary. |\n| `Trigger`, `AckReaction`, label names | `Dispatch` | Work-intake and forge presentation policy. Label vocabulary is configurable per forge/identity association. |\n| `Engine` | `MemoryConfig` | Memory engine selection. `internal/domain/memory` is the only engine family; per-engine settings remain owned and validated by that engine. |\n| `Name`, `Transport`, `Command`, `Args`, `WorkDir`, `URL` | `MCPServer` | One typed MCP server instance specification. Transport-specific variants must reject fields belonging to other variants. |\n| `MaxResultChars` | `ToolPolicy` | Tool-execution policy values, not generic daemon configuration. |\n| `URL`, `TokenEnv` | `NATSConfig` | NATS transport settings with a secret reference replacing the environment-name field. |\n| `Image`, `MaxConcurrency`, `MaxUptime`, `VolumeTTL`, `PullPolicy`, `Network` | `ContainerConfig` | Mandatory managed-worker infrastructure settings and resource policy. The legacy `enabled` input is decode-only. |\n| model mappings, email, webhook, Telegram | `ChatConfig` | Separate channel instance settings plus channel-routing associations; there is no chat-wide settings owner. |\n| `AllowedUserIDs`, `Token` / `TokenEnv` (older form), update commands | Telegram config | Telegram adapter settings and access policy. `Token` is the preferred `secret.SecretRef`; `TokenEnv` remains a backwards-compatible fallback. |\n| listen/relay addresses | email config | Email adapter instance settings. |\n| webhook target | `Notify` | Notification adapter instance settings. |\n| web listen address | `Web` | Web transport settings. |\n| `listen` under `[services.state]` and `[services.gateway]` | `ServiceConnection` | The bind address of the process that owns the service. Read only by that process, and defaulted to the address its clients dial so the pair cannot drift. |\n| forge, dispatch, models, providers, containers, memory, tools and repositories | `IdentityConfig` | Existing composite duplicates unrelated settings. Replace it with associations from an `IdentityID` to independently owned configured instances. |\n| owner, name, base, gate, protection, ecosystem, preflight, storage, retry and concurrency | `Repo` | Split across work intake, workspace, workflow execution, policy, gate, and scheduling as described in Repository settings. |\n| `TaskConfig`, `TaskForge` | worker snapshot | Replace with the minimum immutable, versioned WorkflowExecution input assembled by the application. It is not a second configuration model. |\n| `Extra` and arbitrary maps | root/provider config | Allowed only at an extension boundary whose owning plugin/provider supplies schema and validation. Core configuration never consumes an untyped catch-all. |\n\n## Behavioural literals and incorrectly fixed values\n\nThe following current values affect useful behaviour but are not represented as\nowned typed settings or policy. They MUST be classified and migrated before the\nold configuration package is removed. The listed target is the architectural\ndecision; exact setting names are implementation details.\n\n| Capability | Values fixed in current code | Required classification |\n| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Chat agent | maximum steps `8` in `cmd/archied` | Workflow/chat execution policy. |\n| Context compression | enabled, threshold `0.5`, context size `128000`, protect first `3` and last `20`, compression marker, token estimate of four characters per token | Model/context runtime settings. The marker is presentation; the estimate is provider/model strategy, not a universal truth. |\n| Tool guardrails | escalation thresholds `3`, `5`, `10`, `2` | Tool-use policy with named meanings and bounded defaults. |\n| Tool output | per-result cap `50000` characters, no aggregate turn cap, preview `500` characters | Tool-execution resource and presentation settings. Per-result spill/truncation remains; turn continuation is not character- or token-gated. |\n| Pairing/access | code TTL `1h`, maximum pending `3`, rate window `10m`, failures `5`, lockout `1h`, maximum lockout `24h` | Access policy. Code length/alphabet and salt length require an explicit security-constraint decision; they are not automatically valid constants. |\n| NATS daemon consumer | stream `ARCHIE_TASKS`, consumer `archie-daemon`, deduplication `2m`, poll `2s`, deliveries `3`, acknowledgement wait `5m`, inactive threshold `24h`, subjects | NATS topology settings plus delivery policy. Subject grammar or protocol headers may be invariants only if declared as versioned wire contracts. |\n| NATS worker consumer | stream `ARCHIE_TASKS`, consumer `archie-agent`, deduplication `2m`, poll `5s`, acknowledgement wait `30m`, deliveries `3`, inactive threshold `24h`, queue group and retry delay `1s` | Worker transport settings and delivery/retry policy. Shared stream defaults must have one owner rather than two copies. |\n| Worker readiness | readiness timeout `20s`, retry delay `250ms` | Worker lifecycle and retry policy. |\n| Execution/RPC | agent execution `30m`, task-run RPC `60s`, worktree RPC `15m` | Per-operation timeout policy. |\n| Plugin/tool cleanup | plugin rollback `10s`, tool-provider cleanup `10s`, MCP-provider cleanup `10s` | Lifecycle shutdown/rollback policy owned by each engine, with a shared policy vocabulary where appropriate. |\n| Memory read path | per-scope record limit `20`, rendered `\u003cmemory\u003e` block byte cap `8192` | Memory read policy owned by the engine family (`internal/domain/memory`). |\n| Built-in memory store | per-file size limit `100KiB`, query result limit `100` | Split into memory resource policy, input policy, and presentation settings. |\n| Memory safety scanner | compiled prompt-injection and sensitive-data patterns with fixed warn/block consequences | Versioned memory-domain policy definitions. Patterns and consequences must be inspectable and replaceable without editing scanner code. |\n| MCP stdio | initial backoff `500ms`, maximum backoff `30s`, unlimited default retries, send and shutdown `5s` | MCP transport retry, request, and lifecycle policy. Unlimited retries must be explicit rather than a zero-value accident. |\n| MCP SSE/HTTP | reconnect `1s` to `30s`, POST/request timeout `30s` | MCP transport settings and retry policy. |\n| Message framing | MCP frame `4MiB`, worker protocol frame `8MiB`, diagnostic `64KiB`, media/base64 `64MiB` | Resource/security policy unless a published wire protocol mandates the limit. Limits must be configurable only within safe bounds. |\n| Webhooks and web UI | webhook bind `0.0.0.0:8644`, webhook shutdown `5s`, web read-header `5s`, web shutdown `2s` | Transport instance and lifecycle settings. |\n| Telegram | typing refresh `4s`, draft interval `700ms`, update-action TTL `10m` | Telegram adapter presentation and lifecycle settings. |\n| Telegram limits | message length `4000` and callback/update prefixes | Record the upstream platform constraint and version it as an adapter compatibility limit; prefixes may remain wire constants when changing them would invalidate outstanding callbacks. |\n| Containers | stop and pull timeouts `10s` and `30s` | Container lifecycle policy. |\n| HTTP clients | feasibility notification timeout `10s`; other paths use `http.DefaultClient` | Adapter request policy. Every outbound client requires an explicit timeout profile. |\n| Workflow routing | labels `bug`, `feature`, `bootstrap`; workflows `tdd`, `feasibility`, `bootstrap`; fallback `implement` | Agent-system routing policy. Workflow identifiers are references, not compiled routing decisions. |\n| Filesystem/deployment | `.agents/skills`, `.archie/stages`, `.archie/gate.go`, `/data/worktree`, `/data/repo`, cache names/paths and prepared sentinel | Split into deployment settings, repository conventions, and genuine internal protocol markers. Paths an installation may relocate are settings. |\n| Packaged release data | `/usr/share/archie/CHANGELOG.md` | Packaging/deployment settings or paths derived from an installation prefix. |\n| Forge presentation | default label colour `bfd4f2` and state-label colours | Forge adapter presentation settings. |\n| Fallback system prompt | compiled fallback prompt | Agent-system default content, versioned and replaceable through its owning settings; safety constraints remain policy/code. |\n| Default paths and identities | config path, data paths, listen addresses, model roles and workflow names | Typed defaults belonging to their respective owners, never global constants consumed throughout the application. |\n\nThis inventory is intentionally broader than exported Go `const` declarations. A\nliteral hidden in a constructor, timeout, switch, retry loop, path join, prompt,\nlabel map, or client setup restricts behaviour just as effectively as an\nincorrectly used constant.\n\n## Constants that may remain\n\nA value may remain constant only when its owner documents why runtime variation\nwould be invalid. Typical valid cases are:\n\n- domain enum values and state names;\n- a versioned wire-protocol version or framing token;\n- reserved internal headers and callback prefixes that peers must interpret\n identically;\n- compile-time mathematical relationships;\n- sentinel errors and internal type identifiers.\n\nNames such as a stream, consumer, label, workflow, file, directory, network,\nmodel role, timeout, retry count, capacity, size limit, colour, prompt, command,\nor executable are not invariants merely because the current application has only\none of them.\n\nArchitecture tests SHOULD reject exported behavioural constants outside approved\ninvariant packages and SHOULD detect direct consumption of private default\nconstants outside their owning `DefaultSettings` construction.\n\n## Current mutation and reload ledger\n\nThe application currently changes behavioural values through unrelated paths:\n\n| Current change seam | Current behaviour | Required target |\n| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Telegram `/restart` in `cmd/archied` | Re-reads the global document and directly replaces Telegram token and allowed-user IDs; failure retains old values. | Submit a Telegram-settings candidate through the universal change protocol. Validate the complete candidate, promote atomically, audit the acting identity, and retain last-known-good. |\n| Active model selection | `chatModelManager.SetActiveModel` changes a model reference. | A conversation/session command changing runtime state. Validate against configured model instances and audit it; do not rewrite configuration. |\n| Active provider selection | `chatModelManager.SetActiveProvider` changes provider selection. | A conversation/session command changing runtime state, with the same distinction from provider-instance configuration. |\n| Persona selection | `PersonaRegistry.SetActive` mutates an in-memory per-session choice. | Agent-system session state. Persistence and audit requirements are owned by that domain, not the configuration loader. |\n| File and environment overlays | Applied while loading `internal/config`; defaults are filled by mutation in `finalize`. | Configuration-infrastructure input processing that produces separately typed candidates. Defaults and derivations remain visible in provenance. |\n| SIGHUP config reload (2026-08) | Re-resolves the file config, re-applies the database layer over it, validates, then atomically republishes through `config.Holder`. A failed reload, whether the file or the control plane caused it, keeps the running config and records `last_error`/`last_error_at` in `/api/config`. | Candidate promotion with last-known-good retention: the running snapshot is untouched on failure. A bounded observation period and actor audit are not yet implemented. |\n| Control-plane resource command (2026-09) | `ControlPlaneService.Command` validates a whole resource document against its owning feature, then writes it with an audit record (actor, source, request ID, expected version) under optimistic concurrency. Hosted on the State Store's gRPC server; the Web UI and messaging channels are its only clients. See `docs/prds/runtime-control-plane.md`. | Met for validate-stage-promote, actor audit and versioned history. Per-field health observation and validate-before-switch on live apply are not yet implemented. |\n\nAny further mutation seam discovered during migration MUST be added here before\nit is changed. Direct field assignment on a live settings object is prohibited.\n\n## Universal change protocol\n\nChanging configuration is an application command performed by an identity. It\nMUST:\n\n1. resolve the target owner and current version;\n2. parse into that owner's typed settings;\n3. validate syntax, semantic constraints, policy, references, and safe bounds;\n4. stage a candidate without altering the active value;\n5. run owner-defined readiness and health checks;\n6. atomically promote the candidate and record actor, source, reason, diff,\n prior version, and new version;\n7. observe the affected capability for a bounded period;\n8. retain the new version on success, or automatically restore last-known-good\n on failure;\n9. report the rejected or rolled-back candidate without terminating unaffected\n capabilities.\n\nStartup loading uses the same candidate rules. A bad user value therefore does\nnot partially mutate the application or make a previously healthy capability\nunavailable.\n\nSettings that cannot safely reload in-process are marked restart-required. The\nchange may still be validated and versioned immediately, but promotion occurs\nthrough a controlled restart with the same health and rollback semantics.\n\n## Implemented runtime reload and settings writes (2026-09)\n\nTwo mechanisms change behavioural values at runtime: SIGHUP re-reads the file\nconfig, and the control plane owns every database-backed setting. The dashboard\nruntime overlay that used to sit between them was deleted once the control plane\ncovered its surface; `-config-overlay` names a file overlay and is unrelated.\n\n### Published snapshot: `config.Holder`\n\nThe running configuration is a published snapshot held in `config.Holder`\n(`internal/config/holder.go`). Get returns a value under RLock; Set replaces the\nwhole snapshot. A published Config is immutable: reload constructs a fresh one\nand calls Set. `config.Clone` deep-copies every reference-type field so a failed\ndecode cannot mutate the published snapshot's shared maps. `Holder.Get` panics\non a nil receiver — a forgotten `NewHolder` must fail loudly, not boot a daemon\nthat silently runs on an empty config.\n\n### The reloadable-field criterion\n\nThe diff-and-warn on reload asks whether a changed field takes effect without a\nrestart. The criterion is: **a field is reloadable when every consumer re-reads\nit.** That question read as \"what does the daemon re-read\" while the daemon owned\nthe only Holder; it stopped being sufficient when webui began sharing that\nHolder, because webui handlers re-read on every request. The allowlist lives in\n`cmd/archied/reload.go` and is deliberately a deny-by-default list: a field added\nto Config later defaults to requires-restart, forcing whoever adds it to decide\ndeliberately. A reload that changes any non-allowlisted field logs\n\"requires a restart\" with the field names; a fully-applied reload logs Info.\n\n### SIGHUP reload\n\nSIGHUP re-resolves the file config, re-applies the database layer over it\n(`boot.runtimeConfig`, the same call boot makes), validates, and atomically\nrepublishes through the shared Holder. A failed reload keeps the running config\nand records `last_error`/`last_error_at` in `/api/config` so the stale state is\nvisible where the operator is looking. The three poll tickers re-read\n`PollInterval` per iteration and reset on change, so a reloaded interval\ngenuinely takes effect.\n\n### Settings writes go through the control plane\n\nDatabase-backed settings are read and written through\n`ControlPlaneService` on the State Store's gRPC server: catalog, query,\ncommand, watch. Each feature owns its resource's validation and commands;\nstorage writes versions and audit records only. `docs/prds/runtime-control-plane.md`\nis the specification.\n\nTwo properties matter to anyone changing this area:\n\n- **Bootstrap stays file-owned.** The daemon reads the file config first and\n layers database resources over it at boot (`boot.loadRuntimeConfig`). Values\n the daemon needs before it can reach the State Store — the database path,\n listen addresses, credentials — stay file-only by design.\n- **Reload re-reads them.** SIGHUP resolves the file document and then runs the\n same `boot.runtimeConfig` layering boot does, so a reload cannot revert a\n database-owned setting to its file value. A control plane that cannot answer\n fails the reload rather than publishing the file's values over the running\n ones. Workflow execution budgets arrive on a watch rather than in a resource\n query, so boot records the last published settings and the reload re-applies\n them from there.\n\n## Completion criteria\n\nThe configuration rearchitecture is complete only when:\n\n- `internal/config` no longer exists;\n- every behavioural value has one classification and owner;\n- every current field and hard-coded value in the ledgers has a recorded\n disposition;\n- domains and plugins receive only their typed settings;\n- all changes use semantic commands and the universal change protocol;\n- active, candidate, rejected, rolled-back, and last-known-good versions are\n queryable with secret-safe provenance;\n- generated documentation lists every external key, default, constraint,\n mutability class, owner, and deprecation;\n- architecture tests prevent a new central runtime configuration model, direct\n live-setting mutation, and incorrectly fixed behavioural values.\n\n## Startup policy: missing credential degrades, invalid config is fatal\n\nDecided 2026-07-30. Recovered 2026-08-09 from the pre-migration issue tracker.\n\nA missing **credential** disables a capability. Only an **invalid config** stops\nthe daemon. \"Forge is a feature, not a requirement.\"\n\nBefore `518a1a0`, an unresolvable forge token did `return 1` in\n`cmd/archied/main.go`, so a chat-only user who never configured GitHub got no\nchat, no gateway, nothing — and under a systemd unit with `Restart=on-failure`\nand no start limit it crash-looped every five seconds with the error scrolling\npast unread. Forge construction now lives in `resolveForge`, shared by the\nprimary forge and every configured identity, and degrades to `forge.NewNoop`\nwith a warning. The identity path had the same defect and was worse: one\nidentity's missing credential killed all the others.\n\nMalformed configuration still fails fast in `configuration.Validate`, well\nbefore this point. That distinction is the whole design.\n\nApply the same reasoning to any future capability credential, LLM keys\nespecially: start, report the capability as unavailable, let the operator see it.\nGuarded by `TestResolveForgeDegradesInsteadOfFailing`.\n\n**Known consequence, tracked separately:** `worktree.Manager` still receives the\nempty token, so pushes fail late rather than at startup.\n\n## Config generation belongs to `archied setup`, not `install.sh`\n\nDecided 2026-07-30 ([#368](https://github.com/samcharles93/archie-core/issues/368)).\nRecovered 2026-08-09 from the pre-migration issue tracker.\n\n`install.sh` must stop generating `config.toml`; `archied setup` owns it. The\ninstaller hand-rolling TOML has already drifted from the schema once: the\ngenerated-config branch hardcoded `key = ARCHIE_GITHUB_TOKEN` and\n`host = ${GITEA_URL}` regardless of the answers given, so choosing Gitea produced\na config naming a token the installer never wrote (archied then exited 1), and\nchoosing GitHub pointed `host` at `gitea.example.com`. `61a5b35` patched this\nwith a `forge_block` shell helper shared by both config paths — that helper is a\n**workaround**, deleted by\n[#72](https://github.com/samcharles93/archie-core/issues/72). The durable fix is\nthat the code writing the config is the code that reads it.\n\nWhy Go and not Python: the installer must run where nothing is installed yet, and\n`python3` with `tomllib` means 3.11+, which is not safe on LTS distros. `archied`\nis already being built two steps earlier.\n\n**Key constraint for the port:** `BurntSushi/toml` does **not** survive a\ndecode/encode round trip with comments intact, and `config.example.toml` carries\ndocumentation users are meant to hand-edit. Decide the write strategy\n([#412](https://github.com/samcharles93/archie-core/issues/412)) before writing\nany prompting code. Do not port a separate `config.yaml`/`auth.yaml` secret\nsplit — Archie already has `secret.SecretRef{engine,key}`, so setup writes the\nreference to TOML and the value to the secret engine.\n" + "body": "# Configuration\n\n**Status:** Approved; existing-type assignment is ongoing \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Ownership\n\nConfiguration has two separate responsibilities:\n\n1. `internal/infrastructure/configuration` loads files, environment variables,\n secret references, overlays, and compatibility representations.\n2. Each domain and plugin owns the typed runtime settings required for its\n behaviour.\n\n`internal/app` translates external input into those settings, validates\ncross-capability combinations, and supplies each component only its own\nsettings.\n\nThe following are prohibited:\n\n- passing complete application configuration into a domain;\n- making a domain depend on TOML, YAML, environment-variable names, secret\n references, or a configuration library;\n- defining a central runtime configuration type imported across the system;\n- placing plugin options in an unrelated global configuration model.\n\n## Private input document\n\n`internal/infrastructure/configuration` MAY use a complete decoded document\nwhile discovering files, decoding sections, and applying overlays. It is an\ninput-boundary DTO only.\n\nThe document:\n\n- remains private to the configuration infrastructure;\n- is composed from separately defined feature sections;\n- is immediately translated by application composition;\n- is never returned as the application's runtime model;\n- is never passed through runtime components.\n\nExternal DTOs and domain runtime settings are separate types.\n\nPlugins supply external configuration definitions through a narrow extension\ncontract. The core loader MUST NOT gain a field or recognized filename for every\nplugin.\n\nInvalid configuration input MUST NOT replace active valid settings or terminate\nunaffected capabilities. Configuration changes use the candidate validation,\npromotion, audit, and recovery requirements in\n[safe-change-and-recovery.md](/dev/architecture/safe-change-and-recovery/).\n\n## Dissolution of `internal/config`\n\n`internal/config` has no target location. It MUST be dissolved and MUST NOT\nsurvive as a shared package, configuration domain, or renamed global model.\n\nIts loading behaviour moves to `internal/infrastructure/configuration`. Existing\ntypes and methods are reassigned by behaviour. Repository execution policy,\ndispatch policy, retry policy, ecosystem defaults, tool policy, memory settings,\nprovider settings, and channel settings move to their\nrespective owners.\n\n## Repository settings\n\nThe existing `config.Repo` MUST NOT move intact into a repository domain or\nshared settings package. A managed repository is an application association:\n\n- work intake owns which repository references an identity monitors;\n- task execution owns gates, preflight, test protection, ecosystem behaviour,\n and change limits;\n- task lifecycle owns retry behaviour;\n- scheduling owns concurrency policy;\n- workspace management owns base branch and persistent-storage requirements.\n\nApplication composition correlates these settings by repository reference. No\ndomain receives a universal repository profile.\n\n## Documentation\n\nGenerated configuration reference describes external sections and their owning\ncapabilities. It MUST NOT establish a global Go configuration structure as the\npublic contract.\n\n## Value classification rule\n\nEvery value that controls behaviour MUST have an explicit owner and MUST be\nclassified as exactly one of:\n\n1. **Invariant** — changing it changes the meaning of the program, a wire\n protocol, or a domain concept. It may be a Go constant.\n2. **Runtime setting** — an operator, identity, domain, plugin, or deployment\n may legitimately choose a different value. It is typed configuration owned by\n the affected capability.\n3. **Policy value** — it expresses what is allowed, limited, routed, retried,\n protected, or gated. Its typed definition belongs to the owning domain and it\n is evaluated through the policy system.\n4. **Derived value** — it is computed from other settings and is not\n independently mutable.\n5. **Runtime state** — it changes as the application operates, such as the\n active model for a conversation. It is changed by a domain command and is not\n configuration merely because it can be changed.\n\nAn unclassified behavioural literal is a defect.\n\n`const` MUST NOT be used merely because a value currently has one default.\nConstants are reserved for actual invariants: enum discriminants, protocol\nversions, fixed compatibility tokens, and values whose alteration would change\nthe program's semantics rather than configure an installation.\n\nIn Go, a constant is a compile-time value and does not represent mutable runtime\nstorage. It generally has no address that can be observed or changed.\nArchitecturally, a constant therefore asserts that the value cannot change for\nthe lifetime of the application. Any value that may legitimately change while\nthe application is running, between application instances, or between\ndeployments is not a constant.\n\nA runtime setting may have a default, but the default is not its only access\npath. The owner exposes a typed settings structure, validation, and a\n`DefaultSettings` constructor or equivalent. Private constants used to implement\ndefaults do not turn those defaults into invariants and MUST NOT be consumed\ndirectly by runtime code.\n\nEach runtime setting and policy value MUST declare:\n\n- its owner;\n- type, units, validation constraints, and safe bounds;\n- default and derivation, if any;\n- whether it is startup-only, live-reloadable, or restart-required;\n- whether it may differ by installation, identity, repository, workflow,\n channel, or plugin instance;\n- whether it is secret, sensitive, or safe to report;\n- its current source, effective value, version, and last-known-good version;\n- the command and identity required to change it.\n\n## Current configuration migration ledger\n\nThis ledger records the current meanings hidden inside `internal/config`. It is\nthe source inventory for dissolving that package; rows are split where the\nexisting structure combines different owners.\n\n| Current values | Current location | Target owner and requirement |\n| ----------------------------------------------------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `WorkDir`, `StateDir`, `DBPath` | root `Config` | Workspace/storage infrastructure settings, bootstrap-only. `StateDir` holds the embedded NATS store and endpoint file, task logs and the readiness disk target. `DBPath` only locates the legacy SQLite files (`\u003cdb_path\u003e-tasks.sqlite`, `-eda.sqlite`, `-conversations.sqlite`) for `archie-state-store import` and the boot gate that refuses to serve until it has run; it goes when the importer does. Defaults derive from the platform data directory. |\n| `SkillsDir`, `PluginDir` | root `Config` | Skill and plugin infrastructure settings, independently typed. |\n| `PollInterval` | root `Config` | Work-intake scheduling setting; not a daemon-wide constant. |\n| `Label`, `BotUser`, `BotEmail` | root `Config` | Identity and dispatch associations. `BotEmail` may be derived from the selected forge account but remains overrideable. |\n| `MaxRetries` | root `Config`, `Repo` | Retry policy owned by the operation being retried. The duplicate global/repository representation must disappear. |\n| `DiffCapLines` | root `Config` | Agent-change policy value. |\n| `MaxSteps`, `WallClock`, `GateMaxFailures` | `Budgets` | Workflow-execution policy, with units and zero/unlimited semantics made explicit. |\n| `Class`, `APIKeyEnv`, `BaseURL` | `Provider` | Provider instance settings. Secret lookup becomes a `SecretRef`, not an environment variable name leaking into the provider domain. |\n| `Mode`, `Command`, `Env` | `LegacyAgent` | Decode-only compatibility input for the removed executor selector. Delete after the config migration window; it has no runtime settings owner. |\n| `Type`, `Host`, `Token` / `TokenEnv` (older form) | `Forge` | Forge adapter settings. The older `TokenEnv` input is accepted only at the loader boundary. |\n| `Trigger`, `AckReaction`, label names | `Dispatch` | Work-intake and forge presentation policy. Label vocabulary is configurable per forge/identity association. |\n| `Engine` | `MemoryConfig` | Memory engine selection. `internal/domain/memory` is the only engine family; per-engine settings remain owned and validated by that engine. |\n| `Name`, `Transport`, `Command`, `Args`, `WorkDir`, `URL` | `MCPServer` | One typed MCP server instance specification. Transport-specific variants must reject fields belonging to other variants. |\n| `MaxResultChars` | `ToolPolicy` | Tool-execution policy values, not generic daemon configuration. |\n| `URL`, `TokenEnv` | `NATSConfig` | NATS transport settings with a secret reference replacing the environment-name field. |\n| `Image`, `MaxConcurrency`, `MaxUptime`, `VolumeTTL`, `PullPolicy`, `Network` | `ContainerConfig` | Mandatory managed-worker infrastructure settings and resource policy. The legacy `enabled` input is decode-only. |\n| model mappings, email, webhook, Telegram | `ChatConfig` | Separate channel instance settings plus channel-routing associations; there is no chat-wide settings owner. |\n| `AllowedUserIDs`, `Token` / `TokenEnv` (older form), update commands | Telegram config | Telegram adapter settings and access policy. `Token` is the preferred `secret.SecretRef`; `TokenEnv` remains a backwards-compatible fallback. |\n| listen/relay addresses | email config | Email adapter instance settings. |\n| webhook target | `Notify` | Notification adapter instance settings. |\n| web listen address | `Web` | Web transport settings. |\n| `listen` under `[services.state]` and `[services.gateway]` | `ServiceConnection` | The bind address of the process that owns the service. Read only by that process, and defaulted to the address its clients dial so the pair cannot drift. |\n| forge, dispatch, models, providers, containers, memory, tools and repositories | `IdentityConfig` | Existing composite duplicates unrelated settings. Replace it with associations from an `IdentityID` to independently owned configured instances. |\n| owner, name, base, gate, protection, ecosystem, preflight, storage, retry and concurrency | `Repo` | Split across work intake, workspace, workflow execution, policy, gate, and scheduling as described in Repository settings. |\n| `TaskConfig`, `TaskForge` | worker snapshot | Replace with the minimum immutable, versioned WorkflowExecution input assembled by the application. It is not a second configuration model. |\n| `Extra` and arbitrary maps | root/provider config | Allowed only at an extension boundary whose owning plugin/provider supplies schema and validation. Core configuration never consumes an untyped catch-all. |\n\n## Behavioural literals and incorrectly fixed values\n\nThe following current values affect useful behaviour but are not represented as\nowned typed settings or policy. They MUST be classified and migrated before the\nold configuration package is removed. The listed target is the architectural\ndecision; exact setting names are implementation details.\n\n| Capability | Values fixed in current code | Required classification |\n| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Chat agent | maximum steps `8` in `cmd/archied` | Workflow/chat execution policy. |\n| Context compression | enabled, threshold `0.5`, context size `128000`, protect first `3` and last `20`, compression marker, token estimate of four characters per token | Model/context runtime settings. The marker is presentation; the estimate is provider/model strategy, not a universal truth. |\n| Tool guardrails | escalation thresholds `3`, `5`, `10`, `2` | Tool-use policy with named meanings and bounded defaults. |\n| Tool output | per-result cap `50000` characters, no aggregate turn cap, preview `500` characters | Tool-execution resource and presentation settings. Per-result spill/truncation remains; turn continuation is not character- or token-gated. |\n| Pairing/access | code TTL `1h`, maximum pending `3`, rate window `10m`, failures `5`, lockout `1h`, maximum lockout `24h` | Access policy. Code length/alphabet and salt length require an explicit security-constraint decision; they are not automatically valid constants. |\n| NATS daemon consumer | stream `ARCHIE_TASKS`, consumer `archie-daemon`, deduplication `2m`, poll `2s`, deliveries `3`, acknowledgement wait `5m`, inactive threshold `24h`, subjects | NATS topology settings plus delivery policy. Subject grammar or protocol headers may be invariants only if declared as versioned wire contracts. |\n| NATS worker consumer | stream `ARCHIE_TASKS`, consumer `archie-agent`, deduplication `2m`, poll `5s`, acknowledgement wait `30m`, deliveries `3`, inactive threshold `24h`, queue group and retry delay `1s` | Worker transport settings and delivery/retry policy. Shared stream defaults must have one owner rather than two copies. |\n| Worker readiness | readiness timeout `20s`, retry delay `250ms` | Worker lifecycle and retry policy. |\n| Execution/RPC | agent execution `30m`, task-run RPC `60s`, worktree RPC `15m` | Per-operation timeout policy. |\n| Plugin/tool cleanup | plugin rollback `10s`, tool-provider cleanup `10s`, MCP-provider cleanup `10s` | Lifecycle shutdown/rollback policy owned by each engine, with a shared policy vocabulary where appropriate. |\n| Memory read path | per-scope record limit `20`, rendered `\u003cmemory\u003e` block byte cap `8192` | Memory read policy owned by the engine family (`internal/domain/memory`). |\n| Built-in memory store | per-file size limit `100KiB`, query result limit `100` | Split into memory resource policy, input policy, and presentation settings. |\n| Memory safety scanner | compiled prompt-injection and sensitive-data patterns with fixed warn/block consequences | Versioned memory-domain policy definitions. Patterns and consequences must be inspectable and replaceable without editing scanner code. |\n| MCP stdio | initial backoff `500ms`, maximum backoff `30s`, unlimited default retries, send and shutdown `5s` | MCP transport retry, request, and lifecycle policy. Unlimited retries must be explicit rather than a zero-value accident. |\n| MCP SSE/HTTP | reconnect `1s` to `30s`, POST/request timeout `30s` | MCP transport settings and retry policy. |\n| Message framing | MCP frame `4MiB`, worker protocol frame `8MiB`, diagnostic `64KiB`, media/base64 `64MiB` | Resource/security policy unless a published wire protocol mandates the limit. Limits must be configurable only within safe bounds. |\n| Webhooks and web UI | webhook bind `0.0.0.0:8644`, webhook shutdown `5s`, web read-header `5s`, web shutdown `2s` | Transport instance and lifecycle settings. |\n| Telegram | typing refresh `4s`, draft interval `700ms`, update-action TTL `10m` | Telegram adapter presentation and lifecycle settings. |\n| Telegram limits | message length `4000` and callback/update prefixes | Record the upstream platform constraint and version it as an adapter compatibility limit; prefixes may remain wire constants when changing them would invalidate outstanding callbacks. |\n| Containers | stop and pull timeouts `10s` and `30s` | Container lifecycle policy. |\n| HTTP clients | feasibility notification timeout `10s`; other paths use `http.DefaultClient` | Adapter request policy. Every outbound client requires an explicit timeout profile. |\n| Workflow routing | labels `bug`, `feature`, `bootstrap`; workflows `tdd`, `feasibility`, `bootstrap`; fallback `implement` | Agent-system routing policy. Workflow identifiers are references, not compiled routing decisions. |\n| Filesystem/deployment | `.agents/skills`, `.archie/stages`, `.archie/gate.go`, `/data/worktree`, `/data/repo`, cache names/paths and prepared sentinel | Split into deployment settings, repository conventions, and genuine internal protocol markers. Paths an installation may relocate are settings. |\n| Packaged release data | `/usr/share/archie/CHANGELOG.md` | Packaging/deployment settings or paths derived from an installation prefix. |\n| Forge presentation | default label colour `bfd4f2` and state-label colours | Forge adapter presentation settings. |\n| Fallback system prompt | compiled fallback prompt | Agent-system default content, versioned and replaceable through its owning settings; safety constraints remain policy/code. |\n| Default paths and identities | config path, data paths, listen addresses, model roles and workflow names | Typed defaults belonging to their respective owners, never global constants consumed throughout the application. |\n\nThis inventory is intentionally broader than exported Go `const` declarations. A\nliteral hidden in a constructor, timeout, switch, retry loop, path join, prompt,\nlabel map, or client setup restricts behaviour just as effectively as an\nincorrectly used constant.\n\n## Constants that may remain\n\nA value may remain constant only when its owner documents why runtime variation\nwould be invalid. Typical valid cases are:\n\n- domain enum values and state names;\n- a versioned wire-protocol version or framing token;\n- reserved internal headers and callback prefixes that peers must interpret\n identically;\n- compile-time mathematical relationships;\n- sentinel errors and internal type identifiers.\n\nNames such as a stream, consumer, label, workflow, file, directory, network,\nmodel role, timeout, retry count, capacity, size limit, colour, prompt, command,\nor executable are not invariants merely because the current application has only\none of them.\n\nArchitecture tests SHOULD reject exported behavioural constants outside approved\ninvariant packages and SHOULD detect direct consumption of private default\nconstants outside their owning `DefaultSettings` construction.\n\n## Current mutation and reload ledger\n\nThe application currently changes behavioural values through unrelated paths:\n\n| Current change seam | Current behaviour | Required target |\n| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Telegram `/restart` in `cmd/archied` | Re-reads the global document and directly replaces Telegram token and allowed-user IDs; failure retains old values. | Submit a Telegram-settings candidate through the universal change protocol. Validate the complete candidate, promote atomically, audit the acting identity, and retain last-known-good. |\n| Active model selection | `chatModelManager.SetActiveModel` changes a model reference. | A conversation/session command changing runtime state. Validate against configured model instances and audit it; do not rewrite configuration. |\n| Active provider selection | `chatModelManager.SetActiveProvider` changes provider selection. | A conversation/session command changing runtime state, with the same distinction from provider-instance configuration. |\n| Persona selection | `PersonaRegistry.SetActive` mutates an in-memory per-session choice. | Agent-system session state. Persistence and audit requirements are owned by that domain, not the configuration loader. |\n| File and environment overlays | Applied while loading `internal/config`; defaults are filled by mutation in `finalize`. | Configuration-infrastructure input processing that produces separately typed candidates. Defaults and derivations remain visible in provenance. |\n| SIGHUP config reload (2026-08) | Re-resolves the file config, re-applies the database layer over it, validates, then atomically republishes through `config.Holder`. A failed reload, whether the file or the control plane caused it, keeps the running config and records `last_error`/`last_error_at` in `/api/config`. | Candidate promotion with last-known-good retention: the running snapshot is untouched on failure. A bounded observation period and actor audit are not yet implemented. |\n| Control-plane resource command (2026-09) | `ControlPlaneService.Command` validates a whole resource document against its owning feature, then writes it with an audit record (actor, source, request ID, expected version) under optimistic concurrency. Hosted on the State Store's gRPC server; the Web UI and messaging channels are its only clients. See `docs/prds/runtime-control-plane.md`. | Met for validate-stage-promote, actor audit and versioned history. Per-field health observation and validate-before-switch on live apply are not yet implemented. |\n\nAny further mutation seam discovered during migration MUST be added here before\nit is changed. Direct field assignment on a live settings object is prohibited.\n\n## Universal change protocol\n\nChanging configuration is an application command performed by an identity. It\nMUST:\n\n1. resolve the target owner and current version;\n2. parse into that owner's typed settings;\n3. validate syntax, semantic constraints, policy, references, and safe bounds;\n4. stage a candidate without altering the active value;\n5. run owner-defined readiness and health checks;\n6. atomically promote the candidate and record actor, source, reason, diff,\n prior version, and new version;\n7. observe the affected capability for a bounded period;\n8. retain the new version on success, or automatically restore last-known-good\n on failure;\n9. report the rejected or rolled-back candidate without terminating unaffected\n capabilities.\n\nStartup loading uses the same candidate rules. A bad user value therefore does\nnot partially mutate the application or make a previously healthy capability\nunavailable.\n\nSettings that cannot safely reload in-process are marked restart-required. The\nchange may still be validated and versioned immediately, but promotion occurs\nthrough a controlled restart with the same health and rollback semantics.\n\n## Implemented runtime reload and settings writes (2026-09)\n\nTwo mechanisms change behavioural values at runtime: SIGHUP re-reads the file\nconfig, and the control plane owns every database-backed setting. The dashboard\nruntime overlay that used to sit between them was deleted once the control plane\ncovered its surface; `-config-overlay` names a file overlay and is unrelated.\n\n### Published snapshot: `config.Holder`\n\nThe running configuration is a published snapshot held in `config.Holder`\n(`internal/config/holder.go`). Get returns a value under RLock; Set replaces the\nwhole snapshot. A published Config is immutable: reload constructs a fresh one\nand calls Set. `config.Clone` deep-copies every reference-type field so a failed\ndecode cannot mutate the published snapshot's shared maps. `Holder.Get` panics\non a nil receiver — a forgotten `NewHolder` must fail loudly, not boot a daemon\nthat silently runs on an empty config.\n\n### The reloadable-field criterion\n\nThe diff-and-warn on reload asks whether a changed field takes effect without a\nrestart. The criterion is: **a field is reloadable when every consumer re-reads\nit.** That question read as \"what does the daemon re-read\" while the daemon owned\nthe only Holder; it stopped being sufficient when webui began sharing that\nHolder, because webui handlers re-read on every request. The allowlist lives in\n`cmd/archied/reload.go` and is deliberately a deny-by-default list: a field added\nto Config later defaults to requires-restart, forcing whoever adds it to decide\ndeliberately. A reload that changes any non-allowlisted field logs\n\"requires a restart\" with the field names; a fully-applied reload logs Info.\n\n### SIGHUP reload\n\nSIGHUP re-resolves the file config, re-applies the database layer over it\n(`boot.runtimeConfig`, the same call boot makes), validates, and atomically\nrepublishes through the shared Holder. A failed reload keeps the running config\nand records `last_error`/`last_error_at` in `/api/config` so the stale state is\nvisible where the operator is looking. The three poll tickers re-read\n`PollInterval` per iteration and reset on change, so a reloaded interval\ngenuinely takes effect.\n\n### Settings writes go through the control plane\n\nDatabase-backed settings are read and written through\n`ControlPlaneService` on the State Store's gRPC server: catalog, query,\ncommand, watch. Each feature owns its resource's validation and commands;\nstorage writes versions and audit records only. `docs/prds/runtime-control-plane.md`\nis the specification.\n\nTwo properties matter to anyone changing this area:\n\n- **Bootstrap stays file-owned.** The daemon reads the file config first and\n layers database resources over it at boot (`boot.loadRuntimeConfig`). Values\n the daemon needs before it can reach the State Store — the database path,\n listen addresses, credentials — stay file-only by design.\n- **Reload re-reads them.** SIGHUP resolves the file document and then runs the\n same `boot.runtimeConfig` layering boot does, so a reload cannot revert a\n database-owned setting to its file value. A control plane that cannot answer\n fails the reload rather than publishing the file's values over the running\n ones. Workflow execution budgets arrive on a watch rather than in a resource\n query, so boot records the last published settings and the reload re-applies\n them from there.\n\n## Completion criteria\n\nThe configuration rearchitecture is complete only when:\n\n- `internal/config` no longer exists;\n- every behavioural value has one classification and owner;\n- every current field and hard-coded value in the ledgers has a recorded\n disposition;\n- domains and plugins receive only their typed settings;\n- all changes use semantic commands and the universal change protocol;\n- active, candidate, rejected, rolled-back, and last-known-good versions are\n queryable with secret-safe provenance;\n- generated documentation lists every external key, default, constraint,\n mutability class, owner, and deprecation;\n- architecture tests prevent a new central runtime configuration model, direct\n live-setting mutation, and incorrectly fixed behavioural values.\n\n## Startup policy: missing credential degrades, invalid config is fatal\n\nDecided 2026-07-30. Recovered 2026-08-09 from the pre-migration issue tracker.\n\nA missing **credential** disables a capability. Only an **invalid config** stops\nthe daemon. \"Forge is a feature, not a requirement.\"\n\nBefore `518a1a0`, an unresolvable forge token did `return 1` in\n`cmd/archied/main.go`, so a chat-only user who never configured GitHub got no\nchat, no gateway, nothing — and under a systemd unit with `Restart=on-failure`\nand no start limit it crash-looped every five seconds with the error scrolling\npast unread. Forge construction now lives in `resolveForge`, shared by the\nprimary forge and every configured identity, and degrades to `forge.NewNoop`\nwith a warning. The identity path had the same defect and was worse: one\nidentity's missing credential killed all the others.\n\nMalformed configuration still fails fast in `configuration.Validate`, well\nbefore this point. That distinction is the whole design.\n\nApply the same reasoning to any future capability credential, LLM keys\nespecially: start, report the capability as unavailable, let the operator see it.\nGuarded by `TestResolveForgeDegradesInsteadOfFailing`.\n\n**Known consequence, tracked separately:** `worktree.Manager` still receives the\nempty token, so pushes fail late rather than at startup.\n\n## Config generation belongs to `archied setup`, not `install.sh`\n\nDecided 2026-07-30 ([#368](https://github.com/samcharles93/archie-core/issues/368)).\nRecovered 2026-08-09 from the pre-migration issue tracker.\n\n`install.sh` must stop generating `config.toml`; `archied setup` owns it. The\ninstaller hand-rolling TOML has already drifted from the schema once: the\ngenerated-config branch hardcoded `key = ARCHIE_GITHUB_TOKEN` and\n`host = ${GITEA_URL}` regardless of the answers given, so choosing Gitea produced\na config naming a token the installer never wrote (archied then exited 1), and\nchoosing GitHub pointed `host` at `gitea.example.com`. `61a5b35` patched this\nwith a `forge_block` shell helper shared by both config paths — that helper is a\n**workaround**, deleted by\n[#72](https://github.com/samcharles93/archie-core/issues/72). The durable fix is\nthat the code writing the config is the code that reads it.\n\nWhy Go and not Python: the installer must run where nothing is installed yet, and\n`python3` with `tomllib` means 3.11+, which is not safe on LTS distros. `archied`\nis already being built two steps earlier.\n\n**Key constraint for the port:** `BurntSushi/toml` does **not** survive a\ndecode/encode round trip with comments intact, and `config.example.toml` carries\ndocumentation users are meant to hand-edit. Decide the write strategy\n([#412](https://github.com/samcharles93/archie-core/issues/412)) before writing\nany prompting code. Do not port a separate `config.yaml`/`auth.yaml` secret\nsplit — Archie already has `secret.SecretRef{engine,key}`, so setup writes the\nreference to TOML and the value to the secret engine.\n" }, { "url": "/dev/architecture/dashboard-agent-tools/", @@ -52,25 +52,25 @@ "url": "/dev/architecture/identity/", "title": "Identity", "section": "Architecture", - "body": "# Identity\n\n**Status:** Approved foundation; lifecycle details are deferred \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nIdentity answers which persistent actor owns or performed an action. It supplies\nstable cross-domain attribution and lifecycle semantics.\n\nIdentity kinds include:\n\n- the Archie system identity, such as `System` or `archied`;\n- agent or bot identities;\n- service-account identities;\n- future user identities.\n\nA user is another kind of identity rather than a separate foundation concept.\n\n## Ownership\n\n`internal/domain/identity` owns:\n\n- immutable canonical `IdentityID` values;\n- identity kind;\n- lifecycle state;\n- ownership and attribution semantics;\n- identity-related commands and events.\n\nDomains may record an identity reference on their own state and events when\nownership or attribution is required. They MUST NOT receive a complete\nper-identity application configuration bundle.\n\nPersona and communication style are separate from identity.\n\nEvery Agent has its own immutable acting `IdentityID`. Agent actions use that\nidentity for ownership and attribution in the same way an organisation records\nwhich user accessed a resource, changed a permission, or performed another\nauditable action.\n\nAgent and Identity remain separate aggregates. Identity supplies the stable\nactor reference; Agent owns assistant behaviour, memory, specialisation,\nconversations, and capability context.\n\nAgent identities MUST NOT be shared between Agents. Automated WorkflowExecutions\nMAY eventually act through a dedicated service-account identity when a specific\nunattended automation requires it. Such use is an explicit execution decision\nand audit principal, not the default for Agents or a substitute for recording\nthe initiating user when one exists.\n\nA Workflow is owned by an IdentityID and executes as its owner by default.\nFuture delegated execution MAY explicitly assign a different `RunAsIdentityID`,\nincluding a user or service-account identity. Ownership, effective execution\nidentity, initiating identity, and delegation provenance remain distinct audit\nfacts.\n\n## Canonical identity\n\nEvery identity has an immutable `IdentityID`. It is the only value used for\ncross-domain ownership, attribution, routing, persistence, commands, and events.\n\nIdentity kind is explicit:\n\n- system;\n- bot;\n- service account;\n- user.\n\nDisplay names are mutable and MUST NOT be used as identifiers. Forge usernames,\nchannel bot names, Git authors, provider account names, and other external\nidentifiers are bindings associated with an identity rather than identity\nitself.\n\n`System` is represented by a real identity and `IdentityID`. The empty string\nMUST NOT retain special meaning as a legacy system identity.\n\nConfiguration refers to identities by `IdentityID` and supplies external\nbindings and capability settings separately. Renaming an identity or changing an\nexternal account MUST NOT change ownership or historical attribution.\n\nSecret-bearing channel bindings remain configuration concerns rather than\nidentity data. In particular, `[chat.telegram].token` is a\n`secret.SecretRef` resolved by the application through the configured secret\nregistry; the older `token_env` field remains a compatibility fallback. Neither\nform changes the identity that owns or performs a chat action.\n\n## Persistence\n\nIdentity records are durable domain state. The Identity domain's repository is\nthe source of truth; infrastructure implements its persistence contract.\n\nConfiguration MAY bootstrap a missing identity through an explicit, audited\ncreation operation and MAY propose runtime capability bindings for an existing\n`IdentityID`. Configuration is not the identity registry.\n\nOmitting an identity from configuration stops assembling the omitted runtime\ncapabilities. It MUST NOT delete the identity record, erase history, or make\nhistorical references unresolvable.\n\nDisabling or retiring an identity requires an explicit Identity-domain command.\nRetired identities remain resolvable for historical attribution. A failed\nexternal binding affects capability availability without destroying or silently\nreplacing the identity.\n\n## Lifecycle\n\nIdentity lifecycle is deliberately small:\n\n- `active`: may participate in runtime composition;\n- `suspended`: deliberately prevented from initiating new work while ownership\n and history remain;\n- `retired`: terminal, unable to initiate work, and permanently retained for\n attribution.\n\nAllowed transitions are:\n\n```text\ncreate -\u003e active\nactive -\u003e suspended\nsuspended -\u003e active\nactive -\u003e retired\nsuspended -\u003e retired\n```\n\n`retired` has no outgoing transition.\n\nCapability health is separate from identity lifecycle. Forge, model, channel,\nworkspace, and other bindings report health independently. An active identity\nmay be partially degraded without becoming suspended or retired. Runtime\nsupervision isolates and restores affected bindings through the safe-change and\nrecovery protocol.\n\n## System identity\n\nArchie has exactly one built-in system identity. It is created during initial\nstore bootstrap, is strictly always active, and cannot be suspended or retired.\nIts `IdentityID` and kind are immutable.\n\nStartup, migration, supervision, remediation, and rollback actions performed by\nthe application use the system identity. The system identity MUST NOT be\ndelegated to individual identities. Every user, bot, service account, or other\nactor mutation records the identity that actually performed it.\n\nSystem capabilities may independently become unhealthy or unavailable without\nchanging the system identity.\n\n## Target placement and contracts\n\nThe Identity domain lives at `internal/domain/identity`.\n\nIt owns:\n\n- `IdentityID`, identity kind, display name, lifecycle, version, and audit\n timestamps;\n- create, rename, suspend, reactivate, and retire commands;\n- corresponding lifecycle events;\n- the repository contract required for durable identity records;\n- lookup of current and historical identities.\n\nExternal account bindings and capability settings remain with the domains that\nuse them and reference `IdentityID`.\n\n## Migration constraints\n\nMigration from the current string model MUST:\n\n- bootstrap the permanent system identity;\n- assign stable IDs to configured identities without losing their names;\n- replace empty-string legacy ownership with the system identity;\n- migrate tasks and sessions to canonical identity references;\n- include identity where task uniqueness and transport deduplication require\n independent ownership;\n- route container RPC through the task owner's capability bindings;\n- retain historical resolution after rename, suspension, or retirement;\n- remove identity-string equality from authentication or access semantics.\n\nThis completes the foundation required to restructure the current Identity\nbehaviour. Exact storage schemas and Go method signatures belong to\nimplementation design, not this architecture review.\n\n## Lifecycle\n\nIdentity lifecycle is deliberately small:\n\n- `active`: may participate in runtime composition;\n- `suspended`: deliberately prevented from initiating new work while ownership\n and history remain;\n- `retired`: terminal, unable to initiate work, and permanently retained for\n attribution.\n\nAllowed transitions are:\n\n```text\ncreate -\u003e active\nactive -\u003e suspended\nsuspended -\u003e active\nactive -\u003e retired\nsuspended -\u003e retired\n```\n\n`retired` has no outgoing transition.\n\nCapability health is separate from identity lifecycle. Forge, model, channel,\nworkspace, and other bindings report health independently. An active identity\nmay be partially degraded without becoming suspended or retired. Runtime\nsupervision isolates and restores affected bindings through the safe-change and\nrecovery protocol.\n\n## Explicit non-responsibilities\n\nIdentity is not authentication and is not access control.\n\nThe current architecture does not require authentication or authorization. Those\ncapabilities MAY be introduced later if deployment or product needs justify\nthem. They MUST remain separate from the identity model.\n\nAn identity reference therefore proves attribution within Archie's trusted\nsystem model; it does not by itself prove credentials or permission.\n\n## Per-identity configuration\n\nValues selected for an identity remain external configuration values. At\ncomposition time they are split into the runtime settings owned by the affected\ndomains and correlated by identity reference.\n\nThe existing `IdentityConfig` aggregate MUST be dissolved. Forge accounts,\nrepositories, models, providers, workspace behaviour, polling, budgets,\nnotifications, and transport namespaces retain their respective owners.\n\n## Audit\n\nConfiguration changes and other state changes are actions attributed to an\nidentity. System-initiated changes use the system identity. Their audit records\nMUST preserve the acting identity, the target, and the resulting change.\n\nThe command ownership and persistence mechanism for configuration changes is a\nseparate configuration-management decision.\n\n## Current implementation\n\nIdentity is not currently implemented as a domain. It is a collection of string\nfields and process-local associations:\n\n- `config.IdentityConfig.Name` is the daemon routing key.\n- `config.IdentityConfig.BotUser` is the external forge and Git author name.\n- `daemon.IdentityRunner` bundles the configured name with a forge client,\n worktree manager, repository list, configuration subset, logger, and polling\n goroutine.\n- `store.Task.Identity` persists the configured name on tasks.\n- NATS task messages copy the same configured name.\n- gateway routing accepts an identity name and uses it to select a repository\n profile.\n- gateway sessions persist `BotUser`, not the configured identity name.\n\nThere is no identity registry, identity record, identity lifecycle state,\nidentity persistence, or identity command/event model. Removing an identity from\nconfiguration makes existing task ownership unresolvable.\n\nAn empty identity string selects the legacy single-identity path. This gives the\nempty string behavioural meaning rather than representing a real system\nidentity.\n\n## Current enforcement and isolation\n\n## Withheld judgement verbs\n\n`approve`, `reject`, `abandon` and `archive` are not exposed to an agent. An agent\nthat can approve can approve its own work, and the one control this surface exists\nto keep is a human accepting it.\n\nThe constraint is attribution, not capability: a task action records the acting\nidentity and the authorising principal, so exposing the verbs requires that an\nagent's action be recorded as an agent's and the authority it used recorded\nseparately. Until a surface carries both, the verbs stay withheld. This is the\nreason the four verbs are absent from the dashboard's in-browser tools, and it is\nwhy releasing them is a change to this boundary rather than a change to the tools.\n\n\nThe current implementation uses identity equality as an authorization check for\nchat task approval and cancellation. This conflicts with the approved rule that\nidentity is attribution rather than authentication or access control and must be\nreassigned to an appropriate task-ownership or future access policy.\n\nFocused tests verify that:\n\n- a task resolves the forge client and repository list associated with its\n configured identity name;\n- cross-identity repository profiles do not leak;\n- chat task creation propagates the selected identity;\n- task persistence retains the identity string;\n- chat task control rejects a different identity string.\n\n## Current hazards\n\n- SQLite task uniqueness is `(owner, repo, issue_number)` and excludes identity.\n Two identities discovering the same forge issue cannot own distinct task\n records.\n- NATS deduplication uses `owner/repo/issue` and excludes identity, so discovery\n by a second identity can be discarded as a duplicate.\n- Container-mode forge and worktree RPC servers are wired to the root clients,\n not the task owner's clients. The source documents this as an unresolved\n identity leak.\n- The shared task queue claims the oldest task without identity partitioning and\n relies on later string lookup to recover the correct runner.\n- Sessions use forge bot usernames where tasks use configured identity names, so\n there is no canonical cross-domain identity reference.\n- Identity lifecycle is process lifetime only; there is no disabled, degraded,\n retired, or unavailable state and no audit history.\n\nThe focused identity tests pass. Broader daemon and NATS suites could not open\nembedded listener ports in the current sandbox; their failures occurred before\nidentity behaviour executed.\n\n## Three-layer identity model\n\nAgreed 2026-07-30; the frame for SOUL\n([#68](https://github.com/samcharles93/archie-core/issues/68)) and the curator\n([#435](https://github.com/samcharles93/archie-core/issues/435)).\nRecovered 2026-08-09 from the pre-migration issue tracker.\n\n**Layer 1 — invariants.** Instruction precedence, core rules, and the `\u003ctools\u003e`\nand `\u003cenv\u003e` blocks live in `internal/gateway/templates/archie.md.tpl` and are\nunreachable at runtime by anyone.\n\n**Layer 2 — SOUL.** User-authored identity: agent name, register, warmth.\nEditable without a rebuild.\n\n**Enforcing the Layer 1 / Layer 2 boundary.** SOUL is rendered as the\n`Persona` field of `SystemPromptConfig` (`internal/gateway/prompt.go`) into a\n`\u003csoul purpose=\"identity_and_style\" trust=\"data\"\u003e` block, XML-escaped through\nthe same `xml` template helper used for tool and env metadata, and placed\n*before* `\u003cinstruction_precedence\u003e` in `archie.md.tpl`. Escaping means SOUL\ntext can never forge a live `\u003ccore_rules\u003e`, `\u003cinstruction_precedence\u003e`,\n`\u003ctools\u003e` or `\u003cenv\u003e` tag -- an attempt renders as inert escaped text\n(`\u0026lt;core_rules\u0026gt;...`), and `\u003cinstruction_precedence\u003e` explicitly tells the\nmodel the `\u003csoul\u003e` block is reference data, not an authority. This is\nguarded by `TestBuildSystemPromptSoulCannotOverrideInvariants` in\n`internal/gateway/prompt_test.go`, which renders a hostile SOUL that attempts\nto countermand `core_rules` and asserts the real invariant tags stay singular\nand their text stays intact. Keep this test in sync if the template's escaping\nor block structure changes -- do not relax it to make a future SOUL feature\n(e.g. curator-proposed edits) fit.\n\n**Layer 3 — representation.** Curator-derived knowledge about the user. It may\nonly **propose** changes to SOUL and never apply them. If derived conclusions can\nsilently rewrite tone, personality drifts and the user loses the control SOUL\nexists to give them. A proposal queue also makes the visibility requirement fall\nout for free, since pending conclusions are inherently reviewable.\n\n**Why SOUL comes first:** tone currently lives in Go string constants\n(`DefaultPersonas` in `internal/gateway/persona.go`), so fixing a rude reply on\n2026-07-29 required editing Go and rebuilding. SOUL is small, fixes a real\ndefect, and defines the boundary the curator must respect.\n" + "body": "# Identity\n\n**Status:** Approved foundation; lifecycle details are deferred \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nIdentity answers which persistent actor owns or performed an action. It supplies\nstable cross-domain attribution and lifecycle semantics.\n\nIdentity kinds include:\n\n- the Archie system identity, such as `System` or `archied`;\n- agent or bot identities;\n- service-account identities;\n- future user identities.\n\nA user is another kind of identity rather than a separate foundation concept.\n\n## Ownership\n\n`internal/domain/identity` owns:\n\n- immutable canonical `IdentityID` values;\n- identity kind;\n- lifecycle state;\n- ownership and attribution semantics;\n- identity-related commands and events.\n\nDomains may record an identity reference on their own state and events when\nownership or attribution is required. They MUST NOT receive a complete\nper-identity application configuration bundle.\n\nPersona and communication style are separate from identity.\n\nEvery Agent has its own immutable acting `IdentityID`. Agent actions use that\nidentity for ownership and attribution in the same way an organisation records\nwhich user accessed a resource, changed a permission, or performed another\nauditable action.\n\nAgent and Identity remain separate aggregates. Identity supplies the stable\nactor reference; Agent owns assistant behaviour, memory, specialisation,\nconversations, and capability context.\n\nAgent identities MUST NOT be shared between Agents. Automated WorkflowExecutions\nMAY eventually act through a dedicated service-account identity when a specific\nunattended automation requires it. Such use is an explicit execution decision\nand audit principal, not the default for Agents or a substitute for recording\nthe initiating user when one exists.\n\nA Workflow is owned by an IdentityID and executes as its owner by default.\nFuture delegated execution MAY explicitly assign a different `RunAsIdentityID`,\nincluding a user or service-account identity. Ownership, effective execution\nidentity, initiating identity, and delegation provenance remain distinct audit\nfacts.\n\n## Canonical identity\n\nEvery identity has an immutable `IdentityID`. It is the only value used for\ncross-domain ownership, attribution, routing, persistence, commands, and events.\n\nIdentity kind is explicit:\n\n- system;\n- bot;\n- service account;\n- user.\n\nDisplay names are mutable and MUST NOT be used as identifiers. Forge usernames,\nchannel bot names, Git authors, provider account names, and other external\nidentifiers are bindings associated with an identity rather than identity\nitself.\n\n`System` is represented by a real identity and `IdentityID`. The empty string\nMUST NOT retain special meaning as a legacy system identity.\n\nConfiguration refers to identities by `IdentityID` and supplies external\nbindings and capability settings separately. Renaming an identity or changing an\nexternal account MUST NOT change ownership or historical attribution.\n\nSecret-bearing channel bindings remain configuration concerns rather than\nidentity data. In particular, `[chat.telegram].token` is a\n`secret.SecretRef` resolved by the application through the configured secret\nregistry; the older `token_env` field remains a compatibility fallback. Neither\nform changes the identity that owns or performs a chat action.\n\n## Persistence\n\nIdentity records are durable domain state. The Identity domain's repository is\nthe source of truth; infrastructure implements its persistence contract.\n\nConfiguration MAY bootstrap a missing identity through an explicit, audited\ncreation operation and MAY propose runtime capability bindings for an existing\n`IdentityID`. Configuration is not the identity registry.\n\nOmitting an identity from configuration stops assembling the omitted runtime\ncapabilities. It MUST NOT delete the identity record, erase history, or make\nhistorical references unresolvable.\n\nDisabling or retiring an identity requires an explicit Identity-domain command.\nRetired identities remain resolvable for historical attribution. A failed\nexternal binding affects capability availability without destroying or silently\nreplacing the identity.\n\n## Lifecycle\n\nIdentity lifecycle is deliberately small:\n\n- `active`: may participate in runtime composition;\n- `suspended`: deliberately prevented from initiating new work while ownership\n and history remain;\n- `retired`: terminal, unable to initiate work, and permanently retained for\n attribution.\n\nAllowed transitions are:\n\n```text\ncreate -\u003e active\nactive -\u003e suspended\nsuspended -\u003e active\nactive -\u003e retired\nsuspended -\u003e retired\n```\n\n`retired` has no outgoing transition.\n\nCapability health is separate from identity lifecycle. Forge, model, channel,\nworkspace, and other bindings report health independently. An active identity\nmay be partially degraded without becoming suspended or retired. Runtime\nsupervision isolates and restores affected bindings through the safe-change and\nrecovery protocol.\n\n## System identity\n\nArchie has exactly one built-in system identity. It is created during initial\nstore bootstrap, is strictly always active, and cannot be suspended or retired.\nIts `IdentityID` and kind are immutable.\n\nStartup, migration, supervision, remediation, and rollback actions performed by\nthe application use the system identity. The system identity MUST NOT be\ndelegated to individual identities. Every user, bot, service account, or other\nactor mutation records the identity that actually performed it.\n\nSystem capabilities may independently become unhealthy or unavailable without\nchanging the system identity.\n\n## Target placement and contracts\n\nThe Identity domain lives at `internal/domain/identity`.\n\nIt owns:\n\n- `IdentityID`, identity kind, display name, lifecycle, version, and audit\n timestamps;\n- create, rename, suspend, reactivate, and retire commands;\n- corresponding lifecycle events;\n- the repository contract required for durable identity records;\n- lookup of current and historical identities.\n\nExternal account bindings and capability settings remain with the domains that\nuse them and reference `IdentityID`.\n\n## Migration constraints\n\nMigration from the current string model MUST:\n\n- bootstrap the permanent system identity;\n- assign stable IDs to configured identities without losing their names;\n- replace empty-string legacy ownership with the system identity;\n- migrate tasks and sessions to canonical identity references;\n- include identity where task uniqueness and transport deduplication require\n independent ownership;\n- route container RPC through the task owner's capability bindings;\n- retain historical resolution after rename, suspension, or retirement;\n- remove identity-string equality from authentication or access semantics.\n\nThis completes the foundation required to restructure the current Identity\nbehaviour. Exact storage schemas and Go method signatures belong to\nimplementation design, not this architecture review.\n\n## Lifecycle\n\nIdentity lifecycle is deliberately small:\n\n- `active`: may participate in runtime composition;\n- `suspended`: deliberately prevented from initiating new work while ownership\n and history remain;\n- `retired`: terminal, unable to initiate work, and permanently retained for\n attribution.\n\nAllowed transitions are:\n\n```text\ncreate -\u003e active\nactive -\u003e suspended\nsuspended -\u003e active\nactive -\u003e retired\nsuspended -\u003e retired\n```\n\n`retired` has no outgoing transition.\n\nCapability health is separate from identity lifecycle. Forge, model, channel,\nworkspace, and other bindings report health independently. An active identity\nmay be partially degraded without becoming suspended or retired. Runtime\nsupervision isolates and restores affected bindings through the safe-change and\nrecovery protocol.\n\n## Explicit non-responsibilities\n\nIdentity is not authentication and is not access control.\n\nThe current architecture does not require authentication or authorization. Those\ncapabilities MAY be introduced later if deployment or product needs justify\nthem. They MUST remain separate from the identity model.\n\nAn identity reference therefore proves attribution within Archie's trusted\nsystem model; it does not by itself prove credentials or permission.\n\n## Per-identity configuration\n\nValues selected for an identity remain external configuration values. At\ncomposition time they are split into the runtime settings owned by the affected\ndomains and correlated by identity reference.\n\nThe existing `IdentityConfig` aggregate MUST be dissolved. Forge accounts,\nrepositories, models, providers, workspace behaviour, polling, budgets,\nnotifications, and transport namespaces retain their respective owners.\n\n## Audit\n\nConfiguration changes and other state changes are actions attributed to an\nidentity. System-initiated changes use the system identity. Their audit records\nMUST preserve the acting identity, the target, and the resulting change.\n\nThe command ownership and persistence mechanism for configuration changes is a\nseparate configuration-management decision.\n\n## Current implementation\n\nIdentity is not currently implemented as a domain. It is a collection of string\nfields and process-local associations:\n\n- `config.IdentityConfig.Name` is the daemon routing key.\n- `config.IdentityConfig.BotUser` is the external forge and Git author name.\n- `daemon.IdentityRunner` bundles the configured name with a forge client,\n worktree manager, repository list, configuration subset, logger, and polling\n goroutine.\n- `store.Task.Identity` persists the configured name on tasks.\n- NATS task messages copy the same configured name.\n- gateway routing accepts an identity name and uses it to select a repository\n profile.\n- gateway sessions persist `BotUser`, not the configured identity name.\n\nThere is no identity registry, identity record, identity lifecycle state,\nidentity persistence, or identity command/event model. Removing an identity from\nconfiguration makes existing task ownership unresolvable.\n\nAn empty identity string selects the legacy single-identity path. This gives the\nempty string behavioural meaning rather than representing a real system\nidentity.\n\n## Current enforcement and isolation\n\n## Withheld judgement verbs\n\n`approve`, `reject`, `abandon` and `archive` are not exposed to an agent. An agent\nthat can approve can approve its own work, and the one control this surface exists\nto keep is a human accepting it.\n\nThe constraint is attribution, not capability: a task action records the acting\nidentity and the authorising principal, so exposing the verbs requires that an\nagent's action be recorded as an agent's and the authority it used recorded\nseparately. Until a surface carries both, the verbs stay withheld. This is the\nreason the four verbs are absent from the dashboard's in-browser tools, and it is\nwhy releasing them is a change to this boundary rather than a change to the tools.\n\n\nThe current implementation uses identity equality as an authorization check for\nchat task approval and cancellation. This conflicts with the approved rule that\nidentity is attribution rather than authentication or access control and must be\nreassigned to an appropriate task-ownership or future access policy.\n\nFocused tests verify that:\n\n- a task resolves the forge client and repository list associated with its\n configured identity name;\n- cross-identity repository profiles do not leak;\n- chat task creation propagates the selected identity;\n- task persistence retains the identity string;\n- chat task control rejects a different identity string.\n\n## Current hazards\n\n- Task uniqueness is `(owner, repo, issue_number)` and excludes identity.\n Two identities discovering the same forge issue cannot own distinct task\n records.\n- NATS deduplication uses `owner/repo/issue` and excludes identity, so discovery\n by a second identity can be discarded as a duplicate.\n- Container-mode forge and worktree RPC servers are wired to the root clients,\n not the task owner's clients. The source documents this as an unresolved\n identity leak.\n- The shared task queue claims the oldest task without identity partitioning and\n relies on later string lookup to recover the correct runner.\n- Sessions use forge bot usernames where tasks use configured identity names, so\n there is no canonical cross-domain identity reference.\n- Identity lifecycle is process lifetime only; there is no disabled, degraded,\n retired, or unavailable state and no audit history.\n\nThe focused identity tests pass. Broader daemon and NATS suites could not open\nembedded listener ports in the current sandbox; their failures occurred before\nidentity behaviour executed.\n\n## Three-layer identity model\n\nAgreed 2026-07-30; the frame for SOUL\n([#68](https://github.com/samcharles93/archie-core/issues/68)) and the curator\n([#435](https://github.com/samcharles93/archie-core/issues/435)).\nRecovered 2026-08-09 from the pre-migration issue tracker.\n\n**Layer 1 — invariants.** Instruction precedence, core rules, and the `\u003ctools\u003e`\nand `\u003cenv\u003e` blocks live in `internal/gateway/templates/archie.md.tpl` and are\nunreachable at runtime by anyone.\n\n**Layer 2 — SOUL.** User-authored identity: agent name, register, warmth.\nEditable without a rebuild.\n\n**Enforcing the Layer 1 / Layer 2 boundary.** SOUL is rendered as the\n`Persona` field of `SystemPromptConfig` (`internal/gateway/prompt.go`) into a\n`\u003csoul purpose=\"identity_and_style\" trust=\"data\"\u003e` block, XML-escaped through\nthe same `xml` template helper used for tool and env metadata, and placed\n*before* `\u003cinstruction_precedence\u003e` in `archie.md.tpl`. Escaping means SOUL\ntext can never forge a live `\u003ccore_rules\u003e`, `\u003cinstruction_precedence\u003e`,\n`\u003ctools\u003e` or `\u003cenv\u003e` tag -- an attempt renders as inert escaped text\n(`\u0026lt;core_rules\u0026gt;...`), and `\u003cinstruction_precedence\u003e` explicitly tells the\nmodel the `\u003csoul\u003e` block is reference data, not an authority. This is\nguarded by `TestBuildSystemPromptSoulCannotOverrideInvariants` in\n`internal/gateway/prompt_test.go`, which renders a hostile SOUL that attempts\nto countermand `core_rules` and asserts the real invariant tags stay singular\nand their text stays intact. Keep this test in sync if the template's escaping\nor block structure changes -- do not relax it to make a future SOUL feature\n(e.g. curator-proposed edits) fit.\n\n**Layer 3 — representation.** Curator-derived knowledge about the user. It may\nonly **propose** changes to SOUL and never apply them. If derived conclusions can\nsilently rewrite tone, personality drifts and the user loses the control SOUL\nexists to give them. A proposal queue also makes the visibility requirement fall\nout for free, since pending conclusions are inherently reviewable.\n\n**Why SOUL comes first:** tone currently lives in Go string constants\n(`DefaultPersonas` in `internal/gateway/persona.go`), so fixing a rude reply on\n2026-07-29 required editing Go and rebuilding. SOUL is small, fixes a real\ndefect, and defines the boundary the curator must respect.\n" }, { "url": "/dev/architecture/messaging-and-work-intake/", "title": "Messaging and Work Intake", "section": "Architecture", - "body": "# Messaging and Work Intake\n\n**Status:** Current state researched; foundation decisions in progress \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie is a personal assistant and general agentic system. Its interactions\ninclude conversation, questions, notes, reminders, goals, message handling,\ntroubleshooting, infrastructure operations, server management, source changes,\nand open-ended future capabilities.\n\nA message to an agent is therefore not inherently a work request, and invoking\nan agent is not inherently a WorkflowExecution.\n\nMessaging and work intake are separate from the Agent System's durable workflow\nexecution model:\n\n- messaging preserves interactions with people and external systems;\n- an agent turn interprets and responds to an interaction using its available\n capabilities;\n- work intake handles the optional promotion of an interaction into durable,\n workflow-backed work;\n- the Agent System owns Workflow definitions and creates WorkflowExecutions.\n\nForge issues, Jira issues, Telegram messages, emails, webhooks, and future\nchannels remain peer external message or artifact sources. None becomes an\nexecution model merely because it can lead to work.\n\n## Confirmed boundary\n\nWork intake MUST NOT create a WorkflowExecution directly.\n\nInbound interactions normally route to the Agent selected by the channel\nbinding. The selected Agent retains its own memory, user knowledge, conversation\ncontinuity, specialisation, and capability context. This continuity is\nindependent of the underlying model or provider.\n\nEach selected Agent acts through its own immutable IdentityID. Interaction and\nresulting action records preserve both that acting identity and the initiating\nuser identity when one exists.\n\nAgent memory is private by default. Interaction context MAY additionally use\nuser-wide memory and intentionally global shared memory, but another Agent's\nagent-wide memory is not part of the selected Agent's context.\n\nMultiple users may interact with the same Agent. The channel binding and inbound\nmessage MUST resolve both AgentID and initiating User IdentityID before\nconversation or memory context is loaded. Sharing an Agent MUST NOT implicitly\nshare one user's private memory with another user.\n\nAn interaction may assemble memory from four scopes: global shared, the selected\nAgent's agent-wide memory, the initiating user's authorized user-wide memory,\nand the private Agent-user relationship memory addressed by both IDs.\n\n## Conversation identity\n\nEvery conversation is identified by the complete tuple:\n\n```text\nAgentID\n+ UserIdentityID\n+ ChannelBindingID\n+ ExternalConversationID\n+ ThreadID\n```\n\n`ThreadID` has an explicit empty value for channels without threads. No element\nof this key may be replaced by a display name, configured bot username, channel\ntype, or other mutable label.\n\nAgent and user memory continue across conversations according to their memory\nscope. Immediate conversation history and delivery correlation remain isolated\nto the identified channel conversation or thread. Moving between Telegram,\nemail, or another channel does not create a new Agent or user, but it does\ncreate a distinct conversation unless an explicit linking operation relates\nthem.\n\nConversation and message events support projections for channel-entry metrics,\nuse cases, volumes, outcomes, latency, capability usage, and other product or\noperational analysis. Metrics are derived read models; they MUST NOT become\nmutable counters embedded in the conversation aggregate or change message\nrouting semantics.\n\n## Canonical messages\n\nEvery inbound, assistant, tool-call, and tool-result message has an immutable\napplication-generated MessageID. Platform identifiers are external correlation,\nnot substitutes for the canonical ID.\n\nMessages are addressable because an interaction is more than text. A message may\ncontain assistant content, reasoning data subject to its retention policy, one\nor more tool calls, a tool result with authoritative execution metadata,\nattachments, reply relationships, edits, or other typed content. Stable\nMessageIDs also allow sessions to branch from an exact prior point.\n\nThe canonical session and message implementation must supply:\n\n- application-generated stable message IDs;\n- user, assistant, and tool roles;\n- assistant tool calls and correlated tool-result messages;\n- authoritative tool-result status, timing, error, truncation, and size\n metadata;\n- session lifecycle and active-request state;\n- session-to-agent-instance and parent-session lineage;\n- durable SQLite round trips and timestamp validation.\n\nThese semantics must use the confirmed Agent, user, channel-binding, and\nconversation ownership model.\n\nThe canonical message store MUST preserve messages as immutable records and\nrepresent later edits, deletions, and branches through related records or\nevents.\n\nArchie requires exact branch-point correlation using stable MessageIDs.\n\n## Conversation branches\n\nA branch is a continuation of a previous conversation at an exact message. A\nchild conversation records:\n\n```text\nParentConversationID\nForkMessageID\n```\n\nMessages remain an append-only linear sequence within each conversation. The\nchild's effective initial context is the parent's history through\n`ForkMessageID`; messages after that point in the parent are not inherited.\n\nAfter the fork, parent and child are isolated:\n\n- child messages, tool calls, tool results, capability actions, and errors do\n not appear in the parent conversation;\n- later parent messages do not silently enter the child;\n- returning to the parent resumes its unchanged history;\n- the child may itself be branched using the same rules;\n- branch lineage remains queryable for navigation, audit, and metrics.\n\nThis supports building substantial context about a subject and then performing\none or more contextual subtasks without polluting the original conversation. For\nexample, a user may branch from a project discussion, create Jira issues in the\nchild using inherited context, and then return to the project discussion without\nadding the Jira tool transcript to the parent.\n\nBranching does not copy mutable message rows. History is resolved through the\nimmutable parent conversation and fork MessageID, followed by the child's own\nmessage sequence.\n\nBranch isolation applies to conversation history, not to Agent learning or\nmemory. A memory action performed in a branch has exactly the same semantics as\nthe same action performed in any other conversation. It writes to its selected\nglobal, agent-wide, user-wide, or Agent-user relationship scope.\n\nMemory provenance MAY retain the source ConversationID and MessageIDs for\nexplanation and audit. Retrieval and effect are determined by memory scope and\naccess rules, not by the conversation from which the memory originated.\nReturning to a parent does not insert the child's tool transcript into the\nparent history, but scoped memories learned in the child remain available to\nfuture retrieval wherever that scope is visible.\n\nChanging sessions or branching therefore controls immediate conversational\ncontext without preventing the Agent from learning durable knowledge.\n\n## Target ownership\n\nThe Messaging domain lives at `internal/domain/messaging`. It owns:\n\n- canonical Messages and typed message content;\n- Conversations, branches, and lineage;\n- channel bindings and external source correlation;\n- replies, acknowledgements, and outbound delivery attempts;\n- message and conversation lifecycle events;\n- domain-defined repository and delivery service contracts;\n- projections required for conversation navigation and metrics.\n\nWork intake is a separate cohesive domain at `internal/domain/workintake`. It\nowns admission, validation, deterministic routing, and accepted work requests\nfor the optional transition into durable workflow-backed execution.\n\nTelegram, email, webhook, forge, Jira, and future messaging systems are\ninfrastructure adapters implementing Messaging contracts. They are users of the\ndomain; they do not own Messages, Conversations, work admission, or\nWorkflowExecution semantics.\n\nThe Agent decides whether an ordinary interaction is answered directly, uses a\ncapability, changes or retrieves Agent-owned information, or proposes durable\nworkflow-backed work.\n\nExplicitly configured automation MAY enter work intake without a conversational\nAgent turn when the external trigger and admission rule already unambiguously\nrequest durable work. This avoids spending an LLM turn to rediscover a\ndeterministic routing decision.\n\nWhen an interaction requires durable workflow-backed execution:\n\n1. a channel adapter produces a channel-neutral inbound message;\n2. the acting agent handles or interprets the interaction;\n3. work intake validates, admits, and routes a proposed work request;\n4. work intake emits an accepted work request with its source correlation,\n acting identity, requested outcome, inputs, and routing evidence;\n5. the Agent System consumes that request, selects a specific Workflow version,\n and creates the WorkflowExecution.\n\nThe accepted work request is a cross-domain handoff, not an alternate execution\naggregate. The Agent System is the sole authority that creates a valid\nWorkflowExecution.\n\nAn interaction MAY instead be completed directly by an agent turn, delegated to\na tool or capability, recorded as a note or reminder, answered conversationally,\nor retained for later context. Those paths MUST NOT require a Workflow or\nWorkflowExecution.\n\nA Workflow MAY add reusable, versioned structure to any of those behaviours.\nScheduled personal-assistant routines, message triage, reminders, infrastructure\noperations, and other capability sequences are valid Workflow definitions. The\ndistinction is whether durable structured execution is useful, not whether the\nsubject originated from a forge.\n\n## Canonical execution language\n\n- **Workflow**: reusable, versioned behaviour definition.\n- **WorkflowExecution**: durable execution of a specific Workflow version.\n- **WorkflowStep**: a defined operation in a Workflow.\n- **StepExecution**: the recorded execution and outcome of a WorkflowStep.\n- **Attempt**: a retry within the same WorkflowExecution.\n\nExternal issues, messages, requests, goals, and notes are not synonyms for these\nconcepts.\n\n## Current implementation\n\nCurrent intake is fragmented:\n\n- Telegram, email, and webhook adapters translate payloads into\n `gateway.Message`, while an unused richer `MessageEvent` contract exists\n alongside it.\n- `gateway.Router` combines local channel commands, conversational LLM routing,\n and direct task lifecycle operations.\n- Telegram is the only channel with production LLM and durable conversation\n wiring. Email and webhook free text currently have no LLM responder.\n- Forge polling bypasses the gateway and directly enqueues store rows or NATS\n task messages.\n- Chat `/spawn` directly creates a `store.Task` and invents a synthetic forge\n issue number to satisfy the existing persistence key.\n- Forge replies and chat approval commands implement separate waiting-human\n paths.\n- Message delivery, acknowledgement, retry, deduplication, and source\n correlation are platform-specific side effects rather than one durable model.\n- `store.Task` collapses an external source reference and durable execution\n state.\n\n## Intake provenance\n\nA task's origin determines which enqueue path created it, though every origin\nconverges on the same `tasks` table and the same `Route`/`Run` lifecycle once\nqueued.\n\n- Forge-sourced work -- discovered by polling or, in single-identity\n deployments, delivered by the forge webhook receiver (`internal/forge/webhook`)\n -- enqueues through the same `workintake.TaskEnvelope` and `PublishTask` path.\n `TaskEnvelope.IdempotencyKey` is keyed on `owner/repo/number` and not on the\n delivery source, so a webhook-delivered issue and a later poll of the same\n issue dedup through `PublishUnique` instead of double-enqueuing.\n- Chat-spawned and playbook-binding-dispatched work enqueues through\n `EnqueueChatTask`/`EnqueueBindingTask` (`internal/store`), stamped with\n `binding_id`/`binding_version` for provenance.\n\nWebhook intake is opt-in per deployment (`Forge.Intake` = `poll` (default),\n`webhook`, or `both`) and single-identity only: a config naming\n`webhook`/`both` alongside `[[identities]]`, or setting intake on a\n`[identities.forge]` entry, is rejected at validation rather than started with\nthe setting ignored. One receiver binds one secret and one dispatch predicate,\nso it cannot judge a second identity's events by the right rules. A\nmulti-identity deployment polls per identity until per-identity intake lands.\nThe decided contract is `docs/prds/event-sources-and-reactions.md`; the binding\nmodel and its threat model are `docs/architecture/bindings.md`.\n\n## Current hazards\n\n- Treating every agent interaction as a work request would remove Archie's\n personal-assistant role and force conversation and immediate capability use\n through an inappropriate workflow lifecycle.\n- Treating every accepted work request as a WorkflowExecution would let intake\n or channel code select versions and construct invalid execution state outside\n the Agent System.\n- Source identity is incomplete: sessions use configured bot usernames, tasks\n use identity strings, and external message IDs are generally absent.\n- SQLite and NATS deduplication are based on forge coordinates rather than a\n channel-neutral source-message reference.\n- Replies and acknowledgements have no durable delivery-attempt lifecycle, so\n partial success and retry behaviour differ by adapter.\n- Production code has two competing channel-neutral message contracts.\n- The existing Archie session implementation lacks the required tool-call\n semantics and exact branch lineage and MUST be brought to the canonical\n contract.\n\n## Decisions still required\n\n- the cohesive owner and final location of messaging behaviour;\n- the cohesive owner and final location of work-intake admission and routing;\n- the canonical inbound message, conversation, and source-correlation model;\n- the boundary between an agent turn, direct capability use, and a proposed\n durable work request;\n- the exact relationship between an Agent, its acting IdentityID, its user, and\n its channel bindings;\n- outbound response, acknowledgement, delivery-attempt, retry, deduplication,\n and failure semantics;\n- the smallest accepted-work-request contract consumed by the Agent System;\n- migration of forge polling, replies, chat commands, email, webhook, Telegram,\n and Jira integrations.\n\n## Non-goal: semantic search over chat messages\n\nSemantic/vector search over chat messages is an **explicit non-goal** until usage\ndata shows a real need.\n\nMessage search is lexical only: session-scoped SQLite FTS5 queries paged via\n`MessageQuery{Query,Limit,Offset}` → `MessagePage`. Do not add embeddings\nconfiguration or vector columns to message records.\n\nThe rationale is cost without a demonstrated use case. Measured 2026-08-03 with\n`embeddinggemma:300m` — note this was never an infrastructure problem, the model\nwas already pulled and `ai-sdk/provider/ollama` already has `embed.go`:\n\n- 121 ms median warm per message on the write path, 12.5 s cold;\n- 768 dims = 3072 bytes per vector, roughly 15× a typical message payload;\n- chronological reads would carry vector data they do not use on the hot path\n of every turn;\n- similarity search requires an explicit indexing and retention design.\n\nIf this is revisited, vectors belong in a sibling table so chronological reads\nnever load them.\n" + "body": "# Messaging and Work Intake\n\n**Status:** Current state researched; foundation decisions in progress \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie is a personal assistant and general agentic system. Its interactions\ninclude conversation, questions, notes, reminders, goals, message handling,\ntroubleshooting, infrastructure operations, server management, source changes,\nand open-ended future capabilities.\n\nA message to an agent is therefore not inherently a work request, and invoking\nan agent is not inherently a WorkflowExecution.\n\nMessaging and work intake are separate from the Agent System's durable workflow\nexecution model:\n\n- messaging preserves interactions with people and external systems;\n- an agent turn interprets and responds to an interaction using its available\n capabilities;\n- work intake handles the optional promotion of an interaction into durable,\n workflow-backed work;\n- the Agent System owns Workflow definitions and creates WorkflowExecutions.\n\nForge issues, Jira issues, Telegram messages, emails, webhooks, and future\nchannels remain peer external message or artifact sources. None becomes an\nexecution model merely because it can lead to work.\n\n## Confirmed boundary\n\nWork intake MUST NOT create a WorkflowExecution directly.\n\nInbound interactions normally route to the Agent selected by the channel\nbinding. The selected Agent retains its own memory, user knowledge, conversation\ncontinuity, specialisation, and capability context. This continuity is\nindependent of the underlying model or provider.\n\nEach selected Agent acts through its own immutable IdentityID. Interaction and\nresulting action records preserve both that acting identity and the initiating\nuser identity when one exists.\n\nAgent memory is private by default. Interaction context MAY additionally use\nuser-wide memory and intentionally global shared memory, but another Agent's\nagent-wide memory is not part of the selected Agent's context.\n\nMultiple users may interact with the same Agent. The channel binding and inbound\nmessage MUST resolve both AgentID and initiating User IdentityID before\nconversation or memory context is loaded. Sharing an Agent MUST NOT implicitly\nshare one user's private memory with another user.\n\nAn interaction may assemble memory from four scopes: global shared, the selected\nAgent's agent-wide memory, the initiating user's authorized user-wide memory,\nand the private Agent-user relationship memory addressed by both IDs.\n\n## Conversation identity\n\nEvery conversation is identified by the complete tuple:\n\n```text\nAgentID\n+ UserIdentityID\n+ ChannelBindingID\n+ ExternalConversationID\n+ ThreadID\n```\n\n`ThreadID` has an explicit empty value for channels without threads. No element\nof this key may be replaced by a display name, configured bot username, channel\ntype, or other mutable label.\n\nAgent and user memory continue across conversations according to their memory\nscope. Immediate conversation history and delivery correlation remain isolated\nto the identified channel conversation or thread. Moving between Telegram,\nemail, or another channel does not create a new Agent or user, but it does\ncreate a distinct conversation unless an explicit linking operation relates\nthem.\n\nConversation and message events support projections for channel-entry metrics,\nuse cases, volumes, outcomes, latency, capability usage, and other product or\noperational analysis. Metrics are derived read models; they MUST NOT become\nmutable counters embedded in the conversation aggregate or change message\nrouting semantics.\n\n## Canonical messages\n\nEvery inbound, assistant, tool-call, and tool-result message has an immutable\napplication-generated MessageID. Platform identifiers are external correlation,\nnot substitutes for the canonical ID.\n\nMessages are addressable because an interaction is more than text. A message may\ncontain assistant content, reasoning data subject to its retention policy, one\nor more tool calls, a tool result with authoritative execution metadata,\nattachments, reply relationships, edits, or other typed content. Stable\nMessageIDs also allow sessions to branch from an exact prior point.\n\nThe canonical session and message implementation must supply:\n\n- application-generated stable message IDs;\n- user, assistant, and tool roles;\n- assistant tool calls and correlated tool-result messages;\n- authoritative tool-result status, timing, error, truncation, and size\n metadata;\n- session lifecycle and active-request state;\n- session-to-agent-instance and parent-session lineage;\n- durable database round trips and timestamp validation.\n\nThese semantics must use the confirmed Agent, user, channel-binding, and\nconversation ownership model.\n\nThe canonical message store MUST preserve messages as immutable records and\nrepresent later edits, deletions, and branches through related records or\nevents.\n\nArchie requires exact branch-point correlation using stable MessageIDs.\n\n## Conversation branches\n\nA branch is a continuation of a previous conversation at an exact message. A\nchild conversation records:\n\n```text\nParentConversationID\nForkMessageID\n```\n\nMessages remain an append-only linear sequence within each conversation. The\nchild's effective initial context is the parent's history through\n`ForkMessageID`; messages after that point in the parent are not inherited.\n\nAfter the fork, parent and child are isolated:\n\n- child messages, tool calls, tool results, capability actions, and errors do\n not appear in the parent conversation;\n- later parent messages do not silently enter the child;\n- returning to the parent resumes its unchanged history;\n- the child may itself be branched using the same rules;\n- branch lineage remains queryable for navigation, audit, and metrics.\n\nThis supports building substantial context about a subject and then performing\none or more contextual subtasks without polluting the original conversation. For\nexample, a user may branch from a project discussion, create Jira issues in the\nchild using inherited context, and then return to the project discussion without\nadding the Jira tool transcript to the parent.\n\nBranching does not copy mutable message rows. History is resolved through the\nimmutable parent conversation and fork MessageID, followed by the child's own\nmessage sequence.\n\nBranch isolation applies to conversation history, not to Agent learning or\nmemory. A memory action performed in a branch has exactly the same semantics as\nthe same action performed in any other conversation. It writes to its selected\nglobal, agent-wide, user-wide, or Agent-user relationship scope.\n\nMemory provenance MAY retain the source ConversationID and MessageIDs for\nexplanation and audit. Retrieval and effect are determined by memory scope and\naccess rules, not by the conversation from which the memory originated.\nReturning to a parent does not insert the child's tool transcript into the\nparent history, but scoped memories learned in the child remain available to\nfuture retrieval wherever that scope is visible.\n\nChanging sessions or branching therefore controls immediate conversational\ncontext without preventing the Agent from learning durable knowledge.\n\n## Target ownership\n\nThe Messaging domain lives at `internal/domain/messaging`. It owns:\n\n- canonical Messages and typed message content;\n- Conversations, branches, and lineage;\n- channel bindings and external source correlation;\n- replies, acknowledgements, and outbound delivery attempts;\n- message and conversation lifecycle events;\n- domain-defined repository and delivery service contracts;\n- projections required for conversation navigation and metrics.\n\nWork intake is a separate cohesive domain at `internal/domain/workintake`. It\nowns admission, validation, deterministic routing, and accepted work requests\nfor the optional transition into durable workflow-backed execution.\n\nTelegram, email, webhook, forge, Jira, and future messaging systems are\ninfrastructure adapters implementing Messaging contracts. They are users of the\ndomain; they do not own Messages, Conversations, work admission, or\nWorkflowExecution semantics.\n\nThe Agent decides whether an ordinary interaction is answered directly, uses a\ncapability, changes or retrieves Agent-owned information, or proposes durable\nworkflow-backed work.\n\nExplicitly configured automation MAY enter work intake without a conversational\nAgent turn when the external trigger and admission rule already unambiguously\nrequest durable work. This avoids spending an LLM turn to rediscover a\ndeterministic routing decision.\n\nWhen an interaction requires durable workflow-backed execution:\n\n1. a channel adapter produces a channel-neutral inbound message;\n2. the acting agent handles or interprets the interaction;\n3. work intake validates, admits, and routes a proposed work request;\n4. work intake emits an accepted work request with its source correlation,\n acting identity, requested outcome, inputs, and routing evidence;\n5. the Agent System consumes that request, selects a specific Workflow version,\n and creates the WorkflowExecution.\n\nThe accepted work request is a cross-domain handoff, not an alternate execution\naggregate. The Agent System is the sole authority that creates a valid\nWorkflowExecution.\n\nAn interaction MAY instead be completed directly by an agent turn, delegated to\na tool or capability, recorded as a note or reminder, answered conversationally,\nor retained for later context. Those paths MUST NOT require a Workflow or\nWorkflowExecution.\n\nA Workflow MAY add reusable, versioned structure to any of those behaviours.\nScheduled personal-assistant routines, message triage, reminders, infrastructure\noperations, and other capability sequences are valid Workflow definitions. The\ndistinction is whether durable structured execution is useful, not whether the\nsubject originated from a forge.\n\n## Canonical execution language\n\n- **Workflow**: reusable, versioned behaviour definition.\n- **WorkflowExecution**: durable execution of a specific Workflow version.\n- **WorkflowStep**: a defined operation in a Workflow.\n- **StepExecution**: the recorded execution and outcome of a WorkflowStep.\n- **Attempt**: a retry within the same WorkflowExecution.\n\nExternal issues, messages, requests, goals, and notes are not synonyms for these\nconcepts.\n\n## Current implementation\n\nCurrent intake is fragmented:\n\n- Telegram, email, and webhook adapters translate payloads into\n `gateway.Message`, while an unused richer `MessageEvent` contract exists\n alongside it.\n- `gateway.Router` combines local channel commands, conversational LLM routing,\n and direct task lifecycle operations.\n- Telegram is the only channel with production LLM and durable conversation\n wiring. Email and webhook free text currently have no LLM responder.\n- Forge polling bypasses the gateway and directly enqueues store rows or NATS\n task messages.\n- Chat `/spawn` directly creates a `store.Task` and invents a synthetic forge\n issue number to satisfy the existing persistence key.\n- Forge replies and chat approval commands implement separate waiting-human\n paths.\n- Message delivery, acknowledgement, retry, deduplication, and source\n correlation are platform-specific side effects rather than one durable model.\n- `store.Task` collapses an external source reference and durable execution\n state.\n\n## Intake provenance\n\nA task's origin determines which enqueue path created it, though every origin\nconverges on the same `tasks` table and the same `Route`/`Run` lifecycle once\nqueued.\n\n- Forge-sourced work -- discovered by polling or, in single-identity\n deployments, delivered by the forge webhook receiver (`internal/forge/webhook`)\n -- enqueues through the same `workintake.TaskEnvelope` and `PublishTask` path.\n `TaskEnvelope.IdempotencyKey` is keyed on `owner/repo/number` and not on the\n delivery source, so a webhook-delivered issue and a later poll of the same\n issue dedup through `PublishUnique` instead of double-enqueuing.\n- Chat-spawned and playbook-binding-dispatched work enqueues through\n `EnqueueChatTask`/`EnqueueBindingTask` (`internal/infrastructure/postgres`), stamped with\n `binding_id`/`binding_version` for provenance.\n\nWebhook intake is opt-in per deployment (`Forge.Intake` = `poll` (default),\n`webhook`, or `both`) and single-identity only: a config naming\n`webhook`/`both` alongside `[[identities]]`, or setting intake on a\n`[identities.forge]` entry, is rejected at validation rather than started with\nthe setting ignored. One receiver binds one secret and one dispatch predicate,\nso it cannot judge a second identity's events by the right rules. A\nmulti-identity deployment polls per identity until per-identity intake lands.\nThe decided contract is `docs/prds/event-sources-and-reactions.md`; the binding\nmodel and its threat model are `docs/architecture/bindings.md`.\n\n## Current hazards\n\n- Treating every agent interaction as a work request would remove Archie's\n personal-assistant role and force conversation and immediate capability use\n through an inappropriate workflow lifecycle.\n- Treating every accepted work request as a WorkflowExecution would let intake\n or channel code select versions and construct invalid execution state outside\n the Agent System.\n- Source identity is incomplete: sessions use configured bot usernames, tasks\n use identity strings, and external message IDs are generally absent.\n- Database and NATS deduplication are based on forge coordinates rather than a\n channel-neutral source-message reference.\n- Replies and acknowledgements have no durable delivery-attempt lifecycle, so\n partial success and retry behaviour differ by adapter.\n- Production code has two competing channel-neutral message contracts.\n- The existing Archie session implementation lacks the required tool-call\n semantics and exact branch lineage and MUST be brought to the canonical\n contract.\n\n## Decisions still required\n\n- the cohesive owner and final location of messaging behaviour;\n- the cohesive owner and final location of work-intake admission and routing;\n- the canonical inbound message, conversation, and source-correlation model;\n- the boundary between an agent turn, direct capability use, and a proposed\n durable work request;\n- the exact relationship between an Agent, its acting IdentityID, its user, and\n its channel bindings;\n- outbound response, acknowledgement, delivery-attempt, retry, deduplication,\n and failure semantics;\n- the smallest accepted-work-request contract consumed by the Agent System;\n- migration of forge polling, replies, chat commands, email, webhook, Telegram,\n and Jira integrations.\n\n## Non-goal: semantic search over chat messages\n\nSemantic/vector search over chat messages is an **explicit non-goal** until usage\ndata shows a real need.\n\nMessage search is lexical only: session-scoped PostgreSQL full-text queries paged via\n`MessageQuery{Query,Limit,Offset}` → `MessagePage`. Do not add embeddings\nconfiguration or vector columns to message records.\n\nThe rationale is cost without a demonstrated use case. Measured 2026-08-03 with\n`embeddinggemma:300m` — note this was never an infrastructure problem, the model\nwas already pulled and `ai-sdk/provider/ollama` already has `embed.go`:\n\n- 121 ms median warm per message on the write path, 12.5 s cold;\n- 768 dims = 3072 bytes per vector, roughly 15× a typical message payload;\n- chronological reads would carry vector data they do not use on the hot path\n of every turn;\n- similarity search requires an explicit indexing and retention design.\n\nIf this is revisited, vectors belong in a sibling table so chronological reads\nnever load them.\n" }, { "url": "/dev/architecture/migration-decisions/", "title": "Architecture Migration Decisions", "section": "Architecture", - "body": "# Architecture Migration Decisions\n\n**Status:** Active migration inventory \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nThis document records what remains to be decided before Archie's current\nimplementation can be migrated into the approved domain-oriented structure.\n\nThe product and core domain model are not being redesigned. Existing Archie\nbehaviour and the existing session implementation are the migration\nbaseline. Remaining decisions MUST be grounded in current packages, consumers,\ntests, persistence, and runtime wiring.\n\n## Confirmed domain packages\n\n```text\ninternal/domain/\n identity/ (approved, not yet materialised as a package)\n agent/ (approved, not yet materialised as a package)\n workflow/\n messaging/ (canonical Message/Conversation types only; gateway migration pending, see section 2)\n workintake/\n plugin/ (approved, not yet materialised as a package)\n curator/\n mapping/\n scheduling/\n```\n\n- `identity` owns persistent actors, lifecycle, ownership, attribution, and\n audit identity.\n- `agent` owns persistent assistants, specialisation, instructions,\n model-independent continuity, capability context, and associations with\n identity and memory.\n- `workflow` is a separate domain. It owns reusable, versioned Workflow\n definitions, WorkflowSteps, WorkflowExecutions, StepExecutions, Attempts,\n outcomes, and typed Workflow plugin contracts.\n- `messaging` owns canonical Messages, Conversations, branches, channel and\n source correlation, replies, acknowledgements, delivery attempts, events, and\n projections.\n- `workintake` owns admission, validation, deterministic routing, and accepted\n work requests for the optional transition into durable workflow-backed\n execution.\n- `plugin` owns generic plugin identity, discovery, compatibility, and lifecycle\n mechanics. It remains metadata-only; capability semantics and typed\n registration belong to the owning domain.\n- `curator` owns the curator engine family: long-running agent loops that\n reason over memory and perform maintenance tasks on a declared shape\n (check-in interval, tool set, attached memory engine). See epic\n archie-core-yp9.\n- `mapping` owns the payload field-mapping vocabulary: binding named fields to\n JSON paths inside a captured webhook payload, and detecting drift. See\n `docs/prds/payload-field-mapping.md`.\n- `scheduling` owns the ticker engine: deciding when a scheduled job runs and\n with what concurrency. It declares `JobSource`/`Runner` contracts and owns\n no storage or delivery. See epic archie-core-1786637496320-249-be6d9a20.\n\n**Reconciliation (2026-08-31):** `curator`, `mapping`, and `scheduling` existed\nin the tree without being added to this list when their epics landed. They are\nconfirmed as legitimate, narrowly-scoped domains under the same rules as the\noriginal six — each owns one capability family, declares typed contracts, and\nhas no cross-domain reach. This list was out of date, not the tree.\n\nConversely, `identity`, `agent`, `messaging`, and `plugin` are approved target\ndomains that do not yet exist as packages. Their behaviour currently lives in\npre-migration packages (see \"Complete package destination map\" below for the\nfull inventory). Do **not** create empty stub packages for them ahead of an\nactual migration PR — an unused stub is dead code and gives no test coverage.\nEach materialises when its corresponding migration decision below is resolved\nand implemented in the same change.\n\nMemory's final package placement remains undecided pending focused review of the\nexisting implementation.\n\nThe following are approved shared or composition packages, not domains:\n\n```text\ninternal/eventbus/\ninternal/policy/\ninternal/infrastructure/\ninternal/app/archied/\ninternal/app/agentworker/\n```\n\n## Remaining migration decisions\n\n### 1. Complete package destination map\n\nEvery current package MUST receive one explicit disposition:\n\n- move into a confirmed domain;\n- move under infrastructure as an implementation of a domain-owned contract;\n- move into application composition;\n- merge with an identified duplicate;\n- remain as an approved shared application contract;\n- move to a plugin, extra, deployment, example, or generated artifact;\n- be deleted after its replacement is proven.\n\nThe largest unreviewed areas include memory, tools and capabilities, workspace\nand worktree behaviour, gates, storage, events, web UI, containers, RPC\npackages, release management, and scheduling.\n\nThe destination matrix MUST name the behaviour owner, target path, dependencies\nto remove, state and contracts to preserve, migration prerequisites, and\ndeletion criteria.\n\n**`internal/webui` / UI Service — RATIFIED Phase 3 disposition (2026-09-07).**\nThe current owner is the in-process HTTP/SSE dashboard: SPA hosting, operator\nauthentication, UI response projection, configuration reads and mutations,\ntask/control APIs, and direct adapters to State Store, channel, event/log,\nmemory, curator, workflow, forge, and Gateway runtime values. The target owner\nis a standalone UI application process; `internal/webui` remains its HTTP\ntransport adapter, while Gateway, State Store, configuration, Work Intake,\nMessaging, and other capability owners expose the contracts behind those\nroutes. The migration preserves retained route schemas, auth/CSRF behavior,\ntask/capture/mapping/binding/session continuity, SSE replay semantics, and\ncommitted generated SPA assets. Prerequisites are the ratified UI boundary,\nroute-owner inventory, narrow remote adapters, configuration/auth/readiness\ncontracts, and process/conformance tests. Remove the shared live\n`config.Holder`, direct event/log bus, daemon stopper, forge client, channel\nmanager, capability registries, composite store access, and in-process UI\nlistener only after the focused criteria in\n[`docs/prds/ui-service-boundary.md`](/dev/prds/ui-service-boundary/) pass in\nproduction composition. The old construction and listener are deleted in the\nsame cutover change; no second UI authority remains.\n\n### 2. Session and Messaging migration\n\nThe existing session implementation is the migration baseline. The migration must\ndecide:\n\n- which session and message components are retained or adapted;\n- how Agent, user, channel, Conversation, and branch ownership are added;\n- how Archie's parallel gateway session implementation is replaced;\n- how canonical Messages become immutable persisted records;\n- how existing sessions and messages are migrated;\n- how outbound delivery, retry, acknowledgement, deduplication, and failure\n semantics are represented;\n- how current Telegram, email, webhook, forge, and Jira integrations migrate to\n Messaging infrastructure adapters.\n\nThe following are already decided and MUST NOT be reopened without contradictory\nimplementation evidence:\n\n- the composite Conversation identity;\n- immutable MessageIDs;\n- typed tool-call and tool-result messages;\n- exact branch lineage through ParentConversationID and ForkMessageID;\n- parent and child transcript isolation;\n- normal scoped-memory behaviour regardless of the originating Conversation;\n- Messaging and Work Intake ownership.\n\n**Decision (2026-08-31), resolving the open questions above:**\n\n`internal/gateway/` is the \"parallel gateway session implementation\" this\nsection names. It is retained as-is for now — a big-bang rewrite of live\nsession/compression/approval/branch logic across Telegram, web UI, email, and\nwebhook channels in one change is exactly the \"convoluted to satisfy edge\ncases\" failure mode CLAUDE.md's Revert-on-Flail rule warns about, and would\nrisk silent behavioural regressions in the one path every user-facing message\ntravels through.\n\nInstead, `internal/domain/messaging/` is materialised now with the pieces\nthat are already decided and don't require touching gateway: the canonical\n`Message` and `Conversation` types (composite Conversation identity, immutable\n`MessageID`, typed tool-call/tool-result message variants, branch lineage via\n`ParentConversationID`/`ForkMessageID`). These are pure domain types with no\ngateway dependency — a new package, not a rewrite of an existing one.\n\nMigrating `internal/gateway/`'s session, compression, approval, and branch\nlogic onto these types is real, cross-channel-risk work and is **not** done in\nthis pass. It is tracked as separate, appropriately-scoped bd issues (see\n`bd list --label messaging-migration`), sequenced channel-by-channel so each\nlands with its own behavioural verification instead of one large diff no one\ncan safely review.\n\n**Amended 2026-09-15.** `internal/infrastructure/gatewayrpc.StoreClient` is the\nadapter this migration will need: it implements `messaging.SessionStore` over\nthe chat service's session-store RPCs and is proven by\n`TestStoreClientPreservesCanonicalRecords`. It is deliberately unwired.\nProduction `archied` dials the standalone gateway for `ChatContract` only\n(`composeChatContract` → `gatewayrpc.Dial` returns a `*Client`, not a\n`*StoreClient`), and the daemon's session store is in-process SQLite via\n`makeTelegramSessionStore`. The test comment that says it mirrors production\ndescribes the target, not today's tree. Kept rather than deleted because the\nmigration it serves is deferred by this very section, and re-deriving a tested\nwire adapter once that lands is pure cost.\n\n**Phase 4 Messaging Service extraction (2026-09-19, `archie-core-8cda.6.1`).**\nThe process boundary for external channels (Telegram, email, webhook) is ratified\nin [`docs/prds/messaging-service-boundary.md`](/dev/prds/messaging-service-boundary/).\nMessaging is extracted into `cmd/archie-messaging` / `internal/app/archiemessaging`,\ncalling Gateway's `messaging.ChatContract` exclusively over gRPC. Channel frontends\nhold no direct database handles, daemon state, or model runtimes.\n\n**Telegram operator surface after extraction (2026-09-19, `archie-core-8cda.6.6`).**\n`setupTelegramGateway` wired eight `telegram.Gateway` seams the daemon could\nsatisfy from its own process. Each is resolved by where the fact it needs is\nactually owned, not by where it used to be built:\n\n| Seam | Resolution |\n|---|---|\n| `Version` | Sourced from the Gateway via `ChatSnapshot.Version`. The Messaging Service never stamps or infers a build version; `cmd/archie-messaging` deliberately takes no version ldflags, so it cannot report a partial upgrade as a matched one. |\n| `Updates` (`/update`) | Built in the Messaging Service from `[chat.telegram].update_check_command` / `update_install_command` — its own configuration. `Enrich` is dropped: `componentInstallTypeEnricher` reads `[nats]`, which this service must not decode, so the check command's own reported install type stands unmodified. |\n| `UpdateReportPath` | A local state file under `work_dir`, keyed by the same `sha256(bot_user)[:8]` scheme the daemon used, so a pending report is found rather than written afresh. |\n| `ReleaseAnnouncements` | **Deliberately left nil**, for the same reason as `RunningVersions`. `releaseannounce.Announcer` announces nothing unless each `Component` carries a parseable version, and the only versions this service can obtain are the Gateway's own build stamps, not archied's. Wiring the announcer without them yields a no-op that reads as configured, so it stays unwired until component self-reporting exists. |\n| `SetShowToolCalls` | Projected from `[chat].show_tool_calls`. |\n| `Reload` | Local: re-resolves the token and allowlist from this service's own config file and overlay. Reload was never a daemon fact. |\n| `RunningVersions` | **Deliberately left nil.** It exists to turn an installer's claim into a checked one, and only a component's own compiled-in build can vouch for it. The Messaging Service knows neither archied's build nor the observed archie-agent version (`daemon.AgentStatus`), and the Gateway's own `gatewayVersion` is a different binary's stamp — reporting it as the daemon's would manufacture exactly the false success the check exists to catch. Update reports therefore relay as unverified claims until a component self-report contract exists. |\n| `Dangerous` | Stays nil. No daemon composition ever set it; `/rollback` and `/stop` reported \"not configured\" before the extraction and still do. |\n\nThe Messaging Service consequently reads `work_dir`, `bot_user`, `[chat]` and\n`[health]` in addition to the PRD's `[chat.*]` / `[services.gateway]` list.\n`[health]` is the daemon's health URL, which `releaseupdate.CommandInstaller`\npolls from outside to confirm an update came back up; the rest of the PRD's\nprohibition list is untouched.\n\n**Phase 4 messaging severance: what must move, and the decision the PRD does\nnot cover (2026-09-22, `archie-core-1ng1`).** The PRD's deletion gate cannot\npass today. `cmd/archie-messaging/architecture_test.go` does not exist, and the\ncheck that section specifies -- `go list -deps ./cmd/archie-messaging` links\nzero banned runtime packages -- reports four:\n\n| banned package | PRD category |\n| --- | --- |\n| `modernc.org/sqlite` | State Store |\n| `internal/store` | State Store |\n| `internal/domain/workflow` | Workflow engine |\n| `internal/agentexec` | Workflow engine, via `internal/domain/workflow/agent.go` |\n\nAll four arrive through one import: `internal/app/archiemessaging/run.go`\nimports `internal/app/controlplane`, whose own dependency set is exactly those\nfour. The Messaging Service uses three symbols from that package\n(`NewRPCClient`, `RuntimeChatConfig`, `ChannelSettingsKind`); the package also\ncarries the store-backed `ResourceStore` server, the `Definition` registry and\nthe workflow step vocabulary. Sever the package. Amending the gate to tolerate\n`modernc.org/sqlite` and `internal/store` is rejected: a boundary you whitelist\naround is no longer a boundary.\n\n**What moves.**\n\n- Every kind constant. They are strings with no dependency, and the client names\n them.\n- The channel-settings vocabulary and its decode: `channelSettings`,\n `telegramSettings`, `webhookChannelSettings`, `emailSettings`,\n `rateLimitSettings`, `channelDuration`, and `decodeRenamingLegacyKeys`.\n- The generic client: `Client`, `NewRPCClient`, `RuntimeConfig` with\n `runtimeConfigFrom`, `runtimeChatConfigFrom`, the other `runtime*From`\n projections, and the `resourceReader` interface.\n- The persona watch (`WatchPersonas`, `AppliedPersonas`), which needs only\n `internal/domain/agent`.\n- `executionSettingsDocument`, the document alone.\n\n**What stays, and why.**\n\n- The store-backed server: `Server`, `NewServer`, `ResourceStore`,\n `ImportConfig`, `SeedSkip`, the `Definition` registry, and every other kind's\n seed, validator and `Normalize` hook.\n- `WorkflowDefinitionsClient` and `stepRegistry`, which decode through\n `internal/domain/workflow`.\n- `WatchWorkflowExecutionSettings` and `decodeSettings`: `decodeSettings`\n returns and validates `workflow.ExecutionSettings`, so any package holding it\n links the workflow domain and therefore `internal/agentexec`. This is why the\n client cannot move as one package -- see open question 2.\n- The schedules document, whose type is `scheduling.JobSpec` in\n `internal/domain/scheduling` -- the vocabulary moved there when the legacy\n file-backed cronstore was deleted, which is also what satisfies the rule\n that a domain vocabulary package may not import infrastructure.\n- The schema derivation, which describes the documents the server advertises.\n\n**Placement as landed (2026-09-22, `archie-core-1ng1`).** One new package,\n`internal/infrastructure/controlplanerpc`, holding both the vocabulary (kinds, the\nchannel-settings document and its decode) and the client\n(`NewRPCClient`, `Query`, `RuntimeChatConfig`, the exported `ResourceReader`, the\nwire sentinels and `ClientError`). A second `internal/domain/controlplane` package\nfor the vocabulary was planned and then measured out of the design: the vocabulary\nhas exactly one consumer, the client that projects it, and the store-backed side\nreads the same definitions through type aliases rather than a copy -- so a separate\ndomain package would have been a package with one consumer and a second name for\none thing. What made that safe is a measurement, not a preference:\n`internal/app/controlplane` was the ONLY direct import of the Messaging Service\nthat dragged a banned runtime (`gatewayrpc`, `staterpc`, `applystatus`, `messaging`,\n`configuration`, `readiness` and `health` were all clean), so severing that one\ndependency is sufficient, and the severance needed no third package. Promoting the\nvocabulary to a domain package stays open for the day a second consumer appears.\n\n**Settled by the implementation.** Two of the four open questions this decision\noriginally left are answered, in both cases by measuring rather than by preference:\n\n- The client did NOT become two packages. `WatchWorkflowExecutionSettings` and\n `WatchPersonas` stayed on `internal/app/controlplane`'s `Client`, which now\n *embeds* the new one, so the daemon's validated watch is still defined once and\n the read path still has one implementation -- which is what the second package\n was for -- with one fewer package to keep in step.\n- The schedules and workflow documents stayed put, and\n `internal/domain/workflow/task` is linked deliberately: it carries the Task\n status and view vocabulary, its own dependency set holds none of the banned\n runtimes, and `cmd/archie-messaging/architecture_test.go` carries it as an\n exception checked in both directions (the same entry, for the same reason, as\n `cmd/archie-ui/architecture_test.go`).\n\n**Still open.**\n\n1. Does the server also leave `internal/app/`? `organisation.md` reserves\n `internal/app/` for wiring and composition, and a store-backed gRPC server is\n infrastructure -- but the gate does not require the move, and it would multiply\n the diff across archied's wiring and the State Store binary. Tracked as\n `archie-core-8f4n`.\n2. Does `internal/domain/workflow`'s contract/engine split happen, and when? A\n contract type (`WorkflowDefinitionCollection`) drags `internal/agentexec` only\n because `agent.go` sits beside it. Independent of this severance, tracked as\n `archie-core-inh6`, and the shape `archie-core-8cda.7` will meet again.\n3. Does the vocabulary move to a domain package when a second consumer appears?\n\n**Both consumers keep compiling.** archied imports the new vocabulary for the\nkinds it names and the new client package for `RuntimeConfig` and the persona\nwatch, and keeps `internal/app/controlplane` for the server, the registry,\n`ErrVersionConflict` and the workflow-aware pieces. The State Store binary\ncontinues to call `NewServer` and `ResourceStore` unchanged. The Messaging\nService then imports only `internal/infrastructure/controlplanerpc`,\n`gatewayrpc` and `staterpc`, so `go list -deps ./cmd/archie-messaging` loses all\nfour banned packages and the gate can land green.\n\n**Channel state to the dashboard: one administrative surface, following the\nconfig-snapshot precedent (2026-09-22, `archie-core-8cda.6.8`).** The Messaging\nService now records its channels' lifecycle -- `internal/channels/status.Manager`,\none entry per composed channel, written by each channel's own report through\n`channels.Lifecycle` -- and nothing reads that across a process boundary.\n`/api/channels` answers with nothing because `webui.Server.Channels` has no\nproduction source, and POST `/api/channels/{id}/reload` answers 501 for the same\nreason. The carrier is the State Store, and the shape is the one\n`archie-core-ymut` already established for config snapshots\n(`docs/prds/state-store-contract.md`, \"the one surface whose writer is the daemon\nand whose reader is the UI process\"): a pair of administrative RPCs, deny-by-default\nunder `authorizesTaskScopedCall`, so a task-scoped grant reaches neither.\n\nWhat that decides, so the implementation need not invent it:\n\n- A channel's state is **runtime** state, not a settings document. It does not\n belong in the `ControlPlaneService` resource kinds, which are validated,\n versioned documents seeded from config; a channel that is failed is not a\n document anybody edits.\n- The reader is the UI process, which already dials the State Store and pumps its\n cursor, so the surface needs no new transport in either direction.\n- Adding the pair changes the contract's RPC count, so\n `docs/prds/state-store-contract.md`'s grouped table gains a group in the same\n change that adds the RPCs -- the table is the contract's own index, and a\n surface it does not list is a surface nobody finds.\n\n**Reload gets its own surface, with its own lifetime (decided 2026-09-22).** Reload\nruns the other way -- the UI asks the service to reload a channel -- and the two\ncandidates were a request surface the service observes and clears, or a field on\nthe status surface the service watches. It is the first. The reasoning, recorded\nbecause it would look arbitrary later:\n\n- A request queue and a status report do not share a lifetime, so they should not\n share a surface. A pending request is transient and has an outcome; a channel's\n state is durable and has a current value.\n- A \"please reload\" field on the status resource makes READING status capable of\n RETRIGGERING an action, and makes a transient request indistinguishable from\n state the moment anything polls it.\n- It would also invert the writer/reader relationship the precedent establishes:\n the service would have to clear a field on an object whose whole contract is\n that the service reports it.\n\nSo the status surface stays writer-is-service, and the request carrier is its own\nobject with its own states.\n\n`archie-core-8cda.6.9`'s component self-report is the **same shape**: a process\nreporting something about itself for the UI to read. Whether it shares this\nsurface or gets its own is a decision for `6.9`, and the one outcome to avoid is\ntwo homes for \"what a component reports about itself\".\n\n### 3. Identity data migration\n\nThe migration must define:\n\n- the durable identity schema and repository implementation;\n- bootstrap of the permanent system identity;\n- stable IdentityIDs for existing configured Agents;\n- migration from configured identity names, empty-string ownership, and bot\n usernames;\n- preservation of historical attribution;\n- identity-aware uniqueness and transport deduplication;\n- correction of container and RPC capability resolution so execution uses the\n owning identity's bindings;\n- removal of identity-string equality from access-control behaviour.\n\nExact schema and method signatures are implementation decisions. The approved\nIdentity lifecycle and attribution semantics are fixed migration constraints.\n\n### 4. Workflow migration\n\nThe current implementation must be transformed deliberately:\n\n```text\nstore.Task -\u003e WorkflowExecution\nmutable Stage -\u003e StepExecution history\nRetryCount / Attempt -\u003e Attempt records\nworkflow.Registry -\u003e versioned Workflow definitions\ncurrent stages -\u003e WorkflowStep definitions or Workflow plugins\n```\n\nThe migration must decide:\n\n- Workflow definition and version persistence;\n- the smallest accepted-work-request contract consumed from Work Intake;\n- the enforced WorkflowExecution state machine;\n- atomic state and domain-event persistence;\n- migration of existing task rows and transition history;\n- disposition of current skill, Yaegi, and stage-plugin implementations;\n- the versioned worker-handoff contract;\n- scheduler, claim, retry, crash-recovery, and waiting-human cutover.\n\nWorkflow remains a separate domain from Agent. A user or Agent may define or\ninvoke a Workflow, but Workflow semantics and plugins remain Workflow-owned.\n\n### 5. Memory placement and storage\n\nThe four required scopes are fixed:\n\n- global shared;\n- Agent-wide;\n- user-wide;\n- Agent-user relationship.\n\nA focused review of `internal/memory`, its providers, consumers, persistence,\nand tests had these decisions:\n\n- `internal/domain/memory` versus ownership within the Agent domain — settled\n (2026-09-02, below);\n- the authoritative memory record and revision model — settled by the PRD\n (2026-09-16, below);\n- provider and infrastructure boundaries — settled by the PRD (2026-09-16,\n below);\n- retrieval and access enforcement — settled by the PRD (2026-09-16, below);\n ranked retrieval remains open (below);\n- migration of existing memory data — open: existing data is not migrated;\n- provenance representation — settled by the PRD (2026-09-16, below).\n\nConversation branches do not create another memory scope. A memory action has\nthe same scoped effect regardless of which Conversation originated it.\n\n**`internal/domain/memory` ownership — DECIDED (2026-09-02).** The typed\nengine family lives at `internal/domain/memory`\n(archie-core-1786637490069-11-e245c170), mirroring `internal/domain/curator`'s\ncontract/registrar/registry split. Not owned within the Agent domain: memory\nis consumed by more than agent turns (curators, per archie-core-1786637499636)\nand the plugin engine rule requires its own family, not a capability bolted\nonto another domain's contract.\n\n**Authoritative record and revision model for the builtin engine — DECIDED\n(2026-09-02),** scoped to unblock archie-core-1786637499480 (ship the\nfile-writing store as the default engine). This decides only the builtin\nengine's mapping onto the new `Record{ID}`/`Forget(id)` contract, not the\nmemory model in general or a future non-file engine's model:\n\n- `internal/memory/builtin.Store` is unchanged. It has no concept of a\n stable per-block ID — blocks are addressed by content substring — and nothing\n in this decision requires changing that; existing `internal/memory` tests\n keep passing unmodified, per that issue's acceptance criteria.\n- A new infrastructure adapter (`internal/infrastructure/memory`, per\n `organisation.md`'s \"infrastructure implements domain contracts\" rule — not\n under `internal/domain/memory`, which stays contract-only) wraps one\n `builtin.Store` per identity.\n- **Identity → directory:** `hex(sha256(identity))`, not the identity string\n itself. Closes the path-traversal question without an allowlist regex, and\n sidesteps collisions from identities that differ only in characters a\n filesystem treats as equivalent.\n- **Record ID:** assigned once at `Write`, independent of content, so a later\n edit elsewhere in the file can never orphan a `Forget`. Realized as a\n leading marker line (`\u003c!--mem:\u003culid\u003e--\u003e`) on the block `builtin.Store.Add`\n writes; `Query`/`List` strip it back out of `Record.Content` before\n returning, and `Forget(id)` locates the block by that unique marker via the\n store's existing substring-based `Remove`. The marker is a pragmatic,\n reversible choice — plain-text records stay human-readable and\n human-editable, at the cost of one marker line an operator editing the file\n by hand should leave alone. Revisit if/when provenance representation\n (still open, below) settles on something richer.\n- **Kind → section:** `Observation.Kind` (default `\"general\"` when empty) is\n the section name, reusing `builtin.ValidateSectionName` — so an\n invalid `Kind` fails `Write` with the same validation error `memory_edit`\n already returns for an invalid section today, not a new error shape.\n\n**Memory engine: scopes, CRUD with retained revisions, and store placement —\nDECIDED (2026-09-16).** Supersedes the 2026-09-02 \"Authoritative record and\nrevision model for the builtin engine\" entry above, whose `\u003c!--mem:\u003culid\u003e--\u003e`\nlive marker cannot carry provenance. Authority:\n[`docs/prds/memory-engine-unification.md`](/dev/prds/memory-engine-unification/)\n(Approved); the contract is `internal/domain/memory/contract.go`. This register\nsupersedes; it does not erase.\n\n- **Four typed scopes, caller-named.** `internal/domain/memory` is addressed by\n `Scope{Kind, Agent, User}` over `global`, `agent`, `user` and `agent-user`,\n with `Validate` rejecting a component present where it is not required. A\n caller names the read and write sets (`Subject.Scopes()` /\n `WritableScopes()`; global is never model-writable) and the engine applies no\n policy of its own: **access control is naming a scope, and nothing else.** An\n engine that enforced policy would need to know about channels, sessions and\n bindings, and every backend would reimplement it. `Scope.Key()` length-prefixes\n each component, so two scopes can never share a key even when an opaque\n channel-native id embeds the separator.\n- **The caller resolves identity; the engine never does.** Resolution happens in\n the gateway turn path, where the native sender id and the session are both in\n hand, and is supplied per channel by the composition. A webhook's sender id is\n a route path, not a person, and treating it as one would give a URL an\n identity. An absent user id yields global and agent scopes only: \"show none of\n it\", never a wider fallback.\n- **CRUD supersedes, never overwrites.** `Store` is `Create`/`Get`/`Query`/\n `List`/`Update`/`Forget`/`Revisions` over the four scopes. `Update` appends the\n superseded state to the scope's `HISTORY.md` before replacing the live block;\n `Update.Expected` names the revision being replaced and returns\n `ErrStaleRevision` on mismatch, making a lost update between two processes\n sharing one scope file detectable. `Forget` records a deleted revision before\n removing the live block, so a record's provenance outlives its content; `Get`\n and `Forget` take a scope, because an id alone does not locate one.\n- **Provenance representation.** The live marker becomes JSON, since a\n `\u003c!--mem:\u003culid\u003e--\u003e` line cannot carry provenance, and `Record` retains the\n provenance `agent-system.md` requires — scope, author, originating user,\n source, and revision history. `Metadata` is dropped: it never round-tripped.\n- **Store and scanner placement.** The markdown store relocates to\n `internal/infrastructure/memory/builtin`, its correct home under\n `organisation.md`'s \"a package owns its on-disk format end-to-end\". The content\n scanner moves into `internal/domain/memory` and is applied in `Create` and\n `Update` — the single choke point every producer crosses, so curator-extracted\n content is covered too.\n\n**Open in this section.** Ranked retrieval is deferred: `Query.Text` stays in\nthe contract, unused by the prompt path. Existing memory data is not migrated:\n`\u003cworkDir\u003e/memory` and `\u003cworkDir\u003e/memory-engine` are left on disk, unreferenced.\nThe PRD leaves these undetermined and they remain open here: group-chat\nextraction, where the curator fails closed on mixed-sender sessions; two\nprocesses (`archied` and `archie-gateway`) holding one scope file, where\n`Update.Expected` detects but does not prevent a lost block; unbounded\n`HISTORY.md` with no compaction policy; whether the per-scope size bound is\nright; and whether the dashboard should carry a user identity.\n\n### 6. Runtime and process boundaries — DECIDED\n\n`archied` is the native resident orchestration and interactive-chat process.\nEvery autonomous repository workflow crosses one full-task core-NATS handoff\ninto a task-scoped `archie-agent` container. Embedded versus external NATS\nchanges broker deployment only; it never selects an executor. The former host\nin-process runner, per-stage NATS protocol, and subprocess runner are removed.\n\nForge, store, and worktree authority remains in `archied` and is exposed to the\ntask container through scoped RPC services. The worker runs the workflow and\nits worker-local ai-sdk loop; it does not receive forge credentials or direct\nstore access. See `docs/prds/embedded-nats.md`.\n\n**UI Service boundary — RATIFIED (2026-09-07).** Phase 3 has a focused\nauthority record in [`docs/prds/ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\nThe UI Service owns HTTP/SPA delivery, UI DTOs, browser authentication, and\nits own endpoint/credential/readiness settings. It consumes Gateway and State\nStore contracts and does not receive `config.Holder`, daemon pointers, SQL or\nstore implementations, workflow registries, forge/channel/model runtimes, or\nresolved secrets. The current `internal/webui` implementation remains inside\n`archied` until the ratified migration gates prove route parity, adapter\nselection, failure/readiness behavior, and deletion of the shared-holder and\ndirect-access paths. The extraction cutover removes the in-process UI\nlistener in the same change; there is no dual-live UI authority.\n\n**Dashboard live event delivery — DECIDED (2026-09-09, `archie-core-za9f`).**\nThe extracted UI receives live activity by polling the State Store's existing\n`EventsSince` cursor, not by a new push-subscription RPC. No proto change, no\ncontract amendment beyond this record.\n\nThe UI process runs one event pump. It primes its watermark by paging\n`EventsSince` to the end at startup without delivering, then polls\n`EventsSince(watermark, limit)` on an interval and hands each new event to\n`webui.Server.Broadcast`, exactly where `internal/app/archied/main.go`'s\n`persistAndBroadcastEvents` hands events to it today. One pump serves every\nconnected browser, so cost is one indexed query per interval regardless of\nviewer count, and `Broadcast` is a no-op when nobody is watching.\n\nThree properties make the cursor sufficient, and a push hub unnecessary:\n\n- **The cursor cannot skip.** `internal/store/store.go` opens the database with\n `SetMaxOpenConns(1)`, so writes serialise and a row's `id` is assigned in\n commit order. There is no window in which a lower id becomes visible after a\n higher one, which is the failure mode that normally rules out polling an\n autoincrement cursor. SSE catch-up already depends on this same property.\n- **A broadcast is a wakeup, not the payload.** `sseStream.drain` treats each\n broadcast event as a signal to fill the persisted gap ahead of it via\n `catchUp`, then delivers the event only if catch-up did not already cover\n it, deduplicating on the `since` watermark. Substituting a poller for the\n in-process bus therefore changes no SSE semantics, and needs no change to\n `internal/webui/sse.go`. It also means a broadcast dropped to a stalled\n client self-heals on the next one.\n- **The table is the only complete fan-out point.** Events reach the store\n through several paths that never touch the daemon's bus, including the\n audit event `store.ArchiveTask` writes inside its transaction\n (`internal/store/store.go`) and the binding-dispatch failure\n `internal/daemon/daemon.go` inserts directly. A push hub would have to be\n hooked at every write site and at post-commit, and would silently miss any\n site added later that forgets to notify. Reading the table misses nothing.\n\nThe cost is one poll interval of latency on the activity feed, default 1s and\noperator-tunable. That is accepted: the feed reports task lifecycle history,\nand an operator action already gets its own synchronous HTTP response.\n\nThis resolves the `/events` row of the route-owner inventory in\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/). `/api/logs` and\n`/api/logs/stream` are not resolved by it, they read the daemon's in-process\ndiagnostic feed and remain host-local (see `archie-core-8cda.5.4`).\n\nRevisit only if sub-second feed latency becomes a requirement, or if a second\nconsumer needs the same stream, at which point the fan-out belongs in the State\nStore process behind a `SubscribeEvents` RPC and this record is superseded.\n\n**Dashboard configuration page — DECIDED (2026-09-09, `archie-core-ymut`).**\nThe configuration page becomes read-only in the UI process for Phase 3. Reads\ncross the boundary as a published snapshot; writes are descoped, not\ncontracted. This amends the ratified boundary, which assumed a single admin\ncontract carrying both.\n\n**Reads.** `webui.ConfigView` is already the whole answer. It is a UI-owned,\nsecret-free value projection built in `handleConfig`: `ProviderView` carries\nthe API key's environment variable name and a `Configured` boolean, never a\nvalue, and `RepoView` is repository metadata only. It is already the exact\nshape the browser receives. So the daemon publishes that projection, plus\nprovenance, reload status and the overridden-key list, as a snapshot the UI\nreads; nothing new has to be designed, only moved. Two State Store RPCs carry\nit, a `PutConfigSnapshot` restricted to the administrative token and a\n`GetConfigSnapshot` the UI calls. The daemon republishes on boot and after\neach successful reload, which is precisely when the projection changes.\n\nThe State Store is the right holder because a published configuration snapshot\nis shared state, it is already dialled by the UI, and it costs no new listener,\nport, token or interceptor. The alternative, a config admin service on\n`archied`, would make the daemon a gRPC server for the first time. Today it is\npurely a client of Gateway and State Store; `grpc.NewServer` appears only in\n`RunGateway` and `RunStateStore`, which are the standalone binaries' entry\npoints, not the daemon's.\n\n**Writes.** `PATCH /api/config`, `POST /api/config/reset` and\n`PATCH /api/config/repos/{owner}/{name}` answer 503 in the UI process, and the\nSPA hides those controls rather than offering a button that fails. Editing is\n`config.toml` plus reload, which is already the only path for every setting\noutside the runtime-tunable allowlist.\n\nThis is a deliberate feature reduction and worth naming as one. The write path\nis not hard because of the transport; it is hard because the policy is\ndaemon-local and stays that way: `installUpdateConfigHandler` denies\nnon-runtime-tunable keys, applies a dotted overlay to a clone of the published\nconfig, validates the materialised result, writes overlay rows, and rebuilds\nprovenance. Contracting that means either moving the policy, which the\nboundary forbids, or exposing it verbatim as remote procedure calls, which\nbuilds a Configuration Service inside a migration bead. Neither is work this\nphase should be doing to keep one page's buttons alive.\n\nRevisit when dashboard configuration editing is picked up as a feature in its\nown right (`archie-core-1786637498420-327`, still open). At that point it gets\nan admin contract designed as a feature, and the Configuration Service this\ndocument already anticipates is its natural owner. Until then the UI renders\nconfiguration and does not write it, which is what the boundary always said\nabout ownership even when it assumed a wider contract.\n\nAmended 2026-09-11. This descope is a deferral, not a decision that the\ndashboard should be read-only. Name it that way in review: the daemon-side\npolicy was built and tested, then deleted by the cutover (`e037b95` removed\n`internal/app/archied/config_update.go` and its tests), so reviving it is\nrecoverable work, not a rewrite. `archie-core-j28m` carries the remaining\nquestion -- which process serves the policy and what transport carries the\nwrite -- and blocks the feature bead. The handlers, their error sentinels and\nthe `UpdateConfig`/`ResetConfig`/`UpdateRepoField` seams on `webui.Server` are\nkept deliberately for it; `ConfigView.Editable` follows whichever process holds\n`UpdateConfig`, so wiring that one field makes the page editable\n(`archie-core-ml30`).\n\nImplementation belongs to `archie-core-8cda.5.4`, the cutover that removes the\ndaemon's own dashboard. Nothing here is built yet.\n\n**Superseded 2026-09-21 by the runtime control plane.** The open question this\ndescope left -- which process serves the policy and what transport carries the\nwrite -- was answered by `docs/prds/runtime-control-plane.md`: neither a daemon\nadmin service nor a revival of the daemon-local policy. `ControlPlaneService` on\nthe State Store's existing gRPC server owns every database-backed setting, and\neach feature validates its own resource, so there is no daemon-local policy left\nto contract. `archie-core-j28m` is closed.\n\nConsequently the three write seams described above no longer exist.\n`PATCH /api/config`, `POST /api/config/reset` and\n`PATCH /api/config/repos/{owner}/{name}`, the `UpdateConfig`/`ResetConfig`/\n`UpdateRepoField` fields on `webui.Server`, and `ConfigView.Editable` were\ndeleted in `31622ec5` along with the runtime overlay store they wrote to.\n`GET /api/config` remains a read of the published projection, as this decision\nalways intended.\n\n### 7. Shared mechanics\n\nThe following cross-domain mechanics require exact contracts before dependent\npackages migrate:\n\n- event-bus delivery and failure semantics;\n- policy evaluation API, composition, and precedence;\n- configuration candidate preparation, health validation, atomic promotion,\n observation, and rollback;\n- service-specific health, bounded remediation, isolation, and escalation;\n- changelog association and grouping.\n\nThese shared mechanisms do not acquire ownership of domain meaning. Domains\ncontinue to own their commands, events, policies, settings, and consequences.\n\nThe Go documentation generator, its output placement, and drift checking are\ndefined in `generated-documentation.md`. The VitePress site and its Pages\ndeployment were removed on 2026-09-12 as an unnecessary build step; `docs/` is\nrepository documentation only. The rendering and publishing surface is an open\ndecision, and so are the renderer's page templates and any developer commands\nor CI sequence that would accompany one. Adapting the generator and implementing\nArchie's normalized documentation model remain migration work rather than open\nproduct architecture.\n\n**Amended 2026-09-15.** `internal/infrastructure/servicediscovery/nats`\nimplements the `servicediscovery.ServiceRegistry` contract (Connect, Register,\nResolve, Watch) over NATS KV and is fully tested, but no production code\nconstructs it and no config surface exposes it. This is a deliberate deferral,\nnot an oversight: service-to-service addressing belongs to the runtime and\nprocess boundary work in section 6, which currently pins container addressing\nby injecting the daemon's startup NATS URL rather than resolving through a\nregistry. The scaffold is kept for that work; re-derive it only if the\naddressing model changes.\n\n### 8. Cutover sequence\n\nThe final migration plan must order:\n\n- compatibility adapters;\n- schema migrations and data backfills;\n- any required dual-read or transitional paths;\n- domain-by-domain package replacement;\n- production wiring changes;\n- legacy package removal;\n- architecture tests preventing dependency regression;\n- focused, integration, race, and full quality gates.\n\nEach legacy package requires objective removal criteria. No compatibility path\nmay become an indefinite second implementation.\n\n\n## Completion criteria\n\nMigration design is complete when:\n\n- every current package has one approved owner and destination;\n- every mutable record has one authoritative owner and migration path;\n- all external and cross-process contracts have versioned owners;\n- existing behaviour and data have explicit parity requirements;\n- process boundaries have evidence-backed justification;\n- the cutover order preserves a working application;\n- legacy deletion criteria and deterministic architecture tests are defined;\n- no unresolved decision requires an implementer to invent domain semantics.\n\nExact Go APIs, SQL statements, and mechanical refactoring steps MAY be finalized\nduring implementation when they do not change these semantics.\n\n## Recorded migration position (2026-07-29)\n\nRecovered 2026-08-09 from the pre-migration issue tracker. This section records\ndecisions the maintainer approved that the repository did not otherwise capture.\n\n**Completed:** `internal/infrastructure/eventbus/nats` (rebuilt, not relocated);\n`internal/infrastructure/configuration` (loading moved out of `internal/config`,\nwhich is now types-only and does no I/O); `internal/eventbus` (broker-neutral\ncontract); `internal/domain/workintake` (the first `internal/domain/` package,\nowning `TaskEnvelope`, the routing `Kind`, the label vocabulary, and task\nsubjects).\n\n**DECISION — private input DTO deferred (approved).**\n`configuration.Document.Config` is still `internal/config.Config` rather than the\nprivate per-domain input DTO this document mandates. Building an ~800-line\nparallel type tree while every consumer still reads `config.Config` would mean\ntwo shapes kept in sync by hand, with no consumer for the new one. It is named as\na field so the compromise is visible at each use site, with the deletion\ncondition recorded in that package's `doc.go`. Do this **with** the\n`internal/config` dissolution, not ahead of it.\n\n**Agreed next: `internal/app/{archied,agentworker}`.** `run()` in\n`cmd/archied/main.go` is ~460 lines, against `organisation.md`'s rule that\n`cmd/*` must not contain substantive wiring or act as a service locator. It is\nalso the last non-infrastructure holder of the NATS SDK, and it is where the\nconfig DTO translation will land, so it unblocks the dissolution above.\n\n**Method constraint.** Do not write a separate recorded package-review document\nper area when the decisions are already dictated by\n`dependencies-and-contracts.md`. That is the ceremony pattern this project\ndeliberately avoids.\n\n## OPEN — `[chat.*]` config section naming\n\n`configuration.md:160` already states the position: `ChatConfig` holds \"separate\nchannel instance settings... there is no chat-wide settings owner\". The config\nsection name has not followed, so a settings block named for a product surface\nowns what the architecture calls channel state.\n\nTwo candidates, neither settled: `[channel.*]` and `[messaging.channel.*]`. The\nsecond reads as a domain path, and no config section in this repo is currently\nnamed after a domain package, so adopting it sets a precedent for every other\nsection rather than fixing one name.\n\nBlocked on a maintainer decision, not on evidence. This is a rename with an\noperator-visible TOML break, so it should land with the `internal/config`\ndissolution recorded above rather than on its own.\n" + "body": "# Architecture Migration Decisions\n\n**Status:** Active migration inventory \n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nThis document records what remains to be decided before Archie's current\nimplementation can be migrated into the approved domain-oriented structure.\n\nThe product and core domain model are not being redesigned. Existing Archie\nbehaviour and the existing session implementation are the migration\nbaseline. Remaining decisions MUST be grounded in current packages, consumers,\ntests, persistence, and runtime wiring.\n\n## Confirmed domain packages\n\n```text\ninternal/domain/\n identity/ (approved, not yet materialised as a package)\n agent/ (approved, not yet materialised as a package)\n workflow/\n messaging/ (canonical Message/Conversation types only; gateway migration pending, see section 2)\n workintake/\n plugin/ (approved, not yet materialised as a package)\n curator/\n mapping/\n scheduling/\n```\n\n- `identity` owns persistent actors, lifecycle, ownership, attribution, and\n audit identity.\n- `agent` owns persistent assistants, specialisation, instructions,\n model-independent continuity, capability context, and associations with\n identity and memory.\n- `workflow` is a separate domain. It owns reusable, versioned Workflow\n definitions, WorkflowSteps, WorkflowExecutions, StepExecutions, Attempts,\n outcomes, and typed Workflow plugin contracts.\n- `messaging` owns canonical Messages, Conversations, branches, channel and\n source correlation, replies, acknowledgements, delivery attempts, events, and\n projections.\n- `workintake` owns admission, validation, deterministic routing, and accepted\n work requests for the optional transition into durable workflow-backed\n execution.\n- `plugin` owns generic plugin identity, discovery, compatibility, and lifecycle\n mechanics. It remains metadata-only; capability semantics and typed\n registration belong to the owning domain.\n- `curator` owns the curator engine family: long-running agent loops that\n reason over memory and perform maintenance tasks on a declared shape\n (check-in interval, tool set, attached memory engine). See epic\n archie-core-yp9.\n- `mapping` owns the payload field-mapping vocabulary: binding named fields to\n JSON paths inside a captured webhook payload, and detecting drift. See\n `docs/prds/payload-field-mapping.md`.\n- `scheduling` owns the ticker engine: deciding when a scheduled job runs and\n with what concurrency. It declares `JobSource`/`Runner` contracts and owns\n no storage or delivery. See epic archie-core-1786637496320-249-be6d9a20.\n\n**Reconciliation (2026-08-31):** `curator`, `mapping`, and `scheduling` existed\nin the tree without being added to this list when their epics landed. They are\nconfirmed as legitimate, narrowly-scoped domains under the same rules as the\noriginal six — each owns one capability family, declares typed contracts, and\nhas no cross-domain reach. This list was out of date, not the tree.\n\nConversely, `identity`, `agent`, `messaging`, and `plugin` are approved target\ndomains that do not yet exist as packages. Their behaviour currently lives in\npre-migration packages (see \"Complete package destination map\" below for the\nfull inventory). Do **not** create empty stub packages for them ahead of an\nactual migration PR — an unused stub is dead code and gives no test coverage.\nEach materialises when its corresponding migration decision below is resolved\nand implemented in the same change.\n\nMemory's final package placement remains undecided pending focused review of the\nexisting implementation.\n\nThe following are approved shared or composition packages, not domains:\n\n```text\ninternal/eventbus/\ninternal/policy/\ninternal/infrastructure/\ninternal/app/archied/\ninternal/app/agentworker/\n```\n\n## Remaining migration decisions\n\n### 1. Complete package destination map\n\nEvery current package MUST receive one explicit disposition:\n\n- move into a confirmed domain;\n- move under infrastructure as an implementation of a domain-owned contract;\n- move into application composition;\n- merge with an identified duplicate;\n- remain as an approved shared application contract;\n- move to a plugin, extra, deployment, example, or generated artifact;\n- be deleted after its replacement is proven.\n\nThe largest unreviewed areas include memory, tools and capabilities, workspace\nand worktree behaviour, gates, storage, events, web UI, containers, RPC\npackages, release management, and scheduling.\n\nThe destination matrix MUST name the behaviour owner, target path, dependencies\nto remove, state and contracts to preserve, migration prerequisites, and\ndeletion criteria.\n\n**`internal/webui` / UI Service — RATIFIED Phase 3 disposition (2026-09-07).**\nThe current owner is the in-process HTTP/SSE dashboard: SPA hosting, operator\nauthentication, UI response projection, configuration reads and mutations,\ntask/control APIs, and direct adapters to State Store, channel, event/log,\nmemory, curator, workflow, forge, and Gateway runtime values. The target owner\nis a standalone UI application process; `internal/webui` remains its HTTP\ntransport adapter, while Gateway, State Store, configuration, Work Intake,\nMessaging, and other capability owners expose the contracts behind those\nroutes. The migration preserves retained route schemas, auth/CSRF behavior,\ntask/capture/mapping/binding/session continuity, SSE replay semantics, and\ncommitted generated SPA assets. Prerequisites are the ratified UI boundary,\nroute-owner inventory, narrow remote adapters, configuration/auth/readiness\ncontracts, and process/conformance tests. Remove the shared live\n`config.Holder`, direct event/log bus, daemon stopper, forge client, channel\nmanager, capability registries, composite store access, and in-process UI\nlistener only after the focused criteria in\n[`docs/prds/ui-service-boundary.md`](/dev/prds/ui-service-boundary/) pass in\nproduction composition. The old construction and listener are deleted in the\nsame cutover change; no second UI authority remains.\n\n### 2. Session and Messaging migration\n\nThe existing session implementation is the migration baseline. The migration must\ndecide:\n\n- which session and message components are retained or adapted;\n- how Agent, user, channel, Conversation, and branch ownership are added;\n- how Archie's parallel gateway session implementation is replaced;\n- how canonical Messages become immutable persisted records;\n- how existing sessions and messages are migrated;\n- how outbound delivery, retry, acknowledgement, deduplication, and failure\n semantics are represented;\n- how current Telegram, email, webhook, forge, and Jira integrations migrate to\n Messaging infrastructure adapters.\n\nThe following are already decided and MUST NOT be reopened without contradictory\nimplementation evidence:\n\n- the composite Conversation identity;\n- immutable MessageIDs;\n- typed tool-call and tool-result messages;\n- exact branch lineage through ParentConversationID and ForkMessageID;\n- parent and child transcript isolation;\n- normal scoped-memory behaviour regardless of the originating Conversation;\n- Messaging and Work Intake ownership.\n\n**Decision (2026-08-31), resolving the open questions above:**\n\n`internal/gateway/` is the \"parallel gateway session implementation\" this\nsection names. It is retained as-is for now — a big-bang rewrite of live\nsession/compression/approval/branch logic across Telegram, web UI, email, and\nwebhook channels in one change is exactly the \"convoluted to satisfy edge\ncases\" failure mode CLAUDE.md's Revert-on-Flail rule warns about, and would\nrisk silent behavioural regressions in the one path every user-facing message\ntravels through.\n\nInstead, `internal/domain/messaging/` is materialised now with the pieces\nthat are already decided and don't require touching gateway: the canonical\n`Message` and `Conversation` types (composite Conversation identity, immutable\n`MessageID`, typed tool-call/tool-result message variants, branch lineage via\n`ParentConversationID`/`ForkMessageID`). These are pure domain types with no\ngateway dependency — a new package, not a rewrite of an existing one.\n\nMigrating `internal/gateway/`'s session, compression, approval, and branch\nlogic onto these types is real, cross-channel-risk work and is **not** done in\nthis pass. It is tracked as separate, appropriately-scoped bd issues (see\n`bd list --label messaging-migration`), sequenced channel-by-channel so each\nlands with its own behavioural verification instead of one large diff no one\ncan safely review.\n\n**Amended 2026-09-15.** `internal/infrastructure/gatewayrpc.StoreClient` is the\nadapter this migration will need: it implements `messaging.SessionStore` over\nthe chat service's session-store RPCs and is proven by\n`TestStoreClientPreservesCanonicalRecords`. It is deliberately unwired.\nProduction `archied` dials the standalone gateway for `ChatContract` only\n(`composeChatContract` → `gatewayrpc.Dial` returns a `*Client`, not a\n`*StoreClient`), and the Gateway's session store is PostgreSQL via\n`gateway.NewPostgresSessionStore`. The test comment that says it mirrors production\ndescribes the target, not today's tree. Kept rather than deleted because the\nmigration it serves is deferred by this very section, and re-deriving a tested\nwire adapter once that lands is pure cost.\n\n**Phase 4 Messaging Service extraction (2026-09-19, `archie-core-8cda.6.1`).**\nThe process boundary for external channels (Telegram, email, webhook) is ratified\nin [`docs/prds/messaging-service-boundary.md`](/dev/prds/messaging-service-boundary/).\nMessaging is extracted into `cmd/archie-messaging` / `internal/app/archiemessaging`,\ncalling Gateway's `messaging.ChatContract` exclusively over gRPC. Channel frontends\nhold no direct database handles, daemon state, or model runtimes.\n\n**Telegram operator surface after extraction (2026-09-19, `archie-core-8cda.6.6`).**\n`setupTelegramGateway` wired eight `telegram.Gateway` seams the daemon could\nsatisfy from its own process. Each is resolved by where the fact it needs is\nactually owned, not by where it used to be built:\n\n| Seam | Resolution |\n|---|---|\n| `Version` | Sourced from the Gateway via `ChatSnapshot.Version`. The Messaging Service never stamps or infers a build version; `cmd/archie-messaging` deliberately takes no version ldflags, so it cannot report a partial upgrade as a matched one. |\n| `Updates` (`/update`) | Built in the Messaging Service from `[chat.telegram].update_check_command` / `update_install_command` — its own configuration. `Enrich` is dropped: `componentInstallTypeEnricher` reads `[nats]`, which this service must not decode, so the check command's own reported install type stands unmodified. |\n| `UpdateReportPath` | A local state file under `work_dir`, keyed by the same `sha256(bot_user)[:8]` scheme the daemon used, so a pending report is found rather than written afresh. |\n| `ReleaseAnnouncements` | **Deliberately left nil**, for the same reason as `RunningVersions`. `releaseannounce.Announcer` announces nothing unless each `Component` carries a parseable version, and the only versions this service can obtain are the Gateway's own build stamps, not archied's. Wiring the announcer without them yields a no-op that reads as configured, so it stays unwired until component self-reporting exists. |\n| `SetShowToolCalls` | Projected from `[chat].show_tool_calls`. |\n| `Reload` | Local: re-resolves the token and allowlist from this service's own config file and overlay. Reload was never a daemon fact. |\n| `RunningVersions` | **Deliberately left nil.** It exists to turn an installer's claim into a checked one, and only a component's own compiled-in build can vouch for it. The Messaging Service knows neither archied's build nor the observed archie-agent version (`daemon.AgentStatus`), and the Gateway's own `gatewayVersion` is a different binary's stamp — reporting it as the daemon's would manufacture exactly the false success the check exists to catch. Update reports therefore relay as unverified claims until a component self-report contract exists. |\n| `Dangerous` | Stays nil. No daemon composition ever set it; `/rollback` and `/stop` reported \"not configured\" before the extraction and still do. |\n\nThe Messaging Service consequently reads `work_dir`, `bot_user`, `[chat]` and\n`[health]` in addition to the PRD's `[chat.*]` / `[services.gateway]` list.\n`[health]` is the daemon's health URL, which `releaseupdate.CommandInstaller`\npolls from outside to confirm an update came back up; the rest of the PRD's\nprohibition list is untouched.\n\n**Phase 4 messaging severance: what must move, and the decision the PRD does\nnot cover (2026-09-22, `archie-core-1ng1`).** The PRD's deletion gate cannot\npass today. `cmd/archie-messaging/architecture_test.go` does not exist, and the\ncheck that section specifies -- `go list -deps ./cmd/archie-messaging` links\nzero banned runtime packages -- reported four at the time (the task store was\nthen `internal/store` over `modernc.org/sqlite`; the gate now bans\n`modernc.org/sqlite`, `github.com/jackc/pgx`, `internal/infrastructure/postgres`\nand `internal/infrastructure/legacyread` in its place):\n\n| banned package | PRD category |\n| --- | --- |\n| `modernc.org/sqlite` | State Store |\n| `internal/store` | State Store |\n| `internal/domain/workflow` | Workflow engine |\n| `internal/agentexec` | Workflow engine, via `internal/domain/workflow/agent.go` |\n\nAll four arrive through one import: `internal/app/archiemessaging/run.go`\nimports `internal/app/controlplane`, whose own dependency set is exactly those\nfour. The Messaging Service uses three symbols from that package\n(`NewRPCClient`, `RuntimeChatConfig`, `ChannelSettingsKind`); the package also\ncarries the store-backed `ResourceStore` server, the `Definition` registry and\nthe workflow step vocabulary. Sever the package. Amending the gate to tolerate\n`modernc.org/sqlite` and `internal/store` is rejected: a boundary you whitelist\naround is no longer a boundary.\n\n**What moves.**\n\n- Every kind constant. They are strings with no dependency, and the client names\n them.\n- The channel-settings vocabulary and its decode: `channelSettings`,\n `telegramSettings`, `webhookChannelSettings`, `emailSettings`,\n `rateLimitSettings`, `channelDuration`, and `decodeRenamingLegacyKeys`.\n- The generic client: `Client`, `NewRPCClient`, `RuntimeConfig` with\n `runtimeConfigFrom`, `runtimeChatConfigFrom`, the other `runtime*From`\n projections, and the `resourceReader` interface.\n- The persona watch (`WatchPersonas`, `AppliedPersonas`), which needs only\n `internal/domain/agent`.\n- `executionSettingsDocument`, the document alone.\n\n**What stays, and why.**\n\n- The store-backed server: `Server`, `NewServer`, `ResourceStore`,\n `ImportConfig`, `SeedSkip`, the `Definition` registry, and every other kind's\n seed, validator and `Normalize` hook.\n- `WorkflowDefinitionsClient` and `stepRegistry`, which decode through\n `internal/domain/workflow`.\n- `WatchWorkflowExecutionSettings` and `decodeSettings`: `decodeSettings`\n returns and validates `workflow.ExecutionSettings`, so any package holding it\n links the workflow domain and therefore `internal/agentexec`. This is why the\n client cannot move as one package -- see open question 2.\n- The schedules document, whose type is `scheduling.JobSpec` in\n `internal/domain/scheduling` -- the vocabulary moved there when the legacy\n file-backed cronstore was deleted, which is also what satisfies the rule\n that a domain vocabulary package may not import infrastructure.\n- The schema derivation, which describes the documents the server advertises.\n\n**Placement as landed (2026-09-22, `archie-core-1ng1`).** One new package,\n`internal/infrastructure/controlplanerpc`, holding both the vocabulary (kinds, the\nchannel-settings document and its decode) and the client\n(`NewRPCClient`, `Query`, `RuntimeChatConfig`, the exported `ResourceReader`, the\nwire sentinels and `ClientError`). A second `internal/domain/controlplane` package\nfor the vocabulary was planned and then measured out of the design: the vocabulary\nhas exactly one consumer, the client that projects it, and the store-backed side\nreads the same definitions through type aliases rather than a copy -- so a separate\ndomain package would have been a package with one consumer and a second name for\none thing. What made that safe is a measurement, not a preference:\n`internal/app/controlplane` was the ONLY direct import of the Messaging Service\nthat dragged a banned runtime (`gatewayrpc`, `staterpc`, `applystatus`, `messaging`,\n`configuration`, `readiness` and `health` were all clean), so severing that one\ndependency is sufficient, and the severance needed no third package. Promoting the\nvocabulary to a domain package stays open for the day a second consumer appears.\n\n**Settled by the implementation.** Two of the four open questions this decision\noriginally left are answered, in both cases by measuring rather than by preference:\n\n- The client did NOT become two packages. `WatchWorkflowExecutionSettings` and\n `WatchPersonas` stayed on `internal/app/controlplane`'s `Client`, which now\n *embeds* the new one, so the daemon's validated watch is still defined once and\n the read path still has one implementation -- which is what the second package\n was for -- with one fewer package to keep in step.\n- The schedules and workflow documents stayed put, and\n `internal/domain/workflow/task` is linked deliberately: it carries the Task\n status and view vocabulary, its own dependency set holds none of the banned\n runtimes, and `cmd/archie-messaging/architecture_test.go` carries it as an\n exception checked in both directions (the same entry, for the same reason, as\n `cmd/archie-ui/architecture_test.go`).\n\n**Still open.**\n\n1. Does the server also leave `internal/app/`? `organisation.md` reserves\n `internal/app/` for wiring and composition, and a store-backed gRPC server is\n infrastructure -- but the gate does not require the move, and it would multiply\n the diff across archied's wiring and the State Store binary. Tracked as\n `archie-core-8f4n`.\n2. Does `internal/domain/workflow`'s contract/engine split happen, and when? A\n contract type (`WorkflowDefinitionCollection`) drags `internal/agentexec` only\n because `agent.go` sits beside it. Independent of this severance, tracked as\n `archie-core-inh6`, and the shape `archie-core-8cda.7` will meet again.\n3. Does the vocabulary move to a domain package when a second consumer appears?\n\n**Both consumers keep compiling.** archied imports the new vocabulary for the\nkinds it names and the new client package for `RuntimeConfig` and the persona\nwatch, and keeps `internal/app/controlplane` for the server, the registry,\n`ErrVersionConflict` and the workflow-aware pieces. The State Store binary\ncontinues to call `NewServer` and `ResourceStore` unchanged. The Messaging\nService then imports only `internal/infrastructure/controlplanerpc`,\n`gatewayrpc` and `staterpc`, so `go list -deps ./cmd/archie-messaging` loses all\nfour banned packages and the gate can land green.\n\n**Channel state to the dashboard: one administrative surface, following the\nconfig-snapshot precedent (2026-09-22, `archie-core-8cda.6.8`).** The Messaging\nService now records its channels' lifecycle -- `internal/channels/status.Manager`,\none entry per composed channel, written by each channel's own report through\n`channels.Lifecycle` -- and nothing reads that across a process boundary.\n`/api/channels` answers with nothing because `webui.Server.Channels` has no\nproduction source, and POST `/api/channels/{id}/reload` answers 501 for the same\nreason. The carrier is the State Store, and the shape is the one\n`archie-core-ymut` already established for config snapshots\n(`docs/prds/state-store-contract.md`, \"the one surface whose writer is the daemon\nand whose reader is the UI process\"): a pair of administrative RPCs, deny-by-default\nunder `authorizesTaskScopedCall`, so a task-scoped grant reaches neither.\n\nWhat that decides, so the implementation need not invent it:\n\n- A channel's state is **runtime** state, not a settings document. It does not\n belong in the `ControlPlaneService` resource kinds, which are validated,\n versioned documents seeded from config; a channel that is failed is not a\n document anybody edits.\n- The reader is the UI process, which already dials the State Store and pumps its\n cursor, so the surface needs no new transport in either direction.\n- Adding the pair changes the contract's RPC count, so\n `docs/prds/state-store-contract.md`'s grouped table gains a group in the same\n change that adds the RPCs -- the table is the contract's own index, and a\n surface it does not list is a surface nobody finds.\n\n**Reload gets its own surface, with its own lifetime (decided 2026-09-22).** Reload\nruns the other way -- the UI asks the service to reload a channel -- and the two\ncandidates were a request surface the service observes and clears, or a field on\nthe status surface the service watches. It is the first. The reasoning, recorded\nbecause it would look arbitrary later:\n\n- A request queue and a status report do not share a lifetime, so they should not\n share a surface. A pending request is transient and has an outcome; a channel's\n state is durable and has a current value.\n- A \"please reload\" field on the status resource makes READING status capable of\n RETRIGGERING an action, and makes a transient request indistinguishable from\n state the moment anything polls it.\n- It would also invert the writer/reader relationship the precedent establishes:\n the service would have to clear a field on an object whose whole contract is\n that the service reports it.\n\nSo the status surface stays writer-is-service, and the request carrier is its own\nobject with its own states.\n\n`archie-core-8cda.6.9`'s component self-report is the **same shape**: a process\nreporting something about itself for the UI to read. Whether it shares this\nsurface or gets its own is a decision for `6.9`, and the one outcome to avoid is\ntwo homes for \"what a component reports about itself\".\n\n### 3. Identity data migration\n\nThe migration must define:\n\n- the durable identity schema and repository implementation;\n- bootstrap of the permanent system identity;\n- stable IdentityIDs for existing configured Agents;\n- migration from configured identity names, empty-string ownership, and bot\n usernames;\n- preservation of historical attribution;\n- identity-aware uniqueness and transport deduplication;\n- correction of container and RPC capability resolution so execution uses the\n owning identity's bindings;\n- removal of identity-string equality from access-control behaviour.\n\nExact schema and method signatures are implementation decisions. The approved\nIdentity lifecycle and attribution semantics are fixed migration constraints.\n\n### 4. Workflow migration\n\nThe current implementation must be transformed deliberately:\n\n```text\nstore.Task -\u003e WorkflowExecution\nmutable Stage -\u003e StepExecution history\nRetryCount / Attempt -\u003e Attempt records\nworkflow.Registry -\u003e versioned Workflow definitions\ncurrent stages -\u003e WorkflowStep definitions or Workflow plugins\n```\n\nThe migration must decide:\n\n- Workflow definition and version persistence;\n- the smallest accepted-work-request contract consumed from Work Intake;\n- the enforced WorkflowExecution state machine;\n- atomic state and domain-event persistence;\n- migration of existing task rows and transition history;\n- disposition of current skill, Yaegi, and stage-plugin implementations;\n- the versioned worker-handoff contract;\n- scheduler, claim, retry, crash-recovery, and waiting-human cutover.\n\nWorkflow remains a separate domain from Agent. A user or Agent may define or\ninvoke a Workflow, but Workflow semantics and plugins remain Workflow-owned.\n\n### 5. Memory placement and storage\n\nThe four required scopes are fixed:\n\n- global shared;\n- Agent-wide;\n- user-wide;\n- Agent-user relationship.\n\nA focused review of `internal/memory`, its providers, consumers, persistence,\nand tests had these decisions:\n\n- `internal/domain/memory` versus ownership within the Agent domain — settled\n (2026-09-02, below);\n- the authoritative memory record and revision model — settled by the PRD\n (2026-09-16, below);\n- provider and infrastructure boundaries — settled by the PRD (2026-09-16,\n below);\n- retrieval and access enforcement — settled by the PRD (2026-09-16, below);\n ranked retrieval remains open (below);\n- migration of existing memory data — open: existing data is not migrated;\n- provenance representation — settled by the PRD (2026-09-16, below).\n\nConversation branches do not create another memory scope. A memory action has\nthe same scoped effect regardless of which Conversation originated it.\n\n**`internal/domain/memory` ownership — DECIDED (2026-09-02).** The typed\nengine family lives at `internal/domain/memory`\n(archie-core-1786637490069-11-e245c170), mirroring `internal/domain/curator`'s\ncontract/registrar/registry split. Not owned within the Agent domain: memory\nis consumed by more than agent turns (curators, per archie-core-1786637499636)\nand the plugin engine rule requires its own family, not a capability bolted\nonto another domain's contract.\n\n**Authoritative record and revision model for the builtin engine — DECIDED\n(2026-09-02),** scoped to unblock archie-core-1786637499480 (ship the\nfile-writing store as the default engine). This decides only the builtin\nengine's mapping onto the new `Record{ID}`/`Forget(id)` contract, not the\nmemory model in general or a future non-file engine's model:\n\n- `internal/memory/builtin.Store` is unchanged. It has no concept of a\n stable per-block ID — blocks are addressed by content substring — and nothing\n in this decision requires changing that; existing `internal/memory` tests\n keep passing unmodified, per that issue's acceptance criteria.\n- A new infrastructure adapter (`internal/infrastructure/memory`, per\n `organisation.md`'s \"infrastructure implements domain contracts\" rule — not\n under `internal/domain/memory`, which stays contract-only) wraps one\n `builtin.Store` per identity.\n- **Identity → directory:** `hex(sha256(identity))`, not the identity string\n itself. Closes the path-traversal question without an allowlist regex, and\n sidesteps collisions from identities that differ only in characters a\n filesystem treats as equivalent.\n- **Record ID:** assigned once at `Write`, independent of content, so a later\n edit elsewhere in the file can never orphan a `Forget`. Realized as a\n leading marker line (`\u003c!--mem:\u003culid\u003e--\u003e`) on the block `builtin.Store.Add`\n writes; `Query`/`List` strip it back out of `Record.Content` before\n returning, and `Forget(id)` locates the block by that unique marker via the\n store's existing substring-based `Remove`. The marker is a pragmatic,\n reversible choice — plain-text records stay human-readable and\n human-editable, at the cost of one marker line an operator editing the file\n by hand should leave alone. Revisit if/when provenance representation\n (still open, below) settles on something richer.\n- **Kind → section:** `Observation.Kind` (default `\"general\"` when empty) is\n the section name, reusing `builtin.ValidateSectionName` — so an\n invalid `Kind` fails `Write` with the same validation error `memory_edit`\n already returns for an invalid section today, not a new error shape.\n\n**Memory engine: scopes, CRUD with retained revisions, and store placement —\nDECIDED (2026-09-16).** Supersedes the 2026-09-02 \"Authoritative record and\nrevision model for the builtin engine\" entry above, whose `\u003c!--mem:\u003culid\u003e--\u003e`\nlive marker cannot carry provenance. Authority:\n[`docs/prds/memory-engine-unification.md`](/dev/prds/memory-engine-unification/)\n(Approved); the contract is `internal/domain/memory/contract.go`. This register\nsupersedes; it does not erase.\n\n- **Four typed scopes, caller-named.** `internal/domain/memory` is addressed by\n `Scope{Kind, Agent, User}` over `global`, `agent`, `user` and `agent-user`,\n with `Validate` rejecting a component present where it is not required. A\n caller names the read and write sets (`Subject.Scopes()` /\n `WritableScopes()`; global is never model-writable) and the engine applies no\n policy of its own: **access control is naming a scope, and nothing else.** An\n engine that enforced policy would need to know about channels, sessions and\n bindings, and every backend would reimplement it. `Scope.Key()` length-prefixes\n each component, so two scopes can never share a key even when an opaque\n channel-native id embeds the separator.\n- **The caller resolves identity; the engine never does.** Resolution happens in\n the gateway turn path, where the native sender id and the session are both in\n hand, and is supplied per channel by the composition. A webhook's sender id is\n a route path, not a person, and treating it as one would give a URL an\n identity. An absent user id yields global and agent scopes only: \"show none of\n it\", never a wider fallback.\n- **CRUD supersedes, never overwrites.** `Store` is `Create`/`Get`/`Query`/\n `List`/`Update`/`Forget`/`Revisions` over the four scopes. `Update` appends the\n superseded state to the scope's `HISTORY.md` before replacing the live block;\n `Update.Expected` names the revision being replaced and returns\n `ErrStaleRevision` on mismatch, making a lost update between two processes\n sharing one scope file detectable. `Forget` records a deleted revision before\n removing the live block, so a record's provenance outlives its content; `Get`\n and `Forget` take a scope, because an id alone does not locate one.\n- **Provenance representation.** The live marker becomes JSON, since a\n `\u003c!--mem:\u003culid\u003e--\u003e` line cannot carry provenance, and `Record` retains the\n provenance `agent-system.md` requires — scope, author, originating user,\n source, and revision history. `Metadata` is dropped: it never round-tripped.\n- **Store and scanner placement.** The markdown store relocates to\n `internal/infrastructure/memory/builtin`, its correct home under\n `organisation.md`'s \"a package owns its on-disk format end-to-end\". The content\n scanner moves into `internal/domain/memory` and is applied in `Create` and\n `Update` — the single choke point every producer crosses, so curator-extracted\n content is covered too.\n\n**Open in this section.** Ranked retrieval is deferred: `Query.Text` stays in\nthe contract, unused by the prompt path. Existing memory data is not migrated:\n`\u003cworkDir\u003e/memory` and `\u003cworkDir\u003e/memory-engine` are left on disk, unreferenced.\nThe PRD leaves these undetermined and they remain open here: group-chat\nextraction, where the curator fails closed on mixed-sender sessions; two\nprocesses (`archied` and `archie-gateway`) holding one scope file, where\n`Update.Expected` detects but does not prevent a lost block; unbounded\n`HISTORY.md` with no compaction policy; whether the per-scope size bound is\nright; and whether the dashboard should carry a user identity.\n\n### 6. Runtime and process boundaries — DECIDED\n\n`archied` is the native resident orchestration and interactive-chat process.\nEvery autonomous repository workflow crosses one full-task core-NATS handoff\ninto a task-scoped `archie-agent` container. Embedded versus external NATS\nchanges broker deployment only; it never selects an executor. The former host\nin-process runner, per-stage NATS protocol, and subprocess runner are removed.\n\nForge, store, and worktree authority remains in `archied` and is exposed to the\ntask container through scoped RPC services. The worker runs the workflow and\nits worker-local ai-sdk loop; it does not receive forge credentials or direct\nstore access. See `docs/prds/embedded-nats.md`.\n\n**UI Service boundary — RATIFIED (2026-09-07).** Phase 3 has a focused\nauthority record in [`docs/prds/ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\nThe UI Service owns HTTP/SPA delivery, UI DTOs, browser authentication, and\nits own endpoint/credential/readiness settings. It consumes Gateway and State\nStore contracts and does not receive `config.Holder`, daemon pointers, SQL or\nstore implementations, workflow registries, forge/channel/model runtimes, or\nresolved secrets. The current `internal/webui` implementation remains inside\n`archied` until the ratified migration gates prove route parity, adapter\nselection, failure/readiness behavior, and deletion of the shared-holder and\ndirect-access paths. The extraction cutover removes the in-process UI\nlistener in the same change; there is no dual-live UI authority.\n\n**Dashboard live event delivery — DECIDED (2026-09-09, `archie-core-za9f`).**\nThe extracted UI receives live activity by polling the State Store's existing\n`EventsSince` cursor, not by a new push-subscription RPC. No proto change, no\ncontract amendment beyond this record.\n\nThe UI process runs one event pump. It primes its watermark by paging\n`EventsSince` to the end at startup without delivering, then polls\n`EventsSince(watermark, limit)` on an interval and hands each new event to\n`webui.Server.Broadcast`, exactly where `internal/app/archied/main.go`'s\n`persistAndBroadcastEvents` hands events to it today. One pump serves every\nconnected browser, so cost is one indexed query per interval regardless of\nviewer count, and `Broadcast` is a no-op when nobody is watching.\n\nThree properties make the cursor sufficient, and a push hub unnecessary:\n\n- **The cursor cannot skip.** `internal/infrastructure/postgres` takes an\n advisory event-append lock around each insert (`TestEventInsertsCommitInIDOrder`),\n so a row's `id` is assigned in commit order. There is no window in which a lower id becomes visible after a\n higher one, which is the failure mode that normally rules out polling an\n autoincrement cursor. SSE catch-up already depends on this same property.\n- **A broadcast is a wakeup, not the payload.** `sseStream.drain` treats each\n broadcast event as a signal to fill the persisted gap ahead of it via\n `catchUp`, then delivers the event only if catch-up did not already cover\n it, deduplicating on the `since` watermark. Substituting a poller for the\n in-process bus therefore changes no SSE semantics, and needs no change to\n `internal/webui/sse.go`. It also means a broadcast dropped to a stalled\n client self-heals on the next one.\n- **The table is the only complete fan-out point.** Events reach the store\n through several paths that never touch the daemon's bus, including the\n audit event `store.ArchiveTask` writes inside its transaction\n (`internal/infrastructure/postgres/store.go`) and the binding-dispatch failure\n `internal/daemon/daemon.go` inserts directly. A push hub would have to be\n hooked at every write site and at post-commit, and would silently miss any\n site added later that forgets to notify. Reading the table misses nothing.\n\nThe cost is one poll interval of latency on the activity feed, default 1s and\noperator-tunable. That is accepted: the feed reports task lifecycle history,\nand an operator action already gets its own synchronous HTTP response.\n\nThis resolves the `/events` row of the route-owner inventory in\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/). `/api/logs` and\n`/api/logs/stream` are not resolved by it, they read the daemon's in-process\ndiagnostic feed and remain host-local (see `archie-core-8cda.5.4`).\n\nRevisit only if sub-second feed latency becomes a requirement, or if a second\nconsumer needs the same stream, at which point the fan-out belongs in the State\nStore process behind a `SubscribeEvents` RPC and this record is superseded.\n\n**Dashboard configuration page — DECIDED (2026-09-09, `archie-core-ymut`).**\nThe configuration page becomes read-only in the UI process for Phase 3. Reads\ncross the boundary as a published snapshot; writes are descoped, not\ncontracted. This amends the ratified boundary, which assumed a single admin\ncontract carrying both.\n\n**Reads.** `webui.ConfigView` is already the whole answer. It is a UI-owned,\nsecret-free value projection built in `handleConfig`: `ProviderView` carries\nthe API key's environment variable name and a `Configured` boolean, never a\nvalue, and `RepoView` is repository metadata only. It is already the exact\nshape the browser receives. So the daemon publishes that projection, plus\nprovenance, reload status and the overridden-key list, as a snapshot the UI\nreads; nothing new has to be designed, only moved. Two State Store RPCs carry\nit, a `PutConfigSnapshot` restricted to the administrative token and a\n`GetConfigSnapshot` the UI calls. The daemon republishes on boot and after\neach successful reload, which is precisely when the projection changes.\n\nThe State Store is the right holder because a published configuration snapshot\nis shared state, it is already dialled by the UI, and it costs no new listener,\nport, token or interceptor. The alternative, a config admin service on\n`archied`, would make the daemon a gRPC server for the first time. Today it is\npurely a client of Gateway and State Store; `grpc.NewServer` appears only in\n`RunGateway` and `RunStateStore`, which are the standalone binaries' entry\npoints, not the daemon's.\n\n**Writes.** `PATCH /api/config`, `POST /api/config/reset` and\n`PATCH /api/config/repos/{owner}/{name}` answer 503 in the UI process, and the\nSPA hides those controls rather than offering a button that fails. Editing is\n`config.toml` plus reload, which is already the only path for every setting\noutside the runtime-tunable allowlist.\n\nThis is a deliberate feature reduction and worth naming as one. The write path\nis not hard because of the transport; it is hard because the policy is\ndaemon-local and stays that way: `installUpdateConfigHandler` denies\nnon-runtime-tunable keys, applies a dotted overlay to a clone of the published\nconfig, validates the materialised result, writes overlay rows, and rebuilds\nprovenance. Contracting that means either moving the policy, which the\nboundary forbids, or exposing it verbatim as remote procedure calls, which\nbuilds a Configuration Service inside a migration bead. Neither is work this\nphase should be doing to keep one page's buttons alive.\n\nRevisit when dashboard configuration editing is picked up as a feature in its\nown right (`archie-core-1786637498420-327`, still open). At that point it gets\nan admin contract designed as a feature, and the Configuration Service this\ndocument already anticipates is its natural owner. Until then the UI renders\nconfiguration and does not write it, which is what the boundary always said\nabout ownership even when it assumed a wider contract.\n\nAmended 2026-09-11. This descope is a deferral, not a decision that the\ndashboard should be read-only. Name it that way in review: the daemon-side\npolicy was built and tested, then deleted by the cutover (`e037b95` removed\n`internal/app/archied/config_update.go` and its tests), so reviving it is\nrecoverable work, not a rewrite. `archie-core-j28m` carries the remaining\nquestion -- which process serves the policy and what transport carries the\nwrite -- and blocks the feature bead. The handlers, their error sentinels and\nthe `UpdateConfig`/`ResetConfig`/`UpdateRepoField` seams on `webui.Server` are\nkept deliberately for it; `ConfigView.Editable` follows whichever process holds\n`UpdateConfig`, so wiring that one field makes the page editable\n(`archie-core-ml30`).\n\nImplementation belongs to `archie-core-8cda.5.4`, the cutover that removes the\ndaemon's own dashboard. Nothing here is built yet.\n\n**Superseded 2026-09-21 by the runtime control plane.** The open question this\ndescope left -- which process serves the policy and what transport carries the\nwrite -- was answered by `docs/prds/runtime-control-plane.md`: neither a daemon\nadmin service nor a revival of the daemon-local policy. `ControlPlaneService` on\nthe State Store's existing gRPC server owns every database-backed setting, and\neach feature validates its own resource, so there is no daemon-local policy left\nto contract. `archie-core-j28m` is closed.\n\nConsequently the three write seams described above no longer exist.\n`PATCH /api/config`, `POST /api/config/reset` and\n`PATCH /api/config/repos/{owner}/{name}`, the `UpdateConfig`/`ResetConfig`/\n`UpdateRepoField` fields on `webui.Server`, and `ConfigView.Editable` were\ndeleted in `31622ec5` along with the runtime overlay store they wrote to.\n`GET /api/config` remains a read of the published projection, as this decision\nalways intended.\n\n### 7. Shared mechanics\n\nThe following cross-domain mechanics require exact contracts before dependent\npackages migrate:\n\n- event-bus delivery and failure semantics;\n- policy evaluation API, composition, and precedence;\n- configuration candidate preparation, health validation, atomic promotion,\n observation, and rollback;\n- service-specific health, bounded remediation, isolation, and escalation;\n- changelog association and grouping.\n\nThese shared mechanisms do not acquire ownership of domain meaning. Domains\ncontinue to own their commands, events, policies, settings, and consequences.\n\nThe Go documentation generator, its output placement, and drift checking are\ndefined in `generated-documentation.md`. The VitePress site and its Pages\ndeployment were removed on 2026-09-12 as an unnecessary build step; `docs/` is\nrepository documentation only. The rendering and publishing surface is an open\ndecision, and so are the renderer's page templates and any developer commands\nor CI sequence that would accompany one. Adapting the generator and implementing\nArchie's normalized documentation model remain migration work rather than open\nproduct architecture.\n\n**Amended 2026-09-15.** `internal/infrastructure/servicediscovery/nats`\nimplements the `servicediscovery.ServiceRegistry` contract (Connect, Register,\nResolve, Watch) over NATS KV and is fully tested, but no production code\nconstructs it and no config surface exposes it. This is a deliberate deferral,\nnot an oversight: service-to-service addressing belongs to the runtime and\nprocess boundary work in section 6, which currently pins container addressing\nby injecting the daemon's startup NATS URL rather than resolving through a\nregistry. The scaffold is kept for that work; re-derive it only if the\naddressing model changes.\n\n### 8. Cutover sequence\n\nThe final migration plan must order:\n\n- compatibility adapters;\n- schema migrations and data backfills;\n- any required dual-read or transitional paths;\n- domain-by-domain package replacement;\n- production wiring changes;\n- legacy package removal;\n- architecture tests preventing dependency regression;\n- focused, integration, race, and full quality gates.\n\nEach legacy package requires objective removal criteria. No compatibility path\nmay become an indefinite second implementation.\n\n\n## Completion criteria\n\nMigration design is complete when:\n\n- every current package has one approved owner and destination;\n- every mutable record has one authoritative owner and migration path;\n- all external and cross-process contracts have versioned owners;\n- existing behaviour and data have explicit parity requirements;\n- process boundaries have evidence-backed justification;\n- the cutover order preserves a working application;\n- legacy deletion criteria and deterministic architecture tests are defined;\n- no unresolved decision requires an implementer to invent domain semantics.\n\nExact Go APIs, SQL statements, and mechanical refactoring steps MAY be finalized\nduring implementation when they do not change these semantics.\n\n## Recorded migration position (2026-07-29)\n\nRecovered 2026-08-09 from the pre-migration issue tracker. This section records\ndecisions the maintainer approved that the repository did not otherwise capture.\n\n**Completed:** `internal/infrastructure/eventbus/nats` (rebuilt, not relocated);\n`internal/infrastructure/configuration` (loading moved out of `internal/config`,\nwhich is now types-only and does no I/O); `internal/eventbus` (broker-neutral\ncontract); `internal/domain/workintake` (the first `internal/domain/` package,\nowning `TaskEnvelope`, the routing `Kind`, the label vocabulary, and task\nsubjects).\n\n**DECISION — private input DTO deferred (approved).**\n`configuration.Document.Config` is still `internal/config.Config` rather than the\nprivate per-domain input DTO this document mandates. Building an ~800-line\nparallel type tree while every consumer still reads `config.Config` would mean\ntwo shapes kept in sync by hand, with no consumer for the new one. It is named as\na field so the compromise is visible at each use site, with the deletion\ncondition recorded in that package's `doc.go`. Do this **with** the\n`internal/config` dissolution, not ahead of it.\n\n**Agreed next: `internal/app/{archied,agentworker}`.** `run()` in\n`cmd/archied/main.go` is ~460 lines, against `organisation.md`'s rule that\n`cmd/*` must not contain substantive wiring or act as a service locator. It is\nalso the last non-infrastructure holder of the NATS SDK, and it is where the\nconfig DTO translation will land, so it unblocks the dissolution above.\n\n**Method constraint.** Do not write a separate recorded package-review document\nper area when the decisions are already dictated by\n`dependencies-and-contracts.md`. That is the ceremony pattern this project\ndeliberately avoids.\n\n## OPEN — `[chat.*]` config section naming\n\n`configuration.md:160` already states the position: `ChatConfig` holds \"separate\nchannel instance settings... there is no chat-wide settings owner\". The config\nsection name has not followed, so a settings block named for a product surface\nowns what the architecture calls channel state.\n\nTwo candidates, neither settled: `[channel.*]` and `[messaging.channel.*]`. The\nsecond reads as a domain path, and no config section in this repo is currently\nnamed after a domain package, so adopting it sets a precedent for every other\nsection rather than fixing one name.\n\nBlocked on a maintainer decision, not on evidence. This is a rename with an\noperator-visible TOML break, so it should land with the `internal/config`\ndissolution recorded above rather than on its own.\n" }, { "url": "/dev/architecture/observability/", "title": "Task Observability", "section": "Architecture", - "body": "# Task Observability\n\n**Status:** Implemented. The per-run detail surface was added on 2026-09-18 and\nis described below.\n**Date:** 2026-08-09 (revised 2026-09-18)\n\n## Historical problem\n\narchied's own daemon logging (`internal/logging/`) was already solid: a\nrotating file sink, a live `Feed` the dashboard subscribes to, and a reader\nthe `/api/logs` endpoint serves from. Before this implementation, none of\nthat reached inside a task.\n\nA task's actual work happens in a task-scoped `archie-agent` container,\ndispatched as one complete workflow over NATS.\nBefore this design was implemented, five things destroyed the evidence of what\nthat container did:\n\n1. **Worker startup** constructed an stderr-only logger and never replaced its\n handler after discovering a dedicated boot task, so runtime records could\n not leave the container.\n2. **`internal/container/pool.go`** creates every task container with\n `AutoRemove: true`. The container — and its stderr — is gone at exit,\n success or failure alike.\n3. **`internal/domain/workflow/implement.go`, `StageBaselineGate`** captures\n `go build`/`go vet`/`go test` output via `cmd.CombinedOutput()`, then\n clips it to 200 characters into a `Warn` log line and discards the rest.\n The eventual park reason is built from `res.Status` alone — the actual\n compiler error is never persisted anywhere a human or Archie's own chat\n tools can read it back.\n4. **`agentexec.Result`** (`internal/agentexec/protocol.go`) carries only\n `Summary`/`Detail` — no step-by-step transcript (tool calls, durations,\n errors). There is no transcript to recover even in principle.\n5. **`agentexec.SubjectForSystem`** is documented as the return channel for\n exactly this — *\"log dumps, health, and PII warnings... the daemon reads\n these for observability and never forwards them\"* — but has no publisher\n and no consumer anywhere in the codebase except a subject-format\n assertion in `nats_test.go`. The designed return path was never wired.\n\nThe practical consequence: a task parks with\n`stage baseline: baseline red -- go build ./... fails and builder could not\nauto-fix (status: parked)` and there is no way — not from the dashboard, not\nfrom Telegram chat, not by SSH'ing into the deployment host — to see *why*\n`go build` failed. The container that knew is already gone.\n\nThis matters beyond debugging convenience: supported deployments can combine\na resident daemon with ephemeral Docker containers that run unattended for\nweeks between operator logins, and the system is\nintended to be usable by people other than the original operator. A system\nthat is only debuggable by an operator who happens to be watching `docker ps`\nin real time does not meet that bar.\n\n## Decision: NATS is the sole autonomous handoff\n\nThe agent image must reach NATS in every deployment, because the broker is the\nonly autonomous task handoff and the only channel carrying task results back\nto the daemon. `archied` has no workflow runner or host fallback. A container\nthat dies before establishing a NATS connection is a distinct, narrower\nfailure (connectivity/config) from the one this document addresses (a connected\ncontainer's own output being unrecoverable), and is out of scope here.\n\n## Implemented design\n\nThe key property: **logs leave the container while it is alive**, so its\ndeath at `AutoRemove` no longer matters. Persistence happens in the daemon\nprocess, which already owns durable state.\n\n### Sink: extend `internal/logging`, not a new package\n\n`internal/logging` already owns its on-disk format end to end per\n`organisation.md` (\"a package owns its own format end to end\"). Add a\ntask-scoped writer producing the same `logging.Entry` JSONL shape at\n`\u003cstate_dir\u003e/logs/tasks/\u003ctask_id\u003e/attempt-\u003cn\u003e.jsonl`. `logging.Tail`,\n`logging.Query`, and `logging.Components` operate on this unchanged;\n`internal/webui/api_logs.go` stays transport-only, as it does today.\n\n### Agent side: make the designed subject real\n\n`internal/app/agentworker` replaces the stderr-only `slog.Handler` with one\nthat tees to stderr (kept for container-level operator debugging when a\nshell is attached) and publishes each record to\n`agentexec.SubjectForSystem(taskID)`, which already exists for this purpose.\nThe application layer obtains only a narrow publisher from\n`internal/infrastructure/agenttransport/nats`; raw core-NATS types remain in\ninfrastructure. `internal/infrastructure/agentboot.TaskID` reads the task ID\nfrom `\u003cworktree\u003e/.git/task.json` per the container/pool.go trap documented in\n`CLAUDE.md`. Publishing is fire-and-forget: log shipping must never block or\nfail the run it is reporting on.\n\n### Daemon side\n\n`cmd/archied/main.go` registers `subscribeSystemLogs` once at startup on\n`agentexec.SubjectSystemWildcard`. It demultiplexes by task ID, appends to that task's\nsink, and publishes into the existing `logging.Feed` so live task output\nappears on the dashboard exactly like daemon output. The approved target is\nfor this composition to move to `internal/app/archied`; that package does not\nyet exist, so this document does not claim the migration is complete.\n\nThe daemon is also a writer in its own right for the decisions only it makes.\nEvery park records its reason into that attempt's sink (`daemon.recordPark`),\non the daemon side after the guarded transition lands and in\n`workflow.park` on the agent side, because \"why did this park?\" is the question\nthe log exists to answer and the container path cannot answer a run that never\nreached a container. The entry is written only once the park actually\nhappened, so a transition another actor won does not leave a log claiming an\noutcome the task board contradicts.\n\n### Gate output: stop discarding it\n\n`StageBaselineGate` and the equivalent TDD-stage gate runs write full\n`cmd.CombinedOutput()` to the task's sink under a `gate` component, and put a\nbounded tail (~2KB, not 200 bytes) into the park reason. This is the cheapest\nfix and independently valuable — it alone would have made the `go build`\nfailure diagnosable without any NATS plumbing.\n\n### Transcript\n\nStep-level events (tool name, argument digest, duration, error) are emitted\nas log entries on the same NATS system subject rather than added as new\nfields on `agentexec.Result`. This keeps the wire protocol\n(`internal/agentexec/protocol.go`) stable and the transcript lands in the\nsink automatically, with no separate delivery path to keep consistent.\n\n### Retention\n\nReuse the existing rotation (`MaxSizeMB`/`Keep`) semantics per task\ndirectory. Prune a task's log directory when `store.ArchiveTask` archives\nit, so task logs share the task's own\nlifecycle rather than accumulating independently.\n\n### Surfaces\n\n- `/api/tasks/{id}/logs`, built on `logging.Reader` the same way\n `/api/logs` is today.\n- A chat tool so Archie can answer \"why did task 4 park?\" directly from\n Telegram or the webui chat — this is the surface that actually matters\n for the deployment model in `CLAUDE.md`, where an operator may not be\n looking at a dashboard at all.\n- The per-attempt read surface described in the next section, on top of the\n same task-scoped stream.\n\n## Per-run task detail (attempts, stages, changes, config)\n\n**Status:** landed, together with the change set that carries this revision\n(workflow run `76f8e877`). The provenance half is in the tree: `events.Event.Attempt`,\nthe `changes_captured` and `config_captured` event kinds, and the two protobuf field\nadditions. So is the read half: the three per-attempt endpoints\n(`/api/tasks/{id}/attempts`, `/changes`, `/debug`) and the dashboard run-detail page\nthat consumes them. The decision record, the rejected alternatives and the honest\nlimits are `docs/prds/task-run-detail.md`.\n\nThe task log surface above answers *why did this park?*. This surface answers\n*what did this run actually do?*, per attempt.\n\n### Attempt attribution\n\nOne task can run many times, and the task's event stream is the durable record\nof all of them. `events.Event.Attempt` attributes each event to one run; `0`\nmeans **not attributable** — every event written before this change, plus the\ndeliberately task-agnostic producers. An unattributed event is reported as a\ncount and is never presented as an attempt.\n\nAttribution is a column, not a fold over `task_queued`/`task_retried`: a fold\nwould have to guess which side of a boundary an event belongs to across a daemon\nrestart, a stale-run recovery, or two interleaved attempts, and a guess rendered\nas structure is exactly the false fidelity this surface exists to avoid.\n\nThe column reaches an existing database through the presence-gated arm in\n`internal/store`'s `migrateTasks` — `eventsSchema` is `CREATE TABLE IF NOT\nEXISTS`, so on an existing database the migrator is the *only* thing that adds\nit. `PRAGMA user_version` is unchanged because the loop is presence-gated\nrather than version-gated, and nothing is backfilled.\n\n### Read surface\n\nThe routes are registered beside the existing task routes in\n`internal/webui`'s `registerTaskRoutes` and all resolve the task through the\nstore's `TaskByID`; a task that does not exist is a `404`, never a `200` with an\nempty body.\n\n- `GET /api/tasks/{id}/attempts` — every attempt with its stages inline, so the\n rail and the attempt selector can never disagree. Stages are ordered by\n occurrence within their attempt. An attempt with no stages is legitimate and\n renders as \"no stages recorded\", never as an unexplained empty rail.\n- `GET /api/tasks/{id}/changes?attempt=N` — the change captures for one\n attempt, ordered by capture time; more than one per attempt is normal (a\n mid-workflow commit such as TDD's `commit-repro`, then `commit-push`, then a\n final `open-pr` capture that carries the pull-request number).\n `found: false` means no capture was recorded for that attempt, which is\n **not** \"no files changed\"; a capture that was recorded but could not be read\n is reported as exactly that, and never as a diffstat of zeroes.\n- `GET /api/tasks/{id}` — unchanged: the raw event array, each event now\n carrying `attempt`. The per-attempt configuration is read from this response,\n selected by kind **and** attempt.\n- `GET /api/tasks/{id}/logs` — gains `stage` beside its existing filters. The\n match is exact and case-insensitive, and an entry that records no stage never\n matches (see `logging.Query.Stage`).\n- `GET /api/tasks/{id}/debug?attempt=N` — the stored task record and the task's\n events, deliberately unfiltered, so an operator can attribute everything the\n task did even when a panel has nothing to show.\n\nThe `attempt` parameter follows one rule on every route: absent or `0` selects\nthe task's current attempt, and the resolved attempt is echoed in the response.\n\n### Stage status\n\n`stage_start` and `stage_finish` are the only inputs, so the derivation is\nexplicit and each case is pinned by a test:\n\n| Events seen for the stage | Reported |\n| --- | --- |\n| `stage_finish` with `data.error` | `failed`, with the error text |\n| `stage_finish` with `data.interrupted` | `interrupted` |\n| `stage_finish`, neither | `ok` — \"returned without error\" |\n| `stage_start` unmatched, and this is the task's current in-flight attempt | `running` |\n| `stage_start` unmatched on any dead attempt | `interrupted` — the run ended without finishing it |\n| `stage_finish` with no matching start | the stage, with no start time invented |\n| no stage information at all for that attempt | `unknown` |\n\n**There is no exit code.** `stage_finish` carries a duration and an optional\nerror string and nothing else, so no stage can be shown as pass/fail and the\npage must not imply verification. A stage that finished cleanly and then parked\nstays `ok` on a failed task, which is why the attempt status is read together\nwith the task status rather than on its own.\n\n### Changed files\n\nThe change an attempt produced is read from the worktree at the moment it is\ncommitted or pushed, and persisted as a `changes_captured` durable event — plus\none final capture when `OpenPR` records the pull-request number, which is the\nonly point at which a capture can carry one. It has to be captured there: the\nworktree is deleted on merge, close and no-PR terminal states, a retry resets\nthe branch onto its base, and no forge contract exposes a changed-file read.\n`internal/domain/workflow/task/changes.go` owns the value types and the status\nstrings they persist.\n\n- One capture carries per-file `added`/`modified`/`deleted`/`renamed`/\n `typechange` status with added and deleted line counts, and totals.\n- Each capture names its schema (`archie/task-changes@1`, in `internal/events`)\n and a reader refuses a payload whose schema or key set it does not know,\n rather than rendering the zero value of every field it could not find. That\n refusal surfaces as \"a capture was recorded but could not be read\", which is\n a different statement from \"no capture was recorded\".\n- `base_sha` is the merge base the file list was diffed against, not the base\n branch's tip, so the pair describes one change set even when the base branch\n moved during the run.\n- `files` is capped; `totals` always covers the **full** set, and a truncated\n capture says so rather than silently dropping entries.\n- `binary` means \"no textual hunks\" — never a claim about the file's type on\n disk — and a path the repository recorded as a rename pair is reported as the\n repository recorded it.\n- Capture **degrades and never parks**: an implementation without the capture\n seam captures nothing, logs, and lets the run continue.\n\n### Per-attempt configuration\n\nThe effective configuration an attempt ran under is persisted as a once-per-\nattempt `config_captured` durable event carrying the schema constant\n`archie/task-config@1` and the non-secret task-runtime subset from\n`config.Config.ForTask()`. It is not stored in a table and has no endpoint of\nits own: the dashboard reads it out of the events response it already fetches.\nThe single-row `config_snapshot` table is left alone — it answers what the\nrunning configuration is now and is replaced on every publish.\n\nRedaction is proven positively by a test that builds a full configuration\ncontaining canary values and asserts none of them survive serialising, rather\nthan being inferred from the producer's contract. The timeline carries one\nhuman line for this kind; the document itself is rendered by the panel.\n\n**Trust boundary, stated honestly.** A task-scoped grant may insert events on\nits own task ID — that is how `tool_call` and `agent_finish` already work — so a\ntask could in principle write its own `config_captured` event. This widens no\ncapability, because the grant could already insert arbitrary events on itself,\nbut it does mean this provenance is exactly as trustworthy as the rest of that\ntask's event stream, and no more. It is not tamper-proofing and must never be\ndescribed as such.\n\n### The wire\n\nThe whole wire change is two fields on existing messages: `Event.attempt`\n(field 11) and `ReadTaskLogRequest.stage` (field 9) in\n`proto/state/v1/state.proto`. No RPC, no table and no configuration field was\nadded, so the ratified state-store contract in\n`docs/prds/state-store-contract.md` rev 2c is unchanged.\n\nVersion skew degrades instead of failing, and each degrade is a statement the\npage has to make rather than hide: an old server makes `attempt` read as `0` and\n`stage` match nothing, so the page says it cannot attribute events to attempts\ninstead of labelling the whole stream \"attempt 1\"; an old client sees\n`config_captured` as an unfamiliar kind, and the panel must then say it cannot\nread per-run configuration — a different claim from \"not captured for this\nattempt\".\n\n### Rules the surface keeps\n\n- \"No record\" and \"this deployment cannot read this\" stay different answers,\n for logs and for configuration alike.\n- No run-duration figure is shown. A run can spend days in `waiting_human`, so\n one number would understate wall time; per-stage durations only.\n- No duration is invented for a stage with no start; it reads \"not recorded\".\n- The stage filter is a narrowing, not a coverage claim: agent and tool output\n carries no stage and is never matched, and the pane says so beside the\n control.\n- An attempt predating the change shows \"not captured for this run\" for its\n configuration and its change capture, never an empty view that implies\n defaults applied or nothing changed.\n\n### Failure and rollback\n\nEvery part is additive. A capture or a configuration emit that cannot happen\nleaves the run alone — reporting must never park a task. Reverting the change\nset removes the endpoints and the page; the extra column and the extra event\nkinds are inert to the previous revision (SQLite ignores a column no statement\nnames, and an unfamiliar event kind already renders as unfamiliar), and there is\nno data transformation to undo.\n\n## Dependency direction\n\n`internal/logging` remains cross-cutting per `organisation.md` — it imports\nno domain, infrastructure, or app package, so extending it to be task-scoped\ndoes not change its classification. The workflow domain writes through the\ncross-cutting logging contract. On the agent side,\n`internal/app/agentworker` composes the narrow publisher supplied by\n`internal/infrastructure/agenttransport/nats`. On the daemon side,\n`cmd/archied/main.go` currently composes the NATS subscription with the task\nsink until the approved `internal/app/archied` migration is implemented.\n\n## Implementation sequence (completed)\n\n1. Gate-output capture (~30 lines, no NATS changes) — fixes the immediate\n symptom that prompted this document.\n2. Task log sink in `internal/logging`.\n3. Agent-side publisher on `SubjectForSystem`.\n4. Daemon-side subscriber + `logging.Feed` integration + retention on\n archive.\n5. `/api/tasks/{id}/logs` + chat tool.\n\nTracked as GitHub epic/subtasks per the issue list linked from this\ndocument's PR.\n" + "body": "# Task Observability\n\n**Status:** Implemented. The per-run detail surface was added on 2026-09-18 and\nis described below.\n**Date:** 2026-08-09 (revised 2026-09-18)\n\n## Historical problem\n\narchied's own daemon logging (`internal/logging/`) was already solid: a\nrotating file sink, a live `Feed` the dashboard subscribes to, and a reader\nthe `/api/logs` endpoint serves from. Before this implementation, none of\nthat reached inside a task.\n\nA task's actual work happens in a task-scoped `archie-agent` container,\ndispatched as one complete workflow over NATS.\nBefore this design was implemented, five things destroyed the evidence of what\nthat container did:\n\n1. **Worker startup** constructed an stderr-only logger and never replaced its\n handler after discovering a dedicated boot task, so runtime records could\n not leave the container.\n2. **`internal/container/pool.go`** creates every task container with\n `AutoRemove: true`. The container — and its stderr — is gone at exit,\n success or failure alike.\n3. **`internal/domain/workflow/implement.go`, `StageBaselineGate`** captures\n `go build`/`go vet`/`go test` output via `cmd.CombinedOutput()`, then\n clips it to 200 characters into a `Warn` log line and discards the rest.\n The eventual park reason is built from `res.Status` alone — the actual\n compiler error is never persisted anywhere a human or Archie's own chat\n tools can read it back.\n4. **`agentexec.Result`** (`internal/agentexec/protocol.go`) carries only\n `Summary`/`Detail` — no step-by-step transcript (tool calls, durations,\n errors). There is no transcript to recover even in principle.\n5. **`agentexec.SubjectForSystem`** is documented as the return channel for\n exactly this — *\"log dumps, health, and PII warnings... the daemon reads\n these for observability and never forwards them\"* — but has no publisher\n and no consumer anywhere in the codebase except a subject-format\n assertion in `nats_test.go`. The designed return path was never wired.\n\nThe practical consequence: a task parks with\n`stage baseline: baseline red -- go build ./... fails and builder could not\nauto-fix (status: parked)` and there is no way — not from the dashboard, not\nfrom Telegram chat, not by SSH'ing into the deployment host — to see *why*\n`go build` failed. The container that knew is already gone.\n\nThis matters beyond debugging convenience: supported deployments can combine\na resident daemon with ephemeral Docker containers that run unattended for\nweeks between operator logins, and the system is\nintended to be usable by people other than the original operator. A system\nthat is only debuggable by an operator who happens to be watching `docker ps`\nin real time does not meet that bar.\n\n## Decision: NATS is the sole autonomous handoff\n\nThe agent image must reach NATS in every deployment, because the broker is the\nonly autonomous task handoff and the only channel carrying task results back\nto the daemon. `archied` has no workflow runner or host fallback. A container\nthat dies before establishing a NATS connection is a distinct, narrower\nfailure (connectivity/config) from the one this document addresses (a connected\ncontainer's own output being unrecoverable), and is out of scope here.\n\n## Implemented design\n\nThe key property: **logs leave the container while it is alive**, so its\ndeath at `AutoRemove` no longer matters. Persistence happens in the daemon\nprocess, which already owns durable state.\n\n### Sink: extend `internal/logging`, not a new package\n\n`internal/logging` already owns its on-disk format end to end per\n`organisation.md` (\"a package owns its own format end to end\"). Add a\ntask-scoped writer producing the same `logging.Entry` JSONL shape at\n`\u003cstate_dir\u003e/logs/tasks/\u003ctask_id\u003e/attempt-\u003cn\u003e.jsonl`. `logging.Tail`,\n`logging.Query`, and `logging.Components` operate on this unchanged;\n`internal/webui/api_logs.go` stays transport-only, as it does today.\n\n### Agent side: make the designed subject real\n\n`internal/app/agentworker` replaces the stderr-only `slog.Handler` with one\nthat tees to stderr (kept for container-level operator debugging when a\nshell is attached) and publishes each record to\n`agentexec.SubjectForSystem(taskID)`, which already exists for this purpose.\nThe application layer obtains only a narrow publisher from\n`internal/infrastructure/agenttransport/nats`; raw core-NATS types remain in\ninfrastructure. `internal/infrastructure/agentboot.TaskID` reads the task ID\nfrom `\u003cworktree\u003e/.git/task.json` per the container/pool.go trap documented in\n`CLAUDE.md`. Publishing is fire-and-forget: log shipping must never block or\nfail the run it is reporting on.\n\n### Daemon side\n\n`cmd/archied/main.go` registers `subscribeSystemLogs` once at startup on\n`agentexec.SubjectSystemWildcard`. It demultiplexes by task ID, appends to that task's\nsink, and publishes into the existing `logging.Feed` so live task output\nappears on the dashboard exactly like daemon output. The approved target is\nfor this composition to move to `internal/app/archied`; that package does not\nyet exist, so this document does not claim the migration is complete.\n\nThe daemon is also a writer in its own right for the decisions only it makes.\nEvery park records its reason into that attempt's sink (`daemon.recordPark`),\non the daemon side after the guarded transition lands and in\n`workflow.park` on the agent side, because \"why did this park?\" is the question\nthe log exists to answer and the container path cannot answer a run that never\nreached a container. The entry is written only once the park actually\nhappened, so a transition another actor won does not leave a log claiming an\noutcome the task board contradicts.\n\n### Gate output: stop discarding it\n\n`StageBaselineGate` and the equivalent TDD-stage gate runs write full\n`cmd.CombinedOutput()` to the task's sink under a `gate` component, and put a\nbounded tail (~2KB, not 200 bytes) into the park reason. This is the cheapest\nfix and independently valuable — it alone would have made the `go build`\nfailure diagnosable without any NATS plumbing.\n\n### Transcript\n\nStep-level events (tool name, argument digest, duration, error) are emitted\nas log entries on the same NATS system subject rather than added as new\nfields on `agentexec.Result`. This keeps the wire protocol\n(`internal/agentexec/protocol.go`) stable and the transcript lands in the\nsink automatically, with no separate delivery path to keep consistent.\n\n### Retention\n\nReuse the existing rotation (`MaxSizeMB`/`Keep`) semantics per task\ndirectory. Prune a task's log directory when `store.ArchiveTask` archives\nit, so task logs share the task's own\nlifecycle rather than accumulating independently.\n\n### Surfaces\n\n- `/api/tasks/{id}/logs`, built on `logging.Reader` the same way\n `/api/logs` is today.\n- A chat tool so Archie can answer \"why did task 4 park?\" directly from\n Telegram or the webui chat — this is the surface that actually matters\n for the deployment model in `CLAUDE.md`, where an operator may not be\n looking at a dashboard at all.\n- The per-attempt read surface described in the next section, on top of the\n same task-scoped stream.\n\n## Per-run task detail (attempts, stages, changes, config)\n\n**Status:** landed, together with the change set that carries this revision\n(workflow run `76f8e877`). The provenance half is in the tree: `events.Event.Attempt`,\nthe `changes_captured` and `config_captured` event kinds, and the two protobuf field\nadditions. So is the read half: the three per-attempt endpoints\n(`/api/tasks/{id}/attempts`, `/changes`, `/debug`) and the dashboard run-detail page\nthat consumes them. The decision record, the rejected alternatives and the honest\nlimits are `docs/prds/task-run-detail.md`.\n\nThe task log surface above answers *why did this park?*. This surface answers\n*what did this run actually do?*, per attempt.\n\n### Attempt attribution\n\nOne task can run many times, and the task's event stream is the durable record\nof all of them. `events.Event.Attempt` attributes each event to one run; `0`\nmeans **not attributable** — every event written before this change, plus the\ndeliberately task-agnostic producers. An unattributed event is reported as a\ncount and is never presented as an attempt.\n\nAttribution is a column, not a fold over `task_queued`/`task_retried`: a fold\nwould have to guess which side of a boundary an event belongs to across a daemon\nrestart, a stale-run recovery, or two interleaved attempts, and a guess rendered\nas structure is exactly the false fidelity this surface exists to avoid.\n\nThe column is part of the events table in\n`internal/infrastructure/postgres/migrations/0001_state_store.sql`, and nothing\nis backfilled.\n\n### Read surface\n\nThe routes are registered beside the existing task routes in\n`internal/webui`'s `registerTaskRoutes` and all resolve the task through the\nstore's `TaskByID`; a task that does not exist is a `404`, never a `200` with an\nempty body.\n\n- `GET /api/tasks/{id}/attempts` — every attempt with its stages inline, so the\n rail and the attempt selector can never disagree. Stages are ordered by\n occurrence within their attempt. An attempt with no stages is legitimate and\n renders as \"no stages recorded\", never as an unexplained empty rail.\n- `GET /api/tasks/{id}/changes?attempt=N` — the change captures for one\n attempt, ordered by capture time; more than one per attempt is normal (a\n mid-workflow commit such as TDD's `commit-repro`, then `commit-push`, then a\n final `open-pr` capture that carries the pull-request number).\n `found: false` means no capture was recorded for that attempt, which is\n **not** \"no files changed\"; a capture that was recorded but could not be read\n is reported as exactly that, and never as a diffstat of zeroes.\n- `GET /api/tasks/{id}` — unchanged: the raw event array, each event now\n carrying `attempt`. The per-attempt configuration is read from this response,\n selected by kind **and** attempt.\n- `GET /api/tasks/{id}/logs` — gains `stage` beside its existing filters. The\n match is exact and case-insensitive, and an entry that records no stage never\n matches (see `logging.Query.Stage`).\n- `GET /api/tasks/{id}/debug?attempt=N` — the stored task record and the task's\n events, deliberately unfiltered, so an operator can attribute everything the\n task did even when a panel has nothing to show.\n\nThe `attempt` parameter follows one rule on every route: absent or `0` selects\nthe task's current attempt, and the resolved attempt is echoed in the response.\n\n### Stage status\n\n`stage_start` and `stage_finish` are the only inputs, so the derivation is\nexplicit and each case is pinned by a test:\n\n| Events seen for the stage | Reported |\n| --- | --- |\n| `stage_finish` with `data.error` | `failed`, with the error text |\n| `stage_finish` with `data.interrupted` | `interrupted` |\n| `stage_finish`, neither | `ok` — \"returned without error\" |\n| `stage_start` unmatched, and this is the task's current in-flight attempt | `running` |\n| `stage_start` unmatched on any dead attempt | `interrupted` — the run ended without finishing it |\n| `stage_finish` with no matching start | the stage, with no start time invented |\n| no stage information at all for that attempt | `unknown` |\n\n**There is no exit code.** `stage_finish` carries a duration and an optional\nerror string and nothing else, so no stage can be shown as pass/fail and the\npage must not imply verification. A stage that finished cleanly and then parked\nstays `ok` on a failed task, which is why the attempt status is read together\nwith the task status rather than on its own.\n\n### Changed files\n\nThe change an attempt produced is read from the worktree at the moment it is\ncommitted or pushed, and persisted as a `changes_captured` durable event — plus\none final capture when `OpenPR` records the pull-request number, which is the\nonly point at which a capture can carry one. It has to be captured there: the\nworktree is deleted on merge, close and no-PR terminal states, a retry resets\nthe branch onto its base, and no forge contract exposes a changed-file read.\n`internal/domain/workflow/task/changes.go` owns the value types and the status\nstrings they persist.\n\n- One capture carries per-file `added`/`modified`/`deleted`/`renamed`/\n `typechange` status with added and deleted line counts, and totals.\n- Each capture names its schema (`archie/task-changes@1`, in `internal/events`)\n and a reader refuses a payload whose schema or key set it does not know,\n rather than rendering the zero value of every field it could not find. That\n refusal surfaces as \"a capture was recorded but could not be read\", which is\n a different statement from \"no capture was recorded\".\n- `base_sha` is the merge base the file list was diffed against, not the base\n branch's tip, so the pair describes one change set even when the base branch\n moved during the run.\n- `files` is capped; `totals` always covers the **full** set, and a truncated\n capture says so rather than silently dropping entries.\n- `binary` means \"no textual hunks\" — never a claim about the file's type on\n disk — and a path the repository recorded as a rename pair is reported as the\n repository recorded it.\n- Capture **degrades and never parks**: an implementation without the capture\n seam captures nothing, logs, and lets the run continue.\n\n### Per-attempt configuration\n\nThe effective configuration an attempt ran under is persisted as a once-per-\nattempt `config_captured` durable event carrying the schema constant\n`archie/task-config@1` and the non-secret task-runtime subset from\n`config.Config.ForTask()`. It is not stored in a table and has no endpoint of\nits own: the dashboard reads it out of the events response it already fetches.\nThe single-row `config_snapshot` table is left alone — it answers what the\nrunning configuration is now and is replaced on every publish.\n\nRedaction is proven positively by a test that builds a full configuration\ncontaining canary values and asserts none of them survive serialising, rather\nthan being inferred from the producer's contract. The timeline carries one\nhuman line for this kind; the document itself is rendered by the panel.\n\n**Trust boundary, stated honestly.** A task-scoped grant may insert events on\nits own task ID — that is how `tool_call` and `agent_finish` already work — so a\ntask could in principle write its own `config_captured` event. This widens no\ncapability, because the grant could already insert arbitrary events on itself,\nbut it does mean this provenance is exactly as trustworthy as the rest of that\ntask's event stream, and no more. It is not tamper-proofing and must never be\ndescribed as such.\n\n### The wire\n\nThe whole wire change is two fields on existing messages: `Event.attempt`\n(field 11) and `ReadTaskLogRequest.stage` (field 9) in\n`proto/state/v1/state.proto`. No RPC, no table and no configuration field was\nadded, so the ratified state-store contract in\n`docs/prds/state-store-contract.md` rev 2c is unchanged.\n\nVersion skew degrades instead of failing, and each degrade is a statement the\npage has to make rather than hide: an old server makes `attempt` read as `0` and\n`stage` match nothing, so the page says it cannot attribute events to attempts\ninstead of labelling the whole stream \"attempt 1\"; an old client sees\n`config_captured` as an unfamiliar kind, and the panel must then say it cannot\nread per-run configuration — a different claim from \"not captured for this\nattempt\".\n\n### Rules the surface keeps\n\n- \"No record\" and \"this deployment cannot read this\" stay different answers,\n for logs and for configuration alike.\n- No run-duration figure is shown. A run can spend days in `waiting_human`, so\n one number would understate wall time; per-stage durations only.\n- No duration is invented for a stage with no start; it reads \"not recorded\".\n- The stage filter is a narrowing, not a coverage claim: agent and tool output\n carries no stage and is never matched, and the pane says so beside the\n control.\n- An attempt predating the change shows \"not captured for this run\" for its\n configuration and its change capture, never an empty view that implies\n defaults applied or nothing changed.\n\n### Failure and rollback\n\nEvery part is additive. A capture or a configuration emit that cannot happen\nleaves the run alone — reporting must never park a task. Reverting the change\nset removes the endpoints and the page; the extra column and the extra event\nkinds are inert to the previous revision (a column no statement names is\nignored, and an unfamiliar event kind already renders as unfamiliar), and there is\nno data transformation to undo.\n\n## Dependency direction\n\n`internal/logging` remains cross-cutting per `organisation.md` — it imports\nno domain, infrastructure, or app package, so extending it to be task-scoped\ndoes not change its classification. The workflow domain writes through the\ncross-cutting logging contract. On the agent side,\n`internal/app/agentworker` composes the narrow publisher supplied by\n`internal/infrastructure/agenttransport/nats`. On the daemon side,\n`cmd/archied/main.go` currently composes the NATS subscription with the task\nsink until the approved `internal/app/archied` migration is implemented.\n\n## Implementation sequence (completed)\n\n1. Gate-output capture (~30 lines, no NATS changes) — fixes the immediate\n symptom that prompted this document.\n2. Task log sink in `internal/logging`.\n3. Agent-side publisher on `SubjectForSystem`.\n4. Daemon-side subscriber + `logging.Feed` integration + retention on\n archive.\n5. `/api/tasks/{id}/logs` + chat tool.\n\nTracked as GitHub epic/subtasks per the issue list linked from this\ndocument's PR.\n" }, { "url": "/dev/architecture/organisation/", @@ -100,7 +100,7 @@ "url": "/dev/architecture/safe-change-and-recovery/", "title": "Safe Change and Recovery", "section": "Architecture", - "body": "# Safe Change and Recovery\n\n**Status:** Requirements approved; mechanics are under design. Database-backed settings are implemented through the control plane (2026-09): see the degrade/recovery section below, which records a gap rather than a solution. The full runtime-supervision protocol (bounded observation, versioned audit trail) remains under design.\n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie is built and improved by agents as well as humans. The architecture MUST\nmake a change easy to locate, understand, test, audit, extend, and reverse\nwithout requiring its author to understand unrelated parts of the application.\n\nSeparation of concerns is therefore an operational safety boundary, not only a\ncode-organisation preference.\n\n## Required outcomes\n\n- An agent can improve a focused feature without receiving authority over\n unrelated domains.\n- Every event, function, action, setting, and policy has an identifiable owner\n and provenance.\n- Source and runtime changes pass deterministic checks outside the actor making\n the change.\n- A rejected or unhealthy candidate cannot replace the last known working\n version.\n- Invalid user configuration does not terminate an otherwise healthy\n application.\n- Configuration and service failures are observable, attributed, and eligible\n for bounded remediation or rollback.\n- The application continues serving unaffected capabilities when one\n capability is misconfigured or unhealthy.\n- Recovery failure is reported and contained rather than hidden in an\n unbounded retry loop.\n\n## Source changes\n\nAgent-authored source changes occur through the task-execution boundary in an\nisolated workspace. The agent does not directly mutate the running\ninstallation.\n\nPromotion is controlled by deterministic repository-owned policy, including\ntests, generated-reference drift checks, protected paths, change scope, and\nother required gates. The actor proposing a change cannot waive these checks.\n\nRepository and domain separation SHOULD allow a task to load and modify only\nthe governing requirements and code for its declared scope.\n\n## Runtime changes\n\nConfiguration and other runtime changes are proposed as candidates rather than\nmutating the active state in place.\n\nThe affected domain owns:\n\n- the command describing the semantic change;\n- validation of its typed settings;\n- health signals that demonstrate correct operation;\n- domain-specific remediation and rollback consequences.\n\nShared policy supplies consistent evaluation, evidence, and audit mechanics.\nIdentity supplies actor attribution. Infrastructure supplies external\nconfiguration storage, service control, and other operational adapters.\n\n## Runtime change protocol\n\nLast-known-good preservation and candidate promotion are the universal runtime\nchange protocol.\n\n1. An identity proposes an immutable candidate version.\n2. Infrastructure parses it without changing active state.\n3. Each affected domain validates its typed settings.\n4. Shared policy evaluates cross-cutting constraints.\n5. Each affected capability prepares the candidate without destroying its\n active instance.\n6. Capability-specific health checks prove the candidate works.\n7. Runtime supervision atomically promotes the candidate.\n8. A bounded observation period watches for delayed failure.\n9. Failure restores the last-known-good version and records the reason,\n evidence, actor, and involved versions.\n10. Repeated remediation failure stops retrying, isolates the capability, and\n reports the fault.\n\nThe affected domain owns the semantic change command, typed validation, health\nsignals, and remediation consequences. Shared runtime supervision owns the\npromotion transaction. Infrastructure performs storage and service-control\noperations.\n\nThe protocol's exact Go contracts, storage model, atomicity boundary, and\nobservation rules require a focused runtime-supervision design pass.\n\n## Failure isolation\n\nInvalid input MUST leave the active valid configuration unchanged and produce a\ndiagnostic record.\n\nA capability that cannot start or reload SHOULD fail independently where its\nabsence does not violate a declared application invariant. Unaffected\ncapabilities continue operating.\n\nRemediation MUST be bounded, observable, and policy-driven. Recovery MUST NOT\nsilently oscillate between versions or retry forever.\n\n## Database settings: degrade paths and recovery (2026-09)\n\nThe control plane owns runtime-tunable settings and stores them in `archie.db`.\nThis replaced a dashboard overlay store that degraded to file config on any\nfailure and had three documented ways back. The replacement has none, and the\nposture is the opposite: **the daemon fails closed.**\n\n### Degrade paths\n\n| Failure | Boot behaviour | Where the operator sees it |\n| --- | --- | --- |\n| State Store unreachable | `boot.loadRuntimeConfig` returns the dial error and `archied` exits non-zero | stderr: `runtime settings unavailable` |\n| A stored resource fails validation | Same: `validate database settings` wraps the error and the daemon exits | Same |\n| SIGHUP reload with a bad file, or a control plane that cannot be read | Running config is kept; the reload records `last_error` / `last_error_at`. The reload re-applies the database layer, so it never publishes file values over the running database-owned ones | `/api/config` → `reload.last_error` → dashboard banner |\n\nFailing closed is deliberate for settings the database owns: falling back to the\nfile values would restore the second source of truth the control plane exists to\nremove. The cost is that a bad stored value stops the daemon starting, and the\nvalue can only be corrected through the daemon's own API.\n\n### Upgrading an install from SQLite\n\nThe State Store, the daemon and the Gateway serve from `database_url` only. An\ninstall that still holds the SQLite stores (`\u003cdb_path\u003e-tasks.sqlite`,\n`\u003cdb_path\u003e-eda.sqlite`, `\u003cdb_path\u003e-conversations.sqlite`) upgrades in three\nsteps:\n\n1. Stop every archie process.\n2. Run `archie-state-store import`. It moves all three stores into Postgres in\n one transaction, verifies the rows, and records completion. The SQLite files\n are read, never written.\n3. Start archie.\n\nEach serving process checks this at boot, after migrating the schema. With no\ncompletion record and a non-empty legacy file it refuses to start and names the\nfiles and the import command, rather than serve empty tables over existing\ndata. There is no override. With no legacy data it is a fresh install: it writes\nthe completion record itself, so a legacy file that appears later is ignored.\nA database that recorded a fresh start refuses a later import.\n\nThe Gateway holds a Postgres serve-ownership claim for its whole life, so a\nsecond Gateway against the same database refuses to start (its turn recovery\nwould fail the first one's in-flight turns). The State Store holds its own.\n\n### Recovery: offline commands on the State Store\n\nNothing skips the control plane at boot, and the only writer for a resource is\n`ControlPlaneService.Command` on a running State Store -- which is exactly what\nthe daemon fails closed without. `archie-state-store` therefore carries the\noffline recovery commands (`archie-core-6zw0`). They work on the task database\nfile directly -- the configured `db_path` with `-tasks.sqlite` appended, not\n`db_path` itself -- and need neither `archied` nor the Web UI running: `validate`\nreads the configuration file the daemon boots with, which is what makes its\nverdict the daemon's verdict.\n\n| Command | What it does | State Store |\n| --- | --- | --- |\n| `backup -db F -out F` | writes a transactionally consistent snapshot (`VACUUM INTO`), verified as a usable store before it is published | may be running |\n| `restore -db F -from F` | replaces the database with a snapshot and drops the replaced file's `-wal`/`-shm` | must be stopped |\n| `validate -db F [-config C]` | runs the checks that stop archied starting: the file is readable, uncorrupted and no newer than the binary; every stored resource decodes under the definition that owns it; and `configuration.Validate` accepts the settings the daemon would read -- the stored ones, or the seed the State Store writes for a kind the file does not hold yet -- layered onto `C`, the check the daemon reports as `validate database settings` | may be running |\n| `rollback -db F -kind K [-revision N]` | replays an earlier revision of a stored resource through the ordinary replace, recording the rollback as a new revision | must be stopped |\n\nThe two validation layers `validate` reports are not the same check, and it\nreports both rather than merging them: the write path's own decode is stricter\nabout shape (it rejects unknown fields boot's decode ignores), while boot is\nstricter about meaning (only boot checks `dispatch.trigger`, and a positive\n`poll_interval`). Either one refusing is a store somebody has to look at.\n\nNo rollback RPC exists and none is needed: a revision carries its own value, so\nrestoring one is the same replace the dashboard and the channels already use.\nThe offline command is that operation with the store stopped -- the one state in\nwhich the daemon's own API cannot serve it.\n\n\"Must be stopped\" is checked rather than assumed. The serving process takes an\nexclusive flock on `\u003cdb\u003e.lock` for its whole life, and `restore` and `rollback`\nrefuse with `task database is owned by another process` while another process\nholds it. The read-only commands -- `backup` and `validate` -- take no lock and\nare meant to run against a serving store: `backup` because the update installer\nruns inside the process it is replacing and cannot stop the State Store, and\n`validate` because an operator checks a running deployment with it. That\n`backup` exemption is only sound while backup never replaces the database, so\nnaming the database as its own `-out` is refused. `scripts/archie-update-install`\ntakes its pre-flight snapshot through this subcommand rather than through its\nown `sqlite3` shell, so the installer and an operator run one implementation.\n\n`rollback` is the surgical route back from a bad stored value; `restore` is the\nblunt one, and is also the way back from a schema migration a release cannot\ngrow out of.\n\n### The same commands on PostgreSQL\n\nWithout `-db`, each command works on the PostgreSQL database the configuration's\n`database_url` names (`internal/app/archied/state_store_recovery_postgres.go`,\n`internal/infrastructure/postgres/maintenance.go`). One database backs every\nservice, so the scope is always the whole database.\n\n- `backup -out F` runs `pg_dump --format=custom`, reads the archive back with\n `pg_restore --list` and only then replaces `F`. It takes no claim and runs\n against a serving database.\n- `restore -from F` is destructive and offline. It claims every service role's\n ownership lock (`state-store` and `gateway`) without waiting and refuses if\n any is held or the check cannot complete, then drops tables the snapshot does\n not hold and runs `pg_restore --clean --single-transaction`. Every write made\n after the snapshot is lost.\n- `validate` refuses a schema newer than the binary's highest migration, then\n runs the same two validation layers.\n- `rollback` holds the `state-store` claim for the replay.\n\nThere is no automatic schema rollback. goose `Down` drops tables, so it is data\nloss, not a rollback. `scripts/archie-update-install` refuses a daemon update\nit cannot snapshot first, and on a failed update `scripts/archie-update-watchdog`\nrolls the binaries back, leaves the database alone and prints the `restore`\ncommand for the pre-update snapshot. Whether to run it is the operator's call.\n\n## Gateway restart: two constraints learned the hard way\n\nTelegram `/restart` (`internal/channels/telegram/restart.go` plus the `Start`\nsupervisor loop). Recovered 2026-08-09 from the pre-migration issue tracker.\nBoth constraints must survive any refactor.\n\n**1. Deadlock.** The `/restart` handler runs *on* the bot instance the supervisor\nis about to stop. Tearing down inline from the handler deadlocks the restart it\njust asked for. So the handler sends on a **buffered** channel (`restartCh`,\ncapacity 1) and returns immediately; the supervisor loop in `Start` does the\nteardown. Duplicate requests hit the default branch and are dropped, not queued.\n\n**2. Lockout.** A failed config reload must **not** be fatal. It logs and\nrelaunches with the previous in-memory settings. If a bad config edit killed the\ngateway, it would also destroy the only means of fixing it remotely — which is\nthe entire reason `/restart` exists.\n\n**Scope is gateway-only by design.** The daemon keeps running so in-flight agent\ntasks survive. Exiting the process would also restart under a `unless-stopped`\npolicy, but would kill running work. `Reload` (supplied by `cmd/archied/main.go`)\nre-reads config from disk and refreshes **only** the token and `AllowedUserIDs`;\neverything else is wired into the daemon at construction and still needs a full\nrestart.\n\n**Authorisation is inherited:** `authorizedMessage` rejects non-allowlisted\nsenders before any handler runs, so all gateway commands are admin-only by\nconstruction. Do not add per-command auth without checking that invariant still\nholds.\n" + "body": "# Safe Change and Recovery\n\n**Status:** Requirements approved; mechanics are under design. Database-backed settings are implemented through the control plane (2026-09): see the degrade/recovery section below, which records a gap rather than a solution. The full runtime-supervision protocol (bounded observation, versioned audit trail) remains under design.\n**Date:** 2026-07-28 \n**Tracking issue:** [#73](https://github.com/samcharles93/archie-core/issues/73)\n\n## Purpose\n\nArchie is built and improved by agents as well as humans. The architecture MUST\nmake a change easy to locate, understand, test, audit, extend, and reverse\nwithout requiring its author to understand unrelated parts of the application.\n\nSeparation of concerns is therefore an operational safety boundary, not only a\ncode-organisation preference.\n\n## Required outcomes\n\n- An agent can improve a focused feature without receiving authority over\n unrelated domains.\n- Every event, function, action, setting, and policy has an identifiable owner\n and provenance.\n- Source and runtime changes pass deterministic checks outside the actor making\n the change.\n- A rejected or unhealthy candidate cannot replace the last known working\n version.\n- Invalid user configuration does not terminate an otherwise healthy\n application.\n- Configuration and service failures are observable, attributed, and eligible\n for bounded remediation or rollback.\n- The application continues serving unaffected capabilities when one\n capability is misconfigured or unhealthy.\n- Recovery failure is reported and contained rather than hidden in an\n unbounded retry loop.\n\n## Source changes\n\nAgent-authored source changes occur through the task-execution boundary in an\nisolated workspace. The agent does not directly mutate the running\ninstallation.\n\nPromotion is controlled by deterministic repository-owned policy, including\ntests, generated-reference drift checks, protected paths, change scope, and\nother required gates. The actor proposing a change cannot waive these checks.\n\nRepository and domain separation SHOULD allow a task to load and modify only\nthe governing requirements and code for its declared scope.\n\n## Runtime changes\n\nConfiguration and other runtime changes are proposed as candidates rather than\nmutating the active state in place.\n\nThe affected domain owns:\n\n- the command describing the semantic change;\n- validation of its typed settings;\n- health signals that demonstrate correct operation;\n- domain-specific remediation and rollback consequences.\n\nShared policy supplies consistent evaluation, evidence, and audit mechanics.\nIdentity supplies actor attribution. Infrastructure supplies external\nconfiguration storage, service control, and other operational adapters.\n\n## Runtime change protocol\n\nLast-known-good preservation and candidate promotion are the universal runtime\nchange protocol.\n\n1. An identity proposes an immutable candidate version.\n2. Infrastructure parses it without changing active state.\n3. Each affected domain validates its typed settings.\n4. Shared policy evaluates cross-cutting constraints.\n5. Each affected capability prepares the candidate without destroying its\n active instance.\n6. Capability-specific health checks prove the candidate works.\n7. Runtime supervision atomically promotes the candidate.\n8. A bounded observation period watches for delayed failure.\n9. Failure restores the last-known-good version and records the reason,\n evidence, actor, and involved versions.\n10. Repeated remediation failure stops retrying, isolates the capability, and\n reports the fault.\n\nThe affected domain owns the semantic change command, typed validation, health\nsignals, and remediation consequences. Shared runtime supervision owns the\npromotion transaction. Infrastructure performs storage and service-control\noperations.\n\nThe protocol's exact Go contracts, storage model, atomicity boundary, and\nobservation rules require a focused runtime-supervision design pass.\n\n## Failure isolation\n\nInvalid input MUST leave the active valid configuration unchanged and produce a\ndiagnostic record.\n\nA capability that cannot start or reload SHOULD fail independently where its\nabsence does not violate a declared application invariant. Unaffected\ncapabilities continue operating.\n\nRemediation MUST be bounded, observable, and policy-driven. Recovery MUST NOT\nsilently oscillate between versions or retry forever.\n\n## Database settings: degrade paths and recovery (2026-09)\n\nThe control plane owns runtime-tunable settings and stores them in the State\nStore's PostgreSQL database.\nThis replaced a dashboard overlay store that degraded to file config on any\nfailure and had three documented ways back. The replacement has none, and the\nposture is the opposite: **the daemon fails closed.**\n\n### Degrade paths\n\n| Failure | Boot behaviour | Where the operator sees it |\n| --- | --- | --- |\n| State Store unreachable | `boot.loadRuntimeConfig` returns the dial error and `archied` exits non-zero | stderr: `runtime settings unavailable` |\n| A stored resource fails validation | Same: `validate database settings` wraps the error and the daemon exits | Same |\n| SIGHUP reload with a bad file, or a control plane that cannot be read | Running config is kept; the reload records `last_error` / `last_error_at`. The reload re-applies the database layer, so it never publishes file values over the running database-owned ones | `/api/config` → `reload.last_error` → dashboard banner |\n\nFailing closed is deliberate for settings the database owns: falling back to the\nfile values would restore the second source of truth the control plane exists to\nremove. The cost is that a bad stored value stops the daemon starting, and the\nvalue can only be corrected through the daemon's own API.\n\n### Upgrading an install from SQLite\n\nThe State Store, the daemon and the Gateway serve from `database_url` only. An\ninstall that still holds the SQLite stores (`\u003cdb_path\u003e-tasks.sqlite`,\n`\u003cdb_path\u003e-eda.sqlite`, `\u003cdb_path\u003e-conversations.sqlite`) upgrades in three\nsteps:\n\n1. Stop every archie process.\n2. Run `archie-state-store import`. It moves all three stores into Postgres in\n one transaction, verifies the rows, and records completion. The SQLite files\n are read, never written.\n3. Start archie.\n\n`db_path` exists only to locate these files for the import and the boot gate\nbelow, and goes with the importer. The embedded NATS store, its endpoint file\nand task logs live under `state_dir` (default: the data home's `archie`\ndirectory). An install whose `db_path` sat outside the data home sets\n`state_dir` to that directory to keep them where they were.\n\nEach serving process checks this at boot, after migrating the schema. With no\ncompletion record and a non-empty legacy file it refuses to start and names the\nfiles and the import command, rather than serve empty tables over existing\ndata. There is no override. With no legacy data it is a fresh install: it writes\nthe completion record itself, so a legacy file that appears later is ignored.\nA database that recorded a fresh start refuses a later import.\n\nThe Gateway holds a Postgres serve-ownership claim for its whole life, so a\nsecond Gateway against the same database refuses to start (its turn recovery\nwould fail the first one's in-flight turns). The State Store holds its own.\n\n### Recovery: offline commands on the State Store\n\nNothing skips the control plane at boot, and the only writer for a resource is\n`ControlPlaneService.Command` on a running State Store -- which is exactly what\nthe daemon fails closed without. `archie-state-store` therefore carries the\noffline recovery commands (`archie-core-6zw0`). They work on the PostgreSQL\ndatabase the configuration's `database_url` names\n(`internal/app/archied/state_store_recovery_postgres.go`,\n`internal/infrastructure/postgres/maintenance.go`) and need neither `archied`\nnor the Web UI running: `validate` reads the configuration file the daemon\nboots with, which is what makes its verdict the daemon's verdict. One database\nbacks every service, so the scope is always the whole database.\n\n| Command | What it does | Serving processes |\n| --- | --- | --- |\n| `backup -out F` | runs `pg_dump --format=custom`, reads the archive back with `pg_restore --list` and only then replaces `F` | may be running |\n| `restore -from F` | drops tables the snapshot does not hold and runs `pg_restore --clean --single-transaction`; every write made after the snapshot is lost | every one must be stopped |\n| `validate [-config C]` | refuses a schema newer than the binary's highest migration; every stored resource decodes under the definition that owns it; and `configuration.Validate` accepts the settings the daemon would read -- the stored ones, or the seed the State Store writes for a kind the database does not hold yet -- layered onto `C`, the check the daemon reports as `validate database settings` | may be running |\n| `rollback -kind K [-revision N]` | replays an earlier revision of a stored resource through the ordinary replace, recording the rollback as a new revision | the State Store must be stopped |\n\nThe two validation layers `validate` reports are not the same check, and it\nreports both rather than merging them: the write path's own decode is stricter\nabout shape (it rejects unknown fields boot's decode ignores), while boot is\nstricter about meaning (only boot checks `dispatch.trigger`, and a positive\n`poll_interval`). Either one refusing is a store somebody has to look at.\n\nNo rollback RPC exists and none is needed: a revision carries its own value, so\nrestoring one is the same replace the dashboard and the channels already use.\nThe offline command is that operation with the store stopped -- the one state in\nwhich the daemon's own API cannot serve it.\n\n\"Must be stopped\" is checked rather than assumed. Each serving process holds a\nPostgres ownership lock for its whole life. `rollback` holds the `state-store`\nclaim for the replay. `restore` claims every service role's lock (`state-store`\nand `gateway`) without waiting and refuses if any is held or the check cannot\ncomplete. The read-only commands -- `backup` and `validate` -- take no claim and\nare meant to run against a serving database: `backup` because the update\ninstaller runs inside the process it is replacing and cannot stop the State\nStore, and `validate` because an operator checks a running deployment with it.\n`scripts/archie-update-install` takes its pre-flight snapshot through this\nsubcommand, so the installer and an operator run one implementation.\n\n`rollback` is the surgical route back from a bad stored value; `restore` is the\nblunt one, and is also the way back from a schema migration a release cannot\ngrow out of.\n\nThere is no automatic schema rollback. goose `Down` drops tables, so it is data\nloss, not a rollback. `scripts/archie-update-install` refuses a daemon update\nit cannot snapshot first, and on a failed update `scripts/archie-update-watchdog`\nrolls the binaries back, leaves the database alone and prints the `restore`\ncommand for the pre-update snapshot. Whether to run it is the operator's call.\n\n## Gateway restart: two constraints learned the hard way\n\nTelegram `/restart` (`internal/channels/telegram/restart.go` plus the `Start`\nsupervisor loop). Recovered 2026-08-09 from the pre-migration issue tracker.\nBoth constraints must survive any refactor.\n\n**1. Deadlock.** The `/restart` handler runs *on* the bot instance the supervisor\nis about to stop. Tearing down inline from the handler deadlocks the restart it\njust asked for. So the handler sends on a **buffered** channel (`restartCh`,\ncapacity 1) and returns immediately; the supervisor loop in `Start` does the\nteardown. Duplicate requests hit the default branch and are dropped, not queued.\n\n**2. Lockout.** A failed config reload must **not** be fatal. It logs and\nrelaunches with the previous in-memory settings. If a bad config edit killed the\ngateway, it would also destroy the only means of fixing it remotely — which is\nthe entire reason `/restart` exists.\n\n**Scope is gateway-only by design.** The daemon keeps running so in-flight agent\ntasks survive. Exiting the process would also restart under a `unless-stopped`\npolicy, but would kill running work. `Reload` (supplied by `cmd/archied/main.go`)\nre-reads config from disk and refreshes **only** the token and `AllowedUserIDs`;\neverything else is wired into the daemon at construction and still needs a full\nrestart.\n\n**Authorisation is inherited:** `authorizedMessage` rejects non-allowlisted\nsenders before any handler runs, so all gateway commands are admin-only by\nconstruction. Do not add per-command auth without checking that invariant still\nholds.\n" }, { "url": "/dev/prds/01-project-management/", @@ -142,7 +142,7 @@ "url": "/dev/prds/binding-secret-encryption/", "title": "Binding secret at-rest encryption -- decision", "section": "PRDs", - "body": "# Binding secret at-rest encryption -- decision\n\n**Status:** Approved\n**Date:** 2026-09-02\n**Beads issue:** `archie-core-t2db.7` (P2), parent `archie-core-t2db`\n\n## Problem\n\n`binding.Binding.Secret` (the per-source HMAC shared secret) is persisted as\nplaintext in the `bindings.secret TEXT` column. Anyone with read access to the\nSQLite file -- the same access level as `captured_events.body` and `tasks.body`\n-- sees sender-side shared secrets. A leaked database file leaks secrets even\nwhen the filesystem ACLs were correct at rest.\n\n## Decision\n\nEncrypt the `secret` column at rest with **AES-256-GCM** (stdlib\n`crypto/aes` + `crypto/cipher`, no new dependency), transparently inside the\nstore: encrypt on write, decrypt on read. The store is the data boundary; its\nconsumers (dispatch loop, HMAC verification) receive the plaintext secret\nexactly as and the webui already blanks `Secret` on every response.\n\n### Cipher parameters\n\n- **AEAD:** AES-256-GCM, 12-byte random nonce (`crypto/rand`), tag appended.\n- **Key length:** 32 bytes, derived as `SHA-256(resolved_material)`. Kept as\n SHA-256 (no Argon2id) because the source is a **32-byte high-entropy** value\n resolved through the secret registry, so re-deriving to 32 bytes merely\n re-shapes entropy rather than defending a weak passphrase.\n- **AAD:** a static domain separator, `[]byte(\"arcie-binding-secret\")`, so a\n ciphertext cannot be relocated to a different field/column and still\n authenticate. AAD is intentionally **not** row-bound: binding secrets are\n random HMAC keys and are never used as lookup keys, so cross-row swap is not\n a meaningful attack here.\n\n### Envelope format\n\n```\narcie-binding:v1:\u003ckeyFingerprint\u003e:\u003cbase64url(nonce || ciphertext || tag)\u003e\n```\n\n- `arcie-binding` -- domain marker\n- `v1` -- format version (future cipher/KDF change is a new version, old rows\n stay readable)\n- `\u003ckeyFingerprint\u003e` -- `hex(SHA-256(material))[:16]`, the identity of the key\n that sealed the row, enabling rotation without a one-shot rewrite\n- payload -- `base64url` of `nonce ‖ GCM.Seal(plaintext, aad)`\n\n### Key source (confirmed with maintainer)\n\nResolved through the existing secret registry (`internal/secret`), the\nestablished `SecretRef` pattern already used for `forge.webhook_secret`:\n\n```toml\n[bindings]\nencryption_key = { engine = \"env\", key = \"ARCHIE_BINDING_KEY\" }\n# Rotation: after swapping encryption_key to a new value, retain the previous\n# material so rows sealed under the old key still decrypt.\nprevious_encryption_keys = [ { engine = \"env\", key = \"ARCHIE_BINDING_KEY_OLD\" } ]\n```\n\nThe active key encrypts new writes; `previous_encryption_keys` are retained for\ndecrypt-only. Key identity is intrinsic to the material (the fingerprint), so\nno manual key IDs are configured.\n\n### Rotation story\n\nRotate by changing `encryption_key`, moving the old material into\n`previous_encryption_keys`. Rows sealed under the old key still decrypt via\ntheir embedded fingerprint; new writes use the new active key. Once no\nremaining row references an old fingerprint, remove it from the config. Losing\na fingerprint's material before all its rows are rewritten makes those rows\nunreadable -- the same operational rule as any key rotation, and the reason the\nold material must be retained.\n\n## Non-goals / deferred\n\n- **`captured_events.body` redaction hardening** was completed alongside this\n work (not deferred): `webhookguard.RedactPayload` now redacts an expanded set\n of secret key markers (compound names like `access_token`, `client_secret`,\n `signing_key`, `x_api_key`, …) as well as unambiguous value shapes (compact\n JWTs, PEM private-key blocks) under any key. `gateway.sensitiveParameterKey`\n was broadened identically to preserve the documented agreement. It remains\n best-effort per `docs/prds/webhook-intake-security.md` point 5 -- a sender\n that picks a name the heuristic misses is still captured.\n- **Key re-derivation on config reload** is a startup concern: the store holds\n the cipher for its lifetime, so a reload that changes `[bindings]` keys\n requires a daemon restart to take effect. Documented, accepted.\n\n## Wiring\n\n- `internal/config`: add `BindingsConfig` (`encryption_key SecretRef`,\n `previous_encryption_keys []SecretRef`) on `Config.Bindings`.\n- `internal/app/archied/bootstrap.go` `openStores`: resolve both refs through\n the already-built `secrets` registry, pass the material to `store.Open` via a\n new option `store.WithBindingCipher(cipher)`.\n- `internal/store`: add a `BindingCipher` interface (concrete AES-GCM impl in\n `binding_cipher.go`), a `bindingCipher` field on `Store`, and the `Open`\n option. **Nil cipher = plaintext**, so existing call sites and tests change\n nothing and legacy deployments keep current behaviour until they set a key.\n\n## Back-compat \u0026 tests\n\n- `store.Open` gains the option variadically; all current call sites compile\n unchanged.\n- Existing plaintext rows are unaffected until a cipher is configured; rows\n written before a cipher is installed remain plaintext and read as plaintext.\n- Tests: cipher round-trip (Encrypt/Decrypt), wrong-key and tampered-ciphertext\n integrity failure, envelope parse errors, and a store round-trip that asserts\n the persisted column is NOT the plaintext while `GetBinding` returns it.\n" + "body": "# Binding secret at-rest encryption -- decision\n\n**Status:** Approved\n**Date:** 2026-09-02\n**Beads issue:** `archie-core-t2db.7` (P2), parent `archie-core-t2db`\n\n## Problem\n\n\u003e Storage superseded: every store now lives in PostgreSQL behind\n\u003e `internal/domain/storecontract`, implemented in `internal/infrastructure/postgres`\n\u003e (`docs/prds/state-store-contract.md`). SQLite names below describe the\n\u003e original design.\n\n`binding.Binding.Secret` (the per-source HMAC shared secret) is persisted as\nplaintext in the `bindings.secret TEXT` column. Anyone with read access to the\nSQLite file -- the same access level as `captured_events.body` and `tasks.body`\n-- sees sender-side shared secrets. A leaked database file leaks secrets even\nwhen the filesystem ACLs were correct at rest.\n\n## Decision\n\nEncrypt the `secret` column at rest with **AES-256-GCM** (stdlib\n`crypto/aes` + `crypto/cipher`, no new dependency), transparently inside the\nstore: encrypt on write, decrypt on read. The store is the data boundary; its\nconsumers (dispatch loop, HMAC verification) receive the plaintext secret\nexactly as and the webui already blanks `Secret` on every response.\n\n### Cipher parameters\n\n- **AEAD:** AES-256-GCM, 12-byte random nonce (`crypto/rand`), tag appended.\n- **Key length:** 32 bytes, derived as `SHA-256(resolved_material)`. Kept as\n SHA-256 (no Argon2id) because the source is a **32-byte high-entropy** value\n resolved through the secret registry, so re-deriving to 32 bytes merely\n re-shapes entropy rather than defending a weak passphrase.\n- **AAD:** a static domain separator, `[]byte(\"arcie-binding-secret\")`, so a\n ciphertext cannot be relocated to a different field/column and still\n authenticate. AAD is intentionally **not** row-bound: binding secrets are\n random HMAC keys and are never used as lookup keys, so cross-row swap is not\n a meaningful attack here.\n\n### Envelope format\n\n```\narcie-binding:v1:\u003ckeyFingerprint\u003e:\u003cbase64url(nonce || ciphertext || tag)\u003e\n```\n\n- `arcie-binding` -- domain marker\n- `v1` -- format version (future cipher/KDF change is a new version, old rows\n stay readable)\n- `\u003ckeyFingerprint\u003e` -- `hex(SHA-256(material))[:16]`, the identity of the key\n that sealed the row, enabling rotation without a one-shot rewrite\n- payload -- `base64url` of `nonce ‖ GCM.Seal(plaintext, aad)`\n\n### Key source (confirmed with maintainer)\n\nResolved through the existing secret registry (`internal/secret`), the\nestablished `SecretRef` pattern already used for `forge.webhook_secret`:\n\n```toml\n[bindings]\nencryption_key = { engine = \"env\", key = \"ARCHIE_BINDING_KEY\" }\n# Rotation: after swapping encryption_key to a new value, retain the previous\n# material so rows sealed under the old key still decrypt.\nprevious_encryption_keys = [ { engine = \"env\", key = \"ARCHIE_BINDING_KEY_OLD\" } ]\n```\n\nThe active key encrypts new writes; `previous_encryption_keys` are retained for\ndecrypt-only. Key identity is intrinsic to the material (the fingerprint), so\nno manual key IDs are configured.\n\n### Rotation story\n\nRotate by changing `encryption_key`, moving the old material into\n`previous_encryption_keys`. Rows sealed under the old key still decrypt via\ntheir embedded fingerprint; new writes use the new active key. Once no\nremaining row references an old fingerprint, remove it from the config. Losing\na fingerprint's material before all its rows are rewritten makes those rows\nunreadable -- the same operational rule as any key rotation, and the reason the\nold material must be retained.\n\n## Non-goals / deferred\n\n- **`captured_events.body` redaction hardening** was completed alongside this\n work (not deferred): `webhookguard.RedactPayload` now redacts an expanded set\n of secret key markers (compound names like `access_token`, `client_secret`,\n `signing_key`, `x_api_key`, …) as well as unambiguous value shapes (compact\n JWTs, PEM private-key blocks) under any key. `gateway.sensitiveParameterKey`\n was broadened identically to preserve the documented agreement. It remains\n best-effort per `docs/prds/webhook-intake-security.md` point 5 -- a sender\n that picks a name the heuristic misses is still captured.\n- **Key re-derivation on config reload** is a startup concern: the store holds\n the cipher for its lifetime, so a reload that changes `[bindings]` keys\n requires a daemon restart to take effect. Documented, accepted.\n\n## Wiring\n\n- `internal/config`: add `BindingsConfig` (`encryption_key SecretRef`,\n `previous_encryption_keys []SecretRef`) on `Config.Bindings`.\n- `internal/app/archied/bootstrap.go` `openStores`: resolve both refs through\n the already-built `secrets` registry, pass the material to `store.Open` via a\n new option `store.WithBindingCipher(cipher)`.\n- `internal/infrastructure/bindingcipher`: the `BindingCipher` interface and\n concrete AES-GCM impl, consumed by the event-capture store; a `bindingCipher` field on `Store`, and the `Open`\n option. **Nil cipher = plaintext**, so existing call sites and tests change\n nothing and legacy deployments keep current behaviour until they set a key.\n\n## Back-compat \u0026 tests\n\n- `store.Open` gains the option variadically; all current call sites compile\n unchanged.\n- Existing plaintext rows are unaffected until a cipher is configured; rows\n written before a cipher is installed remain plaintext and read as plaintext.\n- Tests: cipher round-trip (Encrypt/Decrypt), wrong-key and tampered-ciphertext\n integrity failure, envelope parse errors, and a store round-trip that asserts\n the persisted column is NOT the plaintext while `GetBinding` returns it.\n" }, { "url": "/dev/prds/channel-forge-positions/", @@ -220,7 +220,7 @@ "url": "/dev/prds/dynamic-curators/", "title": "Dynamically-defined curators -- decision", "section": "PRDs", - "body": "# Dynamically-defined curators -- decision\n\n**Status:** Approved\n**Date:** 2026-09-12\n**GitHub epic:** [#789](https://github.com/samcharles93/archie-core/issues/789)\n**Parent epic (curator engine):** [#435](https://github.com/samcharles93/archie-core/issues/435) (closed)\n**Depends on:** nothing shipping; this resolves the open question that epic #435's\n\"registered and configured declaratively, not hardcoded\" criterion left open.\n\n## §0 Implementation status\n\nEvery claim in this document is mapped to what exists in the tree.\n\n| Claim | Status | Where it stands |\n|---|---|---|\n| Curator engine family: typed contract, owning registry, start/health/stop, narrow host access | **Implemented** | `internal/domain/curator/{contract,registry,runtime,registrar,activity,wake}.go` |\n| Curator declared shape (`Manifest`: interval, cooldown, on_input, tools, skills, memory engine, conversations, model) | **Implemented** | `internal/domain/curator/contract.go`; `Validate()` requires `Interval \u003e 0` |\n| Two reference curators run through the registry | **Implemented** | `internal/infrastructure/skillcurator`, `internal/infrastructure/sessioncurator` |\n| Curator definitions as persisted, API-editable data | **Aspirational -- zero code** | No table, no entity, no RPC, no config block |\n| Generic definition-driven engine that executes a declared tool set + free-form instructions | **Aspirational -- zero code** | Only the two code-registered curators exist |\n| `Registrar.Tools` (`ToolBuilder`) resolved and bound at registration | **Partial** | Interface declared (`registrar.go`); **no implementation anywhere**; `bootstrap.go` builds the `Registrar` with no `Tools` field |\n| A curator can declare tools at all | **Partial** | `Manifest.Tools` exists, but `registry.go` fails registration with `\"curator manifest: declares tools but the registrar has no ToolBuilder\"` |\n| Curator passes that actually reach a model with tools | **Cosmetic only** | `curator.ChatRequest` carries `Tools`/`MaxSteps`; `curatorLLMRunner` (`internal/app/archied/main.go`) drops both and returns `ChatResult{Text}` only |\n| Dynamic registration / removal (no restart) | **Aspirational -- zero code** | `Registry` has `Register`, no `Unregister`; `Runtime` has `Start`/`Nudge`/`Stop`, no `Add`/`Remove`; `Start()` snapshots membership |\n| Curator definition CRUD over REST | **Partial** | `GET /api/curators` read-only (`internal/webui/server.go`); no write routes |\n| Curator definition config | **Aspirational -- zero code** | No `[curator]` block in `internal/config/config.go` |\n| Store/API as definition authority | **Aspirational -- zero code** | No curator persistence of any kind |\n| No live-path defaults for interval/cooldown/tools/model/memory engine | **Partial** | `skillcurator.DefaultInterval` / `sessioncurator.DefaultInterval` are Go consts read at registration; `curator.NewRuntime(..., RuntimeConfig{})` uses code defaults for pass timeout and concurrency |\n| Idle curators are observable in logs | **Partial** | `runtime.go` logs nothing when `Check` reports not-due; the session-memory curator's hourly pass is the only curator activity ever logged |\n| Sampler family | **Implemented** | `internal/domain/sampling`, shipped; no curator consumes a `Sampler` yet |\n| Forge/issue reach from chat | **Implemented** (via `shell`) | No dedicated issue tool exists, but the chat agent's `shell` is unconfined (`chat.unrestricted_filesystem = true`), so `gh issue edit|comment|create|close` works. Dedicated tools (#161) are convenience, not a blocker |\n\n\u003e **Target state, not current state.** Everything below marked aspirational is\n\u003e design, not a description of the daemon as it runs.\n\n## Problem\n\nA curator exists only if a Go package implements `curator.CuratorEngine`\nand `bootstrap.go/1025` names it in a `Register` call. Nothing about a\ncurator is definable as data. Worse, no curator can declare a tool set at all:\nthe `ToolBuilder` interface has no implementation in the repo, so the registry\nrefuses (correctly, fail-closed) any curator that declares tools -- and the\ndaemon's model adapter drops the tools even if it were bound. The curator family\ntherefore has exactly one shape (cheap, tool-less maintenance pass) and its only\nextension point is compiling Go.\n\nSam's requirement is the opposite: *\"Curators should be dynamically created and\ndefined, in addition to being able to be defined in code/config/webui\"* and *\"No\nconfiguration should ever be hard-coded.\"* The motivating consumer is a\nproject/orchestrator curator -- declared as data, given forge issue tools, and\npointed at a repo to review, refine, and propose issues.\n\n## Decision\n\n**A curator definition is data: a persisted entity whose declared shape is\nexactly `curator.Manifest` plus identity, free-form instructions, and an enabled\nflag. One generic engine interprets any definition.** The shape is not a second\ninvention -- `Manifest` already encodes interval, cooldown, on-input, tools,\nskills, memory engine, conversations, and model, and `Validate()` already\nenforces what a valid shape means. The definition adds only what `Manifest`\ncannot express:\n\n| Definition field | Relationship to `Manifest` |\n|---|---|\n| `name`, `enabled` | Identity and lifecycle; the registry's key is the name |\n| `instructions` | **Free-form** system prompt for the generic engine -- the piece that makes a definition more than a knob set |\n| interval, cooldown, on_input, tools, skills, memory_engine, conversations, model | The existing `Manifest` fields verbatim; the definition **is** a `Manifest` |\n\nCode-registered engines keep `CuratorEngine` unchanged. A data-defined curator is\nserved by one registered engine whose `Manifest()` is the stored definition and\nwhose `Pass` composes `instructions` with the built tool set.\n\nThis is the curator domain's missing half, not a standalone tool: it lands in\n`internal/domain/curator` (entity + engine), `internal/store` (persistence),\n`internal/webui` (CRUD), and `internal/app/archied` (seed + apply wiring).\n\n## Definition sources and the merge rule\n\nThree sources, per Sam's requirement. **The store is authoritative.**\n\n| Source | Role | Collision behaviour |\n|---|---|---|\n| **Code** (a registered Go engine) | The escape hatch: a curator whose strategy is Go, not tools+instructions | Wins over data for its own name -- a data definition cannot silently replace a compiled engine; the store refuses the collision and logs it |\n| **Config** (`[[curators]]`) | **Seed data only**, merged at startup | Insert-if-absent. A store row of the same name is never overwritten, and deleting a store row is not undone by config on the next boot |\n| **WebUI/API** (the store) | **Authoritative** | Wins over config on a name collision; the only source of truth after first boot |\n\nConfig existing at all is what makes a definition reproducible on a fresh host;\nmaking it authoritative would make the API a lie, so it is seed data with a\none-way merge. The alternative -- config authoritative, store a cache -- would\nmean an API edit silently reverts on restart. Rejected.\n\n## How the generic engine executes a definition\n\nA pass is: resolve the declared tools, compose the prompt, run the model loop,\nrecord actions.\n\n1. **Tool resolution.** `Registrar.Tools.Build(ctx, declared)` returns exactly\n the declared set, never a broader registry. A declared name with no\n implementation fails the pass rather than silently running with fewer tools.\n This is the enforcement point, and it is why the `ToolBuilder` binding is\n load-bearing rather than a convenience: without an implementation, no curator\n can declare tools at all (the `registry.go` guard).\n2. **Tool-carrying model call.** `curator.ChatRequest` already carries\n `Tools []tools.ToolEntry` and `MaxSteps`; `curatorLLMRunner` must pass them\n into `core.GenerateOptions` (which has `Tools`, `ToolChoice`, `MaxSteps`, and\n already executes the tool loop) instead of dropping them. `ChatResult` is\n widened only as far as attributing the pass's actions requires.\n3. **Prompt.** The system prompt is the definition's `instructions` plus the\n declared tool summaries -- rendered from the same set that was built, never\n from a wider list. Each pass records `Action`s so activity stays attributable\n through the existing registry `activity` surface and events.\n\nA bespoke non-tool strategy (the skill curator's structural validation, the\nsession-memory curator's own extraction) remains a registered Go engine. That is\nthe escape hatch, not the mode: the mode is tools + instructions as data.\n\n## Dynamic registration and mutation\n\nA definition change must apply without a daemon restart, which the runtime\ncannot do (`Start()` snapshots membership into one goroutine per curator;\nno `Unregister`, no `Add`/`Remove`).\n\n- `Registry.Unregister(name)` -- valid while running, same failure isolation as\n `Register` (a rejected removal affects only that curator).\n- `Runtime.Add(engine)` / `Runtime.Remove(name)` -- `Remove` cancels the removed\n curator's in-flight pass context and stops its goroutine within a bounded\n wait; it never holds the registry mutex across a pass.\n- The app layer owns the apply sequence (stop old, register new, start) so the\n registry stays the membership authority and the runtime the scheduler.\n\nRemoval must not be able to orphan a loop, and a pass that ignores cancellation\nmust not block removal forever.\n\n## REST / WebUI surface\n\nMirror `internal/webui/api_binding.go`, not a new pattern:\n\n| Route | Handler shape |\n|---|---|\n| `GET /api/curators` | Existing read-only list, fields unchanged (dashboard keeps working) |\n| `POST /api/curators` | `handleCuratorCreate` |\n| `GET /api/curators/{id}` / `PATCH` / `DELETE` | `handleCuratorGet` / `Update` / `Delete` |\n\nAll mutations behind `authorizeTaskMutation`, same as bindings and mappings. A\nsuccessful mutation goes through the runtime apply path above, so it is live\nwithout a restart.\n\n## Sequencing: persistence and the State Store\n\nThe State Store is a standalone process that solely owns `archie.db`; the daemon\nand Gateway never open it (`TestOpenStoresNeverOwnsTaskDB` guards this), and\n`proto/state/v1/state.proto` is one gRPC service fronting every ratified store\ncontract. AGENTS.md forbids adding a Go interface method without a matching RPC.\nThat fixes the order:\n\n| Phase | Ships | Proto work |\n|---|---|---|\n| 1. Engine + config source | `ToolBuilder`, tool-carrying passes, generic engine, `[[curators]]`, idle logging | None -- nothing persisted yet |\n| 2. Runtime mutation | `Unregister`, `Add`/`Remove`, apply path | None -- in-memory membership |\n| 3. Persistence | definition entity + store schema, config seed (store authoritative), REST CRUD | **Same change**: request/response messages + RPCs, `staterpc.Client` façade, `mapError`/`unmapError`, and an update to `docs/prds/state-store-contract.md` |\n\nPhases 1 and 2 are genuinely independent of the wire boundary, so they are not\nblocked on it. Phase 3 is where the boundary is unavoidable: a definition the\nWebUI can edit must be reachable from the WebUI, and the WebUI reaches the store\nthrough the State Store, not by opening `archie.db`. Splitting the RPCs into a\nlater change would leave a table nothing can read -- rejected. Deferring the\nwhole phase is the same rejection, later.\n\n## No live-path constants\n\nNo default may be a live-path constant. Interval, cooldown, tools, model, memory\nengine, pass timeout, and concurrency are definition or config values. A\ndefinition missing `interval` is refused with a clear error, never defaulted at\npass time; `skillcurator.DefaultInterval` and `sessioncurator.DefaultInterval`\nstop being read at registration (the value becomes seed data), and\n`curator.NewRuntime(..., RuntimeConfig{})` stops relying on code defaults.\n\n## What this deliberately does NOT do\n\n- **No fixed catalogue of curator \"types\".** A definition is arbitrary tools +\n instructions, not a union of known kinds.\n- **No chat tool that mints curators.** A definition grants an LLM a tool set; a\n curator able to create curators is privilege escalation. Mutation is REST with\n the same authority as bindings -- not exposed on the chat tool surface.\n- **No dashboard SPA editor page.** The REST surface is the WebUI definition\n source; the dashboard can adopt it separately.\n- **No forge issue tools here.** This design only ensures a curator *can declare*\n tools. Dedicated issue tools (#161) are a convenience, not a prerequisite: the\n chat agent already reaches issues through the unconfined `shell` tool\n (`gh issue edit|comment|create|close`). See the correction on #789.\n- **No second authority.** No file-per-curator definition store, no parallel\n config reader.\n" + "body": "# Dynamically-defined curators -- decision\n\n**Status:** Approved\n**Date:** 2026-09-12\n**GitHub epic:** [#789](https://github.com/samcharles93/archie-core/issues/789)\n**Parent epic (curator engine):** [#435](https://github.com/samcharles93/archie-core/issues/435) (closed)\n**Depends on:** nothing shipping; this resolves the open question that epic #435's\n\"registered and configured declaratively, not hardcoded\" criterion left open.\n\n## §0 Implementation status\n\nEvery claim in this document is mapped to what exists in the tree.\n\n| Claim | Status | Where it stands |\n|---|---|---|\n| Curator engine family: typed contract, owning registry, start/health/stop, narrow host access | **Implemented** | `internal/domain/curator/{contract,registry,runtime,registrar,activity,wake}.go` |\n| Curator declared shape (`Manifest`: interval, cooldown, on_input, tools, skills, memory engine, conversations, model) | **Implemented** | `internal/domain/curator/contract.go`; `Validate()` requires `Interval \u003e 0` |\n| Two reference curators run through the registry | **Implemented** | `internal/infrastructure/skillcurator`, `internal/infrastructure/sessioncurator` |\n| Curator definitions as persisted, API-editable data | **Aspirational -- zero code** | No table, no entity, no RPC, no config block |\n| Generic definition-driven engine that executes a declared tool set + free-form instructions | **Aspirational -- zero code** | Only the two code-registered curators exist |\n| `Registrar.Tools` (`ToolBuilder`) resolved and bound at registration | **Partial** | Interface declared (`registrar.go`); **no implementation anywhere**; `bootstrap.go` builds the `Registrar` with no `Tools` field |\n| A curator can declare tools at all | **Partial** | `Manifest.Tools` exists, but `registry.go` fails registration with `\"curator manifest: declares tools but the registrar has no ToolBuilder\"` |\n| Curator passes that actually reach a model with tools | **Cosmetic only** | `curator.ChatRequest` carries `Tools`/`MaxSteps`; `curatorLLMRunner` (`internal/app/archied/main.go`) drops both and returns `ChatResult{Text}` only |\n| Dynamic registration / removal (no restart) | **Aspirational -- zero code** | `Registry` has `Register`, no `Unregister`; `Runtime` has `Start`/`Nudge`/`Stop`, no `Add`/`Remove`; `Start()` snapshots membership |\n| Curator definition CRUD over REST | **Partial** | `GET /api/curators` read-only (`internal/webui/server.go`); no write routes |\n| Curator definition config | **Aspirational -- zero code** | No `[curator]` block in `internal/config/config.go` |\n| Store/API as definition authority | **Aspirational -- zero code** | No curator persistence of any kind |\n| No live-path defaults for interval/cooldown/tools/model/memory engine | **Partial** | `skillcurator.DefaultInterval` / `sessioncurator.DefaultInterval` are Go consts read at registration; `curator.NewRuntime(..., RuntimeConfig{})` uses code defaults for pass timeout and concurrency |\n| Idle curators are observable in logs | **Partial** | `runtime.go` logs nothing when `Check` reports not-due; the session-memory curator's hourly pass is the only curator activity ever logged |\n| Sampler family | **Implemented** | `internal/domain/sampling`, shipped; no curator consumes a `Sampler` yet |\n| Forge/issue reach from chat | **Implemented** (via `shell`) | No dedicated issue tool exists, but the chat agent's `shell` is unconfined (`chat.unrestricted_filesystem = true`), so `gh issue edit|comment|create|close` works. Dedicated tools (#161) are convenience, not a blocker |\n\n\u003e **Target state, not current state.** Everything below marked aspirational is\n\u003e design, not a description of the daemon as it runs.\n\n## Problem\n\nA curator exists only if a Go package implements `curator.CuratorEngine`\nand `bootstrap.go/1025` names it in a `Register` call. Nothing about a\ncurator is definable as data. Worse, no curator can declare a tool set at all:\nthe `ToolBuilder` interface has no implementation in the repo, so the registry\nrefuses (correctly, fail-closed) any curator that declares tools -- and the\ndaemon's model adapter drops the tools even if it were bound. The curator family\ntherefore has exactly one shape (cheap, tool-less maintenance pass) and its only\nextension point is compiling Go.\n\nSam's requirement is the opposite: *\"Curators should be dynamically created and\ndefined, in addition to being able to be defined in code/config/webui\"* and *\"No\nconfiguration should ever be hard-coded.\"* The motivating consumer is a\nproject/orchestrator curator -- declared as data, given forge issue tools, and\npointed at a repo to review, refine, and propose issues.\n\n## Decision\n\n**A curator definition is data: a persisted entity whose declared shape is\nexactly `curator.Manifest` plus identity, free-form instructions, and an enabled\nflag. One generic engine interprets any definition.** The shape is not a second\ninvention -- `Manifest` already encodes interval, cooldown, on-input, tools,\nskills, memory engine, conversations, and model, and `Validate()` already\nenforces what a valid shape means. The definition adds only what `Manifest`\ncannot express:\n\n| Definition field | Relationship to `Manifest` |\n|---|---|\n| `name`, `enabled` | Identity and lifecycle; the registry's key is the name |\n| `instructions` | **Free-form** system prompt for the generic engine -- the piece that makes a definition more than a knob set |\n| interval, cooldown, on_input, tools, skills, memory_engine, conversations, model | The existing `Manifest` fields verbatim; the definition **is** a `Manifest` |\n\nCode-registered engines keep `CuratorEngine` unchanged. A data-defined curator is\nserved by one registered engine whose `Manifest()` is the stored definition and\nwhose `Pass` composes `instructions` with the built tool set.\n\nThis is the curator domain's missing half, not a standalone tool: it lands in\n`internal/domain/curator` (entity + engine), `internal/infrastructure/postgres` (persistence),\n`internal/webui` (CRUD), and `internal/app/archied` (seed + apply wiring).\n\n## Definition sources and the merge rule\n\nThree sources, per Sam's requirement. **The store is authoritative.**\n\n| Source | Role | Collision behaviour |\n|---|---|---|\n| **Code** (a registered Go engine) | The escape hatch: a curator whose strategy is Go, not tools+instructions | Wins over data for its own name -- a data definition cannot silently replace a compiled engine; the store refuses the collision and logs it |\n| **Config** (`[[curators]]`) | **Seed data only**, merged at startup | Insert-if-absent. A store row of the same name is never overwritten, and deleting a store row is not undone by config on the next boot |\n| **WebUI/API** (the store) | **Authoritative** | Wins over config on a name collision; the only source of truth after first boot |\n\nConfig existing at all is what makes a definition reproducible on a fresh host;\nmaking it authoritative would make the API a lie, so it is seed data with a\none-way merge. The alternative -- config authoritative, store a cache -- would\nmean an API edit silently reverts on restart. Rejected.\n\n## How the generic engine executes a definition\n\nA pass is: resolve the declared tools, compose the prompt, run the model loop,\nrecord actions.\n\n1. **Tool resolution.** `Registrar.Tools.Build(ctx, declared)` returns exactly\n the declared set, never a broader registry. A declared name with no\n implementation fails the pass rather than silently running with fewer tools.\n This is the enforcement point, and it is why the `ToolBuilder` binding is\n load-bearing rather than a convenience: without an implementation, no curator\n can declare tools at all (the `registry.go` guard).\n2. **Tool-carrying model call.** `curator.ChatRequest` already carries\n `Tools []tools.ToolEntry` and `MaxSteps`; `curatorLLMRunner` must pass them\n into `core.GenerateOptions` (which has `Tools`, `ToolChoice`, `MaxSteps`, and\n already executes the tool loop) instead of dropping them. `ChatResult` is\n widened only as far as attributing the pass's actions requires.\n3. **Prompt.** The system prompt is the definition's `instructions` plus the\n declared tool summaries -- rendered from the same set that was built, never\n from a wider list. Each pass records `Action`s so activity stays attributable\n through the existing registry `activity` surface and events.\n\nA bespoke non-tool strategy (the skill curator's structural validation, the\nsession-memory curator's own extraction) remains a registered Go engine. That is\nthe escape hatch, not the mode: the mode is tools + instructions as data.\n\n## Dynamic registration and mutation\n\nA definition change must apply without a daemon restart, which the runtime\ncannot do (`Start()` snapshots membership into one goroutine per curator;\nno `Unregister`, no `Add`/`Remove`).\n\n- `Registry.Unregister(name)` -- valid while running, same failure isolation as\n `Register` (a rejected removal affects only that curator).\n- `Runtime.Add(engine)` / `Runtime.Remove(name)` -- `Remove` cancels the removed\n curator's in-flight pass context and stops its goroutine within a bounded\n wait; it never holds the registry mutex across a pass.\n- The app layer owns the apply sequence (stop old, register new, start) so the\n registry stays the membership authority and the runtime the scheduler.\n\nRemoval must not be able to orphan a loop, and a pass that ignores cancellation\nmust not block removal forever.\n\n## REST / WebUI surface\n\nMirror `internal/webui/api_binding.go`, not a new pattern:\n\n| Route | Handler shape |\n|---|---|\n| `GET /api/curators` | Existing read-only list, fields unchanged (dashboard keeps working) |\n| `POST /api/curators` | `handleCuratorCreate` |\n| `GET /api/curators/{id}` / `PATCH` / `DELETE` | `handleCuratorGet` / `Update` / `Delete` |\n\nAll mutations behind `authorizeTaskMutation`, same as bindings and mappings. A\nsuccessful mutation goes through the runtime apply path above, so it is live\nwithout a restart.\n\n## Sequencing: persistence and the State Store\n\nThe State Store is a standalone process that solely owns the task database; the daemon\nand Gateway never open it (`TestOpenStoresNeverOwnsTaskDB` guards this), and\n`proto/state/v1/state.proto` is one gRPC service fronting every ratified store\ncontract. AGENTS.md forbids adding a Go interface method without a matching RPC.\nThat fixes the order:\n\n| Phase | Ships | Proto work |\n|---|---|---|\n| 1. Engine + config source | `ToolBuilder`, tool-carrying passes, generic engine, `[[curators]]`, idle logging | None -- nothing persisted yet |\n| 2. Runtime mutation | `Unregister`, `Add`/`Remove`, apply path | None -- in-memory membership |\n| 3. Persistence | definition entity + store schema, config seed (store authoritative), REST CRUD | **Same change**: request/response messages + RPCs, `staterpc.Client` façade, `mapError`/`unmapError`, and an update to `docs/prds/state-store-contract.md` |\n\nPhases 1 and 2 are genuinely independent of the wire boundary, so they are not\nblocked on it. Phase 3 is where the boundary is unavoidable: a definition the\nWebUI can edit must be reachable from the WebUI, and the WebUI reaches the store\nthrough the State Store, not by opening the database. Splitting the RPCs into a\nlater change would leave a table nothing can read -- rejected. Deferring the\nwhole phase is the same rejection, later.\n\n## No live-path constants\n\nNo default may be a live-path constant. Interval, cooldown, tools, model, memory\nengine, pass timeout, and concurrency are definition or config values. A\ndefinition missing `interval` is refused with a clear error, never defaulted at\npass time; `skillcurator.DefaultInterval` and `sessioncurator.DefaultInterval`\nstop being read at registration (the value becomes seed data), and\n`curator.NewRuntime(..., RuntimeConfig{})` stops relying on code defaults.\n\n## What this deliberately does NOT do\n\n- **No fixed catalogue of curator \"types\".** A definition is arbitrary tools +\n instructions, not a union of known kinds.\n- **No chat tool that mints curators.** A definition grants an LLM a tool set; a\n curator able to create curators is privilege escalation. Mutation is REST with\n the same authority as bindings -- not exposed on the chat tool surface.\n- **No dashboard SPA editor page.** The REST surface is the WebUI definition\n source; the dashboard can adopt it separately.\n- **No forge issue tools here.** This design only ensures a curator *can declare*\n tools. Dedicated issue tools (#161) are a convenience, not a prerequisite: the\n chat agent already reaches issues through the unconfined `shell` tool\n (`gh issue edit|comment|create|close`). See the correction on #789.\n- **No second authority.** No file-per-curator definition store, no parallel\n config reader.\n" }, { "url": "/dev/prds/dynamic-workflow-triage/", @@ -232,7 +232,7 @@ "url": "/dev/prds/eda-playbook-engine/", "title": "EDA playbook engine -- design", "section": "PRDs", - "body": "# EDA playbook engine -- design\n\n**Status:** Draft\n**Date:** 2026-09-03\n**Beads epic:** `archie-core-t2db`\n\n## Problem\n\n\"Which workflow runs for which trigger\" is hardcoded Go: a literal\nlabel vocabulary (`workintake.labelKinds`) and a literal label-\u003eworkflow map\n(`workflow/routing.go`), extended only by adding a Go case or shipping a\nskill whose `metadata.archie.workflow` overrides one *named* built-in\nworkflow. Extending the system to a new trigger, a new action, or a new\nevent source means a feature branch, a code change in a hardcoded\npath, and new Go tests -- for every addition. That is the barrier this\ndocument exists to remove.\n\nThe goal: triggers, the actions they run, and the\nconditions that connect them should be **drop-in and load**, not **drop-in\nand touch application code**. Yaegi-evaluated Go is the unit of imperative\n\"do a thing\" logic; YAML playbooks are pure orchestration data -- no logic,\njust \"on this event, with this data, call these actions, in this order,\nunder these conditions.\" The daemon's event coordinator doesn't know what a\n`bugfix` or `image-gen` action does, it resolves the name against a\nregistry and pipes typed data through.\n\nThis is explicitly **not** scoped to code-generation workflows. A playbook\naction can be any registered capability -- run a codegen workflow, send a\nmessage on a channel, open something on a forge, generate an image -- so\nlong as it's implemented behind one of the typed engine families below.\n\n## Relationship to existing decisions\n\n- `docs/prds/event-sources-and-reactions.md` ruled\n that reactions in the two epics it covers are **producer-only** (no veto,\n no mutation of in-flight work) and explicitly rejected a single generic\n `Source` interface as \"the untyped-hook shape the plugin engine rule\n exists to prevent.\" **This document extends that scope**: playbooks can\n dispatch to multiple typed action kinds, not just \"produce a\n `TaskEnvelope`.\" It does not relitigate producer-only-ness (still no\n veto/mutation of in-flight work here) and it does not introduce one\n generic interface -- see \"Typed positions, not one generic hook\" below,\n which is how it stays compliant with the plugin engine rule instead of\n reopening that document's #4.\n- `docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`\n governs every extension point\n introduced here. Every position below is a typed domain contract with an\n owning registry, not a generic callback map. This is a hard constraint,\n confirmed against the existing rule text before writing this doc.\n- Yaegi is already a proven in-process extension mechanism\n (`internal/plugin`, `internal/domain/workflow/wfeval/yaegi.go`,\n `internal/gate/gateeval/yaegi.go`, `internal/skill/plugin.go`) -- this\n design generalizes its existing role, it does not introduce a new\n execution mechanism.\n- `internal/domain/workflow/skillbuild.BuildRegistry` already composes a\n `workflow.Registry` from a skills catalog, plugin-defined overriding\n built-in. The playbook loader described here supersedes label routing\n specifically; it does not replace `skillbuild`'s workflow-composition role,\n it becomes an additional input the daemon composes at startup.\n- Webhook event intake is a separate design. This\n document defines the router each webhook-sourced event should be able to\n reach; it does not redesign webhook receipt itself.\n\n## Shape: four logic positions, each a typed engine family\n\nEvery entrypoint kind (mechanism, action, loop, schedule, route, trigger) is itself a schema emitter: defining a new kind produces the\ngenerated contract downstream tooling consumes. The kinds slot into a fixed\nhierarchy of logic positions:\n\n```\nWorkflow -\u003e Module -\u003e Channel (telegram/whatsapp/email/...)\n -\u003e Forge (GitHub/Gitea/GitLab/...)\n```\n\nEach position is its own typed engine family per the plugin engine rule:\nidentity + capability-specific operations, an owning `Registry`/`Manager`,\nexplicit lifecycle where the implementation owns resources, narrow host\naccess. Concretely:\n\n- **Workflow** -- already exists (`internal/domain/workflow`). A playbook\n action of kind `workflow` invokes a registered `workflow.Registry` entry\n by name, unchanged from the existing execution path.\n- **Module** -- the new general-purpose action position. A `Module` is a\n Yaegi-evaluated implementation of a generated interface (args in, typed\n result out) for one declared action kind (e.g. `image-gen`, `notify`,\n `open-pr-comment`). This is the position that makes actions \"not\n restricted to code-generation.\" Full design (schema-per-kind, the\n registry shape, the trust boundary, the recommended first slice):\n `docs/prds/module-position.md`.\n- **Channel** -- already exists as `channels.Channel` (telegram, email,\n webhook). A playbook action of kind `channel` sends through a named,\n already-registered channel. No new mechanism; this document just makes it\n addressable from a playbook.\n- **Forge** -- forge clients (GitHub, Gitea, GitLab) become an addressable\n action position the same way Channel is, so a playbook can say \"comment\n on this forge issue\" without that being a special case wired only into\n the poller.\n\n**Typed positions, not one generic hook.** There is no single `Action`\ninterface with an `any` payload. Each position keeps its own typed\ncontract; the playbook YAML names a position and an implementation, and the\nevent coordinator resolves that pair against the matching typed registry.\nThis is what keeps the design compliant with the plugin engine rule and\nwith `event-sources-and-reactions.md`'s rejection of a generic `Source`/\nhook interface -- the generality lives in the YAML data model (which\npositions exist, which named implementations are registered), not in a\nruntime `any`-typed dispatch path.\n\n## `go:generate`-driven schemas, from day one\n\nSchemas are not a future nicety bolted onto an ad hoc\nYAML shape. They are the struct+interface contracts for each logic\nposition, and they drive code generation from the start.\n\n- Each logic position (Workflow/Module/Channel/Forge) has a schema source\n (Go-annotated types, following the existing pattern elsewhere in the\n repo) from which `go:generate` emits the interface and struct scaffolding\n an implementation must satisfy.\n- A Yaegi module implements the generated interface for its position. The\n generated contract is what makes an implementation valid, not hand-review.\n- The schema is also what the linter and LSP (below) validate a playbook\n YAML against -- one source of truth for codegen, startup loading, and\n authoring-time tooling, so the three can't drift from each other.\n- Schema versioning is explicit per position, independent of a playbook's\n own workflow version (see collision handling below).\n\nThe generator itself is not designed here. The constraint is\nschema-first, `go:generate`-driven, per logic position.\n\n## Playbook YAML: pure orchestration data\n\nA playbook declares no embedded logic -- only:\n\n- **trigger**: the event predicate (e.g. forge issue assigned + labelled\n `bugfix`, a schedule tick, a webhook payload shape).\n- **actions**: an ordered list, each naming a logic position and a\n registered implementation, with typed argument data and optional\n conditions.\n- **data flow between actions**: results/facts from one action available to\n later actions in the same playbook run -- exact interpolation syntax is\n an open point, not decided here (see Open questions).\n\nPlaybooks live in a configured directory (repo/directory-scoped, per Sam's\n\"define repos, directories, playbooks\" framing) and are loaded at archied\nstartup, alongside the existing skills-catalog composition.\n\n## Dedup: detect and log, no arbitration mechanism\n\nNo runtime auto-resolution rule. An automatic \"pick a winner\" algorithm\n(e.g. by version) is itself a mechanism that can fail and need debugging\nwhen a collision happens -- the wrong kind of complexity to introduce for\na case that should just be visible and fixed by a human. Instead:\n\n- **At daemon startup:** if independently-sourced playbook directories\n declare a collision on the same trigger, that is a definition failure.\n The colliding definitions are dropped (not loaded) and the failure is\n logged plainly (which sources, which trigger) and reported back to the\n caller that requested the load -- not swallowed as a background log line\n only. The schema is the definition; a schema conflict is a reported\n failure, not something the daemon silently arbitrates.\n- **Linter**: a CI/dev-time tool, reading the same generated schemas, that\n flags duplicate or conflicting trigger definitions *within* a single\n maintained tree before merge.\n\n **Resolved** (`archie-core-t2db.12`). A standalone\n gopls-shaped binary (`cmd/archie-playbooks`, lint subcommand) validates\n playbook directories against the exact loaders the daemon uses\n (`LoadPlaybookDirs`/`LoadKindWorkflowsYAML`/`LoadLabelWorkflowsYAML`),\n exiting non-zero on any collision / malformed file / invalid binding.\n A finding about one binding key leads with the key's `file:line`, read\n from the `yaml.Node` the loader decodes; the linter agrees with runtime\n validation by construction, which is the load-bearing property. `-eda-dir`\n lints an `eda_playbook_dir` the same way, through `playbook.Load` with its\n CEL, action-id and args checks.\n Discoverable via `task lint:playbooks` or direct `go run\n ./cmd/archie-playbooks lint -dir ...`. The LSP/serve mode is a later\n entrypoint of the SAME binary, per the shared-package decision above.\n- **LSP**: same schema source again, feeding author-time hover/completion/\n inline errors while someone edits a playbook YAML in an editor. Kept\n explicitly in scope per Sam (not deferred), because it's the same schema\n artifact as the linter's, just a different consumer.\n\n `archie-playbooks serve` is the language server, over stdio. When a file is\n opened or saved it lints the file's saved directory with the loader the\n daemon runs for it (a file with a top-level `trigger` key is an EDA\n playbook, anything else a routing binding file) and publishes the findings\n for that file as error diagnostics, each on the line of the key, action,\n `when` or args key it names; a file-level finding sits on the first line.\n Unsaved edits are not validated. Hover and completion are later additions.\n\nThe startup log-and-refuse path exists for what the linter/LSP structurally\ncannot see -- sources composed together only at daemon startup, never\nchecked against each other by either tool. If the linter and LSP do the job\nthey're built for, a real collision reaching that startup path should be\nrare in practice, not a routine occurrence the design leans on.\n\n**Shared package, thin consumers -- not a bundled-vs-standalone choice.**\nThe validation logic (`LoadKindWorkflowsYAML`, `LoadLabelWorkflowsYAML`, the\ndirectory-merge from t2db.11) lives in `internal/domain/workflow`, a plain\nGo package with no knowledge of which binary calls it. This makes \"should\nthe linter/LSP be bundled into archied or a separate tool\" a non-question:\n- `archied` already imports the package directly and validates in-process\n at startup -- no subprocess, no external tool call.\n- An agent that needs to validate a playbook in-process can import the same\n package directly, for the same reason.\n- The standalone `cmd/` binary (t2db.12, gopls-shaped: one binary, a lint\n mode now, an LSP/serve mode later) exists only because CI and editors are\n out-of-process consumers by nature -- a CI step needs an exit code, an\n editor speaks LSP over a process boundary, neither can `import` a Go\n package. It is a thin wrapper per `organisation.md`'s `cmd/` rule, not a\n second implementation.\n\nFolding CLI/LSP protocol handling into `archied` itself would grow the\ndaemon into unrelated concerns, against the plugin-engine rule's spirit of\nnarrow capability families.\n\n## Execution-time gaps (both resolved)\n\nEverything above covers *loading* a playbook. Nothing above covers what\nhappens while one *runs*. Two gaps, both load-bearing enough to resolve\nbefore implementation, not defer:\n\n### 1. Mid-playbook action failure\n\n**Resolved** (`archie-core-t2db.18`). The default below\nis a committed decision, not a candidate list; changing it later would\nrequire its own decision. Gap 2 below is resolved by `archie-core-t2db.17`.\n\nA playbook is an ordered action list with data flowing between steps. If\naction 2 of 4 fails at runtime (an image-gen API times out, a forge call\n403s), the run needs a defined outcome. This\ncodebase already answers a structurally identical question for its\nexisting workflow engine, precisely enough to reuse rather than\nre-derive: `workflow.Run` (`internal/domain/workflow/workflow.go`)\ndraws a three-way distinction on a stage failure, and a playbook run\ndraws the same one on an action failure.\n\n**Decision: reuse `Run`'s three-way distinction directly.** The stated\ndefault is **stop-on-first-failure**: a real action error halts the\nplaybook run immediately and is reported, with no continue-to-next-action\nand no automatic retry or compensating follow-up. The three points below\nare the complete decision.\n\n1. **A real action error** (the action's own logic failed -- an API\n error, a validation failure inside the action) stops the run\n immediately. No later actions execute. The failure is reported the\n same way a load collision is reported -- dropped, logged plainly\n (which action and why), and returned to the caller, never swallowed\n as a background log line only (the Dedup section's drop-and-report\n rule). `Run` is the mechanical\n precedent for the stop itself: a stage error sets the park reason and\n returns without running further stages (`workflow.go`). This\n is stop-on-first-failure, matching the document's existing bias\n against clever automatic recovery, and it is now a stated decision\n rather than a nobody-chose default.\n2. **Interruption** (the daemon is shutting down mid-dispatch, `ctx.Err()\n != nil`) is explicitly **not** a failure. `Run` already treats this\n case specially -- \"Daemon shutdown is not a workflow failure. Leave\n the task running ...; parking here would publish a false failure and\n require manual intervention\" (`workflow.go`) -- and the same\n reasoning applies to a playbook run: an in-flight dispatch interrupted\n by restart must not be recorded as a broken playbook. What \"leave it\n alone for restart\" means for a playbook run (there is no `store.Task`\n row a playbook run inherently owns the way a workflow stage does) is\n an open implementation detail for whoever builds the coordinator, not\n decided here -- but the *semantic* (interruption ≠ failure) is decided.\n3. **A run that produces no outcome at all** (every action ran without\n error, but nothing was assigned to happen) is itself an error\n condition, matching `Run`'s own \"a workflow must end with an explicit\n outcome; not doing so is a definition bug, which still must not vanish\n silently\" (`workflow.go`). A playbook with actions that all ran\n but never reached a coherent terminal state is a definition bug in the\n playbook, reported as such.\n\n**Rollback:** explicitly **not attempted**, for exactly the reason the\ngap's own description named -- a posted forge comment is not revocable.\nThis is not a limitation to work around later; it is the correct\nsemantic. An operator who needs a corrective action after a failure\nwrites a *new* playbook/action for that, they do not get automatic\nundo.\n\n**This stays inside the producer-only rule.** Stop-on-first-failure is the\nrun deciding its own terminal state from its own action's error. Nothing\nalready in flight is blocked or altered; the run only declines to start\nactions that had not started yet, and it records the failure on its own\nunit the way `Run` sets a park reason and transitions the task to\n`StatusParked`. A future compensating action (\"if the notify action\nfailed, also log an incident\") would be producing new work, which the rule\nalready permits. No syntax for it is designed here.\n\n**J3 relationship, stated explicitly:** J3 (a CEL `when` condition\nerroring at eval time → treated as false → skip that action → run\ncontinues) is a **different case from an action failure** and is not\nrevisited by this resolution. A condition error means \"we couldn't\ndetermine whether to run this action,\" which this document already\nresolved as \"then don't, and move on.\" An action failure means \"we tried\nto run this action and it broke,\" which stops the run per point 1 above.\nThe two must not be conflated: a coordinator implementation should be\nable to point at one `when`-evaluation code path and one\naction-invocation code path and show they use different outcomes.\n\n### 2. Idempotency at execution time\n\n**Resolved** (`archie-core-t2db.17`, a\ndesign-investigation ticket: resolve, do not implement). The decision\nbelow is the answer; the keying scheme itself is separate follow-up work\nand is intentionally not built here. It **extends**\n`event-sources-and-reactions.md`'s existing contract -- it does not\ncontradict it (relationship argued at the end).\n\n**Problem (unchanged):** `event-sources-and-reactions.md` decided\nreactions ride `internal/eventbus`'s at-least-once delivery specifically\nso `PublishUnique`/idempotency keys absorb redelivery without duplicate\nside effects. This document inherits that for free only for\n`workflow`-kind actions, because `TaskEnvelope` already carries an\nidempotency key. It said nothing for `channel`, `forge`, or `module`\nactions with real side effects -- a posted comment, a sent message, a\nbilled image generation.\n\nOne correction to the problem statement, checked against the code rather\nthan assumed, because it changes what the answer must be:\n**`PublishUnique`'s dedup is not a consumer-redelivery guard at all.**\n`PublishUnique` only sets the JetStream `Nats-Msg-Id` header on the\noutgoing message (`internal/infrastructure/eventbus/nats/publisher.go`;\nkey constant `message.go`); JetStream suppresses *republished*\nmessages carrying a repeated key inside `Config.DedupWindow`\n(`client.go`, `Duplicates: cfg.DedupWindow`), default 2 minutes\n(`config.go`, `DefaultDedupWindow = 2 * time.Minute`). A consumer\nredelivery is the same *stored* message re-delivered on Nak or\nacknowledge-timeout (`internal/eventbus/eventbus.go`: \"a handler\nreturning an error causes redelivery\") -- not a republish; its\n`Nats-Msg-Id` is never re-firewalled, and even if it were, two minutes is\nnot an at-most-once guarantee for a side effect that must never repeat.\n**Conclusion: side-effecting action kinds need a consumer-side, durable\ndedup record; the bus cannot supply it.** That is precisely the situation\n`binding_dispatches` was built for -- so the decision below reuses its\nmechanism rather than designing a new one.\n\n#### Decision\n\n**Granularity: per-action-per-event -- one dedup record per\n(action, event) pair, not per playbook run.** Whole-run keying, the\nobvious alternative, has two crash windows: the run row is written when the run starts, so a crash mid-run\nleaves the run \"started\" and a redelivery skips *every remaining\naction* (work silently lost); a crash before the row write re-runs\n*every already-fired action* (duplicate side effects -- the exact harm\nthis gap exists to prevent). Per-action keys collapse both into one\nwindow and one behavior: an action's record is written immediately\nbefore that action's side effect is invoked, so an action fires iff its\nown record is absent, no fired action ever re-fires, and a redelivered\nevent resumes at the first unrecorded action. For the shipped\nsingle-action playbooks the two granularities are equivalent in effect;\nthe difference starts with multi-action runs -- where whole-run keying\nwould already be wrong.\n\n**Storage/lookup: a new durable ledger table `playbook_dispatches` in\n`internal/store`, copying `binding_dispatches`'s conventions exactly**\n(durable table, `INSERT OR IGNORE`, sentinel error, rows freed with\ntheir parent, survives daemon restarts -- `internal/store/bindings.go`):\n\n```sql\nCREATE TABLE playbook_dispatches (\n playbook_id TEXT NOT NULL,\n playbook_version TEXT NOT NULL,\n event_id TEXT NOT NULL,\n action_id TEXT NOT NULL,\n dispatched_at TEXT NOT NULL,\n PRIMARY KEY (playbook_id, playbook_version, event_id, action_id)\n);\n```\n\nWritten with the same `INSERT OR IGNORE` + sentinel-error convention\n(`ErrAlreadyDispatched`, `bindings.go`), so a duplicate is a no-op\nwrite rather than a constraint error, and the caller matches it with\n`errors.Is` the way the binding dispatch loop already does\n(`internal/daemon/daemon.go` -- \"another cycle already won this\nrace,\" not an error to surface). The record is consumed by the playbook\ncoordinator at dispatch time through a domain-side interface implemented\nby `internal/store` -- the same shape as `BindingDispatcher`\n(`internal/store/interface.go`) -- and the table lands in the\nstore's schema application (`internal/store/store.go`). No change\nto `internal/eventbus`; no per-message dedup state on the client.\n\n**Lifetime: no time-based expiry, and no automatic reclamation.** Rows are\nretained until an explicit `DeletePlaybookDispatches` for their playbook.\nRemoving a playbook file does not delete its rows: playbooks are read only\nat boot, so the only place to notice a removal is a boot-time reconcile,\nand a boot that finds the directory missing or unmounted loads as \"no\nplaybooks\" and would reclaim the whole ledger, letting redelivered events\nfire their side effects again. A binding's rows go with an explicit\noperator delete (`bindings.go`, same transaction); a file disappearing is\nnot an equally deliberate act. The at-most-once guarantee must not\nsilently decay, which is exactly the property that disqualifies\n`DedupWindow` above; table growth is bounded by distinct (playbook,\nevent, action) pairs and is accepted, the same tradeoff\n`binding_dispatches` already makes.\n\n**Redelivery behavior: skip the recorded action and continue the run.**\nA detected duplicate is not re-invoked, is not reported to the caller as\nan error, and is logged at debug level rather than as a warning -- this\nis the expected, correct behavior of at-least-once delivery, not an\nanomaly (same treatment as `ErrAlreadyDispatched` at\n`daemon.go`).\nThe run continues with the next action (or ends, for a single-action\nplaybook whose one action is recorded), which is what makes a redelivery\na resume rather than a dead restart.\n\n**Key derivation: a fixed composition of four structural identities,\ncomputed by the coordinator -- not a CEL expression.** The key is the\n`(playbook_id, playbook_version, event_id, action_id)` tuple above, where:\n\n- `playbook_id` is `Playbook.ID` -- the playbook file's path relative to\n its configured directory root, slash-normalized, unique within the load\n composition by construction (`internal/domain/eda/playbook`).\n- `playbook_version` is `Playbook.Version` -- a SHA-256 of the loaded\n file, recomputed on every load (`internal/domain/eda/playbook`),\n mirroring `Binding.Version`'s provenance-pinning purpose: the dispatch\n is pinned to the exact definition that was active when it fired. A\n changed playbook definition is a different key and may fire again for\n the same event -- deliberate, not a bug.\n- `event_id` is `DispatchInput.TaskID` (`playbook.go`), which\n carries the `TaskEnvelope.IdempotencyKey()` value\n (`\"archie:\" + owner/repo/number`,\n `internal/domain/workintake/envelope.go`). It is chosen over\n `store.Task.ID` because the trigger vocabulary is issue-level\n workintake label/kind, dispatched at discovery time (pollNATS, forge\n webhook receiver), where no task row exists yet (`playbook.go`\n comment; `playbook_test.go`: \"the value available at the\n discovery/dispatch point... NOT a store.Task.ID int64\"). Keying on the\n issue identity rather than the delivery source is the load-bearing\n property the existing contract already established for `TaskEnvelope`\n (`event-sources-and-reactions.md` question 4: a webhook-sourced\n envelope for the same issue produces the *same* idempotency key as a\n poll-discovered one; `internal/forge/webhook/receiver.go` --\n \"Idempotency is not this package's job\"); this resolution inherits that\n property rather than re-deciding it. Consequence, stated explicitly:\n same-issue re-discoveries (a re-label that re-triggers the same kind)\n are the *same event* by design -- the workintake trigger vocabulary is\n issue-granular, exactly as `TaskEnvelope`'s own key is. A future\n capture/binding-originated trigger type keys on `captured_events.id`\n instead; derivation is per-trigger-type, not one formula for all\n triggers (rule carried over unchanged from the draft).\n- `action_id` is the action's declared `id` -- J1's named ids, \"a\n required field on every action that later actions reference\", unique\n and stable-identifier-shaped, validated at load (the CEL resolution's\n J1 and its context table above in this document). For an action declaring no `id`\n (permitted by J1 while unreferenced), the key uses the action's 1-based\n position in the playbook: deterministic and stable for the\n single-action playbooks, and a later reorder is a file edit whose\n content hash changes `playbook_version` with it, so a positional\n fallback cannot silently collide with a prior definition.\n\n**No CEL in the key -- stated explicitly because the CEL resolution\nflagged this as gap 2's \"first real exercise\"** (\"CEL's read-only result\naccess makes gap 2's keying derivation readable\" -- the CEL resolution's\nclosing note). Readability\nis not the binding constraint; **determinism under redelivery is**, and\nCEL key expressions lose three ways. First, they are not load-checkable:\n`event` stays `dyn` by the resolved J4 (schema-by-example), so\na key expression over event fields cannot be type-checked at load,\nviolating the document's reject-at-load philosophy. Second, they can\nfail at dispatch (a missing field is a CEL error), and a key that cannot\nbe computed is an unfireable state -- neither \"skip\" nor \"fire\" is safe\nwithout losing or duplicating work. Third, the contract needs one\nderivation, and two authors writing different key expressions would\nfragment the ledger. The key is therefore structural by construction:\nevery component above is an immutable identity of the run's inputs, so\ntwo deliveries of the same event derive the same key with no evaluation.\nThe exercise's actual outcome is the derivation rule above; CEL remains\na *condition* surface (`when`), never a key surface. An action kind that\ngenuinely needs finer granularity than (action, event), such as \"per\ncomment, not per issue\", arrives with its own trigger type and its own\nevent identity, per the per-trigger-type rule.\n\n**Record-before-invoke -- one deliberate divergence from\n`binding_dispatches`, justified by the harm profile.** The binding\ndispatch loop enqueues the task first and records the ledger row after\n(`daemon.go`), accepting that racing cycles may both enqueue\n(\"the duplicate task remains queued... delete-on-races is its own can of\nworms\"); the selection query excludes already-dispatched captures\n(`bindings.go`), but the race window remains. That ordering is\ncorrect there because the duplicate artifact is an internal task row --\nrecoverable, low-harm. The playbook ledger's ordering is reversed on\npurpose: **the row is written and committed before the action's side\neffect is invoked** -- the `INSERT OR IGNORE` is the atomic gate, and\nthe lost-write loser skips without invoking. The cost is stated, not\nhidden: a crash between the row commit and the invoke loses that one\naction's side effect, repairable by hand -- the reverse direction of the\nsame window that would otherwise duplicate a posted comment or\nre-bill an image generation. For side effects whose duplication is\nexternally visible and not revocable, at-most-once is the required\ndirection; the binding system's at-least-once-tolerant ordering is not\nthe precedent for this ledger.\n\n**Gap-1 interplay, stated to keep the two mechanisms separate:** the\nledger answers one question per action -- has this action already fired\nfor this event? It does not answer whether the run continues, stops, or\nis re-attempted; that is gap 1's coordinator concern (a run-outcome\nrecord; gap 1 explicitly leaves \"what leave it alone for restart means\nfor a playbook run\" to whoever builds the coordinator). One consequence\nof record-before: an action that FAILED (gap 1's real-action-error case)\nhas a ledger row, so a later redelivery of the same event skips it --\nconsistent with gap 1's \"no automatic retry\" resolve (a corrective\naction is a *new* playbook/action, per gap 1's rollback paragraph). A\nfailed action is never re-attempted by the bus; if a run must be\nrecovered, an operator edits the playbook -- new version, new key --\nrather than fighting ledger state.\n\n**Relationship to `event-sources-and-reactions.md` -- extension, not\ncontradiction.** Keyed to that document's sections:\n\n1. Its decision 2 commits reactions to at-least-once delivery via\n `PublishUnique`'s \"without duplicate enqueue\" guarantee -- which, with\n the code facts above, is accurate for the *publish* path only:\n `PublishUnique` absorbs duplicate publishes of the same event (poll\n vs. webhook), which is what that decision was actually about. This\n document's own problem statement, taken literally, overstates it\n (\"keys absorb redelivery without duplicate side effects\") -- that\n overstatement is exactly what this gap exists to correct. The\n existing contract never claimed consumer redeliveries of\n side-effecting reactions were absorbed, so nothing it says contradicts\n a consumer-side ledger; this resolution pins down the boundary the\n sentence left implicit (publish-path dedup vs. consumer-path dedup)\n rather than changing either.\n2. Its decision 3 is untouched: the ledger is the run's own bookkeeping\n about its own actions, suppressing a re-fire and nothing else.\n3. Its decision 4 (no generic `Source` interface; `TaskEnvelope` is the\n typed contract, keyed on issue identity, not delivery source): the\n event half of the playbook key IS that same identity\n (`TaskEnvelope.IdempotencyKey()`), and the per-trigger-type rule\n means any future trigger type brings its own identity rather than a\n new generic one -- the playbook ledger introduces no second identity\n vocabulary and no untyped payload. The only new type in this\n resolution is the ledger's own value object, consumed through a\n narrow interface like `BindingDispatcher` -- the same shape the\n document's \"typed families\" rule (decision 1, plugin engine rule)\n already blesses.\n\n**Scope: this is the `channel`/`forge`/`module` decision, not a\n`workflow`-position change.** Workflow-kind actions keep their existing\nkeying claim unchanged -- the `TaskEnvelope` path that this document\nalready inherits for free (`daemon.go`, `PublishTask` publishes\nwith `task.IdempotencyKey()`), with its cross-poll/webhook property\nuntouched. The ledger gate applies to the side-effecting positions\n(channel, forge, module), which is why the closing statement below\nstill holds: the first slice (Module + loader + workflow-kind dispatch)\nwas buildable without this resolution, and channel/forge/multi-action\nplaybooks were not. A coordinator MAY implement one position-uniform\ngate as a simplification; what this resolution requires is only that\nthe side-effecting positions are gated.\n\n## Resolved questions\n\n1. **Data-flow/condition syntax.** Does this project want a small existing\n Go expression evaluator (there may be one already in the codebase worth\n checking, e.g. anything backing gate conditions), or a bespoke minimal\n grammar?\n\n **Resolved** (`archie-core-t2db.14`). CEL (`cel.dev/cel-go`, pinned) is\n the one mechanism for both an action's `when` condition and its `args`\n values. The trust boundary, evaluation context, load-time validation and\n the five judgment calls are in\n `docs/prds/playbook-expression-syntax.md`.\n\n2. **Playbook directory config field.** Where this lives in\n `internal/config` / `configuration.Document`, and whether it's one\n directory or a list (mirroring `SkillsDir`'s shape) -- follow existing\n config precedent, not invented fresh.\n\n **Resolved** (`archie-core-t2db.11`). The directory\n shape landed as `playbook_dirs` (a LIST of directories of\n `*.yaml`/`*.yml` binding files), loaded at startup as an additional\n input to the two single-file fields, which remain supported unchanged.\n A single `playbook_dir` would contradict\n supporting multiple independently-maintained playbook sources, which\n the Dedup section already assumes ('independently-sourced playbook\n directories collide'), so the field is a list. Cross-source collision\n (same key in two directories, or in a single-file field and a\n directory) is reported like an in-directory collision -- nothing is\n arbitrated by source precedence. This is what finally exercises the\n design doc's drop-and-report collision rule, since a single file\n cannot collide with itself. Repo-scoping (tying directories to\n `Config.Repos` owner/name) is explicitly deferred to a separate,\n later decision; org/tenant keying is out of scope (no auth/identity\n model).\n\n Decisions recorded from the label-vocabulary slice:\n - Arbitrary labels bind to registered workflow names via\n `LabelWorkflows` + `LoadLabelWorkflowsYAML`/`SetLabelWorkflows`\n (`internal/domain/workflow/routing.go`).\n - The closed `Kind`/NATS-subject set is untouched; the label map only\n extends binding authority for labels the kind layer does not own.\n - Collision rule: a label already owned by the\n kind set (bug/feature/bootstrap), an empty label, an empty workflow\n name, or a duplicate binding is a reported load failure -- dropped\n and logged, the error returned to the caller. Nothing is silently\n arbitrated by schema/version. This is the \"schema defines the\n accepted message; anything else is the caller's fault\" rule applied\n to the label layer.\n - Precedence in `Route()`: explicit `t.Workflow` → arbitrary-label\n binding → kind binding → triage → implement → default.\n3. **Trust boundary for Module Yaegi code.** The plugin engine rule's\n invariant 6 draws a line between operator-trusted in-process code and\n repository-supplied code that must run in a container.\n\n **Resolved** in `docs/prds/module-position.md`, \"Trust boundary\": a\n Module is operator-installed, in-process and daemon-privileged, the same\n tier as `PluginDir` and `SecretEngineDir`, never repository-supplied\n task code.\n4. **First implementation slice.** Recommend: Module position + the\n playbook loader + trigger-to-workflow dispatch only (subsuming the existing\n label routing) first, proving the schema-gen -\u003e Yaegi -\u003e playbook path\n end to end on the smallest useful case, before adding Channel/Forge\n action kinds or the linter/LSP.\n\n **Resolved** (`archie-core-t2db.13`, `.14`, `.15`). The Module position\n (registry + log kind, t2db.13), the CEL expression environment\n (t2db.14), and the playbook document + event coordinator with\n single-action workflow-kind dispatch (t2db.15) are shipped.\n\n **Resolved** (`archie-core-t2db.23`): the coordinator selects a real\n task's workflow. The wiring point is the daemon's definition pin\n (`Daemon.resolveWorkflowID`, consumed by `pinWorkflowFromCollection` in\n `internal/daemon`), not `pollNATS`/`publishTask`. Routing belongs to the\n pin because a workflow definition is a\n database row pinned before dispatch, which makes the pin the one place a\n task's workflow is chosen. Precedence there is explicit `Task.Workflow`\n (the waiting_human -\u003e approved requeue) → matching playbook →\n `workflow.ResolveWorkflowID`'s label binding → kind binding → triage →\n implement. A playbook naming an undefined workflow is a reported failure\n that parks the task rather than a silent fall-through to the binding's\n choice.\n\n `Store.Dispatch` returns the selected workflow *name* and the matching\n playbook's id and version, not a compiled `workflow.Workflow`: the\n production caller decides against the active definition collection, and\n the id and version are the first two components of the gap-2 idempotency\n key. The playbook package does not import `internal/domain/workflow`.\n\n A `when` condition at this point reads what the intake knows about the\n originating issue: `kind`, `labels`, `owner`, `repo`, `number`, `title`,\n `body`, `source`, built by `playbookInput` in `internal/daemon`.\n\n The trigger shape reuses the existing workintake kind/label vocabulary.\n The single-action hard boundary is superseded by\n `docs/prds/multi-action-playbooks.md` (two-shape rule); Module, Channel,\n and Forge positions still have no production dispatch path.\n## Standing constraint\n\nThis may be commercialized, and is to be discernible from other open-source\nevent-driven-automation tooling. No design decision follows from that yet;\nit bears on licensing or feature-gating questions about playbook authoring\nand the LSP when they arise.\n\n## Packages this touches (first slice)\n\n- New: a schema-generation package/convention (`go:generate` source +\n emitted contracts) for the Module position.\n- New: playbook loader + event coordinator (name/location TBD -- likely\n `internal/domain/eda` or similar, following the domain-layer rule: no\n infrastructure imports, no untyped hooks).\n- `internal/domain/workflow`: dispatch target for `workflow`-kind actions,\n unchanged in its own contracts.\n- `internal/domain/workintake`: `labelKinds`/routing becomes one playbook\n among others once the loader subsumes it; existing Go fallback stays for\n backward compatibility until playbooks fully replace it (mirrors the\n `dynamic-workflow-triage.md` precedent of a two-line fallback change, not\n a rip-and-replace).\n" + "body": "# EDA playbook engine -- design\n\n**Status:** Draft\n**Date:** 2026-09-03\n**Beads epic:** `archie-core-t2db`\n\n## Problem\n\n\"Which workflow runs for which trigger\" is hardcoded Go: a literal\nlabel vocabulary (`workintake.labelKinds`) and a literal label-\u003eworkflow map\n(`workflow/routing.go`), extended only by adding a Go case or shipping a\nskill whose `metadata.archie.workflow` overrides one *named* built-in\nworkflow. Extending the system to a new trigger, a new action, or a new\nevent source means a feature branch, a code change in a hardcoded\npath, and new Go tests -- for every addition. That is the barrier this\ndocument exists to remove.\n\nThe goal: triggers, the actions they run, and the\nconditions that connect them should be **drop-in and load**, not **drop-in\nand touch application code**. Yaegi-evaluated Go is the unit of imperative\n\"do a thing\" logic; YAML playbooks are pure orchestration data -- no logic,\njust \"on this event, with this data, call these actions, in this order,\nunder these conditions.\" The daemon's event coordinator doesn't know what a\n`bugfix` or `image-gen` action does, it resolves the name against a\nregistry and pipes typed data through.\n\nThis is explicitly **not** scoped to code-generation workflows. A playbook\naction can be any registered capability -- run a codegen workflow, send a\nmessage on a channel, open something on a forge, generate an image -- so\nlong as it's implemented behind one of the typed engine families below.\n\n## Relationship to existing decisions\n\n- `docs/prds/event-sources-and-reactions.md` ruled\n that reactions in the two epics it covers are **producer-only** (no veto,\n no mutation of in-flight work) and explicitly rejected a single generic\n `Source` interface as \"the untyped-hook shape the plugin engine rule\n exists to prevent.\" **This document extends that scope**: playbooks can\n dispatch to multiple typed action kinds, not just \"produce a\n `TaskEnvelope`.\" It does not relitigate producer-only-ness (still no\n veto/mutation of in-flight work here) and it does not introduce one\n generic interface -- see \"Typed positions, not one generic hook\" below,\n which is how it stays compliant with the plugin engine rule instead of\n reopening that document's #4.\n- `docs/architecture/plugins-and-extensions.md#plugin-engine-rule-strict`\n governs every extension point\n introduced here. Every position below is a typed domain contract with an\n owning registry, not a generic callback map. This is a hard constraint,\n confirmed against the existing rule text before writing this doc.\n- Yaegi is already a proven in-process extension mechanism\n (`internal/plugin`, `internal/domain/workflow/wfeval/yaegi.go`,\n `internal/gate/gateeval/yaegi.go`, `internal/skill/plugin.go`) -- this\n design generalizes its existing role, it does not introduce a new\n execution mechanism.\n- `internal/domain/workflow/skillbuild.BuildRegistry` already composes a\n `workflow.Registry` from a skills catalog, plugin-defined overriding\n built-in. The playbook loader described here supersedes label routing\n specifically; it does not replace `skillbuild`'s workflow-composition role,\n it becomes an additional input the daemon composes at startup.\n- Webhook event intake is a separate design. This\n document defines the router each webhook-sourced event should be able to\n reach; it does not redesign webhook receipt itself.\n\n## Shape: four logic positions, each a typed engine family\n\nEvery entrypoint kind (mechanism, action, loop, schedule, route, trigger) is itself a schema emitter: defining a new kind produces the\ngenerated contract downstream tooling consumes. The kinds slot into a fixed\nhierarchy of logic positions:\n\n```\nWorkflow -\u003e Module -\u003e Channel (telegram/whatsapp/email/...)\n -\u003e Forge (GitHub/Gitea/GitLab/...)\n```\n\nEach position is its own typed engine family per the plugin engine rule:\nidentity + capability-specific operations, an owning `Registry`/`Manager`,\nexplicit lifecycle where the implementation owns resources, narrow host\naccess. Concretely:\n\n- **Workflow** -- already exists (`internal/domain/workflow`). A playbook\n action of kind `workflow` invokes a registered `workflow.Registry` entry\n by name, unchanged from the existing execution path.\n- **Module** -- the new general-purpose action position. A `Module` is a\n Yaegi-evaluated implementation of a generated interface (args in, typed\n result out) for one declared action kind (e.g. `image-gen`, `notify`,\n `open-pr-comment`). This is the position that makes actions \"not\n restricted to code-generation.\" Full design (schema-per-kind, the\n registry shape, the trust boundary, the recommended first slice):\n `docs/prds/module-position.md`.\n- **Channel** -- already exists as `channels.Channel` (telegram, email,\n webhook). A playbook action of kind `channel` sends through a named,\n already-registered channel. No new mechanism; this document just makes it\n addressable from a playbook.\n- **Forge** -- forge clients (GitHub, Gitea, GitLab) become an addressable\n action position the same way Channel is, so a playbook can say \"comment\n on this forge issue\" without that being a special case wired only into\n the poller.\n\n**Typed positions, not one generic hook.** There is no single `Action`\ninterface with an `any` payload. Each position keeps its own typed\ncontract; the playbook YAML names a position and an implementation, and the\nevent coordinator resolves that pair against the matching typed registry.\nThis is what keeps the design compliant with the plugin engine rule and\nwith `event-sources-and-reactions.md`'s rejection of a generic `Source`/\nhook interface -- the generality lives in the YAML data model (which\npositions exist, which named implementations are registered), not in a\nruntime `any`-typed dispatch path.\n\n## `go:generate`-driven schemas, from day one\n\nSchemas are not a future nicety bolted onto an ad hoc\nYAML shape. They are the struct+interface contracts for each logic\nposition, and they drive code generation from the start.\n\n- Each logic position (Workflow/Module/Channel/Forge) has a schema source\n (Go-annotated types, following the existing pattern elsewhere in the\n repo) from which `go:generate` emits the interface and struct scaffolding\n an implementation must satisfy.\n- A Yaegi module implements the generated interface for its position. The\n generated contract is what makes an implementation valid, not hand-review.\n- The schema is also what the linter and LSP (below) validate a playbook\n YAML against -- one source of truth for codegen, startup loading, and\n authoring-time tooling, so the three can't drift from each other.\n- Schema versioning is explicit per position, independent of a playbook's\n own workflow version (see collision handling below).\n\nThe generator itself is not designed here. The constraint is\nschema-first, `go:generate`-driven, per logic position.\n\n## Playbook YAML: pure orchestration data\n\nA playbook declares no embedded logic -- only:\n\n- **trigger**: the event predicate (e.g. forge issue assigned + labelled\n `bugfix`, a schedule tick, a webhook payload shape).\n- **actions**: an ordered list, each naming a logic position and a\n registered implementation, with typed argument data and optional\n conditions.\n- **data flow between actions**: results/facts from one action available to\n later actions in the same playbook run -- exact interpolation syntax is\n an open point, not decided here (see Open questions).\n\nPlaybooks live in a configured directory (repo/directory-scoped, per Sam's\n\"define repos, directories, playbooks\" framing) and are loaded at archied\nstartup, alongside the existing skills-catalog composition.\n\n## Dedup: detect and log, no arbitration mechanism\n\nNo runtime auto-resolution rule. An automatic \"pick a winner\" algorithm\n(e.g. by version) is itself a mechanism that can fail and need debugging\nwhen a collision happens -- the wrong kind of complexity to introduce for\na case that should just be visible and fixed by a human. Instead:\n\n- **At daemon startup:** if independently-sourced playbook directories\n declare a collision on the same trigger, that is a definition failure.\n The colliding definitions are dropped (not loaded) and the failure is\n logged plainly (which sources, which trigger) and reported back to the\n caller that requested the load -- not swallowed as a background log line\n only. The schema is the definition; a schema conflict is a reported\n failure, not something the daemon silently arbitrates.\n- **Linter**: a CI/dev-time tool, reading the same generated schemas, that\n flags duplicate or conflicting trigger definitions *within* a single\n maintained tree before merge.\n\n **Resolved** (`archie-core-t2db.12`). A standalone\n gopls-shaped binary (`cmd/archie-playbooks`, lint subcommand) validates\n playbook directories against the exact loaders the daemon uses\n (`LoadPlaybookDirs`/`LoadKindWorkflowsYAML`/`LoadLabelWorkflowsYAML`),\n exiting non-zero on any collision / malformed file / invalid binding.\n A finding about one binding key leads with the key's `file:line`, read\n from the `yaml.Node` the loader decodes; the linter agrees with runtime\n validation by construction, which is the load-bearing property. `-eda-dir`\n lints an `eda_playbook_dir` the same way, through `playbook.Load` with its\n CEL, action-id and args checks.\n Discoverable via `task lint:playbooks` or direct `go run\n ./cmd/archie-playbooks lint -dir ...`. The LSP/serve mode is a later\n entrypoint of the SAME binary, per the shared-package decision above.\n- **LSP**: same schema source again, feeding author-time hover/completion/\n inline errors while someone edits a playbook YAML in an editor. Kept\n explicitly in scope per Sam (not deferred), because it's the same schema\n artifact as the linter's, just a different consumer.\n\n `archie-playbooks serve` is the language server, over stdio. When a file is\n opened or saved it lints the file's saved directory with the loader the\n daemon runs for it (a file with a top-level `trigger` key is an EDA\n playbook, anything else a routing binding file) and publishes the findings\n for that file as error diagnostics, each on the line of the key, action,\n `when` or args key it names; a file-level finding sits on the first line.\n Unsaved edits are not validated. Hover and completion are later additions.\n\nThe startup log-and-refuse path exists for what the linter/LSP structurally\ncannot see -- sources composed together only at daemon startup, never\nchecked against each other by either tool. If the linter and LSP do the job\nthey're built for, a real collision reaching that startup path should be\nrare in practice, not a routine occurrence the design leans on.\n\n**Shared package, thin consumers -- not a bundled-vs-standalone choice.**\nThe validation logic (`LoadKindWorkflowsYAML`, `LoadLabelWorkflowsYAML`, the\ndirectory-merge from t2db.11) lives in `internal/domain/workflow`, a plain\nGo package with no knowledge of which binary calls it. This makes \"should\nthe linter/LSP be bundled into archied or a separate tool\" a non-question:\n- `archied` already imports the package directly and validates in-process\n at startup -- no subprocess, no external tool call.\n- An agent that needs to validate a playbook in-process can import the same\n package directly, for the same reason.\n- The standalone `cmd/` binary (t2db.12, gopls-shaped: one binary, a lint\n mode now, an LSP/serve mode later) exists only because CI and editors are\n out-of-process consumers by nature -- a CI step needs an exit code, an\n editor speaks LSP over a process boundary, neither can `import` a Go\n package. It is a thin wrapper per `organisation.md`'s `cmd/` rule, not a\n second implementation.\n\nFolding CLI/LSP protocol handling into `archied` itself would grow the\ndaemon into unrelated concerns, against the plugin-engine rule's spirit of\nnarrow capability families.\n\n## Execution-time gaps (both resolved)\n\nEverything above covers *loading* a playbook. Nothing above covers what\nhappens while one *runs*. Two gaps, both load-bearing enough to resolve\nbefore implementation, not defer:\n\n### 1. Mid-playbook action failure\n\n**Resolved** (`archie-core-t2db.18`). The default below\nis a committed decision, not a candidate list; changing it later would\nrequire its own decision. Gap 2 below is resolved by `archie-core-t2db.17`.\n\nA playbook is an ordered action list with data flowing between steps. If\naction 2 of 4 fails at runtime (an image-gen API times out, a forge call\n403s), the run needs a defined outcome. This\ncodebase already answers a structurally identical question for its\nexisting workflow engine, precisely enough to reuse rather than\nre-derive: `workflow.Run` (`internal/domain/workflow/workflow.go`)\ndraws a three-way distinction on a stage failure, and a playbook run\ndraws the same one on an action failure.\n\n**Decision: reuse `Run`'s three-way distinction directly.** The stated\ndefault is **stop-on-first-failure**: a real action error halts the\nplaybook run immediately and is reported, with no continue-to-next-action\nand no automatic retry or compensating follow-up. The three points below\nare the complete decision.\n\n1. **A real action error** (the action's own logic failed -- an API\n error, a validation failure inside the action) stops the run\n immediately. No later actions execute. The failure is reported the\n same way a load collision is reported -- dropped, logged plainly\n (which action and why), and returned to the caller, never swallowed\n as a background log line only (the Dedup section's drop-and-report\n rule). `Run` is the mechanical\n precedent for the stop itself: a stage error sets the park reason and\n returns without running further stages (`workflow.go`). This\n is stop-on-first-failure, matching the document's existing bias\n against clever automatic recovery, and it is now a stated decision\n rather than a nobody-chose default.\n2. **Interruption** (the daemon is shutting down mid-dispatch, `ctx.Err()\n != nil`) is explicitly **not** a failure. `Run` already treats this\n case specially -- \"Daemon shutdown is not a workflow failure. Leave\n the task running ...; parking here would publish a false failure and\n require manual intervention\" (`workflow.go`) -- and the same\n reasoning applies to a playbook run: an in-flight dispatch interrupted\n by restart must not be recorded as a broken playbook. What \"leave it\n alone for restart\" means for a playbook run (there is no `store.Task`\n row a playbook run inherently owns the way a workflow stage does) is\n an open implementation detail for whoever builds the coordinator, not\n decided here -- but the *semantic* (interruption ≠ failure) is decided.\n3. **A run that produces no outcome at all** (every action ran without\n error, but nothing was assigned to happen) is itself an error\n condition, matching `Run`'s own \"a workflow must end with an explicit\n outcome; not doing so is a definition bug, which still must not vanish\n silently\" (`workflow.go`). A playbook with actions that all ran\n but never reached a coherent terminal state is a definition bug in the\n playbook, reported as such.\n\n**Rollback:** explicitly **not attempted**, for exactly the reason the\ngap's own description named -- a posted forge comment is not revocable.\nThis is not a limitation to work around later; it is the correct\nsemantic. An operator who needs a corrective action after a failure\nwrites a *new* playbook/action for that, they do not get automatic\nundo.\n\n**This stays inside the producer-only rule.** Stop-on-first-failure is the\nrun deciding its own terminal state from its own action's error. Nothing\nalready in flight is blocked or altered; the run only declines to start\nactions that had not started yet, and it records the failure on its own\nunit the way `Run` sets a park reason and transitions the task to\n`StatusParked`. A future compensating action (\"if the notify action\nfailed, also log an incident\") would be producing new work, which the rule\nalready permits. No syntax for it is designed here.\n\n**J3 relationship, stated explicitly:** J3 (a CEL `when` condition\nerroring at eval time → treated as false → skip that action → run\ncontinues) is a **different case from an action failure** and is not\nrevisited by this resolution. A condition error means \"we couldn't\ndetermine whether to run this action,\" which this document already\nresolved as \"then don't, and move on.\" An action failure means \"we tried\nto run this action and it broke,\" which stops the run per point 1 above.\nThe two must not be conflated: a coordinator implementation should be\nable to point at one `when`-evaluation code path and one\naction-invocation code path and show they use different outcomes.\n\n### 2. Idempotency at execution time\n\n**Resolved** (`archie-core-t2db.17`, a\ndesign-investigation ticket: resolve, do not implement). The decision\nbelow is the answer; the keying scheme itself is separate follow-up work\nand is intentionally not built here. It **extends**\n`event-sources-and-reactions.md`'s existing contract -- it does not\ncontradict it (relationship argued at the end).\n\n**Problem (unchanged):** `event-sources-and-reactions.md` decided\nreactions ride `internal/eventbus`'s at-least-once delivery specifically\nso `PublishUnique`/idempotency keys absorb redelivery without duplicate\nside effects. This document inherits that for free only for\n`workflow`-kind actions, because `TaskEnvelope` already carries an\nidempotency key. It said nothing for `channel`, `forge`, or `module`\nactions with real side effects -- a posted comment, a sent message, a\nbilled image generation.\n\nOne correction to the problem statement, checked against the code rather\nthan assumed, because it changes what the answer must be:\n**`PublishUnique`'s dedup is not a consumer-redelivery guard at all.**\n`PublishUnique` only sets the JetStream `Nats-Msg-Id` header on the\noutgoing message (`internal/infrastructure/eventbus/nats/publisher.go`;\nkey constant `message.go`); JetStream suppresses *republished*\nmessages carrying a repeated key inside `Config.DedupWindow`\n(`client.go`, `Duplicates: cfg.DedupWindow`), default 2 minutes\n(`config.go`, `DefaultDedupWindow = 2 * time.Minute`). A consumer\nredelivery is the same *stored* message re-delivered on Nak or\nacknowledge-timeout (`internal/eventbus/eventbus.go`: \"a handler\nreturning an error causes redelivery\") -- not a republish; its\n`Nats-Msg-Id` is never re-firewalled, and even if it were, two minutes is\nnot an at-most-once guarantee for a side effect that must never repeat.\n**Conclusion: side-effecting action kinds need a consumer-side, durable\ndedup record; the bus cannot supply it.** That is precisely the situation\n`binding_dispatches` was built for -- so the decision below reuses its\nmechanism rather than designing a new one.\n\n#### Decision\n\n**Granularity: per-action-per-event -- one dedup record per\n(action, event) pair, not per playbook run.** Whole-run keying, the\nobvious alternative, has two crash windows: the run row is written when the run starts, so a crash mid-run\nleaves the run \"started\" and a redelivery skips *every remaining\naction* (work silently lost); a crash before the row write re-runs\n*every already-fired action* (duplicate side effects -- the exact harm\nthis gap exists to prevent). Per-action keys collapse both into one\nwindow and one behavior: an action's record is written immediately\nbefore that action's side effect is invoked, so an action fires iff its\nown record is absent, no fired action ever re-fires, and a redelivered\nevent resumes at the first unrecorded action. For the shipped\nsingle-action playbooks the two granularities are equivalent in effect;\nthe difference starts with multi-action runs -- where whole-run keying\nwould already be wrong.\n\n\u003e Storage superseded: every store now lives in PostgreSQL behind\n\u003e `internal/domain/storecontract`, implemented in `internal/infrastructure/postgres`\n\u003e (`docs/prds/state-store-contract.md`). SQLite names below describe the\n\u003e original design.\n\n**Storage/lookup: a new durable ledger table `playbook_dispatches` in\nthe event-capture store, copying `binding_dispatches`'s conventions exactly**\n(durable table, `INSERT OR IGNORE`, sentinel error, rows freed with\ntheir parent, survives daemon restarts -- `RecordDispatch`):\n\n```sql\nCREATE TABLE playbook_dispatches (\n playbook_id TEXT NOT NULL,\n playbook_version TEXT NOT NULL,\n event_id TEXT NOT NULL,\n action_id TEXT NOT NULL,\n dispatched_at TEXT NOT NULL,\n PRIMARY KEY (playbook_id, playbook_version, event_id, action_id)\n);\n```\n\nWritten with the same `INSERT OR IGNORE` + sentinel-error convention\n(`ErrAlreadyDispatched`, `bindings.go`), so a duplicate is a no-op\nwrite rather than a constraint error, and the caller matches it with\n`errors.Is` the way the binding dispatch loop already does\n(`internal/daemon/daemon.go` -- \"another cycle already won this\nrace,\" not an error to surface). The record is consumed by the playbook\ncoordinator at dispatch time through a domain-side interface implemented\nby the event-capture store -- the same shape as `BindingDispatcher`\n(`internal/domain/storecontract`) -- and the table lands in the\nstore's schema migrations. No change\nto `internal/eventbus`; no per-message dedup state on the client.\n\n**Lifetime: no time-based expiry, and no automatic reclamation.** Rows are\nretained until an explicit `DeletePlaybookDispatches` for their playbook.\nRemoving a playbook file does not delete its rows: playbooks are read only\nat boot, so the only place to notice a removal is a boot-time reconcile,\nand a boot that finds the directory missing or unmounted loads as \"no\nplaybooks\" and would reclaim the whole ledger, letting redelivered events\nfire their side effects again. A binding's rows go with an explicit\noperator delete (`bindings.go`, same transaction); a file disappearing is\nnot an equally deliberate act. The at-most-once guarantee must not\nsilently decay, which is exactly the property that disqualifies\n`DedupWindow` above; table growth is bounded by distinct (playbook,\nevent, action) pairs and is accepted, the same tradeoff\n`binding_dispatches` already makes.\n\n**Redelivery behavior: skip the recorded action and continue the run.**\nA detected duplicate is not re-invoked, is not reported to the caller as\nan error, and is logged at debug level rather than as a warning -- this\nis the expected, correct behavior of at-least-once delivery, not an\nanomaly (same treatment as `ErrAlreadyDispatched` at\n`daemon.go`).\nThe run continues with the next action (or ends, for a single-action\nplaybook whose one action is recorded), which is what makes a redelivery\na resume rather than a dead restart.\n\n**Key derivation: a fixed composition of four structural identities,\ncomputed by the coordinator -- not a CEL expression.** The key is the\n`(playbook_id, playbook_version, event_id, action_id)` tuple above, where:\n\n- `playbook_id` is `Playbook.ID` -- the playbook file's path relative to\n its configured directory root, slash-normalized, unique within the load\n composition by construction (`internal/domain/eda/playbook`).\n- `playbook_version` is `Playbook.Version` -- a SHA-256 of the loaded\n file, recomputed on every load (`internal/domain/eda/playbook`),\n mirroring `Binding.Version`'s provenance-pinning purpose: the dispatch\n is pinned to the exact definition that was active when it fired. A\n changed playbook definition is a different key and may fire again for\n the same event -- deliberate, not a bug.\n- `event_id` is `DispatchInput.TaskID` (`playbook.go`), which\n carries the `TaskEnvelope.IdempotencyKey()` value\n (`\"archie:\" + owner/repo/number`,\n `internal/domain/workintake/envelope.go`). It is chosen over\n `store.Task.ID` because the trigger vocabulary is issue-level\n workintake label/kind, dispatched at discovery time (pollNATS, forge\n webhook receiver), where no task row exists yet (`playbook.go`\n comment; `playbook_test.go`: \"the value available at the\n discovery/dispatch point... NOT a store.Task.ID int64\"). Keying on the\n issue identity rather than the delivery source is the load-bearing\n property the existing contract already established for `TaskEnvelope`\n (`event-sources-and-reactions.md` question 4: a webhook-sourced\n envelope for the same issue produces the *same* idempotency key as a\n poll-discovered one; `internal/forge/webhook/receiver.go` --\n \"Idempotency is not this package's job\"); this resolution inherits that\n property rather than re-deciding it. Consequence, stated explicitly:\n same-issue re-discoveries (a re-label that re-triggers the same kind)\n are the *same event* by design -- the workintake trigger vocabulary is\n issue-granular, exactly as `TaskEnvelope`'s own key is. A future\n capture/binding-originated trigger type keys on `captured_events.id`\n instead; derivation is per-trigger-type, not one formula for all\n triggers (rule carried over unchanged from the draft).\n- `action_id` is the action's declared `id` -- J1's named ids, \"a\n required field on every action that later actions reference\", unique\n and stable-identifier-shaped, validated at load (the CEL resolution's\n J1 and its context table above in this document). For an action declaring no `id`\n (permitted by J1 while unreferenced), the key uses the action's 1-based\n position in the playbook: deterministic and stable for the\n single-action playbooks, and a later reorder is a file edit whose\n content hash changes `playbook_version` with it, so a positional\n fallback cannot silently collide with a prior definition.\n\n**No CEL in the key -- stated explicitly because the CEL resolution\nflagged this as gap 2's \"first real exercise\"** (\"CEL's read-only result\naccess makes gap 2's keying derivation readable\" -- the CEL resolution's\nclosing note). Readability\nis not the binding constraint; **determinism under redelivery is**, and\nCEL key expressions lose three ways. First, they are not load-checkable:\n`event` stays `dyn` by the resolved J4 (schema-by-example), so\na key expression over event fields cannot be type-checked at load,\nviolating the document's reject-at-load philosophy. Second, they can\nfail at dispatch (a missing field is a CEL error), and a key that cannot\nbe computed is an unfireable state -- neither \"skip\" nor \"fire\" is safe\nwithout losing or duplicating work. Third, the contract needs one\nderivation, and two authors writing different key expressions would\nfragment the ledger. The key is therefore structural by construction:\nevery component above is an immutable identity of the run's inputs, so\ntwo deliveries of the same event derive the same key with no evaluation.\nThe exercise's actual outcome is the derivation rule above; CEL remains\na *condition* surface (`when`), never a key surface. An action kind that\ngenuinely needs finer granularity than (action, event), such as \"per\ncomment, not per issue\", arrives with its own trigger type and its own\nevent identity, per the per-trigger-type rule.\n\n**Record-before-invoke -- one deliberate divergence from\n`binding_dispatches`, justified by the harm profile.** The binding\ndispatch loop enqueues the task first and records the ledger row after\n(`daemon.go`), accepting that racing cycles may both enqueue\n(\"the duplicate task remains queued... delete-on-races is its own can of\nworms\"); the selection query excludes already-dispatched captures\n(`bindings.go`), but the race window remains. That ordering is\ncorrect there because the duplicate artifact is an internal task row --\nrecoverable, low-harm. The playbook ledger's ordering is reversed on\npurpose: **the row is written and committed before the action's side\neffect is invoked** -- the `INSERT OR IGNORE` is the atomic gate, and\nthe lost-write loser skips without invoking. The cost is stated, not\nhidden: a crash between the row commit and the invoke loses that one\naction's side effect, repairable by hand -- the reverse direction of the\nsame window that would otherwise duplicate a posted comment or\nre-bill an image generation. For side effects whose duplication is\nexternally visible and not revocable, at-most-once is the required\ndirection; the binding system's at-least-once-tolerant ordering is not\nthe precedent for this ledger.\n\n**Gap-1 interplay, stated to keep the two mechanisms separate:** the\nledger answers one question per action -- has this action already fired\nfor this event? It does not answer whether the run continues, stops, or\nis re-attempted; that is gap 1's coordinator concern (a run-outcome\nrecord; gap 1 explicitly leaves \"what leave it alone for restart means\nfor a playbook run\" to whoever builds the coordinator). One consequence\nof record-before: an action that FAILED (gap 1's real-action-error case)\nhas a ledger row, so a later redelivery of the same event skips it --\nconsistent with gap 1's \"no automatic retry\" resolve (a corrective\naction is a *new* playbook/action, per gap 1's rollback paragraph). A\nfailed action is never re-attempted by the bus; if a run must be\nrecovered, an operator edits the playbook -- new version, new key --\nrather than fighting ledger state.\n\n**Relationship to `event-sources-and-reactions.md` -- extension, not\ncontradiction.** Keyed to that document's sections:\n\n1. Its decision 2 commits reactions to at-least-once delivery via\n `PublishUnique`'s \"without duplicate enqueue\" guarantee -- which, with\n the code facts above, is accurate for the *publish* path only:\n `PublishUnique` absorbs duplicate publishes of the same event (poll\n vs. webhook), which is what that decision was actually about. This\n document's own problem statement, taken literally, overstates it\n (\"keys absorb redelivery without duplicate side effects\") -- that\n overstatement is exactly what this gap exists to correct. The\n existing contract never claimed consumer redeliveries of\n side-effecting reactions were absorbed, so nothing it says contradicts\n a consumer-side ledger; this resolution pins down the boundary the\n sentence left implicit (publish-path dedup vs. consumer-path dedup)\n rather than changing either.\n2. Its decision 3 is untouched: the ledger is the run's own bookkeeping\n about its own actions, suppressing a re-fire and nothing else.\n3. Its decision 4 (no generic `Source` interface; `TaskEnvelope` is the\n typed contract, keyed on issue identity, not delivery source): the\n event half of the playbook key IS that same identity\n (`TaskEnvelope.IdempotencyKey()`), and the per-trigger-type rule\n means any future trigger type brings its own identity rather than a\n new generic one -- the playbook ledger introduces no second identity\n vocabulary and no untyped payload. The only new type in this\n resolution is the ledger's own value object, consumed through a\n narrow interface like `BindingDispatcher` -- the same shape the\n document's \"typed families\" rule (decision 1, plugin engine rule)\n already blesses.\n\n**Scope: this is the `channel`/`forge`/`module` decision, not a\n`workflow`-position change.** Workflow-kind actions keep their existing\nkeying claim unchanged -- the `TaskEnvelope` path that this document\nalready inherits for free (`daemon.go`, `PublishTask` publishes\nwith `task.IdempotencyKey()`), with its cross-poll/webhook property\nuntouched. The ledger gate applies to the side-effecting positions\n(channel, forge, module), which is why the closing statement below\nstill holds: the first slice (Module + loader + workflow-kind dispatch)\nwas buildable without this resolution, and channel/forge/multi-action\nplaybooks were not. A coordinator MAY implement one position-uniform\ngate as a simplification; what this resolution requires is only that\nthe side-effecting positions are gated.\n\n## Resolved questions\n\n1. **Data-flow/condition syntax.** Does this project want a small existing\n Go expression evaluator (there may be one already in the codebase worth\n checking, e.g. anything backing gate conditions), or a bespoke minimal\n grammar?\n\n **Resolved** (`archie-core-t2db.14`). CEL (`cel.dev/cel-go`, pinned) is\n the one mechanism for both an action's `when` condition and its `args`\n values. The trust boundary, evaluation context, load-time validation and\n the five judgment calls are in\n `docs/prds/playbook-expression-syntax.md`.\n\n2. **Playbook directory config field.** Where this lives in\n `internal/config` / `configuration.Document`, and whether it's one\n directory or a list (mirroring `SkillsDir`'s shape) -- follow existing\n config precedent, not invented fresh.\n\n **Resolved** (`archie-core-t2db.11`). The directory\n shape landed as `playbook_dirs` (a LIST of directories of\n `*.yaml`/`*.yml` binding files), loaded at startup as an additional\n input to the two single-file fields, which remain supported unchanged.\n A single `playbook_dir` would contradict\n supporting multiple independently-maintained playbook sources, which\n the Dedup section already assumes ('independently-sourced playbook\n directories collide'), so the field is a list. Cross-source collision\n (same key in two directories, or in a single-file field and a\n directory) is reported like an in-directory collision -- nothing is\n arbitrated by source precedence. This is what finally exercises the\n design doc's drop-and-report collision rule, since a single file\n cannot collide with itself. Repo-scoping (tying directories to\n `Config.Repos` owner/name) is explicitly deferred to a separate,\n later decision; org/tenant keying is out of scope (no auth/identity\n model).\n\n Decisions recorded from the label-vocabulary slice:\n - Arbitrary labels bind to registered workflow names via\n `LabelWorkflows` + `LoadLabelWorkflowsYAML`/`SetLabelWorkflows`\n (`internal/domain/workflow/routing.go`).\n - The closed `Kind`/NATS-subject set is untouched; the label map only\n extends binding authority for labels the kind layer does not own.\n - Collision rule: a label already owned by the\n kind set (bug/feature/bootstrap), an empty label, an empty workflow\n name, or a duplicate binding is a reported load failure -- dropped\n and logged, the error returned to the caller. Nothing is silently\n arbitrated by schema/version. This is the \"schema defines the\n accepted message; anything else is the caller's fault\" rule applied\n to the label layer.\n - Precedence in `Route()`: explicit `t.Workflow` → arbitrary-label\n binding → kind binding → triage → implement → default.\n3. **Trust boundary for Module Yaegi code.** The plugin engine rule's\n invariant 6 draws a line between operator-trusted in-process code and\n repository-supplied code that must run in a container.\n\n **Resolved** in `docs/prds/module-position.md`, \"Trust boundary\": a\n Module is operator-installed, in-process and daemon-privileged, the same\n tier as `PluginDir` and `SecretEngineDir`, never repository-supplied\n task code.\n4. **First implementation slice.** Recommend: Module position + the\n playbook loader + trigger-to-workflow dispatch only (subsuming the existing\n label routing) first, proving the schema-gen -\u003e Yaegi -\u003e playbook path\n end to end on the smallest useful case, before adding Channel/Forge\n action kinds or the linter/LSP.\n\n **Resolved** (`archie-core-t2db.13`, `.14`, `.15`). The Module position\n (registry + log kind, t2db.13), the CEL expression environment\n (t2db.14), and the playbook document + event coordinator with\n single-action workflow-kind dispatch (t2db.15) are shipped.\n\n **Resolved** (`archie-core-t2db.23`): the coordinator selects a real\n task's workflow. The wiring point is the daemon's definition pin\n (`Daemon.resolveWorkflowID`, consumed by `pinWorkflowFromCollection` in\n `internal/daemon`), not `pollNATS`/`publishTask`. Routing belongs to the\n pin because a workflow definition is a\n database row pinned before dispatch, which makes the pin the one place a\n task's workflow is chosen. Precedence there is explicit `Task.Workflow`\n (the waiting_human -\u003e approved requeue) → matching playbook →\n `workflow.ResolveWorkflowID`'s label binding → kind binding → triage →\n implement. A playbook naming an undefined workflow is a reported failure\n that parks the task rather than a silent fall-through to the binding's\n choice.\n\n `Store.Dispatch` returns the selected workflow *name* and the matching\n playbook's id and version, not a compiled `workflow.Workflow`: the\n production caller decides against the active definition collection, and\n the id and version are the first two components of the gap-2 idempotency\n key. The playbook package does not import `internal/domain/workflow`.\n\n A `when` condition at this point reads what the intake knows about the\n originating issue: `kind`, `labels`, `owner`, `repo`, `number`, `title`,\n `body`, `source`, built by `playbookInput` in `internal/daemon`.\n\n The trigger shape reuses the existing workintake kind/label vocabulary.\n The single-action hard boundary is superseded by\n `docs/prds/multi-action-playbooks.md` (two-shape rule); Module, Channel,\n and Forge positions still have no production dispatch path.\n## Standing constraint\n\nThis may be commercialized, and is to be discernible from other open-source\nevent-driven-automation tooling. No design decision follows from that yet;\nit bears on licensing or feature-gating questions about playbook authoring\nand the LSP when they arise.\n\n## Packages this touches (first slice)\n\n- New: a schema-generation package/convention (`go:generate` source +\n emitted contracts) for the Module position.\n- New: playbook loader + event coordinator (name/location TBD -- likely\n `internal/domain/eda` or similar, following the domain-layer rule: no\n infrastructure imports, no untyped hooks).\n- `internal/domain/workflow`: dispatch target for `workflow`-kind actions,\n unchanged in its own contracts.\n- `internal/domain/workintake`: `labelKinds`/routing becomes one playbook\n among others once the loader subsumes it; existing Go fallback stays for\n backward compatibility until playbooks fully replace it (mirrors the\n `dynamic-workflow-triage.md` precedent of a two-line fallback change, not\n a rip-and-replace).\n" }, { "url": "/dev/prds/embedded-nats/", @@ -292,13 +292,13 @@ "url": "/dev/prds/memory-engine-unification/", "title": "Memory engine unification: four scopes, CRUD, and one write path -- decision", "section": "PRDs", - "body": "# Memory engine unification: four scopes, CRUD, and one write path -- decision\n\n**Status:** Approved\n**Date:** 2026-09-16\n**Findings:** `memory-4` and its dependants in the adversarial sweep\n(`.local/issue-tracker`); the real defect is the split described below, for\nwhich `memory-4` is a symptom\n**Supersedes:** the draft `memory-turn-context.md` (its synchronous-read\nargument survives and is reused; its `Manager` method and consumer-owned port do\nnot)\n**Settles:** four open items in `docs/architecture/migration-decisions.md#5`\n(listed at the end)\n\n## Problem\n\nThere are **two live memory stores for one assistant**, split by which component\nwrites them, and the chat turn reads neither.\n\n| Family | Written by | Root | Format |\n|---|---|---|---|\n| `internal/memory` (`Manager`, `builtin.Provider`) | the chat agent's `memory_edit` tool via `memorytoolprovider` (`bootstrap.go`) | `\u003cworkDir\u003e/memory` | `MEMORY.md` / `USER.md` |\n| `internal/domain/memory` + `internal/infrastructure/memory` | the session curator (`sessioncurator.go`) | `\u003cworkDir\u003e/memory-engine` | identity-keyed blocks |\n\nThey share no code, no format and no scope vocabulary. Neither reaches a prompt:\n`Manager.SystemPromptBlock` (`internal/memory/manager.go`) has no production\ncaller anywhere, so the agent can write memory through a tool and never read it\nback.\n\nThree further defects fall out of the same split:\n\n1. **The engine's contract is not CRUD.** `Store` is `Write`/`Query`/`List`/\n `Forget` — create, read, delete, and **no Update**. The retired file provider\n *does* have `handleAdd`/`handleReplace`/`handleRemove`, so the family being\n deleted is operationally richer than its replacement.\n2. **There is no scope model.** The contract carries one opaque `Identity\n string`, and the curator fills it with a **session id** (`sessioncurator.go`)\n — which is none of the four scopes `docs/architecture/agent-system.md`\n requires, and stops meaning anything when the session ends.\n3. **The chat turn has no user identity on half the channels.** Telegram and\n email populate `SenderID` with a person; the dashboard populates it with\n nothing, and a webhook populates it with nothing either — a route path names\n a source, not a person, so `webhook.go` leaves `SenderID` empty and carries\n the route in `Inbound.BudgetKey` for rate limiting. `sender_id` is persisted\n (`session_store_sqlite.go`), so whatever a channel puts there reaches the\n consumers that read it.\n\n`agent-system.md` says the write-scope selection and cross-scope sharing\nrules \"require a focused memory design decision\". This is that decision.\n\n## Decision\n\n### 1. One engine, addressed by four typed scopes\n\n`internal/domain/memory` is the only memory engine. `Observation.Identity`,\n`Query.Identity`, `Record.Identity` and `List(identity)` are deleted, not\nextended: a field whose meaning depends on the caller cannot carry a scope.\n\n```go\ntype ScopeKind string\n\nconst (\n ScopeGlobal ScopeKind = \"global\" // shared across the application\n ScopeAgent ScopeKind = \"agent\" // private to one Agent\n ScopeUser ScopeKind = \"user\" // one user, across their Agents\n ScopeAgentUser ScopeKind = \"agent-user\" // one Agent's relationship with one user\n)\n\ntype Scope struct {\n Kind ScopeKind\n Agent AgentID // required for ScopeAgent, ScopeAgentUser\n User IdentityID // required for ScopeUser, ScopeAgentUser\n}\n\nfunc (s Scope) Validate() error // rejects a component present where it is not required\nfunc (s Scope) Key() string // canonical, collision-free storage key\n```\n\n`AgentID` and `IdentityID` are opaque. This package never resolves or\ncanonicalises them.\n\nA turn's resolved identity is one value:\n\n```go\ntype Subject struct {\n AgentID AgentID\n UserID IdentityID // \"\" when the channel carries no per-person identity\n}\n\nfunc (s Subject) Scopes() []Scope // readable, most specific first\nfunc (s Subject) WritableScopes() []Scope // never includes ScopeGlobal\n```\n\n**Access control is naming a scope, and nothing else.** The read path names\nexactly the scopes of the resolved `Subject`; the engine never unions beyond what\nit is handed and applies no policy of its own. An engine that enforced policy\nwould need to know about channels, sessions and bindings, and every backend would\nreimplement it. \"Agent-wide is private to its owning Agent by default\"\n(`agent-system.md`) falls out mechanically, because `ScopeAgent{a}` and\n`ScopeAgent{b}` are different storage keys.\n\n`ScopeGlobal` is never model-writable. Writing it is an operator action; nothing\nproduces one yet, but the contract models it because the read set includes it.\n\n### 2. CRUD, with revisions retained\n\n```go\ntype Store interface {\n Create(ctx context.Context, in NewRecord) (Record, error)\n Get(ctx context.Context, scope Scope, id RecordID) (Record, error) // ErrNotFound\n Query(ctx context.Context, q Query) ([]Record, error) // heads only\n List(ctx context.Context, scope Scope) ([]Record, error) // heads only\n Update(ctx context.Context, in RecordUpdate) (Record, error) // ErrNotFound, ErrStaleRevision\n Forget(ctx context.Context, scope Scope, id RecordID) error // idempotent\n Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)\n}\n```\n\n`Get` and `Forget` take a scope. The current `Forget` scans every identity's\nstore to find a marker (`builtin_engine.go`) purely because the contract\ngave it nothing better.\n\n`Record` carries what `agent-system.md` requires be retained: `Scope`,\n`Kind`, `Content`, `Revision`, `Author`, `OriginUser`, `Source`, `CreatedAt`,\n`UpdatedAt`. `Metadata` is dropped — nothing sets it and the parser never\nrestores it, so it is a field that silently does not round-trip.\n\n`Update` supersedes rather than overwrites. The superseded state is appended to\nthe scope's `HISTORY.md` **before** the live block is replaced, so a crash\nbetween the two leaves an extra history entry and an unchanged record —\nrecoverable — never a record whose previous state was lost. `Update` carries no\n`Scope`: a record may never move between scopes, and the API makes that\nunrepresentable rather than validating it. `Expected` optionally names the\nrevision being replaced and returns `ErrStaleRevision` on mismatch, which matters\nbecause two processes (`archied` and `archie-gateway`) can hold the same scope's\nfile.\n\n`Forget` records a deleted revision before removing the live block, so a record's\nprovenance outlives its content.\n\n### 3. The caller resolves identity; the engine never does\n\nResolution happens in the gateway turn path, where the native sender id and the\nsession are both in hand, and it is supplied **per channel by the composition**:\n\n```go\n// TurnRunnerConfig\n// UserIdentity resolves the initiating user's identity for one inbound message.\n// false means the channel carries no per-person identity, and the turn gets\n// global and agent scopes only. Nil means the same.\nUserIdentity func(msg messaging.Message) (IdentityID, bool)\n```\n\nThis must be injectable rather than a blanket `channel + \":\" + SenderID`\nconcatenation, because a channel's sender id is not necessarily a person: a\nwebhook's route path names a source, and treating it as an identity would give\na URL one -- which is why `webhook.go` leaves `SenderID` empty and carries the\nroute in `Inbound.BudgetKey` instead. Telegram and email supply their sender\nid; the dashboard and webhook resolve nothing.\n\n**Fail closed.** When `UserID` is empty the turn proceeds with global and agent\nscopes only, and a write targeting `user` or `agent-user` fails with a clear\nerror. `agent-system.md` forbids showing one user's memory to another, so\n\"we do not know who this is\" must mean \"show none of it\", never \"fall back to a\nwider scope\".\n\n### 4. The chat read path\n\nSynchronous and total. It runs inside `prepareTurn` (`turn.go`) before the\nprompt is built, never returns an error to the caller, and degrades to an empty\nblock on any engine failure or provider panic. A fire-and-forget hook would race\nthe prompt build in the same turn, which is why this is not a lifecycle hook.\n\nThe read is a scope-only `Query` over the subject's scopes — **not** ranked\nsearch, which remains open in `migration-decisions.md#5`. A ranked read on the\nhot path with no ranking implementation would be a lie.\n\n`SystemPromptConfig` gains a `Memory` field, rendered into\n`templates/archie.md.tpl` between the `\u003ctools\u003e` and `\u003cenv\u003e` blocks:\n\n```\n\u003cmemory purpose=\"durable_context\" trust=\"data\"\u003e\n```\n\n`trust=\"data\"` and inclusion in the precedence clause follow the `\u003ctools\u003e`/`\u003cenv\u003e`\nprecedent exactly: memory content is model-written from untrusted input and must\nbe framed as data, never instructions. The rendered block carries each record's\nshort id, because the write path addresses records by id — one source of truth\nfor both directions.\n\nBoth a per-scope record limit and a byte cap on the rendered block are named\nconstants. The cap is about size, not latency; a local read needs no timeout.\n\n**Prerequisite:** `setupGateways` runs before `setupMemoryAll`\n(`main.go` vs `:428`), and `startGatewayRuntime` before `setupMemory`\n(`gateway.go` vs `:115`), so the engine is nil at every turn-runner\nconstruction site. The ordering is the root-cause fix and nothing in either\nmemory step reads what those later steps produce.\n\n### 5. The chat write path\n\nThe memory tool is built **per turn**, not registered once at boot, because it\nmust write into a scope derived from the turn's resolved subject:\n\n| action | engine call |\n|---|---|\n| `create` | `Create` |\n| `update` | `Update` (by id) |\n| `delete` | `Forget` (by id) |\n| `list` | `List` |\n\n**The model chooses the scope kind; it never chooses the ids.** Agent and user\nids come from the turn's resolution and there is no schema field to put anything\nelse in — even a model that emits another user's id has nowhere to write it. That\nis the root-cause form of `agent-system.md`, matching this system's other\nnon-waivable boundaries (the model never runs git; gates cannot be waived by the\nagent). `global` is not offered at all.\n\nThe tool takes a record **id** where the retired provider took a content\nsubstring (`handleReplace`/`handleRemove`). The substring addressing existed\nbecause a markdown file was hand-edited and the model had no stable handle on a\nblock; with ids rendered into the prompt it disappears, and one addressing scheme\nreplaces two.\n\n### 6. The curator's write\n\n`sessioncurator.reviewOne` stops writing `Identity: sessionID` and writes\n`ScopeAgentUser{AgentID: \u003csession's agent\u003e, UserID: \u003cparticipant\u003e}`. Relationship\nscope is the only defensible home: agent-wide is explicitly unsafe for user facts\nand would leak user A's extracted facts to user B through the same agent;\nuser-wide would pool facts across agents; and a session id cannot survive the\nsession ending, which is the whole requirement.\n\nThis needs data the tree does not persist: the SQLite `messages` table\ngains `sender_id`, and `curator.SessionSummary` / `ConversationMessage` gain the\nagent and participant. The participant is derived from the session's own\nuser-role messages: exactly one distinct non-empty sender writes; zero (a\ndashboard or webhook session) or several (a group chat) writes nothing and\nrecords why. Ambiguity fails closed rather than guessing.\n\nThe payoff is the point of the exercise: chat-written and curator-derived memory\nnow occupy the same scopes, so a future turn's prompt contains both, and a\ncurator's bad note becomes id-addressable and therefore correctable by the agent.\n\n### 7. The content scanner moves to the engine\n\n`internal/memory/scanner.go` and its installation at `bootstrap.go` are what\nmake `configuration.md`'s \"Memory safety scanner\" row describe a control that\nactually runs. Deleting the package would silently remove prompt-injection\nscanning on memory writes. The scanner moves into the engine and is applied in\n`Create` and `Update`, which is the single choke point every producer crosses —\nso the curator's model-extracted content is covered too, not just the chat tool's.\n\n### 8. `internal/memory` is deleted\n\nNot shimmed, not partially, not behind a flag: `manager.go`, `provider.go`,\n`lifecycle.go`, `sync.go`, `scanner.go` (moved), `schema.go`, `backup.go`,\n`config.go`, `builtin/provider.go`, and their tests. Its markdown store\n(`builtin/store.go`) is the one survivor, relocated to\n`internal/infrastructure/memory/builtin/` — `organisation.md`'s \"a package owns\nits on-disk format end-to-end\" makes infrastructure its correct home, and the\nengine already parses that format.\n\nAlso deleted: `internal/tools/provider/memory`'s manager wrapping (rewired to the\nengine), `config.Memory.{Provider,ProviderConfig}` and the\n`validateMemory` rejection that exists only to complain about `Provider`\n(`validate.go`), and `webui.MemoryStatus`/`MemoryProviderHandle` — never\nimplemented in production, so `/api/memory` reports empty forever unless it is\nrepointed at the engine registry.\n\nExisting data at `\u003cworkDir\u003e/memory` is left on disk, unreferenced. No migration.\n`\u003cworkDir\u003e/memory-engine` is likewise orphaned: its directories were keyed by\n`sha256(sessionID)` and no scope key can hash to one.\n\n## What we deliberately do NOT do\n\n- **No cross-scope sharing or grants in this wave.** Promoting a record between\n scopes is a future explicit command, and `Update` cannot change scope, so it\n cannot happen by accident.\n- **No canonical user identity.** `SenderID` is channel-native, so one person on\n Telegram and email is two identities. Unifying needs an Identity aggregate that\n does not exist (`internal/domain` has no identity package and `IdentityID`\n appears nowhere in Go). The contract does not change when it lands.\n- **No ranked retrieval.** `Query.Text` stays, unused by the prompt path.\n- **No data migration**, per the decision above.\n- **No prompt-memory timeout mechanism.** The read is bounded by construction.\n- **No `Metadata` field.** It is write-only; a caller needing it later\n needs one field, not a speculative contract.\n\n## Acceptance criteria\n\n1. A record created through the memory tool in turn 1 appears in turn 2's prompt,\n asserted on content the test planted, with its id.\n2. The same record is **absent** from a different user's prompt on the same\n channel and agent. Isolation asserted end to end, not at the engine boundary.\n3. An agent-scope record is visible to both of that agent's users; a global\n record is visible to a second agent; an agent-user record is not.\n4. A curator pass over a session writes a record a later turn reads, and which\n survives that session being deleted.\n5. `create` → prompt shows it; `update` → prompt shows new content and\n `Revisions` returns the prior content with author and time; `delete` → gone\n from prompt and `List`. Round-trip is asserted as `parse(render(r)) == r`,\n never against a fixture string.\n6. A panicking engine and a failing engine both complete the turn with no memory\n block.\n7. A channel with no resolvable user renders only global and agent scopes, and\n the tool refuses `user`/`agent-user` with a reason. A webhook route path never\n becomes a user identity.\n8. A write that would exceed the file cap fails loudly rather than silently\n dropping content.\n9. `internal/memory` is gone, `task check` is green, and no document describes a\n manager that no longer exists.\n\nEvery criterion asserts behaviour or planted content. None asserts on the text of\na hand-written file, and none pins that a method was called. In particular none\nasserts the composition calls setup in a given order: if the ordering were\nswapped back, criteria 1 and 4 would fail, and that is the assertion.\n\n## Sequencing\n\nEach step is independently verifiable and lands green on its own.\n\n1. **Persist the participant.** `sender_id` column on `messages`,\n insert/select/migration, `curator.SessionSummary.AgentID` and\n `ConversationMessage.SenderID`. Touches no memory contract and is shippable\n alone — nothing consumes it yet.\n2. **Scope, CRUD and revisions.** The `internal/domain/memory` contract, the\n builtin engine with `HISTORY.md`, the store relocation, the scanner move, and\n the curator rewritten onto agent-user scope (it is the engine's only live\n consumer, so it belongs in this slice). *Depends on 1 for the curator half.*\n3. **Chat read path.** The `Memory` seam, the template block, subject resolution,\n and the composition-ordering fix. *Depends on 2.*\n4. **Chat write path.** The tool adapter onto CRUD and the turn's subject on the\n context. *Depends on 3, because a write is only observable through the read.*\n5. **Delete `internal/memory` and the dead config/dashboard surface**, and\n rewrite the \"Memory engine family\" section of\n `docs/architecture/plugins-and-extensions.md` to describe the\n engine family rather than a manager it deleted. *Depends on 3 and 4.*\n\nSteps 3 and 4 are separable from each other; step 1 is separable from everything.\n\n## Settles these open items in `migration-decisions.md` §5\n\n| §5 open item | Settled by |\n|---|---|\n| authoritative memory record and revision model | §1, §2 — the live marker becomes JSON, since the ratified `\u003c!--mem:\u003culid\u003e--\u003e` cannot carry provenance; superseded states live in a per-scope `HISTORY.md` |\n| provider and infrastructure boundaries | §2 — contract in `internal/domain/memory`, builtin format in `internal/infrastructure/memory/builtin` |\n| retrieval and access enforcement | §1 — caller-named scope sets, no engine-side policy |\n| provenance representation | §2 |\n\n## Not determined\n\n- **Group chats produce no derived memory**, because the curator fails closed on\n mixed-sender sessions. Whether extraction should instead become\n per-participant is a product call.\n- **Two processes, one scope file.** After step 3 both `archied` and\n `archie-gateway` hold an engine over the same root, and the engine caches a\n store per scope and rewrites the whole file, so concurrent writers on one scope\n can lose a block. `Update.Expected` makes that detectable for one record; it\n does not make the file safe. A reload-before-mutate or advisory lock is the fix\n and is not designed here.\n- **Unbounded history.** `HISTORY.md` grows without limit; no compaction policy.\n- **Size bounds.** `defaultMaxFileBytes` is now per scope rather than per\n identity; whether that is right is unexamined.\n- **Whether the dashboard should carry a user identity.** It has one bearer\n token, not users. Until it has users, web turns are correctly limited to global\n and agent scope.\n" + "body": "# Memory engine unification: four scopes, CRUD, and one write path -- decision\n\n**Status:** Approved\n**Date:** 2026-09-16\n**Findings:** `memory-4` and its dependants in the adversarial sweep\n(`.local/issue-tracker`); the real defect is the split described below, for\nwhich `memory-4` is a symptom\n**Supersedes:** the draft `memory-turn-context.md` (its synchronous-read\nargument survives and is reused; its `Manager` method and consumer-owned port do\nnot)\n**Settles:** four open items in `docs/architecture/migration-decisions.md#5`\n(listed at the end)\n\n## Problem\n\nThere are **two live memory stores for one assistant**, split by which component\nwrites them, and the chat turn reads neither.\n\n| Family | Written by | Root | Format |\n|---|---|---|---|\n| `internal/memory` (`Manager`, `builtin.Provider`) | the chat agent's `memory_edit` tool via `memorytoolprovider` (`bootstrap.go`) | `\u003cworkDir\u003e/memory` | `MEMORY.md` / `USER.md` |\n| `internal/domain/memory` + `internal/infrastructure/memory` | the session curator (`sessioncurator.go`) | `\u003cworkDir\u003e/memory-engine` | identity-keyed blocks |\n\nThey share no code, no format and no scope vocabulary. Neither reaches a prompt:\n`Manager.SystemPromptBlock` (`internal/memory/manager.go`) has no production\ncaller anywhere, so the agent can write memory through a tool and never read it\nback.\n\nThree further defects fall out of the same split:\n\n1. **The engine's contract is not CRUD.** `Store` is `Write`/`Query`/`List`/\n `Forget` — create, read, delete, and **no Update**. The retired file provider\n *does* have `handleAdd`/`handleReplace`/`handleRemove`, so the family being\n deleted is operationally richer than its replacement.\n2. **There is no scope model.** The contract carries one opaque `Identity\n string`, and the curator fills it with a **session id** (`sessioncurator.go`)\n — which is none of the four scopes `docs/architecture/agent-system.md`\n requires, and stops meaning anything when the session ends.\n3. **The chat turn has no user identity on half the channels.** Telegram and\n email populate `SenderID` with a person; the dashboard populates it with\n nothing, and a webhook populates it with nothing either — a route path names\n a source, not a person, so `webhook.go` leaves `SenderID` empty and carries\n the route in `Inbound.BudgetKey` for rate limiting. `sender_id` is persisted\n (`session_store_postgres.go`), so whatever a channel puts there reaches the\n consumers that read it.\n\n`agent-system.md` says the write-scope selection and cross-scope sharing\nrules \"require a focused memory design decision\". This is that decision.\n\n## Decision\n\n### 1. One engine, addressed by four typed scopes\n\n`internal/domain/memory` is the only memory engine. `Observation.Identity`,\n`Query.Identity`, `Record.Identity` and `List(identity)` are deleted, not\nextended: a field whose meaning depends on the caller cannot carry a scope.\n\n```go\ntype ScopeKind string\n\nconst (\n ScopeGlobal ScopeKind = \"global\" // shared across the application\n ScopeAgent ScopeKind = \"agent\" // private to one Agent\n ScopeUser ScopeKind = \"user\" // one user, across their Agents\n ScopeAgentUser ScopeKind = \"agent-user\" // one Agent's relationship with one user\n)\n\ntype Scope struct {\n Kind ScopeKind\n Agent AgentID // required for ScopeAgent, ScopeAgentUser\n User IdentityID // required for ScopeUser, ScopeAgentUser\n}\n\nfunc (s Scope) Validate() error // rejects a component present where it is not required\nfunc (s Scope) Key() string // canonical, collision-free storage key\n```\n\n`AgentID` and `IdentityID` are opaque. This package never resolves or\ncanonicalises them.\n\nA turn's resolved identity is one value:\n\n```go\ntype Subject struct {\n AgentID AgentID\n UserID IdentityID // \"\" when the channel carries no per-person identity\n}\n\nfunc (s Subject) Scopes() []Scope // readable, most specific first\nfunc (s Subject) WritableScopes() []Scope // never includes ScopeGlobal\n```\n\n**Access control is naming a scope, and nothing else.** The read path names\nexactly the scopes of the resolved `Subject`; the engine never unions beyond what\nit is handed and applies no policy of its own. An engine that enforced policy\nwould need to know about channels, sessions and bindings, and every backend would\nreimplement it. \"Agent-wide is private to its owning Agent by default\"\n(`agent-system.md`) falls out mechanically, because `ScopeAgent{a}` and\n`ScopeAgent{b}` are different storage keys.\n\n`ScopeGlobal` is never model-writable. Writing it is an operator action; nothing\nproduces one yet, but the contract models it because the read set includes it.\n\n### 2. CRUD, with revisions retained\n\n```go\ntype Store interface {\n Create(ctx context.Context, in NewRecord) (Record, error)\n Get(ctx context.Context, scope Scope, id RecordID) (Record, error) // ErrNotFound\n Query(ctx context.Context, q Query) ([]Record, error) // heads only\n List(ctx context.Context, scope Scope) ([]Record, error) // heads only\n Update(ctx context.Context, in RecordUpdate) (Record, error) // ErrNotFound, ErrStaleRevision\n Forget(ctx context.Context, scope Scope, id RecordID) error // idempotent\n Revisions(ctx context.Context, scope Scope, id RecordID) ([]Revision, error)\n}\n```\n\n`Get` and `Forget` take a scope. The current `Forget` scans every identity's\nstore to find a marker (`builtin_engine.go`) purely because the contract\ngave it nothing better.\n\n`Record` carries what `agent-system.md` requires be retained: `Scope`,\n`Kind`, `Content`, `Revision`, `Author`, `OriginUser`, `Source`, `CreatedAt`,\n`UpdatedAt`. `Metadata` is dropped — nothing sets it and the parser never\nrestores it, so it is a field that silently does not round-trip.\n\n`Update` supersedes rather than overwrites. The superseded state is appended to\nthe scope's `HISTORY.md` **before** the live block is replaced, so a crash\nbetween the two leaves an extra history entry and an unchanged record —\nrecoverable — never a record whose previous state was lost. `Update` carries no\n`Scope`: a record may never move between scopes, and the API makes that\nunrepresentable rather than validating it. `Expected` optionally names the\nrevision being replaced and returns `ErrStaleRevision` on mismatch, which matters\nbecause two processes (`archied` and `archie-gateway`) can hold the same scope's\nfile.\n\n`Forget` records a deleted revision before removing the live block, so a record's\nprovenance outlives its content.\n\n### 3. The caller resolves identity; the engine never does\n\nResolution happens in the gateway turn path, where the native sender id and the\nsession are both in hand, and it is supplied **per channel by the composition**:\n\n```go\n// TurnRunnerConfig\n// UserIdentity resolves the initiating user's identity for one inbound message.\n// false means the channel carries no per-person identity, and the turn gets\n// global and agent scopes only. Nil means the same.\nUserIdentity func(msg messaging.Message) (IdentityID, bool)\n```\n\nThis must be injectable rather than a blanket `channel + \":\" + SenderID`\nconcatenation, because a channel's sender id is not necessarily a person: a\nwebhook's route path names a source, and treating it as an identity would give\na URL one -- which is why `webhook.go` leaves `SenderID` empty and carries the\nroute in `Inbound.BudgetKey` instead. Telegram and email supply their sender\nid; the dashboard and webhook resolve nothing.\n\n**Fail closed.** When `UserID` is empty the turn proceeds with global and agent\nscopes only, and a write targeting `user` or `agent-user` fails with a clear\nerror. `agent-system.md` forbids showing one user's memory to another, so\n\"we do not know who this is\" must mean \"show none of it\", never \"fall back to a\nwider scope\".\n\n### 4. The chat read path\n\nSynchronous and total. It runs inside `prepareTurn` (`turn.go`) before the\nprompt is built, never returns an error to the caller, and degrades to an empty\nblock on any engine failure or provider panic. A fire-and-forget hook would race\nthe prompt build in the same turn, which is why this is not a lifecycle hook.\n\nThe read is a scope-only `Query` over the subject's scopes — **not** ranked\nsearch, which remains open in `migration-decisions.md#5`. A ranked read on the\nhot path with no ranking implementation would be a lie.\n\n`SystemPromptConfig` gains a `Memory` field, rendered into\n`templates/archie.md.tpl` between the `\u003ctools\u003e` and `\u003cenv\u003e` blocks:\n\n```\n\u003cmemory purpose=\"durable_context\" trust=\"data\"\u003e\n```\n\n`trust=\"data\"` and inclusion in the precedence clause follow the `\u003ctools\u003e`/`\u003cenv\u003e`\nprecedent exactly: memory content is model-written from untrusted input and must\nbe framed as data, never instructions. The rendered block carries each record's\nshort id, because the write path addresses records by id — one source of truth\nfor both directions.\n\nBoth a per-scope record limit and a byte cap on the rendered block are named\nconstants. The cap is about size, not latency; a local read needs no timeout.\n\n**Prerequisite:** `setupGateways` runs before `setupMemoryAll`\n(`main.go` vs `:428`), and `startGatewayRuntime` before `setupMemory`\n(`gateway.go` vs `:115`), so the engine is nil at every turn-runner\nconstruction site. The ordering is the root-cause fix and nothing in either\nmemory step reads what those later steps produce.\n\n### 5. The chat write path\n\nThe memory tool is built **per turn**, not registered once at boot, because it\nmust write into a scope derived from the turn's resolved subject:\n\n| action | engine call |\n|---|---|\n| `create` | `Create` |\n| `update` | `Update` (by id) |\n| `delete` | `Forget` (by id) |\n| `list` | `List` |\n\n**The model chooses the scope kind; it never chooses the ids.** Agent and user\nids come from the turn's resolution and there is no schema field to put anything\nelse in — even a model that emits another user's id has nowhere to write it. That\nis the root-cause form of `agent-system.md`, matching this system's other\nnon-waivable boundaries (the model never runs git; gates cannot be waived by the\nagent). `global` is not offered at all.\n\nThe tool takes a record **id** where the retired provider took a content\nsubstring (`handleReplace`/`handleRemove`). The substring addressing existed\nbecause a markdown file was hand-edited and the model had no stable handle on a\nblock; with ids rendered into the prompt it disappears, and one addressing scheme\nreplaces two.\n\n### 6. The curator's write\n\n`sessioncurator.reviewOne` stops writing `Identity: sessionID` and writes\n`ScopeAgentUser{AgentID: \u003csession's agent\u003e, UserID: \u003cparticipant\u003e}`. Relationship\nscope is the only defensible home: agent-wide is explicitly unsafe for user facts\nand would leak user A's extracted facts to user B through the same agent;\nuser-wide would pool facts across agents; and a session id cannot survive the\nsession ending, which is the whole requirement.\n\nThis needs data the tree does not persist: the `messages` table\ngains `sender_id`, and `curator.SessionSummary` / `ConversationMessage` gain the\nagent and participant. The participant is derived from the session's own\nuser-role messages: exactly one distinct non-empty sender writes; zero (a\ndashboard or webhook session) or several (a group chat) writes nothing and\nrecords why. Ambiguity fails closed rather than guessing.\n\nThe payoff is the point of the exercise: chat-written and curator-derived memory\nnow occupy the same scopes, so a future turn's prompt contains both, and a\ncurator's bad note becomes id-addressable and therefore correctable by the agent.\n\n### 7. The content scanner moves to the engine\n\n`internal/memory/scanner.go` and its installation at `bootstrap.go` are what\nmake `configuration.md`'s \"Memory safety scanner\" row describe a control that\nactually runs. Deleting the package would silently remove prompt-injection\nscanning on memory writes. The scanner moves into the engine and is applied in\n`Create` and `Update`, which is the single choke point every producer crosses —\nso the curator's model-extracted content is covered too, not just the chat tool's.\n\n### 8. `internal/memory` is deleted\n\nNot shimmed, not partially, not behind a flag: `manager.go`, `provider.go`,\n`lifecycle.go`, `sync.go`, `scanner.go` (moved), `schema.go`, `backup.go`,\n`config.go`, `builtin/provider.go`, and their tests. Its markdown store\n(`builtin/store.go`) is the one survivor, relocated to\n`internal/infrastructure/memory/builtin/` — `organisation.md`'s \"a package owns\nits on-disk format end-to-end\" makes infrastructure its correct home, and the\nengine already parses that format.\n\nAlso deleted: `internal/tools/provider/memory`'s manager wrapping (rewired to the\nengine), `config.Memory.{Provider,ProviderConfig}` and the\n`validateMemory` rejection that exists only to complain about `Provider`\n(`validate.go`), and `webui.MemoryStatus`/`MemoryProviderHandle` — never\nimplemented in production, so `/api/memory` reports empty forever unless it is\nrepointed at the engine registry.\n\nExisting data at `\u003cworkDir\u003e/memory` is left on disk, unreferenced. No migration.\n`\u003cworkDir\u003e/memory-engine` is likewise orphaned: its directories were keyed by\n`sha256(sessionID)` and no scope key can hash to one.\n\n## What we deliberately do NOT do\n\n- **No cross-scope sharing or grants in this wave.** Promoting a record between\n scopes is a future explicit command, and `Update` cannot change scope, so it\n cannot happen by accident.\n- **No canonical user identity.** `SenderID` is channel-native, so one person on\n Telegram and email is two identities. Unifying needs an Identity aggregate that\n does not exist (`internal/domain` has no identity package and `IdentityID`\n appears nowhere in Go). The contract does not change when it lands.\n- **No ranked retrieval.** `Query.Text` stays, unused by the prompt path.\n- **No data migration**, per the decision above.\n- **No prompt-memory timeout mechanism.** The read is bounded by construction.\n- **No `Metadata` field.** It is write-only; a caller needing it later\n needs one field, not a speculative contract.\n\n## Acceptance criteria\n\n1. A record created through the memory tool in turn 1 appears in turn 2's prompt,\n asserted on content the test planted, with its id.\n2. The same record is **absent** from a different user's prompt on the same\n channel and agent. Isolation asserted end to end, not at the engine boundary.\n3. An agent-scope record is visible to both of that agent's users; a global\n record is visible to a second agent; an agent-user record is not.\n4. A curator pass over a session writes a record a later turn reads, and which\n survives that session being deleted.\n5. `create` → prompt shows it; `update` → prompt shows new content and\n `Revisions` returns the prior content with author and time; `delete` → gone\n from prompt and `List`. Round-trip is asserted as `parse(render(r)) == r`,\n never against a fixture string.\n6. A panicking engine and a failing engine both complete the turn with no memory\n block.\n7. A channel with no resolvable user renders only global and agent scopes, and\n the tool refuses `user`/`agent-user` with a reason. A webhook route path never\n becomes a user identity.\n8. A write that would exceed the file cap fails loudly rather than silently\n dropping content.\n9. `internal/memory` is gone, `task check` is green, and no document describes a\n manager that no longer exists.\n\nEvery criterion asserts behaviour or planted content. None asserts on the text of\na hand-written file, and none pins that a method was called. In particular none\nasserts the composition calls setup in a given order: if the ordering were\nswapped back, criteria 1 and 4 would fail, and that is the assertion.\n\n## Sequencing\n\nEach step is independently verifiable and lands green on its own.\n\n1. **Persist the participant.** `sender_id` column on `messages`,\n insert/select/migration, `curator.SessionSummary.AgentID` and\n `ConversationMessage.SenderID`. Touches no memory contract and is shippable\n alone — nothing consumes it yet.\n2. **Scope, CRUD and revisions.** The `internal/domain/memory` contract, the\n builtin engine with `HISTORY.md`, the store relocation, the scanner move, and\n the curator rewritten onto agent-user scope (it is the engine's only live\n consumer, so it belongs in this slice). *Depends on 1 for the curator half.*\n3. **Chat read path.** The `Memory` seam, the template block, subject resolution,\n and the composition-ordering fix. *Depends on 2.*\n4. **Chat write path.** The tool adapter onto CRUD and the turn's subject on the\n context. *Depends on 3, because a write is only observable through the read.*\n5. **Delete `internal/memory` and the dead config/dashboard surface**, and\n rewrite the \"Memory engine family\" section of\n `docs/architecture/plugins-and-extensions.md` to describe the\n engine family rather than a manager it deleted. *Depends on 3 and 4.*\n\nSteps 3 and 4 are separable from each other; step 1 is separable from everything.\n\n## Settles these open items in `migration-decisions.md` §5\n\n| §5 open item | Settled by |\n|---|---|\n| authoritative memory record and revision model | §1, §2 — the live marker becomes JSON, since the ratified `\u003c!--mem:\u003culid\u003e--\u003e` cannot carry provenance; superseded states live in a per-scope `HISTORY.md` |\n| provider and infrastructure boundaries | §2 — contract in `internal/domain/memory`, builtin format in `internal/infrastructure/memory/builtin` |\n| retrieval and access enforcement | §1 — caller-named scope sets, no engine-side policy |\n| provenance representation | §2 |\n\n## Not determined\n\n- **Group chats produce no derived memory**, because the curator fails closed on\n mixed-sender sessions. Whether extraction should instead become\n per-participant is a product call.\n- **Two processes, one scope file.** After step 3 both `archied` and\n `archie-gateway` hold an engine over the same root, and the engine caches a\n store per scope and rewrites the whole file, so concurrent writers on one scope\n can lose a block. `Update.Expected` makes that detectable for one record; it\n does not make the file safe. A reload-before-mutate or advisory lock is the fix\n and is not designed here.\n- **Unbounded history.** `HISTORY.md` grows without limit; no compaction policy.\n- **Size bounds.** `defaultMaxFileBytes` is now per scope rather than per\n identity; whether that is right is unexamined.\n- **Whether the dashboard should carry a user identity.** It has one bearer\n token, not users. Until it has users, web turns are correctly limited to global\n and agent scope.\n" }, { "url": "/dev/prds/messaging-service-boundary/", "title": "Messaging Service boundary and transport design", "section": "PRDs", - "body": "# Messaging Service boundary and transport design\n\n**Status:** Approved\n**Revision:** 1 \n**Date:** 2026-09-19 \n**Beads issue:** `archie-core-8cda.6.1` \n**Parent:** [`service-decomposition.md`](/dev/prds/service-decomposition/) \n**Prerequisites:** Gateway contract extraction (Phase 1), State Store extraction (Phase 2), UI extraction (Phase 3)\n\nThis document is the authoritative specification for Phase 4: Messaging Service extraction. It fixes the process boundary, configuration ownership, client contracts, security posture, readiness behavior, channel invariants, and deletion criteria required by `archie-core-8cda.6` and its child tasks.\n\n## Decision\n\nThe Messaging Service (`cmd/archie-messaging`, composed via `internal/app/archiemessaging`) owns external channel connections (Telegram, email, webhook) and dispatches inbound messages and commands to the standalone Gateway Service over gRPC via `messaging.ChatContract`.\n\nIt is a peer service to the UI Service and the daemon:\n- It serves external chat participants across configured communication platforms.\n- It consumes Gateway's `messaging.ChatContract` exclusively for chat turn execution, streaming, session queries, cancellation, persona selection, and task control actions.\n- It holds no direct database connections (`*store.Store` or SQLite), owns no LLM execution or model provider credentials, and has no dependency on the daemon runtime or the UI HTTP server.\n\n## Boundary and Ownership\n\n| Concern | Messaging Service owns | Messaging Service consumes | Messaging Service must not own or link |\n|---|---|---|---|\n| **Channel transports** | Telegram bot long-polling, Email IMAP/SMTP loops, Inbound Webhook HTTP listener | Gateway `ChatContract` | Direct LLM providers, model runtimes, tokenizer libraries |\n| **Turn dispatch** | Inbound message packaging, typing indicators, live streaming updates | `ChatContract.Route`, `ChatContract.Stream` | `gateway.Router`, `runtime.Runtime`, agent prompt builders |\n| **Sessions \u0026 History** | Platform-to-session key mapping, conversation thread ID translation | `ChatContract.GetSession`, `RecentMessages`, `RecentTurns` | `*store.Store`, `SessionStore` SQLite handles, direct database files |\n| **Control \u0026 Actions** | Platform command parsing (`/status`, `/tasks`, `/cancel`, `/stop`, `/persona`) | `ChatContract.Snapshot`, `ApplyTaskAction`, `Cancel`, `SetPersona` | `daemon.Daemon`, workflow execution engine, worktree operations |\n| **Task Approvals** | Channel interactive keyboards, approval callbacks | `ChatContract.ApplyTaskAction` | Direct task status mutations, forge comment posters |\n| **Configuration** | Platform bot tokens, allowed sender/user IDs, webhook listen ports | `[chat.*]` channel configs, Gateway target/token | `config.Holder`, forge tokens, repo configs, agent container specs |\n| **Readiness** | Messaging process liveness, channel connectivity, Gateway probe | Gateway `ChatContract.Snapshot` | Internal daemon subsystem probes |\n\n## Target Service Shape\n\n1. **Binary entry point**: `cmd/archie-messaging/main.go` parses CLI flags (`-config`, `-config-overlay`, `-gateway-target`, `-gateway-token`) and hands execution to `internal/app/archiemessaging.Run`.\n2. **Application composition**: `internal/app/archiemessaging/` wires:\n - Config loading scoped strictly to `[chat.*]` and `[services.gateway]`.\n - Secret resolution for channel credentials (e.g. Telegram bot token).\n - Gateway gRPC client (`internal/infrastructure/gatewayrpc.Client`).\n - Channel lifecycle managers (`internal/channels/status`).\n - Channel instances (`internal/channels/telegram`, `email`, `webhook`).\n3. **Turn and Stream Execution**:\n - Channels adapt inbound platform messages to `messaging.Inbound`.\n - Streaming channels (Telegram) consume `ChatContract.Stream(ctx, inbound)` which returns `\u003c-chan messaging.ChatEvent`, mapping `started`, `delta`, `tool`, `media`, `done`, and `error` events directly to live message frames.\n - Non-streaming channels (email, synchronous webhooks) consume `ChatContract.Route(ctx, inbound)`.\n\n## Load-Bearing Channel Invariants\n\nThe Messaging Service preserves existing invariants defined in `AGENTS.md` and channel documentation:\n1. **Telegram long-polling**: Must call `dropPendingUpdates(ctx, b)` before `b.Start(ctx)` and on `/restart`.\n2. **Telegram command menus**: Must register command menus simultaneously to `default`, `all_private_chats`, and `all_group_chats` via `commands.go` to prevent menu shadowing.\n3. **No UI detour**: Turns and task actions route directly to Gateway's `ChatContract`; no HTTP request may ever be sent to `archie-ui` or `internal/webui`.\n4. **Sender Confinement**: Telegram sender checks against `allowed_user_ids` must continue to fail closed before initiating any turn dispatch.\n\n## Configuration Ownership\n\n`cmd/archie-messaging` reads only:\n- `[services.gateway].target` and `[services.gateway].target_token` (or CLI flags).\n- `[chat.telegram]` (token, allowed user IDs).\n- `[chat.email]` (IMAP/SMTP host, ports, credentials).\n- `[chat.webhook]` (listen address, secret).\n- `[secrets]` (for resolving token secret references).\n\nIt does not read or decode `[forge]`, `[repos]`, `[models]`, `[providers]`, `[runners]`, `[containers]`, or `[nats]`.\n\n## Deletion Gate and Verification\n\nThe extraction is guarded by `cmd/archie-messaging/architecture_test.go`, verifying that `go list -deps ./cmd/archie-messaging` links zero banned runtime packages:\n- `modernc.org/sqlite` and `internal/store` (State Store)\n- `internal/daemon`, `internal/app/archied`, `internal/container`, `internal/worktree` (Daemon runtime)\n- `internal/domain/workflow`, `internal/agentexec`, `internal/taskrun` (Workflow engine)\n- `internal/forge`, `internal/forgerpc` (Forge client)\n- `internal/gateway` runtime (only `internal/domain/messaging` and `internal/infrastructure/gatewayrpc` are permitted)\n- `internal/webui` (Dashboard)\n" + "body": "# Messaging Service boundary and transport design\n\n**Status:** Approved\n**Revision:** 1 \n**Date:** 2026-09-19 \n**Beads issue:** `archie-core-8cda.6.1` \n**Parent:** [`service-decomposition.md`](/dev/prds/service-decomposition/) \n**Prerequisites:** Gateway contract extraction (Phase 1), State Store extraction (Phase 2), UI extraction (Phase 3)\n\nThis document is the authoritative specification for Phase 4: Messaging Service extraction. It fixes the process boundary, configuration ownership, client contracts, security posture, readiness behavior, channel invariants, and deletion criteria required by `archie-core-8cda.6` and its child tasks.\n\n## Decision\n\nThe Messaging Service (`cmd/archie-messaging`, composed via `internal/app/archiemessaging`) owns external channel connections (Telegram, email, webhook) and dispatches inbound messages and commands to the standalone Gateway Service over gRPC via `messaging.ChatContract`.\n\nIt is a peer service to the UI Service and the daemon:\n- It serves external chat participants across configured communication platforms.\n- It consumes Gateway's `messaging.ChatContract` exclusively for chat turn execution, streaming, session queries, cancellation, persona selection, and task control actions.\n- It holds no direct database connections, owns no LLM execution or model provider credentials, and has no dependency on the daemon runtime or the UI HTTP server.\n\n## Boundary and Ownership\n\n| Concern | Messaging Service owns | Messaging Service consumes | Messaging Service must not own or link |\n|---|---|---|---|\n| **Channel transports** | Telegram bot long-polling, Email IMAP/SMTP loops, Inbound Webhook HTTP listener | Gateway `ChatContract` | Direct LLM providers, model runtimes, tokenizer libraries |\n| **Turn dispatch** | Inbound message packaging, typing indicators, live streaming updates | `ChatContract.Route`, `ChatContract.Stream` | `gateway.Router`, `runtime.Runtime`, agent prompt builders |\n| **Sessions \u0026 History** | Platform-to-session key mapping, conversation thread ID translation | `ChatContract.GetSession`, `RecentMessages`, `RecentTurns` | store implementations, `SessionStore` database handles |\n| **Control \u0026 Actions** | Platform command parsing (`/status`, `/tasks`, `/cancel`, `/stop`, `/persona`) | `ChatContract.Snapshot`, `ApplyTaskAction`, `Cancel`, `SetPersona` | `daemon.Daemon`, workflow execution engine, worktree operations |\n| **Task Approvals** | Channel interactive keyboards, approval callbacks | `ChatContract.ApplyTaskAction` | Direct task status mutations, forge comment posters |\n| **Configuration** | Platform bot tokens, allowed sender/user IDs, webhook listen ports | `[chat.*]` channel configs, Gateway target/token | `config.Holder`, forge tokens, repo configs, agent container specs |\n| **Readiness** | Messaging process liveness, channel connectivity, Gateway probe | Gateway `ChatContract.Snapshot` | Internal daemon subsystem probes |\n\n## Target Service Shape\n\n1. **Binary entry point**: `cmd/archie-messaging/main.go` parses CLI flags (`-config`, `-config-overlay`, `-gateway-target`, `-gateway-token`) and hands execution to `internal/app/archiemessaging.Run`.\n2. **Application composition**: `internal/app/archiemessaging/` wires:\n - Config loading scoped strictly to `[chat.*]` and `[services.gateway]`.\n - Secret resolution for channel credentials (e.g. Telegram bot token).\n - Gateway gRPC client (`internal/infrastructure/gatewayrpc.Client`).\n - Channel lifecycle managers (`internal/channels/status`).\n - Channel instances (`internal/channels/telegram`, `email`, `webhook`).\n3. **Turn and Stream Execution**:\n - Channels adapt inbound platform messages to `messaging.Inbound`.\n - Streaming channels (Telegram) consume `ChatContract.Stream(ctx, inbound)` which returns `\u003c-chan messaging.ChatEvent`, mapping `started`, `delta`, `tool`, `media`, `done`, and `error` events directly to live message frames.\n - Non-streaming channels (email, synchronous webhooks) consume `ChatContract.Route(ctx, inbound)`.\n\n## Load-Bearing Channel Invariants\n\nThe Messaging Service preserves existing invariants defined in `AGENTS.md` and channel documentation:\n1. **Telegram long-polling**: Must call `dropPendingUpdates(ctx, b)` before `b.Start(ctx)` and on `/restart`.\n2. **Telegram command menus**: Must register command menus simultaneously to `default`, `all_private_chats`, and `all_group_chats` via `commands.go` to prevent menu shadowing.\n3. **No UI detour**: Turns and task actions route directly to Gateway's `ChatContract`; no HTTP request may ever be sent to `archie-ui` or `internal/webui`.\n4. **Sender Confinement**: Telegram sender checks against `allowed_user_ids` must continue to fail closed before initiating any turn dispatch.\n\n## Configuration Ownership\n\n`cmd/archie-messaging` reads only:\n- `[services.gateway].target` and `[services.gateway].target_token` (or CLI flags).\n- `[chat.telegram]` (token, allowed user IDs).\n- `[chat.email]` (IMAP/SMTP host, ports, credentials).\n- `[chat.webhook]` (listen address, secret).\n- `[secrets]` (for resolving token secret references).\n\nIt does not read or decode `[forge]`, `[repos]`, `[models]`, `[providers]`, `[runners]`, `[containers]`, or `[nats]`.\n\n## Deletion Gate and Verification\n\nThe extraction is guarded by `cmd/archie-messaging/architecture_test.go`, verifying that `go list -deps ./cmd/archie-messaging` links zero banned runtime packages:\n- `modernc.org/sqlite`, `github.com/jackc/pgx`, `internal/infrastructure/postgres` and `internal/infrastructure/legacyread` (State Store)\n- `internal/daemon`, `internal/app/archied`, `internal/container`, `internal/worktree` (Daemon runtime)\n- `internal/domain/workflow`, `internal/agentexec`, `internal/taskrun` (Workflow engine)\n- `internal/forge`, `internal/forgerpc` (Forge client)\n- `internal/gateway` runtime (only `internal/domain/messaging` and `internal/infrastructure/gatewayrpc` are permitted)\n- `internal/webui` (Dashboard)\n" }, { "url": "/dev/prds/module-position/", @@ -334,7 +334,7 @@ "url": "/dev/prds/payload-field-mapping/", "title": "Payload field mapping -- decision", "section": "PRDs", - "body": "# Payload field mapping -- decision\n\n**Status:** Approved\n**Date:** 2026-08-22\n**Beads issue:** `archie-core-t2db.3`, depends on `t2db.1`, blocks `t2db.4`\n\nAnswers what a \"mapping\" is, where it lives, and what \"resolve against a\nreal event\" means precisely enough to implement. `t2db.4` (binding: matcher\n+ mapping + workflow) is out of scope -- this bead ships the mapping\nconcept and its preview only, as a standalone entity `t2db.4` will later\nreference by ID.\n\n## Decision: a new `internal/domain/mapping` package, SQLite storage\n\n`internal/domain/mapping` owns the mapping vocabulary and the pure\nresolve/preview rule; `internal/store` persists it (mirrors\n`captures.go`'s shape exactly -- inline schema const, narrow store\ninterface, `*store.Store` implementation); `internal/webui/api_mapping.go`\nis the transport. This follows `docs/prds/event-capture-storage.md`'s own\nplacement precedent: `internal/store` is where this daemon's persistence\nalready lives, not `internal/infrastructure` (that layer holds\ncross-domain services -- config, eventbus, modelcatalog -- not this\ndaemon's primary store).\n\n### Why a domain package and not just a store table\n\nResolving a path against a payload and deciding \"missing vs null vs\ntype-changed\" is a rule, not a query -- CLAUDE.md's domain layer owns\n\"language, state, rules, operations\" precisely so this logic is testable\nwithout a database and reusable by `t2db.4`'s runtime trigger path without\neither package importing the other's transport concerns.\n\n## Mapping shape\n\n```go\ntype Mapping struct {\n\tID int64\n\tName string\n\tSourceHint string // free-text label (e.g. \"sentry\"), organisational only -- not enforced against Capture.Source\n\tFields []Field\n\tCreatedAt time.Time\n\tUpdatedAt time.Time\n}\n\ntype Field struct {\n\tName string // the name the workflow receives\n\tPath string // dot/bracket path into the payload, see below\n\tType FieldType // expected type, checked at resolve time\n\tRequired bool // Required=false + missing is not a failure; Required=false + wrong type still is\n}\n\ntype FieldType string // \"string\" | \"number\" | \"bool\" | \"object\" | \"array\" | \"any\"\n```\n\n`SourceHint` is advisory only: an operator authors a mapping by clicking\nthrough one example capture, but the mapping is not pinned to that\ncapture's exact source string, because `t2db.4`'s matcher (not this bead)\ndecides which events a binding applies to. Pinning here would duplicate\nthat decision in the wrong layer.\n\n## Path syntax\n\nDot-separated keys with `[N]` for array indices: `data.items[0].id`. No\nwildcards, no filters, no external library -- the grammar is exactly what a\nclick-through UI needs to express (this key, then this key, then this\nindex), and every path in a mapping was generated by clicking a value in a\nreal payload, never hand-written against a spec. A hand-rolled parser\n(~30 lines: split on `.`, strip `[N]` suffixes) is less risk than a new\ndependency for a grammar this small.\n\n## Resolve semantics (the preview AND the future runtime path)\n\n```go\nfunc Resolve(fields []Field, payload []byte) (values map[string]any, failures []Failure)\n\ntype Failure struct {\n\tFieldName string\n\tPath string\n\tReason string // \"missing\" | \"null\" | \"type_mismatch\"\n}\n```\n\nDecode with `json.Decoder.UseNumber()` (matches `webhookguard.RedactPayload`'s\nexisting convention) so int/float distinction survives for type checking.\nWalk each field's path independently -- one field's failure does not stop\nresolution of the others, because the whole point of preview is showing\nthe operator every problem at once, not the first one.\n\n- Path segment doesn't exist, or indexes past an array's length -\u003e `missing`\n- Path resolves to JSON `null` -\u003e `null`\n- Path resolves to a value whose JSON type doesn't match `Field.Type`\n (`\"any\"` always matches) -\u003e `type_mismatch`\n- `Required: false` suppresses `missing` only, never `null` or\n `type_mismatch` -- a field that's present but the wrong shape is always\n worth surfacing, since silently coercing it is exactly the \"agent\n confabulates around a blank field\" failure this bead exists to prevent.\n\n**This is the same function both consumers use.** `t2db.3`'s preview calls\nit against an operator-picked capture and renders every result, pass or\nfail, in the UI -- that satisfies this bead's \"fail loudly\" acceptance\ncriterion, because at this stage the only runtime consumer of a mapping\n*is* the human previewing it. `t2db.4` will call the identical function\nwhen a live event triggers a binding and turn any non-empty `failures` into\na visible operator-facing failure (park the task / surface in the\ninspector) instead of starting a workflow with blank fields -- that wiring\nis `t2db.4`'s job, not duplicated here.\n\n## Storage\n\n`internal/store/mappings.go`, `field_mappings` table: `id, name,\nsource_hint, fields_json, created_at, updated_at`. `fields_json` is the\n`[]Field` slice JSON-encoded into one column -- fields have no independent\nquery need (nothing filters \"mappings with a field named X\"), so a\njoin table would be structure with no consumer, the same failure mode\n`organisation.md` bans for empty layering.\n\n`MappingStore` interface (narrow, mirrors `CaptureStore`):\n`InsertMapping`, `ListMappings`, `GetMapping`, `UpdateMapping`,\n`DeleteMapping`.\n\n## HTTP surface\n\nAll under `/api/mappings`, token-gated like every other `/api/*` route (no\nbypass -- unlike capture, nothing here accepts unauthenticated input):\n\n- `GET /api/mappings` / `POST /api/mappings` -- list / create\n- `GET|PATCH|DELETE /api/mappings/{id}`\n- `POST /api/mappings/preview` -- body `{fields: []Field, capture_id:\n int64}`, calls `mapping.Resolve` against that capture's stored (already\n redacted) `Body` and returns `{values, failures}`. Deliberately takes\n loose `fields` rather than a saved mapping ID, so an operator can iterate\n before saving anything -- \"never ship a mapping the operator has not seen\n resolve against real data\" requires preview to work pre-save.\n\nPreview resolves against `CapturedEvent.Body`, which is already\nredaction-passed (`webhookguard.RedactPayload`, per\n`event-capture-storage.md`) -- a mapped field pointing at a\nsensitive-keyed path previews as the literal string `\"[redacted]\"`, not the\nreal value. This is accepted, not fixed here: it is the stored ground\ntruth, the same payload `t2db.4`'s runtime path will resolve against, and a\npreview mismatch would be worse than an honest one.\n\n## Frontend\n\nNew `ui/src/mappings/` feature folder, same three-file split as\n`ui/src/captures/`: `mappings.js` (page: list, fetch, create/edit flow),\n`mapping-editor.js` (pure, unit-testable: renders a captured payload as a\nclickable tree -- clicking a leaf appends a field row with `Path`\npre-filled -- plus the field table and preview-result rendering),\n`mappings.css`. Creating a mapping starts from an existing captured event\n(reuses `api.captures`), so the editor always has a real payload to click\nthrough and preview against, never a blank form.\n\n## What this does not decide\n\nThe binding entity, its matcher, its draft/pending_approval/armed state\nmachine, and turning a runtime `Resolve` failure into a parked task are\n`t2db.4`. Per-source HMAC secret registration is `t2db.5`. This bead ships\nthe mapping concept, its storage, and its preview only.\n" + "body": "# Payload field mapping -- decision\n\n**Status:** Approved\n**Date:** 2026-08-22\n**Beads issue:** `archie-core-t2db.3`, depends on `t2db.1`, blocks `t2db.4`\n\nAnswers what a \"mapping\" is, where it lives, and what \"resolve against a\nreal event\" means precisely enough to implement. `t2db.4` (binding: matcher\n+ mapping + workflow) is out of scope -- this bead ships the mapping\nconcept and its preview only, as a standalone entity `t2db.4` will later\nreference by ID.\n\n## Decision: a new `internal/domain/mapping` package, SQLite storage\n\n\u003e Storage superseded: every store now lives in PostgreSQL behind\n\u003e `internal/domain/storecontract`, implemented in `internal/infrastructure/postgres`\n\u003e (`docs/prds/state-store-contract.md`). SQLite names below describe the\n\u003e original design.\n\n`internal/domain/mapping` owns the mapping vocabulary and the pure\nresolve/preview rule; `internal/store` persists it (mirrors\n`captures.go`'s shape exactly -- inline schema const, narrow store\ninterface, `*store.Store` implementation); `internal/webui/api_mapping.go`\nis the transport. This follows `docs/prds/event-capture-storage.md`'s own\nplacement precedent: `internal/store` is where this daemon's persistence\nalready lives, not `internal/infrastructure` (that layer holds\ncross-domain services -- config, eventbus, modelcatalog -- not this\ndaemon's primary store).\n\n### Why a domain package and not just a store table\n\nResolving a path against a payload and deciding \"missing vs null vs\ntype-changed\" is a rule, not a query -- CLAUDE.md's domain layer owns\n\"language, state, rules, operations\" precisely so this logic is testable\nwithout a database and reusable by `t2db.4`'s runtime trigger path without\neither package importing the other's transport concerns.\n\n## Mapping shape\n\n```go\ntype Mapping struct {\n\tID int64\n\tName string\n\tSourceHint string // free-text label (e.g. \"sentry\"), organisational only -- not enforced against Capture.Source\n\tFields []Field\n\tCreatedAt time.Time\n\tUpdatedAt time.Time\n}\n\ntype Field struct {\n\tName string // the name the workflow receives\n\tPath string // dot/bracket path into the payload, see below\n\tType FieldType // expected type, checked at resolve time\n\tRequired bool // Required=false + missing is not a failure; Required=false + wrong type still is\n}\n\ntype FieldType string // \"string\" | \"number\" | \"bool\" | \"object\" | \"array\" | \"any\"\n```\n\n`SourceHint` is advisory only: an operator authors a mapping by clicking\nthrough one example capture, but the mapping is not pinned to that\ncapture's exact source string, because `t2db.4`'s matcher (not this bead)\ndecides which events a binding applies to. Pinning here would duplicate\nthat decision in the wrong layer.\n\n## Path syntax\n\nDot-separated keys with `[N]` for array indices: `data.items[0].id`. No\nwildcards, no filters, no external library -- the grammar is exactly what a\nclick-through UI needs to express (this key, then this key, then this\nindex), and every path in a mapping was generated by clicking a value in a\nreal payload, never hand-written against a spec. A hand-rolled parser\n(~30 lines: split on `.`, strip `[N]` suffixes) is less risk than a new\ndependency for a grammar this small.\n\n## Resolve semantics (the preview AND the future runtime path)\n\n```go\nfunc Resolve(fields []Field, payload []byte) (values map[string]any, failures []Failure)\n\ntype Failure struct {\n\tFieldName string\n\tPath string\n\tReason string // \"missing\" | \"null\" | \"type_mismatch\"\n}\n```\n\nDecode with `json.Decoder.UseNumber()` (matches `webhookguard.RedactPayload`'s\nexisting convention) so int/float distinction survives for type checking.\nWalk each field's path independently -- one field's failure does not stop\nresolution of the others, because the whole point of preview is showing\nthe operator every problem at once, not the first one.\n\n- Path segment doesn't exist, or indexes past an array's length -\u003e `missing`\n- Path resolves to JSON `null` -\u003e `null`\n- Path resolves to a value whose JSON type doesn't match `Field.Type`\n (`\"any\"` always matches) -\u003e `type_mismatch`\n- `Required: false` suppresses `missing` only, never `null` or\n `type_mismatch` -- a field that's present but the wrong shape is always\n worth surfacing, since silently coercing it is exactly the \"agent\n confabulates around a blank field\" failure this bead exists to prevent.\n\n**This is the same function both consumers use.** `t2db.3`'s preview calls\nit against an operator-picked capture and renders every result, pass or\nfail, in the UI -- that satisfies this bead's \"fail loudly\" acceptance\ncriterion, because at this stage the only runtime consumer of a mapping\n*is* the human previewing it. `t2db.4` will call the identical function\nwhen a live event triggers a binding and turn any non-empty `failures` into\na visible operator-facing failure (park the task / surface in the\ninspector) instead of starting a workflow with blank fields -- that wiring\nis `t2db.4`'s job, not duplicated here.\n\n## Storage\n\nThe `mappings` table: `id, name,\nsource_hint, fields_json, created_at, updated_at`. `fields_json` is the\n`[]Field` slice JSON-encoded into one column -- fields have no independent\nquery need (nothing filters \"mappings with a field named X\"), so a\njoin table would be structure with no consumer, the same failure mode\n`organisation.md` bans for empty layering.\n\n`MappingStore` interface (narrow, mirrors `CaptureStore`):\n`InsertMapping`, `ListMappings`, `GetMapping`, `UpdateMapping`,\n`DeleteMapping`.\n\n## HTTP surface\n\nAll under `/api/mappings`, token-gated like every other `/api/*` route (no\nbypass -- unlike capture, nothing here accepts unauthenticated input):\n\n- `GET /api/mappings` / `POST /api/mappings` -- list / create\n- `GET|PATCH|DELETE /api/mappings/{id}`\n- `POST /api/mappings/preview` -- body `{fields: []Field, capture_id:\n int64}`, calls `mapping.Resolve` against that capture's stored (already\n redacted) `Body` and returns `{values, failures}`. Deliberately takes\n loose `fields` rather than a saved mapping ID, so an operator can iterate\n before saving anything -- \"never ship a mapping the operator has not seen\n resolve against real data\" requires preview to work pre-save.\n\nPreview resolves against `CapturedEvent.Body`, which is already\nredaction-passed (`webhookguard.RedactPayload`, per\n`event-capture-storage.md`) -- a mapped field pointing at a\nsensitive-keyed path previews as the literal string `\"[redacted]\"`, not the\nreal value. This is accepted, not fixed here: it is the stored ground\ntruth, the same payload `t2db.4`'s runtime path will resolve against, and a\npreview mismatch would be worse than an honest one.\n\n## Frontend\n\nNew `ui/src/mappings/` feature folder, same three-file split as\n`ui/src/captures/`: `mappings.js` (page: list, fetch, create/edit flow),\n`mapping-editor.js` (pure, unit-testable: renders a captured payload as a\nclickable tree -- clicking a leaf appends a field row with `Path`\npre-filled -- plus the field table and preview-result rendering),\n`mappings.css`. Creating a mapping starts from an existing captured event\n(reuses `api.captures`), so the editor always has a real payload to click\nthrough and preview against, never a blank form.\n\n## What this does not decide\n\nThe binding entity, its matcher, its draft/pending_approval/armed state\nmachine, and turning a runtime `Resolve` failure into a parked task are\n`t2db.4`. Per-source HMAC secret registration is `t2db.5`. This bead ships\nthe mapping concept, its storage, and its preview only.\n" }, { "url": "/dev/prds/playbook-expression-syntax/", @@ -346,7 +346,7 @@ "url": "/dev/prds/pr-review-remediation/", "title": "PR review comment remediation -- decision", "section": "PRDs", - "body": "# PR review comment remediation -- decision\n\n**Status:** Approved\n**Date:** 2026-09-10, amended and ratified 2026-09-19\n**Beads issue:** (filed as `archie-core-…`, parent: `archie-core-7d5u` event sources and typed reactions)\n\nAnswers: how archie reacts to review comments on a PR it opened, from a human\nor from another review bot, and remediates them. Built on two settled decisions — the adversarial review\nepic's isolation contract (`docs/prds/adversarial-self-review.md`) and the\nevent-sources decision (`docs/prds/event-sources-and-reactions.md`) — plus the\nwebhook intake security contract (`docs/prds/webhook-intake-security.md`).\nNothing here re-decides those; it names the one concrete reaction each one's\nmachinery was built to carry.\n\n## Problem\n\nArchie opens a PR (`pr_open`) and then never learns what a human said about\nit. `forge.Forge` has no method to list PR reviews or review comments; the\ndaemon poll loop finds only new issues; the forge webhook receiver\n(`internal/forge/webhook/receiver.go`) decodes only `issues` events and\nexplicitly ignores `pull_request` events. A \"requested changes\" review, or an\ninline comment on a line, is invisible work.\n\n## Decisions\n\n### 1. Read surface: a narrow `PullRequestReviewReader`, type-asserted\n\nA new forge interface, implemented by GitHub and Gitea, not folded into the\nfat `Forge` interface (the h019.8 `PullRequestReader` precedent — the noop\nforge cannot honestly provide it):\n\n```go\ntype ReviewComment struct { ID int64; Author, Body, Path string; Line int; InReplyTo int64; CreatedAt time.Time }\ntype Review struct { ID int64; Author, State string /* requested_changes | approved | commented | dismissed */; SubmittedAt time.Time }\n\ntype PullRequestReviewReader interface {\n ListReviews(ctx, owner, repo string, number, sinceID int64) ([]Review, error)\n ListReviewComments(ctx, owner, repo string, number, sinceID int64) ([]ReviewComment, error)\n // ReplyToReview posts a reply to a review or comment thread.\n ReplyToReview(ctx, owner, repo string, number, inReplyTo int64, body string) error\n}\n```\n\nGitHub: `pulls/{n}/reviews` + `pulls/{n}/comments`; reply via\n`pulls/{n}/comments/{id}/replies`. Gitea: `pulls/{n}/reviews` +\n`pulls/{n}/comments`; reply via `pulls/{n}/comments/{id}`. `sinceID` is the\ncursor for both poll and webhook dedup.\n\n### 2. Event source: webhook first, poll as the universal backstop\n\n- **Webhook (GitHub, low latency):** extend `receiver.go` to also decode\n `pull_request_review` and `pull_request_review_comment` events, behind the\n existing HMAC verification, producing the same reaction as the poller. The\n receiver stays a forge-intake surface — not the generic chat webhook, not a\n new generic receiver (7d5u decision 4).\n- **Poll (both forges, no webhook dependency):** a periodic scan of `pr_open`\n tasks calls `ListReviews`/`ListReviewComments` with a persisted `sinceID`\n cursor per task. This is the Gitea path and the deployment-without-webhook\n path; it must stay first-class, not a legacy fallback (the repo's own\n webhook-intake rule). It is **one shared scan over all `pr_open` tasks**,\n not a scan inside each identity's poll loop, and it calls the forge client\n the scanned task's own identity owns. This is `reconcilePRs`'s shape\n exactly (`internal/daemon/daemon.go`: `Store.OpenPRs` then `forgeFor(\u0026t)`).\n A per-identity scan would call one identity's forge client against another\n identity's PR.\n\nBoth paths emit the **same** typed reaction, so `PublishUnique` dedups them\nfor free — the `TaskEnvelope.IdempotencyKey` pattern (7d5u decision 4) applied\nto reviews.\n\n### 3. Reaction contract: producer-only, resolves to an existing task\n\nA new `workintake.ReviewCommentEnvelope{Owner, Repo, PRNumber, Kind, CommentID,\nReviewID, Author, State, Body, Path, Line}`. Idempotency key is\n`owner/repo/pr-number/kind/id` — source-independent, so a webhook delivery\nand a poll of the same comment collapse.\n\n`Kind` (`review` or `comment`) is part of the key because a review ID and a\nreview-comment ID come from independent forge sequences. Keying on the bare id\nlets a review and an inline comment on the same PR collide, silently dropping\none of the two. This is a producer-only reaction\n(7d5u decision 3): it never vetoes or mutates in-flight work; the target task\nis idle (`pr_open`).\n\nThe daemon resolves `PRNumber` → the task that owns it. An unmatched PR (not\nan archie task) is dropped with a counter — a public webhook must not be able\nto start an agent against an arbitrary repository. A PR whose task is merged,\nclosed or archived is dropped through the same counter: the reaction is only\never work against a PR archie itself opened *and still owns*.\n\nThat lookup does not exist. `pr_number` is written by\n`internal/store/store.go` and never read in a `WHERE` clause, so this needs a\n`TaskStore` method keyed on `(owner, repo, pr_number)`, a matching\n`StateStoreService` RPC (a Go interface method on a store facade without one\nis forbidden, `CLAUDE.md`), and an index on those three columns. It is the\nauthorization boundary, so it is sequenced as its own step below rather than\ncarried along with the envelope.\n\n**Identity is derived from the resolved task, never carried in the reaction.**\nThe task row already stores `identity` (`internal/store/store.go`), and it is\nthe resolved task's identity that selects the forge client, worktree manager,\nrepo config and `bot_user` for the remediation run and the `ReplyToReview`\nreply, through the existing `forgeFor`/`treesFor`/`repoFor` helpers. That is\nwhat decision 5's \"the identity that owns the task\" means, and it is the only\nplace identity enters this path.\n\nIdentity is deliberately **not** added to the envelope, the idempotency key or\nthe index:\n\n- A PR number is unique within an `owner/repo`, and the tasks table is keyed\n `UNIQUE(owner, repo, issue_number)` with no host or identity column, so\n archie already treats `owner/repo` as globally unique across identities. A\n `(owner, repo, pr_number)` lookup cannot return a task other than the one\n that owns that PR; there is no cross-identity resolution to guard against.\n- The webhook producer has no identity to carry: webhook intake refuses to\n start when `[[identities]]` is configured (`setupForgeWebhook`,\n `internal/app/archied/bootstrap.go`).\n- Keying on a field only one of the two producers can populate is precisely\n the delivery-source-specific key `CLAUDE.md` forbids, and it would break the\n webhook/poll dedup the key exists for. An index column that never appears in\n the `WHERE` clause is dead weight.\n\n### 4. Remediation: a `remediate` workflow on the existing branch\n\nA new workflow that runs when a reaction resolves to an archie-owned PR. It is\nnot a new branch/PR — it reuses the task's prepared worktree, refreshes it\nonto the PR branch (the `worktree.Manager.refresh` idempotency), and:\n\n1. Runs a builder agent whose mission is the review's actionable comments\n (each with body, path, line): \"Address these review comments with the\n smallest change; run the gate.\" One agent run per review, per decision 5. The toolset is the normal builder toolset — edit + gate + (never\n git; the orchestrator commits/pushes) — not the reviewer's read-only set.\n2. Runs the repo gate (`[[repos.gate]]`) and `diff_cap_lines` — the same\n enforcement as any other work (webhook-intake-security point 4; no\n relaxation for review-triggered edits).\n3. Commits and pushes to the **same** PR branch.\n4. Replies to the review/comment via `ReplyToReview`, summarising what\n changed.\n\nThe comment is injected through the task record (e.g. `Task.Notes` or a\ndedicated review field), not through a prompt-only path, so the reaction is\nrecoverable and auditable.\n\n### 5. Routing and lifecycle\n\nA **review** is the unit of remediation, never an individual comment.\n\n- `requested_changes` review → one remediation run addressing every actionable\n comment in that review.\n- `commented` review with actionable inline comments → one remediation run\n addressing every actionable comment in that review.\n- `approved` review → no remediation. Record the review state on the task;\n do **not** auto-merge — merging stays an operator action (the dashboard/chat\n `/approve` precedent). Auto-merge is a separate decision if ever wanted.\n\nOne run per review, strictly. A review carrying eight inline comments arrives\nas one `pull_request_review` event plus eight `pull_request_review_comment`\nevents; remediating each separately means eight builder runs, eight gate runs\nand eight pushes to answer one review. The comment events are collected\nagainst their parent `ReviewID` and remediated together. A standalone comment\nwith no parent review is its own unit of one.\n\nRemediation runs for a single task are **serialised**, whatever\n`[[repos]].allow_concurrent` says. Two runs against one PR branch would race\non the shared worktree and on the push; `allow_concurrent` is about unrelated\ntasks in a repo, not about one task reacting to itself.\n\n#### Whose comments count\n\nOnly archie's **own** comments are excluded, matched on the forge identity\nthat owns the task (`bot_user`, or the identity's `bot_user` under\n`[[identities]]`), so its `ReplyToReview` replies do not retrigger it.\n\nEvery other author is actionable, including other bots. The external review\nbot on this repository is the main source of findings on an archie-opened PR,\nand discarding comments because their author is a bot would discard the review\nthis feature exists to act on. There is no \"is a bot\" test anywhere in this\npath.\n\n(The original draft justified a broader exclusion by citing a `RepliesAfter`\n`exclude` precedent. No such mechanism exists in the tree; the citation is\nwithdrawn.)\n\n#### Bounding the exchange\n\nBecause bot comments are actionable, the exchange can run away without a human\never touching it: archie pushes a remediation, the review bot re-reviews the\nnew commit, comments, archie remediates again. Nothing in the reaction path\nterminates that on its own.\n\nA task therefore carries a remediation round count, bounded by the repo's\nexisting `max_retries`. On the cap, archie stops remediating, posts one\ncomment saying it has stopped and why, and parks the task for an operator.\nThis is a hard stop, not a backoff: the failure it guards against is\nunbounded spend, so slowing it down does not address it.\n\n### 6. Security posture\n\nThe webhook reaction is not a public no-code binding, so it does not pass\nthrough the `draft → pending_approval → armed` gate (webhook-intake-security\npoint 2) — but every other constraint holds: HMAC (already in the receiver),\nper-source rate limiting (add when the receiver gains a second event family),\nand blast radius bounded by the existing gate/sandbox/worktree enforcement,\n**plus** the new archie-owned-task check (decision 3), which is the actual\nauthorization boundary: a review event only ever causes work against a PR\narchie itself opened and still owns.\n\nAccepting other bots' comments (decision 5) does not widen that boundary: an\nauthor is never a reason to act, only the owned-task check is. It does widen\nthe *spend* surface, which the round cap bounds.\n\n### 7. Sequencing\n\n1. **The reaction stream.** `ARCHIE_REACTIONS`, under the fan-out\n `jetstream.LimitsPolicy` the parent decision requires\n (`event-sources-and-reactions.md` §2). `Config.Retention` is\n parameterised for exactly this (`archie-core-7d5u.2`), but no reaction\n stream exists and `ARCHIE_TASKS` carries `archie.task.\u003e` only, so every\n publish to `archie.reaction.*` fails against no matching\n stream. This step is first because nothing after it can be exercised\n without it.\n2. `PullRequestReviewReader` + GitHub/Gitea implementations + `ReplyToReview`.\n *(Landed.)*\n3. `ReviewCommentEnvelope` + the kinded idempotency key.\n4. **The owned-task guard**: `(owner, repo, pr_number)` lookup, its\n `StateStoreService` RPC, its index, and the drop-with-counter path for an\n unmatched, merged, closed or archived PR. Nothing may consume a reaction\n before this exists.\n5. `remediate` workflow: worktree refresh, one run per review, builder, gate,\n push, reply, round cap.\n6. Webhook receiver extension (`pull_request_review` / `_comment` events).\n7. Poll backstop with the persisted `sinceID` cursor.\n8. Tests + `task check`.\n\nSteps 6 and 7 are the producers and come last on purpose. Building a producer\nbefore its consumer and its authorization guard yields a path that publishes\ninto nothing, and becomes an unguarded intake the moment a stream appears.\n\n## Out of scope\n\n- Auto-merge on approval (separate decision).\n- Reacting to comments on PRs archie did not open.\n- The calibration epic's confirm-before-surfacing (h019.5) — orthogonal.\n" + "body": "# PR review comment remediation -- decision\n\n**Status:** Approved\n**Date:** 2026-09-10, amended and ratified 2026-09-19\n**Beads issue:** (filed as `archie-core-…`, parent: `archie-core-7d5u` event sources and typed reactions)\n\nAnswers: how archie reacts to review comments on a PR it opened, from a human\nor from another review bot, and remediates them. Built on two settled decisions — the adversarial review\nepic's isolation contract (`docs/prds/adversarial-self-review.md`) and the\nevent-sources decision (`docs/prds/event-sources-and-reactions.md`) — plus the\nwebhook intake security contract (`docs/prds/webhook-intake-security.md`).\nNothing here re-decides those; it names the one concrete reaction each one's\nmachinery was built to carry.\n\n## Problem\n\nArchie opens a PR (`pr_open`) and then never learns what a human said about\nit. `forge.Forge` has no method to list PR reviews or review comments; the\ndaemon poll loop finds only new issues; the forge webhook receiver\n(`internal/forge/webhook/receiver.go`) decodes only `issues` events and\nexplicitly ignores `pull_request` events. A \"requested changes\" review, or an\ninline comment on a line, is invisible work.\n\n## Decisions\n\n### 1. Read surface: a narrow `PullRequestReviewReader`, type-asserted\n\nA new forge interface, implemented by GitHub and Gitea, not folded into the\nfat `Forge` interface (the h019.8 `PullRequestReader` precedent — the noop\nforge cannot honestly provide it):\n\n```go\ntype ReviewComment struct { ID int64; Author, Body, Path string; Line int; InReplyTo int64; CreatedAt time.Time }\ntype Review struct { ID int64; Author, State string /* requested_changes | approved | commented | dismissed */; SubmittedAt time.Time }\n\ntype PullRequestReviewReader interface {\n ListReviews(ctx, owner, repo string, number, sinceID int64) ([]Review, error)\n ListReviewComments(ctx, owner, repo string, number, sinceID int64) ([]ReviewComment, error)\n // ReplyToReview posts a reply to a review or comment thread.\n ReplyToReview(ctx, owner, repo string, number, inReplyTo int64, body string) error\n}\n```\n\nGitHub: `pulls/{n}/reviews` + `pulls/{n}/comments`; reply via\n`pulls/{n}/comments/{id}/replies`. Gitea: `pulls/{n}/reviews` +\n`pulls/{n}/comments`; reply via `pulls/{n}/comments/{id}`. `sinceID` is the\ncursor for both poll and webhook dedup.\n\n### 2. Event source: webhook first, poll as the universal backstop\n\n- **Webhook (GitHub, low latency):** extend `receiver.go` to also decode\n `pull_request_review` and `pull_request_review_comment` events, behind the\n existing HMAC verification, producing the same reaction as the poller. The\n receiver stays a forge-intake surface — not the generic chat webhook, not a\n new generic receiver (7d5u decision 4).\n- **Poll (both forges, no webhook dependency):** a periodic scan of `pr_open`\n tasks calls `ListReviews`/`ListReviewComments` with a persisted `sinceID`\n cursor per task. This is the Gitea path and the deployment-without-webhook\n path; it must stay first-class, not a legacy fallback (the repo's own\n webhook-intake rule). It is **one shared scan over all `pr_open` tasks**,\n not a scan inside each identity's poll loop, and it calls the forge client\n the scanned task's own identity owns. This is `reconcilePRs`'s shape\n exactly (`internal/daemon/daemon.go`: `Store.OpenPRs` then `forgeFor(\u0026t)`).\n A per-identity scan would call one identity's forge client against another\n identity's PR.\n\nBoth paths emit the **same** typed reaction, so `PublishUnique` dedups them\nfor free — the `TaskEnvelope.IdempotencyKey` pattern (7d5u decision 4) applied\nto reviews.\n\n### 3. Reaction contract: producer-only, resolves to an existing task\n\nA new `workintake.ReviewCommentEnvelope{Owner, Repo, PRNumber, Kind, CommentID,\nReviewID, Author, State, Body, Path, Line}`. Idempotency key is\n`owner/repo/pr-number/kind/id` — source-independent, so a webhook delivery\nand a poll of the same comment collapse.\n\n`Kind` (`review` or `comment`) is part of the key because a review ID and a\nreview-comment ID come from independent forge sequences. Keying on the bare id\nlets a review and an inline comment on the same PR collide, silently dropping\none of the two. This is a producer-only reaction\n(7d5u decision 3): it never vetoes or mutates in-flight work; the target task\nis idle (`pr_open`).\n\nThe daemon resolves `PRNumber` → the task that owns it. An unmatched PR (not\nan archie task) is dropped with a counter — a public webhook must not be able\nto start an agent against an arbitrary repository. A PR whose task is merged,\nclosed or archived is dropped through the same counter: the reaction is only\never work against a PR archie itself opened *and still owns*.\n\nThat lookup does not exist. `pr_number` is written by\n`internal/infrastructure/postgres/store.go` and never read in a `WHERE` clause, so this needs a\n`TaskStore` method keyed on `(owner, repo, pr_number)`, a matching\n`StateStoreService` RPC (a Go interface method on a store facade without one\nis forbidden, `CLAUDE.md`), and an index on those three columns. It is the\nauthorization boundary, so it is sequenced as its own step below rather than\ncarried along with the envelope.\n\n**Identity is derived from the resolved task, never carried in the reaction.**\nThe task row already stores `identity` (`internal/infrastructure/postgres/store.go`), and it is\nthe resolved task's identity that selects the forge client, worktree manager,\nrepo config and `bot_user` for the remediation run and the `ReplyToReview`\nreply, through the existing `forgeFor`/`treesFor`/`repoFor` helpers. That is\nwhat decision 5's \"the identity that owns the task\" means, and it is the only\nplace identity enters this path.\n\nIdentity is deliberately **not** added to the envelope, the idempotency key or\nthe index:\n\n- A PR number is unique within an `owner/repo`, and the tasks table is keyed\n `UNIQUE(owner, repo, issue_number)` with no host or identity column, so\n archie already treats `owner/repo` as globally unique across identities. A\n `(owner, repo, pr_number)` lookup cannot return a task other than the one\n that owns that PR; there is no cross-identity resolution to guard against.\n- The webhook producer has no identity to carry: webhook intake refuses to\n start when `[[identities]]` is configured (`setupForgeWebhook`,\n `internal/app/archied/bootstrap.go`).\n- Keying on a field only one of the two producers can populate is precisely\n the delivery-source-specific key `CLAUDE.md` forbids, and it would break the\n webhook/poll dedup the key exists for. An index column that never appears in\n the `WHERE` clause is dead weight.\n\n### 4. Remediation: a `remediate` workflow on the existing branch\n\nA new workflow that runs when a reaction resolves to an archie-owned PR. It is\nnot a new branch/PR — it reuses the task's prepared worktree, refreshes it\nonto the PR branch (the `worktree.Manager.refresh` idempotency), and:\n\n1. Runs a builder agent whose mission is the review's actionable comments\n (each with body, path, line): \"Address these review comments with the\n smallest change; run the gate.\" One agent run per review, per decision 5. The toolset is the normal builder toolset — edit + gate + (never\n git; the orchestrator commits/pushes) — not the reviewer's read-only set.\n2. Runs the repo gate (`[[repos.gate]]`) and `diff_cap_lines` — the same\n enforcement as any other work (webhook-intake-security point 4; no\n relaxation for review-triggered edits).\n3. Commits and pushes to the **same** PR branch.\n4. Replies to the review/comment via `ReplyToReview`, summarising what\n changed.\n\nThe comment is injected through the task record (e.g. `Task.Notes` or a\ndedicated review field), not through a prompt-only path, so the reaction is\nrecoverable and auditable.\n\n### 5. Routing and lifecycle\n\nA **review** is the unit of remediation, never an individual comment.\n\n- `requested_changes` review → one remediation run addressing every actionable\n comment in that review.\n- `commented` review with actionable inline comments → one remediation run\n addressing every actionable comment in that review.\n- `approved` review → no remediation. Record the review state on the task;\n do **not** auto-merge — merging stays an operator action (the dashboard/chat\n `/approve` precedent). Auto-merge is a separate decision if ever wanted.\n\nOne run per review, strictly. A review carrying eight inline comments arrives\nas one `pull_request_review` event plus eight `pull_request_review_comment`\nevents; remediating each separately means eight builder runs, eight gate runs\nand eight pushes to answer one review. The comment events are collected\nagainst their parent `ReviewID` and remediated together. A standalone comment\nwith no parent review is its own unit of one.\n\nRemediation runs for a single task are **serialised**, whatever\n`[[repos]].allow_concurrent` says. Two runs against one PR branch would race\non the shared worktree and on the push; `allow_concurrent` is about unrelated\ntasks in a repo, not about one task reacting to itself.\n\n#### Whose comments count\n\nOnly archie's **own** comments are excluded, matched on the forge identity\nthat owns the task (`bot_user`, or the identity's `bot_user` under\n`[[identities]]`), so its `ReplyToReview` replies do not retrigger it.\n\nEvery other author is actionable, including other bots. The external review\nbot on this repository is the main source of findings on an archie-opened PR,\nand discarding comments because their author is a bot would discard the review\nthis feature exists to act on. There is no \"is a bot\" test anywhere in this\npath.\n\n(The original draft justified a broader exclusion by citing a `RepliesAfter`\n`exclude` precedent. No such mechanism exists in the tree; the citation is\nwithdrawn.)\n\n#### Bounding the exchange\n\nBecause bot comments are actionable, the exchange can run away without a human\never touching it: archie pushes a remediation, the review bot re-reviews the\nnew commit, comments, archie remediates again. Nothing in the reaction path\nterminates that on its own.\n\nA task therefore carries a remediation round count, bounded by the repo's\nexisting `max_retries`. On the cap, archie stops remediating, posts one\ncomment saying it has stopped and why, and parks the task for an operator.\nThis is a hard stop, not a backoff: the failure it guards against is\nunbounded spend, so slowing it down does not address it.\n\n### 6. Security posture\n\nThe webhook reaction is not a public no-code binding, so it does not pass\nthrough the `draft → pending_approval → armed` gate (webhook-intake-security\npoint 2) — but every other constraint holds: HMAC (already in the receiver),\nper-source rate limiting (add when the receiver gains a second event family),\nand blast radius bounded by the existing gate/sandbox/worktree enforcement,\n**plus** the new archie-owned-task check (decision 3), which is the actual\nauthorization boundary: a review event only ever causes work against a PR\narchie itself opened and still owns.\n\nAccepting other bots' comments (decision 5) does not widen that boundary: an\nauthor is never a reason to act, only the owned-task check is. It does widen\nthe *spend* surface, which the round cap bounds.\n\n### 7. Sequencing\n\n1. **The reaction stream.** `ARCHIE_REACTIONS`, under the fan-out\n `jetstream.LimitsPolicy` the parent decision requires\n (`event-sources-and-reactions.md` §2). `Config.Retention` is\n parameterised for exactly this (`archie-core-7d5u.2`), but no reaction\n stream exists and `ARCHIE_TASKS` carries `archie.task.\u003e` only, so every\n publish to `archie.reaction.*` fails against no matching\n stream. This step is first because nothing after it can be exercised\n without it.\n2. `PullRequestReviewReader` + GitHub/Gitea implementations + `ReplyToReview`.\n *(Landed.)*\n3. `ReviewCommentEnvelope` + the kinded idempotency key.\n4. **The owned-task guard**: `(owner, repo, pr_number)` lookup, its\n `StateStoreService` RPC, its index, and the drop-with-counter path for an\n unmatched, merged, closed or archived PR. Nothing may consume a reaction\n before this exists.\n5. `remediate` workflow: worktree refresh, one run per review, builder, gate,\n push, reply, round cap.\n6. Webhook receiver extension (`pull_request_review` / `_comment` events).\n7. Poll backstop with the persisted `sinceID` cursor.\n8. Tests + `task check`.\n\nSteps 6 and 7 are the producers and come last on purpose. Building a producer\nbefore its consumer and its authorization guard yields a path that publishes\ninto nothing, and becomes an unguarded intake the moment a stream appears.\n\n## Out of scope\n\n- Auto-merge on approval (separate decision).\n- Reacting to comments on PRs archie did not open.\n- The calibration epic's confirm-before-surfacing (h019.5) — orthogonal.\n" }, { "url": "/dev/prds/progressive-tool-disclosure/", @@ -388,7 +388,7 @@ "url": "/dev/prds/service-decomposition/", "title": "Service decomposition -- decision", "section": "PRDs", - "body": "# Service decomposition -- decision\n\n**Status:** Draft\n**Date:** 2026-09-05\n**Beads epic:** archie-core-8cda\n**Prior art:** `docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md`,\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`,\n`docs/inspiration/service-discovery-research-2026-09-05.md`,\n`/work/artefacts/archie-core-current-state-archify.html`,\n`/work/artefacts/archie-core-ideal-state-archify.html`\n\nThe Phase 3 UI boundary and configuration decision is ratified in\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/). That focused document is\nthe implementation authority for `archie-core-8cda.5.2`; this parent remains\nthe broader service-distribution proposal.\n\n## Decision\n\narchie-core moves from one Go binary (`archied`) hosting every domain\nin-process to a set of independently deployable services, each exposing\nexactly one typed contract. Consumers depend on the contract; they never\nreach into another service's internal types, session state, or config.\ngRPC is the synchronous transport; NATS remains for asynchronous fan-out\n(status, telemetry) rather than being replaced.\n\nThis is a **distribution decision**, not a rewrite of Archie's domain\nmodel. The six-domain split already approved in\n`docs/architecture/index.md` and `docs/architecture/organisation.md`\n(Identity, Agent, Messaging, Workflow, Work Intake, Plugin) is unchanged;\nthis PRD is about how the *processes* running that domain logic get\npackaged, contracted, and discovered -- not about redrawing domain\nboundaries.\n\n### Target services\n\n| Service | Contract it owns | Existing code it replaces/wraps |\n|---|---|---|\n| UI Service | thin client only; talks to Gateway and State Store contracts | `internal/webui` (SPA hosting stays; API surface becomes typed contract clients, not a peer with direct access) |\n| Gateway Service | routing / turn / session contract | `internal/gateway` |\n| Messaging Service | channel-plugin contract (Telegram, email, Discord, Teams, ...) | `internal/channels` |\n| Curator Service | curator contract; **optional**, installs as a Gateway plugin | `internal/domain/curator` |\n| Scheduler Service | cron/ticker contract | `internal/domain/scheduling` |\n| Execution Environment | execution-dispatch contract | `internal/domain/workflow`, task dispatch paths |\n| Runner | isolated worktree/container execution | `internal/container`, `internal/agentexec`, `internal/worktree` |\n| State Store | state-access contract | `internal/store` |\n\n### Discovery\n\nPer `docs/inspiration/service-discovery-research-2026-09-05.md`:\n\n- **Primary:** Kubernetes-native discovery (Service + headless-Service/DNS\n + EndpointSlice, served by CoreDNS), delivered as a Helm chart + Operator,\n on k3s for single-host installs and a real cluster for multi-host or a\n hosted offering.\n- **Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV\n registry) for no-K8s single-host installs, with a custom gRPC resolver\n reading `host:port` from the KV bucket.\n- Both implementations sit behind one internal interface:\n\n ```go\n type ServiceRegistry interface {\n Resolve(service string) []Endpoint\n Watch(service string) \u003c-chan Event // join/leave\n }\n ```\n\n- An optional service (Curator; others later) that is not installed\n must resolve to a distinct `NotInstalled` outcome, never a `Down`/\n `Unhealthy` one. Under K8s this falls out of DNS returning no record;\n under the NATS fallback it requires an explicit \"installed\" marker in the\n KV registry, not just absence-implies-down.\n\n### Distribution model this supports\n\nBoth self-hosted (customer runs the Helm chart or the NATS-fallback\nbinaries themselves) and a managed/hosted offering (we run the same Helm\nchart) are in scope -- this is the RHEL / Ansible Automation Platform\nmodel: one architecture, sold both ways. This PRD does not decide the\nbusiness model; it decides that the architecture must not foreclose\neither shape.\n\n## Why this shape\n\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`\nestablished that Archie's existing domain split already does what AWX's\nservice-decomposition post argues for -- boundaries exist. What didn't\nexist was deployability: `internal/webui` was not a passive frontend. The\nre-baselined coupling inventory is below, and the ratified boundary is\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\n\n- Telegram task actions now call Gateway's `ChatContract` directly through\n `chatTaskActorAdapter`; the old Telegram-to-webui HTTP detour is historical.\n- `internal/webui/api_chat.go` now holds `gateway.ChatContract`; direct\n `SessionStore`, `Router`, and `TurnHistory` access is historical.\n- The third coupling is closed. UI extraction shipped (`archie-core-8cda.5`,\n closed 2026-09-11): `cmd/archie-ui` serves the dashboard against remote\n Gateway and State Store contracts, holds no `config.Holder`, and relinks no\n daemon runtime (`cmd/archie-ui/architecture_test.go` enforces that on every\n `task check`). The `b.web.Cfg = b.d.Cfg` alias left in `buildDaemon` is now\n inside the daemon's own process -- it renders the published `ConfigView`\n snapshot and holds the readiness probes -- so it no longer crosses a service\n boundary. That cleanup is `archie-core-ml30` (P2); the UI boundary is\n ratified in [`ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\n\n## Non-goals\n\n- Redesigning the domain model (Identity/Agent/Messaging/Workflow/Work\n Intake/Plugin boundaries are settled elsewhere and unchanged).\n- Replacing NATS. It remains the async transport; this PRD only adds gRPC\n for synchronous request/response and (optionally) K8s-native discovery.\n- Deciding the business model (self-hosted vs. managed vs. both). Kept\n open deliberately; the architecture must serve either.\n- Config relocation (`config.toml` -\u003e ConfigMaps/Secrets) is real work\n implied by a K8s-native deployment but is its own decision, not decided\n here. Flagged in the discovery research as a phased concern.\n- Any concrete adoption of Pi (or another external agent harness) as an\n execution runner. Raised in conversation alongside this decomposition,\n but it is a separate decision (agent execution as a strict data\n boundary, `docs/architecture/agent-system.md`) and does not gate or get gated by this PRD.\n\n## Open questions\n\nAll ten are resolved; each carries its resolution inline. Three phases have\nsince shipped against them (Gateway, State Store, UI), so the framing here is\nhistorical record rather than an outstanding gate.\n\n1. **Extraction order.** Which service gets pulled out of the monolith\n first? The webui entanglement findings above suggest Gateway Service's\n `ChatContract` should exist before UI Service is extracted, since UI is\n the most entangled consumer. Not yet sequenced.\n\n **RESOLVED.** Gateway Service first, then State Store,\n then UI, then Messaging, then Execution/Runner/Scheduler, then Curator\n (optional) — preceded by an in-process \"contract seams\" (modular\n monolith) phase. Confirms the hypothesis and strengthens it: the\n Gateway is already store-decoupled (it uses narrow contract adapters,\n not `internal/store`) and already owns an isolated session SQLite, so it\n is both the stickiest capability *and* the one with a ready seam and\n isolated data — extractable alone, first. Full phase list and reasoning:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#1`.\n2. **Contract definition mechanism.** Protobuf/gRPC service definitions\n need a home (a new `proto/` or `contracts/` tree, versioning scheme,\n codegen wiring into `Taskfile.yml`). Not yet designed.\n\n **RESOLVED.** buf v2 driving `protoc-gen-go` +\n `protoc-gen-go-grpc` (grpc-go, not connect-go — connect-go's net/http\n client breaks the discovery research's gRPC resolver plumbing for K8s\n DNS and NATS-KV). One committed `proto/\u003cservice\u003e/v1/` tree, generated\n Go committed to `internal/contracts/\u003cservice\u003e/v1/`, directory\n versioning (breaking change -\u003e new `vN`, enforced by `buf breaking` in\n the gate), and `task proto:generate`/`proto:lint`/`proto:check` wired\n into `task check` and mirrored into `.github/workflows/deploy.yml`\n (CI does not run `task check`). Full toolchain comparison, layout\n rationale, and scratch-verified `buf`/`buf breaking`/`connect-go`\n results:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#2`.\n3. **In-process vs. out-of-process during migration.** Whether services\n can run multiplexed inside one binary behind the same `ServiceRegistry`\n interface during a transition period (so extraction is incremental and\n testable) or whether each service must be a separate binary from day\n one. Strongly recommend the former for a solo maintainer, but not\n decided here.\n\n **RESOLVED.** Multiplexed in-process is the migration\n default -- this is `docs/architecture/organisation.md`'s standing\n \"Process boundaries\" rule, not a new lean. Mechanism: one wire-safe Go\n contract interface per service; a local adapter and a generated gRPC\n client adapter, both satisfying the same interface via a compile-time\n assertion (the `var _ workflow.Store = (*Client)(nil)` idiom;\n `store.WorkflowStore` is superseded by the consumer-owned\n `workflow.Store`); composition (not `ServiceRegistry`, which stays\n network-only) picks the adapter by the presence of a\n `[services.\u003cname\u003e].target` address -- empty = local (in-process) adapter,\n set = dial the remote gRPC client. This replaced the earlier\n `mode = \"inproc\" | \"remote\"` enum, which was dropped (see the gateway's\n `services.gateway.target`). The adapter is chosen by presence of the\n target; Phase 2 then deleted every local fallback, so remote is now the\n only mode and an empty `[services.state].target` is a startup error\n (docs/prds/state-store-contract.md §12 step 7). A\n per-contract conformance suite runs against both adapters over\n `bufconn`; the flip to a real process is a same-commit deletion of the\n in-process path, no dual-live window. Consequence: Gateway's Phase 0\n (`ChatContract` seam) ships in-process first, with the network protocol\n staying out of the production path until Phase 1 -- no amendment to Q1's\n phase list needed. Full mechanism and evidence:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#3`\n (the State Store's concrete ratified config shape is\n `docs/prds/state-store-contract.md` §10).\n4. **State Store's contract shape.** Whether every service gets its own\n storage (true microservice isolation) or several services share the\n State Store service's contract (less isolation, less migration risk).\n Not yet decided.\n\n **RESOLVED.** One State Store service owns the existing\n single SQLite file behind narrow typed contracts (`TaskStore`,\n `CaptureStore`, `MappingStore`, `BindingStore`, `WorkflowStore`) over\n gRPC. *(Ratified 2026-09-06: the workflow contract is now the\n domain-owned `workflow.Store`, which supersedes\n `internal/store`'s `WorkflowStore`. See note below and\n `docs/prds/state-store-contract.md`.)*\n No physical database-per-service: the solo-maintainer / team-\n topology rule of thumb plus the low value of per-service autonomy for\n one person rule it out, and one store means no distributed-transactions\n problem. The Gateway keeps its own already-separate session SQLite.\n Concrete migration path (generalise `storerpc`, swap consumers one\n contract at a time, stand up the store service, split the file only on\n evidence):\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#4`.\n\n **RATIFIED (2026-09-06, beads archie-core-8cda.4.1).** The authoritative\n contract boundary \u0026 transport design for this is fixed in\n `docs/prds/state-store-contract.md` (rev. 2c): one `StateStore` gRPC service,\n 42 RPCs; **ownership split** — consumer-owned `workflow.Store` +\n `workflow.Task`/`Status`/`Source` in the workflow domain (dependency rules\n #2/#7), producer-owned daemon/webui store surfaces in `internal/store`;\n the `Task`/`Status`/`Source` + interface relocation is a Phase 2\n prerequisite pulled from migration-decisions §4 (`store.Task →\n WorkflowExecution`); `RecordDispatch` drops `*sql.Tx`; `Close` off-wire;\n structured gRPC error codes with sanitised details; explicit agent State\n Store gRPC handoff (`STATE_STORE_URL`/`STATE_STORE_TOKEN`); loopback+bearer\n transport-security boundary; presence-based `[services.state].target`\n defaulting local; migration order `workflow.Store -\u003e archie-state-store -\u003e\n Capture -\u003e Mapping -\u003e Binding -\u003e TaskStore`. `.4.2` and `.4.3` implement\n against that doc.\n5. **Helm chart / Operator ownership.** Net-new work with no existing\n analog in this repository (`deployments/` holds TOML profiles\n and a `docker-compose.yml`, not Kubernetes manifests). Scoping not yet\n started.\n\n **RESOLVED.** Three charts: an `archie-library` chart\n (shared Deployment/Service templates, named ports), an `archie` chart\n for the 7 core services, and an `archie-curator` chart + Curator\n Operator (CRD, controller, RBAC). The Curator reconciler is a\n level-triggered loop that fetches the CR, builds Deployment + Service,\n sets owner refs, applies idempotently (`CreateOrUpdate`), watches via\n `Owns()`, and reports a `Ready` status condition. Smallest real\n milestone (M1): install a Curator CR → Operator creates Deployment +\n Service → CoreDNS serves `curator` → Gateway's resolver flips it from\n `NotInstalled` to Installed with no redeploy; uninstall flips it back.\n Full scoping and first bead-issue list:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#5`.\n\n## Completion criteria\n\nNot applicable yet -- this PRD records the target shape and the discovery\ndecision. It does not authorize implementation. Before work begins: file\na beads epic, resolve the open questions above (at minimum #1-#3), and\nget explicit go-ahead given the scale of this change relative to\n`AGENTS.md`'s \"smallest workable change\" default for this solo project.\n" + "body": "# Service decomposition -- decision\n\n**Status:** Draft\n**Date:** 2026-09-05\n**Beads epic:** archie-core-8cda\n**Prior art:** `docs/inspiration/2026-09-05-awx-service-decomposition-and-ansible-task-model.md`,\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`,\n`docs/inspiration/service-discovery-research-2026-09-05.md`,\n`/work/artefacts/archie-core-current-state-archify.html`,\n`/work/artefacts/archie-core-ideal-state-archify.html`\n\nThe Phase 3 UI boundary and configuration decision is ratified in\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/). That focused document is\nthe implementation authority for `archie-core-8cda.5.2`; this parent remains\nthe broader service-distribution proposal.\n\n## Decision\n\narchie-core moves from one Go binary (`archied`) hosting every domain\nin-process to a set of independently deployable services, each exposing\nexactly one typed contract. Consumers depend on the contract; they never\nreach into another service's internal types, session state, or config.\ngRPC is the synchronous transport; NATS remains for asynchronous fan-out\n(status, telemetry) rather than being replaced.\n\nThis is a **distribution decision**, not a rewrite of Archie's domain\nmodel. The six-domain split already approved in\n`docs/architecture/index.md` and `docs/architecture/organisation.md`\n(Identity, Agent, Messaging, Workflow, Work Intake, Plugin) is unchanged;\nthis PRD is about how the *processes* running that domain logic get\npackaged, contracted, and discovered -- not about redrawing domain\nboundaries.\n\n### Target services\n\n| Service | Contract it owns | Existing code it replaces/wraps |\n|---|---|---|\n| UI Service | thin client only; talks to Gateway and State Store contracts | `internal/webui` (SPA hosting stays; API surface becomes typed contract clients, not a peer with direct access) |\n| Gateway Service | routing / turn / session contract | `internal/gateway` |\n| Messaging Service | channel-plugin contract (Telegram, email, Discord, Teams, ...) | `internal/channels` |\n| Curator Service | curator contract; **optional**, installs as a Gateway plugin | `internal/domain/curator` |\n| Scheduler Service | cron/ticker contract | `internal/domain/scheduling` |\n| Execution Environment | execution-dispatch contract | `internal/domain/workflow`, task dispatch paths |\n| Runner | isolated worktree/container execution | `internal/container`, `internal/agentexec`, `internal/worktree` |\n| State Store | state-access contract | `internal/infrastructure/postgres` |\n\n### Discovery\n\nPer `docs/inspiration/service-discovery-research-2026-09-05.md`:\n\n- **Primary:** Kubernetes-native discovery (Service + headless-Service/DNS\n + EndpointSlice, served by CoreDNS), delivered as a Helm chart + Operator,\n on k3s for single-host installs and a real cluster for multi-host or a\n hosted offering.\n- **Fallback:** NATS-based discovery (micro `$SRV` announce + JetStream KV\n registry) for no-K8s single-host installs, with a custom gRPC resolver\n reading `host:port` from the KV bucket.\n- Both implementations sit behind one internal interface:\n\n ```go\n type ServiceRegistry interface {\n Resolve(service string) []Endpoint\n Watch(service string) \u003c-chan Event // join/leave\n }\n ```\n\n- An optional service (Curator; others later) that is not installed\n must resolve to a distinct `NotInstalled` outcome, never a `Down`/\n `Unhealthy` one. Under K8s this falls out of DNS returning no record;\n under the NATS fallback it requires an explicit \"installed\" marker in the\n KV registry, not just absence-implies-down.\n\n### Distribution model this supports\n\nBoth self-hosted (customer runs the Helm chart or the NATS-fallback\nbinaries themselves) and a managed/hosted offering (we run the same Helm\nchart) are in scope -- this is the RHEL / Ansible Automation Platform\nmodel: one architecture, sold both ways. This PRD does not decide the\nbusiness model; it decides that the architecture must not foreclose\neither shape.\n\n## Why this shape\n\n`docs/inspiration/module-reference-awx-decomposition-vs-archie.md`\nestablished that Archie's existing domain split already does what AWX's\nservice-decomposition post argues for -- boundaries exist. What didn't\nexist was deployability: `internal/webui` was not a passive frontend. The\nre-baselined coupling inventory is below, and the ratified boundary is\n[`ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\n\n- Telegram task actions now call Gateway's `ChatContract` directly through\n `chatTaskActorAdapter`; the old Telegram-to-webui HTTP detour is historical.\n- `internal/webui/api_chat.go` now holds `gateway.ChatContract`; direct\n `SessionStore`, `Router`, and `TurnHistory` access is historical.\n- The third coupling is closed. UI extraction shipped (`archie-core-8cda.5`,\n closed 2026-09-11): `cmd/archie-ui` serves the dashboard against remote\n Gateway and State Store contracts, holds no `config.Holder`, and relinks no\n daemon runtime (`cmd/archie-ui/architecture_test.go` enforces that on every\n `task check`). The `b.web.Cfg = b.d.Cfg` alias left in `buildDaemon` is now\n inside the daemon's own process -- it renders the published `ConfigView`\n snapshot and holds the readiness probes -- so it no longer crosses a service\n boundary. That cleanup is `archie-core-ml30` (P2); the UI boundary is\n ratified in [`ui-service-boundary.md`](/dev/prds/ui-service-boundary/).\n\n## Non-goals\n\n- Redesigning the domain model (Identity/Agent/Messaging/Workflow/Work\n Intake/Plugin boundaries are settled elsewhere and unchanged).\n- Replacing NATS. It remains the async transport; this PRD only adds gRPC\n for synchronous request/response and (optionally) K8s-native discovery.\n- Deciding the business model (self-hosted vs. managed vs. both). Kept\n open deliberately; the architecture must serve either.\n- Config relocation (`config.toml` -\u003e ConfigMaps/Secrets) is real work\n implied by a K8s-native deployment but is its own decision, not decided\n here. Flagged in the discovery research as a phased concern.\n- Any concrete adoption of Pi (or another external agent harness) as an\n execution runner. Raised in conversation alongside this decomposition,\n but it is a separate decision (agent execution as a strict data\n boundary, `docs/architecture/agent-system.md`) and does not gate or get gated by this PRD.\n\n## Open questions\n\nAll ten are resolved; each carries its resolution inline. Three phases have\nsince shipped against them (Gateway, State Store, UI), so the framing here is\nhistorical record rather than an outstanding gate.\n\n1. **Extraction order.** Which service gets pulled out of the monolith\n first? The webui entanglement findings above suggest Gateway Service's\n `ChatContract` should exist before UI Service is extracted, since UI is\n the most entangled consumer. Not yet sequenced.\n\n **RESOLVED.** Gateway Service first, then State Store,\n then UI, then Messaging, then Execution/Runner/Scheduler, then Curator\n (optional) — preceded by an in-process \"contract seams\" (modular\n monolith) phase. Confirms the hypothesis and strengthens it: the\n Gateway is already store-decoupled (it uses narrow contract adapters,\n not `internal/store`) and already owns an isolated session SQLite, so it\n is both the stickiest capability *and* the one with a ready seam and\n isolated data — extractable alone, first. Full phase list and reasoning:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#1`.\n2. **Contract definition mechanism.** Protobuf/gRPC service definitions\n need a home (a new `proto/` or `contracts/` tree, versioning scheme,\n codegen wiring into `Taskfile.yml`). Not yet designed.\n\n **RESOLVED.** buf v2 driving `protoc-gen-go` +\n `protoc-gen-go-grpc` (grpc-go, not connect-go — connect-go's net/http\n client breaks the discovery research's gRPC resolver plumbing for K8s\n DNS and NATS-KV). One committed `proto/\u003cservice\u003e/v1/` tree, generated\n Go committed to `internal/contracts/\u003cservice\u003e/v1/`, directory\n versioning (breaking change -\u003e new `vN`, enforced by `buf breaking` in\n the gate), and `task proto:generate`/`proto:lint`/`proto:check` wired\n into `task check` and mirrored into `.github/workflows/deploy.yml`\n (CI does not run `task check`). Full toolchain comparison, layout\n rationale, and scratch-verified `buf`/`buf breaking`/`connect-go`\n results:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#2`.\n3. **In-process vs. out-of-process during migration.** Whether services\n can run multiplexed inside one binary behind the same `ServiceRegistry`\n interface during a transition period (so extraction is incremental and\n testable) or whether each service must be a separate binary from day\n one. Strongly recommend the former for a solo maintainer, but not\n decided here.\n\n **RESOLVED.** Multiplexed in-process is the migration\n default -- this is `docs/architecture/organisation.md`'s standing\n \"Process boundaries\" rule, not a new lean. Mechanism: one wire-safe Go\n contract interface per service; a local adapter and a generated gRPC\n client adapter, both satisfying the same interface via a compile-time\n assertion (the `var _ workflow.Store = (*Client)(nil)` idiom;\n `store.WorkflowStore` is superseded by the consumer-owned\n `workflow.Store`); composition (not `ServiceRegistry`, which stays\n network-only) picks the adapter by the presence of a\n `[services.\u003cname\u003e].target` address -- empty = local (in-process) adapter,\n set = dial the remote gRPC client. This replaced the earlier\n `mode = \"inproc\" | \"remote\"` enum, which was dropped (see the gateway's\n `services.gateway.target`). The adapter is chosen by presence of the\n target; Phase 2 then deleted every local fallback, so remote is now the\n only mode and an empty `[services.state].target` is a startup error\n (docs/prds/state-store-contract.md §12 step 7). A\n per-contract conformance suite runs against both adapters over\n `bufconn`; the flip to a real process is a same-commit deletion of the\n in-process path, no dual-live window. Consequence: Gateway's Phase 0\n (`ChatContract` seam) ships in-process first, with the network protocol\n staying out of the production path until Phase 1 -- no amendment to Q1's\n phase list needed. Full mechanism and evidence:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#3`\n (the State Store's concrete ratified config shape is\n `docs/prds/state-store-contract.md` §10).\n4. **State Store's contract shape.** Whether every service gets its own\n storage (true microservice isolation) or several services share the\n State Store service's contract (less isolation, less migration risk).\n Not yet decided.\n\n **RESOLVED.** One State Store service owns the task data\n (now PostgreSQL, see `docs/prds/state-store-contract.md`) behind narrow typed contracts (`TaskStore`,\n `CaptureStore`, `MappingStore`, `BindingStore`, `WorkflowStore`) over\n gRPC. *(Ratified 2026-09-06: the workflow contract is now the\n domain-owned `workflow.Store`, which supersedes\n `internal/store`'s `WorkflowStore`. See note below and\n `docs/prds/state-store-contract.md`.)*\n No physical database-per-service: the solo-maintainer / team-\n topology rule of thumb plus the low value of per-service autonomy for\n one person rule it out, and one store means no distributed-transactions\n problem.\n Concrete migration path (generalise `storerpc`, swap consumers one\n contract at a time, stand up the store service, split the file only on\n evidence):\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#4`.\n\n **RATIFIED (2026-09-06, beads archie-core-8cda.4.1).** The authoritative\n contract boundary \u0026 transport design for this is fixed in\n `docs/prds/state-store-contract.md` (rev. 2c): one `StateStore` gRPC service,\n 42 RPCs; **ownership split** — consumer-owned `workflow.Store` +\n `workflow.Task`/`Status`/`Source` in the workflow domain (dependency rules\n #2/#7), producer-owned daemon/webui store surfaces in `internal/domain/storecontract`;\n the `Task`/`Status`/`Source` + interface relocation is a Phase 2\n prerequisite pulled from migration-decisions §4 (`store.Task →\n WorkflowExecution`); `RecordDispatch` drops `*sql.Tx`; `Close` off-wire;\n structured gRPC error codes with sanitised details; explicit agent State\n Store gRPC handoff (`STATE_STORE_URL`/`STATE_STORE_TOKEN`); loopback+bearer\n transport-security boundary; presence-based `[services.state].target`\n defaulting local; migration order `workflow.Store -\u003e archie-state-store -\u003e\n Capture -\u003e Mapping -\u003e Binding -\u003e TaskStore`. `.4.2` and `.4.3` implement\n against that doc.\n5. **Helm chart / Operator ownership.** Net-new work with no existing\n analog in this repository (`deployments/` holds TOML profiles\n and a `docker-compose.yml`, not Kubernetes manifests). Scoping not yet\n started.\n\n **RESOLVED.** Three charts: an `archie-library` chart\n (shared Deployment/Service templates, named ports), an `archie` chart\n for the 7 core services, and an `archie-curator` chart + Curator\n Operator (CRD, controller, RBAC). The Curator reconciler is a\n level-triggered loop that fetches the CR, builds Deployment + Service,\n sets owner refs, applies idempotently (`CreateOrUpdate`), watches via\n `Owns()`, and reports a `Ready` status condition. Smallest real\n milestone (M1): install a Curator CR → Operator creates Deployment +\n Service → CoreDNS serves `curator` → Gateway's resolver flips it from\n `NotInstalled` to Installed with no redeploy; uninstall flips it back.\n Full scoping and first bead-issue list:\n `docs/inspiration/service-decomposition-open-questions-research-2026-09-05.md#5`.\n\n## Completion criteria\n\nNot applicable yet -- this PRD records the target shape and the discovery\ndecision. It does not authorize implementation. Before work begins: file\na beads epic, resolve the open questions above (at minimum #1-#3), and\nget explicit go-ahead given the scale of this change relative to\n`AGENTS.md`'s \"smallest workable change\" default for this solo project.\n" }, { "url": "/dev/prds/service-registry/", @@ -412,7 +412,7 @@ "url": "/dev/prds/state-store-contract/", "title": "State Store — contract boundary \u0026 transport (ratification)", "section": "PRDs", - "body": "# State Store — contract boundary \u0026 transport (ratification)\n\n**Status:** Approved\n**Revision:** 2e\n**Date:** 2026-09-06\n\nThis is the pre-implementation design document that Phase 2 subtasks `.4.2` (generalize\n`storerpc` as the State Store gRPC contract) and `.4.3` (stand up `archie-state-store`)\nimplement against.\n**Beads milestone:** archie-core-8cda.4.1\n**Parent:** `docs/prds/service-decomposition.md` (open-question Q4 **RESOLVED**)\n\nThis document fixes the **contract boundary** and **transport** for the State Store service.\nIt does not redesign Archie's domain model, does not invent a second store, and does **not**\nsplit `archie.db`. It records method ownership, DTO/wire representation, error semantics,\nlimits, mode/config shape, local-adapter compatibility, and cutover/deletion rules so that\n`.4.2` and `.4.3` have exactly one authoritative surface to build against.\n\n\u003e **Current state — read before `.4.2` starts.** The §12 step 2 relocation (`archie-core-8cda.\n\u003e 4.8`) has landed: `Task`/`Status*`/`Source*` and the 3-method `workflow.Store` now live in\n\u003e `internal/domain/workflow`, the six production workflow files import no `internal/store`\n\u003e symbol, and `storerpc`/the NATS transport assert against `workflow.Store` instead of\n\u003e `store.WorkflowStore`. The broader Phase 2 acceptance criterion (State Store as its own\n\u003e service behind gRPC) is still open — that is what `.4.2`/`.4.3` build next.\n\n\u003e **Relationship to migration-decisions §4 — not replaced.** `docs/architecture/\n\u003e migration-decisions.md` §4 (`store.Task → WorkflowExecution`, `mutable Stage →\n\u003e StepExecution history`, `RetryCount/Attempt → Attempt records`, versioned Workflow defs)\n\u003e **remains the authoritative, open document for the long-term Workflow migration.** This PRD\n\u003e does **not** replace it. It only pulls the minimal `Task`/`Status`/`Source` type + interface\n\u003e relocation forward as a Phase 2 prerequisite (because the acceptance criterion requires it);\n\u003e the rest of §4 stays its own migration and stays open.\n\n---\n\n## 1. Decision (the invariant everything follows)\n\nArchie is composed such that a single **`archie-state-store`** service owns the one existing\nSQLite file (`archie.db`) and exposes it over **gRPC behind narrow typed contracts**. There is\nexactly one store, one file, one service. No database-per-service, no distributed\ntransactions, no second store abstraction.\n\nTwo adapter families satisfy the contract interfaces:\n\n- **Local adapter** — `*store.Store` (the in-process SQLite implementation).\n- **Remote adapter** — `*staterpc.Client` (wraps the generated gRPC client).\n\nEach adapter carries an explicit compile-time assertion against the contract interfaces it\nserves. Composition picks local vs. remote by the presence of `[services.state].target`\n(empty = local; set = dial the gRPC service). Default is **local** (the State Store is not yet\nextracted), which is the PRD §3 posture; it differs from the gateway's remote-default only\nbecause the gateway is already a separate deployment and the State Store is not.\n\n\u003e **Contract-ownership fork, decided (rev. 2c — resolves the domain contradiction).**\n\u003e Ownership splits by the consumer's *layer*, per the dependency contract's rule #2\n\u003e (\"a domain defines the smallest interfaces required to perform its work\") and rule #7\n\u003e (\"database rows … MUST NOT leak into domain models\"):\n\u003e\n\u003e - The **workflow domain** (`internal/domain/workflow`) is the only domain-layer consumer of\n\u003e store state. It defines its own **consumer-owned** narrow `Store` interface\n\u003e (Update/Transition/InsertEvent) and its own **domain-owned** `Task` type plus\n\u003e `Status`/`Source` vocabulary. It imports **zero** `internal/store` symbols — satisfying the\n\u003e Phase 2 acceptance criterion. This is the minimal bound of migration-decisions §4\n\u003e (`store.Task → WorkflowExecution`), pulled forward **because** the acceptance criterion\n\u003e requires it.\n\u003e - The **daemon / webui / intake** store surfaces (`TaskStore`, `CaptureStore`,\n\u003e `MappingStore`, `BindingStore`, `BindingDispatcher`, `BindingTaskCreator`) stay\n\u003e **producer-owned** in `internal/store`, because those consumers are application and\n\u003e infrastructure layers, not domains — rule #2 does not apply to them.\n\nSo the State Store contract is **not** uniformly producer-owned: the domain-facing workflow\ncontract is consumer-owned (in the domain), while the daemon/webui store surfaces remain in\n`internal/store`. `internal/store` imports `internal/domain/workflow` for the `Task`/`Status`/\n`Source` types it persists (persistence → domain, the correct direction), and its `Store`\nimplementation satisfies `workflow.Store`. This is the single decision that makes the whole\nPhase 2 acceptance criterion and the architecture's rules simultaneously satisfiable.\n\n\u003e **`internal/store` is a TRANSITIONAL compatibility location (rev. 2c).** The repository's target\n\u003e architecture (`docs/architecture/dependencies-and-contracts.md`, `organisation.md`) places\n\u003e persistence implementations under `internal/infrastructure/\u003ccapability\u003e`. `internal/store`\n\u003e holding the task/capture/mapping/binding implementation and ALSO being the home of the\n\u003e producer-owned daemon/webui interfaces is acceptable **only as a transitional location**. The\n\u003e long-term home for the store implementation is `internal/infrastructure/store`; the\n\u003e producer-owned interfaces may move to the same infrastructure package (or an approved\n\u003e contract location) when the store implementation relocates. Agents must not treat\n\u003e `internal/store` as the final infrastructure boundary.\n\u003e\n\u003e **REV. 2d (archie-core-8cda.5.6): the producer-owned interfaces HAVE moved to an approved\n\u003e contract location** — `internal/domain/storecontract` (package `storecontract`), ahead of the\n\u003e store implementation's own relocation. The UI Service deletion gate required the archie-ui\n\u003e process to hold the contracts without linking `internal/store`'s SQL implementation, and\n\u003e `internal/contracts/` itself is ruled out: that tree is the `buf generate` output and\n\u003e `tools/proto.sh check` deletes hand-written files there. `internal/store` keeps type aliases\n\u003e (`type TaskStore = storecontract.TaskStore`, the sentinel `var`s, the value types) so\n\u003e daemon-side callers are unaffected; the sentinel message strings are unchanged and remain\n\u003e the wire contract (§4).\n\n---\n\n## 2. Contract boundary\n\n**Ownership split** (see the fork in §1): the **workflow-domain consumer contract** is\n`workflow.Store` (consumer-owned, in `internal/domain/workflow`); the **daemon/webui/intake\nstore surfaces** are the producer-owned interfaces in `internal/store/interface.go`.\n`store.WorkflowStore` is **superseded** — it is replaced by the domain-owned `workflow.Store`,\nand `internal/store`'s `Store` implementation satisfies `workflow.Store` directly. `TaskStore`\nremains the only composite; every other interface is flat. Narrow slices are split along\nconsumer-capability lines so no consumer acquires a surface it does not need\n(`interfacebloat` cap = 8).\n\n| Interface (owner) | Embeds | Methods | Local impl | Remote impl |\n|---|---|---|---|---|\n| `TaskStore` (store) | `TaskLifecycle`, `TaskEvents`, `TaskQueries`, `TaskArchiver`, `TaskRetryer` | 24 (union) | `*Store` | `*staterpc.Client` |\n| `TaskLifecycle` (store) | — | 8 | `*Store` | `*staterpc.Client` |\n| `TaskEvents` (store) | — | 7 | `*Store` | `*staterpc.Client` |\n| `TaskQueries` (store) | — | 7 | `*Store` | `*staterpc.Client` |\n| `TaskArchiver` (store) | — | 1 | `*Store` | `*staterpc.Client` |\n| `TaskRetryer` (store) | — | 1 | `*Store` | `*staterpc.Client` |\n| `workflow.Store` (workflow domain) | — | 3 | `*Store` | `*staterpc.Client` |\n| `CaptureStore` (store) | — | 2 | `*Store` | `*staterpc.Client` |\n| `MappingStore` (store) | — | 5 | `*Store` | `*staterpc.Client` |\n| `BindingStore` (store) | — | 6 | `*Store` | `*staterpc.Client` |\n| `BindingDispatcher` (store) | — | 3 | `*Store` | `*staterpc.Client` |\n| `BindingTaskCreator` (store) | — | 1 | `*Store` | `*staterpc.Client` |\n| `PlaybookDispatcher` (store) | — | 2 | `*Store` | `*staterpc.Client` |\n\n**Key property of the remote client:** `*staterpc.Client` is a *single* adapter that wraps one\n`StateStoreClient` and is asserted against **whichever narrow Go interfaces its caller needs**\n(exactly as `gatewayrpc.Client` asserts both `gateway.ChatContract` and\n`gateway.SessionStore`). The Go contracts stay narrow (≤8, the `interfacebloat` cap); the wire\nis one service. `workflow.Store` introduces **no new RPCs** — it is a narrow consumer face over\nthe same `Update`/`Transition`/`InsertEvent` RPCs.\n\n---\n\n## 3. Method ownership\n\n`*Store` remains the single implementation of the daemon/webui store surfaces and of\n`workflow.Store`. Ownership below is about which **process** is the primary consumer of each\nmethod and therefore which methods are genuinely cross-process on the wire vs.\ndaemon-internal.\n\nProcess legend: **DAEMON** = `archied` (incl. in-process webui + gateway runtime +\n`storerpc.Server`); **AGENT** = `archie-agent` worker (no DB connection); **GATEWAY** = store\naccess through daemon-side narrow adapter interfaces.\n\n| Contract | Primary consumers (process) | Remote on the wire? |\n|---|---|---|\n| `TaskLifecycle` (8) | DAEMON (dispatch, claim, action) + **AGENT** (`Transition`, `Update` via storerpc) | `Transition`/`Update` = **YES** (agent); rest DAEMON |\n| `TaskArchiver` (1) | DAEMON (`taskactions` archive) | not yet |\n| `TaskRetryer` (1) | DAEMON (`taskactions` retry) | not yet |\n| `TaskQueries` (7) | DAEMON + GATEWAY (status/list adapters) | not yet (`OpenPRs`/`Tasks`/`StatusCounts` cross to gateway adapters in-process) |\n| `TaskEvents` (7) | DAEMON + **AGENT** (`InsertEvent` via storerpc) | `InsertEvent` = **YES** (agent); rest DAEMON |\n| `workflow.Store` (3) | **AGENT** only | **YES** (the sole genuinely remote consumer) |\n| `CaptureStore` (2) | DAEMON (webhook intake + mapping editor) | not yet |\n| `MappingStore` (5) | DAEMON (dashboard editor + dispatch loop) | not yet |\n| `BindingStore` (6) | DAEMON (dashboard editor + dispatch loop) | not yet |\n| `BindingDispatcher` (3) | DAEMON (dispatch loop) | not yet |\n| `BindingTaskCreator` (1) | DAEMON (dispatch loop) | not yet |\n| `PlaybookDispatcher` (2) | DAEMON (EDA playbook dispatch loop) | **YES** (side-effecting-action idempotency ledger over gRPC) |\n\n**Consequence for migration order:** the **agent's `workflow.Store` is already cross-process\nalready** (via the NATS `storerpc` surrogate), so it is the first contract to move to gRPC. The\ndaemon-internal surfaces (`Capture`/`Mapping`/`Binding`) stay in-process until the daemon\nitself points at `archie-state-store`. `TaskStore` (the large composite) moves last.\n\n**Two orphan methods** — `TaskQueries.ClearTerminalTasks` and\n`TaskQueries.IncrementRetryCount` have **no production consumer** (only `*Store` tests). They\nremain part of the Go `TaskQueries`/`TaskStore` surface, and therefore part of the wire\ncontract, but are flagged for deletion: they must **not** gain dedicated consumers or wire\nmethods in this work. Do not add the rest of the codebase to them.\n\n---\n\n## 4. Domain-facing types \u0026 consumer-owned contract — resolves the contradiction\n\nThe Phase 2 acceptance criterion (\"`internal/domain/workflow` … no longer import\n`internal/store` directly — they call the State Store contract\") **cannot be met while**\n`store.Task`, `store.Status*`, `store.Source*`, and `store.WorkflowStore` live in\n`internal/store`. This section fixes where they move.\n\n**The relocation (a prerequisite that must land before or with `.4.2`):**\n\n| Current symbol (`internal/store`) | Moves to | Notes |\n|---|---|---|\n| `Task` type | `internal/domain/workflow.Task` | The task-execution record the workflow domain operates on. `internal/store` persists this type (persistence → domain). |\n| `Status` + `Status*` constants | `internal/domain/workflow` | e.g. `StatusClosedWontDo`, `StatusMerged`, `StatusPROpen`, `StatusWaitingHuman`, `StatusRunning` — task-lifecycle vocabulary. |\n| `Source` + `Source*` constants | `internal/domain/workflow` | e.g. `SourceChat` — provenance vocabulary, follows `Task`. |\n| `WorkflowStore` interface (3 methods) | `internal/domain/workflow.Store` | **Consumer-owned** (dependency rule #2). Supersedes `store.WorkflowStore`. |\n\n**Resulting dependency direction (correct per the architecture):**\n\n- `internal/domain/workflow` imports **no** `internal/store` symbols → the acceptance\n criterion is satisfied. It consumes `store.Task`'s replacement as `workflow.Task`, uses\n `workflow.Status*`/`workflow.Source*`, and depends on its own `workflow.Store` interface.\n- `internal/store` imports `internal/domain/workflow` for the `Task`/`Status`/`Source` types it\n reads and writes (infrastructure → domain). **NOTE: `internal/store` is a transitional\n compatibility location only** — its long-term home is `internal/infrastructure/store` (see\n the callout in §1). Do not treat `internal/store` as the final infrastructure boundary.\n- Every store test that constructs `store.Task{…}` is updated to the relocated type as part of\n this prerequisite. This is a mechanical rename across ~6 workflow production files\n (`workflow.go`, `steps.go`, `triage.go`, `review.go`, `feasibility.go`, `implement.go`) and\n their tests plus the store tests. `TestRecordDispatchViaExplicitTx` is removed rather than\n updated — its non-nil `*sql.Tx` path has no production consumer (see §5's `RecordDispatch`\n blocker).\n\n**What this does NOT do:** it does not implement the full §4 Workflow migration\n(`mutable Stage → StepExecution history`, `RetryCount/Attempt → Attempt records`,\n`workflow.Registry → versioned Workflow definitions`, etc.). It lands only the minimal\ntask-type + `Status`/`Source` + consumer-owned-interface relocation that the Phase 2\ndomain-decoupling criterion requires. The residue of §4 remains its own migration.\n\n**Forward-compatibility:** because the wire DTOs are derived from the domain `Task`/`Status`\n(§5), when the full §4 migration later renames `workflow.Task` (or refines it into\n`WorkflowExecution`), only `values.go` and the proto field mapping change — the RPC surface and\nconsumer interfaces are unaffected. The wire contract is deliberately type-agnostic at this\nlevel.\n\n---\n\n## 5. DTO / wire representation\n\n### Proto layout\n\n- **Service:** `proto/state/v1/state.proto`, `package state.v1`, **one** `StateStore` service\n (mirrors the gateway's single `ChatService`; proto bypasses the Go `interfacebloat` cap, as\n `ChatContract` does via `//nolint:interfacebloat`).\n- **Codegen:** the existing buf v2 pipeline. `proto/buf.yaml` (module, lint STANDARD, breaking\n FILE against `main`), `proto/buf.gen.yaml` **managed mode** with\n `go_package_prefix → github.com/samcharles93/archie-core/internal/contracts`, plugins\n `protoc-gen-go` + `protoc-gen-go-grpc`, `out: ../internal/contracts`,\n `opt: paths=source_relative`. **No `go_package` in source** — managed mode owns it. Generated\n Go lands in `internal/contracts/state/v1/` (package `statev1`), committed (`proto:check`\n fails on uncommitted/unmodified generated files).\n- **Adapter package:** `internal/infrastructure/staterpc/` with `server.go` (registers the\n generated server over the store), `client.go` (wraps the generated client into `store.*` and\n `workflow.Store` facades), `values.go` (proto ⇄ domain mappers), and `conformance_test.go`\n (bufconn local-vs-grpc dual-adapter suite), mirroring `internal/infrastructure/gatewayrpc/`.\n- **Binary:** `cmd/archie-state-store/main.go`, mirroring `cmd/archie-gateway/main.go` and the\n serve pattern in `internal/app/archied/gateway.go` (`serveGateway`).\n\n### The `StateStore` RPC set (union of the contracts, grouped)\n\nMethods are named after the store methods so the mapping is unambiguous. `workflow.Store` is a\n*view*, not new RPCs — `Update`/`Transition`/`InsertEvent` are shared with\n`TaskLifecycle`/`TaskEvents` and appear once.\n\n| Group | RPCs |\n|---|---|\n| Lifecycle | `EnqueueIssue`, `EnqueueChatTask`, `ClaimNext`, `ClaimByIssue`, `Transition`, `Update`, `Requeue`, `RecoverStale` |\n| Archive / Retry | `ArchiveTask`, `RetryTask` |\n| Queries | `TaskByIssue`, `OpenTaskByPR`, `TaskByID`, `OpenPRs`, `ClearTerminalTasks`, `Tasks`, `StatusCounts`, `IncrementRetryCount` |\n| Events | `InsertEvent`, `EventsSince`, `TaskEvents`, `WorkflowStats`, `StageStats`, `TokensByDay` |\n| Capture | `InsertCapture`, `ListCaptures` |\n| Mapping | `InsertMapping`, `GetMapping`, `ListMappings`, `UpdateMapping`, `DeleteMapping` |\n| Binding | `InsertBinding`, `GetBinding`, `ListBindings`, `UpdateBinding`, `DeleteBinding`, `ApproveBinding` |\n| Dispatch | `ArmedBindingsForSource`, `RecordDispatch`, `ListUndispatchedCaptures` |\n| Playbook | `RecordPlaybookDispatch`, `DeletePlaybookDispatches` |\n| BindingTaskCreator | `EnqueueBindingTask` |\n| Config snapshot | `PutConfigSnapshot`, `GetConfigSnapshot` |\n| Apply status | `PutApplyStatus`, `ListApplyStatus` |\n| Channel status | `PutChannelStatus`, `ListChannelStatus` |\n| Task log | `ReadTaskLog`, `StreamTaskLogContent` |\n\nThat is **51 unique contract RPCs** across one service (the `TaskEvents.Close` method is dropped).\n`Close()` is **excluded** from the wire (it is server lifecycle, not a client call) — see §11.\n\nThe channel-status pair follows the config snapshot's split for the same reason: the writer is the process that HOSTS the channels -- the Messaging Service, which alone observes whether a channel is starting, running or failed -- and the reader is the UI process. Both RPCs are administrative, so a task-scoped grant reaches neither. A channel's state is runtime state rather than a settings document, which is why it is not a `ControlPlaneService` resource kind: a failed channel is not a document anybody edits. Asking for a reload travels on its own surface rather than through this one; see `docs/architecture/migration-decisions.md`, \"Channel state to the dashboard\".\n\nThe config-snapshot pair was added by `archie-core-ymut` (see\n`docs/architecture/migration-decisions.md`, \"Dashboard configuration page\"). It is the one\nsurface whose writer is the daemon and whose reader is the UI process, and both RPCs are\nadministrative: `authorizesTaskScopedCall` is deny-by-default, so a task-scoped grant reaches\nneither. `ConfigSnapshot.document` is opaque to this contract — the store neither parses nor\nvalidates it, and `schema` names the projection so a reader can refuse a shape it does not\nunderstand.\n\nThe apply-status pair was added by `archie-core-pskb` (see\n`docs/prds/control-plane-apply-status.md`). It is administrative for the same reason as the\nconfig-snapshot pair, and deny-by-default in `authorizesTaskScopedCall` covers it without a\nnew rule. Unlike that pair, the read is a list: `ListApplyStatus` returns every process's\nrecord and an empty slice means nothing has reported yet, so the `found=false` convention for\nabsent single rows does not apply. The rows are keyed on `(process, kind)` and replaced in\nplace, because each process re-stamps its record on an interval and no reader wants a history\nof re-stamps.\n\nThe task-log pair was added by `archie-core-iaqx` (see `docs/prds/ui-service-boundary.md`'s\nroute inventory). It is the one surface whose payload is not a row: the store service reads\nthem out of the state directory it owns with `internal/logging`'s own reader, so the wire\ncarries `logging.Entry`/`logging.Query` values and neither side restates the JSONL format.\n`ReadTaskLog` returns a decoded page and `StreamTaskLogContent` returns the raw bytes\nserver-streamed, because a log file rotates at 20 MB — past gRPC's 4 MiB unary cap — and the\nsame reasoning that made `StreamCaptures` supersede `ListCaptures` applies. `found=false` with\nno error means the attempt has no log; a service with no reader answers `Unavailable` instead,\nand the two answers are load-bearing for the dashboard: only the second is a deployment\ncondition, and the panel that collapsed them told operators task logging was switched off when\nit was not. The Go facade is `storecontract.TaskLogStore` (2 methods), carried by\n`*staterpc.Client` alongside the rest.\n\nThe playbook pair was added by `archie-core-t2db.17`'s implementation (see\n`docs/prds/eda-playbook-engine.md`, execution-time gap 2). It is administrative\nfor the same reason as the config-snapshot and apply-status pairs, and\ndeny-by-default in `authorizesTaskScopedCall` covers it without a new rule. The\nrow is the structural `(playbook_id, playbook_version, event_id, action_id)`\ntuple, written **before** the action's side effect fires (the reverse of\n`binding_dispatches`), so a duplicate returns `store.ErrAlreadyDispatched` and\nthe coordinator skips the invoke rather than repeating a non-revocable side\neffect. `DeletePlaybookDispatches` is the explicit reclamation primitive; nothing\ncalls it automatically, because a playbook file's absence at boot is not a\ndeliberate removal (`eda-playbook-engine.md`, \"Lifetime\"). The Go facade is\n`storecontract.PlaybookDispatcher` (2 methods), carried by `*staterpc.Client`\nalongside the rest.\n\n### Domain types and `values.go` mapping\n\n`values.go` maps proto ⇄ these types. The task model for the **agent** path is the domain-owned\n`workflow.Task` (§4); the daemon/webui store surfaces return the same `workflow.Task` (the store\npersists the domain type).\n\n| Domain type | Package | Notes for `values.go` |\n|---|---|---|\n| `workflow.Task` | `internal/domain/workflow` | time fields stored as layout strings; convert to/from `google.protobuf.Timestamp` |\n| `events.Event` | `internal/events` | `Timestamp` → `google.protobuf.Timestamp` |\n| `store.CapturedEvent` | `internal/store` | headers/body/remote_addr; body capped 256 KiB |\n| `mapping.Mapping` | `internal/domain/mapping` | JSON-path field bindings |\n| `binding.Binding` | `internal/domain/binding` | secret is **already decrypted** before returning; keep decrypted form out of any durable/logged representation |\n| `workflow.Status` / `workflow.Source` | `internal/domain/workflow` | enum-like; map to proto enum or string, keeping the names stable for compatibility |\n| `store.WorkflowStat` | `internal/store` | aggregate row |\n| `store.StageStat` | `internal/store` | aggregate row |\n| `store.DayTokens` | `internal/store` | aggregate row |\n\n### Wire-contract blocker fixed here: `RecordDispatch`\n\n`store.BindingDispatcher.RecordDispatch(ctx, tx *sql.Tx, bindingID, bindingVersion int64,\ncaptureID int64, taskID int64) error` takes a **`*sql.Tx`** that cannot cross a gRPC boundary.\n\n**Evidence that the `*sql.Tx` is removable:** the dispatch loop calls `RecordDispatch(ctx, nil,\n…)` (`internal/daemon/daemon.go`). The implementation falls back to the store's own\n`*sql.DB` when `tx == nil` (\"best-effort, not atomic\"; the comment and the\n`TestRecordDispatchViaExplicitTx` test are the only consumers of the non-nil `tx` path).\nProduction never opens the explicit transaction.\n\n**Decision:** the contract drops `*sql.Tx` from `RecordDispatch` (the method becomes\n`RecordDispatch(ctx, bindingID, bindingVersion, captureID, taskID) error`). The store service\nowns any transaction boundary internally when atomicity is wanted; the at-most-once\nguarantee is the `INSERT OR IGNORE` dedup row, which is already transactionless in production.\n`TestRecordDispatchViaExplicitTx` is superseded by the store service's own internal-transaction\ntest. **This is a required interface migration that must land before or with `.4.2`** (the\ncurrent `interface.go` signature and test still use the transaction-bearing form).\n\n---\n\n## 6. Agent State Store client handoff\n\nMoving the agent from NATS `storerpc` to gRPC `workflow.Store` requires the agent process to\nreceive a State Store endpoint. The agent already receives its NATS connection settings via\nboot-time env/flags (see `cmd/archie-agent/main.go`: `-nats-url` / `NATS_URL`, `NATS_TOKEN`\nenv), and the daemon injects them into the task container. The State Store handoff follows the\nsame seam.\n\n**Boot handoff contract (additive, not replacing the NATS settings):**\n\n| Field | Where | Source | Notes |\n|---|---|---|---|\n| `StateStoreTarget` | `agentworker.Settings` | env `STATE_STORE_URL` (or `-state-store-url` flag), set by the daemon in the container env | `host:port` of the State Store gRPC server. Loopback/Docker-bridge address — the agent reaches it the same way it reaches the daemon's NATS. |\n| `StateStoreToken` | `agentworker.Settings` | env `STATE_STORE_TOKEN` (or `-state-store-token` flag) | Per-task bearer token carried in gRPC metadata by a client interceptor (see §9). |\n| `StoreTimeout` | `agentworker.Settings` | a bounded per-call timeout | Mirrors `natsrpc.Client.Timeout` / the existing `rpcTimeout`; bounds each State Store call when the caller `ctx` has no deadline. |\n\n**Wiring change in the transport façade:**\n\n- `internal/infrastructure/agenttransport/nats.Transport` — `Store(timeout) store.WorkflowStore`\n returns `\u0026storerpc.Client{Conn: t.conn, Timeout: timeout}`. It becomes\n `Store(timeout) workflow.Store` and returns a *single long-lived* `staterpc.Client` dialed to\n `settings.StateStoreTarget` (with the token interceptor and per-call timeout).\n- `internal/app/agentworker/worker.go` `Settings` gains the three fields above; `taskServiceTransport`\n and `taskDependencies` change their `store` type from `store.WorkflowStore` to\n `workflow.Store` (via `domain/workflow.TaskContext{Store: …}`).\n- Reconnect/deadline: `grpc.NewClient` (or the existing dial path) provides transparent reconnect;\n each RPC is bounded by the per-call context deadline (`StoreTimeout`), matching the\n request/reply timeout the NATS path used. No streaming is involved, so no server-sent\n reconnect semantics apply.\n\n**Deletion rule for the NATS path:** the NATS `storerpc` (Server, Client, subjects\n`archie.store.update|transition|insert_event`) is deleted **only after** the agent has fully\nmoved to gRPC and the container env carries `STATE_STORE_URL`. Until then both paths may coexist\n(agent flips when the daemon starts injecting `STATE_STORE_URL`); there is no dual-live window\nacross a single agent process, which is either NATS-backed or gRPC-backed, never both.\n\n### Operational specifics (in-process listener → address → per-task injection)\n\n- **Who binds the in-process State Store listener:** the daemon (`archied`), at bootstrap, when\n the State Store is served in-process/multiplexed (mirroring `serveGateway` in\n `internal/app/archied/gateway.go`): a `grpc.NewServer()` + `RegisterServer` + graceful stop on\n `ctx.Done()`.\n- **Address selection (Docker-bridge reachable):** reuse the host-gateway resolution already used\n for embedded NATS (`internal/container/network.go` `RequireHostGateway` /\n `resolveHostGateway`). Per §9's single listener-topology rule, the listener binds the\n **host-gateway bridge address** when agent containers consume it (the Phase 2 default), or\n loopback-only when the daemon is the sole consumer. The chosen address is stored on the daemon\n as `d.ConnectedStateStore.URL` (analogous to `d.ConnectedNATS.URL`).\n- **Per-task injection:** `containerEnv(task)` (`internal/daemon/daemon.go`) appends\n `STATE_STORE_URL=\u003cd.ConnectedStateStore.URL\u003e` and `STATE_STORE_TOKEN=\u003ctoken\u003e` alongside the\n existing `NATS_URL`/`NATS_TOKEN`. The agent reads them via env (`-state-store-url` /\n `STATE_STORE_URL`, `-state-store-token` / `STATE_STORE_TOKEN`), the same seam as `NATS_URL`. A\n reloaded `[services.state]` must not point new containers at a store the daemon is not serving\n — the same immutability rule as NATS.\n\n---\n\n## 7. Error semantics\n\n### Current state (both boundaries lose error identity)\n\n- **NATS `natsrpc.Envelope`** (`internal/natsrpc/natsrpc.go`) serialises any error to a single\n `err.Error()` string and rehydrates a fresh `errors.New(string)`. The original type, `%w`\n chain, and sentinel identity are lost → `errors.Is(err, store.ErrStaleTransition)` is\n **false** across the boundary.\n- **gRPC `gatewayrpc`** returns raw `err` from handlers; gRPC-Go converts to\n `codes.Unknown` with only the message. The **only** explicit code in the repo is\n `codes.Unavailable` for a missing session store (`server.go`). No status-code mapping for\n store errors exists.\n\n### Decision — structured mapping so `errors.Is` survives\n\nThe `StateStore` service must map the store's sentinel errors to canonical gRPC codes and\nrehydrate them to the *same* sentinels on the client, so consumer `errors.Is` checks keep\nworking — an improvement over the flattening it replaces, not a regression.\n\n| Store sentinel / outcome | gRPC code | Client rehydrates to |\n|---|---|---|\n| `store.ErrStaleTransition` | `FailedPrecondition` | `store.ErrStaleTransition` |\n| `store.ErrBindingNotFound` | `NotFound` | `store.ErrBindingNotFound` |\n| `store.ErrMappingNotFound` | `NotFound` | `store.ErrMappingNotFound` |\n| `store.ErrBindingOverlap` | `FailedPrecondition` | `store.ErrBindingOverlap` |\n| `store.ErrBindingTransition` | `FailedPrecondition` | `store.ErrBindingTransition` |\n| `store.ErrAlreadyDispatched` | `AlreadyExists` | `store.ErrAlreadyDispatched` |\n| not-found-as-`(nil,nil)` (`TaskByID`, `GetBinding`, `GetMapping`, `ClaimNext`, `ClaimByIssue`, …) | `NotFound` with a `found=false` field (**not** an error) | `(nil, nil)` |\n| infra / wrapped error | `Internal` (**sanitised** message, see below) | wrapped `fmt.Errorf(...%w)` |\n| capability absent | `Unavailable` | context/typed error |\n\nThe `found=false`-not-error convention matters: the repo deliberately returns `(nil, nil)` for\nnot-found in reads and a sentinel for not-found in writes. The wire must preserve **both**\nconventions (a `GoogleRPCStatus` with `NotFound` is an *error*, which would wrongly surface to\ncallers that expect `(nil, nil)`). Implement `found` as an explicit boolean field on read\nresponses, and map `(nil,nil)` ⇄ `found=false` + `OK`.\n\n### Error-detail sanitisation (rev. 2c)\n\nFull internal error messages must **not** be placed in gRPC `details` for remote callers. SQL\npaths, provider/forge details, secrets, and stack traces are sensitive internals. The rule:\n\n- **Sentinel errors** — the client rehydrates the sentinel from the code; the message string in\n `status.Error` is a short, public, stable phrase (e.g. `\"stale transition\"`), not the raw\n wrapped chain.\n- **Infra errors** — the wire carries a **sanitised** public message + the `Internal` code.\n The **full** error (with `%w` chain, SQL, provider detail) is logged **server-side only** by\n the store service. The client rehydrates a generic wrapped error and must not surface the raw\n text to users.\n\n---\n\n## 8. Limits\n\n**Interface bloat (`interfacebloat` cap = 8)** — `.golangci.yml` `interfacebloat: max: 8`,\nwhich overrides the upstream default of 10. The Go consumer facades must stay ≤8 methods.\n`TaskLifecycle` is exactly 8 (at the cap); `TaskStore` is the 24-method composite that the cap\nis why it is decomposed; proto services bypass the cap (like `ChatContract`).\n\n**Payload caps:**\n- Capture body: `defaultCaptureMaxBodyBytes = 256 KiB` (may reduce further; the wire message\n must not exceed gRPC's default 4 MiB receive limit for a single capture). A single capture\n fits, but a *batch* of them did not: the caller-side cap of up to 100 captures at 256 KiB each\n could exceed 4 MiB in one unary `ListCaptures`/`ListUndispatchedCaptures` response. `.4.7` adds\n `StreamCaptures`/`StreamUndispatchedCaptures` (server-streaming, one capture per message) as\n the path both `staterpc.Client` methods actually call; the original unary RPCs stay defined,\n unmodified and deprecated, only so `buf breaking` protects an in-flight rolling deploy.\n- Task `detail`/`park_reason`/event `Detail`: 4000-char hard cap (`clip(s, 4000)`).\n\n**Retention (stays server-side in the store; the wire does not carry pruning decisions):**\n- `events` table: **no prune, unbounded growth** — a known risk to revisit, not changed here.\n- `captured_events`: prune-on-write, default 7 days / 5000 events.\n\n**Pagination/arity:** store read methods take a raw `int limit` (no store-side cap); the\ncaller-side caps (task list 100, log 2000, captures 100, capture scan window 5000, binding\ndispatch batch 100, session list 20) remain caller-side. The wire passes `limit int` and returns\n`[]*` slices, mirroring the existing Go signatures — **no cursor/paging protocol is introduced**\nin this work (the current `EventsSince(sinceID, limit)` offset cursor is the only one, and it\nstays as-is).\n\n---\n\n## 9. Transport security boundary\n\nThe gateway precedent is loopback-only, unauthenticated gRPC. The State Store holds the\ncrown-jewel `archie.db` (task/event/capture/mapping/binding data), so its boundary must be\nstated explicitly rather than left implicit.\n\n**The State Store's equivalent boundary (rev. 2c):**\n\n- **Listener topology — single rule (rev. 2c).** One in-process listener; its bind address is\n chosen by the consumer set, never both at once:\n - *Agent containers consume it* (the Phase 2 driver during multiplexed serving) → bind to the\n **host-gateway bridge address** so containers can reach it, and **token auth is mandatory**\n (a bridge host is reachable by any process on that bridge). A container **cannot reach the\n host's `127.0.0.1`**, so loopback-only is NOT usable for the agent-consumed path.\n - *Only the daemon consumes it* (no agent) → bind **loopback-only** (`--listen 127.0.0.1`),\n `insecure` is fine because it is not network-reachable; no token required.\n - There is **no dual-listener** design. The practical default during `.4.2` (the reason\n in-process serving exists is to give the agent a gRPC target) is the **bridge address +\n token**. `127.0.0.1` is reserved for the daemon-local-only case.\n- **Agent access → the container bridge.** The agent runs in a task-scoped container on the\n host and reaches the State Store over the Docker bridge gateway (the bind address from the\n topology rule above), exactly as it reaches the daemon's NATS. To close the \"any host\n process could reach the store\" gap, the agent **authenticates with a per-task bearer token**\n carried in gRPC metadata by a client interceptor (`StateStoreToken`, §6). Network\n reachability + the token is the trust boundary for the default bridge topology.\n- **The daemon** uses the local adapter (`*store.Store`) in-process by default, so no network\n boundary exists until `[services.state].target` is set; when it is set, the daemon client\n authenticates with a token supplied alongside the target.\n- **Non-loopback / k8s:** TLS/mTLS is an **operator decision**, not a default. Any non-loopback\n exposure (e.g. `--listen 0.0.0.0`, a CDN, a cross-host service) requires an explicit decision\n and mTLS or token auth — mirroring the gateway's `--listen`/`--enable-cors-header`\n confirmation caveat. The NATS-KV discovery fallback supplies endpoints but does not add\n transport security; that stays the operator's responsibility.\n\n### Token lifecycle (rev. 2c; scoping added in `.4.7`)\n\n- **Generation:** the daemon generates a per-task bearer token when it acquires the container\n (or per incumbence) and injects it via `containerEnv` as `STATE_STORE_TOKEN`. A single token\n may cover a task's container lifetime.\n- **Registration (`.4.7`, post-standalone-cutover):** the State Store is now its own process\n (`.4.3`/`.4.6`), so the daemon can no longer just hold an in-process issued-token set for the\n server to check — it registers the token at the remote authority via the\n `RegisterTaskGrant`/`RevokeTaskGrant` RPCs (`staterpc.GrantIssuer`, `daemon.StateStoreGrantIssuer`),\n admin-token-authenticated calls a task-scoped token itself can never make.\n- **Lifetime:** tied to the task + container grace period; rotated on a new container acquisition\n for the same task; a token for a released container is invalidated (`RevokeTaskGrant`, deferred\n alongside `ContainerPool.Release`). An unrevoked grant still expires server-side (bounded by\n `RegisterTaskGrantRequest.lifetime_seconds`, capped at 7 days) so a daemon crash before revoke\n cannot leave a grant valid indefinitely.\n- **Validation and scope (`.4.7`):** a gRPC interceptor (`staterpc.TaskGrants.UnaryInterceptor`)\n validates the token against the State Store's own issued-grant set; missing/unknown/expired →\n `codes.Unauthenticated`. Unlike the daemon's own administrative token (full access to all ~40\n RPCs), a task-scoped grant additionally authorizes only `Update`/`Transition`/`InsertEvent` on\n its own task ID — every other RPC, including the streaming capture surface and\n `RegisterTaskGrant`/`RevokeTaskGrant` themselves, is `codes.PermissionDenied` for a task-scoped\n caller. The token is carried in gRPC metadata, never in a URL.\n\n### Fail-closed on non-loopback (rev. 2c)\n\nIf `[services.state].target` points to a non-loopback address and **neither** TLS **nor** a token\nis configured, the State Store client (daemon or agent) **fails closed** — it refuses to\nstart/dial rather than fall back to a plaintext, unauthenticated remote connection. This mirrors\nthe gateway's `--listen`/`--enable-cors-header` confinement caveat and prevents a silent\nunauthenticated remote store. The loopback + token default is the only path that starts without\nan explicit operator decision.\n\n---\n\n## 10. Mode / config shape\n\n**Follow the gateway's presence-based `target` seam, not the NATS `mode` enum.**\n\n- **Services are keyed by name, not named as struct fields.** A\n `State ServiceConnection` field per service is the duplication\n `docs/prds/service-registry.md` removes, and this document's summary describes\n the seam as `[services.\u003cname\u003e]`:\n ```go\n type Services map[string]ServiceConnection\n ```\n registered once via `RegisterService(context, name, target, listen, tokenEnv)`. The TOML\n shape below is unchanged, so nothing else in this section is affected; read\n `cfg.Services.Get(\"state\")` wherever it says `cfg.Services.State`.\n- **Empty `Target` → startup error.** The local `*store.Store` adapter default was\n withdrawn when the State Store was extracted (§12 step 7): the standalone `archie-state-store` process\n exclusively owns `archie.db`, so both `archied` and `archie-gateway` now reject an empty\n target (`internal/app/archied/bootstrap.go` and `state_store_client.go`: `services.state.target\n is required`). **Set `Target` → dial gRPC** with `*staterpc.Client`; no client opens the\n database itself. This differs from the gateway, where `Target` is defaulted to\n `127.0.0.1:8585` because `archie-gateway` is already a separate process whose client and\n server share one default.\n- `[services.state]` is present and **uncommented** in `config.example.toml`, with `target`\n marked REQUIRED, because the value is load-bearing for every client. A commented-out\n `target` cannot boot, so the section ships uncommented.\n- Composition root: `internal/app/archied/bootstrap.go` (the daemon) and\n `internal/app/archieui/config.go` (the UI process) both read `cfg.Services.Get(\"state\")` and\n dial the remote State Store. `openProductionTaskStore` is **not** called from the daemon: it\n lives in the standalone `archie-state-store` process (`internal/app/archied/state_store.go`),\n which is the only path that opens `archie.db`.\n\n\u003e **Mode fork, decided.** The gateway uses a single `target` address (no enum). The State Store\n\u003e boundary uses the same seam for symmetry, and **both now require a non-empty `target`**: the\n\u003e State Store's local-store default was withdrawn when the process was extracted (§12 step 7).\n\u003e No `mode` field is added. A companion `STATE_STORE_TOKEN` secret (or a\n\u003e `[services.state] target_token` key) carries authentication.\n\n---\n\n## 11. Local-adapter compatibility + conformance\n\n- `*store.Store` (local) and `*staterpc.Client` (remote) satisfy the **same** contract\n interfaces. Use **explicit** `var _` assertions on **both** adapters (the gateway's local\n adapters use only structural/return-type assertions; the store precedent's explicit `var _`\n style is stronger and should be adopted on both sides).\n ```go\n var _ workflow.Store = (*staterpc.Client)(nil) // first contract to move (agent)\n var _ store.TaskStore = (*staterpc.Client)(nil) // daemon path, added progressively\n var _ store.CaptureStore = (*staterpc.Client)(nil)\n var _ store.MappingStore = (*staterpc.Client)(nil)\n var _ store.BindingStore = (*staterpc.Client)(nil)\n var _ store.BindingDispatcher = (*staterpc.Client)(nil)\n var _ store.BindingTaskCreator = (*staterpc.Client)(nil)\n var _ store.PlaybookDispatcher = (*staterpc.Client)(nil)\n // local side (in store), already present:\n var _ workflow.Store = (*Store)(nil)\n ```\n- **Conformance suite:** a `staterpc/conformance_test.go` bufconn suite runs the **same**\n contract through both `\"local\"` and `\"grpc\"` adapters (mirroring\n `gatewayrpc/conformance_test.go`), asserting error-sentinel fidelity and the\n not-found-as-`(nil,nil)` convention both ways.\n- **`Close()`:** stays on the Go store interfaces (so `TaskStore`/`TaskEvents` remain fully\n satisfiable) but is **not** a `StateStore` RPC. The remote `staterpc.Client` implements\n `Close()` as a **no-op**; the store service owns its own DB lifecycle. When the daemon hands\n the store to `archie-state-store`, `b.cleanup()` no longer calls `store.Close()` on a remote\n client (it is the daemon's own lifecycle only when it owns the store in-process).\n\n---\n\n## 12. Cutover / deletion rules\n\n1. **Ratify the contract (this document).** No code.\n2. **Domain type + interface relocation (prerequisite, coupled with §4).** Move `Task`/`Status`/\n `Source` and the `WorkflowStore` interface into `internal/domain/workflow` (as\n `workflow.Task`/`workflow.Status`/`workflow.Source`/`workflow.Store`), so\n `internal/domain/workflow` imports no `internal/store`. Update the store tests and the 6\n workflow production files. **This must land before or with `.4.2`, because `.4.2` cannot move\n the agent's store contract off `store.WorkflowStore` while the domain still imports it.**\n3. **`.4.2` — generalize `storerpc` to the gRPC contract.** Move the agent's `workflow.Store`\n (`Update`/`Transition`/`InsertEvent`) from NATS JSON `storerpc` to `staterpc.Client` over\n gRPC, using the §6 handoff. The daemon serves the gRPC `StateStore` server **in-process\n (multiplexed)** first so the agent has a gRPC target before the standalone binary exists\n (PRD §3 \"multiplexed in-process is the migration default\"). Bind the in-process listener and\n inject `STATE_STORE_URL`/`STATE_STORE_TOKEN` per task (see §6 \"Operational specifics\"). Also\n land the `RecordDispatch` `*sql.Tx` drop (see §5) in the same span. Flip the agent's\n `Transport.Store` to a gRPC client once `STATE_STORE_URL` is injected.\n4. **Delete NATS `storerpc` only after the agent is fully on gRPC** — delete `internal/storerpc`\n (Server, Client, subjects `archie.store.update|transition|insert_event`), the\n `natsrpc.RegisterAll` registration, and the now-unused `store.WorkflowStore` NATS path in\n `internal/infrastructure/agenttransport/nats`, in the same commit.\n5. **`[services.state].target` stays empty (inproc)** through `.4.2`; the `archie-state-store`\n `cmd/` binary is **`.4.3`** and hosts the same `StateStore` gRPC service.\n6. **Swap daemon-internal contracts one at a time** (only once the daemon points at\n `archie-state-store`): `Capture → Mapping → Binding → TaskStore`. Each swap is independent —\n add the remote-client composition path, run the per-contract conformance suite, flip\n `[services.state].target`, delete the local use. A consumer mixes in-process and remote\n contracts for a contract-code (no dual-live *window*; each contract is either local or remote\n at a time).\n7. **Final flip / deletion of the in-process serving path:** when the daemon no longer owns the\n store (i.e. `TaskStore` has moved), delete the in-process `openProductionTaskStore` /\n `wireWebStoreSurfaces` path in the daemon in a **same-commit** deletion — no dual-store\n ownership. The single SQLite file remains owned by `archie-state-store`.\n8. **Never** split `archie.db` or add a second store. The gateway keeps its own already-separate\n session SQLite; that is unchanged and out of state-store scope.\n\n---\n\n## 13. Decisions recorded (for the review trail)\n\n- **`internal/store` is a transitional compatibility location** (target\n `internal/infrastructure/store`), not the final infrastructure boundary. The producer-owned\n daemon/webui store surfaces live there now but may relocate with the implementation.\n- **In-process listener + token lifecycle:** the daemon binds the in-process State Store\n listener to the **host-gateway bridge address** when agent containers consume it (the Phase 2\n default) or loopback-only when the daemon is the sole consumer — one listener, one address\n rule, never both (rev. 2c). It selects the bridge address by reusing the embedded-NATS\n host-gateway resolution and injects `STATE_STORE_URL`/`STATE_STORE_TOKEN` per task via\n `containerEnv`. Token is per-incumbence (task + container grace period) and validated by a\n gRPC interceptor → `codes.Unauthenticated` on missing/unknown/expired. Non-loopback targets\n fail closed without TLS or a token.\n- **§12 step 2 (domain type + interface relocation) is done** — `internal/domain/workflow`\n imports no `internal/store` symbol and `store.WorkflowStore` is gone from `TaskContext`. The\n broader Phase 2 acceptance criterion (State Store as its own gRPC service) is still open;\n `.4.2` picks up from here.\n- **Ownership split:** domain-facing workflow contract is **consumer-owned** in the domain\n (`workflow.Store` + `workflow.Task`/`Status`/`Source`, per dependency rules #2 and #7);\n daemon/webui store surfaces stay **producer-owned** in `internal/store`. `store.WorkflowStore`\n is superseded by `workflow.Store`.\n- **One `StateStore` gRPC service** (49 contract RPCs, grouped by contract) + narrow Go consumer facades\n (≤8) on `staterpc.Client` — mirrors the single-`ChatService` precedent.\n- **Domain type relocation** (`Task`/`Status`/`Source` → `internal/domain/workflow`) is a\n Phase 2 prerequisite, pulled forward from migration-decisions §4 (minimal bound,\n `Task → WorkflowExecution`); the rest of §4 stays a separate migration.\n- **`RecordDispatch` drops `*sql.Tx`** (wire-blocker; production passes `nil`) — a\n required interface migration landing before/with `.4.2`.\n- **`Close()` not on the wire**; remote client `Close()` is a no-op.\n- **Structured gRPC error codes** with client rehydration to store sentinels (preserves\n `errors.Is`), a `found=false` + `OK` convention for read not-found, and **sanitised public\n error details** (full detail logged server-side only).\n- **Agent handoff:** `agentworker.Settings` gains `StateStoreTarget`/`StateStoreToken`/\n `StoreTimeout`, injected as `STATE_STORE_URL`/`STATE_STORE_TOKEN` env (mirrors the NATS\n handoff); `Transport.Store` returns a long-lived gRPC `workflow.Store` client.\n- **Transport security + listener topology (rev. 2c):** one in-process listener, bound to the\n host-gateway bridge address when agent containers consume it (the Phase 2 default) or\n loopback-only for a daemon-local-only consumer. `insecure` only on loopback; mandatory\n per-task bearer-token auth on the bridge; TLS/mTLS is an operator choice for any\n non-loopback/k8s topology.\n- **Mode = presence-based `[services.state].target`**, default **local** (not yet extracted);\n add `[services.state]` to `config.example.toml`.\n- **Explicit `var _` assertions on both adapters**; per-contract bufconn conformance.\n- **Migration order:** domain type/interface relocation → `workflow.Store` (agent) → stand up\n `archie-state-store` → `Capture` → `Mapping` → `Binding` → `TaskStore`; NATS `storerpc`\n deleted once the agent is on gRPC; in-process store path deleted in a same-commit flip.\n- **Orphans `ClearTerminalTasks`/`IncrementRetryCount`** retained but not built upon.\n" + "body": "# State Store — contract boundary \u0026 transport (ratification)\n\n**Status:** Approved\n**Revision:** 2e\n**Date:** 2026-09-06\n\nThis is the pre-implementation design document that Phase 2 subtasks `.4.2` (generalize\n`storerpc` as the State Store gRPC contract) and `.4.3` (stand up `archie-state-store`)\nimplement against.\n**Beads milestone:** archie-core-8cda.4.1\n**Parent:** `docs/prds/service-decomposition.md` (open-question Q4 **RESOLVED**)\n\nThis document fixes the **contract boundary** and **transport** for the State Store service.\nIt does not redesign Archie's domain model, does not invent a second store, and does **not**\nsplit `archie.db`. It records method ownership, DTO/wire representation, error semantics,\nlimits, mode/config shape, local-adapter compatibility, and cutover/deletion rules so that\n`.4.2` and `.4.3` have exactly one authoritative surface to build against.\n\n\u003e **Storage superseded.** The store behind this contract is PostgreSQL:\n\u003e `internal/infrastructure/postgres` implements the producer-owned interfaces in\n\u003e `internal/domain/storecontract` and `workflow.Store`, over the database\n\u003e `database_url` names. `internal/store`, the SQLite file and the local adapter\n\u003e are gone; where this document says `archie.db`, `*store.Store` or\n\u003e `internal/store`, read the State Store's PostgreSQL database and\n\u003e `internal/infrastructure/postgres`. The contract, ownership split, error\n\u003e semantics and cutover rules stand.\n\n\u003e **Current state — read before `.4.2` starts.** The §12 step 2 relocation (`archie-core-8cda.\n\u003e 4.8`) has landed: `Task`/`Status*`/`Source*` and the 3-method `workflow.Store` now live in\n\u003e `internal/domain/workflow`, the six production workflow files import no `internal/store`\n\u003e symbol, and `storerpc`/the NATS transport assert against `workflow.Store` instead of\n\u003e `store.WorkflowStore`. The broader Phase 2 acceptance criterion (State Store as its own\n\u003e service behind gRPC) is still open — that is what `.4.2`/`.4.3` build next.\n\n\u003e **Relationship to migration-decisions §4 — not replaced.** `docs/architecture/\n\u003e migration-decisions.md` §4 (`store.Task → WorkflowExecution`, `mutable Stage →\n\u003e StepExecution history`, `RetryCount/Attempt → Attempt records`, versioned Workflow defs)\n\u003e **remains the authoritative, open document for the long-term Workflow migration.** This PRD\n\u003e does **not** replace it. It only pulls the minimal `Task`/`Status`/`Source` type + interface\n\u003e relocation forward as a Phase 2 prerequisite (because the acceptance criterion requires it);\n\u003e the rest of §4 stays its own migration and stays open.\n\n---\n\n## 1. Decision (the invariant everything follows)\n\nArchie is composed such that a single **`archie-state-store`** service owns the one existing\nSQLite file (`archie.db`) and exposes it over **gRPC behind narrow typed contracts**. There is\nexactly one store, one file, one service. No database-per-service, no distributed\ntransactions, no second store abstraction.\n\nTwo adapter families satisfy the contract interfaces:\n\n- **Local adapter** — `*store.Store` (the in-process SQLite implementation).\n- **Remote adapter** — `*staterpc.Client` (wraps the generated gRPC client).\n\nEach adapter carries an explicit compile-time assertion against the contract interfaces it\nserves. Composition picks local vs. remote by the presence of `[services.state].target`\n(empty = local; set = dial the gRPC service). Default is **local** (the State Store is not yet\nextracted), which is the PRD §3 posture; it differs from the gateway's remote-default only\nbecause the gateway is already a separate deployment and the State Store is not.\n\n\u003e **Contract-ownership fork, decided (rev. 2c — resolves the domain contradiction).**\n\u003e Ownership splits by the consumer's *layer*, per the dependency contract's rule #2\n\u003e (\"a domain defines the smallest interfaces required to perform its work\") and rule #7\n\u003e (\"database rows … MUST NOT leak into domain models\"):\n\u003e\n\u003e - The **workflow domain** (`internal/domain/workflow`) is the only domain-layer consumer of\n\u003e store state. It defines its own **consumer-owned** narrow `Store` interface\n\u003e (Update/Transition/InsertEvent) and its own **domain-owned** `Task` type plus\n\u003e `Status`/`Source` vocabulary. It imports **zero** `internal/store` symbols — satisfying the\n\u003e Phase 2 acceptance criterion. This is the minimal bound of migration-decisions §4\n\u003e (`store.Task → WorkflowExecution`), pulled forward **because** the acceptance criterion\n\u003e requires it.\n\u003e - The **daemon / webui / intake** store surfaces (`TaskStore`, `CaptureStore`,\n\u003e `MappingStore`, `BindingStore`, `BindingDispatcher`, `BindingTaskCreator`) stay\n\u003e **producer-owned** in `internal/store`, because those consumers are application and\n\u003e infrastructure layers, not domains — rule #2 does not apply to them.\n\nSo the State Store contract is **not** uniformly producer-owned: the domain-facing workflow\ncontract is consumer-owned (in the domain), while the daemon/webui store surfaces remain in\n`internal/store`. `internal/store` imports `internal/domain/workflow` for the `Task`/`Status`/\n`Source` types it persists (persistence → domain, the correct direction), and its `Store`\nimplementation satisfies `workflow.Store`. This is the single decision that makes the whole\nPhase 2 acceptance criterion and the architecture's rules simultaneously satisfiable.\n\n\u003e **`internal/store` is a TRANSITIONAL compatibility location (rev. 2c).** The repository's target\n\u003e architecture (`docs/architecture/dependencies-and-contracts.md`, `organisation.md`) places\n\u003e persistence implementations under `internal/infrastructure/\u003ccapability\u003e`. `internal/store`\n\u003e holding the task/capture/mapping/binding implementation and ALSO being the home of the\n\u003e producer-owned daemon/webui interfaces is acceptable **only as a transitional location**. The\n\u003e long-term home for the store implementation is `internal/infrastructure/store`; the\n\u003e producer-owned interfaces may move to the same infrastructure package (or an approved\n\u003e contract location) when the store implementation relocates. Agents must not treat\n\u003e `internal/store` as the final infrastructure boundary.\n\u003e\n\u003e **REV. 2d (archie-core-8cda.5.6): the producer-owned interfaces HAVE moved to an approved\n\u003e contract location** — `internal/domain/storecontract` (package `storecontract`), ahead of the\n\u003e store implementation's own relocation. The UI Service deletion gate required the archie-ui\n\u003e process to hold the contracts without linking `internal/store`'s SQL implementation, and\n\u003e `internal/contracts/` itself is ruled out: that tree is the `buf generate` output and\n\u003e `tools/proto.sh check` deletes hand-written files there. `internal/store` keeps type aliases\n\u003e (`type TaskStore = storecontract.TaskStore`, the sentinel `var`s, the value types) so\n\u003e daemon-side callers are unaffected; the sentinel message strings are unchanged and remain\n\u003e the wire contract (§4).\n\n---\n\n## 2. Contract boundary\n\n**Ownership split** (see the fork in §1): the **workflow-domain consumer contract** is\n`workflow.Store` (consumer-owned, in `internal/domain/workflow`); the **daemon/webui/intake\nstore surfaces** are the producer-owned interfaces in `internal/store/interface.go`.\n`store.WorkflowStore` is **superseded** — it is replaced by the domain-owned `workflow.Store`,\nand `internal/store`'s `Store` implementation satisfies `workflow.Store` directly. `TaskStore`\nremains the only composite; every other interface is flat. Narrow slices are split along\nconsumer-capability lines so no consumer acquires a surface it does not need\n(`interfacebloat` cap = 8).\n\n| Interface (owner) | Embeds | Methods | Local impl | Remote impl |\n|---|---|---|---|---|\n| `TaskStore` (store) | `TaskLifecycle`, `TaskEvents`, `TaskQueries`, `TaskArchiver`, `TaskRetryer` | 24 (union) | `*Store` | `*staterpc.Client` |\n| `TaskLifecycle` (store) | — | 8 | `*Store` | `*staterpc.Client` |\n| `TaskEvents` (store) | — | 7 | `*Store` | `*staterpc.Client` |\n| `TaskQueries` (store) | — | 7 | `*Store` | `*staterpc.Client` |\n| `TaskArchiver` (store) | — | 1 | `*Store` | `*staterpc.Client` |\n| `TaskRetryer` (store) | — | 1 | `*Store` | `*staterpc.Client` |\n| `workflow.Store` (workflow domain) | — | 3 | `*Store` | `*staterpc.Client` |\n| `CaptureStore` (store) | — | 2 | `*Store` | `*staterpc.Client` |\n| `MappingStore` (store) | — | 5 | `*Store` | `*staterpc.Client` |\n| `BindingStore` (store) | — | 6 | `*Store` | `*staterpc.Client` |\n| `BindingDispatcher` (store) | — | 3 | `*Store` | `*staterpc.Client` |\n| `BindingTaskCreator` (store) | — | 1 | `*Store` | `*staterpc.Client` |\n| `PlaybookDispatcher` (store) | — | 2 | `*Store` | `*staterpc.Client` |\n\n**Key property of the remote client:** `*staterpc.Client` is a *single* adapter that wraps one\n`StateStoreClient` and is asserted against **whichever narrow Go interfaces its caller needs**\n(exactly as `gatewayrpc.Client` asserts both `gateway.ChatContract` and\n`gateway.SessionStore`). The Go contracts stay narrow (≤8, the `interfacebloat` cap); the wire\nis one service. `workflow.Store` introduces **no new RPCs** — it is a narrow consumer face over\nthe same `Update`/`Transition`/`InsertEvent` RPCs.\n\n---\n\n## 3. Method ownership\n\n`*Store` remains the single implementation of the daemon/webui store surfaces and of\n`workflow.Store`. Ownership below is about which **process** is the primary consumer of each\nmethod and therefore which methods are genuinely cross-process on the wire vs.\ndaemon-internal.\n\nProcess legend: **DAEMON** = `archied` (incl. in-process webui + gateway runtime +\n`storerpc.Server`); **AGENT** = `archie-agent` worker (no DB connection); **GATEWAY** = store\naccess through daemon-side narrow adapter interfaces.\n\n| Contract | Primary consumers (process) | Remote on the wire? |\n|---|---|---|\n| `TaskLifecycle` (8) | DAEMON (dispatch, claim, action) + **AGENT** (`Transition`, `Update` via storerpc) | `Transition`/`Update` = **YES** (agent); rest DAEMON |\n| `TaskArchiver` (1) | DAEMON (`taskactions` archive) | not yet |\n| `TaskRetryer` (1) | DAEMON (`taskactions` retry) | not yet |\n| `TaskQueries` (7) | DAEMON + GATEWAY (status/list adapters) | not yet (`OpenPRs`/`Tasks`/`StatusCounts` cross to gateway adapters in-process) |\n| `TaskEvents` (7) | DAEMON + **AGENT** (`InsertEvent` via storerpc) | `InsertEvent` = **YES** (agent); rest DAEMON |\n| `workflow.Store` (3) | **AGENT** only | **YES** (the sole genuinely remote consumer) |\n| `CaptureStore` (2) | DAEMON (webhook intake + mapping editor) | not yet |\n| `MappingStore` (5) | DAEMON (dashboard editor + dispatch loop) | not yet |\n| `BindingStore` (6) | DAEMON (dashboard editor + dispatch loop) | not yet |\n| `BindingDispatcher` (3) | DAEMON (dispatch loop) | not yet |\n| `BindingTaskCreator` (1) | DAEMON (dispatch loop) | not yet |\n| `PlaybookDispatcher` (2) | DAEMON (EDA playbook dispatch loop) | **YES** (side-effecting-action idempotency ledger over gRPC) |\n\n**Consequence for migration order:** the **agent's `workflow.Store` is already cross-process\nalready** (via the NATS `storerpc` surrogate), so it is the first contract to move to gRPC. The\ndaemon-internal surfaces (`Capture`/`Mapping`/`Binding`) stay in-process until the daemon\nitself points at `archie-state-store`. `TaskStore` (the large composite) moves last.\n\n**Two orphan methods** — `TaskQueries.ClearTerminalTasks` and\n`TaskQueries.IncrementRetryCount` have **no production consumer** (only `*Store` tests). They\nremain part of the Go `TaskQueries`/`TaskStore` surface, and therefore part of the wire\ncontract, but are flagged for deletion: they must **not** gain dedicated consumers or wire\nmethods in this work. Do not add the rest of the codebase to them.\n\n---\n\n## 4. Domain-facing types \u0026 consumer-owned contract — resolves the contradiction\n\nThe Phase 2 acceptance criterion (\"`internal/domain/workflow` … no longer import\n`internal/store` directly — they call the State Store contract\") **cannot be met while**\n`store.Task`, `store.Status*`, `store.Source*`, and `store.WorkflowStore` live in\n`internal/store`. This section fixes where they move.\n\n**The relocation (a prerequisite that must land before or with `.4.2`):**\n\n| Current symbol (`internal/store`) | Moves to | Notes |\n|---|---|---|\n| `Task` type | `internal/domain/workflow.Task` | The task-execution record the workflow domain operates on. `internal/store` persists this type (persistence → domain). |\n| `Status` + `Status*` constants | `internal/domain/workflow` | e.g. `StatusClosedWontDo`, `StatusMerged`, `StatusPROpen`, `StatusWaitingHuman`, `StatusRunning` — task-lifecycle vocabulary. |\n| `Source` + `Source*` constants | `internal/domain/workflow` | e.g. `SourceChat` — provenance vocabulary, follows `Task`. |\n| `WorkflowStore` interface (3 methods) | `internal/domain/workflow.Store` | **Consumer-owned** (dependency rule #2). Supersedes `store.WorkflowStore`. |\n\n**Resulting dependency direction (correct per the architecture):**\n\n- `internal/domain/workflow` imports **no** `internal/store` symbols → the acceptance\n criterion is satisfied. It consumes `store.Task`'s replacement as `workflow.Task`, uses\n `workflow.Status*`/`workflow.Source*`, and depends on its own `workflow.Store` interface.\n- `internal/store` imports `internal/domain/workflow` for the `Task`/`Status`/`Source` types it\n reads and writes (infrastructure → domain). **NOTE: `internal/store` is a transitional\n compatibility location only** — its long-term home is `internal/infrastructure/store` (see\n the callout in §1). Do not treat `internal/store` as the final infrastructure boundary.\n- Every store test that constructs `store.Task{…}` is updated to the relocated type as part of\n this prerequisite. This is a mechanical rename across ~6 workflow production files\n (`workflow.go`, `steps.go`, `triage.go`, `review.go`, `feasibility.go`, `implement.go`) and\n their tests plus the store tests. `TestRecordDispatchViaExplicitTx` is removed rather than\n updated — its non-nil `*sql.Tx` path has no production consumer (see §5's `RecordDispatch`\n blocker).\n\n**What this does NOT do:** it does not implement the full §4 Workflow migration\n(`mutable Stage → StepExecution history`, `RetryCount/Attempt → Attempt records`,\n`workflow.Registry → versioned Workflow definitions`, etc.). It lands only the minimal\ntask-type + `Status`/`Source` + consumer-owned-interface relocation that the Phase 2\ndomain-decoupling criterion requires. The residue of §4 remains its own migration.\n\n**Forward-compatibility:** because the wire DTOs are derived from the domain `Task`/`Status`\n(§5), when the full §4 migration later renames `workflow.Task` (or refines it into\n`WorkflowExecution`), only `values.go` and the proto field mapping change — the RPC surface and\nconsumer interfaces are unaffected. The wire contract is deliberately type-agnostic at this\nlevel.\n\n---\n\n## 5. DTO / wire representation\n\n### Proto layout\n\n- **Service:** `proto/state/v1/state.proto`, `package state.v1`, **one** `StateStore` service\n (mirrors the gateway's single `ChatService`; proto bypasses the Go `interfacebloat` cap, as\n `ChatContract` does via `//nolint:interfacebloat`).\n- **Codegen:** the existing buf v2 pipeline. `proto/buf.yaml` (module, lint STANDARD, breaking\n FILE against `main`), `proto/buf.gen.yaml` **managed mode** with\n `go_package_prefix → github.com/samcharles93/archie-core/internal/contracts`, plugins\n `protoc-gen-go` + `protoc-gen-go-grpc`, `out: ../internal/contracts`,\n `opt: paths=source_relative`. **No `go_package` in source** — managed mode owns it. Generated\n Go lands in `internal/contracts/state/v1/` (package `statev1`), committed (`proto:check`\n fails on uncommitted/unmodified generated files).\n- **Adapter package:** `internal/infrastructure/staterpc/` with `server.go` (registers the\n generated server over the store), `client.go` (wraps the generated client into `store.*` and\n `workflow.Store` facades), `values.go` (proto ⇄ domain mappers), and `conformance_test.go`\n (bufconn local-vs-grpc dual-adapter suite), mirroring `internal/infrastructure/gatewayrpc/`.\n- **Binary:** `cmd/archie-state-store/main.go`, mirroring `cmd/archie-gateway/main.go` and the\n serve pattern in `internal/app/archied/gateway.go` (`serveGateway`).\n\n### The `StateStore` RPC set (union of the contracts, grouped)\n\nMethods are named after the store methods so the mapping is unambiguous. `workflow.Store` is a\n*view*, not new RPCs — `Update`/`Transition`/`InsertEvent` are shared with\n`TaskLifecycle`/`TaskEvents` and appear once.\n\n| Group | RPCs |\n|---|---|\n| Lifecycle | `EnqueueIssue`, `EnqueueChatTask`, `ClaimNext`, `ClaimByIssue`, `Transition`, `Update`, `Requeue`, `RecoverStale` |\n| Archive / Retry | `ArchiveTask`, `RetryTask` |\n| Queries | `TaskByIssue`, `OpenTaskByPR`, `TaskByID`, `OpenPRs`, `ClearTerminalTasks`, `Tasks`, `StatusCounts`, `IncrementRetryCount` |\n| Events | `InsertEvent`, `EventsSince`, `TaskEvents`, `WorkflowStats`, `StageStats`, `TokensByDay` |\n| Capture | `InsertCapture`, `ListCaptures` |\n| Mapping | `InsertMapping`, `GetMapping`, `ListMappings`, `UpdateMapping`, `DeleteMapping` |\n| Binding | `InsertBinding`, `GetBinding`, `ListBindings`, `UpdateBinding`, `DeleteBinding`, `ApproveBinding` |\n| Dispatch | `ArmedBindingsForSource`, `RecordDispatch`, `ListUndispatchedCaptures` |\n| Playbook | `RecordPlaybookDispatch`, `DeletePlaybookDispatches` |\n| BindingTaskCreator | `EnqueueBindingTask` |\n| Config snapshot | `PutConfigSnapshot`, `GetConfigSnapshot` |\n| Apply status | `PutApplyStatus`, `ListApplyStatus` |\n| Channel status | `PutChannelStatus`, `ListChannelStatus` |\n| Task log | `ReadTaskLog`, `StreamTaskLogContent` |\n\nThat is **51 unique contract RPCs** across one service (the `TaskEvents.Close` method is dropped).\n`Close()` is **excluded** from the wire (it is server lifecycle, not a client call) — see §11.\n\nThe channel-status pair follows the config snapshot's split for the same reason: the writer is the process that HOSTS the channels -- the Messaging Service, which alone observes whether a channel is starting, running or failed -- and the reader is the UI process. Both RPCs are administrative, so a task-scoped grant reaches neither. A channel's state is runtime state rather than a settings document, which is why it is not a `ControlPlaneService` resource kind: a failed channel is not a document anybody edits. Asking for a reload travels on its own surface rather than through this one; see `docs/architecture/migration-decisions.md`, \"Channel state to the dashboard\".\n\nThe config-snapshot pair was added by `archie-core-ymut` (see\n`docs/architecture/migration-decisions.md`, \"Dashboard configuration page\"). It is the one\nsurface whose writer is the daemon and whose reader is the UI process, and both RPCs are\nadministrative: `authorizesTaskScopedCall` is deny-by-default, so a task-scoped grant reaches\nneither. `ConfigSnapshot.document` is opaque to this contract — the store neither parses nor\nvalidates it, and `schema` names the projection so a reader can refuse a shape it does not\nunderstand.\n\nThe apply-status pair was added by `archie-core-pskb` (see\n`docs/prds/control-plane-apply-status.md`). It is administrative for the same reason as the\nconfig-snapshot pair, and deny-by-default in `authorizesTaskScopedCall` covers it without a\nnew rule. Unlike that pair, the read is a list: `ListApplyStatus` returns every process's\nrecord and an empty slice means nothing has reported yet, so the `found=false` convention for\nabsent single rows does not apply. The rows are keyed on `(process, kind)` and replaced in\nplace, because each process re-stamps its record on an interval and no reader wants a history\nof re-stamps.\n\nThe task-log pair was added by `archie-core-iaqx` (see `docs/prds/ui-service-boundary.md`'s\nroute inventory). It is the one surface whose payload is not a row: the store service reads\nthem out of the state directory it owns with `internal/logging`'s own reader, so the wire\ncarries `logging.Entry`/`logging.Query` values and neither side restates the JSONL format.\n`ReadTaskLog` returns a decoded page and `StreamTaskLogContent` returns the raw bytes\nserver-streamed, because a log file rotates at 20 MB — past gRPC's 4 MiB unary cap — and the\nsame reasoning that made `StreamCaptures` supersede `ListCaptures` applies. `found=false` with\nno error means the attempt has no log; a service with no reader answers `Unavailable` instead,\nand the two answers are load-bearing for the dashboard: only the second is a deployment\ncondition, and the panel that collapsed them told operators task logging was switched off when\nit was not. The Go facade is `storecontract.TaskLogStore` (2 methods), carried by\n`*staterpc.Client` alongside the rest.\n\nThe playbook pair was added by `archie-core-t2db.17`'s implementation (see\n`docs/prds/eda-playbook-engine.md`, execution-time gap 2). It is administrative\nfor the same reason as the config-snapshot and apply-status pairs, and\ndeny-by-default in `authorizesTaskScopedCall` covers it without a new rule. The\nrow is the structural `(playbook_id, playbook_version, event_id, action_id)`\ntuple, written **before** the action's side effect fires (the reverse of\n`binding_dispatches`), so a duplicate returns `store.ErrAlreadyDispatched` and\nthe coordinator skips the invoke rather than repeating a non-revocable side\neffect. `DeletePlaybookDispatches` is the explicit reclamation primitive; nothing\ncalls it automatically, because a playbook file's absence at boot is not a\ndeliberate removal (`eda-playbook-engine.md`, \"Lifetime\"). The Go facade is\n`storecontract.PlaybookDispatcher` (2 methods), carried by `*staterpc.Client`\nalongside the rest.\n\n### Domain types and `values.go` mapping\n\n`values.go` maps proto ⇄ these types. The task model for the **agent** path is the domain-owned\n`workflow.Task` (§4); the daemon/webui store surfaces return the same `workflow.Task` (the store\npersists the domain type).\n\n| Domain type | Package | Notes for `values.go` |\n|---|---|---|\n| `workflow.Task` | `internal/domain/workflow` | time fields stored as layout strings; convert to/from `google.protobuf.Timestamp` |\n| `events.Event` | `internal/events` | `Timestamp` → `google.protobuf.Timestamp` |\n| `store.CapturedEvent` | `internal/store` | headers/body/remote_addr; body capped 256 KiB |\n| `mapping.Mapping` | `internal/domain/mapping` | JSON-path field bindings |\n| `binding.Binding` | `internal/domain/binding` | secret is **already decrypted** before returning; keep decrypted form out of any durable/logged representation |\n| `workflow.Status` / `workflow.Source` | `internal/domain/workflow` | enum-like; map to proto enum or string, keeping the names stable for compatibility |\n| `store.WorkflowStat` | `internal/store` | aggregate row |\n| `store.StageStat` | `internal/store` | aggregate row |\n| `store.DayTokens` | `internal/store` | aggregate row |\n\n### Wire-contract blocker fixed here: `RecordDispatch`\n\n`store.BindingDispatcher.RecordDispatch(ctx, tx *sql.Tx, bindingID, bindingVersion int64,\ncaptureID int64, taskID int64) error` takes a **`*sql.Tx`** that cannot cross a gRPC boundary.\n\n**Evidence that the `*sql.Tx` is removable:** the dispatch loop calls `RecordDispatch(ctx, nil,\n…)` (`internal/daemon/daemon.go`). The implementation falls back to the store's own\n`*sql.DB` when `tx == nil` (\"best-effort, not atomic\"; the comment and the\n`TestRecordDispatchViaExplicitTx` test are the only consumers of the non-nil `tx` path).\nProduction never opens the explicit transaction.\n\n**Decision:** the contract drops `*sql.Tx` from `RecordDispatch` (the method becomes\n`RecordDispatch(ctx, bindingID, bindingVersion, captureID, taskID) error`). The store service\nowns any transaction boundary internally when atomicity is wanted; the at-most-once\nguarantee is the `INSERT OR IGNORE` dedup row, which is already transactionless in production.\n`TestRecordDispatchViaExplicitTx` is superseded by the store service's own internal-transaction\ntest. **This is a required interface migration that must land before or with `.4.2`** (the\ncurrent `interface.go` signature and test still use the transaction-bearing form).\n\n---\n\n## 6. Agent State Store client handoff\n\nMoving the agent from NATS `storerpc` to gRPC `workflow.Store` requires the agent process to\nreceive a State Store endpoint. The agent already receives its NATS connection settings via\nboot-time env/flags (see `cmd/archie-agent/main.go`: `-nats-url` / `NATS_URL`, `NATS_TOKEN`\nenv), and the daemon injects them into the task container. The State Store handoff follows the\nsame seam.\n\n**Boot handoff contract (additive, not replacing the NATS settings):**\n\n| Field | Where | Source | Notes |\n|---|---|---|---|\n| `StateStoreTarget` | `agentworker.Settings` | env `STATE_STORE_URL` (or `-state-store-url` flag), set by the daemon in the container env | `host:port` of the State Store gRPC server. Loopback/Docker-bridge address — the agent reaches it the same way it reaches the daemon's NATS. |\n| `StateStoreToken` | `agentworker.Settings` | env `STATE_STORE_TOKEN` (or `-state-store-token` flag) | Per-task bearer token carried in gRPC metadata by a client interceptor (see §9). |\n| `StoreTimeout` | `agentworker.Settings` | a bounded per-call timeout | Mirrors `natsrpc.Client.Timeout` / the existing `rpcTimeout`; bounds each State Store call when the caller `ctx` has no deadline. |\n\n**Wiring change in the transport façade:**\n\n- `internal/infrastructure/agenttransport/nats.Transport` — `Store(timeout) store.WorkflowStore`\n returns `\u0026storerpc.Client{Conn: t.conn, Timeout: timeout}`. It becomes\n `Store(timeout) workflow.Store` and returns a *single long-lived* `staterpc.Client` dialed to\n `settings.StateStoreTarget` (with the token interceptor and per-call timeout).\n- `internal/app/agentworker/worker.go` `Settings` gains the three fields above; `taskServiceTransport`\n and `taskDependencies` change their `store` type from `store.WorkflowStore` to\n `workflow.Store` (via `domain/workflow.TaskContext{Store: …}`).\n- Reconnect/deadline: `grpc.NewClient` (or the existing dial path) provides transparent reconnect;\n each RPC is bounded by the per-call context deadline (`StoreTimeout`), matching the\n request/reply timeout the NATS path used. No streaming is involved, so no server-sent\n reconnect semantics apply.\n\n**Deletion rule for the NATS path:** the NATS `storerpc` (Server, Client, subjects\n`archie.store.update|transition|insert_event`) is deleted **only after** the agent has fully\nmoved to gRPC and the container env carries `STATE_STORE_URL`. Until then both paths may coexist\n(agent flips when the daemon starts injecting `STATE_STORE_URL`); there is no dual-live window\nacross a single agent process, which is either NATS-backed or gRPC-backed, never both.\n\n### Operational specifics (in-process listener → address → per-task injection)\n\n- **Who binds the in-process State Store listener:** the daemon (`archied`), at bootstrap, when\n the State Store is served in-process/multiplexed (mirroring `serveGateway` in\n `internal/app/archied/gateway.go`): a `grpc.NewServer()` + `RegisterServer` + graceful stop on\n `ctx.Done()`.\n- **Address selection (Docker-bridge reachable):** reuse the host-gateway resolution already used\n for embedded NATS (`internal/container/network.go` `RequireHostGateway` /\n `resolveHostGateway`). Per §9's single listener-topology rule, the listener binds the\n **host-gateway bridge address** when agent containers consume it (the Phase 2 default), or\n loopback-only when the daemon is the sole consumer. The chosen address is stored on the daemon\n as `d.ConnectedStateStore.URL` (analogous to `d.ConnectedNATS.URL`).\n- **Per-task injection:** `containerEnv(task)` (`internal/daemon/daemon.go`) appends\n `STATE_STORE_URL=\u003cd.ConnectedStateStore.URL\u003e` and `STATE_STORE_TOKEN=\u003ctoken\u003e` alongside the\n existing `NATS_URL`/`NATS_TOKEN`. The agent reads them via env (`-state-store-url` /\n `STATE_STORE_URL`, `-state-store-token` / `STATE_STORE_TOKEN`), the same seam as `NATS_URL`. A\n reloaded `[services.state]` must not point new containers at a store the daemon is not serving\n — the same immutability rule as NATS.\n\n---\n\n## 7. Error semantics\n\n### Current state (both boundaries lose error identity)\n\n- **NATS `natsrpc.Envelope`** (`internal/natsrpc/natsrpc.go`) serialises any error to a single\n `err.Error()` string and rehydrates a fresh `errors.New(string)`. The original type, `%w`\n chain, and sentinel identity are lost → `errors.Is(err, store.ErrStaleTransition)` is\n **false** across the boundary.\n- **gRPC `gatewayrpc`** returns raw `err` from handlers; gRPC-Go converts to\n `codes.Unknown` with only the message. The **only** explicit code in the repo is\n `codes.Unavailable` for a missing session store (`server.go`). No status-code mapping for\n store errors exists.\n\n### Decision — structured mapping so `errors.Is` survives\n\nThe `StateStore` service must map the store's sentinel errors to canonical gRPC codes and\nrehydrate them to the *same* sentinels on the client, so consumer `errors.Is` checks keep\nworking — an improvement over the flattening it replaces, not a regression.\n\n| Store sentinel / outcome | gRPC code | Client rehydrates to |\n|---|---|---|\n| `store.ErrStaleTransition` | `FailedPrecondition` | `store.ErrStaleTransition` |\n| `store.ErrBindingNotFound` | `NotFound` | `store.ErrBindingNotFound` |\n| `store.ErrMappingNotFound` | `NotFound` | `store.ErrMappingNotFound` |\n| `store.ErrBindingOverlap` | `FailedPrecondition` | `store.ErrBindingOverlap` |\n| `store.ErrBindingTransition` | `FailedPrecondition` | `store.ErrBindingTransition` |\n| `store.ErrAlreadyDispatched` | `AlreadyExists` | `store.ErrAlreadyDispatched` |\n| not-found-as-`(nil,nil)` (`TaskByID`, `GetBinding`, `GetMapping`, `ClaimNext`, `ClaimByIssue`, …) | `NotFound` with a `found=false` field (**not** an error) | `(nil, nil)` |\n| infra / wrapped error | `Internal` (**sanitised** message, see below) | wrapped `fmt.Errorf(...%w)` |\n| capability absent | `Unavailable` | context/typed error |\n\nThe `found=false`-not-error convention matters: the repo deliberately returns `(nil, nil)` for\nnot-found in reads and a sentinel for not-found in writes. The wire must preserve **both**\nconventions (a `GoogleRPCStatus` with `NotFound` is an *error*, which would wrongly surface to\ncallers that expect `(nil, nil)`). Implement `found` as an explicit boolean field on read\nresponses, and map `(nil,nil)` ⇄ `found=false` + `OK`.\n\n### Error-detail sanitisation (rev. 2c)\n\nFull internal error messages must **not** be placed in gRPC `details` for remote callers. SQL\npaths, provider/forge details, secrets, and stack traces are sensitive internals. The rule:\n\n- **Sentinel errors** — the client rehydrates the sentinel from the code; the message string in\n `status.Error` is a short, public, stable phrase (e.g. `\"stale transition\"`), not the raw\n wrapped chain.\n- **Infra errors** — the wire carries a **sanitised** public message + the `Internal` code.\n The **full** error (with `%w` chain, SQL, provider detail) is logged **server-side only** by\n the store service. The client rehydrates a generic wrapped error and must not surface the raw\n text to users.\n\n---\n\n## 8. Limits\n\n**Interface bloat (`interfacebloat` cap = 8)** — `.golangci.yml` `interfacebloat: max: 8`,\nwhich overrides the upstream default of 10. The Go consumer facades must stay ≤8 methods.\n`TaskLifecycle` is exactly 8 (at the cap); `TaskStore` is the 24-method composite that the cap\nis why it is decomposed; proto services bypass the cap (like `ChatContract`).\n\n**Payload caps:**\n- Capture body: `defaultCaptureMaxBodyBytes = 256 KiB` (may reduce further; the wire message\n must not exceed gRPC's default 4 MiB receive limit for a single capture). A single capture\n fits, but a *batch* of them did not: the caller-side cap of up to 100 captures at 256 KiB each\n could exceed 4 MiB in one unary `ListCaptures`/`ListUndispatchedCaptures` response. `.4.7` adds\n `StreamCaptures`/`StreamUndispatchedCaptures` (server-streaming, one capture per message) as\n the path both `staterpc.Client` methods actually call; the original unary RPCs stay defined,\n unmodified and deprecated, only so `buf breaking` protects an in-flight rolling deploy.\n- Task `detail`/`park_reason`/event `Detail`: 4000-char hard cap (`clip(s, 4000)`).\n\n**Retention (stays server-side in the store; the wire does not carry pruning decisions):**\n- `events` table: **no prune, unbounded growth** — a known risk to revisit, not changed here.\n- `captured_events`: prune-on-write, default 7 days / 5000 events.\n\n**Pagination/arity:** store read methods take a raw `int limit` (no store-side cap); the\ncaller-side caps (task list 100, log 2000, captures 100, capture scan window 5000, binding\ndispatch batch 100, session list 20) remain caller-side. The wire passes `limit int` and returns\n`[]*` slices, mirroring the existing Go signatures — **no cursor/paging protocol is introduced**\nin this work (the current `EventsSince(sinceID, limit)` offset cursor is the only one, and it\nstays as-is).\n\n---\n\n## 9. Transport security boundary\n\nThe gateway precedent is loopback-only, unauthenticated gRPC. The State Store holds the\ncrown-jewel `archie.db` (task/event/capture/mapping/binding data), so its boundary must be\nstated explicitly rather than left implicit.\n\n**The State Store's equivalent boundary (rev. 2c):**\n\n- **Listener topology — single rule (rev. 2c).** One in-process listener; its bind address is\n chosen by the consumer set, never both at once:\n - *Agent containers consume it* (the Phase 2 driver during multiplexed serving) → bind to the\n **host-gateway bridge address** so containers can reach it, and **token auth is mandatory**\n (a bridge host is reachable by any process on that bridge). A container **cannot reach the\n host's `127.0.0.1`**, so loopback-only is NOT usable for the agent-consumed path.\n - *Only the daemon consumes it* (no agent) → bind **loopback-only** (`--listen 127.0.0.1`),\n `insecure` is fine because it is not network-reachable; no token required.\n - There is **no dual-listener** design. The practical default during `.4.2` (the reason\n in-process serving exists is to give the agent a gRPC target) is the **bridge address +\n token**. `127.0.0.1` is reserved for the daemon-local-only case.\n- **Agent access → the container bridge.** The agent runs in a task-scoped container on the\n host and reaches the State Store over the Docker bridge gateway (the bind address from the\n topology rule above), exactly as it reaches the daemon's NATS. To close the \"any host\n process could reach the store\" gap, the agent **authenticates with a per-task bearer token**\n carried in gRPC metadata by a client interceptor (`StateStoreToken`, §6). Network\n reachability + the token is the trust boundary for the default bridge topology.\n- **The daemon** uses the local adapter (`*store.Store`) in-process by default, so no network\n boundary exists until `[services.state].target` is set; when it is set, the daemon client\n authenticates with a token supplied alongside the target.\n- **Non-loopback / k8s:** TLS/mTLS is an **operator decision**, not a default. Any non-loopback\n exposure (e.g. `--listen 0.0.0.0`, a CDN, a cross-host service) requires an explicit decision\n and mTLS or token auth — mirroring the gateway's `--listen`/`--enable-cors-header`\n confirmation caveat. The NATS-KV discovery fallback supplies endpoints but does not add\n transport security; that stays the operator's responsibility.\n\n### Token lifecycle (rev. 2c; scoping added in `.4.7`)\n\n- **Generation:** the daemon generates a per-task bearer token when it acquires the container\n (or per incumbence) and injects it via `containerEnv` as `STATE_STORE_TOKEN`. A single token\n may cover a task's container lifetime.\n- **Registration (`.4.7`, post-standalone-cutover):** the State Store is now its own process\n (`.4.3`/`.4.6`), so the daemon can no longer just hold an in-process issued-token set for the\n server to check — it registers the token at the remote authority via the\n `RegisterTaskGrant`/`RevokeTaskGrant` RPCs (`staterpc.GrantIssuer`, `daemon.StateStoreGrantIssuer`),\n admin-token-authenticated calls a task-scoped token itself can never make.\n- **Lifetime:** tied to the task + container grace period; rotated on a new container acquisition\n for the same task; a token for a released container is invalidated (`RevokeTaskGrant`, deferred\n alongside `ContainerPool.Release`). An unrevoked grant still expires server-side (bounded by\n `RegisterTaskGrantRequest.lifetime_seconds`, capped at 7 days) so a daemon crash before revoke\n cannot leave a grant valid indefinitely.\n- **Validation and scope (`.4.7`):** a gRPC interceptor (`staterpc.TaskGrants.UnaryInterceptor`)\n validates the token against the State Store's own issued-grant set; missing/unknown/expired →\n `codes.Unauthenticated`. Unlike the daemon's own administrative token (full access to all ~40\n RPCs), a task-scoped grant additionally authorizes only `Update`/`Transition`/`InsertEvent` on\n its own task ID — every other RPC, including the streaming capture surface and\n `RegisterTaskGrant`/`RevokeTaskGrant` themselves, is `codes.PermissionDenied` for a task-scoped\n caller. The token is carried in gRPC metadata, never in a URL.\n\n### Fail-closed on non-loopback (rev. 2c)\n\nIf `[services.state].target` points to a non-loopback address and **neither** TLS **nor** a token\nis configured, the State Store client (daemon or agent) **fails closed** — it refuses to\nstart/dial rather than fall back to a plaintext, unauthenticated remote connection. This mirrors\nthe gateway's `--listen`/`--enable-cors-header` confinement caveat and prevents a silent\nunauthenticated remote store. The loopback + token default is the only path that starts without\nan explicit operator decision.\n\n---\n\n## 10. Mode / config shape\n\n**Follow the gateway's presence-based `target` seam, not the NATS `mode` enum.**\n\n- **Services are keyed by name, not named as struct fields.** A\n `State ServiceConnection` field per service is the duplication\n `docs/prds/service-registry.md` removes, and this document's summary describes\n the seam as `[services.\u003cname\u003e]`:\n ```go\n type Services map[string]ServiceConnection\n ```\n registered once via `RegisterService(context, name, target, listen, tokenEnv)`. The TOML\n shape below is unchanged, so nothing else in this section is affected; read\n `cfg.Services.Get(\"state\")` wherever it says `cfg.Services.State`.\n- **Empty `Target` → startup error.** The local `*store.Store` adapter default was\n withdrawn when the State Store was extracted (§12 step 7): the standalone `archie-state-store` process\n exclusively owns `archie.db`, so both `archied` and `archie-gateway` now reject an empty\n target (`internal/app/archied/bootstrap.go` and `state_store_client.go`: `services.state.target\n is required`). **Set `Target` → dial gRPC** with `*staterpc.Client`; no client opens the\n database itself. This differs from the gateway, where `Target` is defaulted to\n `127.0.0.1:8585` because `archie-gateway` is already a separate process whose client and\n server share one default.\n- `[services.state]` is present and **uncommented** in `config.example.toml`, with `target`\n marked REQUIRED, because the value is load-bearing for every client. A commented-out\n `target` cannot boot, so the section ships uncommented.\n- Composition root: `internal/app/archied/bootstrap.go` (the daemon) and\n `internal/app/archieui/config.go` (the UI process) both read `cfg.Services.Get(\"state\")` and\n dial the remote State Store. The task store is built only in the standalone\n `archie-state-store` process (`internal/app/archied/state_store.go`), which is the only path\n that opens the task tables.\n\n\u003e **Mode fork, decided.** The gateway uses a single `target` address (no enum). The State Store\n\u003e boundary uses the same seam for symmetry, and **both now require a non-empty `target`**: the\n\u003e State Store's local-store default was withdrawn when the process was extracted (§12 step 7).\n\u003e No `mode` field is added. A companion `STATE_STORE_TOKEN` secret (or a\n\u003e `[services.state] target_token` key) carries authentication.\n\n---\n\n## 11. Local-adapter compatibility + conformance\n\n- `*store.Store` (local) and `*staterpc.Client` (remote) satisfy the **same** contract\n interfaces. Use **explicit** `var _` assertions on **both** adapters (the gateway's local\n adapters use only structural/return-type assertions; the store precedent's explicit `var _`\n style is stronger and should be adopted on both sides).\n ```go\n var _ workflow.Store = (*staterpc.Client)(nil) // first contract to move (agent)\n var _ store.TaskStore = (*staterpc.Client)(nil) // daemon path, added progressively\n var _ store.CaptureStore = (*staterpc.Client)(nil)\n var _ store.MappingStore = (*staterpc.Client)(nil)\n var _ store.BindingStore = (*staterpc.Client)(nil)\n var _ store.BindingDispatcher = (*staterpc.Client)(nil)\n var _ store.BindingTaskCreator = (*staterpc.Client)(nil)\n var _ store.PlaybookDispatcher = (*staterpc.Client)(nil)\n // local side (in store), already present:\n var _ workflow.Store = (*Store)(nil)\n ```\n- **Conformance suite:** a `staterpc/conformance_test.go` bufconn suite runs the **same**\n contract through both `\"local\"` and `\"grpc\"` adapters (mirroring\n `gatewayrpc/conformance_test.go`), asserting error-sentinel fidelity and the\n not-found-as-`(nil,nil)` convention both ways.\n- **`Close()`:** stays on the Go store interfaces (so `TaskStore`/`TaskEvents` remain fully\n satisfiable) but is **not** a `StateStore` RPC. The remote `staterpc.Client` implements\n `Close()` as a **no-op**; the store service owns its own DB lifecycle. When the daemon hands\n the store to `archie-state-store`, `b.cleanup()` no longer calls `store.Close()` on a remote\n client (it is the daemon's own lifecycle only when it owns the store in-process).\n\n---\n\n## 12. Cutover / deletion rules\n\n1. **Ratify the contract (this document).** No code.\n2. **Domain type + interface relocation (prerequisite, coupled with §4).** Move `Task`/`Status`/\n `Source` and the `WorkflowStore` interface into `internal/domain/workflow` (as\n `workflow.Task`/`workflow.Status`/`workflow.Source`/`workflow.Store`), so\n `internal/domain/workflow` imports no `internal/store`. Update the store tests and the 6\n workflow production files. **This must land before or with `.4.2`, because `.4.2` cannot move\n the agent's store contract off `store.WorkflowStore` while the domain still imports it.**\n3. **`.4.2` — generalize `storerpc` to the gRPC contract.** Move the agent's `workflow.Store`\n (`Update`/`Transition`/`InsertEvent`) from NATS JSON `storerpc` to `staterpc.Client` over\n gRPC, using the §6 handoff. The daemon serves the gRPC `StateStore` server **in-process\n (multiplexed)** first so the agent has a gRPC target before the standalone binary exists\n (PRD §3 \"multiplexed in-process is the migration default\"). Bind the in-process listener and\n inject `STATE_STORE_URL`/`STATE_STORE_TOKEN` per task (see §6 \"Operational specifics\"). Also\n land the `RecordDispatch` `*sql.Tx` drop (see §5) in the same span. Flip the agent's\n `Transport.Store` to a gRPC client once `STATE_STORE_URL` is injected.\n4. **Delete NATS `storerpc` only after the agent is fully on gRPC** — delete `internal/storerpc`\n (Server, Client, subjects `archie.store.update|transition|insert_event`), the\n `natsrpc.RegisterAll` registration, and the now-unused `store.WorkflowStore` NATS path in\n `internal/infrastructure/agenttransport/nats`, in the same commit.\n5. **`[services.state].target` stays empty (inproc)** through `.4.2`; the `archie-state-store`\n `cmd/` binary is **`.4.3`** and hosts the same `StateStore` gRPC service.\n6. **Swap daemon-internal contracts one at a time** (only once the daemon points at\n `archie-state-store`): `Capture → Mapping → Binding → TaskStore`. Each swap is independent —\n add the remote-client composition path, run the per-contract conformance suite, flip\n `[services.state].target`, delete the local use. A consumer mixes in-process and remote\n contracts for a contract-code (no dual-live *window*; each contract is either local or remote\n at a time).\n7. **Final flip / deletion of the in-process serving path:** when the daemon no longer owns the\n store (i.e. `TaskStore` has moved), delete the in-process task-store /\n `wireWebStoreSurfaces` path in the daemon in a **same-commit** deletion — no dual-store\n ownership. The single SQLite file remains owned by `archie-state-store`.\n8. **Never** split `archie.db` or add a second store. The gateway keeps its own already-separate\n session SQLite; that is unchanged and out of state-store scope.\n\n---\n\n## 13. Decisions recorded (for the review trail)\n\n- **`internal/store` is a transitional compatibility location** (target\n `internal/infrastructure/store`), not the final infrastructure boundary. The producer-owned\n daemon/webui store surfaces live there now but may relocate with the implementation.\n- **In-process listener + token lifecycle:** the daemon binds the in-process State Store\n listener to the **host-gateway bridge address** when agent containers consume it (the Phase 2\n default) or loopback-only when the daemon is the sole consumer — one listener, one address\n rule, never both (rev. 2c). It selects the bridge address by reusing the embedded-NATS\n host-gateway resolution and injects `STATE_STORE_URL`/`STATE_STORE_TOKEN` per task via\n `containerEnv`. Token is per-incumbence (task + container grace period) and validated by a\n gRPC interceptor → `codes.Unauthenticated` on missing/unknown/expired. Non-loopback targets\n fail closed without TLS or a token.\n- **§12 step 2 (domain type + interface relocation) is done** — `internal/domain/workflow`\n imports no `internal/store` symbol and `store.WorkflowStore` is gone from `TaskContext`. The\n broader Phase 2 acceptance criterion (State Store as its own gRPC service) is still open;\n `.4.2` picks up from here.\n- **Ownership split:** domain-facing workflow contract is **consumer-owned** in the domain\n (`workflow.Store` + `workflow.Task`/`Status`/`Source`, per dependency rules #2 and #7);\n daemon/webui store surfaces stay **producer-owned** in `internal/store`. `store.WorkflowStore`\n is superseded by `workflow.Store`.\n- **One `StateStore` gRPC service** (49 contract RPCs, grouped by contract) + narrow Go consumer facades\n (≤8) on `staterpc.Client` — mirrors the single-`ChatService` precedent.\n- **Domain type relocation** (`Task`/`Status`/`Source` → `internal/domain/workflow`) is a\n Phase 2 prerequisite, pulled forward from migration-decisions §4 (minimal bound,\n `Task → WorkflowExecution`); the rest of §4 stays a separate migration.\n- **`RecordDispatch` drops `*sql.Tx`** (wire-blocker; production passes `nil`) — a\n required interface migration landing before/with `.4.2`.\n- **`Close()` not on the wire**; remote client `Close()` is a no-op.\n- **Structured gRPC error codes** with client rehydration to store sentinels (preserves\n `errors.Is`), a `found=false` + `OK` convention for read not-found, and **sanitised public\n error details** (full detail logged server-side only).\n- **Agent handoff:** `agentworker.Settings` gains `StateStoreTarget`/`StateStoreToken`/\n `StoreTimeout`, injected as `STATE_STORE_URL`/`STATE_STORE_TOKEN` env (mirrors the NATS\n handoff); `Transport.Store` returns a long-lived gRPC `workflow.Store` client.\n- **Transport security + listener topology (rev. 2c):** one in-process listener, bound to the\n host-gateway bridge address when agent containers consume it (the Phase 2 default) or\n loopback-only for a daemon-local-only consumer. `insecure` only on loopback; mandatory\n per-task bearer-token auth on the bridge; TLS/mTLS is an operator choice for any\n non-loopback/k8s topology.\n- **Mode = presence-based `[services.state].target`**, default **local** (not yet extracted);\n add `[services.state]` to `config.example.toml`.\n- **Explicit `var _` assertions on both adapters**; per-contract bufconn conformance.\n- **Migration order:** domain type/interface relocation → `workflow.Store` (agent) → stand up\n `archie-state-store` → `Capture` → `Mapping` → `Binding` → `TaskStore`; NATS `storerpc`\n deleted once the agent is on gRPC; in-process store path deleted in a same-commit flip.\n- **Orphans `ClearTerminalTasks`/`IncrementRetryCount`** retained but not built upon.\n" }, { "url": "/dev/prds/status-health-surface/", @@ -430,7 +430,7 @@ "url": "/dev/prds/task-run-detail/", "title": "Per-run task detail: attempts, stages, changes, config, debug", "section": "PRDs", - "body": "# Per-run task detail: attempts, stages, changes, config, debug\n\n**Status:** Approved\n**Authority:** the surface contract itself is\n`docs/architecture/observability.md`.\n**Date:** 2026-09-18\n**Compounds with:** `docs/prds/task-logs-pagination.md` (a bounded read says\nwhat it does not show), `docs/prds/status-health-surface.md` (no surface that\nanswers something already answered elsewhere).\n\n## Decision\n\nA task gets one detail page that answers \"what did this run actually do?\" per\n**attempt**: an ordered stage rail, the change each commit/push produced, the\neffective configuration the attempt ran under, the attempt's log, and the raw\nrecord behind all of it.\n\nThe single organising idea is **attempt attribution**. The dashboard\nrenders a task's whole event stream as one flat timeline, so a retried task\nreads as one long run with two of everything. Every new panel is scoped to one\nattempt and says which one.\n\nThe second organising idea is **provenance, not re-derivation**. What a run\nproduced is captured at the producer, while the evidence exists, and persisted\ndurably. Nothing in this surface tries to reconstruct a past run from state\nthat has since moved on.\n\n## The requirement set\n\n| # | Requirement | Mechanism |\n| --- | --- | --- |\n| R1 | Per-attempt stage rail: stages with name, start, duration, terminal status and error, ordered within one attempt | `GET /api/tasks/{id}/attempts` reads the task's events and groups them by `attempt` |\n| R2 | Events carry an attempt, migrated safely and carried on the wire | `events.attempt` column + presence-gated migrator arm; `Event.attempt = 11` |\n| R3 | Changed files captured at commit/push, persisted per task, exposed, shown with repo/PR links | `changes_captured` durable event written by the commit/push steps; `GET /api/tasks/{id}/changes` |\n| R4 | The effective configuration an attempt ran under, secrets redacted, shown | `config_captured` durable event written once per attempt; read from the events response the page already holds |\n| R5 | Log pane filters by attempt, stage and level; absence stays distinct from an unreadable deployment | `logging.Query.Stage` + `ReadTaskLogRequest.stage = 9`; the pane keeps `disabled` and `found=false` apart |\n| R6 | Retry is reachable from the run view and produces a new attempt shown as a new run | Existing retry action, labelled as a NEW run whose confirm states the previous attempt's commits are discarded |\n| R7 | Raw JSON of the task record and its events for the selected attempt | `GET /api/tasks/{id}/debug` |\n| R8 | Deep-linkable, reload-safe per-task detail page that keeps existing URLs working | `/tasks/{id}` route beside the existing `/tasks?task=N` and `/tasks?status=...` |\n| R9 | Design polish: hierarchy, tokens, both themes, responsive rail, a11y, loading/empty/error states | UI lane; measured at 480px and 700px in both themes |\n| R10 | Documentation of record | This document plus `docs/architecture/observability.md` |\n| R11 | Red-first tests for every new behaviour, full gate, `task ui`, forced `go build ./...`, `go test -race` on touched packages | Per lane; the anti-cost-cutting requirement lives here |\n\n## Attempt attribution: a column, not a guess\n\n`events.Event.Attempt`, persisted by `internal/store` as the `events.attempt`\ncolumn (`internal/store/events.go`, `eventsSchema`), is a real column,\n`NOT NULL DEFAULT 0`. It was chosen over segmenting the stream on `task_queued` /\n`task_retried` because a fold would have to guess which side of a boundary an\nevent belongs to when the daemon restarts mid-attempt, when `RecoverStale`\nrequeues, or when two attempts interleave across a crash. A guess rendered as\nstructure is exactly the false fidelity this surface exists to avoid; a column\nis durable, queryable and ordered by construction.\n\n**`0` means \"not attributable\".** It is every row written before this change\nplus the deliberately task-agnostic producers. It is not an attempt number and\nis never rendered as one. The attempts endpoint reports the count of\nunattributed events so the page can say so instead of presenting old history as\nattempt 1.\n\n## Migration against an existing `archie.db`\n\nThe store's migrator (`migrateTasks` in `internal/store/store.go`) already reads\n`PRAGMA table_info` through `tableColumns` and applies a presence-gated\n`ALTER TABLE ... ADD COLUMN` list. This change adds an `events` arm: a\n`tableColumns(ctx, tx, \"events\")` read beside the other two, and the entry\n`ALTER TABLE events ADD COLUMN attempt INTEGER NOT NULL DEFAULT 0`.\n\nThree consequences a future maintainer must not get wrong:\n\n- `eventsSchema` is `CREATE TABLE IF NOT EXISTS` (`internal/store/events.go`),\n so on an existing database it is a **no-op**. The migrator arm is the only\n thing that adds the column there. This is why the red test builds a raw\n SQLite file with the pre-change `events` DDL and must fail before the arm\n exists: every temp-database test passes either way.\n- **`user_version` stays 3.** The loop is presence-gated, not version-gated;\n bumping it would force an edit to the test that exists to prove a newer\n schema is refused.\n- **No backfill, ever.** Existing rows keep `attempt = 0`. Inferring an attempt\n number for them would be fabrication.\n\n## Changed files: captured at the producer\n\nThe measurement is taken inside the shared step library immediately after a\nsuccessful commit and after a successful push (`StageCommit` and\n`StageCommitPush` in `internal/domain/workflow/steps.go`) — the one place\nwhere the worktree, the branch, the base, the task and the store are all in\nscope, and the only place that knows a push actually happened.\n\nBoth alternatives were rejected on evidence, not taste:\n\n- **Reading the diff on demand** is impossible. The worktree is deleted on\n merge, close and no-PR terminal states (`internal/worktree.(*Manager).Cleanup`,\n called from the daemon's terminal paths) and a retry resets the branch onto\n its base (`internal/worktree.(*Manager).refresh` → `resetOnto`). There is\n nothing left to read.\n- **Reading it from the forge** is impossible. The forge contract exposes\n no changed-file capability, and adding one would cost three implementations\n plus an RPC and would still fail for chat-sourced tasks with no PR.\n\nBinding consequences:\n\n- Capture **degrades, never parks**. A `Trees` implementation without the\n capture seam captures nothing, logs, and lets the run continue. A reporting\n failure must never fail a task.\n- The value types live in `internal/domain/workflow/task` (`changes.go`), not in\n package `workflow` and not in `internal/worktree`. Package `workflow` has no\n import of `internal/worktree`, and this subpackage is not projected into the\n interpreted-stage symbol table, so declaring the types here widens nothing and\n needs no symbol-table regeneration. The capture method stays off that surface\n entirely.\n- `files` is capped at 200 entries because an event's `data` is not\n length-limited by the store. `totals` always covers the **full** set and\n `truncated: true` says when entries were dropped.\n- `binary` means \"no textual hunks\", not \"a binary file\". Rename pairing is\n go-git behaviour the lane pins with a test; where the repository records a\n rename as add + delete, the surface reports what was recorded.\n\n## Per-run configuration: a durable event, not a table\n\n**Decision.** The effective configuration an attempt ran under is persisted as a\n`config_captured` durable event (`internal/events.KindConfigCaptured`), on the\nsame attempt key R2 adds — not in a table, and not behind new RPCs.\n\n**Why.** Three reasons, in order of weight:\n\n1. Events are already this system's durable per-attempt provenance stream, and\n they demonstrably carry aggregation (`StageStats`, `TokensByDay` both\n aggregate over persisted events). R3 puts its diffstat there in this same\n change. A table plus two RPCs would have given one feature **two provenance\n mechanisms for one purpose**.\n2. It adds no RPC, so the ratified state-store surface is untouched and nothing\n in `docs/prds/state-store-contract.md` (rev 2c) changes.\n3. The page already fetches the task's events to build the rail (R1) and to read\n the diffstat (R3), so the document arrives in a fetch it already makes. There\n is nothing to join.\n\nThe single-row `config_snapshot` table is deliberately **left alone**. It\nanswers \"what is the running configuration now\", is replaced on every publish,\nand is pinned to one row by a test. It could not answer \"what did attempt 2 run\nunder\" in any case.\n\n**Payload.** The result of `config.Config.ForTask()` — the non-secret\ntask-runtime subset — plus the schema constant `archie/task-config@1`. Not the\ndashboard's configuration view: that is a larger projection carrying the full\nschema catalog, and it is the wrong object for this. Stored shape and the tab's\nread rule are in `docs/architecture/observability.md`.\n\n**Binding conditions.** It is emitted through the durable path, never the\nlossy bus; it is written once per attempt, not once per stage; redaction is\nproven **positively** by a canary-secrets test rather than assumed from the\nproducer's contract; the timeline gains one human line for it rather than\ndumping a JSON blob; and an attempt with no such event renders \"not captured for\nthis run\", never an empty table implying defaults applied.\n\n**Documented residual — the honest trust claim.** A task-scoped grant may insert\nevents on its own task ID; that is how `tool_call` and `agent_finish` already\nwork, so a task can in principle write its own `config_captured` event. This\n**widens no capability** — the grant could already insert arbitrary events on\nitself — but it does mean this provenance is exactly as trustworthy as the rest\nof that task's event stream, and no more. It is not tamper-proofing and must\nnever be described as such.\n\n## No new RPC and no new table\n\nThe entire wire change is two fields on existing messages:\n\n```proto\n// proto/state/v1/state.proto\nmessage Event { int64 attempt = 11; }\nmessage ReadTaskLogRequest { string stage = 9; }\n```\n\nNo RPC was added, no store table was added, and no new configuration field was\nintroduced. `docs/prds/state-store-contract.md` is unchanged because no ratified\ncontract changed; the JSON document rides an existing `data` column over an\nexisting RPC.\n\nBoth fields are tolerant in both directions, and the degrade is stated rather\nthan hidden: an old server makes `attempt` read as `0` and `stage` match\nnothing, so the page must say the deployment cannot attribute events to attempts\nrather than show one run labelled \"attempt 1\". An old client shows\n`config_captured` as an unknown kind, and the Config tab must then say \"this\ndeployment cannot read per-run configuration\" — a different claim from \"not\ncaptured for this attempt\".\n\n## Retry is a new run\n\nThe retry action is reachable from the run view and produces a new attempt that\nappears as a new run. It is a **fresh** run: it resets the branch onto\n`origin/\u003cbase\u003e` (`internal/worktree.(*Manager).refresh` → `resetOnto`) and\n**discards the previous attempt's commits**. Labelling it \"restart\" would imply\nresumption that does not happen, so the control and its confirm copy say so.\n\n## Honest limits the surface carries\n\nThese are the claims that would otherwise be overstated. Each is a copy or\nrendering rule, not a caveat in a comment:\n\n- **There is no exit code, so no stage shows pass/fail.** `stage_finish` carries\n a duration and an optional error string and nothing else. A stage `ok` means\n \"returned without error\", not \"the work was correct\". The agent's own verdict\n is shown separately from the rail. A stage that parks after finishing cleanly\n leaves that stage `ok` on a failed task, so attempt status is read together\n with task status.\n- **The log stage filter only matches lines the workflow bound a stage to.**\n Agent and tool output carries no stage and never matches. The pane states that\n coverage limit next to the control rather than presenting a filtered view as\n the whole story.\n- **Absence and unreadability are different answers.** A deployment that cannot\n read task logs says so; an attempt whose log does not exist says that. They\n are never collapsed, in either direction, for logs or for configuration.\n- **An empty changed-files view never means \"nothing changed\".** No capture\n exists for runs predating this change and the forge cannot supply one, so the\n empty state reads \"not captured for this run\".\n- **No run-duration figure is shown.** Per-stage durations only: a run can spend\n days in `waiting_human`, and one number would understate wall time.\n- **No duration is invented** for a stage with no `stage_start`; it reads \"not\n recorded\".\n- **Unattributed history is never presented as an attempt.** Events with\n `attempt = 0` are reported as a count, not guessed into a run.\n- **The configuration shown is the attempt's non-secret task-runtime subset** —\n bot identity, models, limits, budgets, dispatch, diff cap, notifications,\n forge host, tool policy. It is not the full dashboard configuration and must\n never imply it contains providers, repositories, identities or credentials.\n\n## Route and compatibility\n\n`/tasks/{id}` is the detail route; it is deep-linkable and reload-safe.\n`/tasks?task=N` and `/tasks?status=...` keep working and keep their existing\nbehaviour, including the nav highlight. The debug view is why the raw view\nexists at all: the task list response is capped, so a client-side join has no\nrow for an older task, and a debug view that silently hid events would be worse\nthan useless.\n\n## What this deliberately does not do\n\n- **No new RPC, table, store interface or configuration field.** Nothing here\n needed one, and each would have widened a ratified surface.\n- **No exit status, no pass/fail gate badge, no \"steps\" in the CI sense.** The\n reference layout is a CI pipeline; archie's stages are agent runs and the page\n must not borrow CI's claims.\n- **No backfill** of attempts, change captures or configurations onto runs that\n predate the change.\n- **No client-side join for configuration.** It arrives in the events response\n the page already fetches; there is nothing to join.\n- **No change to `docs/prds/state-store-contract.md`.** No ratified contract\n moved.\n\n## Files this change touches\n\n- `internal/store/`: the `events.attempt` column, its migrator arm, and the\n schema-presence test.\n- `internal/events/`: `Event.Attempt`, `KindChangesCaptured`,\n `KindConfigCaptured`, `ConfigCapturedSchema`.\n- `internal/domain/workflow/task/changes.go`: the change value types.\n- `internal/worktree/` + `internal/app/agentworker/`: the capture primitive and\n its production forwarding.\n- `internal/domain/workflow/steps.go`: the capture at commit and push.\n- `internal/daemon/`: the per-attempt configuration emit at dispatch.\n- `internal/logging/`: `Query.Stage`.\n- `internal/webui/`: the three read endpoints and the `stage` log parameter.\n- `proto/state/v1/state.proto` + generated stubs: the two field additions.\n- `ui/src/`: the detail page, the tab primitive, the rail, and the panel copy.\n- `docs/`: this document and `docs/architecture/observability.md`.\n\n## Acceptance\n\n1. Two attempts of one task render as two runs, and attempt 1's rail excludes\n attempt 2's stages.\n2. A task with no stages renders \"no stages recorded\", never an unexplained\n empty rail.\n3. A pre-change `archie.db` gains the `attempt` column with `user_version`\n still 3, and its old events are reported as unattributed.\n4. An attempt with a capture shows per-file added/deleted counts, status, repo\n and PR links; an attempt without one says so and never says \"no changes\".\n5. The Config tab shows the attempt's document for the selected attempt only,\n with no secret value present in the serialized document.\n6. Retry from the run view produces a new attempt shown as a new run, with copy\n stating the previous attempt's commits are discarded.\n7. The log stage filter narrows to stage-bound lines and the pane states the\n coverage limit; `disabled` and `found=false` remain distinguishable.\n8. `/tasks/{id}` survives reload, and `/tasks?task=N` / `/tasks?status=...`\n are unchanged.\n9. The debug view returns the task record and its unfiltered events, each\n carrying `attempt`.\n10. Run the gate and the dashboard build: `task check` passes clean, `task ui`\n has been run so `ui/dist` matches the sources, and every touched package\n builds and passes under `-race`.\n" + "body": "# Per-run task detail: attempts, stages, changes, config, debug\n\n**Status:** Approved\n**Authority:** the surface contract itself is\n`docs/architecture/observability.md`.\n**Date:** 2026-09-18\n**Compounds with:** `docs/prds/task-logs-pagination.md` (a bounded read says\nwhat it does not show), `docs/prds/status-health-surface.md` (no surface that\nanswers something already answered elsewhere).\n\n## Decision\n\nA task gets one detail page that answers \"what did this run actually do?\" per\n**attempt**: an ordered stage rail, the change each commit/push produced, the\neffective configuration the attempt ran under, the attempt's log, and the raw\nrecord behind all of it.\n\nThe single organising idea is **attempt attribution**. The dashboard\nrenders a task's whole event stream as one flat timeline, so a retried task\nreads as one long run with two of everything. Every new panel is scoped to one\nattempt and says which one.\n\nThe second organising idea is **provenance, not re-derivation**. What a run\nproduced is captured at the producer, while the evidence exists, and persisted\ndurably. Nothing in this surface tries to reconstruct a past run from state\nthat has since moved on.\n\n## The requirement set\n\n| # | Requirement | Mechanism |\n| --- | --- | --- |\n| R1 | Per-attempt stage rail: stages with name, start, duration, terminal status and error, ordered within one attempt | `GET /api/tasks/{id}/attempts` reads the task's events and groups them by `attempt` |\n| R2 | Events carry an attempt, migrated safely and carried on the wire | `events.attempt` column + presence-gated migrator arm; `Event.attempt = 11` |\n| R3 | Changed files captured at commit/push, persisted per task, exposed, shown with repo/PR links | `changes_captured` durable event written by the commit/push steps; `GET /api/tasks/{id}/changes` |\n| R4 | The effective configuration an attempt ran under, secrets redacted, shown | `config_captured` durable event written once per attempt; read from the events response the page already holds |\n| R5 | Log pane filters by attempt, stage and level; absence stays distinct from an unreadable deployment | `logging.Query.Stage` + `ReadTaskLogRequest.stage = 9`; the pane keeps `disabled` and `found=false` apart |\n| R6 | Retry is reachable from the run view and produces a new attempt shown as a new run | Existing retry action, labelled as a NEW run whose confirm states the previous attempt's commits are discarded |\n| R7 | Raw JSON of the task record and its events for the selected attempt | `GET /api/tasks/{id}/debug` |\n| R8 | Deep-linkable, reload-safe per-task detail page that keeps existing URLs working | `/tasks/{id}` route beside the existing `/tasks?task=N` and `/tasks?status=...` |\n| R9 | Design polish: hierarchy, tokens, both themes, responsive rail, a11y, loading/empty/error states | UI lane; measured at 480px and 700px in both themes |\n| R10 | Documentation of record | This document plus `docs/architecture/observability.md` |\n| R11 | Red-first tests for every new behaviour, full gate, `task ui`, forced `go build ./...`, `go test -race` on touched packages | Per lane; the anti-cost-cutting requirement lives here |\n\n## Attempt attribution: a column, not a guess\n\n`events.Event.Attempt`, persisted as the `events.attempt` column\n(`internal/infrastructure/postgres/migrations/0001_state_store.sql`), is a real column,\n`NOT NULL DEFAULT 0`. It was chosen over segmenting the stream on `task_queued` /\n`task_retried` because a fold would have to guess which side of a boundary an\nevent belongs to when the daemon restarts mid-attempt, when `RecoverStale`\nrequeues, or when two attempts interleave across a crash. A guess rendered as\nstructure is exactly the false fidelity this surface exists to avoid; a column\nis durable, queryable and ordered by construction.\n\n**`0` means \"not attributable\".** It is every row written before this change\nplus the deliberately task-agnostic producers. It is not an attempt number and\nis never rendered as one. The attempts endpoint reports the count of\nunattributed events so the page can say so instead of presenting old history as\nattempt 1.\n\n## Migration against an existing `archie.db`\n\n\u003e Storage superseded: every store now lives in PostgreSQL behind\n\u003e `internal/domain/storecontract`, implemented in `internal/infrastructure/postgres`\n\u003e (`docs/prds/state-store-contract.md`). SQLite names below describe the\n\u003e original design.\n\nThe store's migrator (`migrateTasks` in `internal/store/store.go`) already reads\n`PRAGMA table_info` through `tableColumns` and applies a presence-gated\n`ALTER TABLE ... ADD COLUMN` list. This change adds an `events` arm: a\n`tableColumns(ctx, tx, \"events\")` read beside the other two, and the entry\n`ALTER TABLE events ADD COLUMN attempt INTEGER NOT NULL DEFAULT 0`.\n\nThree consequences a future maintainer must not get wrong:\n\n- `eventsSchema` is `CREATE TABLE IF NOT EXISTS` (`internal/store/events.go`),\n so on an existing database it is a **no-op**. The migrator arm is the only\n thing that adds the column there. This is why the red test builds a raw\n SQLite file with the pre-change `events` DDL and must fail before the arm\n exists: every temp-database test passes either way.\n- **`user_version` stays 3.** The loop is presence-gated, not version-gated;\n bumping it would force an edit to the test that exists to prove a newer\n schema is refused.\n- **No backfill, ever.** Existing rows keep `attempt = 0`. Inferring an attempt\n number for them would be fabrication.\n\n## Changed files: captured at the producer\n\nThe measurement is taken inside the shared step library immediately after a\nsuccessful commit and after a successful push (`StageCommit` and\n`StageCommitPush` in `internal/domain/workflow/steps.go`) — the one place\nwhere the worktree, the branch, the base, the task and the store are all in\nscope, and the only place that knows a push actually happened.\n\nBoth alternatives were rejected on evidence, not taste:\n\n- **Reading the diff on demand** is impossible. The worktree is deleted on\n merge, close and no-PR terminal states (`internal/worktree.(*Manager).Cleanup`,\n called from the daemon's terminal paths) and a retry resets the branch onto\n its base (`internal/worktree.(*Manager).refresh` → `resetOnto`). There is\n nothing left to read.\n- **Reading it from the forge** is impossible. The forge contract exposes\n no changed-file capability, and adding one would cost three implementations\n plus an RPC and would still fail for chat-sourced tasks with no PR.\n\nBinding consequences:\n\n- Capture **degrades, never parks**. A `Trees` implementation without the\n capture seam captures nothing, logs, and lets the run continue. A reporting\n failure must never fail a task.\n- The value types live in `internal/domain/workflow/task` (`changes.go`), not in\n package `workflow` and not in `internal/worktree`. Package `workflow` has no\n import of `internal/worktree`, and this subpackage is not projected into the\n interpreted-stage symbol table, so declaring the types here widens nothing and\n needs no symbol-table regeneration. The capture method stays off that surface\n entirely.\n- `files` is capped at 200 entries because an event's `data` is not\n length-limited by the store. `totals` always covers the **full** set and\n `truncated: true` says when entries were dropped.\n- `binary` means \"no textual hunks\", not \"a binary file\". Rename pairing is\n go-git behaviour the lane pins with a test; where the repository records a\n rename as add + delete, the surface reports what was recorded.\n\n## Per-run configuration: a durable event, not a table\n\n**Decision.** The effective configuration an attempt ran under is persisted as a\n`config_captured` durable event (`internal/events.KindConfigCaptured`), on the\nsame attempt key R2 adds — not in a table, and not behind new RPCs.\n\n**Why.** Three reasons, in order of weight:\n\n1. Events are already this system's durable per-attempt provenance stream, and\n they demonstrably carry aggregation (`StageStats`, `TokensByDay` both\n aggregate over persisted events). R3 puts its diffstat there in this same\n change. A table plus two RPCs would have given one feature **two provenance\n mechanisms for one purpose**.\n2. It adds no RPC, so the ratified state-store surface is untouched and nothing\n in `docs/prds/state-store-contract.md` (rev 2c) changes.\n3. The page already fetches the task's events to build the rail (R1) and to read\n the diffstat (R3), so the document arrives in a fetch it already makes. There\n is nothing to join.\n\nThe single-row `config_snapshot` table is deliberately **left alone**. It\nanswers \"what is the running configuration now\", is replaced on every publish,\nand is pinned to one row by a test. It could not answer \"what did attempt 2 run\nunder\" in any case.\n\n**Payload.** The result of `config.Config.ForTask()` — the non-secret\ntask-runtime subset — plus the schema constant `archie/task-config@1`. Not the\ndashboard's configuration view: that is a larger projection carrying the full\nschema catalog, and it is the wrong object for this. Stored shape and the tab's\nread rule are in `docs/architecture/observability.md`.\n\n**Binding conditions.** It is emitted through the durable path, never the\nlossy bus; it is written once per attempt, not once per stage; redaction is\nproven **positively** by a canary-secrets test rather than assumed from the\nproducer's contract; the timeline gains one human line for it rather than\ndumping a JSON blob; and an attempt with no such event renders \"not captured for\nthis run\", never an empty table implying defaults applied.\n\n**Documented residual — the honest trust claim.** A task-scoped grant may insert\nevents on its own task ID; that is how `tool_call` and `agent_finish` already\nwork, so a task can in principle write its own `config_captured` event. This\n**widens no capability** — the grant could already insert arbitrary events on\nitself — but it does mean this provenance is exactly as trustworthy as the rest\nof that task's event stream, and no more. It is not tamper-proofing and must\nnever be described as such.\n\n## No new RPC and no new table\n\nThe entire wire change is two fields on existing messages:\n\n```proto\n// proto/state/v1/state.proto\nmessage Event { int64 attempt = 11; }\nmessage ReadTaskLogRequest { string stage = 9; }\n```\n\nNo RPC was added, no store table was added, and no new configuration field was\nintroduced. `docs/prds/state-store-contract.md` is unchanged because no ratified\ncontract changed; the JSON document rides an existing `data` column over an\nexisting RPC.\n\nBoth fields are tolerant in both directions, and the degrade is stated rather\nthan hidden: an old server makes `attempt` read as `0` and `stage` match\nnothing, so the page must say the deployment cannot attribute events to attempts\nrather than show one run labelled \"attempt 1\". An old client shows\n`config_captured` as an unknown kind, and the Config tab must then say \"this\ndeployment cannot read per-run configuration\" — a different claim from \"not\ncaptured for this attempt\".\n\n## Retry is a new run\n\nThe retry action is reachable from the run view and produces a new attempt that\nappears as a new run. It is a **fresh** run: it resets the branch onto\n`origin/\u003cbase\u003e` (`internal/worktree.(*Manager).refresh` → `resetOnto`) and\n**discards the previous attempt's commits**. Labelling it \"restart\" would imply\nresumption that does not happen, so the control and its confirm copy say so.\n\n## Honest limits the surface carries\n\nThese are the claims that would otherwise be overstated. Each is a copy or\nrendering rule, not a caveat in a comment:\n\n- **There is no exit code, so no stage shows pass/fail.** `stage_finish` carries\n a duration and an optional error string and nothing else. A stage `ok` means\n \"returned without error\", not \"the work was correct\". The agent's own verdict\n is shown separately from the rail. A stage that parks after finishing cleanly\n leaves that stage `ok` on a failed task, so attempt status is read together\n with task status.\n- **The log stage filter only matches lines the workflow bound a stage to.**\n Agent and tool output carries no stage and never matches. The pane states that\n coverage limit next to the control rather than presenting a filtered view as\n the whole story.\n- **Absence and unreadability are different answers.** A deployment that cannot\n read task logs says so; an attempt whose log does not exist says that. They\n are never collapsed, in either direction, for logs or for configuration.\n- **An empty changed-files view never means \"nothing changed\".** No capture\n exists for runs predating this change and the forge cannot supply one, so the\n empty state reads \"not captured for this run\".\n- **No run-duration figure is shown.** Per-stage durations only: a run can spend\n days in `waiting_human`, and one number would understate wall time.\n- **No duration is invented** for a stage with no `stage_start`; it reads \"not\n recorded\".\n- **Unattributed history is never presented as an attempt.** Events with\n `attempt = 0` are reported as a count, not guessed into a run.\n- **The configuration shown is the attempt's non-secret task-runtime subset** —\n bot identity, models, limits, budgets, dispatch, diff cap, notifications,\n forge host, tool policy. It is not the full dashboard configuration and must\n never imply it contains providers, repositories, identities or credentials.\n\n## Route and compatibility\n\n`/tasks/{id}` is the detail route; it is deep-linkable and reload-safe.\n`/tasks?task=N` and `/tasks?status=...` keep working and keep their existing\nbehaviour, including the nav highlight. The debug view is why the raw view\nexists at all: the task list response is capped, so a client-side join has no\nrow for an older task, and a debug view that silently hid events would be worse\nthan useless.\n\n## What this deliberately does not do\n\n- **No new RPC, table, store interface or configuration field.** Nothing here\n needed one, and each would have widened a ratified surface.\n- **No exit status, no pass/fail gate badge, no \"steps\" in the CI sense.** The\n reference layout is a CI pipeline; archie's stages are agent runs and the page\n must not borrow CI's claims.\n- **No backfill** of attempts, change captures or configurations onto runs that\n predate the change.\n- **No client-side join for configuration.** It arrives in the events response\n the page already fetches; there is nothing to join.\n- **No change to `docs/prds/state-store-contract.md`.** No ratified contract\n moved.\n\n## Files this change touches\n\n- the State Store schema: the `events.attempt` column.\n- `internal/events/`: `Event.Attempt`, `KindChangesCaptured`,\n `KindConfigCaptured`, `ConfigCapturedSchema`.\n- `internal/domain/workflow/task/changes.go`: the change value types.\n- `internal/worktree/` + `internal/app/agentworker/`: the capture primitive and\n its production forwarding.\n- `internal/domain/workflow/steps.go`: the capture at commit and push.\n- `internal/daemon/`: the per-attempt configuration emit at dispatch.\n- `internal/logging/`: `Query.Stage`.\n- `internal/webui/`: the three read endpoints and the `stage` log parameter.\n- `proto/state/v1/state.proto` + generated stubs: the two field additions.\n- `ui/src/`: the detail page, the tab primitive, the rail, and the panel copy.\n- `docs/`: this document and `docs/architecture/observability.md`.\n\n## Acceptance\n\n1. Two attempts of one task render as two runs, and attempt 1's rail excludes\n attempt 2's stages.\n2. A task with no stages renders \"no stages recorded\", never an unexplained\n empty rail.\n3. A pre-change `archie.db` gains the `attempt` column with `user_version`\n still 3, and its old events are reported as unattributed.\n4. An attempt with a capture shows per-file added/deleted counts, status, repo\n and PR links; an attempt without one says so and never says \"no changes\".\n5. The Config tab shows the attempt's document for the selected attempt only,\n with no secret value present in the serialized document.\n6. Retry from the run view produces a new attempt shown as a new run, with copy\n stating the previous attempt's commits are discarded.\n7. The log stage filter narrows to stage-bound lines and the pane states the\n coverage limit; `disabled` and `found=false` remain distinguishable.\n8. `/tasks/{id}` survives reload, and `/tasks?task=N` / `/tasks?status=...`\n are unchanged.\n9. The debug view returns the task record and its unfiltered events, each\n carrying `attempt`.\n10. Run the gate and the dashboard build: `task check` passes clean, `task ui`\n has been run so `ui/dist` matches the sources, and every touched package\n builds and passes under `-race`.\n" }, { "url": "/dev/prds/task-run-master-detail/", @@ -442,7 +442,7 @@ "url": "/dev/prds/ui-service-boundary/", "title": "UI Service boundary and configuration ownership", "section": "PRDs", - "body": "# UI Service boundary and configuration ownership\n\n**Status:** Approved\n**Revision:** 1\n**Date:** 2026-09-07\n**Beads issue:** `archie-core-8cda.5.1`\n**Parent:** [`service-decomposition.md`](/dev/prds/service-decomposition/)\n**Prerequisites:** Gateway contract extraction and State Store contract ratification\n\nThis document is the authority for Phase 3 UI Service implementation. It fixes\nthe process boundary, configuration ownership, client contracts, security\nposture, readiness behavior, and migration evidence required by\n`archie-core-8cda.5.2` and later UI extraction work.\n\n## Decision\n\nThe UI Service owns the operator-facing HTTP surface and the compiled SPA. It\nserves `ui/dist` and exposes UI-specific JSON DTOs for dashboard views and\nactions. It does not own conversations, task state, workflow execution,\nchannel lifecycle, forge access, or the daemon's runtime configuration.\n\nThe UI Service consumes two typed remote capabilities:\n\n- Gateway's `gateway.ChatContract` through the versioned Gateway gRPC\n contract, for chat snapshots, messages, turns, streaming, persona changes,\n cancellation, and Gateway task actions.\n- State Store's narrow read/write contracts through the versioned State Store\n gRPC contract, for task views, events, captures, mappings, bindings, and\n configuration data explicitly projected for the dashboard.\n\nThe UI process may compose local adapters during the migration seam, but the\nadapter implements the same wire-safe contract as the remote client. The\nproduction extraction cutover removes the in-process adapter in the same\nchange; a dual-live UI authority is not permitted.\n\n## Boundary and ownership\n\n| Concern | UI Service owns | UI Service may consume | UI Service must not own or receive |\n|---|---|---|---|\n| HTTP and SPA | routes, auth middleware, UI DTOs, assets, HTTP lifecycle | contract results and health snapshots | domain entities or infrastructure handles |\n| Chat | request validation and response projection | Gateway `ChatContract` | `gateway.Router`, `SessionStore`, turn runner, model/provider runtime |\n| Task dashboard | query/action DTOs and HTTP status mapping | State Store narrow contracts and explicitly ratified task-action contract | `*store.Store`, SQL connections, workflow registry, daemon pointer |\n| Configuration page | secret-free view and update request validation | a UI configuration contract owned by the configuration/application boundary | `config.Config`, `config.Holder`, secret registry, reload implementation |\n| Readiness | UI process liveness and dependency readiness aggregation | bounded health results from Gateway and State Store | direct probing of private databases or providers |\n\nEvery value crossing the process boundary is a versioned DTO or scalar contract\nvalue. `internal/webui` may retain temporary Go-side adapters while the\nmigration is in progress, but handlers must map Gateway/State Store values to\nUI-owned response types at the HTTP edge. A constructor accepting a concrete\nGateway, State Store, daemon, or `config.Holder` is not an approved final\nboundary.\n\nThe existing UI routes remain behaviorally compatible during extraction,\nincluding task actions, SSE event delivery, config views, chat streaming,\ncapture intake, mappings, bindings, update reporting, and health endpoints.\nCompatibility preserves status codes, authentication requirements, redaction,\npagination/limits, event ordering, and failure degradation unless a separately\nratified contract changes them.\n\nThe route families have these target owners. This is the required inventory for\nthe implementation children; a route without an owner is not eligible for\ncutover.\n\n| Route family | Target owner consumed by UI | Migration status |\n|---|---|---|\n| `/api/chat/*` | Gateway `ChatContract` and its versioned client | contract seam exists; DTO and stream parity remain |\n| `/api/tasks/*`, `/api/workflows`, `/api/setup` | State Store read/action contracts plus an explicit execution/admin action contract | State Store adapter exists; UI-facing narrow surface remains |\n| `/events` | State Store `EventsSince`, polled by one UI-process pump feeding `Server.Broadcast` | RESOLVED (`archie-core-za9f`): see migration-decisions, \"Dashboard live event delivery\". No new RPC; replay cursor and drop-recovery are the existing `since` watermark in `sse.go` |\n| `/api/logs`, `/api/logs/stream` | daemon diagnostic feed, host-local | unresolved: `logging.Feed` is in-process on the daemon host and has no contract. The per-task log routes moved off this row: see `/api/tasks/*` above |\n| `/api/tasks/{id}/logs`, `/api/tasks/{id}/logs/download` | State Store task-log read contract (`TaskLogStore`: `ReadTaskLog` + `StreamTaskLogContent`), read by the UI process from the service that owns the state directory | RESOLVED (`archie-core-iaqx`): the UI process composes `TaskLogStore` from its own State Store client and serves the attempt log as JSON (`found` distinguishes \"no log for this attempt\" from \"this process cannot read logs\") and as a `Content-Disposition` attachment named `task-\u003cid\u003e-attempt-\u003cn\u003e.log`. The daemon's in-process registry is wired as the server-side reader in `internal/app/archied/state_store.go` only (`staterpc.Deps.TaskLogs`), so no process opens another process's log files |\n| `/api/captures`, `/api/mappings`, `/api/bindings` | State Store contracts, with webhook verification owned by Work Intake/Messaging | RESOLVED (cutover, `archie-core-8cda.5.4`): the UI process composes CaptureStore/MappingStore/BindingStore and mounts the `captureintake.Receiver` from its own State Store client; the daemon's binding-dispatch loop keeps consuming captures from the same store |\n| `/api/config` (read) | daemon-published `ConfigView` snapshot, read over the State Store contract | RESOLVED (`archie-core-ymut`): see migration-decisions, \"Dashboard configuration page\" |\n| `/api/config` (write), `/api/config/reset`, `/api/config/repos/*` | a configuration write contract; the daemon stays the policy owner | DESCOPED for Phase 3 (`archie-core-ymut`), NOT abandoned: 503 in the UI process, SPA hides the controls, editing is config.toml plus reload. Dashboard editing is a tracked feature (`archie-core-1786637498420-327`) and its transport is `archie-core-j28m`; the handlers and their func-field seams on `webui.Server` are kept for it |\n| `/api/channels`, reload, curators, memory, skills, version/update | owning capability contract or an explicitly removed route | owner and failure semantics remain to define |\n| `/`, `/healthz`, `/health`, `/health/detailed` | UI HTTP process; readiness consumes per-service health contracts | current HTTP behavior is the compatibility baseline |\n\n## Configuration ownership\n\nThe UI Service owns only process-local settings:\n\n- HTTP listen address and read/header/shutdown timeouts;\n- UI asset source or build mode;\n- UI authentication mode and token reference;\n- Gateway endpoint and credential reference;\n- State Store endpoint and credential reference;\n- readiness endpoint and dependency timeout policy.\n\nThese settings are decoded into a UI-owned input DTO by the UI composition root.\nThe DTO contains endpoint references and secret references, never resolved\nsecret values or complete daemon configuration. The UI process resolves its\nown credentials and builds its own clients. It does not read or mutate the\ndaemon's configuration file, runtime overlay, forge credentials, model catalog,\nworkflow routing, NATS settings, filesystem jail, or agent/container settings.\n\nThe dashboard configuration API is a separate capability. Its secret-free,\nallowlisted field projections remain governed by\n[`config-schema.md`](/dev/prds/config-schema/). During the transition the daemon may\ncontinue to perform validate-persist-publish for those updates through a\nnarrow application contract. The UI process must never implement that policy\nby holding or mutating a shared `config.Holder`.\n\nThe daemon's `config.Holder` sharing was migration residue: `buildDaemon`\nassigned `b.web.Cfg = b.d.Cfg`. Removed by `archie-core-ml30` -- the field no\nlonger exists, and boot owns the single Holder the daemon reads through.\n\nFor Phase 3, configuration ownership is split deliberately. The UI process\nowns its listener, browser-auth, asset, dependency-target, and readiness\nsettings. The daemon/application configuration owner remains authoritative for\nthe dashboard's `/api/config` read, validation, persistence, overlay, reload,\nand audit behavior. The UI does not become the writer merely because it\nrenders the configuration page.\n\nAmended 2026-09-11 by `archie-core-ml30`: the daemon's `config.Holder` sharing\nis gone. `webui.Server` has no holder field, so the sharing this paragraph\ndescribes cannot be reintroduced by wiring; the daemon owns one Holder and\npublishes a rendered projection.\n\nThere is no single narrow admin contract carrying both the read and the write\n(`archie-core-ymut`). The\nread crosses as a daemon-published `ConfigView` snapshot over the State Store\ncontract, and the write is descoped for Phase 3 rather than contracted, on the\nreasoning recorded in migration-decisions under \"Dashboard configuration\npage\". A future dedicated Configuration Service is the natural owner of a\nwrite contract, and designing one is that feature's work, not this migration's.\n\n## Listen, authentication, and readiness\n\nThe extracted UI follows the current web UI security posture:\n\n- loopback listeners may omit a dashboard token;\n- non-loopback listeners require a non-empty token and fail closed when it is\n absent;\n- `/healthz` and `/health` are unauthenticated liveness probes;\n- `/health/detailed` is authenticated and reports dependency readiness;\n- capture webhooks retain their separately ratified source authentication and\n rate-limit rules; they are not admitted by the dashboard token;\n- Gateway and State Store credentials are service-to-service credentials, never\n browser-visible dashboard tokens.\n\nThe UI is ready only when its HTTP server is serving and each mandatory remote\ncontract has passed its bounded readiness check. Each dependency must expose a\nversioned health result or health endpoint that the UI can consume; the UI must\nnot inspect another service's concrete manager, registry, or database. Optional\ndashboard sections may remain degraded with the current explicit\n`503`/unavailable behavior. A successful liveness response must never imply\ndependency readiness.\n\n## Migration sequence and evidence\n\nThe implementation must complete these gates in order:\n\n1. Define the UI input DTO, Gateway client surface, State Store client surface,\n and UI response DTOs. Add compile-time contract assertions and mapping\n tests. No handler may depend on a concrete remote implementation.\n2. Make the current in-process web UI consume the same narrow adapters and\n preserve the current route, auth, redaction, stream, SSE, and degradation\n behavior. Add a composition test proving the selected adapter is the one\n used by production wiring.\n3. Remove direct `internal/gateway` session/router/runtime access from web UI\n handlers. Remove direct `internal/store` implementation access, SQL access,\n daemon pointers, and broad runtime callback bundles from the UI constructor.\n4. Replace the shared `config.Holder` with the UI-owned configuration DTO and\n narrow configuration application contract. Prove reload/update atomicity\n remains owned by the configuration/application boundary.\n5. Add the standalone UI process and deployment wiring. It must obtain Gateway\n and State Store endpoints from its own configuration, authenticate each\n client independently, start liveness/readiness endpoints, and shut down\n clients and HTTP listeners in bounded order.\n6. Run dual-adapter contract tests and an HTTP parity suite covering every\n retained route. Test remote failures, missing credentials, malformed DTOs,\n stream cancellation, stale State Store data, and readiness degradation.\n7. Cut over the supported composition to the UI process. The old in-process\n `webui.Run` path is deleted in the same change; no fallback or second HTTP\n authority remains.\n\nThe deletion gate is objective: `go list -deps` from the UI process shows no\ndaemon, SQL/store implementation, workflow, forge, channel, model, or secret\nruntime package; production composition contains one UI HTTP listener; and\narchitecture tests reject `config.Holder`, concrete `*store.Store`, concrete\nGateway runtime types, and direct SQLite imports in the UI service.\n\nRollback is a deployment composition change that points the operator at the\nlast validated UI binary and restores the previous endpoint configuration. It\nmust not restore a second writer or reintroduce shared-holder mutation. UI\nstate is either remote contract state or existing durable state owned by its\ncurrent service; no UI-specific database migration is introduced by this\nboundary decision.\n\n## Current evidence and known delta\n\nCurrent code proves the following migration inputs:\n\n- `internal/app/archied/bootstrap.go:setupObservability` constructs\n `webui.Server` and wires store, channels, events, logs, and config-related\n callbacks.\n- `internal/app/archied/bootstrap.go:buildDaemon` replaces the initial holder\n with `b.d.Cfg`, explicitly sharing the daemon's live configuration.\n- `internal/webui/api_chat.go` calls the Gateway contract but still imports\n Gateway-owned request/value types; handlers must finish the DTO projection.\n- `internal/webui/server.go:Handler` owns the route set, SPA fallback, health\n split, capture bypass, and token wrapper.\n- `internal/app/archied/state_store.go:RunStateStore` owns the task SQLite\n file and serves the State Store gRPC contract; the UI must use that contract\n rather than opening the file.\n- `internal/app/archied/gateway.go:RunGateway` serves Gateway gRPC and owns\n Gateway session persistence; the UI must use `ChatContract` rather than\n Gateway internals.\n\nThe current code therefore satisfies the upstream prerequisites but does not\nyet satisfy this boundary. This document is the ratified target; the\nimplementation child must prove each migration gate before claiming Phase 3.\n\n### Phase 3 gates\n\nEvery gate below is proven by test, not by inspection:\n\n| Gate | Evidence |\n| --- | --- |\n| Deletion | `cmd/archie-ui/architecture_test.go` re-runs `go list -deps` on every `task check` and fails on any banned package, checked in both directions against its two exceptions |\n| Composition | `internal/app/archieui/TestComposeUIServerHoldsNoDaemonState` fails on a concrete `*store.Store` or a Gateway that is not the gRPC client. The holder clause it also carried is enforced by the type: `webui.Server` has no `config.Holder` field (`archie-core-ml30`) |\n| End-to-end | `cmd/archie-ui/main_test.go` builds the binary and drives it over HTTP against a live Gateway and State Store |\n| No dual authority | The dashboard HTTP listener is `internal/app/archieui/run.go` only; the daemon's two `http.Server`s are its own health endpoint and the forge webhook receiver |\n\nAcceptance criterion 1 was amended at the close of Phase 3 to name the UI\nprocess rather than the `internal/webui` package, because the daemon still held\na holder inside its own process to render the published `ConfigView` snapshot.\n`archie-core-ml30` removed that residue, so the criterion holds in its original\nform again: `internal/webui` receives no holder in any process.\n\n### Deletion gate: SATISFIED (archie-core-8cda.5.6, rev. 2)\n\nThe objective gate now holds. `go list -deps ./cmd/archie-ui` links **zero**\nbanned packages: no `modernc.org/sqlite`, no `internal/store`, `internal/gateway`,\n`internal/channels`, `internal/memory`, `internal/secret`, `internal/agentexec`,\n`internal/tools`, `internal/skill`, `internal/yaegiutil`, `internal/domain/curator`,\nor the workflow engine (`internal/domain/workflow`). How the runtime handles were\nsevered, each at the producer:\n\n- **Store contracts** moved to `internal/domain/storecontract` (producer-owned\n read contracts, state-store-contract rev. 2d); `internal/store` keeps aliases.\n- **Chat contracts** moved to `internal/domain/messaging`; `internal/gateway`\n keeps aliases. The UI process holds `ChatContract`, `SessionStore`, turn and\n stream types without linking the gateway runtime or its SQLite session store.\n- **Webui-owned views** replaced concrete runtime handles on `webui.Server`:\n channel lifecycle via `internal/channels/status`, memory via a `MemoryStatus`\n interface, curators via a `CuratorStatus` interface, and the skill catalogue\n via a `SkillCatalog` interface — each supplied by a daemon-side adapter at\n bootstrap, each degrading to an empty page when unwired.\n- **`config.SecretRef`** moved into `internal/config` (it is config vocabulary);\n `internal/secret` keeps an alias and resolves through `Registry.Resolve`.\n- **Task vocabulary** (`Task`, `Store`, `Definition`, status constants) split\n into `internal/domain/workflow/task`; the pipeline engine stays in package\n `workflow`, which keeps aliases. The UI process links the vocabulary only.\n\nThe gate is enforced, not just measured. `cmd/archie-ui/architecture_test.go`\nre-runs `go list -deps` on every `task check` and fails on any package in a\nbanned category, so a single reference to a type sitting beside a runtime\ncannot quietly relink it. Its exception list (`internal/channels/status`,\n`internal/domain/workflow/task`) is checked in both directions: an exception\nthe UI stops needing is reported, so the allowlist cannot decay into an open\nprefix. `internal/app/archieui`'s `TestComposeUIServerHoldsNoDaemonState`\ncovers the composition clauses, failing on a non-nil `config.Holder`, a\nconcrete `*store.Store`, or a Gateway that is not the gRPC client.\n\nOne piece of residue remained visible at the close of Phase 3: `webui.Server`\nstill declared `Cfg *config.Holder`, because the daemon kept a `webui.Server`\nas the renderer that built the published `ConfigView` snapshot. That cleanup\nlanded as `archie-core-ml30`. `webui.Server` no longer has a configuration\nholder, provenance store, reload-status callback or override-listing callback:\nthe projection is built by `webui.BuildConfigView`, a pure function the\nconfiguration owner calls, and the daemon assembles its input from its own\nstate (`boot.configViewInput`). Readiness reads the daemon's own Holder,\nchannel manager and chat contract, and the health registry it produces is a\ndaemon field rather than a field on a server the daemon never serves.\n\n`ConfigView.Editable` moved with it. It describes the rendering process's own\nwrite path, so `handleConfig` sets it from `Server.UpdateConfig` rather than\nany process baking it into the document. A future configuration write contract\n(`archie-core-j28m`) makes the page editable by wiring that one field.\n\n### Real-process evidence (archie-core-8cda.5.6)\n\n`cmd/archie-ui/main_test.go` builds the binary and runs it against a State\nStore and a Gateway on loopback gRPC, then drives it over HTTP: liveness\nwithout a token, 401 for an API request without one, the `?t=` exchange into\nan HttpOnly cookie, task rows with their forge links, the embedded SPA and its\nassets, a chat stream relayed frame by frame from the Gateway, an operator\ntask action crossing to the Gateway (and refused without the CSRF header),\ncapture intake accepted unauthenticated, the ratified degradations, readiness\ndropping when the State Store dies while liveness holds, and a clean SIGTERM.\n\nTwo dashboard values degraded here because the published projection did not\ncarry them: `chat.show_tool_calls` (the chat page never expanded tool calls)\nand the `/api/setup` checklist (the panel was omitted rather than rendered).\nBoth are closed by `archie-core-ml30`: `ConfigView` gained a `chat` section\ncarrying `show_tool_calls`, the operator name, and whether any chat channel is\nconfigured, and both surfaces now read the projection rather than a live\nholder. The end-to-end suite drives the real binary and streams a turn, so the\n`show_tool_calls` assertion is made on the frames a browser receives.\n\nThe checklist stays. The question raised when it was republished -- is a\nread-only setup report wanted in a process that cannot apply the setup it\nreports? -- resolves in favour of keeping it, because the panel never promised\nto apply anything. It renders each step as a title, a detail and a done mark\nwith no control behind it, so it answers \"what is this deployment missing?\"\nrather than \"fix it here\", and the page that CAN act on it (the Configuration\npage) is where an operator goes next, in the daemon's own words. Removing it\nwould delete the one surface that distinguishes a fresh deployment from a\nfinished one for no gain, since the values it reads already cross the boundary\nin `ConfigView` and the read path costs the daemon nothing. The write path\nremains descoped (see `/api/config` (write) in the route table): the checklist\nreports configuration, it does not change it.\n\n`archie-core-ml30` published the channel flag from two of the three chat\nfront-ends, so an email-only deployment read as configured on `/api/channels`\nand unconfigured on the checklist at the same time (GitHub #821). Both now\nderive from `config.ChatConfig.FrontEnds`, the single definition of a configured\nfront-end, and a test pins the two surfaces to the same answer.\n\nA third value degraded for the same reason and is closed by\n`archie-core-pv6t`: in a deployment configuring `[[identities]]` the projection\ncarried the default identity's forge alone, so task rows withheld their links\nrather than pointing every row at the wrong forge. `ConfigView` now also\npublishes each identity's forge coordinates and the repositories it owns\n(`identities`), which is what resolving a task to its owning forge needs —\ncoordinates and repository names only, never a token. The end-to-end suite\ndrives the real binary against a multi-identity projection and asserts each\nrow's issue link is built from the forge that owns it.\n\n## Non-goals and open work\n\nThis ratification does not define a new domain model, move the dashboard's\nallowlisted configuration schema, redesign Gateway or State Store RPCs, choose\nKubernetes versus local discovery, or authorize unrelated package migration.\nThose decisions remain in their existing authority documents. Any new UI\ncapability that requires a wider Gateway or State Store surface must amend that\nservice's contract before implementation.\n" + "body": "# UI Service boundary and configuration ownership\n\n**Status:** Approved\n**Revision:** 1\n**Date:** 2026-09-07\n**Beads issue:** `archie-core-8cda.5.1`\n**Parent:** [`service-decomposition.md`](/dev/prds/service-decomposition/)\n**Prerequisites:** Gateway contract extraction and State Store contract ratification\n\nThis document is the authority for Phase 3 UI Service implementation. It fixes\nthe process boundary, configuration ownership, client contracts, security\nposture, readiness behavior, and migration evidence required by\n`archie-core-8cda.5.2` and later UI extraction work.\n\n## Decision\n\nThe UI Service owns the operator-facing HTTP surface and the compiled SPA. It\nserves `ui/dist` and exposes UI-specific JSON DTOs for dashboard views and\nactions. It does not own conversations, task state, workflow execution,\nchannel lifecycle, forge access, or the daemon's runtime configuration.\n\nThe UI Service consumes two typed remote capabilities:\n\n- Gateway's `gateway.ChatContract` through the versioned Gateway gRPC\n contract, for chat snapshots, messages, turns, streaming, persona changes,\n cancellation, and Gateway task actions.\n- State Store's narrow read/write contracts through the versioned State Store\n gRPC contract, for task views, events, captures, mappings, bindings, and\n configuration data explicitly projected for the dashboard.\n\nThe UI process may compose local adapters during the migration seam, but the\nadapter implements the same wire-safe contract as the remote client. The\nproduction extraction cutover removes the in-process adapter in the same\nchange; a dual-live UI authority is not permitted.\n\n## Boundary and ownership\n\n| Concern | UI Service owns | UI Service may consume | UI Service must not own or receive |\n|---|---|---|---|\n| HTTP and SPA | routes, auth middleware, UI DTOs, assets, HTTP lifecycle | contract results and health snapshots | domain entities or infrastructure handles |\n| Chat | request validation and response projection | Gateway `ChatContract` | `gateway.Router`, `SessionStore`, turn runner, model/provider runtime |\n| Task dashboard | query/action DTOs and HTTP status mapping | State Store narrow contracts and explicitly ratified task-action contract | `*store.Store`, SQL connections, workflow registry, daemon pointer |\n| Configuration page | secret-free view and update request validation | a UI configuration contract owned by the configuration/application boundary | `config.Config`, `config.Holder`, secret registry, reload implementation |\n| Readiness | UI process liveness and dependency readiness aggregation | bounded health results from Gateway and State Store | direct probing of private databases or providers |\n\nEvery value crossing the process boundary is a versioned DTO or scalar contract\nvalue. `internal/webui` may retain temporary Go-side adapters while the\nmigration is in progress, but handlers must map Gateway/State Store values to\nUI-owned response types at the HTTP edge. A constructor accepting a concrete\nGateway, State Store, daemon, or `config.Holder` is not an approved final\nboundary.\n\nThe existing UI routes remain behaviorally compatible during extraction,\nincluding task actions, SSE event delivery, config views, chat streaming,\ncapture intake, mappings, bindings, update reporting, and health endpoints.\nCompatibility preserves status codes, authentication requirements, redaction,\npagination/limits, event ordering, and failure degradation unless a separately\nratified contract changes them.\n\nThe route families have these target owners. This is the required inventory for\nthe implementation children; a route without an owner is not eligible for\ncutover.\n\n| Route family | Target owner consumed by UI | Migration status |\n|---|---|---|\n| `/api/chat/*` | Gateway `ChatContract` and its versioned client | contract seam exists; DTO and stream parity remain |\n| `/api/tasks/*`, `/api/workflows`, `/api/setup` | State Store read/action contracts plus an explicit execution/admin action contract | State Store adapter exists; UI-facing narrow surface remains |\n| `/events` | State Store `EventsSince`, polled by one UI-process pump feeding `Server.Broadcast` | RESOLVED (`archie-core-za9f`): see migration-decisions, \"Dashboard live event delivery\". No new RPC; replay cursor and drop-recovery are the existing `since` watermark in `sse.go` |\n| `/api/logs`, `/api/logs/stream` | daemon diagnostic feed, host-local | unresolved: `logging.Feed` is in-process on the daemon host and has no contract. The per-task log routes moved off this row: see `/api/tasks/*` above |\n| `/api/tasks/{id}/logs`, `/api/tasks/{id}/logs/download` | State Store task-log read contract (`TaskLogStore`: `ReadTaskLog` + `StreamTaskLogContent`), read by the UI process from the service that owns the state directory | RESOLVED (`archie-core-iaqx`): the UI process composes `TaskLogStore` from its own State Store client and serves the attempt log as JSON (`found` distinguishes \"no log for this attempt\" from \"this process cannot read logs\") and as a `Content-Disposition` attachment named `task-\u003cid\u003e-attempt-\u003cn\u003e.log`. The daemon's in-process registry is wired as the server-side reader in `internal/app/archied/state_store.go` only (`staterpc.Deps.TaskLogs`), so no process opens another process's log files |\n| `/api/captures`, `/api/mappings`, `/api/bindings` | State Store contracts, with webhook verification owned by Work Intake/Messaging | RESOLVED (cutover, `archie-core-8cda.5.4`): the UI process composes CaptureStore/MappingStore/BindingStore and mounts the `captureintake.Receiver` from its own State Store client; the daemon's binding-dispatch loop keeps consuming captures from the same store |\n| `/api/config` (read) | daemon-published `ConfigView` snapshot, read over the State Store contract | RESOLVED (`archie-core-ymut`): see migration-decisions, \"Dashboard configuration page\" |\n| `/api/config` (write), `/api/config/reset`, `/api/config/repos/*` | a configuration write contract; the daemon stays the policy owner | DESCOPED for Phase 3 (`archie-core-ymut`), NOT abandoned: 503 in the UI process, SPA hides the controls, editing is config.toml plus reload. Dashboard editing is a tracked feature (`archie-core-1786637498420-327`) and its transport is `archie-core-j28m`; the handlers and their func-field seams on `webui.Server` are kept for it |\n| `/api/channels`, reload, curators, memory, skills, version/update | owning capability contract or an explicitly removed route | owner and failure semantics remain to define |\n| `/`, `/healthz`, `/health`, `/health/detailed` | UI HTTP process; readiness consumes per-service health contracts | current HTTP behavior is the compatibility baseline |\n\n## Configuration ownership\n\nThe UI Service owns only process-local settings:\n\n- HTTP listen address and read/header/shutdown timeouts;\n- UI asset source or build mode;\n- UI authentication mode and token reference;\n- Gateway endpoint and credential reference;\n- State Store endpoint and credential reference;\n- readiness endpoint and dependency timeout policy.\n\nThese settings are decoded into a UI-owned input DTO by the UI composition root.\nThe DTO contains endpoint references and secret references, never resolved\nsecret values or complete daemon configuration. The UI process resolves its\nown credentials and builds its own clients. It does not read or mutate the\ndaemon's configuration file, runtime overlay, forge credentials, model catalog,\nworkflow routing, NATS settings, filesystem jail, or agent/container settings.\n\nThe dashboard configuration API is a separate capability. Its secret-free,\nallowlisted field projections remain governed by\n[`config-schema.md`](/dev/prds/config-schema/). During the transition the daemon may\ncontinue to perform validate-persist-publish for those updates through a\nnarrow application contract. The UI process must never implement that policy\nby holding or mutating a shared `config.Holder`.\n\nThe daemon's `config.Holder` sharing was migration residue: `buildDaemon`\nassigned `b.web.Cfg = b.d.Cfg`. Removed by `archie-core-ml30` -- the field no\nlonger exists, and boot owns the single Holder the daemon reads through.\n\nFor Phase 3, configuration ownership is split deliberately. The UI process\nowns its listener, browser-auth, asset, dependency-target, and readiness\nsettings. The daemon/application configuration owner remains authoritative for\nthe dashboard's `/api/config` read, validation, persistence, overlay, reload,\nand audit behavior. The UI does not become the writer merely because it\nrenders the configuration page.\n\nAmended 2026-09-11 by `archie-core-ml30`: the daemon's `config.Holder` sharing\nis gone. `webui.Server` has no holder field, so the sharing this paragraph\ndescribes cannot be reintroduced by wiring; the daemon owns one Holder and\npublishes a rendered projection.\n\nThere is no single narrow admin contract carrying both the read and the write\n(`archie-core-ymut`). The\nread crosses as a daemon-published `ConfigView` snapshot over the State Store\ncontract, and the write is descoped for Phase 3 rather than contracted, on the\nreasoning recorded in migration-decisions under \"Dashboard configuration\npage\". A future dedicated Configuration Service is the natural owner of a\nwrite contract, and designing one is that feature's work, not this migration's.\n\n## Listen, authentication, and readiness\n\nThe extracted UI follows the current web UI security posture:\n\n- loopback listeners may omit a dashboard token;\n- non-loopback listeners require a non-empty token and fail closed when it is\n absent;\n- `/healthz` and `/health` are unauthenticated liveness probes;\n- `/health/detailed` is authenticated and reports dependency readiness;\n- capture webhooks retain their separately ratified source authentication and\n rate-limit rules; they are not admitted by the dashboard token;\n- Gateway and State Store credentials are service-to-service credentials, never\n browser-visible dashboard tokens.\n\nThe UI is ready only when its HTTP server is serving and each mandatory remote\ncontract has passed its bounded readiness check. Each dependency must expose a\nversioned health result or health endpoint that the UI can consume; the UI must\nnot inspect another service's concrete manager, registry, or database. Optional\ndashboard sections may remain degraded with the current explicit\n`503`/unavailable behavior. A successful liveness response must never imply\ndependency readiness.\n\n## Migration sequence and evidence\n\nThe implementation must complete these gates in order:\n\n1. Define the UI input DTO, Gateway client surface, State Store client surface,\n and UI response DTOs. Add compile-time contract assertions and mapping\n tests. No handler may depend on a concrete remote implementation.\n2. Make the current in-process web UI consume the same narrow adapters and\n preserve the current route, auth, redaction, stream, SSE, and degradation\n behavior. Add a composition test proving the selected adapter is the one\n used by production wiring.\n3. Remove direct `internal/gateway` session/router/runtime access from web UI\n handlers. Remove direct store implementation access, SQL access,\n daemon pointers, and broad runtime callback bundles from the UI constructor.\n4. Replace the shared `config.Holder` with the UI-owned configuration DTO and\n narrow configuration application contract. Prove reload/update atomicity\n remains owned by the configuration/application boundary.\n5. Add the standalone UI process and deployment wiring. It must obtain Gateway\n and State Store endpoints from its own configuration, authenticate each\n client independently, start liveness/readiness endpoints, and shut down\n clients and HTTP listeners in bounded order.\n6. Run dual-adapter contract tests and an HTTP parity suite covering every\n retained route. Test remote failures, missing credentials, malformed DTOs,\n stream cancellation, stale State Store data, and readiness degradation.\n7. Cut over the supported composition to the UI process. The old in-process\n `webui.Run` path is deleted in the same change; no fallback or second HTTP\n authority remains.\n\nThe deletion gate is objective: `go list -deps` from the UI process shows no\ndaemon, SQL/store implementation, workflow, forge, channel, model, or secret\nruntime package; production composition contains one UI HTTP listener; and\narchitecture tests reject `config.Holder`, concrete `*store.Store`, concrete\nGateway runtime types, and direct database drivers in the UI service.\n\nRollback is a deployment composition change that points the operator at the\nlast validated UI binary and restores the previous endpoint configuration. It\nmust not restore a second writer or reintroduce shared-holder mutation. UI\nstate is either remote contract state or existing durable state owned by its\ncurrent service; no UI-specific database migration is introduced by this\nboundary decision.\n\n## Current evidence and known delta\n\nCurrent code proves the following migration inputs:\n\n- `internal/app/archied/bootstrap.go:setupObservability` constructs\n `webui.Server` and wires store, channels, events, logs, and config-related\n callbacks.\n- `internal/app/archied/bootstrap.go:buildDaemon` replaces the initial holder\n with `b.d.Cfg`, explicitly sharing the daemon's live configuration.\n- `internal/webui/api_chat.go` calls the Gateway contract but still imports\n Gateway-owned request/value types; handlers must finish the DTO projection.\n- `internal/webui/server.go:Handler` owns the route set, SPA fallback, health\n split, capture bypass, and token wrapper.\n- `internal/app/archied/state_store.go:RunStateStore` owns the task\n database and serves the State Store gRPC contract; the UI must use that contract\n rather than opening the database.\n- `internal/app/archied/gateway.go:RunGateway` serves Gateway gRPC and owns\n Gateway session persistence; the UI must use `ChatContract` rather than\n Gateway internals.\n\nThe current code therefore satisfies the upstream prerequisites but does not\nyet satisfy this boundary. This document is the ratified target; the\nimplementation child must prove each migration gate before claiming Phase 3.\n\n### Phase 3 gates\n\nEvery gate below is proven by test, not by inspection:\n\n| Gate | Evidence |\n| --- | --- |\n| Deletion | `cmd/archie-ui/architecture_test.go` re-runs `go list -deps` on every `task check` and fails on any banned package, checked in both directions against its two exceptions |\n| Composition | `internal/app/archieui/TestComposeUIServerHoldsNoDaemonState` fails on a concrete `*store.Store` or a Gateway that is not the gRPC client. The holder clause it also carried is enforced by the type: `webui.Server` has no `config.Holder` field (`archie-core-ml30`) |\n| End-to-end | `cmd/archie-ui/main_test.go` builds the binary and drives it over HTTP against a live Gateway and State Store |\n| No dual authority | The dashboard HTTP listener is `internal/app/archieui/run.go` only; the daemon's two `http.Server`s are its own health endpoint and the forge webhook receiver |\n\nAcceptance criterion 1 was amended at the close of Phase 3 to name the UI\nprocess rather than the `internal/webui` package, because the daemon still held\na holder inside its own process to render the published `ConfigView` snapshot.\n`archie-core-ml30` removed that residue, so the criterion holds in its original\nform again: `internal/webui` receives no holder in any process.\n\n### Deletion gate: SATISFIED (archie-core-8cda.5.6, rev. 2)\n\nThe objective gate now holds. `go list -deps ./cmd/archie-ui` links **zero**\nbanned packages: no `modernc.org/sqlite`, no `github.com/jackc/pgx`, no `internal/infrastructure/postgres`, `internal/gateway`,\n`internal/channels`, `internal/memory`, `internal/secret`, `internal/agentexec`,\n`internal/tools`, `internal/skill`, `internal/yaegiutil`, `internal/domain/curator`,\nor the workflow engine (`internal/domain/workflow`). How the runtime handles were\nsevered, each at the producer:\n\n- **Store contracts** moved to `internal/domain/storecontract` (producer-owned\n read contracts, state-store-contract rev. 2d).\n- **Chat contracts** moved to `internal/domain/messaging`; `internal/gateway`\n keeps aliases. The UI process holds `ChatContract`, `SessionStore`, turn and\n stream types without linking the gateway runtime or its session store.\n- **Webui-owned views** replaced concrete runtime handles on `webui.Server`:\n channel lifecycle via `internal/channels/status`, memory via a `MemoryStatus`\n interface, curators via a `CuratorStatus` interface, and the skill catalogue\n via a `SkillCatalog` interface — each supplied by a daemon-side adapter at\n bootstrap, each degrading to an empty page when unwired.\n- **`config.SecretRef`** moved into `internal/config` (it is config vocabulary);\n `internal/secret` keeps an alias and resolves through `Registry.Resolve`.\n- **Task vocabulary** (`Task`, `Store`, `Definition`, status constants) split\n into `internal/domain/workflow/task`; the pipeline engine stays in package\n `workflow`, which keeps aliases. The UI process links the vocabulary only.\n\nThe gate is enforced, not just measured. `cmd/archie-ui/architecture_test.go`\nre-runs `go list -deps` on every `task check` and fails on any package in a\nbanned category, so a single reference to a type sitting beside a runtime\ncannot quietly relink it. Its exception list (`internal/channels/status`,\n`internal/domain/workflow/task`) is checked in both directions: an exception\nthe UI stops needing is reported, so the allowlist cannot decay into an open\nprefix. `internal/app/archieui`'s `TestComposeUIServerHoldsNoDaemonState`\ncovers the composition clauses, failing on a non-nil `config.Holder`, a\nconcrete `*store.Store`, or a Gateway that is not the gRPC client.\n\nOne piece of residue remained visible at the close of Phase 3: `webui.Server`\nstill declared `Cfg *config.Holder`, because the daemon kept a `webui.Server`\nas the renderer that built the published `ConfigView` snapshot. That cleanup\nlanded as `archie-core-ml30`. `webui.Server` no longer has a configuration\nholder, provenance store, reload-status callback or override-listing callback:\nthe projection is built by `webui.BuildConfigView`, a pure function the\nconfiguration owner calls, and the daemon assembles its input from its own\nstate (`boot.configViewInput`). Readiness reads the daemon's own Holder,\nchannel manager and chat contract, and the health registry it produces is a\ndaemon field rather than a field on a server the daemon never serves.\n\n`ConfigView.Editable` moved with it. It describes the rendering process's own\nwrite path, so `handleConfig` sets it from `Server.UpdateConfig` rather than\nany process baking it into the document. A future configuration write contract\n(`archie-core-j28m`) makes the page editable by wiring that one field.\n\n### Real-process evidence (archie-core-8cda.5.6)\n\n`cmd/archie-ui/main_test.go` builds the binary and runs it against a State\nStore and a Gateway on loopback gRPC, then drives it over HTTP: liveness\nwithout a token, 401 for an API request without one, the `?t=` exchange into\nan HttpOnly cookie, task rows with their forge links, the embedded SPA and its\nassets, a chat stream relayed frame by frame from the Gateway, an operator\ntask action crossing to the Gateway (and refused without the CSRF header),\ncapture intake accepted unauthenticated, the ratified degradations, readiness\ndropping when the State Store dies while liveness holds, and a clean SIGTERM.\n\nTwo dashboard values degraded here because the published projection did not\ncarry them: `chat.show_tool_calls` (the chat page never expanded tool calls)\nand the `/api/setup` checklist (the panel was omitted rather than rendered).\nBoth are closed by `archie-core-ml30`: `ConfigView` gained a `chat` section\ncarrying `show_tool_calls`, the operator name, and whether any chat channel is\nconfigured, and both surfaces now read the projection rather than a live\nholder. The end-to-end suite drives the real binary and streams a turn, so the\n`show_tool_calls` assertion is made on the frames a browser receives.\n\nThe checklist stays. The question raised when it was republished -- is a\nread-only setup report wanted in a process that cannot apply the setup it\nreports? -- resolves in favour of keeping it, because the panel never promised\nto apply anything. It renders each step as a title, a detail and a done mark\nwith no control behind it, so it answers \"what is this deployment missing?\"\nrather than \"fix it here\", and the page that CAN act on it (the Configuration\npage) is where an operator goes next, in the daemon's own words. Removing it\nwould delete the one surface that distinguishes a fresh deployment from a\nfinished one for no gain, since the values it reads already cross the boundary\nin `ConfigView` and the read path costs the daemon nothing. The write path\nremains descoped (see `/api/config` (write) in the route table): the checklist\nreports configuration, it does not change it.\n\n`archie-core-ml30` published the channel flag from two of the three chat\nfront-ends, so an email-only deployment read as configured on `/api/channels`\nand unconfigured on the checklist at the same time (GitHub #821). Both now\nderive from `config.ChatConfig.FrontEnds`, the single definition of a configured\nfront-end, and a test pins the two surfaces to the same answer.\n\nA third value degraded for the same reason and is closed by\n`archie-core-pv6t`: in a deployment configuring `[[identities]]` the projection\ncarried the default identity's forge alone, so task rows withheld their links\nrather than pointing every row at the wrong forge. `ConfigView` now also\npublishes each identity's forge coordinates and the repositories it owns\n(`identities`), which is what resolving a task to its owning forge needs —\ncoordinates and repository names only, never a token. The end-to-end suite\ndrives the real binary against a multi-identity projection and asserts each\nrow's issue link is built from the forge that owns it.\n\n## Non-goals and open work\n\nThis ratification does not define a new domain model, move the dashboard's\nallowlisted configuration schema, redesign Gateway or State Store RPCs, choose\nKubernetes versus local discovery, or authorize unrelated package migration.\nThose decisions remain in their existing authority documents. Any new UI\ncapability that requires a wider Gateway or State Store surface must amend that\nservice's contract before implementation.\n" }, { "url": "/dev/prds/vue-migration-checklist/", diff --git a/docs/prds/binding-secret-encryption.md b/docs/prds/binding-secret-encryption.md index bd9c7c4b9..34e8225b4 100644 --- a/docs/prds/binding-secret-encryption.md +++ b/docs/prds/binding-secret-encryption.md @@ -6,6 +6,11 @@ ## Problem +> Storage superseded: every store now lives in PostgreSQL behind +> `internal/domain/storecontract`, implemented in `internal/infrastructure/postgres` +> (`docs/prds/state-store-contract.md`). SQLite names below describe the +> original design. + `binding.Binding.Secret` (the per-source HMAC shared secret) is persisted as plaintext in the `bindings.secret TEXT` column. Anyone with read access to the SQLite file -- the same access level as `captured_events.body` and `tasks.body` @@ -94,8 +99,8 @@ old material must be retained. - `internal/app/archied/bootstrap.go` `openStores`: resolve both refs through the already-built `secrets` registry, pass the material to `store.Open` via a new option `store.WithBindingCipher(cipher)`. -- `internal/store`: add a `BindingCipher` interface (concrete AES-GCM impl in - `binding_cipher.go`), a `bindingCipher` field on `Store`, and the `Open` +- `internal/infrastructure/bindingcipher`: the `BindingCipher` interface and + concrete AES-GCM impl, consumed by the event-capture store; a `bindingCipher` field on `Store`, and the `Open` option. **Nil cipher = plaintext**, so existing call sites and tests change nothing and legacy deployments keep current behaviour until they set a key. diff --git a/docs/prds/dynamic-curators.md b/docs/prds/dynamic-curators.md index 81b91cb51..8902588b2 100644 --- a/docs/prds/dynamic-curators.md +++ b/docs/prds/dynamic-curators.md @@ -71,7 +71,7 @@ served by one registered engine whose `Manifest()` is the stored definition and whose `Pass` composes `instructions` with the built tool set. This is the curator domain's missing half, not a standalone tool: it lands in -`internal/domain/curator` (entity + engine), `internal/store` (persistence), +`internal/domain/curator` (entity + engine), `internal/infrastructure/postgres` (persistence), `internal/webui` (CRUD), and `internal/app/archied` (seed + apply wiring). ## Definition sources and the merge rule @@ -147,7 +147,7 @@ without a restart. ## Sequencing: persistence and the State Store -The State Store is a standalone process that solely owns `archie.db`; the daemon +The State Store is a standalone process that solely owns the task database; the daemon and Gateway never open it (`TestOpenStoresNeverOwnsTaskDB` guards this), and `proto/state/v1/state.proto` is one gRPC service fronting every ratified store contract. AGENTS.md forbids adding a Go interface method without a matching RPC. @@ -162,7 +162,7 @@ That fixes the order: Phases 1 and 2 are genuinely independent of the wire boundary, so they are not blocked on it. Phase 3 is where the boundary is unavoidable: a definition the WebUI can edit must be reachable from the WebUI, and the WebUI reaches the store -through the State Store, not by opening `archie.db`. Splitting the RPCs into a +through the State Store, not by opening the database. Splitting the RPCs into a later change would leave a table nothing can read -- rejected. Deferring the whole phase is the same rejection, later. diff --git a/docs/prds/eda-playbook-engine.md b/docs/prds/eda-playbook-engine.md index b63ea3260..9e1f6334b 100644 --- a/docs/prds/eda-playbook-engine.md +++ b/docs/prds/eda-playbook-engine.md @@ -353,10 +353,15 @@ single-action playbooks the two granularities are equivalent in effect; the difference starts with multi-action runs -- where whole-run keying would already be wrong. +> Storage superseded: every store now lives in PostgreSQL behind +> `internal/domain/storecontract`, implemented in `internal/infrastructure/postgres` +> (`docs/prds/state-store-contract.md`). SQLite names below describe the +> original design. + **Storage/lookup: a new durable ledger table `playbook_dispatches` in -`internal/store`, copying `binding_dispatches`'s conventions exactly** +the event-capture store, copying `binding_dispatches`'s conventions exactly** (durable table, `INSERT OR IGNORE`, sentinel error, rows freed with -their parent, survives daemon restarts -- `internal/store/bindings.go`): +their parent, survives daemon restarts -- `RecordDispatch`): ```sql CREATE TABLE playbook_dispatches ( @@ -376,9 +381,9 @@ write rather than a constraint error, and the caller matches it with (`internal/daemon/daemon.go` -- "another cycle already won this race," not an error to surface). The record is consumed by the playbook coordinator at dispatch time through a domain-side interface implemented -by `internal/store` -- the same shape as `BindingDispatcher` -(`internal/store/interface.go`) -- and the table lands in the -store's schema application (`internal/store/store.go`). No change +by the event-capture store -- the same shape as `BindingDispatcher` +(`internal/domain/storecontract`) -- and the table lands in the +store's schema migrations. No change to `internal/eventbus`; no per-message dedup state on the client. **Lifetime: no time-based expiry, and no automatic reclamation.** Rows are diff --git a/docs/prds/memory-engine-unification.md b/docs/prds/memory-engine-unification.md index e5d5deb66..169f1eef7 100644 --- a/docs/prds/memory-engine-unification.md +++ b/docs/prds/memory-engine-unification.md @@ -41,7 +41,7 @@ Three further defects fall out of the same split: nothing, and a webhook populates it with nothing either — a route path names a source, not a person, so `webhook.go` leaves `SenderID` empty and carries the route in `Inbound.BudgetKey` for rate limiting. `sender_id` is persisted - (`session_store_sqlite.go`), so whatever a channel puts there reaches the + (`session_store_postgres.go`), so whatever a channel puts there reaches the consumers that read it. `agent-system.md` says the write-scope selection and cross-scope sharing @@ -230,7 +230,7 @@ and would leak user A's extracted facts to user B through the same agent; user-wide would pool facts across agents; and a session id cannot survive the session ending, which is the whole requirement. -This needs data the tree does not persist: the SQLite `messages` table +This needs data the tree does not persist: the `messages` table gains `sender_id`, and `curator.SessionSummary` / `ConversationMessage` gain the agent and participant. The participant is derived from the session's own user-role messages: exactly one distinct non-empty sender writes; zero (a diff --git a/docs/prds/messaging-service-boundary.md b/docs/prds/messaging-service-boundary.md index 7387d3acb..5c803caaf 100644 --- a/docs/prds/messaging-service-boundary.md +++ b/docs/prds/messaging-service-boundary.md @@ -16,7 +16,7 @@ The Messaging Service (`cmd/archie-messaging`, composed via `internal/app/archie It is a peer service to the UI Service and the daemon: - It serves external chat participants across configured communication platforms. - It consumes Gateway's `messaging.ChatContract` exclusively for chat turn execution, streaming, session queries, cancellation, persona selection, and task control actions. -- It holds no direct database connections (`*store.Store` or SQLite), owns no LLM execution or model provider credentials, and has no dependency on the daemon runtime or the UI HTTP server. +- It holds no direct database connections, owns no LLM execution or model provider credentials, and has no dependency on the daemon runtime or the UI HTTP server. ## Boundary and Ownership @@ -24,7 +24,7 @@ It is a peer service to the UI Service and the daemon: |---|---|---|---| | **Channel transports** | Telegram bot long-polling, Email IMAP/SMTP loops, Inbound Webhook HTTP listener | Gateway `ChatContract` | Direct LLM providers, model runtimes, tokenizer libraries | | **Turn dispatch** | Inbound message packaging, typing indicators, live streaming updates | `ChatContract.Route`, `ChatContract.Stream` | `gateway.Router`, `runtime.Runtime`, agent prompt builders | -| **Sessions & History** | Platform-to-session key mapping, conversation thread ID translation | `ChatContract.GetSession`, `RecentMessages`, `RecentTurns` | `*store.Store`, `SessionStore` SQLite handles, direct database files | +| **Sessions & History** | Platform-to-session key mapping, conversation thread ID translation | `ChatContract.GetSession`, `RecentMessages`, `RecentTurns` | store implementations, `SessionStore` database handles | | **Control & Actions** | Platform command parsing (`/status`, `/tasks`, `/cancel`, `/stop`, `/persona`) | `ChatContract.Snapshot`, `ApplyTaskAction`, `Cancel`, `SetPersona` | `daemon.Daemon`, workflow execution engine, worktree operations | | **Task Approvals** | Channel interactive keyboards, approval callbacks | `ChatContract.ApplyTaskAction` | Direct task status mutations, forge comment posters | | **Configuration** | Platform bot tokens, allowed sender/user IDs, webhook listen ports | `[chat.*]` channel configs, Gateway target/token | `config.Holder`, forge tokens, repo configs, agent container specs | @@ -66,7 +66,7 @@ It does not read or decode `[forge]`, `[repos]`, `[models]`, `[providers]`, `[ru ## Deletion Gate and Verification The extraction is guarded by `cmd/archie-messaging/architecture_test.go`, verifying that `go list -deps ./cmd/archie-messaging` links zero banned runtime packages: -- `modernc.org/sqlite` and `internal/store` (State Store) +- `modernc.org/sqlite`, `github.com/jackc/pgx`, `internal/infrastructure/postgres` and `internal/infrastructure/legacyread` (State Store) - `internal/daemon`, `internal/app/archied`, `internal/container`, `internal/worktree` (Daemon runtime) - `internal/domain/workflow`, `internal/agentexec`, `internal/taskrun` (Workflow engine) - `internal/forge`, `internal/forgerpc` (Forge client) diff --git a/docs/prds/payload-field-mapping.md b/docs/prds/payload-field-mapping.md index 9074aca42..94189e15f 100644 --- a/docs/prds/payload-field-mapping.md +++ b/docs/prds/payload-field-mapping.md @@ -12,6 +12,11 @@ reference by ID. ## Decision: a new `internal/domain/mapping` package, SQLite storage +> Storage superseded: every store now lives in PostgreSQL behind +> `internal/domain/storecontract`, implemented in `internal/infrastructure/postgres` +> (`docs/prds/state-store-contract.md`). SQLite names below describe the +> original design. + `internal/domain/mapping` owns the mapping vocabulary and the pure resolve/preview rule; `internal/store` persists it (mirrors `captures.go`'s shape exactly -- inline schema const, narrow store @@ -107,7 +112,7 @@ is `t2db.4`'s job, not duplicated here. ## Storage -`internal/store/mappings.go`, `field_mappings` table: `id, name, +The `mappings` table: `id, name, source_hint, fields_json, created_at, updated_at`. `fields_json` is the `[]Field` slice JSON-encoded into one column -- fields have no independent query need (nothing filters "mappings with a field named X"), so a diff --git a/docs/prds/pr-review-remediation.md b/docs/prds/pr-review-remediation.md index 91f70e4db..a6c5d07ae 100644 --- a/docs/prds/pr-review-remediation.md +++ b/docs/prds/pr-review-remediation.md @@ -89,7 +89,7 @@ closed or archived is dropped through the same counter: the reaction is only ever work against a PR archie itself opened *and still owns*. That lookup does not exist. `pr_number` is written by -`internal/store/store.go` and never read in a `WHERE` clause, so this needs a +`internal/infrastructure/postgres/store.go` and never read in a `WHERE` clause, so this needs a `TaskStore` method keyed on `(owner, repo, pr_number)`, a matching `StateStoreService` RPC (a Go interface method on a store facade without one is forbidden, `CLAUDE.md`), and an index on those three columns. It is the @@ -97,7 +97,7 @@ authorization boundary, so it is sequenced as its own step below rather than carried along with the envelope. **Identity is derived from the resolved task, never carried in the reaction.** -The task row already stores `identity` (`internal/store/store.go`), and it is +The task row already stores `identity` (`internal/infrastructure/postgres/store.go`), and it is the resolved task's identity that selects the forge client, worktree manager, repo config and `bot_user` for the remediation run and the `ReplyToReview` reply, through the existing `forgeFor`/`treesFor`/`repoFor` helpers. That is diff --git a/docs/prds/service-decomposition.md b/docs/prds/service-decomposition.md index eee35bad0..a3c3bfa88 100644 --- a/docs/prds/service-decomposition.md +++ b/docs/prds/service-decomposition.md @@ -42,7 +42,7 @@ boundaries. | Scheduler Service | cron/ticker contract | `internal/domain/scheduling` | | Execution Environment | execution-dispatch contract | `internal/domain/workflow`, task dispatch paths | | Runner | isolated worktree/container execution | `internal/container`, `internal/agentexec`, `internal/worktree` | -| State Store | state-access contract | `internal/store` | +| State Store | state-access contract | `internal/infrastructure/postgres` | ### Discovery @@ -191,8 +191,8 @@ historical record rather than an outstanding gate. State Store service's contract (less isolation, less migration risk). Not yet decided. - **RESOLVED.** One State Store service owns the existing - single SQLite file behind narrow typed contracts (`TaskStore`, + **RESOLVED.** One State Store service owns the task data + (now PostgreSQL, see `docs/prds/state-store-contract.md`) behind narrow typed contracts (`TaskStore`, `CaptureStore`, `MappingStore`, `BindingStore`, `WorkflowStore`) over gRPC. *(Ratified 2026-09-06: the workflow contract is now the domain-owned `workflow.Store`, which supersedes @@ -201,7 +201,7 @@ historical record rather than an outstanding gate. No physical database-per-service: the solo-maintainer / team- topology rule of thumb plus the low value of per-service autonomy for one person rule it out, and one store means no distributed-transactions - problem. The Gateway keeps its own already-separate session SQLite. + problem. Concrete migration path (generalise `storerpc`, swap consumers one contract at a time, stand up the store service, split the file only on evidence): @@ -212,7 +212,7 @@ historical record rather than an outstanding gate. `docs/prds/state-store-contract.md` (rev. 2c): one `StateStore` gRPC service, 42 RPCs; **ownership split** — consumer-owned `workflow.Store` + `workflow.Task`/`Status`/`Source` in the workflow domain (dependency rules - #2/#7), producer-owned daemon/webui store surfaces in `internal/store`; + #2/#7), producer-owned daemon/webui store surfaces in `internal/domain/storecontract`; the `Task`/`Status`/`Source` + interface relocation is a Phase 2 prerequisite pulled from migration-decisions §4 (`store.Task → WorkflowExecution`); `RecordDispatch` drops `*sql.Tx`; `Close` off-wire; diff --git a/docs/prds/state-store-contract.md b/docs/prds/state-store-contract.md index ff78e1abc..8c7339e32 100644 --- a/docs/prds/state-store-contract.md +++ b/docs/prds/state-store-contract.md @@ -16,6 +16,15 @@ split `archie.db`. It records method ownership, DTO/wire representation, error s limits, mode/config shape, local-adapter compatibility, and cutover/deletion rules so that `.4.2` and `.4.3` have exactly one authoritative surface to build against. +> **Storage superseded.** The store behind this contract is PostgreSQL: +> `internal/infrastructure/postgres` implements the producer-owned interfaces in +> `internal/domain/storecontract` and `workflow.Store`, over the database +> `database_url` names. `internal/store`, the SQLite file and the local adapter +> are gone; where this document says `archie.db`, `*store.Store` or +> `internal/store`, read the State Store's PostgreSQL database and +> `internal/infrastructure/postgres`. The contract, ownership split, error +> semantics and cutover rules stand. + > **Current state — read before `.4.2` starts.** The §12 step 2 relocation (`archie-core-8cda. > 4.8`) has landed: `Task`/`Status*`/`Source*` and the 3-method `workflow.Store` now live in > `internal/domain/workflow`, the six production workflow files import no `internal/store` @@ -584,9 +593,9 @@ an explicit operator decision. `target` cannot boot, so the section ships uncommented. - Composition root: `internal/app/archied/bootstrap.go` (the daemon) and `internal/app/archieui/config.go` (the UI process) both read `cfg.Services.Get("state")` and - dial the remote State Store. `openProductionTaskStore` is **not** called from the daemon: it - lives in the standalone `archie-state-store` process (`internal/app/archied/state_store.go`), - which is the only path that opens `archie.db`. + dial the remote State Store. The task store is built only in the standalone + `archie-state-store` process (`internal/app/archied/state_store.go`), which is the only path + that opens the task tables. > **Mode fork, decided.** The gateway uses a single `target` address (no enum). The State Store > boundary uses the same seam for symmetry, and **both now require a non-empty `target`**: the @@ -656,7 +665,7 @@ an explicit operator decision. contracts for a contract-code (no dual-live *window*; each contract is either local or remote at a time). 7. **Final flip / deletion of the in-process serving path:** when the daemon no longer owns the - store (i.e. `TaskStore` has moved), delete the in-process `openProductionTaskStore` / + store (i.e. `TaskStore` has moved), delete the in-process task-store / `wireWebStoreSurfaces` path in the daemon in a **same-commit** deletion — no dual-store ownership. The single SQLite file remains owned by `archie-state-store`. 8. **Never** split `archie.db` or add a second store. The gateway keeps its own already-separate diff --git a/docs/prds/task-run-detail.md b/docs/prds/task-run-detail.md index b7ec0a080..277f4f54c 100644 --- a/docs/prds/task-run-detail.md +++ b/docs/prds/task-run-detail.md @@ -43,8 +43,8 @@ that has since moved on. ## Attempt attribution: a column, not a guess -`events.Event.Attempt`, persisted by `internal/store` as the `events.attempt` -column (`internal/store/events.go`, `eventsSchema`), is a real column, +`events.Event.Attempt`, persisted as the `events.attempt` column +(`internal/infrastructure/postgres/migrations/0001_state_store.sql`), is a real column, `NOT NULL DEFAULT 0`. It was chosen over segmenting the stream on `task_queued` / `task_retried` because a fold would have to guess which side of a boundary an event belongs to when the daemon restarts mid-attempt, when `RecoverStale` @@ -60,6 +60,11 @@ attempt 1. ## Migration against an existing `archie.db` +> Storage superseded: every store now lives in PostgreSQL behind +> `internal/domain/storecontract`, implemented in `internal/infrastructure/postgres` +> (`docs/prds/state-store-contract.md`). SQLite names below describe the +> original design. + The store's migrator (`migrateTasks` in `internal/store/store.go`) already reads `PRAGMA table_info` through `tableColumns` and applies a presence-gated `ALTER TABLE ... ADD COLUMN` list. This change adds an `events` arm: a @@ -249,8 +254,7 @@ than useless. ## Files this change touches -- `internal/store/`: the `events.attempt` column, its migrator arm, and the - schema-presence test. +- the State Store schema: the `events.attempt` column. - `internal/events/`: `Event.Attempt`, `KindChangesCaptured`, `KindConfigCaptured`, `ConfigCapturedSchema`. - `internal/domain/workflow/task/changes.go`: the change value types. diff --git a/docs/prds/ui-service-boundary.md b/docs/prds/ui-service-boundary.md index a0f060753..8f153b347 100644 --- a/docs/prds/ui-service-boundary.md +++ b/docs/prds/ui-service-boundary.md @@ -157,7 +157,7 @@ The implementation must complete these gates in order: behavior. Add a composition test proving the selected adapter is the one used by production wiring. 3. Remove direct `internal/gateway` session/router/runtime access from web UI - handlers. Remove direct `internal/store` implementation access, SQL access, + handlers. Remove direct store implementation access, SQL access, daemon pointers, and broad runtime callback bundles from the UI constructor. 4. Replace the shared `config.Holder` with the UI-owned configuration DTO and narrow configuration application contract. Prove reload/update atomicity @@ -177,7 +177,7 @@ The deletion gate is objective: `go list -deps` from the UI process shows no daemon, SQL/store implementation, workflow, forge, channel, model, or secret runtime package; production composition contains one UI HTTP listener; and architecture tests reject `config.Holder`, concrete `*store.Store`, concrete -Gateway runtime types, and direct SQLite imports in the UI service. +Gateway runtime types, and direct database drivers in the UI service. Rollback is a deployment composition change that points the operator at the last validated UI binary and restores the previous endpoint configuration. It @@ -199,9 +199,9 @@ Current code proves the following migration inputs: Gateway-owned request/value types; handlers must finish the DTO projection. - `internal/webui/server.go:Handler` owns the route set, SPA fallback, health split, capture bypass, and token wrapper. -- `internal/app/archied/state_store.go:RunStateStore` owns the task SQLite - file and serves the State Store gRPC contract; the UI must use that contract - rather than opening the file. +- `internal/app/archied/state_store.go:RunStateStore` owns the task + database and serves the State Store gRPC contract; the UI must use that contract + rather than opening the database. - `internal/app/archied/gateway.go:RunGateway` serves Gateway gRPC and owns Gateway session persistence; the UI must use `ChatContract` rather than Gateway internals. @@ -230,17 +230,17 @@ form again: `internal/webui` receives no holder in any process. ### Deletion gate: SATISFIED (archie-core-8cda.5.6, rev. 2) The objective gate now holds. `go list -deps ./cmd/archie-ui` links **zero** -banned packages: no `modernc.org/sqlite`, no `internal/store`, `internal/gateway`, +banned packages: no `modernc.org/sqlite`, no `github.com/jackc/pgx`, no `internal/infrastructure/postgres`, `internal/gateway`, `internal/channels`, `internal/memory`, `internal/secret`, `internal/agentexec`, `internal/tools`, `internal/skill`, `internal/yaegiutil`, `internal/domain/curator`, or the workflow engine (`internal/domain/workflow`). How the runtime handles were severed, each at the producer: - **Store contracts** moved to `internal/domain/storecontract` (producer-owned - read contracts, state-store-contract rev. 2d); `internal/store` keeps aliases. + read contracts, state-store-contract rev. 2d). - **Chat contracts** moved to `internal/domain/messaging`; `internal/gateway` keeps aliases. The UI process holds `ChatContract`, `SessionStore`, turn and - stream types without linking the gateway runtime or its SQLite session store. + stream types without linking the gateway runtime or its session store. - **Webui-owned views** replaced concrete runtime handles on `webui.Server`: channel lifecycle via `internal/channels/status`, memory via a `MemoryStatus` interface, curators via a `CuratorStatus` interface, and the skill catalogue diff --git a/install.sh b/install.sh index a91fabe56..bcb14c393 100755 --- a/install.sh +++ b/install.sh @@ -8,9 +8,9 @@ # - env (environment variables & API secrets) # - persona/ # - skills/ -# Data: ${XDG_DATA_HOME:-~/.local/share}/archie -# - archie.db-tasks.sqlite -# - archie.db-conversations.sqlite +# Data: ${XDG_DATA_HOME:-~/.local/share}/archie (state_dir) +# - nats/ +# - logs/tasks/ # - work/ # - memories/ # - tasks/ @@ -186,7 +186,7 @@ echo "==> Building native archie binaries..." # ask each installed binary what it is (archie-core-k94o). LDFLAGS="${LDFLAGS} -X github.com/samcharles93/archie-core/internal/buildinfo.Version=${GATEWAY_VERSION}" LDFLAGS="${LDFLAGS} -X github.com/samcharles93/archie-core/internal/buildinfo.Runtime=${RUNTIME_VERSION}" - # archied does not run alone: the State Store owns archie.db, the Gateway + # archied does not run alone: the State Store owns the task data, the Gateway # serves the chat contract, the dashboard is its own process, and the # Messaging Service owns the chat channels. Building only archied leaves it # unable to boot, and omitting archie-messaging leaves the Telegram, email and @@ -460,7 +460,7 @@ RestartSec=5s WantedBy=default.target EOF - # The State Store owns archie.db and is the process everything else dials, so + # The State Store owns the task data and is the process everything else dials, so # its unit is the first dependency the others name. cat < "${SYSTEMD_USER_DIR}/archie-state-store.service" [Unit]