Files
archie/docs/prds/dynamic-curators.md
T

193 lines
13 KiB
Markdown

# Dynamically-defined curators -- decision
**Status:** Decided, to be implemented
**Date:** 2026-09-12
**GitHub epic:** [#789](https://github.com/samcharles93/archie-core/issues/789)
**Parent epic (curator engine):** [#435](https://github.com/samcharles93/archie-core/issues/435) (closed)
**Depends on:** nothing shipping; this resolves the open question that epic #435's
"registered and configured declaratively, not hardcoded" criterion left open.
## §0 Implementation status
Every claim in this document is mapped to what exists in the tree today.
| Claim | Status | Where it stands |
|---|---|---|
| Curator engine family: typed contract, owning registry, start/health/stop, narrow host access | **Implemented** | `internal/domain/curator/{contract,registry,runtime,registrar,activity,wake}.go` |
| Curator declared shape (`Manifest`: interval, cooldown, on_input, tools, skills, memory engine, conversations, model) | **Implemented** | `internal/domain/curator/contract.go`; `Validate()` requires `Interval > 0` |
| Two reference curators run through the registry | **Implemented** | `internal/infrastructure/skillcurator`, `internal/infrastructure/sessioncurator` |
| Curator definitions as persisted, API-editable data | **Aspirational -- zero code** | No table, no entity, no RPC, no config block |
| Generic definition-driven engine that executes a declared tool set + free-form instructions | **Aspirational -- zero code** | Only the two code-registered curators exist |
| `Registrar.Tools` (`ToolBuilder`) resolved and bound at registration | **Partial** | Interface declared (`registrar.go:86`); **no implementation anywhere**; `bootstrap.go` builds the `Registrar` with no `Tools` field |
| A curator can declare tools at all | **Partial** | `Manifest.Tools` exists, but `registry.go:123` fails registration with `"curator manifest: declares tools but the registrar has no ToolBuilder"` |
| Curator passes that actually reach a model with tools | **Cosmetic only** | `curator.ChatRequest` carries `Tools`/`MaxSteps`; `curatorLLMRunner` (`internal/app/archied/main.go:551`) drops both and returns `ChatResult{Text}` only |
| Dynamic registration / removal (no restart) | **Aspirational -- zero code** | `Registry` has `Register`, no `Unregister`; `Runtime` has `Start`/`Nudge`/`Stop`, no `Add`/`Remove`; `Start()` snapshots membership |
| Curator definition CRUD over REST | **Partial** | `GET /api/curators` read-only (`internal/webui/server.go:226`); no write routes |
| Curator definition config | **Aspirational -- zero code** | No `[curator]` block in `internal/config/config.go` |
| Store/API as definition authority | **Aspirational -- zero code** | No curator persistence of any kind |
| No live-path defaults for interval/cooldown/tools/model/memory engine | **Partial** | `skillcurator.DefaultInterval` / `sessioncurator.DefaultInterval` are Go consts read at registration; `curator.NewRuntime(..., RuntimeConfig{})` uses code defaults for pass timeout and concurrency |
| Idle curators are observable in logs | **Partial** | `runtime.go:202-211` logs nothing when `Check` reports not-due; the session-memory curator's hourly pass is the only curator activity ever logged |
| Sampler family | **Implemented** | `internal/domain/sampling`, shipped; no curator consumes a `Sampler` yet |
| Forge/issue reach from chat | **Implemented** (via `shell`) | No dedicated issue tool exists, but the chat agent's `shell` is unconfined (`chat.unrestricted_filesystem = true`), so `gh issue edit|comment|create|close` works today. Dedicated tools (#161) are convenience, not a blocker |
> **Target state, not current state.** Everything below marked aspirational is
> design, not a description of the daemon as it runs today.
## Problem
A curator exists today only if a Go package implements `curator.CuratorEngine`
and `bootstrap.go:1022/1025` names it in a `Register` call. Nothing about a
curator is definable as data. Worse, no curator can declare a tool set at all:
the `ToolBuilder` interface has no implementation in the repo, so the registry
refuses (correctly, fail-closed) any curator that declares tools -- and the
daemon's model adapter drops the tools even if it were bound. The curator family
therefore has exactly one shape (cheap, tool-less maintenance pass) and its only
extension point is compiling Go.
Sam's requirement is the opposite: *"Curators should be dynamically created and
defined, in addition to being able to be defined in code/config/webui"* and *"No
configuration should ever be hard-coded."* The motivating consumer is a
project/orchestrator curator -- declared as data, given forge issue tools, and
pointed at a repo to review, refine, and propose issues.
## Decision
**A curator definition is data: a persisted entity whose declared shape is
exactly `curator.Manifest` plus identity, free-form instructions, and an enabled
flag. One generic engine interprets any definition.** The shape is not a second
invention -- `Manifest` already encodes interval, cooldown, on-input, tools,
skills, memory engine, conversations, and model, and `Validate()` already
enforces what a valid shape means. The definition adds only what `Manifest`
cannot express:
| Definition field | Relationship to `Manifest` |
|---|---|
| `name`, `enabled` | Identity and lifecycle; the registry's key is the name |
| `instructions` | **Free-form** system prompt for the generic engine -- the piece that makes a definition more than a knob set |
| interval, cooldown, on_input, tools, skills, memory_engine, conversations, model | The existing `Manifest` fields verbatim; the definition **is** a `Manifest` |
Code-registered engines keep `CuratorEngine` unchanged. A data-defined curator is
served by one registered engine whose `Manifest()` is the stored definition and
whose `Pass` composes `instructions` with the built tool set.
This is the curator domain's missing half, not a standalone tool: it lands in
`internal/domain/curator` (entity + engine), `internal/store` (persistence),
`internal/webui` (CRUD), and `internal/app/archied` (seed + apply wiring).
## Definition sources and the merge rule
Three sources, per Sam's requirement. **The store is authoritative.**
| Source | Role | Collision behaviour |
|---|---|---|
| **Code** (a registered Go engine) | The escape hatch: a curator whose strategy is Go, not tools+instructions | Wins over data for its own name -- a data definition cannot silently replace a compiled engine; the store refuses the collision and logs it |
| **Config** (`[[curators]]`) | **Seed data only**, merged at startup | Insert-if-absent. A store row of the same name is never overwritten, and deleting a store row is not undone by config on the next boot |
| **WebUI/API** (the store) | **Authoritative** | Wins over config on a name collision; the only source of truth after first boot |
Config existing at all is what makes a definition reproducible on a fresh host;
making it authoritative would make the API a lie, so it is seed data with a
one-way merge. The alternative -- config authoritative, store a cache -- would
mean an API edit silently reverts on restart. Rejected.
## How the generic engine executes a definition
A pass is: resolve the declared tools, compose the prompt, run the model loop,
record actions.
1. **Tool resolution.** `Registrar.Tools.Build(ctx, declared)` returns exactly
the declared set, never a broader registry. A declared name with no
implementation fails the pass rather than silently running with fewer tools.
This is the enforcement point, and it is why the `ToolBuilder` binding is
load-bearing rather than a convenience: without an implementation, no curator
can declare tools at all (today's `registry.go:123` guard).
2. **Tool-carrying model call.** `curator.ChatRequest` already carries
`Tools []tools.ToolEntry` and `MaxSteps`; `curatorLLMRunner` must pass them
into `core.GenerateOptions` (which has `Tools`, `ToolChoice`, `MaxSteps`, and
already executes the tool loop) instead of dropping them. `ChatResult` is
widened only as far as attributing the pass's actions requires.
3. **Prompt.** The system prompt is the definition's `instructions` plus the
declared tool summaries -- rendered from the same set that was built, never
from a wider list. Each pass records `Action`s so activity stays attributable
through the existing registry `activity` surface and events.
A bespoke non-tool strategy (the skill curator's structural validation, the
session-memory curator's own extraction) remains a registered Go engine. That is
the escape hatch, not the mode: the mode is tools + instructions as data.
## Dynamic registration and mutation
A definition change must apply without a daemon restart, which the runtime
cannot do today (`Start()` snapshots membership into one goroutine per curator;
no `Unregister`, no `Add`/`Remove`).
- `Registry.Unregister(name)` -- valid while running, same failure isolation as
`Register` (a rejected removal affects only that curator).
- `Runtime.Add(engine)` / `Runtime.Remove(name)` -- `Remove` cancels the removed
curator's in-flight pass context and stops its goroutine within a bounded
wait; it never holds the registry mutex across a pass.
- The app layer owns the apply sequence (stop old, register new, start) so the
registry stays the membership authority and the runtime the scheduler.
Removal must not be able to orphan a loop, and a pass that ignores cancellation
must not block removal forever.
## REST / WebUI surface
Mirror `internal/webui/api_binding.go`, not a new pattern:
| Route | Handler shape |
|---|---|
| `GET /api/curators` | Existing read-only list, fields unchanged (dashboard keeps working) |
| `POST /api/curators` | `handleCuratorCreate` |
| `GET /api/curators/{id}` / `PATCH` / `DELETE` | `handleCuratorGet` / `Update` / `Delete` |
All mutations behind `authorizeTaskMutation`, same as bindings and mappings. A
successful mutation goes through the runtime apply path above, so it is live
without a restart.
## Sequencing: persistence and the State Store
The State Store is a standalone process that solely owns `archie.db`; the daemon
and Gateway never open it (`TestOpenStoresNeverOwnsTaskDB` guards this), and
`proto/state/v1/state.proto` is one gRPC service fronting every ratified store
contract. AGENTS.md forbids adding a Go interface method without a matching RPC.
That fixes the order:
| Phase | Ships | Proto work |
|---|---|---|
| 1. Engine + config source | `ToolBuilder`, tool-carrying passes, generic engine, `[[curators]]`, idle logging | None -- nothing persisted yet |
| 2. Runtime mutation | `Unregister`, `Add`/`Remove`, apply path | None -- in-memory membership |
| 3. Persistence | definition entity + store schema, config seed (store authoritative), REST CRUD | **Same change**: request/response messages + RPCs, `staterpc.Client` façade, `mapError`/`unmapError`, and an update to `docs/prds/state-store-contract.md` |
Phases 1 and 2 are genuinely independent of the wire boundary, so they are not
blocked on it. Phase 3 is where the boundary is unavoidable: a definition the
WebUI can edit must be reachable from the WebUI, and the WebUI reaches the store
through the State Store, not by opening `archie.db`. Splitting the RPCs into a
later change would leave a table nothing can read -- rejected. Deferring the
whole phase is the same rejection, later.
## No live-path constants
No default may be a live-path constant. Interval, cooldown, tools, model, memory
engine, pass timeout, and concurrency are definition or config values. A
definition missing `interval` is refused with a clear error, never defaulted at
pass time; `skillcurator.DefaultInterval` and `sessioncurator.DefaultInterval`
stop being read at registration (the value becomes seed data), and
`curator.NewRuntime(..., RuntimeConfig{})` stops relying on code defaults.
## What this deliberately does NOT do
- **No fixed catalogue of curator "types".** A definition is arbitrary tools +
instructions, not a union of known kinds.
- **No chat tool that mints curators.** A definition grants an LLM a tool set; a
curator able to create curators is privilege escalation. Mutation is REST with
the same authority as bindings -- not exposed on the chat tool surface.
- **No dashboard SPA editor page.** The REST surface is the WebUI definition
source; the dashboard can adopt it separately.
- **No forge issue tools here.** This design only ensures a curator *can declare*
tools. Dedicated issue tools (#161) are a convenience, not a prerequisite: the
chat agent already reaches issues through the unconfined `shell` tool
(`gh issue edit|comment|create|close`). See the correction on #789.
- **No second authority.** No file-per-curator definition store, no parallel
config reader.