diff --git a/docs/data/generated/dev-docs.json b/docs/data/generated/dev-docs.json index 6c8fe1b8c..5679109e8 100644 --- a/docs/data/generated/dev-docs.json +++ b/docs/data/generated/dev-docs.json @@ -408,6 +408,12 @@ "section": "PRDs", "body": "# Playbook data-flow and condition syntax\n\n**Status:** Approved\n**Date:** 2026-09-03\n**Parent:** `docs/prds/eda-playbook-engine.md`, epic `archie-core-t2db`\n**Beads issue:** `archie-core-t2db.14` (expression environment).\n\n**Decision:** CEL via `cel.dev/cel-go` (v0.32.0), pinned. One expression\nmechanism covers both an action's `when` condition and its `args` values.\n\n## The requirement, stated directly\n\nA playbook action is gated and parametrised by runtime data: (a) a condition\ndeciding whether the action runs; (b) a later action's `args` values referencing\nan earlier action's `Result` and the triggering event's fields. Both are **CEL\nexpressions**, evaluated against one evaluation context. There is no separate\ninterpolation syntax: CEL reads nested maps and fields natively, so `args`\nvalues that need data are written as CEL, and only plain literal args remain\nplain YAML scalars.\n\nBoth halves are read-only: expressions may read context values and compute a\nresult from them, never mutate, never call host functions, never perform I/O.\nCEL enforces this structurally (no side effects, linear evaluation when macros\nare bounded, cost-limited).\n\n```yaml\nactions:\n - id: build\n position: module\n kind: log\n args:\n message: '\"build finished\"' # literal\n when: 'event.label == \"bugfix\"' # gate on event field\n - position: module\n kind: log\n args:\n message: '\"done: \" + string(actions.build.result.written)'\n when: \"actions.build.result.written == true\"\n```\n\n## Trust boundary of the expression\n\nA playbook YAML is **operator-installed, in-process, daemon-privileged** -- the\nsame tier as `ModuleDir`/`PluginDir`/`SecretEngineDir` (module-position.md's\ntrust-boundary section): loaded from a configured directory at startup, never\nfrom a webhook body or task worktree. The schema-by-example flow uses live\nevents to _design against_; the saved playbook is operator-authored file content\nin an operator-configured location.\n\nTwo consequences:\n\n1. The expression string is **trusted**. Operator-authored config, not attacker\n input; no sandboxing against a hostile expression string is required. If a\n future playbook source is less trusted (auto-generated bindings from\n unauthenticated captures), that is the playbook-file trust tier moving -- the\n loader's acceptance check is the right place, not the evaluator.\n2. The **data the expression reads is NOT trusted** (decoded webhook body, prior\n results through the same pipeline). Robustness against hostile data is\n required: no panic on wrong-typed/missing fields, no unbounded cost on\n deep/cyclic maps, bounded evaluation time. CEL provides this structurally\n (cost limits, linear non-Turing-complete evaluation, error values instead of\n panics -- verified in the vetting record below).\n\n## The gate precedent is deliberately NOT copied\n\n`internal/gate/gateeval` evaluates arbitrary interpreted Go (`.archie/gate.go`)\nas the repo's conditional-logic precedent. That is the wrong precedent for\nplaybook conditions because this document's premise is: **YAML is pure\norchestration data; Yaegi is where imperative logic lives** (Problem section;\nmodule-position.md's \"no generic hook\" section). Yaegi snippets in YAML would\nmove imperative logic into the data layer -- the inversion this design exists to\nprevent -- and make playbooks un-lintable.\n\nThe line: conditions are **declarative predicates over runtime data, not\nprograms**. CEL is non-Turing-complete and side-effect-free, so it cannot become\na program; a condition that needs real logic gets a Module kind (Yaegi),\nreferenced by the action, not a stronger expression body.\n\n## Plugin-engine-rule interaction\n\nThe plugin engine rule forbids a generic `Module` interface with an `any`\npayload; it exists to keep static implementation contracts typed. Evaluating CEL\nagainst runtime `map[string]any` is **not the same problem**: the rule concerns\n_implementation contracts_ (static operations a capability family exposes), not\n_runtime data values_ (legitimately dynamic -- an event payload's shape is\nunknown until it arrives). The Module contract stays fully typed (generated\n`Args`/`Result` schemas); CEL is a read-only data reader under the playbook\nengine family, not a new capability contract. Stated once.\n\n## Minimum viable evaluation context (what expressions see)\n\nOne flat context, namespaced from the playbook run, built at dispatch time:\n\n| Name | Type | Source |\n| --------- | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `event` | `map(string, dyn)` | The triggering event's decoded payload (webhook body / forge issue / schedule tick). Field access via `event.\u003cfield\u003e`, map access via `event[\"field\"]`, presence via `has(event.\u003cfield\u003e)`. |\n| `actions` | per-playbook object type | Previous actions' results, one field per declared action `id` (`actions.\u003cid\u003e`), each field typed `{ result: \u003cKindResult\u003e }` so a later action reads an earlier one's `Result` struct (`actions.\u003cid\u003e.result.\u003cfield\u003e`) with field-level checking. The current action and later actions are not present. |\n\nExpressions may also read literal-only state (numbers, strings, booleans)\ndirectly. CEL's `has()` macro covers the missing-key case (`has(event.labels)`\n-\u003e bool), which replaces any bespoke `has`/ `contains` operator.\n\n`actions.\u003cid\u003e.result` is the kind's hand-written `Result` struct:\n`ModuleRegistry.DecodeResult` converts `Module.Invoke`'s flat `map[string]any`\n(`archie-core-t2db.13`) back into that struct before a later expression reads\nit, so the checker and the run read the same typed value.\n\nJudgement call J1: `action` id is a required field on every action that later\nactions reference; ids are validated at load (unique, stable-identifier-shaped),\nand a `when`/`args` expression referencing an unknown `actions.\u003cid\u003e` is a **load\nfailure**, not a runtime miss -- the same reject-at-load rule as collisions (see\nbelow).\n\n## Load-time validation (consistent with the collision rule)\n\nThe collision-handling rule is: schema defines the accepted message; anything\noutside it is rejected at load, logged plainly, reported to the caller -- never\ndeferred to runtime. Expressions are validated the same way, at playbook load:\n\n- **Syntax**: every `when` and every CEL-typed `args` value is parsed and\n checked by CEL's checker when the playbook is loaded. A syntax or type error\n is a reported load failure (the playbook is dropped and the error goes to the\n caller), not a runtime surprise.\n- **Context conformance**: with the context declared to CEL (`event` as\n `map(string, dyn)`, `actions` as a per-playbook object type with one typed\n field per declared id), unknown top-level identifiers are rejected at compile\n time. Field-level typing inside `dyn` values is deferred to runtime by CEL's\n design (verified: `data.items.lenght` with `data` dyn compiles clean and\n errors at eval) -- so the loader declares as much type as the schema provides\n (the action kinds' generated `Result` structs become CEL type declarations at\n load, giving field-level rejection for result reads), and `event` stays `dyn`\n because its shape is unknown by design (schema-by-example).\n- **Names**: unknown `actions.\u003cid\u003e` references and unknown roots are\n compile-time/lint-visible errors (J1).\n\nThis means the lint tool and the daemon share one checker -- same single-source\nargument as the playbook loaders.\n\n## Candidate comparison\n\n| Candidate | Maintenance | Safety surface | Cost (measured) | Verdict |\n| ------------------------ | -------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------ |\n| **CEL (cel.dev/cel-go)** | Actively maintained (google/cel, monthly releases) | Non-Turing-complete, linear eval, `CostLimit`/`CostTracking`/`ParserRecursionLimit`, no side effects, error values not panics | **+13.0MB** (2.34MB -\u003e 15.25MB) | **Chosen** |\n| `expr-lang/expr` v1.17.8 | Actively maintained | Side-effect-free, `DisableAllBuiltins`, depth limits (CVE fixed w/ tests) | +3.9MB (2.34MB -\u003e 6.28MB) | Not chosen: two safety CVEs this cycle; smaller ecosystem; no protobuf/type-model fit with future schema-gen |\n| `gval` v1.2.3 | Maintained, slower | No builtin-restriction surface; extras (ternary/`??`) not wanted | +3.0MB (2 transitive deps) | Not chosen |\n| `govaluate` | **ARCHIVED** (author archived 2024) | n/a | n/a | Not chosen: dead |\n| `starlark-go` | Maintained (Bazel) | Turing-complete with step limits | large (full language) | Not chosen: whole-language surface |\n\n## Vetting record\n\nSource reviewed at HEAD:\n\n- **Repo hygiene**: clean. No binaries/media in tree (the only non-source\n artifacts are expected generated protobuf files); 16-line `.gitignore`.\n- **Hard-boundary correctness**: the place correctness is hard for an expression\n evaluator is bounding evaluation and surviving hostile input. Dedicated\n regression tests exist and pass at HEAD: `TestCostLimit`, `TestCostTracking*`,\n `TestParserRecursionLimit`, and a large conformance suite. Verified by\n execution: evaluation against a 2000-deep nested map under `CostLimit`\n completes in ~10 microseconds and returns an error value for a missing key --\n no panic, no unbounded walk.\n- **Dependency footprint**: antlr (parser), protobuf + genproto (CEL's type\n model), `cel.dev/expr` (CEL spec types), yaml.v3. All mainstream, canonical,\n vendored widely by Google-adjacent projects. No runtime-impure third-party\n code; the REPL-only deps (readline, tview, tcell) are not in the library\n import graph.\n- **Security record**: no advisories on record for `cel.dev/cel-go` (GitHub\n Advisory Database).\n- **Module path**: migrated to `cel.dev/cel-go` (v0.32.0 is current); the old\n `cel.dev/cel-go/cel` path is the pre-migration identity. Pin `cel.dev/cel-go`.\n- **Binary size cost**: +13.0MB is the largest of the candidates (measured:\n baseline 2.34MB -\u003e 15.25MB with CEL). Accepted because: the daemon already\n embeds a large Yaegi runtime; CEL's type model is the load-bearing win\n (compile-time rejection of result-path typos, matching the reject-at-load\n rule); and the alternative bespoke grammar would re-implement CEL's checker to\n get the same guarantee. Flagged for sign-off: if 13MB is unacceptable for a\n specific deployment target, the fallback is a bespoke grammar with a\n re-implemented subset of this checker -- a strictly worse trade.\n\n## Why bespoke was not sufficient\n\nA bespoke grammar for the shipped cases (path read + comparison + bool\ncomposition) is small, but the load-time-validation requirement makes the\ncomparison: the reject-at-load rule requires _field-level_ static checking of\nresult paths against generated `Result` schemas, which is CEL's type-checker's\njob. Re-implementing that on a hand-rolled grammar means re-implementing a type\nsystem -- more code than the grammar itself, with worse diagnostics, no\nconformance suite, and no community surface for future expression needs (even\ndeclarative ones). The bespoke option only wins on binary size, and the size\ncost of CEL is measured and bounded.\n\n## Judgement calls (flagged for sign-off)\n\n- **J1 -- `id` required on referenced actions, unknown id = load failure.**\n Matches the reject-at-load rule. Alternative considered: an alternative is\n positional `actions[0]` access, which breaks on reorder; named is the\n operator-friendlier shape.\n- **J2 -- every `args` value is evaluated as a CEL expression.** No split\n between \"literal args\" and \"expression args\": a plain string value is a CEL\n string literal, a number is a CEL number literal, and any value that reads\n context data is written in CEL. One mechanism, no interpolation marker, no\n second syntax to learn. Alternative considered: cEL string literals must be\n quoted inside YAML (e.g. `message: '\"build finished\"'`), which is slightly\n noisy for mostly-literal args; accepted for uniformity -- the linter catches\n quoting mistakes at load.\n- **J3 -- condition failure semantics.** A condition that errors at runtime\n (missing/wrong-typed field) evaluates to **false**: the action is skipped,\n logged, run continues. This is the simplest candidate consistent with gap 1's\n bias against clever recovery; gap 1 (`archie-core-t2db.18`) commits\n stop-on-first-failure, so per-action control is not coming and J3 stands.\n Alternative considered: cEL separates false from error; we treat evaluation\n error as false + log.\n- **J4 -- `event` stays `dyn` (schema-by-example).** Load-time we cannot know\n event field types; declaring them typed would reject valid plays. Result reads\n get field-level checking via generated schema declarations. Alternative\n considered: when schema-by-example ships, tighten event field declarations\n from captured samples.\n- **J5 -- cost limit default.** A `CostLimit` at load time (e.g. 100,000) is\n applied to every expression; the linter and daemon use the same value so they\n agree. Alternative considered: exact number is a constant to tune at first\n use.\n- **J6 -- `args` failure semantics.** An `args` value that cannot be evaluated\n aborts its playbook run: the action and every later action in that playbook do\n not run, because data has no false value to fall back on and an action run\n without its arguments is a silent one. Other playbooks matching the same event\n still run, and the task is unaffected: a playbook run never changes routing.\n The daemon log reports the failure, naming the task, the playbook and the\n action; no task event is written. Alternative considered: mirroring J3,\n rejected because a predicate has a meaningful false and data does not.\n\n## Interaction with the execution-time gaps\n\nThis resolution does not resolve the execution-time gaps (both are resolved\nseparately: gap 1 by `archie-core-t2db.18`, gap 2 by the gap-2 decision).\nConditions make gap 1 concrete (condition-failure vs. action-failure semantics,\nJ3) and CEL's read-only result access makes gap 2's keying derivation readable\n(an idempotency key expression can read event fields) -- both flagged as the\ngaps' first real exercise. Gap 2's exercise is subsequently solved by that\ndecision: the answer is a fixed structural derivation, and idempotency keys are\ndeliberately NOT CEL expressions (see the \"No CEL in the key\" passage).\n\n## Implementation placement (for the follow-up slice)\n\n- New `internal/domain/eda/expr` package: builds the CEL environment (context\n declarations, generated `Result` schema types as CEL type declarations, cost\n limit), compiles/validates playbook expressions at load, evaluates them at\n dispatch. Domain layer; no infrastructure imports; owned by the playbook\n engine family. One new dependency: `cel.dev/cel-go` (pinned), `go.mod`\n otherwise unchanged.\n- Consumption: the playbook event coordinator (open question 4's loader)\n evaluates `when` before dispatch and evaluates `args` values before\n `Module.Invoke`; the lint tool (`archie-core-t2db.12`) shares the same\n compile/validate path for author-time diagnostics.\n- Schema integration: action kinds' generated `Result`/`Args` structs\n (`archie-core-t2db.13`'s `logextract` pattern) are declared to CEL at load so\n `actions.\u003cid\u003e.result.\u003cfield\u003e` type-checks; `args` expressions type-check\n against the target kind's `Args` schema before the playbook is accepted.\n" }, + { + "url": "/dev/prds/plugin-settings-live/", + "title": "Plugin settings reconcile live", + "section": "PRDs", + "body": "# Plugin settings reconcile live\n\n**Status:** Draft\n**Tracking:** GitHub [#1154](https://github.com/samcharles93/archie-core/issues/1154), epic [#998](https://github.com/samcharles93/archie-core/issues/998)\n**Authority:** `docs/prds/runtime-control-plane.md` decides each feature owns how\nits changes take effect; `docs/architecture/plugins-and-extensions.md` owns the\nplugin engine rule; `docs/prds/control-plane-apply-status.md` owns where apply\nreports go.\n\n## Outcome\n\nAn operator drops a plugin, module or secret-engine file into the configured\ndirectory while archied runs, and the daemon loads it without a restart. An\nedit to an installed file replaces what is running. Deleting an installed file\nnever stops running code; apply status says so. A stored change to\n`plugin-settings` applies live for additions instead of requiring a restart.\n\n## Triggers\n\n`plugin-settings` joins the kinds archied watches (`PluginSettingsKind` moves\nto the live list in `internal/app/archied/control_plane.go`). After the daemon\nlayers a new version it reconciles the three directories the document names.\nA second trigger needs no document change: a poller reconciles on an interval,\nso a file dropped into a directory loads on its own. The poll interval is the\napply-status restamp constant; no filesystem-watch dependency is added. The\npoller reads the current document's directories from the running config each\ntick, so a changed `plugin_dir`, `module_dir` or `secret_engine_dir` retargets\nreconciliation without a restart.\n\n## Rules\n\n- Adding and changing load; removing never unloads. Yaegi interpreters are not\n reclaimed while a caller still exists and there is no unload path; the\n running set can only grow.\n- Each file is loaded through a fresh interpreter (the boot pattern). A changed\n file therefore loads cleanly, and replacement switches consumers to the new\n interpretation; files are content-hashed so only new and changed files load.\n- A file that fails to load is skipped and reported; already-running code keeps\n running. This is the degrade-and-skip convention `plugin.LoadDir` and\n `Registry.LoadDir` already apply at boot. Boot keeps its reject-at-load rule\n for the module directory; a broken file at runtime skips without aborting.\n- A directory that does not exist reconciles to an empty set.\n\n## The three registries\n\n- **Secret engines** (`internal/secret`). Reconciliation re-runs\n `Registry.LoadDir` against the live directory. `Register` replaces by name,\n so an edited engine takes effect for the next resolution. A deleted engine\n stays registered: existing references keep resolving through it until\n restart, and the record says so. Engines are stateless resolvers, so a swap\n needs no lifecycle step.\n- **Modules** (`internal/domain/eda/module`): reconciliation re-runs\n `ModuleRegistry.Register` for the kind's file. The invoker maps\n last-write-wins, and playbook dispatch resolves through the registry at\n invoke time, so the swap reaches a running playbook store unchanged. A file\n deleted for a kind leaves the old invoker in place, running; reported.\n- **Plugins** (`internal/plugin`): `Host` grows running-state operations beside\n `Register`, `Start` and `Stop`. One adds a single module while the host is\n running: it is inserted in dependency order, started alone, and on failure\n rolled back alone, leaving every previously started module running. The\n other replaces a module whose manifest id matches: stop it, start the new\n one, and restart the old one when the new start fails. Replacing a module\n that another depends on is refused, not swapped. These operations obey the\n plugin engine rule: the host still invokes start and stop itself, isolates\n the failure to the one module, and keeps a rollback. A deleted file makes no\n host call: the module runs on and is reported.\n\n## `skills_dir`\n\n`skills_dir` is not part of this change. The daemon refuses a candidate\n`plugin-settings` document whose `skills_dir` differs from the value it is\nrunning, reporting that the field applies on restart, until a skill\nreconciliation exists to carry it. The refusal is the existing rejected-edit\napply-status shape: the store keeps the version, the document is not applied,\nlast-known-good keeps running.\n\n## Apply status\n\nOne record: `archied` on `plugin-settings`. The reconciliation outcome is\nreported against the version the process is running. The record's error field\ncarries the outstanding reconciliation problem: a deleted file still running, a\nfile that failed to load, or the directory failing to read. A reconciliation\nwith nothing outstanding clears the error. The version stays the version the\nprocess is running; an add that succeeded is live even while a removal is\noutstanding, and the error says so rather than showing the kind as\nrestart-required. archie-gateway keeps boot-time loading; its record simply\nstays on the version it last applied until it restarts, which the per-process\napply-status page already reads correctly." + }, { "url": "/dev/prds/pr-review-agent/", "title": "PR review agent", @@ -851,6 +857,10 @@ "title": "Playbook data-flow and condition syntax", "url": "/dev/prds/playbook-expression-syntax/" }, + { + "title": "Plugin settings reconcile live", + "url": "/dev/prds/plugin-settings-live/" + }, { "title": "PR review agent", "url": "/dev/prds/pr-review-agent/" diff --git a/docs/prds/plugin-settings-live.md b/docs/prds/plugin-settings-live.md new file mode 100644 index 000000000..07644e398 --- /dev/null +++ b/docs/prds/plugin-settings-live.md @@ -0,0 +1,88 @@ +# Plugin settings reconcile live + +**Status:** Draft +**Tracking:** GitHub [#1154](https://github.com/samcharles93/archie-core/issues/1154), epic [#998](https://github.com/samcharles93/archie-core/issues/998) +**Authority:** `docs/prds/runtime-control-plane.md` decides each feature owns how +its changes take effect; `docs/architecture/plugins-and-extensions.md` owns the +plugin engine rule; `docs/prds/control-plane-apply-status.md` owns where apply +reports go. + +## Outcome + +An operator drops a plugin, module or secret-engine file into the configured +directory while archied runs, and the daemon loads it without a restart. An +edit to an installed file replaces what is running. Deleting an installed file +never stops running code; apply status says so. A stored change to +`plugin-settings` applies live for additions instead of requiring a restart. + +## Triggers + +`plugin-settings` joins the kinds archied watches (`PluginSettingsKind` moves +to the live list in `internal/app/archied/control_plane.go`). After the daemon +layers a new version it reconciles the three directories the document names. +A second trigger needs no document change: a poller reconciles on an interval, +so a file dropped into a directory loads on its own. The poll interval is the +apply-status restamp constant; no filesystem-watch dependency is added. The +poller reads the current document's directories from the running config each +tick, so a changed `plugin_dir`, `module_dir` or `secret_engine_dir` retargets +reconciliation without a restart. + +## Rules + +- Adding and changing load; removing never unloads. Yaegi interpreters are not + reclaimed while a caller still exists and there is no unload path; the + running set can only grow. +- Each file is loaded through a fresh interpreter (the boot pattern). A changed + file therefore loads cleanly, and replacement switches consumers to the new + interpretation; files are content-hashed so only new and changed files load. +- A file that fails to load is skipped and reported; already-running code keeps + running. This is the degrade-and-skip convention `plugin.LoadDir` and + `Registry.LoadDir` already apply at boot. Boot keeps its reject-at-load rule + for the module directory; a broken file at runtime skips without aborting. +- A directory that does not exist reconciles to an empty set. + +## The three registries + +- **Secret engines** (`internal/secret`). Reconciliation re-runs + `Registry.LoadDir` against the live directory. `Register` replaces by name, + so an edited engine takes effect for the next resolution. A deleted engine + stays registered: existing references keep resolving through it until + restart, and the record says so. Engines are stateless resolvers, so a swap + needs no lifecycle step. +- **Modules** (`internal/domain/eda/module`): reconciliation re-runs + `ModuleRegistry.Register` for the kind's file. The invoker maps + last-write-wins, and playbook dispatch resolves through the registry at + invoke time, so the swap reaches a running playbook store unchanged. A file + deleted for a kind leaves the old invoker in place, running; reported. +- **Plugins** (`internal/plugin`): `Host` grows running-state operations beside + `Register`, `Start` and `Stop`. One adds a single module while the host is + running: it is inserted in dependency order, started alone, and on failure + rolled back alone, leaving every previously started module running. The + other replaces a module whose manifest id matches: stop it, start the new + one, and restart the old one when the new start fails. Replacing a module + that another depends on is refused, not swapped. These operations obey the + plugin engine rule: the host still invokes start and stop itself, isolates + the failure to the one module, and keeps a rollback. A deleted file makes no + host call: the module runs on and is reported. + +## `skills_dir` + +`skills_dir` is not part of this change. The daemon refuses a candidate +`plugin-settings` document whose `skills_dir` differs from the value it is +running, reporting that the field applies on restart, until a skill +reconciliation exists to carry it. The refusal is the existing rejected-edit +apply-status shape: the store keeps the version, the document is not applied, +last-known-good keeps running. + +## Apply status + +One record: `archied` on `plugin-settings`. The reconciliation outcome is +reported against the version the process is running. The record's error field +carries the outstanding reconciliation problem: a deleted file still running, a +file that failed to load, or the directory failing to read. A reconciliation +with nothing outstanding clears the error. The version stays the version the +process is running; an add that succeeded is live even while a removal is +outstanding, and the error says so rather than showing the kind as +restart-required. archie-gateway keeps boot-time loading; its record simply +stays on the version it last applied until it restarts, which the per-process +apply-status page already reads correctly. \ No newline at end of file