Files
archie/docs/prds/eda-playbook-engine.md
T
sam a3f82753fc docs(prds): resolve execution-time gap 2 -- idempotency keying for playbook actions (#715)
Decision for channel/forge/module action keying in eda-playbook-engine.md:
one durable dedup record per (action, event) pair via a playbook_dispatches
ledger in internal/store, reusing binding_dispatches' INSERT OR IGNORE +
sentinel conventions, with record-before-invoke ordering (at-most-once, the
reverse of the binding loop's ordering, justified by the external-side-effect
harm profile). Key = (playbook_id, playbook_version, event_id, action_id):
event_id is the TaskEnvelope.IdempotencyKey() value already carried on
DispatchInput.TaskID; the key is a fixed structural composition, deliberately
not a CEL expression (event stays dyn; determinism under redelivery is the
binding constraint). Redelivery of a recorded action skips it and continues
the run at debug level, never surfaced as an error. Rows live as long as the
playbook exists, no time-based expiry.

Extends event-sources-and-reactions.md's contract without contradicting it:
PublishUnique's dedup is publish-path only (Nats-Msg-Id + 2-minute
DedupWindow), so consumer redelivery of side-effecting reactions was never
covered; the ledger adds exactly that consumer-side piece, reusing the
TaskEnvelope identity and the typed-family shape. The gap-2 prerequisites
(Playbook ID/Version, DispatchInput.TaskID) already landed in 1e736ef.
Documentation only: no code changes. Keying implementation is separate
follow-up work.

Refs: archie-core-t2db.17
2026-09-05 10:11:59 +10:00

957 lines
54 KiB
Markdown

# EDA playbook engine -- design
**Status:** Draft, awaiting sign-off (not yet in `docs/architecture/`)
**Date:** 2026-09-03
## Problem
Today, "which workflow runs for which trigger" is hardcoded Go: a literal
label vocabulary (`workintake.labelKinds`) and a literal label->workflow map
(`workflow/routing.go`), extended only by adding a Go case or shipping a
skill whose `metadata.archie.workflow` overrides one *named* built-in
workflow. Extending the system to a new trigger, a new action, or a new
event source today means a feature branch, a code change in a hardcoded
path, and new Go tests -- for every addition. That is the barrier this
document exists to remove.
The goal (Sam, 2026-09-02): triggers, the actions they run, and the
conditions that connect them should be **drop-in and load**, not **drop-in
and touch application code**. Yaegi-evaluated Go is the unit of imperative
"do a thing" logic; YAML playbooks are pure orchestration data -- no logic,
just "on this event, with this data, call these actions, in this order,
under these conditions." The daemon's event coordinator doesn't know what a
`bugfix` or `image-gen` action does, it resolves the name against a
registry and pipes typed data through.
This is explicitly **not** scoped to code-generation workflows. A playbook
action can be any registered capability -- run a codegen workflow, send a
message on a channel, open something on a forge, generate an image -- so
long as it's implemented behind one of the typed engine families below.
## Relationship to existing decisions
- `docs/prds/event-sources-and-reactions.md` (2026-08-22, decided) ruled
that reactions in the two epics it covers are **producer-only** (no veto,
no mutation of in-flight work) and explicitly rejected a single generic
`Source` interface as "the untyped-hook shape the plugin engine rule
exists to prevent." **This document extends that scope**: playbooks can
dispatch to multiple typed action kinds, not just "produce a
`TaskEnvelope`." It does not relitigate producer-only-ness (still no
veto/mutation of in-flight work here) and it does not introduce one
generic interface -- see "Typed positions, not one generic hook" below,
which is how it stays compliant with the plugin engine rule instead of
reopening that document's #4.
- `ARCHITECTURE.md#plugin-engine-rule-strict` governs every extension point
introduced here. Every position below is a typed domain contract with an
owning registry, not a generic callback map. This is a hard constraint,
confirmed against the existing rule text before writing this doc.
- Yaegi is already a proven in-process extension mechanism
(`internal/plugin`, `internal/domain/workflow/wfeval/yaegi.go`,
`internal/gate/gateeval/yaegi.go`, `internal/skill/plugin.go`) -- this
design generalizes its existing role, it does not introduce a new
execution mechanism.
- `internal/domain/workflow/skillbuild.BuildRegistry` already composes a
`workflow.Registry` from a skills catalog, plugin-defined overriding
built-in. The playbook loader described here supersedes label routing
specifically; it does not replace `skillbuild`'s workflow-composition role,
it becomes an additional input the daemon composes at startup.
- Webhook event intake is separately in flight (Sam, in progress). This
document defines the router each webhook-sourced event should be able to
reach; it does not redesign webhook receipt itself.
## Shape: four logic positions, each a typed engine family
Every entrypoint kind (mechanism, action, loop, schedule, route, trigger --
Sam's list) is itself a schema emitter: defining a new kind produces the
generated contract downstream tooling consumes. The kinds slot into a fixed
hierarchy of logic positions:
```
Workflow -> Module -> Channel (telegram/whatsapp/email/...)
-> Forge (GitHub/Gitea/GitLab/...)
```
Each position is its own typed engine family per the plugin engine rule:
identity + capability-specific operations, an owning `Registry`/`Manager`,
explicit lifecycle where the implementation owns resources, narrow host
access. Concretely:
- **Workflow** -- already exists (`internal/domain/workflow`). A playbook
action of kind `workflow` invokes a registered `workflow.Registry` entry
by name, unchanged from today's execution path.
- **Module** -- the new general-purpose action position. A `Module` is a
Yaegi-evaluated implementation of a generated interface (args in, typed
result out) for one declared action kind (e.g. `image-gen`, `notify`,
`open-pr-comment`). This is the position that makes actions "not
restricted to code-generation." Full design (schema-per-kind, the
registry shape, the trust boundary, the recommended first slice):
`docs/prds/module-position.md`.
- **Channel** -- already exists as `channels.Channel` (telegram, email,
webhook). A playbook action of kind `channel` sends through a named,
already-registered channel. No new mechanism; this document just makes it
addressable from a playbook.
- **Forge** -- forge clients (GitHub, Gitea, GitLab) become an addressable
action position the same way Channel is, so a playbook can say "comment
on this forge issue" without that being a special case wired only into
the poller.
**Typed positions, not one generic hook.** There is no single `Action`
interface with an `any` payload. Each position keeps its own typed
contract; the playbook YAML names a position and an implementation, and the
event coordinator resolves that pair against the matching typed registry.
This is what keeps the design compliant with the plugin engine rule and
with `event-sources-and-reactions.md`'s rejection of a generic `Source`/
hook interface -- the generality lives in the YAML data model (which
positions exist, which named implementations are registered), not in a
runtime `any`-typed dispatch path.
## `go:generate`-driven schemas, from day one
Sam's correction: schemas are not a future nicety bolted onto an ad hoc
YAML shape -- they are the struct+interface contracts for each logic
position, and they drive code generation from the start.
- Each logic position (Workflow/Module/Channel/Forge) has a schema source
(Go-annotated types, following the existing pattern elsewhere in the
repo) from which `go:generate` emits the interface and struct scaffolding
an implementation must satisfy.
- A Yaegi module implements the generated interface for its position. The
generated contract is what makes an implementation valid, not hand-review.
- The schema is also what the linter and LSP (below) validate a playbook
YAML against -- one source of truth for codegen, startup loading, and
authoring-time tooling, so the three can't drift from each other.
- Schema versioning is explicit per position, independent of a playbook's
own workflow version (see collision handling below).
This document does not attempt to design the full generator today. It
records the constraint (schema-first, `go:generate`-driven, per logic
position) so the first implementation slice builds toward it instead of
away from it.
## Playbook YAML: pure orchestration data
A playbook declares no embedded logic -- only:
- **trigger**: the event predicate (e.g. forge issue assigned + labelled
`bugfix`, a schedule tick, a webhook payload shape).
- **actions**: an ordered list, each naming a logic position and a
registered implementation, with typed argument data and optional
conditions.
- **data flow between actions**: results/facts from one action available to
later actions in the same playbook run -- exact interpolation syntax is
an open point, not decided here (see Open questions).
Playbooks live in a configured directory (repo/directory-scoped, per Sam's
"define repos, directories, playbooks" framing) and are loaded at archied
startup, alongside the existing skills-catalog composition.
## Dedup: detect and log, no arbitration mechanism
No runtime auto-resolution rule. An automatic "pick a winner" algorithm
(e.g. by version) is itself a mechanism that can fail and need debugging
when a collision happens -- the wrong kind of complexity to introduce for
a case that should just be visible and fixed by a human. Instead:
- **At daemon startup:** if independently-sourced playbook directories
declare a collision on the same trigger, that is a definition failure.
The colliding definitions are dropped (not loaded) and the failure is
logged plainly (which sources, which trigger) and reported back to the
caller that requested the load -- not swallowed as a background log line
only. The schema is the definition; a schema conflict is a reported
failure, not something the daemon silently arbitrates.
- **Linter**: a CI/dev-time tool, reading the same generated schemas, that
flags duplicate or conflicting trigger definitions *within* a single
maintained tree before merge.
**Status 2026-09-03: lint mode shipped (t2db.12).** A standalone
gopls-shaped binary (`cmd/archie-playbooks`, lint subcommand) validates
playbook directories against the exact loaders the daemon uses
(`LoadPlaybookDirs`/`LoadKindWorkflowsYAML`/`LoadLabelWorkflowsYAML`),
exiting non-zero on any collision / malformed file / invalid binding.
Findings are file-granular, not line-granular: the loader decodes with
`yaml.Unmarshal` into a plain map, which discards line numbers. A
compiler-style file:line diagnostic needs a `yaml.Node` decoding
upgrade, tracked separately -- the linter agrees with runtime
validation by construction, which is the load-bearing property.
Discoverable via `task lint:playbooks` or direct `go run
./cmd/archie-playbooks lint -dir ...`. The LSP/serve mode is a later
entrypoint of the SAME binary, per the shared-package decision above.
- **LSP**: same schema source again, feeding author-time hover/completion/
inline errors while someone edits a playbook YAML in an editor. Kept
explicitly in scope per Sam (not deferred), because it's the same schema
artifact as the linter's, just a different consumer.
The startup log-and-refuse path exists for what the linter/LSP structurally
cannot see -- sources composed together only at daemon startup, never
checked against each other by either tool. If the linter and LSP do the job
they're built for, a real collision reaching that startup path should be
rare in practice, not a routine occurrence the design leans on.
**Shared package, thin consumers -- not a bundled-vs-standalone choice.**
The validation logic (`LoadKindWorkflowsYAML`, `LoadLabelWorkflowsYAML`, the
directory-merge from t2db.11) lives in `internal/domain/workflow`, a plain
Go package with no knowledge of which binary calls it. This makes "should
the linter/LSP be bundled into archied or a separate tool" a non-question:
- `archied` already imports the package directly and validates in-process
at startup -- no subprocess, no external tool call.
- An agent that needs to validate a playbook in-process can import the same
package directly, for the same reason.
- The standalone `cmd/` binary (t2db.12, gopls-shaped: one binary, a lint
mode now, an LSP/serve mode later) exists only because CI and editors are
out-of-process consumers by nature -- a CI step needs an exit code, an
editor speaks LSP over a process boundary, neither can `import` a Go
package. It is a thin wrapper per `organisation.md`'s `cmd/` rule, not a
second implementation.
One shared implementation, two thin consumer shapes (in-process and
out-of-process). Folding CLI/LSP protocol handling into `archied` itself
would grow the daemon into unrelated concerns, against the plugin-engine
rule's spirit of narrow capability families -- so the answer is not
"bundle or don't," it's "don't duplicate the logic, and pick the thinnest
consumer for each caller's actual constraint."
## Execution-time gaps (identified 2026-09-03; both resolved 2026-09-05)
Everything above covers *loading* a playbook. Nothing above covers what
happens while one *runs*. Two gaps, both load-bearing enough to resolve
before implementation, not defer:
### 1. Mid-playbook action failure
**Status 2026-09-05: resolved by archie-core-t2db.18.** The default below
is a committed decision, not a candidate list; changing it later would
require its own decision. Gap 2 below is resolved by `archie-core-t2db.17`.
A playbook is an ordered action list with data flowing between steps. If
action 2 of 4 fails at runtime (an image-gen API times out, a forge call
403s), the doc previously said nothing about what happens next. This
codebase already answers a structurally identical question for its
existing workflow engine, precisely enough to reuse rather than
re-derive: `workflow.Run` (`internal/domain/workflow/workflow.go:294`)
draws a three-way distinction on a stage failure, and a playbook run
draws the same one on an action failure.
**Decision: reuse `Run`'s three-way distinction directly.** The stated
default is **stop-on-first-failure**: a real action error halts the
playbook run immediately and is reported, with no continue-to-next-action
and no automatic retry or compensating follow-up. The three points below
are the complete decision; reasoning and evidence follow each.
1. **A real action error** (the action's own logic failed -- an API
error, a validation failure inside the action) stops the run
immediately. No later actions execute. The failure is reported the
same way a load collision is reported -- dropped, logged plainly
(which action and why), and returned to the caller, never swallowed
as a background log line only (this document's Dedup section, lines
155-159, the t2db.9-12 drop-and-report rule). `Run` is the mechanical
precedent for the stop itself: a stage error sets the park reason and
returns without running further stages (`workflow.go:324-326`). This
is stop-on-first-failure, matching the document's existing bias
against clever automatic recovery, and it is now a stated decision
rather than a nobody-chose default.
2. **Interruption** (the daemon is shutting down mid-dispatch, `ctx.Err()
!= nil`) is explicitly **not** a failure. `Run` already treats this
case specially -- "Daemon shutdown is not a workflow failure. Leave
the task running ...; parking here would publish a false failure and
require manual intervention" (`workflow.go:317-323`) -- and the same
reasoning applies to a playbook run: an in-flight dispatch interrupted
by restart must not be recorded as a broken playbook. What "leave it
alone for restart" means for a playbook run (there is no `store.Task`
row a playbook run inherently owns the way a workflow stage does) is
an open implementation detail for whoever builds the coordinator, not
decided here -- but the *semantic* (interruption ≠ failure) is decided.
3. **A run that produces no outcome at all** (every action ran without
error, but nothing was assigned to happen) is itself an error
condition, matching `Run`'s own "a workflow must end with an explicit
outcome; not doing so is a definition bug, which still must not vanish
silently" (`workflow.go:336`). A playbook with actions that all ran
but never reached a coherent terminal state is a definition bug in the
playbook, reported as such.
**Rollback:** explicitly **not attempted**, for exactly the reason the
gap's own description named -- a posted forge comment is not revocable.
This is not a limitation to work around later; it is the correct
semantic. An operator who needs a corrective action after a failure
writes a *new* playbook/action for that, they do not get automatic
undo.
**Producer-only interaction, stated explicitly -- why this decision does
not violate the rule.** `event-sources-and-reactions.md`'s producer-only
rule is "no veto, no mutation of in-flight work"
(`event-sources-and-reactions.md:61-63`), and it explicitly routes any
reaction that "can veto or mutate an *in-flight* task (block a PR from
opening, alter a running stage)" away from this epic to the
adversarial-self-review one (`event-sources-and-reactions.md:78-81`).
This decision touches neither half of that boundary:
- **The rule constrains what an event source may do to *someone else's*
in-flight work.** Stop-on-first-failure is the playbook run deciding
its *own* terminal state from its own action's error. It vetoes
nothing: no task, stage, or PR that was already in flight is blocked
or altered, and the run only declines to start actions that had not
started yet. A run deciding its own continuation is not veto or
mutation of another producer's work; it is simply how the run ends.
- **Everything after the stop is still producer-only.** The failed run is
reported to its caller per the collision precedent, the same way `Run`
records the failure on its own unit -- park reason set and the task
transitioned to `StatusParked` (`workflow.go:324-326`, `park` at
`workflow.go:351-357`) -- never on someone else's in-flight work. A
future compensating action (e.g. "if the notify action failed, also
log an incident") would be **producing new work** -- new events, new
dispatches -- which is exactly the already-permitted shape
(`event-sources-and-reactions.md:73-76`). No playbook syntax for "on
failure, also run X" is designed here; this resolution only confirms
that if that syntax is added later, it stays inside the producer-only
boundary rather than requiring a new exception.
The boundary therefore holds on both sides: the failure default never
hands an event source veto or mutation power over a playbook run, and
anything a run does after stopping is itself still just producing work.
**J3 relationship, stated explicitly:** J3 (a CEL `when` condition
erroring at eval time → treated as false → skip that action → run
continues) is a **different case from an action failure** and is not
revisited by this resolution. A condition error means "we couldn't
determine whether to run this action," which this document already
resolved as "then don't, and move on." An action failure means "we tried
to run this action and it broke," which stops the run per point 1 above.
The two must not be conflated: a coordinator implementation should be
able to point at one `when`-evaluation code path and one
action-invocation code path and show they use different outcomes.
### 2. Idempotency at execution time
**Status 2026-09-05: resolved** (per bead `archie-core-t2db.17`, a
design-investigation ticket: resolve, do not implement). The decision
below is the answer; the keying scheme itself is separate follow-up work
and is intentionally not built here. It **extends**
`event-sources-and-reactions.md`'s existing contract -- it does not
contradict it (relationship argued at the end).
**Problem (unchanged):** `event-sources-and-reactions.md` decided
reactions ride `internal/eventbus`'s at-least-once delivery specifically
so `PublishUnique`/idempotency keys absorb redelivery without duplicate
side effects. This document inherits that for free only for
`workflow`-kind actions, because `TaskEnvelope` already carries an
idempotency key. It said nothing for `channel`, `forge`, or `module`
actions with real side effects -- a posted comment, a sent message, a
billed image generation.
One correction to the problem statement, checked against the code rather
than assumed, because it changes what the answer must be:
**`PublishUnique`'s dedup is not a consumer-redelivery guard at all.**
`PublishUnique` only sets the JetStream `Nats-Msg-Id` header on the
outgoing message (`internal/infrastructure/eventbus/nats/publisher.go:18-23`;
key constant `message.go:11-13`); JetStream suppresses *republished*
messages carrying a repeated key inside `Config.DedupWindow`
(`client.go:67`, `Duplicates: cfg.DedupWindow`), default 2 minutes
(`config.go:15`, `DefaultDedupWindow = 2 * time.Minute`). A consumer
redelivery is the same *stored* message re-delivered on Nak or
acknowledge-timeout (`internal/eventbus/eventbus.go:64-65`: "a handler
returning an error causes redelivery") -- not a republish; its
`Nats-Msg-Id` is never re-firewalled, and even if it were, two minutes is
not an at-most-once guarantee for a side effect that must never repeat.
**Conclusion: side-effecting action kinds need a consumer-side, durable
dedup record; the bus cannot supply it.** That is precisely the situation
`binding_dispatches` was built for -- so the decision below reuses its
mechanism rather than designing a new one.
#### Decision
**Granularity: per-action-per-event -- one dedup record per
(action, event) pair, not per playbook run.** This supersedes the
earlier draft's whole-run default. Whole-run keying has two crash
windows: the run row is written when the run starts, so a crash mid-run
leaves the run "started" and a redelivery skips *every remaining
action* (work silently lost); a crash before the row write re-runs
*every already-fired action* (duplicate side effects -- the exact harm
this gap exists to prevent). Per-action keys collapse both into one
window and one behavior: an action's record is written immediately
before that action's side effect is invoked, so an action fires iff its
own record is absent, no fired action ever re-fires, and a redelivered
event resumes at the first unrecorded action. For today's shipped
single-action playbooks the two granularities are equivalent in effect;
the difference starts with multi-action runs -- where whole-run keying
would already be wrong.
**Storage/lookup: a new durable ledger table `playbook_dispatches` in
`internal/store`, copying `binding_dispatches`'s conventions exactly**
(durable table, `INSERT OR IGNORE`, sentinel error, rows freed with
their parent, survives daemon restarts -- `internal/store/bindings.go:31-37`,
`355-382`):
```sql
CREATE TABLE playbook_dispatches (
playbook_id TEXT NOT NULL,
playbook_version TEXT NOT NULL,
event_id TEXT NOT NULL,
action_id TEXT NOT NULL,
dispatched_at TEXT NOT NULL,
PRIMARY KEY (playbook_id, playbook_version, event_id, action_id)
);
```
Written with the same `INSERT OR IGNORE` + sentinel-error convention
(`ErrAlreadyDispatched`, `bindings.go:59-63`), so a duplicate is a no-op
write rather than a constraint error, and the caller matches it with
`errors.Is` the way the binding dispatch loop already does
(`internal/daemon/daemon.go:591-597` -- "another cycle already won this
race," not an error to surface). The record is consumed by the playbook
coordinator at dispatch time through a domain-side interface implemented
by `internal/store` -- the same shape as `BindingDispatcher`
(`internal/store/interface.go:123-139`) -- and the table lands in the
store's schema application (`internal/store/store.go:137-139`). No change
to `internal/eventbus`; no per-message dedup state on the client.
**Lifetime: no time-based expiry.** Rows live as long as their playbook
exists; when a playbook is removed from the configured directories, the
coordinator deletes its rows -- mirroring binding-dispatches-rows-deleted-
with-binding (`bindings.go:204-214`, same transaction as the delete). The
at-most-once guarantee must not silently decay, which is exactly the
property that disqualifies `DedupWindow` above; table growth is bounded
by distinct (playbook, event, action) pairs over the playbook's lifetime
and is accepted, the same tradeoff `binding_dispatches` already makes.
**Redelivery behavior: skip the recorded action and continue the run.**
A detected duplicate is not re-invoked, is not reported to the caller as
an error, and is logged at debug level rather than as a warning -- this
is the expected, correct behavior of at-least-once delivery, not an
anomaly (same treatment as `ErrAlreadyDispatched` at
`daemon.go:592-597`).
The run continues with the next action (or ends, for a single-action
playbook whose one action is recorded), which is what makes a redelivery
a resume rather than a dead restart.
**Key derivation: a fixed composition of four structural identities,
computed by the coordinator -- not a CEL expression.** The key is the
`(playbook_id, playbook_version, event_id, action_id)` tuple above, where:
- `playbook_id` is `Playbook.ID` -- the playbook file's path relative to
its configured directory root, slash-normalized, unique within the load
composition by construction (`internal/domain/eda/playbook/playbook.go:39-45`,
`149-155`). Note this prerequisite already landed in code: the earlier
draft's "not yet decided in code" paragraphs are stale --
`edd689e` (`feat(eda): thread Playbook ID/Version and DispatchInput task
identity (gap-2 prereqs)`) shipped `Playbook.ID`/`Version` and
`DispatchInput.TaskID`.
- `playbook_version` is `Playbook.Version` -- a SHA-256 of the loaded
file, recomputed on every load (`playbook.go:46-49`, `154-155`),
mirroring `Binding.Version`'s provenance-pinning purpose: the dispatch
is pinned to the exact definition that was active when it fired. A
changed playbook definition is a different key and may fire again for
the same event -- deliberate, not a bug.
- `event_id` is `DispatchInput.TaskID` (`playbook.go:214-222`), which
carries the `TaskEnvelope.IdempotencyKey()` value
(`"archie:" + owner/repo/number`,
`internal/domain/workintake/envelope.go:105-107`). It is chosen over
`store.Task.ID` because the trigger vocabulary is issue-level
workintake label/kind, dispatched at discovery time (pollNATS, forge
webhook receiver), where no task row exists yet (`playbook.go:214-222`
comment; `playbook_test.go:338-345`: "the value available at the
discovery/dispatch point... NOT a store.Task.ID int64"). Keying on the
issue identity rather than the delivery source is the load-bearing
property the existing contract already established for `TaskEnvelope`
(`event-sources-and-reactions.md` question 4: a webhook-sourced
envelope for the same issue produces the *same* idempotency key as a
poll-discovered one; `internal/forge/webhook/receiver.go:14-20` --
"Idempotency is not this package's job"); this resolution inherits that
property rather than re-deciding it. Consequence, stated explicitly:
same-issue re-discoveries (a re-label that re-triggers the same kind)
are the *same event* by design -- the workintake trigger vocabulary is
issue-granular, exactly as `TaskEnvelope`'s own key is. A future
capture/binding-originated trigger type keys on `captured_events.id`
instead; derivation is per-trigger-type, not one formula for all
triggers (rule carried over unchanged from the draft).
- `action_id` is the action's declared `id` -- J1's named ids, "a
required field on every action that later actions reference", unique
and stable-identifier-shaped, validated at load (the CEL resolution's
J1 and its context table above in this document). For an action declaring no `id`
(permitted by J1 while unreferenced), the key uses the action's 1-based
position in the playbook: deterministic and stable for today's
single-action playbooks, and a later reorder is a file edit whose
content hash changes `playbook_version` with it, so a positional
fallback cannot silently collide with a prior definition.
**No CEL in the key -- stated explicitly because the CEL resolution
flagged this as gap 2's "first real exercise"** ("CEL's read-only result
access makes gap 2's keying derivation readable" -- the CEL resolution's
closing note). Readability
is not the binding constraint; **determinism under redelivery is**, and
CEL key expressions lose three ways. First, they are not load-checkable:
`event` stays `dyn` by the resolved J4 (schema-by-example), so
a key expression over event fields cannot be type-checked at load,
violating the document's reject-at-load philosophy. Second, they can
fail at dispatch (a missing field is a CEL error), and a key that cannot
be computed is an unfireable state -- neither "skip" nor "fire" is safe
without losing or duplicating work. Third, the contract needs one
derivation, and two authors writing different key expressions would
fragment the ledger. The key is therefore structural by construction:
every component above is an immutable identity of the run's inputs, so
two deliveries of the same event derive the same key with no evaluation.
The exercise's actual outcome is the derivation rule above; CEL remains
a *condition* surface (`when`), never a key surface. { Call: if a future
action kind genuinely needs finer granularity than (action, event) --
e.g. "per comment, not per issue" -- that is the documented exception
path, and it arrives with its own trigger type and its own event
identity, per the per-trigger-type rule. }
**Record-before-invoke -- one deliberate divergence from
`binding_dispatches`, justified by the harm profile.** The binding
dispatch loop enqueues the task first and records the ledger row after
(`daemon.go:585-599`), accepting that racing cycles may both enqueue
("the duplicate task remains queued... delete-on-races is its own can of
worms"); the selection query excludes already-dispatched captures
(`bindings.go:395-399`), but the race window remains. That ordering is
correct there because the duplicate artifact is an internal task row --
recoverable, low-harm. The playbook ledger's ordering is reversed on
purpose: **the row is written and committed before the action's side
effect is invoked** -- the `INSERT OR IGNORE` is the atomic gate, and
the lost-write loser skips without invoking. The cost is stated, not
hidden: a crash between the row commit and the invoke loses that one
action's side effect, repairable by hand -- the reverse direction of the
same window that would otherwise duplicate a posted comment or
re-bill an image generation. For side effects whose duplication is
externally visible and not revocable, at-most-once is the required
direction; the binding system's at-least-once-tolerant ordering is not
the precedent for this ledger.
**Gap-1 interplay, stated to keep the two mechanisms separate:** the
ledger answers one question per action -- has this action already fired
for this event? It does not answer whether the run continues, stops, or
is re-attempted; that is gap 1's coordinator concern (a run-outcome
record; gap 1 explicitly leaves "what leave it alone for restart means
for a playbook run" to whoever builds the coordinator). One consequence
of record-before: an action that FAILED (gap 1's real-action-error case)
has a ledger row, so a later redelivery of the same event skips it --
consistent with gap 1's "no automatic retry" resolve (a corrective
action is a *new* playbook/action, per gap 1's rollback paragraph). A
failed action is never re-attempted by the bus; if a run must be
recovered, an operator edits the playbook -- new version, new key --
rather than fighting ledger state.
**Relationship to `event-sources-and-reactions.md` -- extension, not
contradiction, argued explicitly.** Three points, keyed to that
document's sections:
1. Its decision 2 commits reactions to at-least-once delivery via
`PublishUnique`'s "without duplicate enqueue" guarantee -- which, with
the code facts above, is accurate for the *publish* path only:
`PublishUnique` absorbs duplicate publishes of the same event (poll
vs. webhook), which is what that decision was actually about. This
document's own problem statement, taken literally, overstates it
("keys absorb redelivery without duplicate side effects") -- that
overstatement is exactly what this gap exists to correct. The
existing contract never claimed consumer redeliveries of
side-effecting reactions were absorbed, so nothing it says contradicts
a consumer-side ledger; this resolution pins down the boundary the
sentence left implicit (publish-path dedup vs. consumer-path dedup)
rather than changing either.
2. Its decision 3 (producer-only, no veto/mutation of in-flight work)
is untouched: the ledger is the running reaction's own bookkeeping
about its own actions -- it suppresses re-firing an action the
coordinator already ran for the same event; it blocks, mutates, or
reorders nothing else, and no other producer's work is touched.
3. Its decision 4 (no generic `Source` interface; `TaskEnvelope` is the
typed contract, keyed on issue identity, not delivery source): the
event half of the playbook key IS that same identity
(`TaskEnvelope.IdempotencyKey()`), and the per-trigger-type rule
means any future trigger type brings its own identity rather than a
new generic one -- the playbook ledger introduces no second identity
vocabulary and no untyped payload. The only new type in this
resolution is the ledger's own value object, consumed through a
narrow interface like `BindingDispatcher` -- the same shape the
document's "typed families" rule (decision 1, plugin engine rule)
already blesses.
**Scope: this is the `channel`/`forge`/`module` decision, not a
`workflow`-position change.** Workflow-kind actions keep their existing
keying claim unchanged -- the `TaskEnvelope` path that this document
already inherits for free (`daemon.go:858-869`, `PublishTask` publishes
with `task.IdempotencyKey()`), with its cross-poll/webhook property
untouched. The ledger gate applies to the side-effecting positions
(channel, forge, module), which is why the closing statement below
still holds: the first slice (Module + loader + workflow-kind dispatch)
was buildable without this resolution, and channel/forge/multi-action
playbooks were not. A coordinator MAY implement one position-uniform
gate as a simplification; what this resolution requires is only that
the side-effecting positions are gated.
Both execution-time gaps are now resolved (gap 1 by
`archie-core-t2db.18` above, gap 2 by `archie-core-t2db.17` above). For
the record, the first slice (Module + loader + workflow-kind dispatch)
never needed gap 2: workflow-kind actions had idempotency for free via
`TaskEnvelope`, and the ledger is only required once side-effecting
positions dispatch. Channel/Forge actions and multi-action playbooks
are unblocked on the execution-time questions.
## Open questions (for sign-off before implementation)
1. **Data-flow/condition syntax.** Does this project want a small existing
Go expression evaluator (there may be one already in the codebase worth
checking, e.g. anything backing gate conditions), or a bespoke minimal
grammar? Not decided here.
**Status 2026-09-03: resolved.** CEL (cel.dev/cel-go) is the single
mechanism for both halves. The resolution below replaces this entry
and is not an open question. Judgment calls are flagged inline.
## Data-flow and condition syntax -- decision (resolves open question 1)
**Status:** Draft, awaiting sign-off (not yet in `docs/architecture/`)
**Date:** 2026-09-03
**Decision:** CEL via `cel.dev/cel-go` (v0.32.0), pinned; one
expression mechanism covers both an action's `when` condition and its
`args` values.
### The requirement, stated directly
A playbook action is gated and parametrised by runtime data: (a) a
condition deciding whether the action runs; (b) a later action's `args`
values referencing an earlier action's `Result` and the triggering
event's fields. Both are **CEL expressions**, evaluated against one
evaluation context. There is no separate interpolation syntax: CEL
reads nested maps and fields natively, so `args` values that need data
are written as CEL, and only plain literal args remain plain YAML
scalars.
Both halves are read-only: expressions may read context values and
compute a result from them, never mutate, never call host functions,
never perform I/O. CEL enforces this structurally (no side effects,
linear evaluation when macros are bounded, cost-limited).
```yaml
actions:
- position: module
kind: log
args:
message: '"build finished"' # literal
when: 'event.label == "bugfix"' # gate on event field
- position: module
kind: notify
args:
message: '"priority " + string(event.priority)'
when: 'actions.notify.result.delivered == true'
```
### Trust boundary of the expression
A playbook YAML is **operator-installed, in-process, daemon-privileged**
-- the same tier as `ModuleDir`/`PluginDir`/`SecretEngineDir`
(module-position.md's trust-boundary section): loaded from a configured
directory at startup, never from a webhook body or task worktree. The
schema-by-example flow uses live events to *design against*; the saved
playbook is operator-authored file content in an operator-configured
location.
Two consequences:
1. The expression string is **trusted**. Operator-authored config, not
attacker input; no sandboxing against a hostile expression string is
required. If a future playbook source is less trusted (auto-generated
bindings from unauthenticated captures), that is the playbook-file
trust tier moving -- the loader's acceptance check is the right
place, not the evaluator.
2. The **data the expression reads is NOT trusted** (decoded webhook
body, prior results through the same pipeline). Robustness against
hostile data is required: no panic on wrong-typed/missing fields,
no unbounded cost on deep/cyclic maps, bounded evaluation time. CEL
provides this structurally (cost limits, linear non-Turing-complete
evaluation, error values instead of panics -- verified in the
vetting record below).
### The gate precedent is deliberately NOT copied
`internal/gate/gateeval` evaluates arbitrary interpreted Go
(`.archie/gate.go`) as the repo's conditional-logic precedent. That is
the wrong precedent for playbook conditions because this document's
premise is: **YAML is pure orchestration data; Yaegi is where
imperative logic lives** (Problem section; module-position.md's
"no generic hook" section). Yaegi snippets in YAML would move
imperative logic into the data layer -- the inversion this design
exists to prevent -- and make playbooks un-lintable.
The line: conditions are **declarative predicates over runtime data,
not programs**. CEL is non-Turing-complete and side-effect-free, so it
cannot become a program; a condition that needs real logic gets a
Module kind (Yaegi), referenced by the action, not a stronger
expression body.
### Plugin-engine-rule interaction: confirmed non-issue (stated once)
The plugin engine rule forbids a generic `Module` interface with an
`any` payload; it exists to keep static implementation contracts
typed. Evaluating CEL against runtime `map[string]any` is **not the
same problem**: the rule concerns *implementation contracts* (static
operations a capability family exposes), not *runtime data values*
(legitimately dynamic -- an event payload's shape is unknown until it
arrives). The Module contract stays fully typed (generated
`Args`/`Result` schemas); CEL is a read-only data reader under the
playbook engine family, not a new capability contract. Stated once.
### Minimum viable evaluation context (what expressions see)
One flat context, namespaced from the playbook run, built at dispatch
time:
| Name | Type | Source |
|---|---|---|
| `event` | `map(string, dyn)` | The triggering event's decoded payload (webhook body / forge issue / schedule tick). Field access via `event.<field>`, map access via `event["field"]`, presence via `has(event.<field>)`. |
| `actions` | `map(string, dyn)` | Previous actions' results, keyed by the action's `id` as declared in the playbook (`actions.<id>.result.<field>`), so a later action reads an earlier one's `Result` map regardless of position. The current action and later actions are not present. |
Expressions may also read literal-only state (numbers, strings,
booleans) directly. CEL's `has()` macro covers the missing-key case
(`has(event.labels)` -> bool), which replaces any bespoke `has`/
`contains` operator.
`actions.<id>.result` is the `map[string]any` `Module.Invoke` already
returns (t2db.13), so no result-shape change is needed for data flow;
channel/forge kinds' results are declared to the same shape.
Judgement call J1: `action` id is a required field on every action
that later actions reference; ids are validated at load (unique,
stable-identifier-shaped), and a `when`/`args` expression referencing
an unknown `actions.<id>` is a **load failure**, not a runtime miss --
the same reject-at-load rule as collisions (see below).
### Load-time validation (consistent with the collision rule)
The collision-handling rule is: schema defines the accepted message;
anything outside it is rejected at load, logged plainly, reported to
the caller -- never deferred to runtime. Expressions are validated the
same way, at playbook load:
- **Syntax**: every `when` and every CEL-typed `args` value is parsed
and checked by CEL's checker when the playbook is loaded. A syntax
or type error is a reported load failure (the playbook is dropped and
the error goes to the caller), not a runtime surprise.
- **Context conformance**: with the context declared to CEL
(`event` as `map(string, dyn)`, `actions` as `map(string, dyn)`),
unknown top-level identifiers are rejected at compile time.
Field-level typing inside `dyn` values is deferred to runtime by
CEL's design (verified: `data.items.lenght` with `data` dyn compiles
clean and errors at eval) -- so the loader declares as much type as
the schema provides (the action kinds' generated `Result` structs
become CEL type declarations at load, giving field-level rejection
for result reads), and `event` stays `dyn` because its shape is
unknown by design (schema-by-example).
- **Names**: unknown `actions.<id>` references and unknown roots are
compile-time/lint-visible errors (J1).
This means the lint tool (t2db.12) and the daemon share one checker
-- same single-source argument as the playbook loaders.
### Candidate comparison (surveyed 2026-09-03, pkg.go.dev + GitHub;
govaluate and gval checked against actual maintenance state, not
training memory)
| Candidate | Maintenance | Safety surface | Cost (measured) | Verdict |
|---|---|---|---|---|
| **CEL (cel.dev/cel-go)** | Actively maintained (google/cel, monthly releases) | Non-Turing-complete, linear eval, `CostLimit`/`CostTracking`/`ParserRecursionLimit`, no side effects, error values not panics | **+13.0MB** (2.34MB -> 15.25MB) | **Chosen** |
| `expr-lang/expr` v1.17.8 | Actively maintained | Side-effect-free, `DisableAllBuiltins`, depth limits (CVE fixed w/ tests) | +3.9MB (2.34MB -> 6.28MB) | Not chosen: two safety CVEs this cycle; smaller ecosystem; no protobuf/type-model fit with future schema-gen |
| `gval` v1.2.3 | Maintained, slower | No builtin-restriction surface; extras (ternary/`??`) not wanted | +3.0MB (2 transitive deps) | Not chosen |
| `govaluate` | **ARCHIVED** (author archived 2024) | n/a | n/a | Not chosen: dead |
| `starlark-go` | Maintained (Bazel) | Turing-complete with step limits | large (full language) | Not chosen: whole-language surface |
### Vetting record (vetting-dependencies skill, 2026-09-03)
Source reviewed at HEAD, not docs:
- **Repo hygiene**: clean. No binaries/media in tree (the only
non-source artifacts are expected generated protobuf files); 16-line
`.gitignore`.
- **Hard-boundary correctness**: the place correctness is hard for an
expression evaluator is bounding evaluation and surviving hostile
input. Dedicated regression tests exist and pass at HEAD:
`TestCostLimit`, `TestCostTracking*`, `TestParserRecursionLimit`,
and a large conformance suite. Verified by execution: evaluation
against a 2000-deep nested map under `CostLimit` completes in ~10
microseconds and returns an error value for a missing key -- no
panic, no unbounded walk.
- **Dependency footprint**: antlr (parser), protobuf + genproto (CEL's
type model), `cel.dev/expr` (CEL spec types), yaml.v3. All
mainstream, canonical, vendored widely by Google-adjacent projects.
No runtime-impure third-party code; the REPL-only deps (readline,
tview, tcell) are not in the library import graph.
- **Security record**: no advisories on record for `cel.dev/cel-go`
(GitHub Advisory Database, queried 2026-09-03).
- **Module path**: migrated to `cel.dev/cel-go` (v0.32.0 is current);
the old `github.com/google/cel-go` path is the pre-migration identity.
Pin `cel.dev/cel-go`.
- **Binary size cost**: +13.0MB is the largest of the candidates
(measured: baseline 2.34MB -> 15.25MB with CEL). Accepted because:
the daemon already embeds a large Yaegi runtime; CEL's type model is
the load-bearing win (compile-time rejection of result-path typos,
matching the reject-at-load rule); and the alternative bespoke
grammar would re-implement CEL's checker to get the same guarantee.
Flagged for sign-off: if 13MB is unacceptable for a specific
deployment target, the fallback is a bespoke grammar with a
re-implemented subset of this checker -- a strictly worse trade.
### Why bespoke was not sufficient
A bespoke grammar for the shipped cases (path read + comparison + bool
composition) is small, but the load-time-validation requirement makes
the comparison: the reject-at-load rule requires *field-level* static
checking of result paths against generated `Result` schemas, which is
CEL's type-checker's job. Re-implementing that on a hand-rolled
grammar means re-implementing a type system -- more code than the
grammar itself, with worse diagnostics, no conformance suite, and no
community surface for future expression needs (even declarative ones).
The bespoke option only wins on binary size, and the size cost of CEL
is measured and bounded.
### Judgment calls (flagged for sign-off)
- **J1 -- `id` required on referenced actions, unknown id = load
failure.** Matches the reject-at-load rule. { Call: an alternative
is positional `actions[0]` access, which breaks on reorder; named is
the operator-friendlier shape. }
- **J2 -- every `args` value is evaluated as a CEL expression.** No
split between "literal args" and "expression args": a plain string
value is a CEL string literal, a number is a CEL number literal, and
any value that reads context data is written in CEL. One mechanism,
no interpolation marker, no second syntax to learn. { Call: CEL
string literals must be quoted inside YAML (e.g. `message:
'"build finished"'`), which is slightly noisy for mostly-literal
args; accepted for uniformity -- the linter catches quoting mistakes
at load. }
- **J3 -- condition failure semantics.** A condition that errors at
runtime (missing/wrong-typed field) evaluates to **false**: the
action is skipped, logged, run continues. This is the simplest
candidate consistent with gap 1's bias against clever recovery;
gap 1 (resolved by archie-core-t2db.18) commits
stop-on-first-failure, so per-action control is not coming and J3
stands. { Call: CEL separates
false from error; we treat evaluation error as false + log. }
- **J4 -- `event` stays `dyn` (schema-by-example).** Load-time we
cannot know event field types; declaring them typed would reject
valid plays. Result reads get field-level checking via generated
schema declarations. { Call: when schema-by-example ships, tighten
event field declarations from captured samples. }
- **J5 -- cost limit default.** A `CostLimit` at load time (e.g.
100,000) is applied to every expression; the linter and daemon use
the same value so they agree. { Call: exact number is a constant to
tune at first use. }
### Interaction with the execution-time gaps
This resolution does not resolve the execution-time gaps (both are
resolved separately: gap 1 by `archie-core-t2db.18`, gap 2 by the
gap-2 decision below). Conditions make gap 1
concrete (condition-failure vs. action-failure semantics, J3) and
CEL's read-only result access makes gap 2's keying derivation
readable (an idempotency key expression can read event fields) -- both
flagged as the gaps' first real exercise. Gap 2's exercise is
subsequently solved by the gap-2 decision above (2026-09-05): the
answer is a fixed structural derivation, and idempotency keys are
deliberately NOT CEL expressions (see the "No CEL in the key"
passage).
### Implementation placement (for the follow-up slice)
- New `internal/domain/eda/expr` package: builds the CEL environment
(context declarations, generated `Result` schema types as CEL type
declarations, cost limit), compiles/validates playbook expressions
at load, evaluates them at dispatch. Domain layer; no infrastructure
imports; owned by the playbook engine family. One new dependency:
`cel.dev/cel-go` (pinned), `go.mod` otherwise unchanged.
- Consumption: the playbook event coordinator (open question 4's
loader, not yet built) evaluates `when` before dispatch and
evaluates `args` values before `Module.Invoke`; the lint tool
(t2db.12) shares the same compile/validate path for author-time
diagnostics.
- Schema integration: action kinds' generated `Result`/`Args` structs
(t2db.13's `logextract` pattern) are declared to CEL at load so
`actions.<id>.result.<field>` type-checks; `args` expressions
type-check against the target kind's `Args` schema before the
playbook is accepted.
2. **Playbook directory config field.** Where this lives in
`internal/config` / `configuration.Document`, and whether it's one
directory or a list (mirroring `SkillsDir`'s shape) -- follow existing
config precedent, not invented fresh.
**Status 2026-09-03: partially resolved.** The label-vocabulary slice
landed with a single-file shape (`workflow_labels_file`, mirroring
`workflow_routing_file` and `SkillsDir`'s single-path precedent) rather
than a directory, because it is the smallest useful case. A directory
(multiple playbooks, per-repo scoping) remains open and larger; the
single-file shape does not preclude it -- a future `workflow_dir` would
compose `workflow_labels_file`'s role into a loader.
**Status 2026-09-03: resolved by t2db.11 (corrected).** The directory
shape landed as `playbook_dirs` (a LIST of directories of
`*.yaml`/`*.yml` binding files), loaded at startup as an additional
input to the two single-file fields, which remain supported unchanged.
Sam's correction 2026-09-03: a single `playbook_dir` would contradict
supporting multiple independently-maintained playbook sources, which
the Dedup section already assumes ('independently-sourced playbook
directories collide'), so the field is a list. Cross-source collision
(same key in two directories, or in a single-file field and a
directory) is reported like an in-directory collision -- nothing is
arbitrated by source precedence. This is what finally exercises the
design doc's drop-and-report collision rule, since a single file
cannot collide with itself. Repo-scoping (tying directories to
`Config.Repos` owner/name) is explicitly deferred to a separate,
later decision; org/tenant keying is out of scope (no auth/identity
model).
Decisions recorded from the label-vocabulary slice (commit pending):
- Arbitrary labels bind to registered workflow names via
`LabelWorkflows` + `LoadLabelWorkflowsYAML`/`SetLabelWorkflows`
(`internal/domain/workflow/routing.go`).
- The closed `Kind`/NATS-subject set is untouched; the label map only
extends binding authority for labels the kind layer does not own.
- Collision rule (per Sam, 2026-09-03): a label already owned by the
kind set (bug/feature/bootstrap), an empty label, an empty workflow
name, or a duplicate binding is a reported load failure -- dropped
and logged, the error returned to the caller. Nothing is silently
arbitrated by schema/version. This is the "schema defines the
accepted message; anything else is the caller's fault" rule applied
to the label layer.
- Precedence in `Route()`: explicit `t.Workflow` → arbitrary-label
binding → kind binding → triage → implement → default.
3. **Trust boundary for Module Yaegi code.** The plugin engine rule's
invariant 6 draws a line between operator-trusted in-process code and
repository-supplied code that must run in a container. A user-authored
Module is closer to "operator-installed plugin" than "repository code
from a task," but this should be stated explicitly before Modules ship,
not assumed.
4. **First implementation slice.** Recommend: Module position + the
playbook loader + trigger-to-workflow dispatch only (subsuming today's
label routing) first, proving the schema-gen -> Yaegi -> playbook path
end to end on the smallest useful case, before adding Channel/Forge
action kinds or the linter/LSP.
**Status 2026-09-03: resolved across t2db.13-15.** The Module position
(registry + log kind, t2db.13), the CEL expression environment
(t2db.14), and the playbook document + event coordinator with
single-action workflow-kind dispatch (t2db.15) are shipped. The
coordinator is NOT yet wired into production task intake
(daemon.go/pollNATS/publishTask) -- that is a follow-up ticket;
production wiring would make a real forge-triggered task flow through
this coordinator instead of only workflow.Route(). The trigger shape
reuses the existing workintake kind/label vocabulary; multi-action and
non-workflow-position playbooks remain rejected at load (hard
boundary, blocked on gap 2, idempotency).
5. **Monetization boundary.** Sam has flagged this may be commercialized,
and explicitly wants it discerned from other OSS event-driven-automation
tooling. No design decision needed yet, but worth a note if/when
licensing or feature-gating questions arise for playbook authoring or
the LSP.
## Packages this touches (first slice, per open question 4)
- New: a schema-generation package/convention (`go:generate` source +
emitted contracts) for the Module position.
- New: playbook loader + event coordinator (name/location TBD -- likely
`internal/domain/eda` or similar, following the domain-layer rule: no
infrastructure imports, no untyped hooks).
- `internal/domain/workflow`: dispatch target for `workflow`-kind actions,
unchanged in its own contracts.
- `internal/domain/workintake`: `labelKinds`/routing becomes one playbook
among others once the loader subsumes it; existing Go fallback stays for
backward compatibility until playbooks fully replace it (mirrors the
`dynamic-workflow-triage.md` precedent of a two-line fallback change, not
a rip-and-replace).