From 164ca8d450cd3afb5a1bfabfd53059a467ec0557 Mon Sep 17 00:00:00 2001 From: Sam Date: Sat, 5 Sep 2026 10:11:59 +1000 Subject: [PATCH] docs(prds): resolve execution-time gap 2 -- idempotency keying for playbook actions (#715) Decision for channel/forge/module action keying in eda-playbook-engine.md: one durable dedup record per (action, event) pair via a playbook_dispatches ledger in internal/store, reusing binding_dispatches' INSERT OR IGNORE + sentinel conventions, with record-before-invoke ordering (at-most-once, the reverse of the binding loop's ordering, justified by the external-side-effect harm profile). Key = (playbook_id, playbook_version, event_id, action_id): event_id is the TaskEnvelope.IdempotencyKey() value already carried on DispatchInput.TaskID; the key is a fixed structural composition, deliberately not a CEL expression (event stays dyn; determinism under redelivery is the binding constraint). Redelivery of a recorded action skips it and continues the run at debug level, never surfaced as an error. Rows live as long as the playbook exists, no time-based expiry. Extends event-sources-and-reactions.md's contract without contradicting it: PublishUnique's dedup is publish-path only (Nats-Msg-Id + 2-minute DedupWindow), so consumer redelivery of side-effecting reactions was never covered; the ledger adds exactly that consumer-side piece, reusing the TaskEnvelope identity and the typed-family shape. The gap-2 prerequisites (Playbook ID/Version, DispatchInput.TaskID) already landed in a4a8ae7. Documentation only: no code changes. Keying implementation is separate follow-up work. Refs: archie-core-t2db.17 --- docs/prds/eda-playbook-engine.md | 357 ++++++++++++++++++++++--------- 1 file changed, 258 insertions(+), 99 deletions(-) diff --git a/docs/prds/eda-playbook-engine.md b/docs/prds/eda-playbook-engine.md index 8d8b6d56a..ab2bca121 100644 --- a/docs/prds/eda-playbook-engine.md +++ b/docs/prds/eda-playbook-engine.md @@ -210,7 +210,7 @@ rule's spirit of narrow capability families -- so the answer is not "bundle or don't," it's "don't duplicate the logic, and pick the thinnest consumer for each caller's actual constraint." -## Execution-time gaps (identified 2026-09-03; gap 1 resolved 2026-09-05, gap 2 unresolved) +## Execution-time gaps (identified 2026-09-03; both resolved 2026-09-05) Everything above covers *loading* a playbook. Nothing above covers what happens while one *runs*. Two gaps, both load-bearing enough to resolve @@ -220,7 +220,7 @@ before implementation, not defer: **Status 2026-09-05: resolved by archie-core-t2db.18.** The default below is a committed decision, not a candidate list; changing it later would -require its own decision. Gap 2 below remains open. +require its own decision. Gap 2 below is resolved by `archie-core-t2db.17`. A playbook is an ordered action list with data flowing between steps. If action 2 of 4 fails at runtime (an image-gen API times out, a forge call @@ -321,113 +321,268 @@ action-invocation code path and show they use different outcomes. ### 2. Idempotency at execution time -**Status 2026-09-03: proposed resolution below, awaiting Sam's sign-off** -(same process as gap 1 and the CEL resolution -- drafted with evidence, -not settled until explicitly approved). +**Status 2026-09-05: resolved** (per bead `archie-core-t2db.17`, a +design-investigation ticket: resolve, do not implement). The decision +below is the answer; the keying scheme itself is separate follow-up work +and is intentionally not built here. It **extends** +`event-sources-and-reactions.md`'s existing contract -- it does not +contradict it (relationship argued at the end). -`event-sources-and-reactions.md` decided reactions ride -`internal/eventbus`'s at-least-once delivery specifically so -`PublishUnique`/idempotency keys absorb redelivery without duplicate side -effects. This document inherits that for free only for `workflow`-kind -actions, because `TaskEnvelope` already carries an idempotency key. It said -nothing for `channel`, `forge`, or `module` actions with real side effects. +**Problem (unchanged):** `event-sources-and-reactions.md` decided +reactions ride `internal/eventbus`'s at-least-once delivery specifically +so `PublishUnique`/idempotency keys absorb redelivery without duplicate +side effects. This document inherits that for free only for +`workflow`-kind actions, because `TaskEnvelope` already carries an +idempotency key. It said nothing for `channel`, `forge`, or `module` +actions with real side effects -- a posted comment, a sent message, a +billed image generation. -**Proposed decision: reuse `internal/store/bindings.go`'s -`binding_dispatches` ledger shape directly -- do not design a new -mechanism.** This codebase already has a proven, tested, at-most-once -guarantee for the structurally identical problem (don't re-fire a -side-effecting action for an event already handled), surviving daemon -restarts because it is a durable table, not in-memory state: +One correction to the problem statement, checked against the code rather +than assumed, because it changes what the answer must be: +**`PublishUnique`'s dedup is not a consumer-redelivery guard at all.** +`PublishUnique` only sets the JetStream `Nats-Msg-Id` header on the +outgoing message (`internal/infrastructure/eventbus/nats/publisher.go:18-23`; +key constant `message.go:11-13`); JetStream suppresses *republished* +messages carrying a repeated key inside `Config.DedupWindow` +(`client.go:67`, `Duplicates: cfg.DedupWindow`), default 2 minutes +(`config.go:15`, `DefaultDedupWindow = 2 * time.Minute`). A consumer +redelivery is the same *stored* message re-delivered on Nak or +acknowledge-timeout (`internal/eventbus/eventbus.go:64-65`: "a handler +returning an error causes redelivery") -- not a republish; its +`Nats-Msg-Id` is never re-firewalled, and even if it were, two minutes is +not an at-most-once guarantee for a side effect that must never repeat. +**Conclusion: side-effecting action kinds need a consumer-side, durable +dedup record; the bus cannot supply it.** That is precisely the situation +`binding_dispatches` was built for -- so the decision below reuses its +mechanism rather than designing a new one. + +#### Decision + +**Granularity: per-action-per-event -- one dedup record per +(action, event) pair, not per playbook run.** This supersedes the +earlier draft's whole-run default. Whole-run keying has two crash +windows: the run row is written when the run starts, so a crash mid-run +leaves the run "started" and a redelivery skips *every remaining +action* (work silently lost); a crash before the row write re-runs +*every already-fired action* (duplicate side effects -- the exact harm +this gap exists to prevent). Per-action keys collapse both into one +window and one behavior: an action's record is written immediately +before that action's side effect is invoked, so an action fires iff its +own record is absent, no fired action ever re-fires, and a redelivered +event resumes at the first unrecorded action. For today's shipped +single-action playbooks the two granularities are equivalent in effect; +the difference starts with multi-action runs -- where whole-run keying +would already be wrong. + +**Storage/lookup: a new durable ledger table `playbook_dispatches` in +`internal/store`, copying `binding_dispatches`'s conventions exactly** +(durable table, `INSERT OR IGNORE`, sentinel error, rows freed with +their parent, survives daemon restarts -- `internal/store/bindings.go:31-37`, +`355-382`): ```sql -CREATE TABLE binding_dispatches ( - binding_id, binding_version, capture_id, task_id, dispatched_at, - PRIMARY KEY (binding_id, capture_id) +CREATE TABLE playbook_dispatches ( + playbook_id TEXT NOT NULL, + playbook_version TEXT NOT NULL, + event_id TEXT NOT NULL, + action_id TEXT NOT NULL, + dispatched_at TEXT NOT NULL, + PRIMARY KEY (playbook_id, playbook_version, event_id, action_id) ); ``` -`RecordDispatch` does `INSERT OR IGNORE`; a duplicate returns the -sentinel `ErrAlreadyDispatched`, which the caller treats as "another -cycle already won this race," not an error to surface. +Written with the same `INSERT OR IGNORE` + sentinel-error convention +(`ErrAlreadyDispatched`, `bindings.go:59-63`), so a duplicate is a no-op +write rather than a constraint error, and the caller matches it with +`errors.Is` the way the binding dispatch loop already does +(`internal/daemon/daemon.go:591-597` -- "another cycle already won this +race," not an error to surface). The record is consumed by the playbook +coordinator at dispatch time through a domain-side interface implemented +by `internal/store` -- the same shape as `BindingDispatcher` +(`internal/store/interface.go:123-139`) -- and the table lands in the +store's schema application (`internal/store/store.go:137-139`). No change +to `internal/eventbus`; no per-message dedup state on the client. -**Concretely, for playbooks:** a new ledger table (e.g. -`playbook_dispatches`), unique on `(playbook_id, playbook_version, -event_id)` -- the same three-column shape as `binding_dispatches` -(`binding_id, binding_version, capture_id`), substituting a playbook's -identity and an event's identity for a binding's and a capture's. Written -with the same `INSERT OR IGNORE` + sentinel-error convention, checked -once per playbook run before any action dispatches. +**Lifetime: no time-based expiry.** Rows live as long as their playbook +exists; when a playbook is removed from the configured directories, the +coordinator deletes its rows -- mirroring binding-dispatches-rows-deleted- +with-binding (`bindings.go:204-214`, same transaction as the delete). The +at-most-once guarantee must not silently decay, which is exactly the +property that disqualifies `DedupWindow` above; table growth is bounded +by distinct (playbook, event, action) pairs over the playbook's lifetime +and is accepted, the same tradeoff `binding_dispatches` already makes. -**This surfaces a concrete prerequisite, checked against the actual -code, not assumed: `t2db.15`'s shipped `Playbook` struct -(`internal/domain/eda/playbook/playbook.go`) is currently `{Trigger, -Actions}` -- no `ID`, no `Version`.** Unlike `Binding`, which gets its -identity and version from being a database row, a playbook is a loaded -file with no persisted identity today. This resolution cannot be -implemented until playbooks have one. Proposed (not yet built): derive -`playbook_id` from a stable path-relative identifier (e.g. the -playbook's file path relative to its configured directory root, which is -already unique per the directory-loader's own collision rule) and -`playbook_version` from a content hash of the loaded file, recomputed on -every load -- mirroring `Binding.Version`'s purpose (pin dispatched-run -provenance to the exact definition that was active when it fired) without -requiring a database row the way `Binding` has one. This identity scheme -is a small, separate implementation detail for whoever builds this ledger, -not a new open question -- but it is *not yet decided in code*, and this -resolution is not fully buildable until it is. +**Redelivery behavior: skip the recorded action and continue the run.** +A detected duplicate is not re-invoked, is not reported to the caller as +an error, and is logged at debug level rather than as a warning -- this +is the expected, correct behavior of at-least-once delivery, not an +anomaly (same treatment as `ErrAlreadyDispatched` at +`daemon.go:592-597`). +The run continues with the next action (or ends, for a single-action +playbook whose one action is recorded), which is what makes a redelivery +a resume rather than a dead restart. -**Granularity: whole-run, not per-action** -- confirmed as the default -per the gap's own original bias ("simpler... unless a specific action -needs finer-grained dedup"). One key per `(playbook, event)` dedups the -entire run; if a specific future action kind genuinely needs -finer-grained dedup than "did this playbook already run for this event," -that is a documented exception to add when such an action kind is -designed, not something to generalize for pre-emptively now. +**Key derivation: a fixed composition of four structural identities, +computed by the coordinator -- not a CEL expression.** The key is the +`(playbook_id, playbook_version, event_id, action_id)` tuple above, where: -**Key derivation (`event_id`) -- a second concrete prerequisite, checked -against the actual code, not assumed:** the natural instinct is "reuse -the capture pipeline's stable identity" (`captured_events.id`), but that -is **wrong for what `t2db.15` actually shipped**. Verified directly: -`DispatchInput` (`internal/domain/eda/playbook/playbook.go:187`) is -`{Labels []string, Kind string, Event map[string]any}` -- a **task-label -trigger**, the same vocabulary `workflow.Route()` already dispatches on, -entirely unrelated to the webhook-capture/binding pipeline -(`docs/architecture/bindings.md`'s system). No capture, and no -`captured_events` row, is involved in this trigger path at all. `Event` -here is only a cheap re-exposure of the label/kind fields for CEL, not a -webhook payload. +- `playbook_id` is `Playbook.ID` -- the playbook file's path relative to + its configured directory root, slash-normalized, unique within the load + composition by construction (`internal/domain/eda/playbook/playbook.go:39-45`, + `149-155`). Note this prerequisite already landed in code: the earlier + draft's "not yet decided in code" paragraphs are stale -- + `edd689e` (`feat(eda): thread Playbook ID/Version and DispatchInput task + identity (gap-2 prereqs)`) shipped `Playbook.ID`/`Version` and + `DispatchInput.TaskID`. +- `playbook_version` is `Playbook.Version` -- a SHA-256 of the loaded + file, recomputed on every load (`playbook.go:46-49`, `154-155`), + mirroring `Binding.Version`'s provenance-pinning purpose: the dispatch + is pinned to the exact definition that was active when it fired. A + changed playbook definition is a different key and may fire again for + the same event -- deliberate, not a bug. +- `event_id` is `DispatchInput.TaskID` (`playbook.go:214-222`), which + carries the `TaskEnvelope.IdempotencyKey()` value + (`"archie:" + owner/repo/number`, + `internal/domain/workintake/envelope.go:105-107`). It is chosen over + `store.Task.ID` because the trigger vocabulary is issue-level + workintake label/kind, dispatched at discovery time (pollNATS, forge + webhook receiver), where no task row exists yet (`playbook.go:214-222` + comment; `playbook_test.go:338-345`: "the value available at the + discovery/dispatch point... NOT a store.Task.ID int64"). Keying on the + issue identity rather than the delivery source is the load-bearing + property the existing contract already established for `TaskEnvelope` + (`event-sources-and-reactions.md` question 4: a webhook-sourced + envelope for the same issue produces the *same* idempotency key as a + poll-discovered one; `internal/forge/webhook/receiver.go:14-20` -- + "Idempotency is not this package's job"); this resolution inherits that + property rather than re-deciding it. Consequence, stated explicitly: + same-issue re-discoveries (a re-label that re-triggers the same kind) + are the *same event* by design -- the workintake trigger vocabulary is + issue-granular, exactly as `TaskEnvelope`'s own key is. A future + capture/binding-originated trigger type keys on `captured_events.id` + instead; derivation is per-trigger-type, not one formula for all + triggers (rule carried over unchanged from the draft). +- `action_id` is the action's declared `id` -- J1's named ids, "a + required field on every action that later actions reference", unique + and stable-identifier-shaped, validated at load (the CEL resolution's + J1 and its context table above in this document). For an action declaring no `id` + (permitted by J1 while unreferenced), the key uses the action's 1-based + position in the playbook: deterministic and stable for today's + single-action playbooks, and a later reorder is a file edit whose + content hash changes `playbook_version` with it, so a positional + fallback cannot silently collide with a prior definition. -`DispatchInput` also carries **no task identity today** -- not even the -originating task's ID. This document's own earlier claim that -"workflow-kind actions inherit idempotency for free via `TaskEnvelope`" -is true of the task itself, but that identity is not currently threaded -into `DispatchInput` for a playbook run to key on. This is a second, -separate prerequisite alongside the Playbook-identity one above: before -this ledger can be built, `DispatchInput` needs to carry the originating -task's identity (its `store.Task.ID`, or the `TaskEnvelope` idempotency -key that produced it -- whoever implements this should confirm which is -actually available at the `Dispatch` call site rather than assuming). If -and when a capture/binding-originated trigger type is added to the -playbook engine later (unifying with the webhook pipeline is an explicit -non-goal of this document today, per `docs/architecture/bindings.md`'s -own "does not attempt to unify them"), that trigger type would key on -`captured_events.id` instead -- the derivation is per-trigger-type, not -one formula for all triggers. +**No CEL in the key -- stated explicitly because the CEL resolution +flagged this as gap 2's "first real exercise"** ("CEL's read-only result +access makes gap 2's keying derivation readable" -- the CEL resolution's +closing note). Readability +is not the binding constraint; **determinism under redelivery is**, and +CEL key expressions lose three ways. First, they are not load-checkable: +`event` stays `dyn` by the resolved J4 (schema-by-example), so +a key expression over event fields cannot be type-checked at load, +violating the document's reject-at-load philosophy. Second, they can +fail at dispatch (a missing field is a CEL error), and a key that cannot +be computed is an unfireable state -- neither "skip" nor "fire" is safe +without losing or duplicating work. Third, the contract needs one +derivation, and two authors writing different key expressions would +fragment the ledger. The key is therefore structural by construction: +every component above is an immutable identity of the run's inputs, so +two deliveries of the same event derive the same key with no evaluation. +The exercise's actual outcome is the derivation rule above; CEL remains +a *condition* surface (`when`), never a key surface. { Call: if a future +action kind genuinely needs finer granularity than (action, event) -- +e.g. "per comment, not per issue" -- that is the documented exception +path, and it arrives with its own trigger type and its own event +identity, per the per-trigger-type rule. } -**What "dedup" does on redelivery:** the **whole run** is skipped, not -just the already-fired action -- consistent with whole-run granularity -above. A redelivered event that already produced a `playbook_dispatches` -row for this `(playbook, event)` pair does nothing further, silently -(logged at debug level, not as a warning -- this is the expected, -correct behavior of at-least-once delivery, not an anomaly). +**Record-before-invoke -- one deliberate divergence from +`binding_dispatches`, justified by the harm profile.** The binding +dispatch loop enqueues the task first and records the ledger row after +(`daemon.go:585-599`), accepting that racing cycles may both enqueue +("the duplicate task remains queued... delete-on-races is its own can of +worms"); the selection query excludes already-dispatched captures +(`bindings.go:395-399`), but the race window remains. That ordering is +correct there because the duplicate artifact is an internal task row -- +recoverable, low-harm. The playbook ledger's ordering is reversed on +purpose: **the row is written and committed before the action's side +effect is invoked** -- the `INSERT OR IGNORE` is the atomic gate, and +the lost-write loser skips without invoking. The cost is stated, not +hidden: a crash between the row commit and the invoke loses that one +action's side effect, repairable by hand -- the reverse direction of the +same window that would otherwise duplicate a posted comment or +re-bill an image generation. For side effects whose duplication is +externally visible and not revocable, at-most-once is the required +direction; the binding system's at-least-once-tolerant ordering is not +the precedent for this ledger. -Gap 2 blocks open question 4's first-slice recommendation only -partially: the first slice (Module + loader + workflow-kind dispatch) -already has idempotency for free via `TaskEnvelope`, so it can proceed -without resolving gap 2. Gap 1 (failure mid-run) is real even for a -single-action first slice's *future* multi-action runs, but is not -required to build the first slice today. Gap 2 must be resolved -before Channel/Forge actions or multi-action playbooks ship. +**Gap-1 interplay, stated to keep the two mechanisms separate:** the +ledger answers one question per action -- has this action already fired +for this event? It does not answer whether the run continues, stops, or +is re-attempted; that is gap 1's coordinator concern (a run-outcome +record; gap 1 explicitly leaves "what leave it alone for restart means +for a playbook run" to whoever builds the coordinator). One consequence +of record-before: an action that FAILED (gap 1's real-action-error case) +has a ledger row, so a later redelivery of the same event skips it -- +consistent with gap 1's "no automatic retry" resolve (a corrective +action is a *new* playbook/action, per gap 1's rollback paragraph). A +failed action is never re-attempted by the bus; if a run must be +recovered, an operator edits the playbook -- new version, new key -- +rather than fighting ledger state. + +**Relationship to `event-sources-and-reactions.md` -- extension, not +contradiction, argued explicitly.** Three points, keyed to that +document's sections: + +1. Its decision 2 commits reactions to at-least-once delivery via + `PublishUnique`'s "without duplicate enqueue" guarantee -- which, with + the code facts above, is accurate for the *publish* path only: + `PublishUnique` absorbs duplicate publishes of the same event (poll + vs. webhook), which is what that decision was actually about. This + document's own problem statement, taken literally, overstates it + ("keys absorb redelivery without duplicate side effects") -- that + overstatement is exactly what this gap exists to correct. The + existing contract never claimed consumer redeliveries of + side-effecting reactions were absorbed, so nothing it says contradicts + a consumer-side ledger; this resolution pins down the boundary the + sentence left implicit (publish-path dedup vs. consumer-path dedup) + rather than changing either. +2. Its decision 3 (producer-only, no veto/mutation of in-flight work) + is untouched: the ledger is the running reaction's own bookkeeping + about its own actions -- it suppresses re-firing an action the + coordinator already ran for the same event; it blocks, mutates, or + reorders nothing else, and no other producer's work is touched. +3. Its decision 4 (no generic `Source` interface; `TaskEnvelope` is the + typed contract, keyed on issue identity, not delivery source): the + event half of the playbook key IS that same identity + (`TaskEnvelope.IdempotencyKey()`), and the per-trigger-type rule + means any future trigger type brings its own identity rather than a + new generic one -- the playbook ledger introduces no second identity + vocabulary and no untyped payload. The only new type in this + resolution is the ledger's own value object, consumed through a + narrow interface like `BindingDispatcher` -- the same shape the + document's "typed families" rule (decision 1, plugin engine rule) + already blesses. + +**Scope: this is the `channel`/`forge`/`module` decision, not a +`workflow`-position change.** Workflow-kind actions keep their existing +keying claim unchanged -- the `TaskEnvelope` path that this document +already inherits for free (`daemon.go:858-869`, `PublishTask` publishes +with `task.IdempotencyKey()`), with its cross-poll/webhook property +untouched. The ledger gate applies to the side-effecting positions +(channel, forge, module), which is why the closing statement below +still holds: the first slice (Module + loader + workflow-kind dispatch) +was buildable without this resolution, and channel/forge/multi-action +playbooks were not. A coordinator MAY implement one position-uniform +gate as a simplification; what this resolution requires is only that +the side-effecting positions are gated. + +Both execution-time gaps are now resolved (gap 1 by +`archie-core-t2db.18` above, gap 2 by `archie-core-t2db.17` above). For +the record, the first slice (Module + loader + workflow-kind dispatch) +never needed gap 2: workflow-kind actions had idempotency for free via +`TaskEnvelope`, and the ledger is only required once side-effecting +positions dispatch. Channel/Forge actions and multi-action playbooks +are unblocked on the execution-time questions. ## Open questions (for sign-off before implementation) @@ -680,13 +835,17 @@ before Channel/Forge actions or multi-action playbooks ship. ### Interaction with the execution-time gaps - This resolution does not resolve the execution-time gaps (gap 1 is - resolved separately by archie-core-t2db.18; gap 2 remains open). - Conditions make gap 1 + This resolution does not resolve the execution-time gaps (both are + resolved separately: gap 1 by `archie-core-t2db.18`, gap 2 by the + gap-2 decision below). Conditions make gap 1 concrete (condition-failure vs. action-failure semantics, J3) and CEL's read-only result access makes gap 2's keying derivation readable (an idempotency key expression can read event fields) -- both - flagged as the gaps' first real exercise, not solved here. + flagged as the gaps' first real exercise. Gap 2's exercise is + subsequently solved by the gap-2 decision above (2026-09-05): the + answer is a fixed structural derivation, and idempotency keys are + deliberately NOT CEL expressions (see the "No CEL in the key" + passage). ### Implementation placement (for the follow-up slice)