eda: resolve mid-playbook action failure semantics (execution-time gap 1) #441

Closed
opened 2026-10-27 13:36:20 +00:00 by sam · 0 comments
Owner

Design-investigation ticket, same shape as the idempotency one above -- produces a design-doc resolution, not implementation. Gates multi-action playbooks (t2db.15 is deliberately single-action to avoid needing this). docs/prds/eda-playbook-engine.md's 'Execution-time gap 1' has no default yet and explicitly says a default must be a stated decision, not something nobody chose.

Grounded by a crew investigation pass (2026-09-03, not committed -- content folded in here): read event-sources-and-reactions.md's producer-only/no-veto/no-mutation language, existing internal/domain/workflow stage-failure conventions (does a failing stage stop/wrap/resume?), and the load-collision report shape (drop+log+report to caller) from t2db.9-.12 as the precedent for how a run outcome should be reported.

Design

Decide and write the resolution into eda-playbook-engine.md:

  1. On action N of M failing: stop-on-first-failure (matches this doc's existing anti-clever-recovery bias) vs continue-to-independent-later-actions vs rollback. State which, and why the alternatives were rejected -- don't just restate the doc's existing hedge.
  2. What 'rollback' means (likely: nothing, since a posted forge comment isn't revocable) -- state this explicitly rather than leaving it implied.
  3. Producer-only interaction: decide whether a playbook reacting to its own action's failure by running compensating actions is in scope -- state the boundary explicitly against event-sources-and-reactions.md's producer-only rule.
  4. Condition-failure vs action-failure: J3 (CEL condition eval error -> false, skip, log, continue) is already decided for conditions specifically. State cleanly how this differs from an actual action failure.

STRONG REUSABLE PRECEDENT FOUND IN THIS CODEBASE (verified directly, 2026-09-03) -- read internal/domain/workflow/workflow.go:294-352 (Run/finish/park) before designing anything new. The existing workflow engine already answers a structurally identical question for its own stages, and the answer is precise, not just 'stop on failure':

  • A real stage error: stop immediately, set ParkReason (which stage, what error), call park() -- status becomes store.StatusParked, requiring human/retry intervention. This IS this codebase's existing 'stop-on-first-failure, log the outcome, report to caller' pattern -- a playbook run failure should almost certainly reuse the SAME park/report shape (or a directly analogous one) rather than inventing new run-outcome semantics.
  • Context cancellation (ctx.Err() != nil, e.g. daemon shutdown mid-stage) is explicitly NOT treated as a failure: the task is left running so existing crash-recovery requeues it; parking here 'would publish a false failure and require manual intervention.' A playbook coordinator dispatching a Module/Workflow action needs the SAME distinction -- a daemon-shutdown interruption mid-action must not be recorded as an action failure, or every restart during a dispatch would falsely mark playbooks as broken.
  • A workflow ending without an explicit terminal outcome is itself parked as 'workflow ended without an outcome (definition bug)' -- i.e. an engine-level invariant violation gets its own distinct, honestly-labeled outcome rather than being silently treated as success. Consider whether a playbook run has an analogous 'must end with X' invariant.

This means gap 1 may reduce to: 'reuse park()'s three-way distinction (real failure / shutdown interruption / definition-bug) for playbook runs' rather than a from-scratch design -- read the actual code before deciding whether that reduction actually holds or where a playbook run's shape genuinely differs (e.g. playbooks may not have a Store.Transition-shaped persistence target the way a store.Task does -- confirm what a playbook run's own state, if any, actually gets recorded to).

Acceptance criteria

eda-playbook-engine.md's Execution-time gap 1 section is marked resolved with a stated default (not a list of candidates) and explicit reasoning against event-sources-and-reactions.md's producer-only rule. No code changes required for this ticket to close.

Notes

Proposed resolution drafted 2026-09-03 in docs/prds/eda-playbook-engine.md, commit 99eaa7a -- reuses workflow.Run's existing three-way stage-failure distinction. Explicitly marked 'awaiting Sam's sign-off,' not settled.

Close reason

Closed


Imported from beads archie-core-t2db.18 · GitHub #725 · type task · priority P1 · created 2026-09-02 · closed 2026-09-05

Design-investigation ticket, same shape as the idempotency one above -- produces a design-doc resolution, not implementation. Gates multi-action playbooks (t2db.15 is deliberately single-action to avoid needing this). docs/prds/eda-playbook-engine.md's 'Execution-time gap 1' has no default yet and explicitly says a default must be a stated decision, not something nobody chose. Grounded by a crew investigation pass (2026-09-03, not committed -- content folded in here): read event-sources-and-reactions.md's producer-only/no-veto/no-mutation language, existing internal/domain/workflow stage-failure conventions (does a failing stage stop/wrap/resume?), and the load-collision report shape (drop+log+report to caller) from t2db.9-.12 as the precedent for how a run outcome should be reported. ### Design Decide and write the resolution into eda-playbook-engine.md: 1. On action N of M failing: stop-on-first-failure (matches this doc's existing anti-clever-recovery bias) vs continue-to-independent-later-actions vs rollback. State which, and why the alternatives were rejected -- don't just restate the doc's existing hedge. 2. What 'rollback' means (likely: nothing, since a posted forge comment isn't revocable) -- state this explicitly rather than leaving it implied. 3. Producer-only interaction: decide whether a playbook reacting to its own action's failure by running compensating actions is in scope -- state the boundary explicitly against event-sources-and-reactions.md's producer-only rule. 4. Condition-failure vs action-failure: J3 (CEL condition eval error -> false, skip, log, continue) is already decided for conditions specifically. State cleanly how this differs from an actual action failure. STRONG REUSABLE PRECEDENT FOUND IN THIS CODEBASE (verified directly, 2026-09-03) -- read internal/domain/workflow/workflow.go:294-352 (Run/finish/park) before designing anything new. The existing workflow engine already answers a structurally identical question for its own stages, and the answer is precise, not just 'stop on failure': - A real stage error: stop immediately, set ParkReason (which stage, what error), call park() -- status becomes store.StatusParked, requiring human/retry intervention. This IS this codebase's existing 'stop-on-first-failure, log the outcome, report to caller' pattern -- a playbook run failure should almost certainly reuse the SAME park/report shape (or a directly analogous one) rather than inventing new run-outcome semantics. - Context cancellation (ctx.Err() != nil, e.g. daemon shutdown mid-stage) is explicitly NOT treated as a failure: the task is left running so existing crash-recovery requeues it; parking here 'would publish a false failure and require manual intervention.' A playbook coordinator dispatching a Module/Workflow action needs the SAME distinction -- a daemon-shutdown interruption mid-action must not be recorded as an action failure, or every restart during a dispatch would falsely mark playbooks as broken. - A workflow ending without an explicit terminal outcome is itself parked as 'workflow ended without an outcome (definition bug)' -- i.e. an engine-level invariant violation gets its own distinct, honestly-labeled outcome rather than being silently treated as success. Consider whether a playbook run has an analogous 'must end with X' invariant. This means gap 1 may reduce to: 'reuse park()'s three-way distinction (real failure / shutdown interruption / definition-bug) for playbook runs' rather than a from-scratch design -- read the actual code before deciding whether that reduction actually holds or where a playbook run's shape genuinely differs (e.g. playbooks may not have a Store.Transition-shaped persistence target the way a store.Task does -- confirm what a playbook run's own state, if any, actually gets recorded to). ### Acceptance criteria eda-playbook-engine.md's Execution-time gap 1 section is marked resolved with a stated default (not a list of candidates) and explicit reasoning against event-sources-and-reactions.md's producer-only rule. No code changes required for this ticket to close. ### Notes Proposed resolution drafted 2026-09-03 in docs/prds/eda-playbook-engine.md, commit 99eaa7a -- reuses workflow.Run's existing three-way stage-failure distinction. Explicitly marked 'awaiting Sam's sign-off,' not settled. ### Close reason Closed --- _Imported from beads `archie-core-t2db.18` · GitHub #725 · type task · priority P1 · created 2026-09-02 · closed 2026-09-05_
sam added the hightask labels 2026-10-27 13:36:20 +00:00
sam self-assigned this 2026-10-27 13:36:20 +00:00
sam closed this issue 2026-10-27 13:36:20 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#441