8.2 KiB
Execution tree and enforced state machine
Status: Approved
Authority: docs/architecture/agent-system.md (canonical workflow model);
settles the "enforced WorkflowExecution state machine" item in
docs/architecture/migration-decisions.md section 4.
Compounds with: docs/prds/task-run-detail.md (attempt attribution),
docs/prds/state-store-contract.md (wire contract and task-scoped grants).
Decision
A WorkflowExecution becomes the root of a tree of StepExecutions. Every stage run, and every agent call a stage makes, is its own durable record with a parent, an attempt, a status and an outcome. Both levels move only through a declared transition table, enforced by the State Store in the same transaction that writes the audit row and the domain event.
Problem
- A run is one
tasksrow. The only per-step record is the mutablestagecolumn, overwritten asworkflow.Runadvances, plus observability events. A failed attempt cannot say which step failed or what its children did without replaying the event stream. Store.Transitionguards on the expectedfromstatus, but anyfromto anytois accepted. Aqueuedtomergedwrite succeeds.- Stop cancels one context in
daemon.runningTasks. Nothing persisted says which work was in flight, so an agent call that outlives its stage (a reviewer, a container) is invisible to the store. workflow.Runwritestc.Store.Updateand discards the error, then emitsKindStageStartseparately. State and events can diverge.
Model
WorkflowExecution (tasks row, id E)
└── attempt N
├── StepExecution stage "plan" depth 0
├── StepExecution stage "implement" depth 0
│ └── StepExecution agent "implement" depth 1
└── StepExecution stage "review" depth 0
├── StepExecution agent "review:1" depth 1
└── StepExecution agent "review:2" depth 1
- A StepExecution belongs to exactly one WorkflowExecution and one attempt.
parent_idis null for a stage and names the enclosing StepExecution otherwise. Depth is derived from the parent, never supplied by the caller. kindisstageoragent. A new kind needs a named consumer before it is added.- A retry starts a new attempt. Earlier attempts' StepExecutions are never rewritten.
Transition tables
WorkflowExecution statuses are the existing taskstate constants, persisted
strings unchanged. taskstate gains the one table every writer checks:
| From | To |
|---|---|
queued |
running, closed_wont_do |
running |
queued (crash recovery), waiting_human, pr_open, completed, parked, closed_wont_do |
waiting_human |
queued (approve), closed_wont_do |
parked |
queued (retry), dead (abandon), closed_wont_do |
pr_open |
merged, rejected, queued (remediation), closed_wont_do |
| terminal | none |
StepExecution statuses:
| From | To |
|---|---|
pending |
running, cancelled |
running |
succeeded, failed, cancelled, interrupted |
| terminal | none |
interrupted means the process running the step died or shut down. It is
distinct from cancelled, which an operator or a parent asked for, and from
failed, which the step reported.
A StepExecution cannot reach running while its WorkflowExecution is not
running, and cannot start under a terminal parent.
Enforcement
- The table lives in
taskstate(no dependencies), so the store, daemon and gateway share one copy. - Every store method that changes
tasks.statuschecks the table and writes the audit and event rows, not onlyTransition. That includes the claim,ParkTask,Requeue,RetryTask,BeginRemediationandRecoverStalepaths, several of which change status without an audit row. - The State Store rejects an illegal transition with a new sentinel,
ErrIllegalTransition, carried across the wire throughmapErrorlikeErrStaleTransition. A stalefromstaysErrStaleTransition. - Each transition is one transaction: guarded row update, audit row, domain
event row. The events table is written by the store, not emitted beside it.
KindStageStartandKindStageFinishare produced by the step transition and published to the bus after commit. workflow.Runstops discarding persistence errors. A failed step write parks the execution; the step does not run unrecorded.
Cancellation
Stopping a run is one store call, CancelExecution(E, reason). In one
transaction it moves every non-terminal StepExecution of the current attempt
to cancelled and the WorkflowExecution to closed_wont_do or parked,
whichever the operator action names. The daemon then cancels the in-memory
context. A worker whose step write returns ErrStaleTransition because its
step is already cancelled stops without writing further.
The store is the record of cancellation. The context cancel is the delivery mechanism, so a lost cancel still leaves the store correct and the worker's next write fails.
Crash recovery
RecoverStale moves every running StepExecution of an interrupted
execution to interrupted in the same transaction that requeues the
execution. The next attempt starts with fresh steps.
Wire contract
New State Store RPCs, each with a matching Go method on a narrow consumer facade:
StartStep(execution, attempt, parent, kind, name) -> step idFinishStep(step, from, to, detail, tokens_used)CancelExecution(execution, reason, to)ListSteps(execution, attempt)
TaskGrants.UnaryInterceptor authorises a task-scoped token for StartStep
and FinishStep on its own execution only. The container records its own
agent calls. CancelExecution and ListSteps require the administrative
token.
Storage
One new table, step_executions: id, execution_id, attempt,
parent_id, depth, kind, name, status, detail, tokens_used,
started_at, finished_at. Indexed on (execution_id, attempt, id).
Deleting an execution (archive, clearing terminal tasks) deletes its steps.
Existing rows gain no backfilled steps. Their history stays in transitions
and events.
tasks.stage is written from the step transition while any reader still uses
it, then dropped once the dashboard reads ListSteps.
Out of scope
- Running sibling steps in parallel. The tree records concurrency; changing the stage runner is separate work.
- Cost in currency.
tokens_usedper step is the only usage field. - Workflow definition versioning and the
taskstoworkflow_executionsrename. - A harness contract for swappable coding agents.
Plan
taskstatetransition tables with table-driven tests covering every legal and a sample of illegal pairs.ErrIllegalTransitionenforced in the Postgres store and mapped over the wire.step_executionsmigration,StartStep/FinishStep, grant rules;workflow.Runrecords stages; events written in the transition transaction.- Agent-call children recorded from the agent runtime.
CancelExecution; daemon stop and chat/dashboard actions route through it.RecoverStaleinterrupts running steps.- Dashboard run detail reads
ListSteps;tasks.stagedropped.
Verification
- An illegal WorkflowExecution or StepExecution transition returns
ErrIllegalTransitionover gRPC, anderrors.Ismatches on the client. - Killing the daemon mid-stage, then restarting, leaves that stage
interruptedand the executionqueuedwith a new attempt. - Stopping a run during review leaves every review child
cancelled, and a lateFinishStepfrom the container returnsErrStaleTransition. - A task-scoped token cannot
StartStepon another execution. - For every step transition there is exactly one event row, written in the same transaction.