# Execution tree and enforced state machine **Status:** Approved **Authority:** `docs/architecture/agent-system.md` (canonical workflow model); settles the "enforced WorkflowExecution state machine" item in `docs/architecture/migration-decisions.md` section 4. **Compounds with:** `docs/prds/task-run-detail.md` (attempt attribution), `docs/prds/state-store-contract.md` (wire contract and task-scoped grants). ## Decision A WorkflowExecution becomes the root of a tree of StepExecutions. Every stage run, and every agent call a stage makes, is its own durable record with a parent, an attempt, a status and an outcome. Both levels move only through a declared transition table, enforced by the State Store in the same transaction that writes the audit row and the domain event. ## Problem - A run is one `tasks` row. The only per-step record is the mutable `stage` column, overwritten as `workflow.Run` advances, plus observability events. A failed attempt cannot say which step failed or what its children did without replaying the event stream. - `Store.Transition` guards on the expected `from` status, but any `from` to any `to` is accepted. A `queued` to `merged` write succeeds. - Stop cancels one context in `daemon.runningTasks`. Nothing persisted says which work was in flight, so an agent call that outlives its stage (a reviewer, a container) is invisible to the store. - `workflow.Run` writes `tc.Store.Update` and discards the error, then emits `KindStageStart` separately. State and events can diverge. ## Model ```text WorkflowExecution (tasks row, id E) └── attempt N ├── StepExecution stage "plan" depth 0 ├── StepExecution stage "implement" depth 0 │ └── StepExecution agent "implement" depth 1 └── StepExecution stage "review" depth 0 ├── StepExecution agent "review:1" depth 1 └── StepExecution agent "review:2" depth 1 ``` - A StepExecution belongs to exactly one WorkflowExecution and one attempt. `parent_id` is null for a stage and names the enclosing StepExecution otherwise. Depth is derived from the parent, never supplied by the caller. - `kind` is `stage` or `agent`. A new kind needs a named consumer before it is added. - A retry starts a new attempt. Earlier attempts' StepExecutions are never rewritten. ## Transition tables WorkflowExecution statuses are the existing `taskstate` constants, persisted strings unchanged. `taskstate` gains the one table every writer checks: | From | To | | --------------- | ---------------------------------------------------------- | | `queued` | `running`, `closed_wont_do` | | `running` | `queued` (crash recovery), `waiting_human`, `pr_open`, `completed`, `parked`, `closed_wont_do` | | `waiting_human` | `queued` (approve), `closed_wont_do` | | `parked` | `queued` (retry), `dead` (abandon), `closed_wont_do` | | `pr_open` | `merged`, `rejected`, `queued` (remediation), `closed_wont_do` | | terminal | none | StepExecution statuses: | From | To | | --------- | ------------------------------------------------------ | | `pending` | `running`, `cancelled` | | `running` | `succeeded`, `failed`, `cancelled`, `interrupted` | | terminal | none | `interrupted` means the process running the step died or shut down. It is distinct from `cancelled`, which an operator or a parent asked for, and from `failed`, which the step reported. A StepExecution cannot reach `running` while its WorkflowExecution is not `running`, and cannot start under a terminal parent. ## Enforcement - The table lives in `taskstate` (no dependencies), so the store, daemon and gateway share one copy. - Every store method that changes `tasks.status` checks the table and writes the audit and event rows, not only `Transition`. That includes the claim, `ParkTask`, `Requeue`, `RetryTask`, `BeginRemediation` and `RecoverStale` paths, several of which change status without an audit row. - The State Store rejects an illegal transition with a new sentinel, `ErrIllegalTransition`, carried across the wire through `mapError` like `ErrStaleTransition`. A stale `from` stays `ErrStaleTransition`. - Each transition is one transaction: guarded row update, audit row, domain event row. The events table is written by the store, not emitted beside it. `KindStageStart` and `KindStageFinish` are produced by the step transition and published to the bus after commit. - `workflow.Run` stops discarding persistence errors. A failed step write parks the execution; the step does not run unrecorded. ## Cancellation Stopping a run is one store call, `CancelExecution(E, reason)`. In one transaction it moves every non-terminal StepExecution of the current attempt to `cancelled` and the WorkflowExecution to `closed_wont_do` or `parked`, whichever the operator action names. The daemon then cancels the in-memory context. A worker whose step write returns `ErrStaleTransition` because its step is already `cancelled` stops without writing further. The store is the record of cancellation. The context cancel is the delivery mechanism, so a lost cancel still leaves the store correct and the worker's next write fails. ## Crash recovery `RecoverStale` moves every `running` StepExecution of an interrupted execution to `interrupted` in the same transaction that requeues the execution. The next attempt starts with fresh steps. ## Wire contract New State Store RPCs, each with a matching Go method on a narrow consumer facade: - `StartStep(execution, attempt, parent, kind, name) -> step id` - `FinishStep(step, from, to, detail, tokens_used)` - `CancelExecution(execution, reason, to)` - `ListSteps(execution, attempt)` `TaskGrants.UnaryInterceptor` authorises a task-scoped token for `StartStep` and `FinishStep` on its own execution only. The container records its own agent calls. `CancelExecution` and `ListSteps` require the administrative token. ## Storage One new table, `step_executions`: `id`, `execution_id`, `attempt`, `parent_id`, `depth`, `kind`, `name`, `status`, `detail`, `tokens_used`, `started_at`, `finished_at`. Indexed on `(execution_id, attempt, id)`. Deleting an execution (archive, clearing terminal tasks) deletes its steps. Existing rows gain no backfilled steps. Their history stays in `transitions` and `events`. `tasks.stage` is written from the step transition while any reader still uses it, then dropped once the dashboard reads `ListSteps`. ## Out of scope - Running sibling steps in parallel. The tree records concurrency; changing the stage runner is separate work. - Cost in currency. `tokens_used` per step is the only usage field. - Workflow definition versioning and the `tasks` to `workflow_executions` rename. - A harness contract for swappable coding agents. ## Plan 1. `taskstate` transition tables with table-driven tests covering every legal and a sample of illegal pairs. 2. `ErrIllegalTransition` enforced in the Postgres store and mapped over the wire. 3. `step_executions` migration, `StartStep`/`FinishStep`, grant rules; `workflow.Run` records stages; events written in the transition transaction. 4. Agent-call children recorded from the agent runtime. 5. `CancelExecution`; daemon stop and chat/dashboard actions route through it. 6. `RecoverStale` interrupts running steps. 7. Dashboard run detail reads `ListSteps`; `tasks.stage` dropped. ## Verification - An illegal WorkflowExecution or StepExecution transition returns `ErrIllegalTransition` over gRPC, and `errors.Is` matches on the client. - Killing the daemon mid-stage, then restarting, leaves that stage `interrupted` and the execution `queued` with a new attempt. - Stopping a run during review leaves every review child `cancelled`, and a late `FinishStep` from the container returns `ErrStaleTransition`. - A task-scoped token cannot `StartStep` on another execution. - For every step transition there is exactly one event row, written in the same transaction.