Files
archie/docs/prds/execution-tree-state-machine.md
T
sam 1ff51a4a15 docs(prds): fit harnesses, steps and the store inside the org boundary and profile model
A harness is an agent profile whose environment is a Kit composition, so
image profiles and existing workflows are unchanged and only Kit runs take
the egress path; the relay replaces the claim that containers reach the
proxy directly. Credentials, step records, store installs and reviews are
org-scoped under the run credential and the Authorizer, and workflow calls
are linked across executions rather than nested.
2026-09-26 02:41:02 +10:00

201 lines
9.4 KiB
Markdown

# Execution tree and enforced state machine
**Status:** Approved
**Authority:** `docs/architecture/agent-system.md` (canonical workflow model);
settles the "enforced WorkflowExecution state machine" item in
`docs/architecture/migration-decisions.md` section 4.
**Compounds with:** `docs/prds/task-run-detail.md` (attempt attribution),
`docs/prds/state-store-contract.md` (wire contract and task-scoped grants).
## Decision
A WorkflowExecution becomes the root of a tree of StepExecutions. Every stage
run, and every agent call a stage makes, is its own durable record with a
parent, an attempt, a status and an outcome. Both levels move only through a
declared transition table, enforced by the State Store in the same transaction
that writes the audit row and the domain event.
## Problem
- A run is one `tasks` row. The only per-step record is the mutable `stage`
column, overwritten as `workflow.Run` advances, plus observability events.
A failed attempt cannot say which step failed or what its children did
without replaying the event stream.
- `Store.Transition` guards on the expected `from` status, but any `from` to
any `to` is accepted. A `queued` to `merged` write succeeds.
- Stop cancels one context in `daemon.runningTasks`. Nothing persisted says
which work was in flight, so an agent call that outlives its stage (a
reviewer, a container) is invisible to the store.
- `workflow.Run` writes `tc.Store.Update` and discards the error, then emits
`KindStageStart` separately. State and events can diverge.
## Model
```text
WorkflowExecution (tasks row, id E)
└── attempt N
├── StepExecution stage "plan" depth 0
├── StepExecution stage "implement" depth 0
│ └── StepExecution agent "implement" depth 1
└── StepExecution stage "review" depth 0
├── StepExecution agent "review:1" depth 1
└── StepExecution agent "review:2" depth 1
```
- A StepExecution belongs to exactly one WorkflowExecution and one attempt.
`parent_id` is null for a stage and names the enclosing StepExecution
otherwise. Depth is derived from the parent, never supplied by the caller.
- `kind` is `stage`, `agent` or `call`. A new kind needs a named consumer
before it is added.
- A `workflow.call` step (`docs/prds/event-automation.md`) is a `call`
StepExecution in the caller's tree. The callee is its own
WorkflowExecution, with its own tree, acting as its own identity, and the
`call` step records the callee's execution ID. Trees never nest across
executions, so each run's tree stays inside that run's authority.
- A retry starts a new attempt. Earlier attempts' StepExecutions are never
rewritten.
## Transition tables
WorkflowExecution statuses are the existing `taskstate` constants, persisted
strings unchanged. `taskstate` gains the one table every writer checks:
| From | To |
| --------------- | ---------------------------------------------------------- |
| `queued` | `running`, `closed_wont_do` |
| `running` | `queued` (crash recovery), `waiting_human`, `pr_open`, `completed`, `parked`, `closed_wont_do` |
| `waiting_human` | `queued` (approve), `closed_wont_do` |
| `parked` | `queued` (retry), `dead` (abandon), `closed_wont_do` |
| `pr_open` | `merged`, `rejected`, `queued` (remediation), `closed_wont_do` |
| terminal | none |
StepExecution statuses:
| From | To |
| --------- | ------------------------------------------------------ |
| `pending` | `running`, `cancelled` |
| `running` | `succeeded`, `failed`, `cancelled`, `interrupted` |
| terminal | none |
`interrupted` means the process running the step died or shut down. It is
distinct from `cancelled`, which an operator or a parent asked for, and from
`failed`, which the step reported.
A StepExecution cannot reach `running` while its WorkflowExecution is not
`running`, and cannot start under a terminal parent.
## Enforcement
- The table lives in `taskstate` (no dependencies), so the store, daemon and
gateway share one copy.
- Every store method that changes `tasks.status` checks the table and writes
the audit and event rows, not only `Transition`. That includes the claim,
`ParkTask`, `Requeue`, `RetryTask`, `BeginRemediation` and `RecoverStale`
paths, several of which change status without an audit row.
- The State Store rejects an illegal transition with a new sentinel,
`ErrIllegalTransition`, carried across the wire through `mapError` like
`ErrStaleTransition`. A stale `from` stays `ErrStaleTransition`.
- Each transition is one transaction: guarded row update, audit row, domain
event row. The events table is written by the store, not emitted beside it.
`KindStageStart` and `KindStageFinish` are produced by the step transition
and published to the bus after commit.
- `workflow.Run` stops discarding persistence errors. A failed step write
parks the execution; the step does not run unrecorded.
## Cancellation
Stopping a run is one store call, `CancelExecution(E, reason)`. In one
transaction it moves every non-terminal StepExecution of the current attempt
to `cancelled` and the WorkflowExecution to `closed_wont_do` or `parked`,
whichever the operator action names. The daemon then cancels the in-memory
context. A worker whose step write returns `ErrStaleTransition` because its
step is already `cancelled` stops without writing further.
Cancelling a caller cancels each callee it is waiting on (`wait: true`),
through `CancelExecution` on the callee under the callee's own lifecycle. A
callee started with `wait: false` runs on.
The store is the record of cancellation. The context cancel is the delivery
mechanism, so a lost cancel still leaves the store correct and the worker's
next write fails.
## Crash recovery
`RecoverStale` moves every `running` StepExecution of an interrupted
execution to `interrupted` in the same transaction that requeues the
execution. The next attempt starts with fresh steps.
## Wire contract
New State Store RPCs, each with a matching Go method on a narrow consumer
facade:
- `StartStep(execution, attempt, parent, kind, name) -> step id`
- `FinishStep(step, from, to, detail, tokens_used)`
- `CancelExecution(execution, reason, to)`
- `ListSteps(execution, attempt)`
The run credential of `docs/prds/orgs-and-access.md` authorises `StartStep`
and `FinishStep` on its own execution only, and the container records its own
agent calls with it. `CancelExecution` and `ListSteps` are dashboard, API and
dispatch actions, decided by the Authorizer: `update` and `read` on the run.
## Organisations
`docs/prds/orgs-and-access.md` is the boundary. A StepExecution belongs to its
WorkflowExecution's org and workspace and carries their `org_id` and
`workspace_id`, like every owned record. The State Store stamps both from the
credential on `StartStep` and ignores them in the request. `ListSteps` for an
execution in another org returns the store's ordinary not-found error.
## Storage
One new table, `step_executions`: `id`, `org_id`, `workspace_id`,
`execution_id`, `attempt`, `parent_id`, `depth`, `kind`, `name`,
`called_execution_id`, `status`, `detail`, `tokens_used`, `started_at`,
`finished_at`. Indexed on `(execution_id, attempt, id)`.
Deleting an execution (archive, clearing terminal tasks) deletes its steps.
Existing rows gain no backfilled steps. Their history stays in `transitions`
and `events`.
`tasks.stage` is written from the step transition while any reader still uses
it, then dropped once the dashboard reads `ListSteps`.
## Out of scope
- Running sibling steps in parallel. The tree records concurrency; changing
the stage runner is separate work.
- Cost in currency. `tokens_used` per step is the only usage field.
- Workflow definition versioning and the `tasks` to `workflow_executions`
rename.
- A harness contract for swappable coding agents.
## Plan
1. `taskstate` transition tables with table-driven tests covering every
legal and a sample of illegal pairs.
2. `ErrIllegalTransition` enforced in the Postgres store and mapped over the
wire.
3. `step_executions` migration, `StartStep`/`FinishStep`, grant rules;
`workflow.Run` records stages; events written in the transition
transaction.
4. Agent-call children recorded from the agent runtime.
5. `CancelExecution`; daemon stop and chat/dashboard actions route through it.
6. `RecoverStale` interrupts running steps.
7. Dashboard run detail reads `ListSteps`; `tasks.stage` dropped.
## Verification
- An illegal WorkflowExecution or StepExecution transition returns
`ErrIllegalTransition` over gRPC, and `errors.Is` matches on the client.
- Killing the daemon mid-stage, then restarting, leaves that stage
`interrupted` and the execution `queued` with a new attempt.
- Stopping a run during review leaves every review child `cancelled`, and a
late `FinishStep` from the container returns `ErrStaleTransition`.
- A run credential cannot `StartStep` on another execution, and a member of
org A listing steps of an org B execution gets not-found.
- Cancelling a caller cancels its `wait: true` callee and leaves a
`wait: false` callee running.
- For every step transition there is exactly one event row, written in the
same transaction.