180 lines
8.2 KiB
Markdown
180 lines
8.2 KiB
Markdown
# Execution tree and enforced state machine
|
|
|
|
**Status:** Approved
|
|
**Authority:** `docs/architecture/agent-system.md` (canonical workflow model);
|
|
settles the "enforced WorkflowExecution state machine" item in
|
|
`docs/architecture/migration-decisions.md` section 4.
|
|
**Compounds with:** `docs/prds/task-run-detail.md` (attempt attribution),
|
|
`docs/prds/state-store-contract.md` (wire contract and task-scoped grants).
|
|
|
|
## Decision
|
|
|
|
A WorkflowExecution becomes the root of a tree of StepExecutions. Every stage
|
|
run, and every agent call a stage makes, is its own durable record with a
|
|
parent, an attempt, a status and an outcome. Both levels move only through a
|
|
declared transition table, enforced by the State Store in the same transaction
|
|
that writes the audit row and the domain event.
|
|
|
|
## Problem
|
|
|
|
- A run is one `tasks` row. The only per-step record is the mutable `stage`
|
|
column, overwritten as `workflow.Run` advances, plus observability events.
|
|
A failed attempt cannot say which step failed or what its children did
|
|
without replaying the event stream.
|
|
- `Store.Transition` guards on the expected `from` status, but any `from` to
|
|
any `to` is accepted. A `queued` to `merged` write succeeds.
|
|
- Stop cancels one context in `daemon.runningTasks`. Nothing persisted says
|
|
which work was in flight, so an agent call that outlives its stage (a
|
|
reviewer, a container) is invisible to the store.
|
|
- `workflow.Run` writes `tc.Store.Update` and discards the error, then emits
|
|
`KindStageStart` separately. State and events can diverge.
|
|
|
|
## Model
|
|
|
|
```text
|
|
WorkflowExecution (tasks row, id E)
|
|
└── attempt N
|
|
├── StepExecution stage "plan" depth 0
|
|
├── StepExecution stage "implement" depth 0
|
|
│ └── StepExecution agent "implement" depth 1
|
|
└── StepExecution stage "review" depth 0
|
|
├── StepExecution agent "review:1" depth 1
|
|
└── StepExecution agent "review:2" depth 1
|
|
```
|
|
|
|
- A StepExecution belongs to exactly one WorkflowExecution and one attempt.
|
|
`parent_id` is null for a stage and names the enclosing StepExecution
|
|
otherwise. Depth is derived from the parent, never supplied by the caller.
|
|
- `kind` is `stage` or `agent`. A new kind needs a named consumer before it is
|
|
added.
|
|
- A retry starts a new attempt. Earlier attempts' StepExecutions are never
|
|
rewritten.
|
|
|
|
## Transition tables
|
|
|
|
WorkflowExecution statuses are the existing `taskstate` constants, persisted
|
|
strings unchanged. `taskstate` gains the one table every writer checks:
|
|
|
|
| From | To |
|
|
| --------------- | ---------------------------------------------------------- |
|
|
| `queued` | `running`, `closed_wont_do` |
|
|
| `running` | `queued` (crash recovery), `waiting_human`, `pr_open`, `completed`, `parked`, `closed_wont_do` |
|
|
| `waiting_human` | `queued` (approve), `closed_wont_do` |
|
|
| `parked` | `queued` (retry), `dead` (abandon), `closed_wont_do` |
|
|
| `pr_open` | `merged`, `rejected`, `queued` (remediation), `closed_wont_do` |
|
|
| terminal | none |
|
|
|
|
StepExecution statuses:
|
|
|
|
| From | To |
|
|
| --------- | ------------------------------------------------------ |
|
|
| `pending` | `running`, `cancelled` |
|
|
| `running` | `succeeded`, `failed`, `cancelled`, `interrupted` |
|
|
| terminal | none |
|
|
|
|
`interrupted` means the process running the step died or shut down. It is
|
|
distinct from `cancelled`, which an operator or a parent asked for, and from
|
|
`failed`, which the step reported.
|
|
|
|
A StepExecution cannot reach `running` while its WorkflowExecution is not
|
|
`running`, and cannot start under a terminal parent.
|
|
|
|
## Enforcement
|
|
|
|
- The table lives in `taskstate` (no dependencies), so the store, daemon and
|
|
gateway share one copy.
|
|
- Every store method that changes `tasks.status` checks the table and writes
|
|
the audit and event rows, not only `Transition`. That includes the claim,
|
|
`ParkTask`, `Requeue`, `RetryTask`, `BeginRemediation` and `RecoverStale`
|
|
paths, several of which change status without an audit row.
|
|
- The State Store rejects an illegal transition with a new sentinel,
|
|
`ErrIllegalTransition`, carried across the wire through `mapError` like
|
|
`ErrStaleTransition`. A stale `from` stays `ErrStaleTransition`.
|
|
- Each transition is one transaction: guarded row update, audit row, domain
|
|
event row. The events table is written by the store, not emitted beside it.
|
|
`KindStageStart` and `KindStageFinish` are produced by the step transition
|
|
and published to the bus after commit.
|
|
- `workflow.Run` stops discarding persistence errors. A failed step write
|
|
parks the execution; the step does not run unrecorded.
|
|
|
|
## Cancellation
|
|
|
|
Stopping a run is one store call, `CancelExecution(E, reason)`. In one
|
|
transaction it moves every non-terminal StepExecution of the current attempt
|
|
to `cancelled` and the WorkflowExecution to `closed_wont_do` or `parked`,
|
|
whichever the operator action names. The daemon then cancels the in-memory
|
|
context. A worker whose step write returns `ErrStaleTransition` because its
|
|
step is already `cancelled` stops without writing further.
|
|
|
|
The store is the record of cancellation. The context cancel is the delivery
|
|
mechanism, so a lost cancel still leaves the store correct and the worker's
|
|
next write fails.
|
|
|
|
## Crash recovery
|
|
|
|
`RecoverStale` moves every `running` StepExecution of an interrupted
|
|
execution to `interrupted` in the same transaction that requeues the
|
|
execution. The next attempt starts with fresh steps.
|
|
|
|
## Wire contract
|
|
|
|
New State Store RPCs, each with a matching Go method on a narrow consumer
|
|
facade:
|
|
|
|
- `StartStep(execution, attempt, parent, kind, name) -> step id`
|
|
- `FinishStep(step, from, to, detail, tokens_used)`
|
|
- `CancelExecution(execution, reason, to)`
|
|
- `ListSteps(execution, attempt)`
|
|
|
|
`TaskGrants.UnaryInterceptor` authorises a task-scoped token for `StartStep`
|
|
and `FinishStep` on its own execution only. The container records its own
|
|
agent calls. `CancelExecution` and `ListSteps` require the administrative
|
|
token.
|
|
|
|
## Storage
|
|
|
|
One new table, `step_executions`: `id`, `execution_id`, `attempt`,
|
|
`parent_id`, `depth`, `kind`, `name`, `status`, `detail`, `tokens_used`,
|
|
`started_at`, `finished_at`. Indexed on `(execution_id, attempt, id)`.
|
|
Deleting an execution (archive, clearing terminal tasks) deletes its steps.
|
|
Existing rows gain no backfilled steps. Their history stays in `transitions`
|
|
and `events`.
|
|
|
|
`tasks.stage` is written from the step transition while any reader still uses
|
|
it, then dropped once the dashboard reads `ListSteps`.
|
|
|
|
## Out of scope
|
|
|
|
- Running sibling steps in parallel. The tree records concurrency; changing
|
|
the stage runner is separate work.
|
|
- Cost in currency. `tokens_used` per step is the only usage field.
|
|
- Workflow definition versioning and the `tasks` to `workflow_executions`
|
|
rename.
|
|
- A harness contract for swappable coding agents.
|
|
|
|
## Plan
|
|
|
|
1. `taskstate` transition tables with table-driven tests covering every
|
|
legal and a sample of illegal pairs.
|
|
2. `ErrIllegalTransition` enforced in the Postgres store and mapped over the
|
|
wire.
|
|
3. `step_executions` migration, `StartStep`/`FinishStep`, grant rules;
|
|
`workflow.Run` records stages; events written in the transition
|
|
transaction.
|
|
4. Agent-call children recorded from the agent runtime.
|
|
5. `CancelExecution`; daemon stop and chat/dashboard actions route through it.
|
|
6. `RecoverStale` interrupts running steps.
|
|
7. Dashboard run detail reads `ListSteps`; `tasks.stage` dropped.
|
|
|
|
## Verification
|
|
|
|
- An illegal WorkflowExecution or StepExecution transition returns
|
|
`ErrIllegalTransition` over gRPC, and `errors.Is` matches on the client.
|
|
- Killing the daemon mid-stage, then restarting, leaves that stage
|
|
`interrupted` and the execution `queued` with a new attempt.
|
|
- Stopping a run during review leaves every review child `cancelled`, and a
|
|
late `FinishStep` from the container returns `ErrStaleTransition`.
|
|
- A task-scoped token cannot `StartStep` on another execution.
|
|
- For every step transition there is exactly one event row, written in the
|
|
same transaction.
|