feat(statestore): step_executions, StartStep/FinishStep under the run credential

Plan item 3 of docs/prds/execution-tree-state-machine.md (archie-core-1220), on
top of the transition tables and ErrIllegalTransition enforcement (1214/1215,
archie-core-qdbo.5):

- Migration 0017 adds step_executions -- id, org_id, workspace_id, execution_id,
  attempt, parent_id, depth, kind, name, called_execution_id, status, detail,
  tokens_used, started_at, finished_at, indexed on (execution_id, attempt, id).
  Ownership fields are stamped from the execution's row; depth is derived from
  the parent, never supplied by a caller. Existing rows gain no backfilled
  steps.
- task.Store (the workflow engine's mid-run facade) gains StartStep/FinishStep
  (5 methods, within the 8 cap), with storecontract.StepRecorder fronting them
  on the server; deadlineStore wraps both with its per-call timeout.
- The store enforces each write as one transaction: the guarded row write
  under the shared step transition table plus the transition's own
  stage_start/stage_finish event row. Staleness is decided before legality,
  the way guardTransition decides it, and the store returns the persisted
  event for the caller's post-commit publish -- the EmitDurable convention
  (the daemon's event sink skips already-persisted rows by assigned ID), which
  is why workflow.Run stops emitting stage events beside the store and no
  longer discards step-write errors: a failed start or finish parks the
  execution, and the stage does not run unrecorded. An interrupted stage
  records interrupted rather than failed; a recording failure on that
  shutdown path is logged and left to RecoverStale, which owns the
  interrupted-execution sweep in plan item 6.
- The run credential of docs/prds/orgs-and-access.md authorises StartStep and
  FinishStep on its own execution only: both requests carry the execution ID
  the grant check verifies (FinishStep carries it because the interceptor
  never reads the database), and FinishStep verifies handler-side that the
  step really is that execution's. authorizesTaskScopedCall's sanctioned
  widenings are now held as a lookup table so the next one stops growing a
  switch.
- The PRD's wire contract is StartStep(execution, attempt, parent, kind, name)
  -> step id; the responses also carry the persisted event (needed for the
  post-commit publish) and FinishStep's request carries execution_id (needed
  for the grant check) -- both noted here as deliberate additions to the PRD's
  shorthand. called_execution_id exists in the table but not yet on the wire:
  its named consumer is the workflow-call recording of plan item 4 (#1216).

Conformance drives both adapters; grant tests pin the own-execution scoping;
the wire sentinel set gains ErrInvalidStep (InvalidArgument, "step execution
invalid") for a step kind no consumer names.
This commit is contained in:
sam committed 2026-09-27 05:42:32 +10:00
1 parent 7aa01c7cfd
commit fa4566fb0d
29 files changed
+2637 -937

No files matched your search

+1 -1
View File
@@ -484,7 +484,7 @@
"url": "/dev/prds/state-store-contract/",
"title": "State Store — contract boundary \u0026 transport (ratification)",
"section": "PRDs",
"body": "# State Store — contract boundary \u0026 transport (ratification)\n\n**Status:** Approved\n**Revision:** 2e\n**Date:** 2026-09-06\n\nThis is the pre-implementation design document that Phase 2 subtasks `.4.2` (generalise\n`storerpc` as the State Store gRPC contract) and `.4.3` (stand up `archie-state-store`)\nimplement against.\n**Beads milestone:** archie-core-8cda.4.1\n**Parent:** `docs/prds/service-decomposition.md` (open-question Q4 **RESOLVED**)\n\nThis document fixes the **contract boundary** and **transport** for the State Store service.\nIt does not redesign Archie's domain model, does not invent a second store, and does **not**\nsplit `archie.db`. It records method ownership, DTO/wire representation, error semantics,\nlimits, mode/config shape, local-adapter compatibility, and cutover/deletion rules so that\n`.4.2` and `.4.3` have exactly one authoritative surface to build against.\n\n\u003e **Storage superseded.** The store behind this contract is PostgreSQL:\n\u003e `internal/infrastructure/postgres` implements the producer-owned interfaces in\n\u003e `internal/domain/storecontract` and `workflow.Store`, over the database\n\u003e `database_url` names. `internal/store`, the SQLite file and the local adapter\n\u003e are gone; where this document says `archie.db`, `*store.Store` or\n\u003e `internal/store`, read the State Store's PostgreSQL database and\n\u003e `internal/infrastructure/postgres`. The contract, ownership split, error\n\u003e semantics and cutover rules stand.\n\n\u003e **Current state — read before `.4.2` starts.** The §12 step 2 relocation (`archie-core-8cda.\n4.8`) has landed: `Task`/`Status*`/`Source*` and the 3-method `workflow.Store` now live in\n\u003e `internal/domain/workflow`, the six production workflow files import no `internal/store`\n\u003e symbol, and `storerpc`/the NATS transport assert against `workflow.Store` instead of\n\u003e `store.WorkflowStore`. The broader Phase 2 acceptance criterion (State Store as its own\n\u003e service behind gRPC) is still open — that is what `.4.2`/`.4.3` build next.\n\n\u003e **Relationship to migration-decisions §4 — not replaced.** `docs/architecture/\nmigration-decisions.md` §4 (`store.Task → WorkflowExecution`, `mutable Stage →\nStepExecution history`, `RetryCount/Attempt → Attempt records`, versioned Workflow defs)\n\u003e **remains the authoritative, open document for the long-term Workflow migration.** This PRD\n\u003e does **not** replace it. It only pulls the minimal `Task`/`Status`/`Source` type + interface\n\u003e relocation forward as a Phase 2 prerequisite (because the acceptance criterion requires it);\n\u003e the rest of §4 stays its own migration and stays open.\n\n---\n\n## 1. Decision (the invariant everything follows)\n\nArchie is composed such that a single **`archie-state-store`** service owns the one existing\nSQLite file (`archie.db`) and exposes it over **gRPC behind narrow typed contracts**. There is\nexactly one store, one file, one service. No database-per-service, no distributed\ntransactions, no second store abstraction.\n\nTwo adapter families satisfy the contract interfaces:\n\n- **Local adapter** — `*store.Store` (the in-process SQLite implementation).\n- **Remote adapter** — `*staterpc.Client` (wraps the generated gRPC client).\n\nEach adapter carries an explicit compile-time assertion against the contract interfaces it\nserves. Composition picks local vs. remote by the presence of `[services.state].target`\n(empty = local; set = dial the gRPC service). Default is **local** (the State Store is not yet\nextracted), which is the PRD §3 posture; it differs from the gateway's remote-default only\nbecause the gateway is already a separate deployment and the State Store is not.\n\n\u003e **Contract-ownership fork, decided (rev. 2c — resolves the domain contradiction).**\n\u003e Ownership splits by the consumer's _layer_, per the dependency contract's rule #2\n\u003e (\"a domain defines the smallest interfaces required to perform its work\") and rule #7\n\u003e (\"database rows … MUST NOT leak into domain models\"):\n\u003e\n\u003e - The **workflow domain** (`internal/domain/workflow`) is the only domain-layer consumer of\n\u003e store state. It defines its own **consumer-owned** narrow `Store` interface\n\u003e (Update/Transition/InsertEvent) and its own **domain-owned** `Task` type plus\n\u003e `Status`/`Source` vocabulary. It imports **zero** `internal/store` symbols — satisfying the\n\u003e Phase 2 acceptance criterion. This is the minimal bound of migration-decisions §4\n\u003e (`store.Task → WorkflowExecution`), pulled forward **because** the acceptance criterion\n\u003e requires it.\n\u003e - The **daemon / webui / intake** store surfaces (`TaskStore`, `CaptureStore`,\n\u003e `MappingStore`, `BindingStore`, `BindingDispatcher`, `BindingTaskCreator`) stay\n\u003e **producer-owned** in `internal/store`, because those consumers are application Line truncated
"body": "# State Store — contract boundary \u0026 transport (ratification)\n\n**Status:** Approved\n**Revision:** 2e\n**Date:** 2026-09-06\n\nThis is the pre-implementation design document that Phase 2 subtasks `.4.2` (generalise\n`storerpc` as the State Store gRPC contract) and `.4.3` (stand up `archie-state-store`)\nimplement against.\n**Beads milestone:** archie-core-8cda.4.1\n**Parent:** `docs/prds/service-decomposition.md` (open-question Q4 **RESOLVED**)\n\nThis document fixes the **contract boundary** and **transport** for the State Store service.\nIt does not redesign Archie's domain model, does not invent a second store, and does **not**\nsplit `archie.db`. It records method ownership, DTO/wire representation, error semantics,\nlimits, mode/config shape, local-adapter compatibility, and cutover/deletion rules so that\n`.4.2` and `.4.3` have exactly one authoritative surface to build against.\n\n\u003e **Storage superseded.** The store behind this contract is PostgreSQL:\n\u003e `internal/infrastructure/postgres` implements the producer-owned interfaces in\n\u003e `internal/domain/storecontract` and `workflow.Store`, over the database\n\u003e `database_url` names. `internal/store`, the SQLite file and the local adapter\n\u003e are gone; where this document says `archie.db`, `*store.Store` or\n\u003e `internal/store`, read the State Store's PostgreSQL database and\n\u003e `internal/infrastructure/postgres`. The contract, ownership split, error\n\u003e semantics and cutover rules stand.\n\n\u003e **Current state — read before `.4.2` starts.** The §12 step 2 relocation (`archie-core-8cda.\n4.8`) has landed: `Task`/`Status*`/`Source*` and the 3-method `workflow.Store` now live in\n\u003e `internal/domain/workflow`, the six production workflow files import no `internal/store`\n\u003e symbol, and `storerpc`/the NATS transport assert against `workflow.Store` instead of\n\u003e `store.WorkflowStore`. The broader Phase 2 acceptance criterion (State Store as its own\n\u003e service behind gRPC) is still open — that is what `.4.2`/`.4.3` build next.\n\n\u003e **Relationship to migration-decisions §4 — not replaced.** `docs/architecture/\nmigration-decisions.md` §4 (`store.Task → WorkflowExecution`, `mutable Stage →\nStepExecution history`, `RetryCount/Attempt → Attempt records`, versioned Workflow defs)\n\u003e **remains the authoritative, open document for the long-term Workflow migration.** This PRD\n\u003e does **not** replace it. It only pulls the minimal `Task`/`Status`/`Source` type + interface\n\u003e relocation forward as a Phase 2 prerequisite (because the acceptance criterion requires it);\n\u003e the rest of §4 stays its own migration and stays open.\n\n---\n\n## 1. Decision (the invariant everything follows)\n\nArchie is composed such that a single **`archie-state-store`** service owns the one existing\nSQLite file (`archie.db`) and exposes it over **gRPC behind narrow typed contracts**. There is\nexactly one store, one file, one service. No database-per-service, no distributed\ntransactions, no second store abstraction.\n\nTwo adapter families satisfy the contract interfaces:\n\n- **Local adapter** — `*store.Store` (the in-process SQLite implementation).\n- **Remote adapter** — `*staterpc.Client` (wraps the generated gRPC client).\n\nEach adapter carries an explicit compile-time assertion against the contract interfaces it\nserves. Composition picks local vs. remote by the presence of `[services.state].target`\n(empty = local; set = dial the gRPC service). Default is **local** (the State Store is not yet\nextracted), which is the PRD §3 posture; it differs from the gateway's remote-default only\nbecause the gateway is already a separate deployment and the State Store is not.\n\n\u003e **Contract-ownership fork, decided (rev. 2c — resolves the domain contradiction).**\n\u003e Ownership splits by the consumer's _layer_, per the dependency contract's rule #2\n\u003e (\"a domain defines the smallest interfaces required to perform its work\") and rule #7\n\u003e (\"database rows … MUST NOT leak into domain models\"):\n\u003e\n\u003e - The **workflow domain** (`internal/domain/workflow`) is the only domain-layer consumer of\n\u003e store state. It defines its own **consumer-owned** narrow `Store` interface\n\u003e (Update/Transition/InsertEvent) and its own **domain-owned** `Task` type plus\n\u003e `Status`/`Source` vocabulary. It imports **zero** `internal/store` symbols — satisfying the\n\u003e Phase 2 acceptance criterion. This is the minimal bound of migration-decisions §4\n\u003e (`store.Task → WorkflowExecution`), pulled forward **because** the acceptance criterion\n\u003e requires it.\n\u003e - The **daemon / webui / intake** store surfaces (`TaskStore`, `CaptureStore`,\n\u003e `MappingStore`, `BindingStore`, `BindingDispatcher`, `BindingTaskCreator`) stay\n\u003e **producer-owned** in `internal/store`, because those consumers are application Line truncated
},
{
"url": "/dev/prds/status-health-surface/",
+14 -7
View File
@@ -266,13 +266,14 @@ _view_, not new RPCs — `Update`/`Transition`/`InsertEvent` are shared with
| Dispatch | `ArmedBindingsForSource`, `RecordDispatch`, `ListUndispatchedCaptures` |
| Playbook | `RecordPlaybookDispatch`, `DeletePlaybookDispatches` |
| WorkflowCaller | `EnqueueCallTask`, `WorkflowCallStatus` |
| Step executions | `StartStep`, `FinishStep` |
| BindingTaskCreator | `EnqueueBindingTask` |
| Config snapshot | `PutConfigSnapshot`, `GetConfigSnapshot` |
| Apply status | `PutApplyStatus`, `ListApplyStatus` |
| Channel status | `PutChannelStatus`, `ListChannelStatus` |
| Task log | `ReadTaskLog`, `StreamTaskLogContent` |
That is **53 unique contract RPCs** across one service (the `TaskEvents.Close` method is dropped).
That is **55 unique contract RPCs** across one service (the `TaskEvents.Close` method is dropped).
`Close()` is **excluded** from the wire (it is server lifecycle, not a client call) — see §11.
The channel-status pair follows the config snapshot's split for the same reason: the writer is the process that HOSTS the channels -- the Messaging Service, which alone observes whether a channel is starting, running or failed -- and the reader is the UI process. Both RPCs are administrative, so a task-scoped grant reaches neither. A channel's state is runtime state rather than a settings document, which is why it is not a `ControlPlaneService` resource kind: a failed channel is not a document anybody edits. Asking for a reload travels on its own surface rather than through this one; see `docs/architecture/migration-decisions.md`, "Channel state to the dashboard".
@@ -307,12 +308,18 @@ condition, and the panel that collapsed them told operators task logging was swi
it was not. The Go facade is `storecontract.TaskLogStore` (2 methods), carried by
`*staterpc.Client` alongside the rest.
The workflow-caller pair was added by `archie-core-t2db.41` (see
`docs/prds/workflow-calls.md`). It is the one surface a task-scoped grant
reaches beyond the three task.Store RPCs: a `workflow.call` step must start
its callee and, with `wait: true`, read it back, and the callee is the
caller's child. Both requests carry the caller's task ID, which the grant
check verifies; `EnqueueCallTask` derives everything else from the caller's
The step-execution pair was added by the execution tree's plan item 3 (see
`docs/prds/execution-tree-state-machine.md`), the surface a task-scoped grant reaches for
its own execution beyond the three `task.Store` RPCs and the workflow-caller pair: the
container records its stages and its own agent calls with the run credential. Both requests
carry the caller's execution ID, which the grant check verifies; `FinishStep` additionally
verifies, handler-side, that the step really is that execution's. The store writes each
transition's `stage_start`/`stage_finish` event row in the same transaction and returns it
for the caller's post-commit bus publish. `called_execution_id` is stored for the `call`
kind but not yet carried on the wire: the named consumer is the workflow-call recording
that will set it.
The workflow-caller pair was added by `archie-core-t2db.41` (see everything else from the caller's
row and refuses a caller that is not running or a call past the depth limit
(`ErrCallCallerNotRunning` / `ErrCallDepthExceeded`), and `WorkflowCallStatus`
re-checks the parent-child relation against the row before answering
@@ -383,6 +383,7 @@ func startStateStoreGRPC(t *testing.T, local *pgstore.TaskDB) string {
staterpc.RegisterServer(server, staterpc.Deps{
Tasks: local, Captures: eda, Mappings: eda, Bindings: eda,
BindingDispatcher: eda, BindingTaskCreator: local, WorkflowCalls: local,
Steps: local,
})
go func() { _ = server.Serve(listener) }()
t.Cleanup(func() { server.Stop(); _ = listener.Close() })
+3
View File
@@ -346,6 +346,9 @@ func (b *boot) stateStoreDeps(grants *staterpc.TaskGrants) staterpc.Deps {
if wc, ok := b.st.(storecontract.WorkflowCaller); ok {
deps.WorkflowCalls = wc
}
if sr, ok := b.st.(storecontract.StepRecorder); ok {
deps.Steps = sr
}
b.accessDeps(&deps)
if css, ok := b.st.(storecontract.ConfigSnapshotStore); ok {
deps.ConfigSnapshots = css
File diff suppressed because it is too large. Load diff
@@ -40,6 +40,8 @@ const (
StateStoreService_ClaimByIssue_FullMethodName = "/state.v1.StateStoreService/ClaimByIssue"
StateStoreService_Transition_FullMethodName = "/state.v1.StateStoreService/Transition"
StateStoreService_Update_FullMethodName = "/state.v1.StateStoreService/Update"
StateStoreService_StartStep_FullMethodName = "/state.v1.StateStoreService/StartStep"
StateStoreService_FinishStep_FullMethodName = "/state.v1.StateStoreService/FinishStep"
StateStoreService_Requeue_FullMethodName = "/state.v1.StateStoreService/Requeue"
StateStoreService_ParkTask_FullMethodName = "/state.v1.StateStoreService/ParkTask"
StateStoreService_RecoverStale_FullMethodName = "/state.v1.StateStoreService/RecoverStale"
@@ -149,6 +151,10 @@ type StateStoreServiceClient interface {
ClaimByIssue(ctx context.Context, in *ClaimByIssueRequest, opts ...grpc.CallOption) (*ClaimByIssueResponse, error)
Transition(ctx context.Context, in *TransitionRequest, opts ...grpc.CallOption) (*TransitionResponse, error)
Update(ctx context.Context, in *UpdateRequest, opts ...grpc.CallOption) (*UpdateResponse, error)
// Step executions (docs/prds/execution-tree-state-machine.md). The run
// credential authorises both on its own execution only.
StartStep(ctx context.Context, in *StartStepRequest, opts ...grpc.CallOption) (*StartStepResponse, error)
FinishStep(ctx context.Context, in *FinishStepRequest, opts ...grpc.CallOption) (*FinishStepResponse, error)
Requeue(ctx context.Context, in *RequeueRequest, opts ...grpc.CallOption) (*RequeueResponse, error)
ParkTask(ctx context.Context, in *ParkTaskRequest, opts ...grpc.CallOption) (*ParkTaskResponse, error)
RecoverStale(ctx context.Context, in *RecoverStaleRequest, opts ...grpc.CallOption) (*RecoverStaleResponse, error)
@@ -492,6 +498,26 @@ func (c *stateStoreServiceClient) Update(ctx context.Context, in *UpdateRequest,
return out, nil
}
func (c *stateStoreServiceClient) StartStep(ctx context.Context, in *StartStepRequest, opts ...grpc.CallOption) (*StartStepResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(StartStepResponse)
err := c.cc.Invoke(ctx, StateStoreService_StartStep_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
func (c *stateStoreServiceClient) FinishStep(ctx context.Context, in *FinishStepRequest, opts ...grpc.CallOption) (*FinishStepResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(FinishStepResponse)
err := c.cc.Invoke(ctx, StateStoreService_FinishStep_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
func (c *stateStoreServiceClient) Requeue(ctx context.Context, in *RequeueRequest, opts ...grpc.CallOption) (*RequeueResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(RequeueResponse)
@@ -1229,6 +1255,10 @@ type StateStoreServiceServer interface {
ClaimByIssue(context.Context, *ClaimByIssueRequest) (*ClaimByIssueResponse, error)
Transition(context.Context, *TransitionRequest) (*TransitionResponse, error)
Update(context.Context, *UpdateRequest) (*UpdateResponse, error)
// Step executions (docs/prds/execution-tree-state-machine.md). The run
// credential authorises both on its own execution only.
StartStep(context.Context, *StartStepRequest) (*StartStepResponse, error)
FinishStep(context.Context, *FinishStepRequest) (*FinishStepResponse, error)
Requeue(context.Context, *RequeueRequest) (*RequeueResponse, error)
ParkTask(context.Context, *ParkTaskRequest) (*ParkTaskResponse, error)
RecoverStale(context.Context, *RecoverStaleRequest) (*RecoverStaleResponse, error)
@@ -1425,6 +1455,12 @@ func (UnimplementedStateStoreServiceServer) Transition(context.Context, *Transit
func (UnimplementedStateStoreServiceServer) Update(context.Context, *UpdateRequest) (*UpdateResponse, error) {
return nil, status.Error(codes.Unimplemented, "method Update not implemented")
}
func (UnimplementedStateStoreServiceServer) StartStep(context.Context, *StartStepRequest) (*StartStepResponse, error) {
return nil, status.Error(codes.Unimplemented, "method StartStep not implemented")
}
func (UnimplementedStateStoreServiceServer) FinishStep(context.Context, *FinishStepRequest) (*FinishStepResponse, error) {
return nil, status.Error(codes.Unimplemented, "method FinishStep not implemented")
}
func (UnimplementedStateStoreServiceServer) Requeue(context.Context, *RequeueRequest) (*RequeueResponse, error) {
return nil, status.Error(codes.Unimplemented, "method Requeue not implemented")
}
@@ -2025,6 +2061,42 @@ func _StateStoreService_Update_Handler(srv interface{}, ctx context.Context, dec
return interceptor(ctx, in, info, handler)
}
func _StateStoreService_StartStep_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(StartStepRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(StateStoreServiceServer).StartStep(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: StateStoreService_StartStep_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(StateStoreServiceServer).StartStep(ctx, req.(*StartStepRequest))
}
return interceptor(ctx, in, info, handler)
}
func _StateStoreService_FinishStep_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(FinishStepRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(StateStoreServiceServer).FinishStep(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: StateStoreService_FinishStep_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(StateStoreServiceServer).FinishStep(ctx, req.(*FinishStepRequest))
}
return interceptor(ctx, in, info, handler)
}
func _StateStoreService_Requeue_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(RequeueRequest)
if err := dec(in); err != nil {
@@ -3301,6 +3373,14 @@ var StateStoreService_ServiceDesc = grpc.ServiceDesc{
MethodName: "Update",
Handler: _StateStoreService_Update_Handler,
},
{
MethodName: "StartStep",
Handler: _StateStoreService_StartStep_Handler,
},
{
MethodName: "FinishStep",
Handler: _StateStoreService_FinishStep_Handler,
},
{
MethodName: "Requeue",
Handler: _StateStoreService_Requeue_Handler,
@@ -65,6 +65,16 @@ type TaskParker interface {
ParkTask(ctx context.Context, taskID int64, from, detail, class string) error
}
// StepRecorder records the StepExecutions of one running WorkflowExecution
// (docs/prds/execution-tree-state-machine.md). It is deliberately separate
// from TaskLifecycle, which is exactly at the interfacebloat cap: the same
// two methods also sit on the workflow engine's mid-run facade (task.Store),
// which is what a container's run credential calls.
type StepRecorder interface {
StartStep(ctx context.Context, s task.StepStart) (int64, events.Event, error)
FinishStep(ctx context.Context, s task.StepFinish) (events.Event, error)
}
// TaskArchiver removes one terminal task's local record with an optimistic
// status guard. It is separate from the already broad lifecycle contract so
// consumers that only run tasks do not acquire an operator-only capability.
@@ -415,6 +425,12 @@ var (
// it. Distinct from ErrStaleTransition, which covers a from that does
// not match the row's actual status.
ErrIllegalTransition = errors.New("store: illegal transition: status pair is not in the transition table")
// ErrInvalidStep is returned when a step-execution request names a kind
// the store does not record (docs/prds/execution-tree-state-machine.md:
// stage, agent or call; a new kind needs a named consumer before it is
// added) or a step whose identifying fields do not resolve. Distinct from
// the transition sentinels, which cover a row's state.
ErrInvalidStep = errors.New("store: invalid step execution request")
// ErrBindingNotFound is returned when a binding ID does not exist.
ErrBindingNotFound = errors.New("store: binding not found")
// ErrBindingOverlap is returned when a binding's matcher overlaps an
+35 -3
View File
@@ -21,9 +21,14 @@ import (
// events it was asked to persist. A capture is durable provenance, so the
// assertions below read this and not the lossy bus.
type recordingStore struct {
events []events.Event
transitions []transition
insertEr error
events []events.Event
transitions []transition
steps []StepStart
finishes []StepFinish
insertEr error
stepEr error
stepNames map[int64]string
stepAttempts map[int64]int
}
// transition is one status change the engine asked the store for. Parking is
@@ -46,6 +51,33 @@ func (s *recordingStore) InsertEvent(_ context.Context, e events.Event) (int64,
return int64(len(s.events)), nil
}
func (s *recordingStore) StartStep(_ context.Context, start StepStart) (int64, events.Event, error) {
if s.stepEr != nil {
return 0, events.Event{}, s.stepEr
}
s.steps = append(s.steps, start)
id := int64(len(s.steps))
if s.stepNames == nil {
s.stepNames = map[int64]string{}
s.stepAttempts = map[int64]int{}
}
s.stepNames[id] = start.Name
s.stepAttempts[id] = start.Attempt
return id, events.Event{ID: id, Kind: events.KindStageStart, Stage: start.Name, Attempt: start.Attempt}, nil
}
func (s *recordingStore) FinishStep(_ context.Context, finish StepFinish) (events.Event, error) {
if s.stepEr != nil {
return events.Event{}, s.stepEr
}
s.finishes = append(s.finishes, finish)
id := int64(len(s.finishes))
return events.Event{
ID: id, Kind: events.KindStageFinish,
Stage: s.stepNames[finish.StepID], Attempt: s.stepAttempts[finish.StepID],
}, nil
}
func (s *recordingStore) ofKind(kind string) []events.Event {
var out []events.Event
for _, e := range s.events {
+59
View File
@@ -0,0 +1,59 @@
package task
import "github.com/samcharles93/archie-core/internal/taskstate"
// The step-execution vocabulary of docs/prds/execution-tree-state-machine.md:
// one stage run, or one agent or workflow.call step an execution's tree
// records, started and finished through the State Store under the run's own
// credential.
// Step kinds. `stage` is what workflow.Run records for every stage it runs;
// `agent` is what the agent runtime records for one agent call; `call` is the
// workflow.call step in a caller's tree, which names its callee's execution.
// A new kind needs a named consumer before it is added.
const (
StepKindStage = "stage"
StepKindAgent = "agent"
StepKindCall = "call"
)
// ValidStepKind reports whether kind is one the store records.
func ValidStepKind(kind string) bool {
switch kind {
case StepKindStage, StepKindAgent, StepKindCall:
return true
default:
return false
}
}
// StepStart names the step a caller is starting. ExecutionID is the
// WorkflowExecution (the tasks row); Attempt is the run of it the step
// belongs to; ParentID is 0 for a stage and names the enclosing StepExecution
// otherwise, from which the store derives depth.
type StepStart struct {
ExecutionID int64
Attempt int
// ParentID is 0 when the step is a stage at the tree's root.
ParentID int64
Kind string
Name string
}
// StepFinish moves one step from From to its outcome. ExecutionID names the
// run the step belongs to: the task-scoped grant check verifies it, and the
// store verifies the step really is that execution's, so a caller cannot move
// another run's step by naming it.
type StepFinish struct {
StepID int64
ExecutionID int64
// From and To are the step transition table's statuses (taskstate).
From taskstate.StepStatus
To taskstate.StepStatus
// Detail is the step's outcome line: an error's text for a failed step, a
// summary otherwise. Clipped store-side like every detail.
Detail string
// TokensUsed is the usage the step itself accounted. Stages record none;
// the agent runtime records its own calls' usage.
TokensUsed int64
}
+11
View File
@@ -160,6 +160,17 @@ type Store interface {
Update(ctx context.Context, t *Task) error
Transition(ctx context.Context, taskID int64, from, to, detail string) error
InsertEvent(ctx context.Context, e events.Event) (int64, error)
// StartStep records a StepExecution entering running
// (docs/prds/execution-tree-state-machine.md): the store creates the row,
// enforces the step transition table and the execution's own status, and
// writes the transition's stage_start event in the same transaction. It
// returns the step's id and the persisted event, which the caller publishes
// to its bus after the write commits.
StartStep(ctx context.Context, s StepStart) (int64, events.Event, error)
// FinishStep moves one step to its outcome under the same table, writing
// the stage_finish event in the same transaction, and returns that event
// for the caller's post-commit publish.
FinishStep(ctx context.Context, s StepFinish) (events.Event, error)
}
// Definition is the operator-safe snapshot of an executable workflow. It
+12
View File
@@ -14,10 +14,22 @@ type (
// Store is the narrow, consumer-owned subset of the State Store
// contract that workflow stages call mid-run.
Store = task.Store
// StepStart and StepFinish are the step-execution writes the store
// records mid-run (docs/prds/execution-tree-state-machine.md).
StepStart = task.StepStart
StepFinish = task.StepFinish
// Definition is the operator-safe snapshot of an executable workflow.
Definition = task.Definition
)
// Step-execution kinds, defined beside the step writes in internal/domain/
// workflow/task with the same aliasing reason.
const (
StepKindStage = task.StepKindStage
StepKindAgent = task.StepKindAgent
StepKindCall = task.StepKindCall
)
// Task lifecycle statuses. Defined in internal/taskstate; these names are
// the workflow domain's public spelling.
const (
+67 -19
View File
@@ -7,7 +7,6 @@ package workflow
import (
"context"
"errors"
"fmt"
"log/slog"
"regexp"
@@ -20,6 +19,7 @@ import (
"github.com/samcharles93/archie-core/internal/domain/workflow/task"
"github.com/samcharles93/archie-core/internal/events"
"github.com/samcharles93/archie-core/internal/skill"
"github.com/samcharles93/archie-core/internal/taskstate"
"github.com/samcharles93/archie-core/internal/tools"
)
@@ -304,6 +304,20 @@ func Run(ctx context.Context, wf Workflow, tc *TaskContext) {
for _, stage := range wf.Stages {
t.Stage = stage.Name
_ = tc.Store.Update(ctx, t)
// Every stage is a StepExecution (docs/prds/execution-tree-state-machine.md):
// the store records pending -> running and writes the stage_start event
// in the same transaction, and refuses a start the state machine
// forbids. A failed recording write parks the execution -- the step
// does not run unrecorded, because a stage whose record never landed
// would park on a reason no dashboard can trace to a step row.
stepID, startEvent, err := tc.Store.StartStep(ctx, StepStart{
ExecutionID: t.ID, Attempt: t.Attempt, Kind: task.StepKindStage, Name: stage.Name,
})
if err != nil {
park(ctx, tc, fmt.Sprintf("stage %s could not be recorded: %v", stage.Name, err))
return
}
publishEvent(tc, startEvent)
// The stage is bound onto the task logger for exactly the stage's own
// execution and removed afterwards, so every line a stage's code writes
// is selectable by stage (internal/logging.Query.Stage). It is restored
@@ -315,28 +329,52 @@ func Run(ctx context.Context, wf Workflow, tc *TaskContext) {
stageLog := previousLog.With("stage", stage.Name)
tc.Log = stageLog
stageLog.Info("stage starting")
tc.Emit(events.KindStageStart, stage.Name, "", nil)
started := time.Now()
err := stage.Run(ctx, tc)
data := map[string]any{"duration_ms": time.Since(started).Milliseconds()}
if err != nil {
data["error"] = err.Error()
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
data["interrupted"] = true
}
}
tc.Emit(events.KindStageFinish, stage.Name, "", data)
err = stage.Run(ctx, tc)
tc.Log = previousLog
if err != nil {
// Daemon shutdown is not a workflow failure. Leave the task running
// so Startup's existing crash recovery requeues it; parking here
// would publish a false failure and require manual intervention.
if ctx.Err() != nil {
stageLog.Info("stage interrupted", "err", err)
return
// Daemon shutdown is not a workflow failure. The step records the
// table's interrupted outcome -- the process knows the step it is
// giving up -- and returns without parking: Startup's existing crash
// recovery requeues the execution, and the next attempt starts fresh
// steps. A recording failure on this path is logged and left to the
// same recovery: the process is on its way out either way.
if ctx.Err() != nil && err != nil {
if finishEvent, recordErr := tc.Store.FinishStep(ctx, StepFinish{
StepID: stepID, ExecutionID: t.ID,
From: taskstate.StepRunning, To: taskstate.StepInterrupted, Detail: err.Error(),
}); recordErr != nil {
stageLog.Warn("interrupted step finish could not be recorded", "stage", stage.Name, "err", recordErr)
} else {
publishEvent(tc, finishEvent)
}
stageLog.Info("stage interrupted", "err", err)
return
}
finishTo := taskstate.StepSucceeded
finishDetail := ""
if err != nil {
finishTo = taskstate.StepFailed
finishDetail = err.Error()
}
// The outcome event is the store's own stage_finish row, written in the
// FinishStep transaction -- not a second copy emitted beside it. A
// failed recording write parks the execution, like a failed start.
finishEvent, recordErr := tc.Store.FinishStep(ctx, StepFinish{
StepID: stepID, ExecutionID: t.ID,
From: taskstate.StepRunning, To: finishTo, Detail: finishDetail,
})
if recordErr != nil {
stageLog.Error("stage finish could not be recorded", "stage", stage.Name, "err", recordErr)
park(ctx, tc, fmt.Sprintf("stage %s could not be recorded: %v", stage.Name, recordErr))
return
}
publishEvent(tc, finishEvent)
if err != nil {
// Parking here publishes the failure; the store has already
// recorded the step that produced it.
t.ParkReason = fmt.Sprintf("stage %s: %v", stage.Name, err)
park(ctx, tc, t.ParkReason)
return
@@ -351,6 +389,16 @@ func Run(ctx context.Context, wf Workflow, tc *TaskContext) {
park(ctx, tc, "workflow ended without an outcome (definition bug)")
}
// publishEvent puts an already-persisted event on the run's bus after its
// write committed: the assigned ID is what the daemon's event sink reads to
// skip the second insert, the same convention EmitDurable established.
func publishEvent(tc *TaskContext, e events.Event) {
if tc.Bus == nil || e.ID == 0 {
return
}
tc.Bus.Publish(e)
}
func finish(ctx context.Context, tc *TaskContext, log *slog.Logger) {
t := tc.Task
if tc.Outcome.Status == StatusParked {
@@ -0,0 +1,116 @@
package workflow
import (
"context"
"errors"
"log/slog"
"slices"
"testing"
task "github.com/samcharles93/archie-core/internal/domain/workflow/task"
"github.com/samcharles93/archie-core/internal/events"
)
// The stage-recording half of docs/prds/execution-tree-state-machine.md:
// workflow.Run records every stage it runs as a StepExecution through the
// store, and a failed step write parks the execution -- the step does not run
// unrecorded.
// TestRunRecordsEveryStageAsAStepExecution pins the recording shape: one
// StartStep before each stage's body and one FinishStep with that stage's
// outcome, both of kind stage, against the task's own execution and attempt.
func TestRunRecordsEveryStageAsAStepExecution(t *testing.T) {
store := &recordingStore{}
stage := func(name string) Stage {
return Stage{Name: name, Run: func(context.Context, *TaskContext) error { return nil }}
}
wf := Workflow{Name: "recording", Stages: []Stage{stage("plan"), stage("implement")}}
tc := &TaskContext{
Task: &Task{ID: 41, Attempt: 3, Owner: "acme", Repo: "widgets", Status: StatusRunning},
Store: store, Log: slog.New(slog.DiscardHandler),
}
Run(context.Background(), wf, tc)
if len(store.steps) != 2 || len(store.finishes) != 2 {
t.Fatalf("steps = %+v finishes = %+v, want one recorded start and finish per stage", store.steps, store.finishes)
}
for i, step := range store.steps {
if step.ExecutionID != 41 || step.Attempt != 3 {
t.Errorf("stage %d: ExecutionID/Attempt = %d/%d, want the run's own 41/3", i, step.ExecutionID, step.Attempt)
}
if step.Kind != task.StepKindStage {
t.Errorf("stage %d: kind = %q, want %q", i, step.Kind, task.StepKindStage)
}
if want := []string{"plan", "implement"}[i]; step.Name != want {
t.Errorf("stage %d: name = %q, want %q", i, step.Name, want)
}
}
if got := store.finishes[0].To; got != "succeeded" {
t.Errorf("stage finish to = %q, want succeeded", got)
}
}
// TestRunPublishesTheRecordedEvents pins the post-commit half: the events the
// store returned, already persisted, are what reach the bus -- not a second,
// unpersisted copy emitted beside the store write.
func TestRunPublishesThePersistedStepEvents(t *testing.T) {
store := &recordingStore{}
wf := Workflow{Name: "recording", Stages: []Stage{{
Name: "implement", Run: func(context.Context, *TaskContext) error { return nil },
}}}
bus := events.NewBus()
sub := bus.Subscribe(16)
defer sub.Close()
tc := &TaskContext{
Task: &Task{ID: 7, Attempt: 1, Owner: "acme", Repo: "widgets", Status: StatusRunning},
Store: store, Bus: bus, Log: slog.New(slog.DiscardHandler),
}
Run(context.Background(), wf, tc)
var seen []string
ids := map[string]int64{}
for len(sub.C) > 0 {
e := <-sub.C
seen = append(seen, e.Kind)
ids[e.Kind] = e.ID
}
for _, want := range []string{events.KindStageStart, events.KindStageFinish} {
if !slices.Contains(seen, want) {
t.Errorf("bus saw %v, want %q among them: the persisted events are what reach the bus", seen, want)
}
if ids[want] == 0 {
t.Errorf("bus event %q carries no ID: an event without the store's assigned ID would be inserted a second time by the daemon's event sink", want)
}
}
}
// TestRunParksWhenAStepWriteFails is the PRD's persistence rule: a failed
// step write parks the execution, and the step does not run unrecorded.
func TestRunParksWhenAStepWriteFails(t *testing.T) {
store := &recordingStore{stepEr: errStepWrite}
stageRan := false
wf := Workflow{Name: "recording", Stages: []Stage{{
Name: "implement",
Run: func(context.Context, *TaskContext) error { stageRan = true; return nil },
}}}
tc := &TaskContext{
Task: &Task{ID: 9, Attempt: 1, Owner: "acme", Repo: "widgets", Status: StatusRunning},
Store: store, Log: slog.New(slog.DiscardHandler),
}
Run(context.Background(), wf, tc)
if stageRan {
t.Fatal("the stage ran despite its recording write failing; the step must not run unrecorded")
}
if len(store.transitions) == 0 || store.transitions[0].to != StatusParked {
t.Fatalf("transitions = %+v, want the execution parked", store.transitions)
}
if len(store.finishes) != 0 {
t.Errorf("finishes = %+v, want none: the stage never ran", store.finishes)
}
}
var errStepWrite = errors.New("recording write failed")
@@ -221,6 +221,18 @@ func (d deadlineStore) InsertEvent(ctx context.Context, e events.Event) (int64,
return d.Store.InsertEvent(ctx, e)
}
func (d deadlineStore) StartStep(ctx context.Context, s task.StepStart) (int64, events.Event, error) {
ctx, cancel := d.withDeadline(ctx)
defer cancel()
return d.Store.StartStep(ctx, s)
}
func (d deadlineStore) FinishStep(ctx context.Context, s task.StepFinish) (events.Event, error) {
ctx, cancel := d.withDeadline(ctx)
defer cancel()
return d.Store.FinishStep(ctx, s)
}
// Trees constructs the identity-scoped worktree RPC client.
func (t *Transport) Trees(identity, grant string, timeout time.Duration) RemoteTrees {
return &worktreerpc.Client{Conn: t.conn, Timeout: timeout, Identity: identity, Grant: grant}
@@ -0,0 +1,32 @@
-- Step executions: one durable record per stage run and per agent or
-- workflow.call step inside a WorkflowExecution
-- (docs/prds/execution-tree-state-machine.md). Statuses are taskstate's
-- StepStatus vocabulary and move only through the step transition table the
-- store enforces; depth is derived from parent_id, never supplied by a
-- caller. Existing rows gain no backfilled steps: their stage history stays
-- in transitions and events.
-- +goose Up
CREATE TABLE step_executions (
id bigint GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
org_id text NOT NULL DEFAULT 'default',
workspace_id text NOT NULL DEFAULT 'default',
execution_id bigint NOT NULL,
attempt bigint NOT NULL,
parent_id bigint,
depth int NOT NULL DEFAULT 0,
kind text NOT NULL,
name text NOT NULL,
called_execution_id bigint NOT NULL DEFAULT 0,
status text NOT NULL DEFAULT 'pending',
detail text NOT NULL DEFAULT '',
tokens_used bigint NOT NULL DEFAULT 0,
started_at timestamptz,
finished_at timestamptz
);
CREATE INDEX idx_step_executions_attempt ON step_executions (execution_id, attempt, id);
-- +goose Down
DROP TABLE step_executions;
@@ -306,6 +306,24 @@ type Source struct {
WorkspaceID string
}
type StepExecution struct {
ID int64
OrgID string
WorkspaceID string
ExecutionID int64
Attempt int64
ParentID pgtype.Int8
Depth int32
Kind string
Name string
CalledExecutionID int64
Status string
Detail string
TokensUsed int64
StartedAt pgtype.Timestamptz
FinishedAt pgtype.Timestamptz
}
type SysAudit struct {
ID int64
At time.Time
@@ -0,0 +1,165 @@
// Code generated by sqlc. DO NOT EDIT.
// versions:
// sqlc v1.31.1
// source: steps.sql
package postgresdb
import (
"context"
"github.com/jackc/pgx/v5/pgtype"
)
const finishStepExecution = `-- name: FinishStepExecution :execrows
UPDATE step_executions
SET status = $1, detail = $2, tokens_used = $3, finished_at = now()
WHERE id = $4 AND status = $5
`
type FinishStepExecutionParams struct {
Status string
Detail string
TokensUsed int64
ID int64
Status_2 string
}
func (q *Queries) FinishStepExecution(ctx context.Context, arg FinishStepExecutionParams) (int64, error) {
result, err := q.db.Exec(ctx, finishStepExecution,
arg.Status,
arg.Detail,
arg.TokensUsed,
arg.ID,
arg.Status_2,
)
if err != nil {
return 0, err
}
return result.RowsAffected(), nil
}
const insertStepExecution = `-- name: InsertStepExecution :one
INSERT INTO step_executions (org_id, workspace_id, execution_id, attempt, parent_id, depth, kind, name, called_execution_id, status)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, 'pending')
RETURNING id
`
type InsertStepExecutionParams struct {
OrgID string
WorkspaceID string
ExecutionID int64
Attempt int64
ParentID pgtype.Int8
Depth int32
Kind string
Name string
CalledExecutionID int64
}
func (q *Queries) InsertStepExecution(ctx context.Context, arg InsertStepExecutionParams) (int64, error) {
row := q.db.QueryRow(ctx, insertStepExecution,
arg.OrgID,
arg.WorkspaceID,
arg.ExecutionID,
arg.Attempt,
arg.ParentID,
arg.Depth,
arg.Kind,
arg.Name,
arg.CalledExecutionID,
)
var id int64
err := row.Scan(&id)
return id, err
}
const lockExecutionForStep = `-- name: LockExecutionForStep :one
SELECT id, status, org_id, workspace_id, owner, repo, issue_number, workflow, attempt
FROM tasks WHERE id = $1 FOR UPDATE
`
type LockExecutionForStepRow struct {
ID int64
Status string
OrgID string
WorkspaceID string
Owner string
Repo string
IssueNumber int64
Workflow string
Attempt int64
}
// Step-execution queries (docs/prds/execution-tree-state-machine.md). The
// store method owns the transaction that pairs each step transition with its
// event row; these are only the guarded row writes and reads.
// FOR UPDATE locks the execution's row, so the status check, the step insert
// and the event append are one serialized operation per execution -- the way
// guardTransition locks a task's row for its transition. The row also carries
// every field the transition's event needs (org, workspace, the task
// identity) and the attempt the step must belong to.
func (q *Queries) LockExecutionForStep(ctx context.Context, id int64) (LockExecutionForStepRow, error) {
row := q.db.QueryRow(ctx, lockExecutionForStep, id)
var i LockExecutionForStepRow
err := row.Scan(
&i.ID,
&i.Status,
&i.OrgID,
&i.WorkspaceID,
&i.Owner,
&i.Repo,
&i.IssueNumber,
&i.Workflow,
&i.Attempt,
)
return i, err
}
const lockStepExecution = `-- name: LockStepExecution :one
SELECT id, org_id, workspace_id, execution_id, attempt, parent_id, depth, kind, name, called_execution_id, status, detail, tokens_used, started_at, finished_at
FROM step_executions WHERE id = $1 FOR UPDATE
`
// FOR UPDATE, shared by a start's parent check and a finish's guarded write:
// the row is held for the caller's transaction, so the state a check reads is
// the state the guarded write sees.
func (q *Queries) LockStepExecution(ctx context.Context, id int64) (StepExecution, error) {
row := q.db.QueryRow(ctx, lockStepExecution, id)
var i StepExecution
err := row.Scan(
&i.ID,
&i.OrgID,
&i.WorkspaceID,
&i.ExecutionID,
&i.Attempt,
&i.ParentID,
&i.Depth,
&i.Kind,
&i.Name,
&i.CalledExecutionID,
&i.Status,
&i.Detail,
&i.TokensUsed,
&i.StartedAt,
&i.FinishedAt,
)
return i, err
}
const startStepExecution = `-- name: StartStepExecution :execrows
UPDATE step_executions SET status = 'running', started_at = now()
WHERE id = $1 AND status = 'pending'
`
// The pending -> running half of StartStep, guarded by the row's own status:
// the step transition table routes the pair, so a row that is not pending is
// stale rather than silently restarted.
func (q *Queries) StartStepExecution(ctx context.Context, id int64) (int64, error) {
result, err := q.db.Exec(ctx, startStepExecution, id)
if err != nil {
return 0, err
}
return result.RowsAffected(), nil
}
@@ -0,0 +1,36 @@
-- Step-execution queries (docs/prds/execution-tree-state-machine.md). The
-- store method owns the transaction that pairs each step transition with its
-- event row; these are only the guarded row writes and reads.
-- name: LockExecutionForStep :one
-- FOR UPDATE locks the execution's row, so the status check, the step insert
-- and the event append are one serialized operation per execution -- the way
-- guardTransition locks a task's row for its transition. The row also carries
-- every field the transition's event needs (org, workspace, the task
-- identity) and the attempt the step must belong to.
SELECT id, status, org_id, workspace_id, owner, repo, issue_number, workflow, attempt
FROM tasks WHERE id = $1 FOR UPDATE;
-- name: InsertStepExecution :one
INSERT INTO step_executions (org_id, workspace_id, execution_id, attempt, parent_id, depth, kind, name, called_execution_id, status)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, 'pending')
RETURNING id;
-- name: StartStepExecution :execrows
-- The pending -> running half of StartStep, guarded by the row's own status:
-- the step transition table routes the pair, so a row that is not pending is
-- stale rather than silently restarted.
UPDATE step_executions SET status = 'running', started_at = now()
WHERE id = $1 AND status = 'pending';
-- name: LockStepExecution :one
-- FOR UPDATE, shared by a start's parent check and a finish's guarded write:
-- the row is held for the caller's transaction, so the state a check reads is
-- the state the guarded write sees.
SELECT id, org_id, workspace_id, execution_id, attempt, parent_id, depth, kind, name, called_execution_id, status, detail, tokens_used, started_at, finished_at
FROM step_executions WHERE id = $1 FOR UPDATE;
-- name: FinishStepExecution :execrows
UPDATE step_executions
SET status = $1, detail = $2, tokens_used = $3, finished_at = now()
WHERE id = $4 AND status = $5;
+232
View File
@@ -0,0 +1,232 @@
package postgres
import (
"context"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgtype"
"github.com/samcharles93/archie-core/internal/domain/storecontract"
task "github.com/samcharles93/archie-core/internal/domain/workflow/task"
"github.com/samcharles93/archie-core/internal/events"
"github.com/samcharles93/archie-core/internal/infrastructure/postgres/postgresdb"
"github.com/samcharles93/archie-core/internal/taskstate"
)
// The step-execution writes of docs/prds/execution-tree-state-machine.md.
// Each is one transaction: the guarded row write, and the domain event row
// the step transition produces, so the events table records exactly the
// transitions the step row does. The events land in the table; the caller
// publishes the returned event to its bus after the write commits, and the
// daemon's event sink skips already-persisted rows by their assigned ID --
// the EmitDurable convention.
// StartStep records a StepExecution entering running: the row is created
// pending, guarded into running under the shared step transition table, and
// the stage_start event is appended in the same transaction. The execution's
// row is locked for the whole write, so its status check cannot race the
// guarded updates, and org and workspace are stamped from it -- the request's
// ownership fields are ignored, like every owned record.
func (s *Store) StartStep(ctx context.Context, start task.StepStart) (int64, events.Event, error) {
if !task.ValidStepKind(start.Kind) {
return 0, events.Event{}, fmt.Errorf("%w: unknown step kind %q", storecontract.ErrInvalidStep, start.Kind)
}
tx, err := s.pool.Begin(ctx)
if err != nil {
return 0, events.Event{}, err
}
defer func() { _ = tx.Rollback(ctx) }()
q := postgresdb.New(tx)
execution, err := q.LockExecutionForStep(ctx, start.ExecutionID)
if errors.Is(err, pgx.ErrNoRows) {
// The execution the caller names does not exist: its view of the run
// is stale, matching what guardTransition returns for a missing row.
return 0, events.Event{}, storecontract.ErrStaleTransition
}
if err != nil {
return 0, events.Event{}, err
}
if execution.Attempt != int64(start.Attempt) {
// A step belongs to exactly one attempt; a caller naming one the
// execution no longer runs is describing a run that is over.
return 0, events.Event{}, storecontract.ErrStaleTransition
}
depth, err := s.startParent(ctx, q, start)
if err != nil {
return 0, events.Event{}, err
}
// A step cannot reach running while its execution is not running, and
// never under a terminal parent. Both are the state machine's rules, so
// they refuse as illegal transitions.
if err := taskstate.CheckStepStart(execution.Status, depth.underTerminal); err != nil {
return 0, events.Event{}, fmt.Errorf("%w: %w", storecontract.ErrIllegalTransition, err)
}
stepID, err := q.InsertStepExecution(ctx, postgresdb.InsertStepExecutionParams{
OrgID: execution.OrgID, WorkspaceID: execution.WorkspaceID,
ExecutionID: execution.ID, Attempt: execution.Attempt,
ParentID: stepParentID(start.ParentID), Depth: int32(depth.depth),
Kind: start.Kind, Name: start.Name, CalledExecutionID: 0,
})
if err != nil {
return 0, events.Event{}, err
}
n, err := q.StartStepExecution(ctx, stepID)
if err != nil {
return 0, events.Event{}, err
}
if n == 0 {
return 0, events.Event{}, storecontract.ErrStaleTransition
}
event := stepEvent(events.KindStageStart, execution.ID,
execution.Owner, execution.Repo, execution.IssueNumber, execution.Workflow,
int(execution.Attempt), start.Name, nil)
eventID, err := insertEventQ(ctx, q, event)
if err != nil {
return 0, events.Event{}, err
}
event.ID = eventID
if err := tx.Commit(ctx); err != nil {
return 0, events.Event{}, err
}
return stepID, event, nil
}
// stepParent resolves the enclosing StepExecution a child step names: its
// row is locked in the same transaction, the tree must stay inside one
// execution and one attempt, and depth is derived from the parent, never
// supplied by the caller.
func (s *Store) startParent(ctx context.Context, q *postgresdb.Queries, start task.StepStart) (stepParentState, error) {
if start.ParentID == 0 {
return stepParentState{}, nil
}
parent, err := q.LockStepExecution(ctx, start.ParentID)
if errors.Is(err, pgx.ErrNoRows) {
return stepParentState{}, storecontract.ErrStaleTransition
}
if err != nil {
return stepParentState{}, err
}
if parent.ExecutionID != start.ExecutionID || parent.Attempt != int64(start.Attempt) {
// Trees never nest across executions, and a parent from another
// attempt is a tree the caller is not running.
return stepParentState{}, storecontract.ErrStaleTransition
}
return stepParentState{
depth: int(parent.Depth) + 1,
underTerminal: taskstate.StepTerminal(taskstate.StepStatus(parent.Status)),
}, nil
}
type stepParentState struct {
depth int
underTerminal bool
}
// FinishStep moves one step to its outcome under the shared step transition
// table, writing the stage_finish event in the same transaction and returning
// it for the caller's post-commit publish. A step that is missing, that is
// not the named execution's, or whose status does not match from is stale; an
// off-table from->to pair is refused as an illegal transition.
func (s *Store) FinishStep(ctx context.Context, finish task.StepFinish) (events.Event, error) {
tx, err := s.pool.Begin(ctx)
if err != nil {
return events.Event{}, err
}
defer func() { _ = tx.Rollback(ctx) }()
q := postgresdb.New(tx)
step, err := q.LockStepExecution(ctx, finish.StepID)
if errors.Is(err, pgx.ErrNoRows) {
return events.Event{}, storecontract.ErrStaleTransition
}
if err != nil {
return events.Event{}, err
}
// The step belongs to another run, or the caller believes a state the row
// does not hold. Either way its view is stale, and writing would move
// another execution's record.
if step.ExecutionID != finish.ExecutionID || taskstate.StepStatus(step.Status) != finish.From {
return events.Event{}, storecontract.ErrStaleTransition
}
// Staleness is decided first, the way guardTransition decides it: a caller
// that is wrong about the row's state gets the stale sentinel even when
// its pair is also unroutable.
if !taskstate.CanStepTransition(finish.From, finish.To) {
return events.Event{}, fmt.Errorf("%w: %s -> %s is not a step transition",
storecontract.ErrIllegalTransition, finish.From, finish.To)
}
execution, err := q.TaskByID(ctx, step.ExecutionID)
if err != nil {
return events.Event{}, err
}
durationMS := int64(0)
if step.StartedAt.Valid {
durationMS = time.Since(step.StartedAt.Time).Milliseconds()
}
n, err := q.FinishStepExecution(ctx, postgresdb.FinishStepExecutionParams{
Status: string(finish.To), Detail: clip(finish.Detail, 4000), TokensUsed: finish.TokensUsed,
ID: finish.StepID, Status_2: string(finish.From),
})
if err != nil {
return events.Event{}, err
}
if n == 0 {
return events.Event{}, storecontract.ErrStaleTransition
}
event := stepEvent(events.KindStageFinish, execution.ID,
execution.Owner, execution.Repo, execution.IssueNumber, execution.Workflow,
int(step.Attempt), step.Name,
stepEventData(finish.To, durationMS, finish.Detail))
eventID, err := insertEventQ(ctx, q, event)
if err != nil {
return events.Event{}, err
}
event.ID = eventID
if err := tx.Commit(ctx); err != nil {
return events.Event{}, err
}
return event, nil
}
// stepEvent builds the event row one step transition writes, stamped with the
// execution's identity the way TaskContext.Emit stamped it -- the same fields
// the timeline and the stage statistics read back.
func stepEvent(kind string, executionID int64, owner, repo string, issue int64, wf string, attempt int, stage string, data map[string]any) events.Event {
return events.Event{
Kind: kind, TaskID: executionID,
Repo: owner + "/" + repo, Issue: int(issue), Workflow: wf,
Attempt: attempt, Stage: stage, Data: data,
}
}
// stepEventData reads a stage_finish event's data the way the dashboard
// (stageOutcome) and the stage statistics (StageStats) do: duration always,
// the error text only when the step reported failure, and the interruption
// marker only for an interrupted step.
func stepEventData(to taskstate.StepStatus, durationMS int64, detail string) map[string]any {
data := map[string]any{"duration_ms": durationMS}
switch to {
case taskstate.StepFailed:
data["error"] = detail
case taskstate.StepInterrupted:
data["interrupted"] = true
}
return data
}
// stepParentID encodes the nullable parent id an insert takes: 0 means "no
// parent" -- a stage at the tree's root -- which the column stores as NULL
// rather than as a reference to a row that does not exist.
func stepParentID(parentID int64) pgtype.Int8 {
return pgtype.Int8{Int64: parentID, Valid: parentID != 0}
}
@@ -0,0 +1,249 @@
package postgres
import (
"errors"
"testing"
"github.com/samcharles93/archie-core/internal/domain/storecontract"
"github.com/samcharles93/archie-core/internal/domain/workflow"
task "github.com/samcharles93/archie-core/internal/domain/workflow/task"
"github.com/samcharles93/archie-core/internal/events"
"github.com/samcharles93/archie-core/internal/taskstate"
)
// The step-execution writes (docs/prds/execution-tree-state-machine.md): each
// of StartStep and FinishStep is one transaction -- guarded row write plus the
// domain event row -- under the step transition table, with the execution's
// own status gating whether a step may run at all.
// runningExecution enqueues and claims one task, the state a step runs in.
func runningExecution(t *testing.T, s *Store) *workflow.Task {
t.Helper()
seed, err := s.EnqueueChatTask(t.Context(), "acme", "widgets", "t", "b", "implement", "")
if err != nil || seed == nil {
t.Fatalf("EnqueueChatTask: %+v %v", seed, err)
}
claimed, err := s.ClaimNext(t.Context())
if err != nil || claimed == nil {
t.Fatalf("ClaimNext: %+v %v", claimed, err)
}
return claimed
}
func TestStartStepRecordsAStageEnteringRunning(t *testing.T) {
s := storeFor(t)
execution := runningExecution(t, s)
stepID, startEvent, err := s.StartStep(t.Context(), task.StepStart{
ExecutionID: execution.ID, Attempt: execution.Attempt, Kind: task.StepKindStage, Name: "implement",
})
if err != nil {
t.Fatalf("StartStep: %v", err)
}
if startEvent.Kind != events.KindStageStart || startEvent.Stage != "implement" ||
startEvent.TaskID != execution.ID || startEvent.Attempt != execution.Attempt ||
startEvent.Repo != "acme/widgets" || startEvent.ID == 0 {
t.Fatalf("stage_start event = %+v, want the transition's row", startEvent)
}
eventsForTask, err := s.TaskEvents(t.Context(), execution.ID)
if err != nil {
t.Fatalf("TaskEvents: %v", err)
}
if n := len(eventsForTask); n != 1 {
t.Fatalf("events after StartStep = %d, want exactly the one the transition wrote", n)
}
step, err := s.FinishStep(t.Context(), task.StepFinish{
StepID: stepID, ExecutionID: execution.ID,
From: taskstate.StepRunning, To: taskstate.StepSucceeded, Detail: "implemented",
})
if err != nil {
t.Fatalf("FinishStep: %v", err)
}
if step.Kind != events.KindStageFinish || step.Stage != "implement" || step.ID == 0 {
t.Fatalf("stage_finish event = %+v, want the transition's row", step)
}
if _, ok := step.Data["duration_ms"]; !ok {
t.Errorf("stage_finish data = %v, want duration_ms: the stage statistics read it", step.Data)
}
if _, has := step.Data["error"]; has {
t.Errorf("stage_finish data = %v, want no error key for a succeeded step", step.Data)
}
}
func TestFinishStepFailedCarriesTheErrorTheDashboardRenders(t *testing.T) {
s := storeFor(t)
execution := runningExecution(t, s)
stepID, _, err := s.StartStep(t.Context(), task.StepStart{
ExecutionID: execution.ID, Attempt: execution.Attempt, Kind: task.StepKindStage, Name: "review",
})
if err != nil {
t.Fatalf("StartStep: %v", err)
}
step, err := s.FinishStep(t.Context(), task.StepFinish{
StepID: stepID, ExecutionID: execution.ID,
From: taskstate.StepRunning, To: taskstate.StepFailed, Detail: "review found 3 findings",
})
if err != nil {
t.Fatalf("FinishStep: %v", err)
}
if step.Data["error"] != "review found 3 findings" {
t.Errorf("stage_finish data = %v, want the error text stageOutcome renders", step.Data)
}
}
func TestStartStepRefusesWhatTheStateMachineForbids(t *testing.T) {
tests := []struct {
name string
seed func(s *Store, taskID int64) error
start task.StepStart
want error
}{
{
name: "an execution that is not running",
start: task.StepStart{Kind: task.StepKindStage, Name: "plan", Attempt: 1},
want: storecontract.ErrIllegalTransition,
},
{
name: "an execution that does not exist",
start: task.StepStart{Kind: task.StepKindStage, Name: "plan", Attempt: 1},
want: storecontract.ErrStaleTransition,
},
{
name: "an attempt the execution no longer runs",
start: task.StepStart{Kind: task.StepKindStage, Name: "plan", Attempt: 99},
want: storecontract.ErrStaleTransition,
},
{
name: "an unknown step kind",
start: task.StepStart{Kind: "quantum", Name: "plan", Attempt: 1},
want: storecontract.ErrInvalidStep,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := storeFor(t)
execution := runningExecution(t, s)
if tt.name == "an execution that is not running" {
// Back on the queue: nothing may run under it. The seed rides
// the legal running->queued edge, not a hand-written pair.
if err := s.Transition(t.Context(), execution.ID, taskstate.Running, taskstate.Queued, "requeue"); err != nil {
t.Fatalf("seed queued: %v", err)
}
}
start := tt.start
start.ExecutionID = execution.ID
if tt.name == "an execution that does not exist" {
start.ExecutionID = execution.ID + 999999
}
if _, _, err := s.StartStep(t.Context(), start); !errors.Is(err, tt.want) {
t.Fatalf("StartStep = %v, want %v", err, tt.want)
}
})
}
}
func TestStartStepUnderATerminalParentIsIllegal(t *testing.T) {
s := storeFor(t)
execution := runningExecution(t, s)
parentID, _, err := s.StartStep(t.Context(), task.StepStart{
ExecutionID: execution.ID, Attempt: execution.Attempt, Kind: task.StepKindStage, Name: "review",
})
if err != nil {
t.Fatalf("StartStep parent: %v", err)
}
if _, err := s.FinishStep(t.Context(), task.StepFinish{
StepID: parentID, ExecutionID: execution.ID,
From: taskstate.StepRunning, To: taskstate.StepSucceeded, Detail: "done",
}); err != nil {
t.Fatalf("FinishStep parent: %v", err)
}
_, _, err = s.StartStep(t.Context(), task.StepStart{
ExecutionID: execution.ID, Attempt: execution.Attempt,
ParentID: parentID, Kind: task.StepKindAgent, Name: "review:2",
})
if !errors.Is(err, storecontract.ErrIllegalTransition) {
t.Fatalf("StartStep under a terminal parent = %v, want ErrIllegalTransition", err)
}
}
func TestStepDepthIsDerivedFromTheParent(t *testing.T) {
s := storeFor(t)
execution := runningExecution(t, s)
parentID, _, err := s.StartStep(t.Context(), task.StepStart{
ExecutionID: execution.ID, Attempt: execution.Attempt, Kind: task.StepKindStage, Name: "implement",
})
if err != nil {
t.Fatalf("StartStep stage: %v", err)
}
childID, _, err := s.StartStep(t.Context(), task.StepStart{
ExecutionID: execution.ID, Attempt: execution.Attempt,
ParentID: parentID, Kind: task.StepKindAgent, Name: "implement",
})
if err != nil {
t.Fatalf("StartStep agent child: %v", err)
}
var depths map[int64]int
if err := s.pool.QueryRow(t.Context(),
`SELECT jsonb_object_agg(id, depth) FROM step_executions WHERE execution_id = $1`,
execution.ID).Scan(&depths); err != nil {
t.Fatalf("read depths: %v", err)
}
if depths[parentID] != 0 || depths[childID] != 1 {
t.Fatalf("depths = %v, want the stage at 0 and its child at 1", depths)
}
}
func TestFinishStepRefusesStaleAndForeign(t *testing.T) {
s := storeFor(t)
execution := runningExecution(t, s)
other := runningExecution(t, s)
stepID, _, err := s.StartStep(t.Context(), task.StepStart{
ExecutionID: execution.ID, Attempt: execution.Attempt, Kind: task.StepKindStage, Name: "plan",
})
if err != nil {
t.Fatalf("StartStep: %v", err)
}
if _, err := s.FinishStep(t.Context(), task.StepFinish{
StepID: stepID, ExecutionID: other.ID,
From: taskstate.StepRunning, To: taskstate.StepSucceeded,
}); !errors.Is(err, storecontract.ErrStaleTransition) {
t.Fatalf("FinishStep naming another execution = %v, want ErrStaleTransition", err)
}
if _, err := s.FinishStep(t.Context(), task.StepFinish{
StepID: stepID, ExecutionID: execution.ID,
From: taskstate.StepSucceeded, To: taskstate.StepFailed,
}); !errors.Is(err, storecontract.ErrStaleTransition) {
t.Fatalf("FinishStep with an untruthful from = %v, want ErrStaleTransition", err)
}
if _, err := s.FinishStep(t.Context(), task.StepFinish{
StepID: stepID, ExecutionID: execution.ID,
From: taskstate.StepRunning, To: taskstate.StepPending,
}); !errors.Is(err, storecontract.ErrIllegalTransition) {
t.Fatalf("FinishStep off-table pair = %v, want ErrIllegalTransition", err)
}
if _, err := s.FinishStep(t.Context(), task.StepFinish{
StepID: stepID + 999999, ExecutionID: execution.ID,
From: taskstate.StepRunning, To: taskstate.StepSucceeded,
}); !errors.Is(err, storecontract.ErrStaleTransition) {
t.Fatalf("FinishStep on a missing step = %v, want ErrStaleTransition", err)
}
}
// A late FinishStep for a step another path already finished is stale, the
// PRD's named wire behaviour: errors.Is matches on the client too.
func TestFinishStepOffTablePairIsIllegalOverTheTable(t *testing.T) {
if !taskstate.CanStepTransition(taskstate.StepPending, taskstate.StepRunning) {
t.Fatal("pending -> running must be in the step transition table")
}
if taskstate.CanStepTransition(taskstate.StepSucceeded, taskstate.StepRunning) {
t.Fatal("succeeded -> running must not be in the step transition table: terminal is terminal")
}
}
@@ -124,6 +124,7 @@ func taskFromRow(t postgresdb.Task) *workflow.Task {
// Compile-time checks: *Store satisfies every surface the SQLite store did.
var (
_ storecontract.TaskStore = (*Store)(nil)
_ storecontract.StepRecorder = (*Store)(nil)
_ storecontract.BindingTaskCreator = (*Store)(nil)
_ storecontract.ConfigSnapshotStore = (*Store)(nil)
_ storecontract.ChannelStatusStore = (*Store)(nil)
@@ -107,6 +107,41 @@ func (c *Client) Transition(ctx context.Context, taskID int64, from, to, detail
return unmapError(err)
}
// StartStep and FinishStep record the run's step executions
// (docs/prds/execution-tree-state-machine.md) under the run credential, which
// the State Store's grant interceptor scopes to the execution the requests
// name. Each returns the transition's persisted event for the caller's
// post-commit publish.
func (c *Client) StartStep(ctx context.Context, s task.StepStart) (int64, events.Event, error) {
r, err := c.client.StartStep(ctx, &pb.StartStepRequest{
ExecutionId: s.ExecutionID, Attempt: int64(s.Attempt),
ParentId: s.ParentID, Kind: s.Kind, Name: s.Name,
})
if err != nil {
return 0, events.Event{}, unmapError(err)
}
event := events.Event{}
if r.Event != nil {
event = eventValue(r.Event)
}
return r.StepId, event, nil
}
func (c *Client) FinishStep(ctx context.Context, s task.StepFinish) (events.Event, error) {
r, err := c.client.FinishStep(ctx, &pb.FinishStepRequest{
ExecutionId: s.ExecutionID, StepId: s.StepID,
From: string(s.From), To: string(s.To), Detail: s.Detail, TokensUsed: s.TokensUsed,
})
if err != nil {
return events.Event{}, unmapError(err)
}
event := events.Event{}
if r.Event != nil {
event = eventValue(r.Event)
}
return event, nil
}
func (c *Client) Update(ctx context.Context, t *task.Task) error {
_, err := c.client.Update(ctx, &pb.UpdateRequest{Task: taskProto(t)})
return unmapError(err)
@@ -14,6 +14,7 @@ import (
"github.com/samcharles93/archie-core/internal/domain/binding"
"github.com/samcharles93/archie-core/internal/domain/mapping"
"github.com/samcharles93/archie-core/internal/domain/storecontract"
"github.com/samcharles93/archie-core/internal/domain/workflow"
"github.com/samcharles93/archie-core/internal/events"
"github.com/samcharles93/archie-core/internal/infrastructure/postgres"
"github.com/samcharles93/archie-core/internal/infrastructure/postgres/pgstore"
@@ -26,6 +27,7 @@ import (
// docs/prds/state-store-contract.md §11's conformance requirement.
type contract interface {
storecontract.TaskStore
storecontract.StepRecorder
storecontract.BindingTaskCreator
storecontract.ConfigSnapshotStore
storecontract.ApplyStatusStore
@@ -81,7 +83,7 @@ func remoteTaskStore(t *testing.T, local *pgstore.TaskDB, logs storecontract.Tas
listener := bufconn.Listen(1 << 20)
server := grpc.NewServer()
deps := Deps{
Tasks: local, BindingTaskCreator: local, WorkflowCalls: local,
Tasks: local, BindingTaskCreator: local, WorkflowCalls: local, Steps: local,
ConfigSnapshots: local, ApplyStatus: local,
TaskLogs: logs,
}
@@ -133,6 +135,37 @@ func TestStateStoreConformance(t *testing.T) {
if err := c.Transition(ctx, task.ID, task.Status, "running", "started"); err != nil {
t.Fatalf("Transition: %v", err)
}
current, err := c.TaskByID(ctx, task.ID)
if err != nil || current == nil {
t.Fatalf("TaskByID: %+v %v", current, err)
}
// The step-execution writes (docs/prds/execution-tree-state-machine.md):
// both adapters record the same guarded pair, and each response
// carries the transition's own persisted event.
stepID, startEvent, err := c.StartStep(ctx, workflow.StepStart{
ExecutionID: task.ID, Attempt: current.Attempt, Kind: workflow.StepKindStage, Name: "implement",
})
if err != nil || stepID == 0 {
t.Fatalf("StartStep: %d %v", stepID, err)
}
if startEvent.Kind != "stage_start" || startEvent.TaskID != task.ID || startEvent.ID == 0 {
t.Fatalf("StartStep event = %+v, want the persisted stage_start row", startEvent)
}
finishEvent, err := c.FinishStep(ctx, workflow.StepFinish{
StepID: stepID, ExecutionID: task.ID,
From: "running", To: "succeeded", Detail: "done", TokensUsed: 12,
})
if err != nil {
t.Fatalf("FinishStep: %v", err)
}
if finishEvent.Kind != "stage_finish" || finishEvent.ID == 0 || finishEvent.Data["duration_ms"] == nil {
t.Fatalf("FinishStep event = %+v, want the persisted stage_finish row with a duration", finishEvent)
}
if _, err := c.FinishStep(ctx, workflow.StepFinish{
StepID: stepID, ExecutionID: task.ID, From: "running", To: "succeeded",
}); !errors.Is(err, storecontract.ErrStaleTransition) {
t.Fatalf("replayed FinishStep = %v, want ErrStaleTransition over both adapters", err)
}
task.Plan = "the plan"
task.Status = "running"
task.PRNumber = 401
@@ -256,24 +289,29 @@ func TestStateStoreConformance(t *testing.T) {
t.Fatalf("OpenPRs: %v", err)
}
// TaskEvents / EventsSince / stats.
// TaskEvents / EventsSince / stats. The two step events the
// battery's step half wrote come first (their transitions ran
// first), then the two the battery inserted directly.
evs, err := c.TaskEvents(ctx, task.ID)
if err != nil || len(evs) != 2 || evs[0].Data["duration_ms"] != 12.0 {
if err != nil || len(evs) != 4 {
t.Fatalf("TaskEvents: %+v %v", evs, err)
}
if evs[0].Attempt != 2 {
t.Errorf("TaskEvents attempt = %d, want 2 (provenance must survive the wire)", evs[0].Attempt)
if evs[2].Data["duration_ms"] != 12.0 {
t.Fatalf("TaskEvents stage_finish = %+v, want the battery's own duration_ms", evs[2])
}
if evs[1].Kind != events.KindConfigCaptured || evs[1].Attempt != 2 || evs[1].Data["schema"] != events.ConfigCapturedSchema {
t.Errorf("config_captured event = %+v, want its kind, attempt and schema preserved", evs[1])
if evs[2].Attempt != 2 {
t.Errorf("TaskEvents attempt = %d, want 2 (provenance must survive the wire)", evs[2].Attempt)
}
if _, renamed := evs[1].Data["config"]; renamed {
if evs[3].Kind != events.KindConfigCaptured || evs[3].Attempt != 2 || evs[3].Data["schema"] != events.ConfigCapturedSchema {
t.Errorf("config_captured event = %+v, want its kind, attempt and schema preserved", evs[3])
}
if _, renamed := evs[3].Data["config"]; renamed {
t.Error(`the document crossed under "config"; the producer writes it under "document"`)
}
if len(evs[1].Data) != 2 {
if len(evs[3].Data) != 2 {
t.Errorf("config_captured data keys = %v, want exactly schema and document", evs[1].Data)
}
doc, ok := evs[1].Data["document"].(map[string]any)
doc, ok := evs[3].Data["document"].(map[string]any)
if !ok || doc["bot_user"] != "archie" {
t.Errorf("config_captured payload = %#v, want the decoded document under its own key", evs[1].Data)
}
@@ -365,7 +403,7 @@ func TestStateStoreConformance(t *testing.T) {
// Archive. RecoverStale above may have requeued task if it was
// still running, so re-fetch its current status rather than
// trusting the in-memory snapshot.
current, err := c.TaskByID(ctx, task.ID)
current, err = c.TaskByID(ctx, task.ID)
if err != nil || current == nil {
t.Fatalf("TaskByID before archive: %+v %v", current, err)
}
@@ -77,6 +77,7 @@ func TestUnmapErrorSentinelFidelity(t *testing.T) {
}{
{name: "stale transition", store: storecontract.ErrStaleTransition, rehyd: storecontract.ErrStaleTransition},
{name: "illegal transition", store: storecontract.ErrIllegalTransition, rehyd: storecontract.ErrIllegalTransition},
{name: "invalid step", store: storecontract.ErrInvalidStep, rehyd: storecontract.ErrInvalidStep},
{name: "binding not found", store: storecontract.ErrBindingNotFound, rehyd: storecontract.ErrBindingNotFound},
{name: "mapping not found", store: storecontract.ErrMappingNotFound, rehyd: storecontract.ErrMappingNotFound},
{name: "binding overlap", store: storecontract.ErrBindingOverlap, rehyd: storecontract.ErrBindingOverlap},
@@ -26,6 +26,7 @@ import (
"github.com/samcharles93/archie-core/internal/domain/storepkg"
"github.com/samcharles93/archie-core/internal/domain/workflow/task"
"github.com/samcharles93/archie-core/internal/logging"
"github.com/samcharles93/archie-core/internal/taskstate"
)
// Unavailable-capability errors, mirroring gatewayrpc's missing-session-store
@@ -96,6 +97,11 @@ type Deps struct {
// answers both RPCs with codes.Unavailable, which is the honest answer
// for a store that owns no task table.
WorkflowCalls storecontract.WorkflowCaller
// Steps is the step-execution write surface
// (docs/prds/execution-tree-state-machine.md): the guarded row writes the
// workflow engine's run credential calls. Optional: nil answers both RPCs
// with codes.Unavailable.
Steps storecontract.StepRecorder
// PlaybookDispatcher is the side-effecting-action idempotency ledger
// (docs/prds/eda-playbook-engine.md gap 2). Optional: nil disables the
// two playbook dispatch RPCs with codes.Unavailable.
@@ -189,6 +195,40 @@ func (s *server) Transition(ctx context.Context, r *pb.TransitionRequest) (*pb.T
return &pb.TransitionResponse{}, nil
}
// StartStep and FinishStep are the step-execution writes
// (docs/prds/execution-tree-state-machine.md). Each is the store's own
// transaction -- guarded row write plus the event row -- and the store's
// response carries the persisted event back for the caller's post-commit
// publish.
func (s *server) StartStep(ctx context.Context, r *pb.StartStepRequest) (*pb.StartStepResponse, error) {
if s.deps.Steps == nil {
return nil, status.Error(codes.Unavailable, "step recorder unavailable")
}
stepID, event, err := s.deps.Steps.StartStep(ctx, task.StepStart{
ExecutionID: r.ExecutionId, Attempt: int(r.Attempt),
ParentID: r.ParentId, Kind: r.Kind, Name: r.Name,
})
if err != nil {
return nil, s.logErr("StartStep", err)
}
return &pb.StartStepResponse{StepId: stepID, Event: eventProto(event)}, nil
}
func (s *server) FinishStep(ctx context.Context, r *pb.FinishStepRequest) (*pb.FinishStepResponse, error) {
if s.deps.Steps == nil {
return nil, status.Error(codes.Unavailable, "step recorder unavailable")
}
event, err := s.deps.Steps.FinishStep(ctx, task.StepFinish{
StepID: r.StepId, ExecutionID: r.ExecutionId,
From: taskstate.StepStatus(r.From), To: taskstate.StepStatus(r.To),
Detail: r.Detail, TokensUsed: r.TokensUsed,
})
if err != nil {
return nil, s.logErr("FinishStep", err)
}
return &pb.FinishStepResponse{Event: eventProto(event)}, nil
}
func (s *server) Update(ctx context.Context, r *pb.UpdateRequest) (*pb.UpdateResponse, error) {
if r.Task == nil {
return nil, status.Error(codes.InvalidArgument, "task is required")
+62 -20
View File
@@ -95,32 +95,74 @@ func (g *TaskGrants) taskFor(token string) int64 {
// (including RegisterTaskGrant/RevokeTaskGrant themselves) falls through to
// false: a task grant can only ever narrow, never expand.
//
// The two workflow.call RPCs are the one sanctioned widening
// (docs/prds/workflow-calls.md): a workflow.call step must start its callee
// and read it back, and the callee is the caller's child. Both requests
// carry the caller's own task ID, which the grant check verifies; the
// parent-child relation is a handler-side row check, so this interceptor
// never reads the database.
// The two workflow.call RPCs and the two step-execution writes are the
// sanctioned widenings: a workflow.call step must start its callee and read it
// back, the callee being the caller's child (docs/prds/workflow-calls.md), and
// the container records its stages and its own agent calls under the run
// credential of docs/prds/execution-tree-state-machine.md. The step requests
// carry the caller's own task ID, which the grant check verifies; the step's
// membership in that execution is a handler-side row check, so this
// interceptor never reads the database.
func authorizesTaskScopedCall(fullMethod string, req any, taskID int64) bool {
switch fullMethod {
case pb.StateStoreService_Update_FullMethodName:
target := taskScopedTargets[fullMethod]
return target != nil && target(req) == taskID
}
// taskScopedTargets maps each RPC a task grant may ever authorize to the
// function naming the task ID its request targets (0 for a request it cannot
// read). Held as data so the sanctioned widenings stop growing a switch: a
// new authorized RPC registers one row here, and the deny-by-default rule
// above covers everything else.
var taskScopedTargets = map[string]func(any) int64{
pb.StateStoreService_Update_FullMethodName: func(req any) int64 {
r, ok := req.(*pb.UpdateRequest)
return ok && r.Task != nil && r.Task.Id == taskID
case pb.StateStoreService_Transition_FullMethodName:
if !ok || r.Task == nil {
return 0
}
return r.Task.Id
},
pb.StateStoreService_Transition_FullMethodName: func(req any) int64 {
r, ok := req.(*pb.TransitionRequest)
return ok && r.TaskId == taskID
case pb.StateStoreService_InsertEvent_FullMethodName:
if !ok {
return 0
}
return r.TaskId
},
pb.StateStoreService_InsertEvent_FullMethodName: func(req any) int64 {
r, ok := req.(*pb.InsertEventRequest)
return ok && r.Event != nil && r.Event.TaskId == taskID
case pb.StateStoreService_EnqueueCallTask_FullMethodName:
if !ok || r.Event == nil {
return 0
}
return r.Event.TaskId
},
pb.StateStoreService_StartStep_FullMethodName: func(req any) int64 {
r, ok := req.(*pb.StartStepRequest)
if !ok {
return 0
}
return r.ExecutionId
},
pb.StateStoreService_FinishStep_FullMethodName: func(req any) int64 {
r, ok := req.(*pb.FinishStepRequest)
if !ok {
return 0
}
return r.ExecutionId
},
pb.StateStoreService_EnqueueCallTask_FullMethodName: func(req any) int64 {
r, ok := req.(*pb.EnqueueCallTaskRequest)
return ok && r.CallerTaskId == taskID
case pb.StateStoreService_WorkflowCallStatus_FullMethodName:
if !ok {
return 0
}
return r.CallerTaskId
},
pb.StateStoreService_WorkflowCallStatus_FullMethodName: func(req any) int64 {
r, ok := req.(*pb.WorkflowCallStatusRequest)
return ok && r.CallerTaskId == taskID
default:
return false
}
if !ok {
return 0
}
return r.CallerTaskId
},
}
// UnaryInterceptor separates administrator access from task-scoped worker
@@ -17,8 +17,10 @@ import (
"github.com/samcharles93/archie-core/internal/domain/mapping"
"github.com/samcharles93/archie-core/internal/domain/storecontract"
"github.com/samcharles93/archie-core/internal/domain/workflow"
task "github.com/samcharles93/archie-core/internal/domain/workflow/task"
"github.com/samcharles93/archie-core/internal/events"
"github.com/samcharles93/archie-core/internal/infrastructure/postgres/pgstore"
"github.com/samcharles93/archie-core/internal/taskstate"
)
// grantsServer wires TaskGrants' Unary/Stream interceptors -- the same
@@ -41,7 +43,7 @@ func grantsServer(t *testing.T, adminToken string) (grants *TaskGrants, dial fun
Tasks: local, ConfigSnapshots: local, ApplyStatus: local,
Captures: eda, Mappings: eda, Bindings: eda,
BindingDispatcher: eda, PlaybookDispatcher: eda, EventTypes: eda,
BindingTaskCreator: local, WorkflowCalls: local,
BindingTaskCreator: local, WorkflowCalls: local, Steps: local,
Grants: grants,
})
go func() { _ = server.Serve(listener) }()
@@ -385,3 +387,73 @@ func TestOnlyAdminCanRegisterOrRevokeTaskGrants(t *testing.T) {
t.Fatal("a task-scoped grant must not be able to revoke grants")
}
}
// TestRunCredentialRecordsItsOwnSteps: the run credential of
// docs/prds/execution-tree-state-machine.md authorises StartStep and
// FinishStep on its own execution only -- the container records its stages
// and its own agent calls with it -- and never another execution's.
func TestRunCredentialRecordsItsOwnSteps(t *testing.T) {
const adminToken = "daemon-admin-token"
grants, dial := grantsServer(t, adminToken)
_ = grants
admin := dial(t, adminToken)
ctx := t.Context()
own, err := admin.EnqueueChatTask(ctx, "acme", "widget", "a", "body", "implement", "")
if err != nil {
t.Fatal(err)
}
if _, err := admin.ClaimNext(ctx); err != nil {
t.Fatalf("ClaimNext: %v", err)
}
// The claim increments the attempt, so the run the grant names is read
// back after it, the way the container's own request does.
own, err = admin.TaskByID(ctx, own.ID)
if err != nil || own == nil {
t.Fatalf("TaskByID: %+v %v", own, err)
}
other, err := admin.EnqueueChatTask(ctx, "acme", "widget", "b", "body", "implement", "")
if err != nil {
t.Fatal(err)
}
workerToken, err := admin.RegisterTaskGrant(ctx, own.ID, time.Hour)
if err != nil {
t.Fatalf("RegisterTaskGrant: %v", err)
}
worker := dial(t, workerToken)
stepID, startEvent, err := worker.StartStep(ctx, task.StepStart{
ExecutionID: own.ID, Attempt: own.Attempt, Kind: task.StepKindStage, Name: "implement",
})
if err != nil {
t.Fatalf("task grant should authorize StartStep on its own execution: %v", err)
}
if startEvent.Kind != events.KindStageStart || startEvent.TaskID != own.ID {
t.Fatalf("start event = %+v, want the transition's stage_start row", startEvent)
}
finishEvent, err := worker.FinishStep(ctx, task.StepFinish{
StepID: stepID, ExecutionID: own.ID,
From: taskstate.StepRunning, To: taskstate.StepSucceeded,
})
if err != nil {
t.Fatalf("task grant should authorize FinishStep on its own execution: %v", err)
}
if finishEvent.Kind != events.KindStageFinish {
t.Fatalf("finish event = %+v, want the transition's stage_finish row", finishEvent)
}
// The refusal is the interceptor's, not the store's: the codes tell the
// difference, so a store-level refusal could not mask a missing grant rule.
if _, _, err := worker.StartStep(ctx, task.StepStart{
ExecutionID: other.ID, Attempt: other.Attempt, Kind: task.StepKindStage, Name: "plan",
}); status.Code(err) != codes.PermissionDenied {
t.Fatalf("StartStep on another execution = %v, want PermissionDenied from the grant check", err)
}
if _, err := worker.FinishStep(ctx, task.StepFinish{
StepID: stepID, ExecutionID: other.ID,
From: taskstate.StepRunning, To: taskstate.StepSucceeded,
}); status.Code(err) != codes.PermissionDenied {
t.Fatalf("FinishStep naming another execution = %v, want PermissionDenied from the grant check", err)
}
}
@@ -334,6 +334,7 @@ func dayTokensValue(d *pb.DayTokens) storecontract.DayTokens {
const (
msgStaleTransition = "stale transition"
msgIllegalTransition = "illegal transition"
msgInvalidStep = "step execution invalid"
msgBindingNotFound = "binding not found"
msgMappingNotFound = "mapping not found"
msgEventTypeNotFound = "event type not found"
@@ -447,6 +448,7 @@ var wireErrors = []struct {
}{
{storecontract.ErrStaleTransition, codes.FailedPrecondition, msgStaleTransition},
{storecontract.ErrIllegalTransition, codes.FailedPrecondition, msgIllegalTransition},
{storecontract.ErrInvalidStep, codes.InvalidArgument, msgInvalidStep},
{storecontract.ErrBindingOverlap, codes.FailedPrecondition, msgBindingOverlap},
{storecontract.ErrBindingTransition, codes.FailedPrecondition, msgBindingTransition},
{storecontract.ErrSourceSigningStale, codes.FailedPrecondition, msgSourceSigning},
+40
View File
@@ -40,6 +40,10 @@ service StateStoreService {
rpc ClaimByIssue(ClaimByIssueRequest) returns (ClaimByIssueResponse);
rpc Transition(TransitionRequest) returns (TransitionResponse);
rpc Update(UpdateRequest) returns (UpdateResponse);
// Step executions (docs/prds/execution-tree-state-machine.md). The run
// credential authorises both on its own execution only.
rpc StartStep(StartStepRequest) returns (StartStepResponse);
rpc FinishStep(FinishStepRequest) returns (FinishStepResponse);
rpc Requeue(RequeueRequest) returns (RequeueResponse);
rpc ParkTask(ParkTaskRequest) returns (ParkTaskResponse);
rpc RecoverStale(RecoverStaleRequest) returns (RecoverStaleResponse);
@@ -518,6 +522,42 @@ message TransitionRequest {
}
message TransitionResponse {}
message StartStepRequest {
// execution_id is the WorkflowExecution (the tasks row) the step belongs
// to, and the task ID the run credential is scoped to.
int64 execution_id = 1;
int64 attempt = 2;
// parent_id is 0 when the step is a stage; otherwise it names the enclosing
// StepExecution, from which the store derives depth.
int64 parent_id = 3;
// kind is stage, agent or call.
string kind = 4;
string name = 5;
}
message StartStepResponse {
int64 step_id = 1;
// event is the transition's stage_start row, already persisted; the caller
// publishes it to its bus after the write commits (the daemon's event sink
// skips already-persisted rows by their assigned ID).
Event event = 2;
}
message FinishStepRequest {
// execution_id names the run the step belongs to. The grant check verifies
// it against the credential, and the store verifies the step really is that
// execution's, so a caller cannot move another run's step by naming it.
int64 execution_id = 1;
int64 step_id = 2;
string from = 3;
string to = 4;
string detail = 5;
int64 tokens_used = 6;
}
message FinishStepResponse {
// event is the transition's stage_finish row, already persisted.
Event event = 1;
}
message UpdateRequest { Task task = 1; }
message UpdateResponse {}