workflow-definitions: a seeded definition set is never refreshed when a built-in workflow's step vocabulary changes, and deleting the resource does not re-seed it
#1706
Once a built-in workflow's step vocabulary changes, a State Store that has already seeded workflow-definitions keeps serving the old definition set forever. Because ValidateDefinitionCollection validates the collection as a single document, one entry naming a removed step type fails every task's pin — regardless of that task's own workflow. On this instance the effect was total: every task parked with
pin workflow definition: workflow "implement" step 9: unknown type "implement.review"
and nothing could run until the resource was replaced by hand.
There is also no working automatic recovery path, because the seed's RequestID is stable and PutResource short-circuits on (kind, request_id) — so the obvious remedy (delete the resource and restart the State Store) writes nothing.
Impact
Total task outage. The daemon pins every task against the whole collection, so a single stale entry blocks all workflows. Observed on implement (step 9), but the collection is failed as one unit.
Silent. No boot or upgrade error; the stale document only surfaces when a task first tries to pin.
Not self-healing. Every subsequent release leaves it stale, so it persists until an operator intervenes.
Unrecoverable by the documented route. See "Defect 2".
Observed
Instance upgraded v1.42.0 → v1.43.0 (State Store migrations 0012–0019 applied 2026-09-27 16:56:19Z, all 19 applied, so migrations are not implicated). The stored resource:
Stored implement entry named three step types that exist nowhere in origin/main:
stored step type
present in current vocabulary
implement.review
no
implement.open-pr
no
implement.post-review-comments
no
Current Implement() replaces those with a single StagePRReviewAndOpenPR (internal/domain/workflow/implement.go:200), and the comment at implement.go:211-218 records the removal explicitly (archie-core-afbk.7).
resource_history shows exactly one revision ever written for this kind:
Confirmed against the running server's own catalog: the shipped document is 1955 bytes vs the stored 1494; the stored one also predates the pr-review workflow entirely.
Defect 1 — a seeded kind is never refreshed
internal/app/controlplane/registry.go:96-115 seeds only for a kind the store does not hold:
iferr==nil{versions[definition.Kind]=resource.Versioncontinue// held -> skip, whatever it contains}if!errors.Is(err,storecontract.ErrResourceNotFound){returnnil,nil,err}...resource,err=s.store.PutResource(ctx,storecontract.ResourceWrite{Kind:definition.Kind,Value:value,Actor:"system:migration",Source:"legacy-config",RequestID:"import:"+definition.Kind,ExpectedVersion:0,At:time.Now().UTC(),})
The Seed func is live (internal/app/controlplane/workflow_definitions.go:33-39 returns workflow.ShippedDefinitions()), so the seed value never drifts — but it is only ever consulted when the kind is absent. ShippedDefinitions() is generated from the code (internal/domain/workflow/definition.go:181), which is exactly why it can't drift, and exactly why the persisted copy should not be trusted as the source of truth once written.
Related asymmetry: the daemon falls back to ShippedDefinitions() when it has no definitions client (internal/daemon/daemon.go:1854, internal/daemon/binding_target.go:26) — so a deployment with no client is immune, and one with a client is stuck. The bug only bites the correct wiring.
Defect 2 — deleting the resource does not re-seed it
The intuitive recovery is "remove the row, restart, let it seed". That silently no-ops:
ResourceByRequest reads resource_history, not resources (internal/infrastructure/postgres/queries/resources.sql):
PutResource returns the existing row when that lookup hits (internal/infrastructure/postgres/resources.go:91-101), committing nothing.
resource_history carries CREATE UNIQUE INDEX idx_resource_history_kind_request ON resource_history (kind, request_id) (migrations/0001_state_store.sql:112).
Deleting resources leaves history intact, so the stable RequestID: "import:workflow-definitions" still matches, the write is skipped, and the kind stays absent. I hit this exactly: after the delete the kind returned ErrResourceNotFound, which parks tasks on a different error rather than fixing anything.
Defect 3 — the operator recovery that does work is undocumented
POST /api/control-plane/resources/{kind}/commands/replace with the catalog's defaults_json and a correct expected_version does fix it, keeping the prior revision in resource_history (internal/app/controlplane/client.go:82-93). It is exercised by internal/app/controlplane/step_vocabulary_agreement_test.go:254-280. Nothing in the failure path points an operator at it, and the error text says only unknown type.
Suggested fix
Version the seed so an upgrade can refresh a built-in. Either persist a seed revision alongside the value and re-seed when SeedRevision > storedRevision, or derive RequestID from the seed's content/revision ("import:workflow-definitions:" + revision) so PutResource's short-circuit can't mask a changed payload. The second needs no new column but relies on the revision being bumped when a shipped definition changes.
Make ShippedDefinitions() authoritative for built-ins. These were never operator-authored edits — the one revision on this instance is actor=system:migration, source=legacy-config. Distinguishing seeded built-ins from operator overrides would let an upgrade replace the former and keep the latter.
Fail loudly at boot. A stored collection that no longer validates should be reported at startup, not discovered by the first task that pins. DecodeDefinitionCollection already returns the precise error (internal/domain/workflow/definition.go:141-152).
Point the failure at the remedy. Include the replace command (or the resource kind and version) in the pin error, so the message names the fix.
Regression test. Seed a collection, advance ShippedDefinitions() to a document with a changed step type, migrate, and assert the effective collection matches the new one. Note internal/app/archied/step_vocabulary_test.go:169 already asserts the no-client fallback; the seeded-client path is the untested one.
Related, but a distinct defect
Tracked separately as an in-repo task: triage re-routing a task to another workflow (internal/domain/workflow/triage.go:85 sets only tc.Task.Workflow = target) leaves the old pin's YAML and digest in place, and pinWorkflowDefinition accepts an existing pin purely on self-consistency (internal/daemon/daemon.go:1845-1850 — it never checks the YAML's id against task.Workflow). internal/app/agentworker/task_execution.go:398-400 then rejects it as pinned workflow definition does not match task identity, parking a transient failure that retries into the identical error. Mentioned here only because it presents identically and was the first thing mistaken for this bug.
Back up the current resources.value bytea first, then requeue the parked tasks through POST /api/tasks/{id}/action {"action":"retry"} rather than editing tasks directly.
Imported from GitHub #1233 · type bug · priority P1 · created 2026-09-29 · closed 2026-09-29
## Summary
Once a built-in workflow's step vocabulary changes, a State Store that has already seeded `workflow-definitions` keeps serving the **old** definition set forever. Because `ValidateDefinitionCollection` validates the collection as a single document, one entry naming a removed step type fails every task's pin — regardless of that task's own workflow. On this instance the effect was total: every task parked with
```
pin workflow definition: workflow "implement" step 9: unknown type "implement.review"
```
and nothing could run until the resource was replaced by hand.
There is also no working automatic recovery path, because the seed's `RequestID` is stable and `PutResource` short-circuits on `(kind, request_id)` — so the obvious remedy (delete the resource and restart the State Store) writes nothing.
## Impact
- **Total task outage.** The daemon pins every task against the whole collection, so a single stale entry blocks all workflows. Observed on `implement` (step 9), but the collection is failed as one unit.
- **Silent.** No boot or upgrade error; the stale document only surfaces when a task first tries to pin.
- **Not self-healing.** Every subsequent release leaves it stale, so it persists until an operator intervenes.
- **Unrecoverable by the documented route.** See "Defect 2".
## Observed
Instance upgraded v1.42.0 → v1.43.0 (State Store migrations `0012`–`0019` applied `2026-09-27 16:56:19Z`, all 19 applied, so **migrations are not implicated**). The stored resource:
```
resources: kind=workflow-definitions version=1 updated_at=2026-09-24 15:47:46Z
```
Stored `implement` entry named three step types that exist nowhere in `origin/main`:
| stored step type | present in current vocabulary |
| --- | --- |
| `implement.review` | no |
| `implement.open-pr` | no |
| `implement.post-review-comments` | no |
Current `Implement()` replaces those with a single `StagePRReviewAndOpenPR` (`internal/domain/workflow/implement.go:200`), and the comment at `implement.go:211-218` records the removal explicitly (`archie-core-afbk.7`).
`resource_history` shows exactly one revision ever written for this kind:
```
id=2 version=1 actor=system:migration request_id=import:workflow-definitions 2026-09-24 15:47:46Z
```
Confirmed against the running server's own catalog: the shipped document is 1955 bytes vs the stored 1494; the stored one also predates the `pr-review` workflow entirely.
## Defect 1 — a seeded kind is never refreshed
`internal/app/controlplane/registry.go:96-115` seeds only for a kind the store **does not hold**:
```go
if err == nil {
versions[definition.Kind] = resource.Version
continue // held -> skip, whatever it contains
}
if !errors.Is(err, storecontract.ErrResourceNotFound) {
return nil, nil, err
}
...
resource, err = s.store.PutResource(ctx, storecontract.ResourceWrite{
Kind: definition.Kind, Value: value, Actor: "system:migration",
Source: "legacy-config", RequestID: "import:" + definition.Kind,
ExpectedVersion: 0, At: time.Now().UTC(),
})
```
The `Seed` func is live (`internal/app/controlplane/workflow_definitions.go:33-39` returns `workflow.ShippedDefinitions()`), so the seed *value* never drifts — but it is only ever consulted when the kind is absent. `ShippedDefinitions()` is generated from the code (`internal/domain/workflow/definition.go:181`), which is exactly why it can't drift, and exactly why the persisted copy should not be trusted as the source of truth once written.
Related asymmetry: the daemon falls back to `ShippedDefinitions()` when it has no definitions client (`internal/daemon/daemon.go:1854`, `internal/daemon/binding_target.go:26`) — so a deployment with no client is *immune*, and one with a client is *stuck*. The bug only bites the correct wiring.
## Defect 2 — deleting the resource does not re-seed it
The intuitive recovery is "remove the row, restart, let it seed". That silently no-ops:
- `ResourceByRequest` reads **`resource_history`**, not `resources` (`internal/infrastructure/postgres/queries/resources.sql`):
```sql
-- name: ResourceByRequest :one
SELECT ... FROM resource_history WHERE kind = $1 AND request_id = $2;
```
- `PutResource` returns the existing row when that lookup hits (`internal/infrastructure/postgres/resources.go:91-101`), committing nothing.
- `resource_history` carries `CREATE UNIQUE INDEX idx_resource_history_kind_request ON resource_history (kind, request_id)` (`migrations/0001_state_store.sql:112`).
Deleting `resources` leaves history intact, so the stable `RequestID: "import:workflow-definitions"` still matches, the write is skipped, and the kind stays absent. I hit this exactly: after the delete the kind returned `ErrResourceNotFound`, which parks tasks on a *different* error rather than fixing anything.
## Defect 3 — the operator recovery that does work is undocumented
`POST /api/control-plane/resources/{kind}/commands/replace` with the catalog's `defaults_json` and a correct `expected_version` does fix it, keeping the prior revision in `resource_history` (`internal/app/controlplane/client.go:82-93`). It is exercised by `internal/app/controlplane/step_vocabulary_agreement_test.go:254-280`. Nothing in the failure path points an operator at it, and the error text says only `unknown type`.
## Suggested fix
1. **Version the seed so an upgrade can refresh a built-in.** Either persist a seed revision alongside the value and re-seed when `SeedRevision > storedRevision`, or derive `RequestID` from the seed's content/revision (`"import:workflow-definitions:" + revision`) so `PutResource`'s short-circuit can't mask a changed payload. The second needs no new column but relies on the revision being bumped when a shipped definition changes.
2. **Make `ShippedDefinitions()` authoritative for built-ins.** These were never operator-authored edits — the one revision on this instance is `actor=system:migration`, `source=legacy-config`. Distinguishing seeded built-ins from operator overrides would let an upgrade replace the former and keep the latter.
3. **Fail loudly at boot.** A stored collection that no longer validates should be reported at startup, not discovered by the first task that pins. `DecodeDefinitionCollection` already returns the precise error (`internal/domain/workflow/definition.go:141-152`).
4. **Point the failure at the remedy.** Include the `replace` command (or the resource kind and version) in the pin error, so the message names the fix.
5. **Regression test.** Seed a collection, advance `ShippedDefinitions()` to a document with a changed step type, migrate, and assert the effective collection matches the new one. Note `internal/app/archied/step_vocabulary_test.go:169` already asserts the no-client fallback; the seeded-client path is the untested one.
## Related, but a distinct defect
Tracked separately as an in-repo task: triage re-routing a task to another workflow (`internal/domain/workflow/triage.go:85` sets only `tc.Task.Workflow = target`) leaves the old pin's YAML and digest in place, and `pinWorkflowDefinition` accepts an existing pin purely on self-consistency (`internal/daemon/daemon.go:1845-1850` — it never checks the YAML's id against `task.Workflow`). `internal/app/agentworker/task_execution.go:398-400` then rejects it as `pinned workflow definition does not match task identity`, parking a `transient` failure that retries into the identical error. Mentioned here only because it presents identically and was the first thing mistaken for this bug.
## Workaround
```bash
curl -sS -X POST \
-H "Authorization: Bearer $(cat ~/.local/share/archie/web-token)" \
-H 'Content-Type: application/json' \
--data "$(jq -c '{value: .defaults_json, expected_version: .version}' <catalog-response>)" \
https://<host>/api/control-plane/resources/workflow-definitions/commands/replace
```
Back up the current `resources.value` bytea first, then requeue the parked tasks through `POST /api/tasks/{id}/action {"action":"retry"}` rather than editing `tasks` directly.
---
_Imported from GitHub #1233 · type bug · priority P1 · created 2026-09-29 · closed 2026-09-29_
sam
added the highbug labels 2026-10-27 13:41:28 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Once a built-in workflow's step vocabulary changes, a State Store that has already seeded
workflow-definitionskeeps serving the old definition set forever. BecauseValidateDefinitionCollectionvalidates the collection as a single document, one entry naming a removed step type fails every task's pin — regardless of that task's own workflow. On this instance the effect was total: every task parked withand nothing could run until the resource was replaced by hand.
There is also no working automatic recovery path, because the seed's
RequestIDis stable andPutResourceshort-circuits on(kind, request_id)— so the obvious remedy (delete the resource and restart the State Store) writes nothing.Impact
implement(step 9), but the collection is failed as one unit.Observed
Instance upgraded v1.42.0 → v1.43.0 (State Store migrations
0012–0019applied2026-09-27 16:56:19Z, all 19 applied, so migrations are not implicated). The stored resource:Stored
implemententry named three step types that exist nowhere inorigin/main:implement.reviewimplement.open-primplement.post-review-commentsCurrent
Implement()replaces those with a singleStagePRReviewAndOpenPR(internal/domain/workflow/implement.go:200), and the comment atimplement.go:211-218records the removal explicitly (archie-core-afbk.7).resource_historyshows exactly one revision ever written for this kind:Confirmed against the running server's own catalog: the shipped document is 1955 bytes vs the stored 1494; the stored one also predates the
pr-reviewworkflow entirely.Defect 1 — a seeded kind is never refreshed
internal/app/controlplane/registry.go:96-115seeds only for a kind the store does not hold:The
Seedfunc is live (internal/app/controlplane/workflow_definitions.go:33-39returnsworkflow.ShippedDefinitions()), so the seed value never drifts — but it is only ever consulted when the kind is absent.ShippedDefinitions()is generated from the code (internal/domain/workflow/definition.go:181), which is exactly why it can't drift, and exactly why the persisted copy should not be trusted as the source of truth once written.Related asymmetry: the daemon falls back to
ShippedDefinitions()when it has no definitions client (internal/daemon/daemon.go:1854,internal/daemon/binding_target.go:26) — so a deployment with no client is immune, and one with a client is stuck. The bug only bites the correct wiring.Defect 2 — deleting the resource does not re-seed it
The intuitive recovery is "remove the row, restart, let it seed". That silently no-ops:
ResourceByRequestreadsresource_history, notresources(internal/infrastructure/postgres/queries/resources.sql):PutResourcereturns the existing row when that lookup hits (internal/infrastructure/postgres/resources.go:91-101), committing nothing.resource_historycarriesCREATE UNIQUE INDEX idx_resource_history_kind_request ON resource_history (kind, request_id)(migrations/0001_state_store.sql:112).Deleting
resourcesleaves history intact, so the stableRequestID: "import:workflow-definitions"still matches, the write is skipped, and the kind stays absent. I hit this exactly: after the delete the kind returnedErrResourceNotFound, which parks tasks on a different error rather than fixing anything.Defect 3 — the operator recovery that does work is undocumented
POST /api/control-plane/resources/{kind}/commands/replacewith the catalog'sdefaults_jsonand a correctexpected_versiondoes fix it, keeping the prior revision inresource_history(internal/app/controlplane/client.go:82-93). It is exercised byinternal/app/controlplane/step_vocabulary_agreement_test.go:254-280. Nothing in the failure path points an operator at it, and the error text says onlyunknown type.Suggested fix
SeedRevision > storedRevision, or deriveRequestIDfrom the seed's content/revision ("import:workflow-definitions:" + revision) soPutResource's short-circuit can't mask a changed payload. The second needs no new column but relies on the revision being bumped when a shipped definition changes.ShippedDefinitions()authoritative for built-ins. These were never operator-authored edits — the one revision on this instance isactor=system:migration,source=legacy-config. Distinguishing seeded built-ins from operator overrides would let an upgrade replace the former and keep the latter.DecodeDefinitionCollectionalready returns the precise error (internal/domain/workflow/definition.go:141-152).replacecommand (or the resource kind and version) in the pin error, so the message names the fix.ShippedDefinitions()to a document with a changed step type, migrate, and assert the effective collection matches the new one. Noteinternal/app/archied/step_vocabulary_test.go:169already asserts the no-client fallback; the seeded-client path is the untested one.Related, but a distinct defect
Tracked separately as an in-repo task: triage re-routing a task to another workflow (
internal/domain/workflow/triage.go:85sets onlytc.Task.Workflow = target) leaves the old pin's YAML and digest in place, andpinWorkflowDefinitionaccepts an existing pin purely on self-consistency (internal/daemon/daemon.go:1845-1850— it never checks the YAML's id againsttask.Workflow).internal/app/agentworker/task_execution.go:398-400then rejects it aspinned workflow definition does not match task identity, parking atransientfailure that retries into the identical error. Mentioned here only because it presents identically and was the first thing mistaken for this bug.Workaround
Back up the current
resources.valuebytea first, then requeue the parked tasks throughPOST /api/tasks/{id}/action {"action":"retry"}rather than editingtasksdirectly.Imported from GitHub #1233 · type bug · priority P1 · created 2026-09-29 · closed 2026-09-29