fix(controlplane): refresh a stale seeded resource and re-seed a removed one
A seed was consulted only for a resource kind the store did not hold, so a document that changed in the build never reached a database an earlier release had seeded. For workflow-definitions that is not a stale setting but a total outage: the collection validates as one document, so a stored step type this build no longer has fails every task's pin, whatever workflow the task names. The seed's request ID was stable per kind and the store answers a known request ID from resource_history, which outlives the resource, so the documented recovery -- remove the resource and restart -- wrote nothing either. Seeds now carry one identity the reader can name, and a kind the store holds is refreshed only when the seed itself wrote the newest revision: a file-derived value stays the store's, and an operator's replacement is never undone. The request ID is derived from the value, so an idempotent replay is still one revision without making a second, changed document impossible. A replay whose resource is gone re-creates it from the ledger's revision. The shipped-vs-file-derived distinction is the seed's to make, so it is a property of the definition (Defaults is set exactly for the kinds that ship a document) rather than a check at each consumer.
This commit is contained in:
1 parent
9bb3340dce
commit
64a4210b1f
10 files changed
+651
-41
No files matched your search
@@ -118,7 +118,7 @@
|
||||
"url": "/dev/development/control-plane/",
|
||||
"title": "Control plane and settings",
|
||||
"section": "Development",
|
||||
"body": "# Control plane and settings\n\nAuthority: `docs/architecture/configuration.md`. The wiring matrix, the\ndecoded-but-unwired ledger and the three shapes an unwired field takes are in\n`.agents/skills/archie-config-and-flags`.\n\nA field that parses is not a field that works. Before adding one, name the code\nthat reads it and the path the value takes to get there.\n\n## Decide what kind of setting it is\n\n- **Bootstrap-only** (read once at process start: `database_url`, `state_dir`,\n `work_dir`). It stays in the file, is checked by `validateBootstrap` in\n `internal/infrastructure/configuration/validate.go`, and the runtime overlay\n refuses changes to it.\n- **Database-owned** (repository policies, scheduling, tools, plugins,\n channels, containers). The file only seeds it. Once the State Store holds the\n control-plane resource, the stored document is layered over the file\n (`controlplane.Client.RuntimeConfig`) and the file key stops mattering. It is\n checked by `validateDatabaseOwned` and by the resource's validator.\n- **Worker-carried.** A value the agent container needs travels in\n `config.TaskConfig` (`Config.ForTask`) or on `taskrun.Request`. Never pass it\n as an environment variable the daemon did not already define, and never carry\n a secret this way.\n\n## Adding a database-owned field\n\n1. Add it to the wire-shape type the resource's `Document` names — a dedicated\n struct in `internal/app/controlplane/` with explicit snake_case json tags\n (`channel_settings.go`, `tool_settings.go`, `container_settings.go` are the\n precedents), never the internal config struct itself. A struct fed straight\n into the document falls back to Go field names (`Window`, `MaxRequests`),\n and a bare `time.Duration` marshals as nanoseconds — both render unusable\n in the Web UI. The resource's JSON schema and the dashboard's Advanced\n settings editor are generated from that type, so no UI work is needed for\n the editor itself.\n2. Map to and from the internal config type at the resource boundary (`Seed`,\n the layering in `runtime_config.go`, and the validator all use the same\n mapping). Durations in the document are a string-marshalling type\n (`config.Duration` or a document-local one), never a bare `time.Duration`.\n3. A document stored before the shape existed carries Go-cased keys. Read them\n with `controlplanerpc.DecodeRenamingLegacyKeys` (an `UnmarshalJSON` on the\n wire type) while keeping `DisallowUnknownFields` — dropping unknown keys\n would decode a misspelling as \"unset\" — and register a `Normalize` on the\n `Definition` that re-encodes the document, so a legacy document converges\n the next time it is written. A map or a nested struct needs its layering\n checked: a stored map replaces the file's, not merges into it, and a field\n the stored document omits keeps the file's value. Add a case to\n `internal/app/controlplane/runtime_config_test.go` for both.\n4. Validate it in one function both sides call, and add a case to\n `TestResourceValidatorsRejectWhatEffectiveValidationRejects` in\n `validators_agree_test.go`, so the file and the resource cannot disagree.\n5. Decide whether it reloads: it does only if every consumer re-reads it from\n `config.Holder`. The allowlist is `internal/app/archied/reload.go`; anything\n not on it requires a restart.\n6. If the settings page shows it as a plain field, add it to\n `internal/webui/config_schema.go` and the view in `api_config.go`.\n7. A generic catalog resource (`replace` command, not domain-managed, no\n query service) is watched **once per archie-ui process** by `webui.RunLive`,\n then published to every browser through `/api/stream`. Do not add an\n EventSource per resource to a settings store or a gRPC Watch per HTTP client.\n The control-plane server polls `Resource` every 250ms per upstream Watch;\n verify watch count is independent of client count and that late clients get\n the hub's current value (`internal/webui/live_hub_test.go`, real server over\n bufconn). Reopens resume after the last delivered version and charge failed\n attempts; accepting a gRPC stream does not establish its health.\n\n## Every new field\n\n- Test through the layering: build the effective config the way boot does,\n not by setting the field on the consumer.\n- Add a commented example to `config.example.toml`. Touch `deployments/*.toml`\n only when a profile there needs a value different from the default.\n- Update the wiring matrix in `.agents/skills/archie-config-and-flags`.\n- Secrets are `SecretRef`s resolved through the secret registry. Never inspect\n a resolved value, even in a test.\n- Reject a configuration that sets a field with no consumer rather than\n accepting and ignoring it (per-identity `forge.intake` is the precedent).\n"
|
||||
"body": "# Control plane and settings\n\nAuthority: `docs/architecture/configuration.md`. The wiring matrix, the\ndecoded-but-unwired ledger and the three shapes an unwired field takes are in\n`.agents/skills/archie-config-and-flags`.\n\nA field that parses is not a field that works. Before adding one, name the code\nthat reads it and the path the value takes to get there.\n\n## Decide what kind of setting it is\n\n- **Bootstrap-only** (read once at process start: `database_url`, `state_dir`,\n `work_dir`). It stays in the file, is checked by `validateBootstrap` in\n `internal/infrastructure/configuration/validate.go`, and the runtime overlay\n refuses changes to it.\n- **Database-owned** (repository policies, scheduling, tools, plugins,\n channels, containers). The file only seeds it. Once the State Store holds the\n control-plane resource, the stored document is layered over the file\n (`controlplane.Client.RuntimeConfig`) and the file key stops mattering. It is\n checked by `validateDatabaseOwned` and by the resource's validator. A stored\n value is refreshed from the file only when the store has no value for the kind\n at all; a kind that ships a document of its own (workflow definitions,\n personas) is refreshed whenever the seed itself wrote the stored copy, since\n that copy is a previous release's document rather than an operator's\n (`controlplane.Server.seedKind`).\n- **Worker-carried.** A value the agent container needs travels in\n `config.TaskConfig` (`Config.ForTask`) or on `taskrun.Request`. Never pass it\n as an environment variable the daemon did not already define, and never carry\n a secret this way.\n\n## Adding a database-owned field\n\n1. Add it to the wire-shape type the resource's `Document` names — a dedicated\n struct in `internal/app/controlplane/` with explicit snake_case json tags\n (`channel_settings.go`, `tool_settings.go`, `container_settings.go` are the\n precedents), never the internal config struct itself. A struct fed straight\n into the document falls back to Go field names (`Window`, `MaxRequests`),\n and a bare `time.Duration` marshals as nanoseconds — both render unusable\n in the Web UI. The resource's JSON schema and the dashboard's Advanced\n settings editor are generated from that type, so no UI work is needed for\n the editor itself.\n2. Map to and from the internal config type at the resource boundary (`Seed`,\n the layering in `runtime_config.go`, and the validator all use the same\n mapping). Durations in the document are a string-marshalling type\n (`config.Duration` or a document-local one), never a bare `time.Duration`.\n3. A document stored before the shape existed carries Go-cased keys. Read them\n with `controlplanerpc.DecodeRenamingLegacyKeys` (an `UnmarshalJSON` on the\n wire type) while keeping `DisallowUnknownFields` — dropping unknown keys\n would decode a misspelling as \"unset\" — and register a `Normalize` on the\n `Definition` that re-encodes the document, so a legacy document converges\n the next time it is written. A map or a nested struct needs its layering\n checked: a stored map replaces the file's, not merges into it, and a field\n the stored document omits keeps the file's value. Add a case to\n `internal/app/controlplane/runtime_config_test.go` for both.\n4. Validate it in one function both sides call, and add a case to\n `TestResourceValidatorsRejectWhatEffectiveValidationRejects` in\n `validators_agree_test.go`, so the file and the resource cannot disagree.\n5. Decide whether it reloads: it does only if every consumer re-reads it from\n `config.Holder`. The allowlist is `internal/app/archied/reload.go`; anything\n not on it requires a restart.\n6. If the settings page shows it as a plain field, add it to\n `internal/webui/config_schema.go` and the view in `api_config.go`.\n7. A generic catalog resource (`replace` command, not domain-managed, no\n query service) is watched **once per archie-ui process** by `webui.RunLive`,\n then published to every browser through `/api/stream`. Do not add an\n EventSource per resource to a settings store or a gRPC Watch per HTTP client.\n The control-plane server polls `Resource` every 250ms per upstream Watch;\n verify watch count is independent of client count and that late clients get\n the hub's current value (`internal/webui/live_hub_test.go`, real server over\n bufconn). Reopens resume after the last delivered version and charge failed\n attempts; accepting a gRPC stream does not establish its health.\n\n## Every new field\n\n- Test through the layering: build the effective config the way boot does,\n not by setting the field on the consumer.\n- Add a commented example to `config.example.toml`. Touch `deployments/*.toml`\n only when a profile there needs a value different from the default.\n- Update the wiring matrix in `.agents/skills/archie-config-and-flags`.\n- Secrets are `SecretRef`s resolved throLine truncated
|
||||
},
|
||||
{
|
||||
"url": "/dev/development/daemon/",
|
||||
@@ -448,7 +448,7 @@
|
||||
"url": "/dev/prds/runtime-control-plane/",
|
||||
"title": "Runtime control plane",
|
||||
"section": "PRDs",
|
||||
"body": "# Runtime control plane\n\n**Status:** Approved \n**Date:** 2026-09-20 \n**Tracking:** `archie-core-j28m`, `archie-core-1786637498420-327-bacadaae`\n\n## Outcome\n\nArchie can be managed from the Web UI or a messaging channel. Both use the same\nAPI.\n\n## Model\n\nEach feature owns its settings and commands. The database stores them. There is\nno global configuration document.\n\nThis includes:\n\n- workflows;\n- identities and bindings;\n- providers and models;\n- repository policies;\n- channels and schedules;\n- capture mappings and bindings;\n- tool settings;\n- prompts and personas;\n- plugin settings.\n\nPlugin code remains installed code. Only its settings are stored in the\ndatabase.\n\nEach feature defines its data, validation, permissions, editor fields, and how\nchanges take effect. Each feature also defines its allowed actions. For example,\nidentities can be created, suspended, and retired.\n\nRepository and identity settings remain separate. They are not stored as large\nrecords that unrelated features must share.\n\nThe State Store process hosts the control-plane API on its existing gRPC server.\nNo new process is added. Feature code validates changes. Storage code only\nwrites versions and audit records.\n\nEvery write records who made it, where it came from, a request ID, and the\nversion being edited. The API layer sets the actor; the request body cannot.\nThe server validates the change, then writes it with its audit record. It rejects\nedits to an old version.\n\n## API\n\nThe API has four operations:\n\n1. **Catalogue** tells clients what can be viewed or changed and how to edit it.\n2. **Query** reads data and versions without exposing secrets.\n3. **Command** asks the owning feature to make a change.\n4. **Watch** streams changes and errors to clients.\n\nComplex features can use custom screens. They still use the same commands.\n\nA dashboard session or channel sender must map to an Archie identity before it\ncan run a command. Messaging syntax maps to the same commands as the Web UI.\n\n`archie-ui` receives only an API client. It does not receive a database handle,\ndaemon pointer, global configuration object, or secret value.\n\nThe database stores the requested settings. Each process reports the version it\nis using or why it failed. The UI shows both.\n\nSome changes require a restart. The UI shows them as pending. When a live change\nreplaces a running component, Archie starts and checks the new one before\nswitching. If the check fails, the old one keeps running.\n\n## Bootstrap, migration, and recovery\n\nBootstrap contains only the database path, listen addresses, credentials, TLS,\nand encryption-key references. State Store and control-plane RPCs use the same\nendpoint.\n\nWhen upgraded, existing instances are migrated to the database. Existing\ndatabase values are not overwritten. After migration, settings in TOML are\nignored and cannot block State Store startup.\n\nThe Web UI remains available when `archied` or Gateway is down. It shows failed\nchanges and can correct or roll them back.\n\nOffline commands can back up, restore, validate, and roll back the database.\nRecovery does not require `archied` or the Web UI. The database stores secret\nreferences, not secret values.\n\n## Workflows\n\nUsers create and edit workflows in the Web UI. Workflows are stored as versioned\nYAML. Built-in workflows can be overridden and restored.\n\nA workflow is a list of steps. Each step has a known type and its own settings.\nPlugins can add step types. Each run keeps the workflow version it started with.\n\nYaegi, `.archie/stages/*.go`, and `.archie/gate.go` will be removed. Their useful\nbehaviour must first be available as normal workflow steps.\n\n## First implementation\n\nThe first implementation makes task limits editable: maximum model steps,\nmaximum runtime, and maximum consecutive gate failures.\n\nIt includes:\n\n- validated task-limit fields;\n- version history and audit records;\n- the four API operations;\n- authenticated Web UI and messaging access;\n- live updates and apply status;\n- a Pinia-backed editor in the Web UI;\n- a messaging command using the same API;\n- migration of existing `[budgets]` values;\n- an end-to-end restart test.\n\nRunning tasks keep the limits they started with. New tasks use the latest applied\nversion. This replaces the old settings overlay instead of extending it.\n\n## Not included in the first implementation\n\n- Moving bootstrap or secret values into normal settings.\n- A generic JSON editor.\n- Rebuilding every settings screen at once.\n- Changing the limits of a running task.\n"
|
||||
"body": "# Runtime control plane\n\n**Status:** Approved \n**Date:** 2026-09-20 \n**Tracking:** `archie-core-j28m`, `archie-core-1786637498420-327-bacadaae`\n\n## Outcome\n\nArchie can be managed from the Web UI or a messaging channel. Both use the same\nAPI.\n\n## Model\n\nEach feature owns its settings and commands. The database stores them. There is\nno global configuration document.\n\nThis includes:\n\n- workflows;\n- identities and bindings;\n- providers and models;\n- repository policies;\n- channels and schedules;\n- capture mappings and bindings;\n- tool settings;\n- prompts and personas;\n- plugin settings.\n\nPlugin code remains installed code. Only its settings are stored in the\ndatabase.\n\nEach feature defines its data, validation, permissions, editor fields, and how\nchanges take effect. Each feature also defines its allowed actions. For example,\nidentities can be created, suspended, and retired.\n\nRepository and identity settings remain separate. They are not stored as large\nrecords that unrelated features must share.\n\nThe State Store process hosts the control-plane API on its existing gRPC server.\nNo new process is added. Feature code validates changes. Storage code only\nwrites versions and audit records.\n\nEvery write records who made it, where it came from, a request ID, and the\nversion being edited. The API layer sets the actor; the request body cannot.\nThe server validates the change, then writes it with its audit record. It rejects\nedits to an old version.\n\n## API\n\nThe API has four operations:\n\n1. **Catalogue** tells clients what can be viewed or changed and how to edit it.\n2. **Query** reads data and versions without exposing secrets.\n3. **Command** asks the owning feature to make a change.\n4. **Watch** streams changes and errors to clients.\n\nComplex features can use custom screens. They still use the same commands.\n\nA dashboard session or channel sender must map to an Archie identity before it\ncan run a command. Messaging syntax maps to the same commands as the Web UI.\n\n`archie-ui` receives only an API client. It does not receive a database handle,\ndaemon pointer, global configuration object, or secret value.\n\nThe database stores the requested settings. Each process reports the version it\nis using or why it failed. The UI shows both.\n\nSome changes require a restart. The UI shows them as pending. When a live change\nreplaces a running component, Archie starts and checks the new one before\nswitching. If the check fails, the old one keeps running.\n\n## Bootstrap, migration, and recovery\n\nBootstrap contains only the database path, listen addresses, credentials, TLS,\nand encryption-key references. State Store and control-plane RPCs use the same\nendpoint.\n\nWhen upgraded, existing instances are migrated to the database. Existing\ndatabase values are not overwritten, with one exception: a resource whose\ndocument the build ships is refreshed when the seed itself wrote the stored\ncopy, because that copy is a previous release's document rather than an\noperator's. After migration, settings in TOML are ignored and cannot block\nState Store startup.\n\nThe Web UI remains available when `archied` or Gateway is down. It shows failed\nchanges and can correct or roll them back.\n\nOffline commands can back up, restore, validate, and roll back the database.\nRecovery does not require `archied` or the Web UI. The database stores secret\nreferences, not secret values.\n\n## Workflows\n\nUsers create and edit workflows in the Web UI. Workflows are stored as versioned\nYAML. Built-in workflows can be overridden and restored.\n\nA workflow is a list of steps. Each step has a known type and its own settings.\nPlugins can add step types. Each run keeps the workflow version it started with.\n\nYaegi, `.archie/stages/*.go`, and `.archie/gate.go` will be removed. Their useful\nbehaviour must first be available as normal workflow steps.\n\n## First implementation\n\nThe first implementation makes task limits editable: maximum model steps,\nmaximum runtime, and maximum consecutive gate failures.\n\nIt includes:\n\n- validated task-limit fields;\n- version history and audit records;\n- the four API operations;\n- authenticated Web UI and messaging access;\n- live updates and apply status;\n- a Pinia-backed editor in the Web UI;\n- a messaging command using the same API;\n- migration of existing `[budgets]` values;\n- an end-to-end restart test.\n\nRunning tasks keep the limits they started with. New tasks use the latest applied\nversion. This replaces the old settings overlay instead of extending it.\n\n## Not included in the first implementation\n\n- Moving bootstrap or secret values into normal settings.\n- A generic JSON editor.\n- Rebuilding every settings screen at once.\n- Changing the limits of a running task.\n"
|
||||
},
|
||||
{
|
||||
"url": "/dev/prds/self-update-transaction/",
|
||||
|
||||
@@ -17,7 +17,12 @@ that reads it and the path the value takes to get there.
|
||||
channels, containers). The file only seeds it. Once the State Store holds the
|
||||
control-plane resource, the stored document is layered over the file
|
||||
(`controlplane.Client.RuntimeConfig`) and the file key stops mattering. It is
|
||||
checked by `validateDatabaseOwned` and by the resource's validator.
|
||||
checked by `validateDatabaseOwned` and by the resource's validator. A stored
|
||||
value is refreshed from the file only when the store has no value for the kind
|
||||
at all; a kind that ships a document of its own (workflow definitions,
|
||||
personas) is refreshed whenever the seed itself wrote the stored copy, since
|
||||
that copy is a previous release's document rather than an operator's
|
||||
(`controlplane.Server.seedKind`).
|
||||
- **Worker-carried.** A value the agent container needs travels in
|
||||
`config.TaskConfig` (`Config.ForTask`) or on `taskrun.Request`. Never pass it
|
||||
as an environment variable the daemon did not already define, and never carry
|
||||
|
||||
@@ -76,8 +76,11 @@ and encryption-key references. State Store and control-plane RPCs use the same
|
||||
endpoint.
|
||||
|
||||
When upgraded, existing instances are migrated to the database. Existing
|
||||
database values are not overwritten. After migration, settings in TOML are
|
||||
ignored and cannot block State Store startup.
|
||||
database values are not overwritten, with one exception: a resource whose
|
||||
document the build ships is refreshed when the seed itself wrote the stored
|
||||
copy, because that copy is a previous release's document rather than an
|
||||
operator's. After migration, settings in TOML are ignored and cannot block
|
||||
State Store startup.
|
||||
|
||||
The Web UI remains available when `archied` or Gateway is down. It shows failed
|
||||
changes and can correct or roll them back.
|
||||
|
||||
@@ -3,6 +3,8 @@ package controlplane
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
@@ -73,14 +75,47 @@ type SeedSkip struct {
|
||||
Err error
|
||||
}
|
||||
|
||||
// ImportConfig seeds every control-plane resource that has no stored value yet
|
||||
// from the file config, and reports the version of every kind it left alone.
|
||||
// The audit identity every seed writes under. It is a constant rather than a
|
||||
// literal at each call site because a later boot reads it back: it is how a
|
||||
// stored value is told apart from one an operator replaced (seedWroteNewest),
|
||||
// and a writer whose identity its own reader cannot name cannot make that
|
||||
// distinction at all.
|
||||
const (
|
||||
seedActor = "system:migration"
|
||||
seedSource = "legacy-config"
|
||||
)
|
||||
|
||||
// ImportConfig brings every control-plane resource up to date with the value
|
||||
// this build seeds for it, and reports the version each kind is left at.
|
||||
//
|
||||
// A seed the resource's own validator refuses is SKIPPED and returned in skipped,
|
||||
// not fatal. Nothing is written for that kind, so it stays ABSENT and the file
|
||||
// document's value stays the one in effect (Client.RuntimeConfig leaves an absent
|
||||
// kind alone), with the fix still in the file. The seed is retried on the next
|
||||
// start of this process, so correcting config.toml re-seeds it.
|
||||
// For a kind the store does not hold, that is the seed itself. For a kind it
|
||||
// does hold, the stored value wins and the seed writes nothing -- with one
|
||||
// exception, because the two ways a kind gets its content are not symmetric:
|
||||
//
|
||||
// - A kind derived from the file config is seeded once and then owned by the
|
||||
// store. Editing config.toml does not reach a database that holds a value
|
||||
// for it, which is the whole point of the file being only a seed.
|
||||
// - A kind whose document the build ships (the ones that set Definition.Defaults,
|
||||
// the same set the catalog offers as "restore shipped") is not derived from
|
||||
// operator input at all, so a stored copy is a copy of an older build's
|
||||
// document and nothing else. A value like that is brought up to date, or a
|
||||
// State Store upgraded under a running deployment keeps serving the previous
|
||||
// release's document forever. For workflow-definitions that is not a stale
|
||||
// setting but an outage: the collection validates as one document, so a
|
||||
// stored step type this build no longer has fails every task's pin, whatever
|
||||
// workflow the task names.
|
||||
//
|
||||
// A refreshed kind keeps its history: the value it replaced stays a revision in
|
||||
// resource_history. What is replaced is only the seed's own work -- a
|
||||
// definition set an operator replaced is theirs and is left alone, on the boot
|
||||
// that finds it and on every boot after.
|
||||
//
|
||||
// A seed the resource's own validator refuses is SKIPPED and returned in
|
||||
// skipped, not fatal. Nothing is written for that kind, so it stays ABSENT and
|
||||
// the file document's value stays the one in effect (Client.RuntimeConfig
|
||||
// leaves an absent kind alone), with the fix still in the file. The seed is
|
||||
// retried on the next start of this process, so correcting config.toml re-seeds
|
||||
// it.
|
||||
// docs/prds/runtime-control-plane.md, "Bootstrap, migration, and recovery":
|
||||
// after migration, settings in TOML are ignored and cannot block State Store
|
||||
// startup -- and this import runs on the State Store's startup path. Fail closed
|
||||
@@ -91,35 +126,130 @@ func (s *Server) ImportConfig(ctx context.Context, cfg config.Config) (map[strin
|
||||
versions := make(map[string]int64, len(s.ordered))
|
||||
var skipped []SeedSkip
|
||||
for _, definition := range s.ordered {
|
||||
resource, err := s.store.Resource(ctx, definition.Kind)
|
||||
if err == nil {
|
||||
versions[definition.Kind] = resource.Version
|
||||
continue
|
||||
}
|
||||
if !errors.Is(err, storecontract.ErrResourceNotFound) {
|
||||
version, err := s.seedKind(ctx, definition, cfg)
|
||||
var refusal *seedRefusal
|
||||
switch {
|
||||
case errors.As(err, &refusal):
|
||||
skipped = append(skipped, SeedSkip{Kind: definition.Kind, Err: refusal.err})
|
||||
case err != nil:
|
||||
return nil, nil, err
|
||||
default:
|
||||
versions[definition.Kind] = version
|
||||
}
|
||||
value, err := definition.seededValue(cfg)
|
||||
if err != nil {
|
||||
// Skipped and reported, not fatal: nothing is written, so the kind
|
||||
// stays absent and the file document's value is the one in effect,
|
||||
// while the process that would RUN the value still fails closed on
|
||||
// it (boot.runtimeConfig validates the effective document).
|
||||
// seededValue wraps a seed's encode and validation failures together,
|
||||
// which is why both take this path: the alternative is a second seed
|
||||
// implementation here to tell them apart.
|
||||
skipped = append(skipped, SeedSkip{Kind: definition.Kind, Err: err})
|
||||
continue
|
||||
}
|
||||
resource, err = s.store.PutResource(ctx, storecontract.ResourceWrite{Kind: definition.Kind, Value: value, Actor: "system:migration", Source: "legacy-config", RequestID: "import:" + definition.Kind, ExpectedVersion: 0, At: time.Now().UTC()})
|
||||
if err != nil {
|
||||
return nil, nil, fmt.Errorf("seed %s: %w", definition.Kind, err)
|
||||
}
|
||||
versions[definition.Kind] = resource.Version
|
||||
}
|
||||
return versions, skipped, nil
|
||||
}
|
||||
|
||||
// seedRefusal is the seed's own validation refusing the value this build
|
||||
// derived. It is an error because it is one -- there is nothing to write -- and a
|
||||
// distinct type because ImportConfig answers it by reporting the kind rather
|
||||
// than failing the boot (see ImportConfig on why a bad seed is not fatal).
|
||||
type seedRefusal struct{ err error }
|
||||
|
||||
func (r *seedRefusal) Error() string { return r.err.Error() }
|
||||
|
||||
func (r *seedRefusal) Unwrap() error { return r.err }
|
||||
|
||||
// seedKind seeds one kind if the store holds nothing for it, refreshes a
|
||||
// shipped document the seed itself wrote, and otherwise leaves the stored value
|
||||
// alone, returning the version the kind is left at. A *seedRefusal means the
|
||||
// value this build derived is not one the resource's validator accepts, and
|
||||
// nothing was written.
|
||||
func (s *Server) seedKind(ctx context.Context, definition Definition, cfg config.Config) (int64, error) {
|
||||
stored, storedErr := s.store.Resource(ctx, definition.Kind)
|
||||
absent := errors.Is(storedErr, storecontract.ErrResourceNotFound)
|
||||
if storedErr != nil && !absent {
|
||||
return 0, storedErr
|
||||
}
|
||||
value, err := definition.seededValue(cfg)
|
||||
if err != nil {
|
||||
// Skipped and reported, not fatal: nothing is written, so the kind
|
||||
// stays absent and the file document's value is the one in effect,
|
||||
// while the process that would RUN the value still fails closed on
|
||||
// it (boot.runtimeConfig validates the effective document).
|
||||
// seededValue wraps a seed's encode and validation failures together,
|
||||
// which is why both take this path: the alternative is a second seed
|
||||
// implementation here to tell them apart.
|
||||
return 0, &seedRefusal{err: err}
|
||||
}
|
||||
if !absent && bytes.Equal(stored.Value, value) {
|
||||
// The store already holds what this build seeds. Comparing the values
|
||||
// rather than re-writing them is what keeps a boot from adding a
|
||||
// revision for a document that did not change.
|
||||
return stored.Version, nil
|
||||
}
|
||||
if !absent {
|
||||
refresh, err := s.shippedValueIsStale(ctx, definition)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if !refresh {
|
||||
return stored.Version, nil
|
||||
}
|
||||
}
|
||||
write := storecontract.ResourceWrite{
|
||||
Kind: definition.Kind, Value: value,
|
||||
Actor: seedActor, Source: seedSource,
|
||||
RequestID: seedRequestID(definition.Kind, value),
|
||||
ExpectedVersion: expectedVersion(stored, absent), At: time.Now().UTC(),
|
||||
}
|
||||
resource, err := s.store.PutResource(ctx, write)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("seed %s: %w", definition.Kind, err)
|
||||
}
|
||||
return resource.Version, nil
|
||||
}
|
||||
|
||||
// expectedVersion is the optimistic-concurrency guard a seed writes under: the
|
||||
// revision it is replacing, or nothing at all for a kind that has none.
|
||||
func expectedVersion(stored storecontract.Resource, absent bool) int64 {
|
||||
if absent {
|
||||
return 0
|
||||
}
|
||||
return stored.Version
|
||||
}
|
||||
|
||||
// shippedValueIsStale reports whether a stored value is the build's to replace:
|
||||
// the kind ships a document of its own (Defaults is set exactly for those, and
|
||||
// a file-derived seed never overwrites a stored value) and the revision the
|
||||
// seed itself wrote is the newest one. A newer revision carrying any other
|
||||
// identity is an operator's replacement, and an operator's replacement is not
|
||||
// the seed's to undo.
|
||||
func (s *Server) shippedValueIsStale(ctx context.Context, definition Definition) (bool, error) {
|
||||
if definition.Defaults == nil {
|
||||
return false, nil
|
||||
}
|
||||
return s.seedWroteNewest(ctx, definition.Kind)
|
||||
}
|
||||
|
||||
// seedWroteNewest reports whether the newest revision of kind was written by a
|
||||
// seed. It reads the ledger rather than the current row because the ledger is
|
||||
// what records who wrote a value, and the row does not.
|
||||
func (s *Server) seedWroteNewest(ctx context.Context, kind string) (bool, error) {
|
||||
history, err := s.store.ResourceHistory(ctx, kind, 1)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("read %s history: %w", kind, err)
|
||||
}
|
||||
if len(history) == 0 {
|
||||
return false, nil
|
||||
}
|
||||
return history[0].Actor == seedActor && history[0].Source == seedSource, nil
|
||||
}
|
||||
|
||||
// seedRequestID is the idempotency key a seed writes under. It is derived from
|
||||
// the value rather than from the kind alone, because the key is spent the
|
||||
// moment it is in the ledger (resource_history's UNIQUE (kind, request_id), and
|
||||
// PutResource answers a known key from that ledger): a key derived from the
|
||||
// kind makes every later value a replay of the first one ever written, which is
|
||||
// how a document that changed in the build never reached a database the seed
|
||||
// had already written. Deriving it from the value keeps the write idempotent --
|
||||
// the same document twice is still one revision -- without making the second
|
||||
// document impossible.
|
||||
func seedRequestID(kind string, value []byte) string {
|
||||
sum := sha256.Sum256(value)
|
||||
return "import:" + kind + ":" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
// seededValue is the value the State Store writes for a kind it does not hold
|
||||
// yet: the definition's seed derived from cfg, run through the same Decode a
|
||||
// write applies. ImportConfig stores it; the offline config layering reads it
|
||||
|
||||
@@ -0,0 +1,319 @@
|
||||
package controlplane
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/samcharles93/archie-core/internal/config"
|
||||
"github.com/samcharles93/archie-core/internal/domain/storecontract"
|
||||
"github.com/samcharles93/archie-core/internal/domain/workflow"
|
||||
"github.com/samcharles93/archie-core/internal/infrastructure/postgres/pgstore"
|
||||
)
|
||||
|
||||
// retiredStepType names a step type no registry declares. A stored collection
|
||||
// carrying one is what a database seeded by an earlier release holds once a
|
||||
// built-in workflow's step vocabulary changes under it.
|
||||
const retiredStepType = "legacy.retired-step"
|
||||
|
||||
// staleWorkflowDefinitions is a stored workflow-definitions value this build
|
||||
// refuses. ValidateDefinitionCollection validates the collection as one
|
||||
// document, which is what makes one stale entry fail every task's pin, whatever
|
||||
// workflow that task names.
|
||||
//
|
||||
// It is encoded directly rather than through encodeWorkflowDefinitions, because
|
||||
// that runs the validator a write applies: the point of the fixture is the
|
||||
// document an earlier release wrote, which this build's write path would refuse
|
||||
// (the validator cannot bless the value whose rejection is the bug reported).
|
||||
func staleWorkflowDefinitions(t *testing.T) []byte {
|
||||
t.Helper()
|
||||
value, err := json.Marshal(workflow.WorkflowDefinitionCollection{Definitions: []workflow.WorkflowDefinitionEntry{
|
||||
{ID: "legacy", YAML: "id: legacy\nsteps:\n - type: " + retiredStepType + "\n"},
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatalf("encode the stale definitions: %v", err)
|
||||
}
|
||||
if _, err := workflow.DecodeDefinitionCollection(value, testSteps(t).Registry()); err == nil {
|
||||
t.Fatal("the stored value under test decodes; it is not the stale state this test is about")
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
// writeSeeded writes value as the revision an earlier release's seed left
|
||||
// behind: the seed's own audit identity, and the seed's own request ID for that
|
||||
// value. The producer supplies both -- the value is the only thing substituted
|
||||
// -- so a test does not restate a key or an identity it would then be pinning.
|
||||
func writeSeeded(t *testing.T, resources *pgstore.TaskDB, kind string, value []byte) storecontract.Resource {
|
||||
t.Helper()
|
||||
if current, err := resources.Resource(t.Context(), kind); err == nil {
|
||||
t.Fatalf("%s already holds version %d; a seeded fixture writes the first revision", kind, current.Version)
|
||||
} else if !errors.Is(err, storecontract.ErrResourceNotFound) {
|
||||
t.Fatalf("read %s: %v", kind, err)
|
||||
}
|
||||
seeded, err := resources.PutResource(t.Context(), storecontract.ResourceWrite{
|
||||
Kind: kind, Value: value, ExpectedVersion: 0,
|
||||
Actor: seedActor, Source: seedSource, RequestID: seedRequestID(kind, value),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("write the seeded %s revision: %v", kind, err)
|
||||
}
|
||||
return seeded
|
||||
}
|
||||
|
||||
// replaceAsOperator replaces a kind the way the dashboard does: a write carrying
|
||||
// the operator's attribution, which is what marks the value as theirs.
|
||||
func replaceAsOperator(t *testing.T, resources *pgstore.TaskDB, kind string, value []byte, expected int64) storecontract.Resource {
|
||||
t.Helper()
|
||||
replaced, err := resources.PutResource(t.Context(), storecontract.ResourceWrite{
|
||||
Kind: kind, Value: value, ExpectedVersion: expected,
|
||||
Actor: "operator", Source: "archie-ui", RequestID: "operator-replace-" + kind,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("replace %s: %v", kind, err)
|
||||
}
|
||||
return replaced
|
||||
}
|
||||
|
||||
// TestImportConfigRefreshesAStaleShippedValue is defect 1 of #1233. A seed was
|
||||
// consulted only for a kind the store did not hold, so a document that changed
|
||||
// in the build never reached a database an earlier release seeded -- and for
|
||||
// workflow-definitions that is not a stale setting but an outage, because the
|
||||
// collection validates as one document and a step type this build no longer has
|
||||
// fails every task's pin.
|
||||
func TestImportConfigRefreshesAStaleShippedValue(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resources := pgstore.Open(t)
|
||||
defer resources.Close()
|
||||
server := testServer(t, resources)
|
||||
// The state under test is "the seed's own document, one release old" -- the
|
||||
// state the issue reports -- rather than "a value somebody else wrote",
|
||||
// which is the distinction the refresh turns on.
|
||||
seeded := writeSeeded(t, resources, WorkflowDefinitionsKind, staleWorkflowDefinitions(t))
|
||||
|
||||
versions, _, err := server.ImportConfig(t.Context(), config.Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("ImportConfig: %v", err)
|
||||
}
|
||||
refreshed, err := resources.Resource(t.Context(), WorkflowDefinitionsKind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if refreshed.Version <= seeded.Version {
|
||||
t.Fatalf("%s is still at version %d: the stale seed value was not replaced", WorkflowDefinitionsKind, refreshed.Version)
|
||||
}
|
||||
definitions, err := workflow.DecodeDefinitionCollection(refreshed.Value, testSteps(t).Registry())
|
||||
if err != nil {
|
||||
t.Fatalf("the stored workflow definitions still do not decode, so no task can pin: %v", err)
|
||||
}
|
||||
if len(definitions.Definitions) != len(workflow.ShippedDefinitions().Definitions) {
|
||||
t.Fatalf("stored definitions = %d, want the shipped %d", len(definitions.Definitions), len(workflow.ShippedDefinitions().Definitions))
|
||||
}
|
||||
if versions[WorkflowDefinitionsKind] != refreshed.Version {
|
||||
t.Fatalf("the import reported version %d, want the version the store holds, %d", versions[WorkflowDefinitionsKind], refreshed.Version)
|
||||
}
|
||||
// A refresh is a write like any other: what it replaced stays a revision, so
|
||||
// the value the previous release shipped is still readable.
|
||||
history, err := resources.ResourceHistory(t.Context(), WorkflowDefinitionsKind, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(history) != 2 || !bytes.Equal(history[1].Value, staleWorkflowDefinitions(t)) {
|
||||
t.Fatalf("history = %+v, want the replaced value kept as a revision", history)
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportConfigLeavesASeededValueAloneOnceItMatches pins the other half of the
|
||||
// refresh: a boot that finds the value it would write writes nothing. A seed that
|
||||
// re-wrote its own document on every start would add a revision per boot, and the
|
||||
// revision an operator reads as "what I am running" would move for no reason.
|
||||
func TestImportConfigLeavesASeededValueAloneOnceItMatches(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resources := pgstore.Open(t)
|
||||
defer resources.Close()
|
||||
server := testServer(t, resources)
|
||||
for attempt := 1; attempt <= 2; attempt++ {
|
||||
if _, _, err := server.ImportConfig(t.Context(), config.Config{}); err != nil {
|
||||
t.Fatalf("ImportConfig %d: %v", attempt, err)
|
||||
}
|
||||
stored, err := resources.Resource(t.Context(), WorkflowDefinitionsKind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if stored.Version != 1 {
|
||||
t.Fatalf("after %d seed(s) %s is at version %d, want the one revision", attempt, WorkflowDefinitionsKind, stored.Version)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportConfigKeepsAStoredValueThatIsNotTheSeeds pins the half the refresh
|
||||
// must not break. The seed writes a value it produced itself; a value it did not
|
||||
// produce is somebody's, and two kinds of somebody are covered here:
|
||||
//
|
||||
// - an operator's replacement, which the seed must leave alone on the boot
|
||||
// that finds it and on every boot after -- a refresh that consulted the
|
||||
// store's copy rather than the ledger would take the value back the moment
|
||||
// it had written it once;
|
||||
// - a kind derived from the file config, where the file is only a seed by
|
||||
// design, so a changed config.toml does not reach a database that holds a
|
||||
// value for it.
|
||||
func TestImportConfigKeepsAStoredValueThatIsNotTheSeeds(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
kind string
|
||||
// settle leaves the store holding the value the imports must not touch,
|
||||
// and returns that value.
|
||||
settle func(t *testing.T, resources *pgstore.TaskDB) []byte
|
||||
// cfg is the file config the imports run with. For the file-derived
|
||||
// case it names a different value from the one that was seeded.
|
||||
cfg func() config.Config
|
||||
// verify checks the stored value after the imports, given the value
|
||||
// settle left behind.
|
||||
verify func(t *testing.T, got, want []byte)
|
||||
}{
|
||||
{
|
||||
name: "an operator's replacement, whose shipped document also moved on",
|
||||
kind: WorkflowDefinitionsKind,
|
||||
cfg: func() config.Config { return config.Config{} },
|
||||
settle: func(t *testing.T, resources *pgstore.TaskDB) []byte {
|
||||
t.Helper()
|
||||
// The seed wrote a document this build refuses, and the operator
|
||||
// then replaced it. The stored value is therefore neither the
|
||||
// seed's nor the one this build ships, which is exactly the
|
||||
// state a refresh keyed on authorship has to leave alone.
|
||||
seeded := writeSeeded(t, resources, WorkflowDefinitionsKind, staleWorkflowDefinitions(t))
|
||||
value, err := encodeWorkflowDefinitions(workflow.WorkflowDefinitionCollection{Definitions: []workflow.WorkflowDefinitionEntry{
|
||||
{ID: "operator-only", YAML: "id: operator-only\nsteps:\n - type: bootstrap.apply\n"},
|
||||
}}, testSteps(t).Registry())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return replaceAsOperator(t, resources, WorkflowDefinitionsKind, value, seeded.Version).Value
|
||||
},
|
||||
verify: func(t *testing.T, got, want []byte) {
|
||||
t.Helper()
|
||||
definitions, err := workflow.DecodeDefinitionCollection(got, testSteps(t).Registry())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !bytes.Equal(got, want) || len(definitions.Definitions) != 1 || definitions.Definitions[0].ID != "operator-only" {
|
||||
t.Fatalf("stored value = %s, want the operator's replacement kept", got)
|
||||
}
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "a value derived from the file config, whose config.toml changed",
|
||||
kind: RepositoryPoliciesKind,
|
||||
cfg: func() config.Config {
|
||||
cfg := validConfigForValidation()
|
||||
cfg.Repos = []config.Repo{{Owner: "acme", Name: "widget"}}
|
||||
return cfg
|
||||
},
|
||||
settle: func(t *testing.T, resources *pgstore.TaskDB) []byte {
|
||||
t.Helper()
|
||||
resolved := validConfigForValidation()
|
||||
resolved.Repos = []config.Repo{{Owner: "acme", Name: "app"}}
|
||||
if _, _, err := testServer(t, resources).ImportConfig(t.Context(), resolved); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
stored, err := resources.Resource(t.Context(), RepositoryPoliciesKind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return stored.Value
|
||||
},
|
||||
verify: func(t *testing.T, got, want []byte) {
|
||||
t.Helper()
|
||||
if !bytes.Equal(got, want) {
|
||||
t.Fatalf("stored repository policies = %s, want the seeded %s: the file is a seed, not the owner", got, want)
|
||||
}
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resources := pgstore.Open(t)
|
||||
defer resources.Close()
|
||||
server := testServer(t, resources)
|
||||
want := tt.settle(t, resources)
|
||||
before, err := resources.Resource(t.Context(), tt.kind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
for attempt := 1; attempt <= 2; attempt++ {
|
||||
versions, _, err := server.ImportConfig(t.Context(), tt.cfg())
|
||||
if err != nil {
|
||||
t.Fatalf("ImportConfig %d: %v", attempt, err)
|
||||
}
|
||||
after, err := resources.Resource(t.Context(), tt.kind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if after.Version != before.Version {
|
||||
t.Fatalf("import %d moved %s from version %d to %d: the stored value is not the seed's to replace",
|
||||
attempt, tt.kind, before.Version, after.Version)
|
||||
}
|
||||
if versions[tt.kind] != before.Version {
|
||||
t.Fatalf("import %d reported version %d for %s, want the stored %d", attempt, versions[tt.kind], tt.kind, before.Version)
|
||||
}
|
||||
tt.verify(t, after.Value, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestImportConfigRecreatesAKindAnOperatorRemoved is defect 2 of #1233: the
|
||||
// documented recovery -- remove the resource, restart -- wrote nothing. The seed's
|
||||
// request ID is in resource_history, that ledger outlives the row it describes,
|
||||
// and a write whose request ID is already in the ledger is answered as a replay,
|
||||
// so the seed reported a version for a kind that was not there and left it absent.
|
||||
func TestImportConfigRecreatesAKindAnOperatorRemoved(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
resources := pgstore.Open(t)
|
||||
defer resources.Close()
|
||||
server := testServer(t, resources)
|
||||
if _, _, err := server.ImportConfig(t.Context(), config.Config{}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
seeded, err := resources.Resource(t.Context(), WorkflowDefinitionsKind)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
history, err := resources.ResourceHistory(t.Context(), WorkflowDefinitionsKind, 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(history) != 1 {
|
||||
t.Fatalf("seed revisions = %d, want the one the removal leaves behind", len(history))
|
||||
}
|
||||
|
||||
if _, err := resources.Pool.Exec(t.Context(), `DELETE FROM resources WHERE kind = $1`, WorkflowDefinitionsKind); err != nil {
|
||||
t.Fatalf("remove the resource: %v", err)
|
||||
}
|
||||
if _, err := resources.Resource(t.Context(), WorkflowDefinitionsKind); !errors.Is(err, storecontract.ErrResourceNotFound) {
|
||||
t.Fatalf("resource after the removal = %v, want ErrResourceNotFound", err)
|
||||
}
|
||||
|
||||
if _, _, err := server.ImportConfig(t.Context(), config.Config{}); err != nil {
|
||||
t.Fatalf("ImportConfig after the removal: %v", err)
|
||||
}
|
||||
restored, err := resources.Resource(t.Context(), WorkflowDefinitionsKind)
|
||||
if err != nil {
|
||||
t.Fatalf("the removed %s was not re-seeded: %v", WorkflowDefinitionsKind, err)
|
||||
}
|
||||
if restored.Version != seeded.Version || !bytes.Equal(restored.Value, seeded.Value) {
|
||||
t.Fatalf("re-seeded %s = version %d, want the seed back at version %d", WorkflowDefinitionsKind, restored.Version, seeded.Version)
|
||||
}
|
||||
if _, err := workflow.DecodeDefinitionCollection(restored.Value, testSteps(t).Registry()); err != nil {
|
||||
t.Fatalf("the re-seeded workflow definitions do not decode: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -28,7 +28,10 @@ type executionSettingsDocument struct {
|
||||
}
|
||||
|
||||
// ImportWorkflowExecutionSettings preserves the focused bootstrap API for callers
|
||||
// that do not own the complete legacy configuration document.
|
||||
// that do not own the complete legacy configuration document. It is the same
|
||||
// producer rule as ImportConfig, at this narrower entry point: a kind the store
|
||||
// holds is left alone, and a kind it does not hold is seeded under the one seed
|
||||
// identity (see seedActor and seedRequestID).
|
||||
func (s *Server) ImportWorkflowExecutionSettings(ctx context.Context, settings workflow.ExecutionSettings) (int64, error) {
|
||||
resource, err := s.store.Resource(ctx, WorkflowExecutionSettingsKind)
|
||||
if err == nil {
|
||||
@@ -45,7 +48,7 @@ func (s *Server) ImportWorkflowExecutionSettings(ctx context.Context, settings w
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
resource, err = s.store.PutResource(ctx, storecontract.ResourceWrite{Kind: WorkflowExecutionSettingsKind, Value: value, Actor: "system:migration", Source: "legacy-config", RequestID: "import:" + WorkflowExecutionSettingsKind, ExpectedVersion: 0, At: time.Now().UTC()})
|
||||
resource, err = s.store.PutResource(ctx, storecontract.ResourceWrite{Kind: WorkflowExecutionSettingsKind, Value: value, Actor: seedActor, Source: seedSource, RequestID: seedRequestID(WorkflowExecutionSettingsKind, value), ExpectedVersion: 0, At: time.Now().UTC()})
|
||||
return resource.Version, err
|
||||
}
|
||||
|
||||
|
||||
@@ -108,6 +108,43 @@ func (q *Queries) LockResourceWrite(ctx context.Context, hashtextextended string
|
||||
return err
|
||||
}
|
||||
|
||||
const reinsertResource = `-- name: ReinsertResource :one
|
||||
INSERT INTO resources (kind, value, version, updated_at)
|
||||
VALUES ($1, $2, $3, $4)
|
||||
RETURNING kind, value, version, updated_at, org_id
|
||||
`
|
||||
|
||||
type ReinsertResourceParams struct {
|
||||
Kind string
|
||||
Value []byte
|
||||
Version int64
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
// Re-creates the current row for a revision resource_history already records.
|
||||
// The ledger outlives the row it describes, so a resource an operator removed
|
||||
// still has its revisions; putting one back needs the row re-created without a
|
||||
// second ledger entry, which idx_resource_history_kind_request would refuse.
|
||||
// The revision is the ledger's, not a fresh one: this is the row that write
|
||||
// produced, restored.
|
||||
func (q *Queries) ReinsertResource(ctx context.Context, arg ReinsertResourceParams) (Resource, error) {
|
||||
row := q.db.QueryRow(ctx, reinsertResource,
|
||||
arg.Kind,
|
||||
arg.Value,
|
||||
arg.Version,
|
||||
arg.UpdatedAt,
|
||||
)
|
||||
var i Resource
|
||||
err := row.Scan(
|
||||
&i.Kind,
|
||||
&i.Value,
|
||||
&i.Version,
|
||||
&i.UpdatedAt,
|
||||
&i.OrgID,
|
||||
)
|
||||
return i, err
|
||||
}
|
||||
|
||||
const resourceByKind = `-- name: ResourceByKind :one
|
||||
SELECT kind, value, version, updated_at, org_id FROM resources WHERE kind = $1
|
||||
`
|
||||
|
||||
@@ -13,6 +13,17 @@ SELECT kind, value, version, actor, source, request_id, expected_version,
|
||||
FROM resource_history
|
||||
WHERE kind = $1 AND request_id = $2;
|
||||
|
||||
-- name: ReinsertResource :one
|
||||
-- Re-creates the current row for a revision resource_history already records.
|
||||
-- The ledger outlives the row it describes, so a resource an operator removed
|
||||
-- still has its revisions; putting one back needs the row re-created without a
|
||||
-- second ledger entry, which idx_resource_history_kind_request would refuse.
|
||||
-- The revision is the ledger's, not a fresh one: this is the row that write
|
||||
-- produced, restored.
|
||||
INSERT INTO resources (kind, value, version, updated_at)
|
||||
VALUES ($1, $2, $3, $4)
|
||||
RETURNING kind, value, version, updated_at, org_id;
|
||||
|
||||
-- name: ResourceHistory :many
|
||||
SELECT kind, value, version, actor, source, request_id, expected_version,
|
||||
current_version, at
|
||||
|
||||
@@ -78,6 +78,17 @@ func (s *Resources) Audit(ctx context.Context, table string, keys []string, limi
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
// PutResource applies one control-plane resource write.
|
||||
//
|
||||
// It is idempotent per request ID: a write whose (kind, request_id) is already
|
||||
// in resource_history returns that revision instead of writing a second one, so a
|
||||
// retry that reached the store twice is still one revision. The ledger is what
|
||||
// carries the key -- resource_history outlives the row it describes -- so a
|
||||
// replay of a revision whose resource an operator removed re-creates that row
|
||||
// rather than answering with a revision the store no longer holds. The
|
||||
// alternative is a write that reports a version for a kind that is absent: the
|
||||
// caller cannot tell the difference, and a kind can stay gone while every
|
||||
// process believes it holds a value.
|
||||
func (s *Resources) PutResource(ctx context.Context, write storecontract.ResourceWrite) (_ storecontract.Resource, retErr error) {
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
@@ -91,10 +102,11 @@ func (s *Resources) PutResource(ctx context.Context, write storecontract.Resourc
|
||||
if existing, err := queries.ResourceByRequest(ctx, postgresdb.ResourceByRequestParams{
|
||||
Kind: write.Kind, RequestID: write.RequestID,
|
||||
}); err == nil {
|
||||
return resourceFromAudit(
|
||||
existing.Kind, existing.Value, existing.Version, existing.Actor, existing.Source,
|
||||
existing.RequestID, existing.ExpectedVersion, existing.CurrentVersion, existing.At,
|
||||
), nil
|
||||
resource, err := s.replayedResource(ctx, queries, tx, existing)
|
||||
if err != nil {
|
||||
return storecontract.Resource{}, err
|
||||
}
|
||||
return resource, nil
|
||||
} else if !errors.Is(err, pgx.ErrNoRows) {
|
||||
return storecontract.Resource{}, err
|
||||
}
|
||||
@@ -150,6 +162,32 @@ func (s *Resources) PutResource(ctx context.Context, write storecontract.Resourc
|
||||
), nil
|
||||
}
|
||||
|
||||
// replayedResource answers a write whose request ID resource_history already
|
||||
// holds. The ledger's revision is the answer and a replay writes nothing, with
|
||||
// one repair: when the resource that revision describes is no longer the current
|
||||
// row, the row is re-created from it. That is the state an operator removing a
|
||||
// resource leaves behind, and without the repair the store answers with a
|
||||
// revision it does not hold, so the caller reads a version for a kind that is
|
||||
// absent and the kind stays absent.
|
||||
func (s *Resources) replayedResource(ctx context.Context, queries *postgresdb.Queries, tx pgx.Tx, existing postgresdb.ResourceByRequestRow) (storecontract.Resource, error) {
|
||||
if _, err := queries.ResourceByKind(ctx, existing.Kind); errors.Is(err, pgx.ErrNoRows) {
|
||||
if _, err := queries.ReinsertResource(ctx, postgresdb.ReinsertResourceParams{
|
||||
Kind: existing.Kind, Value: existing.Value, Version: existing.Version, UpdatedAt: existing.At,
|
||||
}); err != nil {
|
||||
return storecontract.Resource{}, err
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return storecontract.Resource{}, err
|
||||
}
|
||||
} else if err != nil {
|
||||
return storecontract.Resource{}, err
|
||||
}
|
||||
return resourceFromAudit(
|
||||
existing.Kind, existing.Value, existing.Version, existing.Actor, existing.Source,
|
||||
existing.RequestID, existing.ExpectedVersion, existing.CurrentVersion, existing.At,
|
||||
), nil
|
||||
}
|
||||
|
||||
func resourceFromCurrent(resource postgresdb.Resource) storecontract.Resource {
|
||||
return storecontract.Resource{
|
||||
Kind: resource.Kind, Value: resource.Value, Version: resource.Version, At: resource.UpdatedAt,
|
||||
|
||||
@@ -267,6 +267,70 @@ func TestResourceWritesPreserveOptimisticAndIdempotentSemantics(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A write is idempotent per request ID, and the ledger holding those IDs
|
||||
// outlives the row it describes. So a replay whose resource is gone must
|
||||
// re-create it: answering with the ledger's revision alone reports a version for
|
||||
// a kind the store does not hold, and a caller cannot tell that apart from a
|
||||
// resource that is there. An operator removing a resource and restarting the
|
||||
// process that seeds it is the shape this repairs.
|
||||
func TestResourceReplayRecreatesARemovedResource(t *testing.T) {
|
||||
pool, _ := migrated(t)
|
||||
resources := NewResources(pool)
|
||||
at := time.Date(2026, 9, 27, 9, 0, 0, 0, time.UTC)
|
||||
write := storecontract.ResourceWrite{
|
||||
Kind: "workflow-definitions", Value: []byte(`{"definitions":[]}`), Actor: "system:migration",
|
||||
Source: "legacy-config", RequestID: "import:workflow-definitions", ExpectedVersion: 0, At: at,
|
||||
}
|
||||
seeded, err := resources.PutResource(t.Context(), write)
|
||||
if err != nil {
|
||||
t.Fatalf("seed PutResource: %v", err)
|
||||
}
|
||||
|
||||
if _, err := pool.Exec(t.Context(), `DELETE FROM resources WHERE kind = $1`, write.Kind); err != nil {
|
||||
t.Fatalf("remove the resource: %v", err)
|
||||
}
|
||||
if _, err := resources.Resource(t.Context(), write.Kind); !errors.Is(err, storecontract.ErrResourceNotFound) {
|
||||
t.Fatalf("resource after the removal = %v, want ErrResourceNotFound", err)
|
||||
}
|
||||
|
||||
replayed, err := resources.PutResource(t.Context(), write)
|
||||
if err != nil {
|
||||
t.Fatalf("replay PutResource: %v", err)
|
||||
}
|
||||
if replayed.Version != seeded.Version || string(replayed.Value) != string(seeded.Value) {
|
||||
t.Fatalf("replay = %+v, want the revision the ledger records, %+v", replayed, seeded)
|
||||
}
|
||||
live, err := resources.Resource(t.Context(), write.Kind)
|
||||
if err != nil {
|
||||
t.Fatalf("the replay did not re-create the resource: %v", err)
|
||||
}
|
||||
if live.Version != seeded.Version || string(live.Value) != string(seeded.Value) {
|
||||
t.Fatalf("live resource after the replay = %+v, want the restored %+v", live, seeded)
|
||||
}
|
||||
// The repair re-creates the row the write produced; it does not append a
|
||||
// second ledger entry, which idx_resource_history_kind_request would refuse.
|
||||
history, err := resources.ResourceHistory(t.Context(), write.Kind, 0)
|
||||
if err != nil {
|
||||
t.Fatalf("ResourceHistory: %v", err)
|
||||
}
|
||||
if len(history) != 1 {
|
||||
t.Fatalf("history after the replay = %d revisions, want the one the write made", len(history))
|
||||
}
|
||||
|
||||
// A resource the store still holds is answered from the ledger, unchanged:
|
||||
// the idempotent replay the index exists for.
|
||||
again, err := resources.PutResource(t.Context(), storecontract.ResourceWrite{
|
||||
Kind: write.Kind, Value: []byte(`{"definitions":[{"id":"different"}]}`), Actor: "other",
|
||||
Source: "retry", RequestID: write.RequestID, ExpectedVersion: 0, At: at.Add(time.Minute),
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("second replay: %v", err)
|
||||
}
|
||||
if again.Version != seeded.Version || string(again.Value) != string(seeded.Value) {
|
||||
t.Fatalf("second replay = %+v, want the original %+v", again, seeded)
|
||||
}
|
||||
}
|
||||
|
||||
// A request ID is idempotent per kind: the store API, the resource_history
|
||||
// UNIQUE(kind, request_id) index and every production caller treat it as one,
|
||||
// so a caller that mints one request ID for two different kinds must get two
|
||||
|
||||
Reference in new issue
Block a user