a7d6f71d5a2c9da170a46bfc4471f9979189de87
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
7a8bb8e47b
|
docs: drop one host's service names from port guidance
Five files named a specific host's service and the port it holds as the reason a deployment might move the state store's listen address, including config.example.toml, which ships to every operator. The guidance keeps its point -- move the listen/target pair together when a port is already taken on the host -- without naming a service the reader may not run, or have heard of. Also corrects the Taskfile's dev-stack summary, which still described the state store as binding 127.0.0.1:9090. deployments/dev.toml pairs it with 9091, and has since the pair became configurable, so the comment described a stack that does not start. |
||
|
|
0263bf4e0d
|
fix(configuration): resolve the default config path in one place
Four binaries derived the default with os.UserConfigDir while the daemon resolved it from its own config home. The two agree on Linux with an absolute XDG_CONFIG_HOME and diverge everywhere else, so one deployment booted from a different config file depending on which binary started. Move the rule into the package that owns configuration resolution (internal/infrastructure/configuration, which already owns xdgDataHome and DefaultWorkDir) as DefaultConfigDir/DefaultConfigPath, and have every -config default take it from there: archie-gateway, archie-messaging, archie-state-store (serve and recovery) and archie-ui, plus the daemon's own parseArgs, setup generator and drain-marker path. Fence the claim at the source, since the resolved value is only observable by starting a process: no source under cmd/ or internal/app/archied may call os.UserConfigDir, and every -config registration must default to the helper. |
||
|
|
9a45a35c27
|
fix(config): judge a database-owned setting at the layer that owns it
Review of the two previous commits found the split was incomplete in both directions: the checks moved off the file load path were still enforced anywhere the value could enter the system, and the fold reached only one of the three overlay paths. A stale database-owned TOML value could still stop archie-state-store. The process seeds every control-plane resource from the same file config on a fresh database, and the seed runs the resource's own validator; three of the five values the loader now accepts aborted the boot there (verified by booting RunStateStore before the fix). A seed that does not validate is now SKIPPED and reported instead of fatal: the resource stays absent, so the file's value is the one in effect and editing config.toml is still the fix -- the daemon and the Gateway are where it fails closed, at boot.runtimeConfig. The resource validators were weaker than configuration.Validate for exactly the fields the split reclassified, so a value the file path used to refuse at load became storable in the database, and then un-fixable by editing config.toml. The trigger enum and the test-glob rule now come from the configuration package rather than being restated, the container image is required there as it is in the file, and a table test ties the two layers together value by value. The directory form of -config-overlay kept the archie-core-e2e2 failure: it decoded each overlay file straight into the populated config, so a map-valued entry it only partly addressed still cleared the fields it did not name. Both the overlay layer's main file and its feature files now fold, which is what Loader.Dir's documented parity with Loader.Overlay has always claimed. The overlay file's key report is derived from the apply's own view of the file as well as the file decode's, because TOML matches keys case-insensitively and the fold's yaml decode does not: a key only the file decode matched was consumed by the report and dropped by the apply, a silent no-op that worked in the base file. Tests: the state-store boot is asserted against the real process, the directory overlay and the case-mismatched key each have one, and ImportConfig pins that a refused seed is not stored. |
||
|
|
be321d81bc
|
fix(statestore): build the recovery control-plane server on the shared step vocabulary
After the rebase onto main, controlplane.NewServer takes the step vocabulary it resolves definitions against (controlplane.ResourceStore, *workflow.Manager) and returns (Server, error). Both offline call sites still passed the store alone, so internal/app/archied stopped compiling -- and with it cmd/archie-state-store, cmd/archied and cmd/archie-gateway. validate and rollback now build their server through openStateStoreControlPlane, the root helper the served path already uses, so the offline checks resolve the same shared provider set (internal/infrastructure/workflowsteps) as the process whose verdict they predict, and the constructor's error is returned as a refusal rather than dropped. No second vocabulary call site: the fwmp pin on openStateStoreControlPlane covers these paths unchanged. The test's seeding helper builds its server from the same shared set (workflowsteps.NewManager) because it cannot call the unexported root helper; an empty manager there would seed values the served path never validates against. That is not merely a claim: with an empty manager the three validate tests fail with `workflow "bootstrap" step 1: unknown type "bootstrap.prepare"`, so they already pin that the offline path resolves a live, shared vocabulary. |
||
|
|
08df95684c
|
fix(statestore): read a kind the store does not hold as its seed, and stop validate touching the daemon's log
Four defects from the re-run contract review, all in the recovery path. A store missing a control-plane kind was refused while a store with no resources table at all was accepted -- two stances inside validateStore for states the daemon cannot tell apart. The State Store seeds every kind from its own config before it serves (RunStateStore -> ImportConfig), so a kind the file does not hold yet is one archied reads the seed for: an upgrade that adds a kind, and a validate before the State Store has started again, exited 1 with "read provider-settings: resource not found" for a store the daemon boots on fine. StoredRuntimeConfig now reads a missing kind as the seed the State Store would write for it (Definition.seededValue, shared with ImportConfig so the two cannot drift), and the missing-table branch is the same stance taken to its extreme. defaultConfigPath() derived the daemon's config with os.UserConfigDir, which disagrees with archied's configHome on darwin and rejects a relative XDG_CONFIG_HOME outright -- leaving validate with an empty path and no verdict, on a host where archied boots from relative-dir/archie/config.toml. It is now the daemon's own rule, exported as archied.DefaultConfigPath and used by the daemon's own -config default too, with a test pinning the equality. -h exited 2, the code documented as "the command line was wrong", while the serve path in the same binary exits 0 and main.go advertises -h for these subcommands. ErrHelp now returns 0. validate resolved the boot config through loadConfig -> setupLogging, which created cfg.Log.File's directory, opened the log, and wrote a line stamped component="daemon" -- a diagnostic editing the deployment it inspects, and rotating a log the daemon is writing. boot.stderrLog keeps setupLogging on the stderr logger for the recovery path: no feed, no task-log registry, no durable copy. Tests: TestRecoveryValidateRunsTheGateBootRuns now asserts the unseeded and the missing-kind stores are accepted (both exited 1 before), while the bogus-policy case still refuses; TestRecoveryHelpExitsZero, TestRecoveryValidateLeavesTheDaemonLogAlone and TestRecoveryDefaultConfigFollowsTheDaemonsConfigHome are new. Each was run red first, and the seed fallback and stderrLog were mutation-checked. |
||
|
|
e7e5ddfba5
|
fix(statestore): make validate run boot's gate, and stop backup replacing the database
Two P1 findings from review, both about a claim outrunning the code.
validate answered a different question from boot's. Boot refuses on
configuration.Validate over the file config with every stored resource
layered onto it (internal/app/archied/control_plane.go:27), which is a
stricter check than the write path's per-resource Decode: validateScheduling
never inspects dispatch.trigger and explicitly permits "0s", while boot
rejects an unknown trigger and a non-positive poll interval. A store holding
such a value printed "is a valid store ... N stored resources validate" and
exited 0 while archied exited 1 on it -- a false green pushing an operator
toward the blunt restore instead of the surgical rollback.
validate now runs both layers and reports both, in boot's order:
store.ValidateFile, then the write path's ValidateStored, then the config
gate. The layering itself is now one implementation: controlplane's
runtimeConfigFrom takes a resourceReader, which the gRPC client and a new
Server.StoredRuntimeConfig (reading its own store) both provide, so the
offline check cannot drift from the daemon's. validate takes -config /
-config-overlay (default: the path the serve subcommand uses), because the
question is about the stored values layered onto that document.
backup had no guard against -out naming the database. It renames its
snapshot over the target, so `backup -db F -out F` replaced a database a
serving State Store may still have open and orphaned the WAL of the file it
just unlinked -- the state Restore refuses up front and maintenance.go's own
comment calls unrecoverable. backup now refuses with the same message
Restore uses, which is what makes its exemption from the ownership lock
sound.
Claims corrected where they ran ahead of the code: ValidateStored now says it
is the write path's check and not boot's, the Restore comment no longer
promises "never a mixture" for the sidecars it removes first, the usage text
advertises validate's -config, and the architecture table drops "must be
stopped" for validate (nothing enforces it; read-only opens skip the lock by
design) and states which two layers validate reports.
Tests: TestRecoveryValidateRunsTheGateBootRuns fails on the pre-fix code
("validate accepted a stored policy the daemon refuses to boot with") and
covers a store the State Store never seeded; TestRecoveryBackupRefuses-
TheDatabaseAsItsOwnSnapshot fails on the pre-fix code ("backup over the
database itself succeeded"). The validate-accept test now seeds the store
through ImportConfig and asserts the reported count is non-zero, and the
stored-resource case uses a value both gates refuse instead of one only the
write path does.
|
||
|
|
b8b477fb32
|
feat(statestore): offline backup, restore, validate and rollback
archie-state-store gains the offline recovery commands
docs/prds/runtime-control-plane.md ("Bootstrap, migration, and recovery")
asks for. They work on the task database file directly -- <db_path>-tasks.sqlite,
not db_path -- and need neither archied nor the Web UI, which matters because
the failure they exist for is the one that stops archied starting:
backup -db F -out F consistent snapshot (VACUUM INTO), verified as a
usable store before it is published; safe against
a serving store
restore -db F -from F replace the database with a snapshot, dropping the
replaced file's -wal/-shm
validate -db F readable, not corrupt, schema no newer than this
binary, every stored resource decoding under the
definition that owns it
rollback -db F -kind K replay an earlier revision of a stored resource
[-revision N] through the ordinary replace, recorded as a new
revision
No rollback RPC is added and none is needed: a revision carries its own value,
so restoring one is the same replace the dashboard and the channels already
use, by hand, with the store stopped.
"The State Store is stopped" is checked rather than assumed. The serving
process now takes an exclusive flock on <db>.lock for its whole life
(store.AcquireOwnership), and restore/rollback refuse with
`task database is owned by another process` while it holds it. backup is the
deliberate exception: the update installer runs inside the process it is
replacing and cannot stop the State Store.
scripts/archie-update-install takes its pre-flight snapshot through
`go run ./cmd/archie-state-store backup` from the release being installed
instead of its own sqlite3 shell, so the installer and an operator run one
implementation. The watchdog's refusal semantics are unchanged: an update that
cannot back up still must not start.
Docs: safe-change-and-recovery.md's "Recovery: not yet built" now describes
what exists.
|
||
|
|
e0beb68978
|
feat(config): let the State Store and Gateway listen addresses be configured
Both gRPC listeners took their bind address from a -listen flag default alone, with no config key. The port a service binds and the port its clients dial were therefore two separate facts -- the flag default, and [services.<name>].target -- with nothing keeping them in step. On a host that already owned the default the effect read as a code fault rather than a port collision: the store died with "bind: address already in use" while every client reported "state store: connection error ... error reading server preface: EOF". Cockpit binds 9090 by default, so this is not hypothetical. Verifying the dashboard on such a host needed a hand-written overlay plus a matching -listen, two things to keep in step by hand. - [services.state].listen and [services.gateway].listen now name the bind address. Precedence is -listen, then config, then a default applied by the loader. An empty result is a startup ERROR, never a fallback: net.Listen reads "" as "any free port", so defaulting there would bind the service somewhere no client looks and still report success. - Both flag defaults become "" so configuration can win, mirroring how archie-ui already prefers [web].listen over its flag. - Gateway listen defaults to the same constant as the gateway target, so a default deployment's client and server cannot drift onto different ports. [services.state].target still has no default: an empty target stays a startup error in the daemon and gateway, which no longer serve a local store. - ServiceConnection gains the field. Its doc now says clients read only Target and TargetToken, and Listen belongs to the process that owns the service -- the previous wording claimed the struct starts no listeners while carrying one. - deployments/dev.toml states each port once, beside the target it must match, so `task dev` is retargetable on a host that owns 9090 by editing one overlay. Tests. Unit: config defaults, explicit values, and precedence. Process: the existing real-process smoke test now takes its listen from configuration instead of -listen, so the end-to-end wiring is proven by the test that already starts the binary, with no extra process. Both were verified load-bearing by mutation -- inverting the precedence fails the unit test, and making the app layer ignore config fails the process test with "state: no listen address". Verified: task check exit 0. Also documented in config.example.toml, the configuration architecture table, and the `task dev` summary. |
||
|
|
2ef962912a
|
feat(tasks): attribute events, changes and config to the attempt that produced them
The events table had no attempt column, so two attempts of the same task were indistinguishable: a retried task's stage events interleaved into one timeline and nothing could say which run produced a given result. - events gains an `attempt` column through the existing presence-gated migrateTasks loop rather than a version bump, so user_version stays 3 and an existing archie.db is migrated in place. A legacy-database test builds the pre-change schema and proves both that existing rows survive unattributed (attempt = 0) and that a new value round-trips. - Event.attempt = 11 and ReadTaskLogRequest.stage = 9 are field additions to existing messages: no new RPC, no new table, and no change to any canonical error message that crosses the state-store wire. - Every task event the workflow engine emits now carries the attempt. Producers that have no attempt yet (task_queued, binding_dispatch_failure) stay unattributed rather than guessing one. - Changed files are captured as a durable changes_captured event at commit and push time, reusing the existing three-dot diff primitive: paths from FilePatches(), counts from Stats(), base_sha recorded as the merge base actually diffed rather than the base tip, a 200-file cap with totals kept complete. Consumption goes through an unexported optional interface so workflow.Trees -- the yaegi capability surface for interpreted stage plugins -- is not widened. A capture is also taken once the pull request exists, so the capture can link it. - The per-attempt configuration is captured as a durable config_captured event carrying the non-secret config.TaskConfig, instead of a new table plus two new state-store RPCs: events are already this system's per-attempt provenance stream, so a table would have given one feature two provenance mechanisms. Redaction is proven against canary secrets. Measured payload: 539 bytes typical, 2333 bytes at 20 models. - Captures carry a schema marker and are refused unless their key set matches the declared payload shape, so drift surfaces as "recorded but could not be read" rather than as a capture of zeroes. - Three read endpoints serve the attempt rail, the changed-file captures and a raw debug view, mapping to UI-owned response types rather than dumping store types. - The task-log read gains a stage filter, bound to the stage logger for that stage's own lines; agent and tool output records no stage. A task-scoped grant may already insert events on its own task, so this provenance is exactly as trustworthy as the rest of that task's event stream. That limit is documented rather than papered over. |
||
|
|
af2c8e1f41 |
fix(staterpc): attach the bearer token to State Store streams
The capture batch reads moved onto server-streaming RPCs (StreamCaptures / StreamUndispatchedCaptures) to get past gRPC's 4 MiB unary cap, but the client interceptor suite attached the bearer token to unary calls only: Dial installed WithUnaryInterceptor and nothing for streams, while TaskGrants.StreamInterceptor requires the administrative token. On every token-protected (non-loopback) listener -- every container-mode deployment -- the daemon's binding dispatch and the dashboard's capture inspector got PermissionDenied from the stream interceptor, and the unary fallback those reads replaced is no longer taken, so there was no working path. - Add StreamClientTokenInterceptor, mirroring gatewayrpc's, and install it alongside the unary one in Dial, the single client dial path the daemon, the dashboard and the agent all use. - Route agenttransport's State Store dial through staterpc.Dial, deleting the duplicate dial/fail-closed logic that installed only the unary interceptor -- a second place for this class of bug to hide. - Regression tests: the grants harness now dials through the production Dial so client-side wiring cannot drift again; TestCaptureStreamsCarryTheAdminToken reads both list surfaces (5 MiB batch) through a token-protected listener; TestStateStoreRealProcessTokenProtectedCaptureStream does the same against the real binary; TestUpdateRejectsNilTask covers the Update guard left untested by the earlier finding fix. |
||
|
|
33751abe4e |
test(staterpc): verify State Store end-to-end and preserve context error identity (#775)
* test(state-store): real-process smoke + deadline/sentinel error contract Add end-to-end verification for archie-core-8cda.4.7, the .4.7 State Store verification stage against docs/prds/state-store-contract.md (rev. 2c): - Real-process smoke coverage: build ./cmd/archie-state-store, run it against a minimal config on loopback, poll the readiness surface, and drive the State Store gRPC contract through staterpc.NewClient exactly as a consumer does. Covers the combined task/event/capture view, error-sentinel fidelity across the wire, the found=false-not-error convention, single-owner SQLite (the binary owns the one archie.db file; the consumer never opens a store file), per-call deadline behaviour, restart/recovery (SIGTERM + reopen on the same db), and the remote capture/mapping/binding surfaces. - Fix a real deadline-identity gap in unmapError: a cancelled or deadline-exceeded context surfaced by gRPC-Go as a *status.Error was re-wrapped into a sanitised internal-style error, breaking errors.Is(err, context.Canceled/DeadlineExceeded). The agent's deadlineStore bounds each Store call with context.WithTimeout and the workflow consumer (workflow.go) uses errors.Is(err, context.DeadlineExceeded) to mark a stage interrupted rather than failed, so the re-wrap would wrongly park an interrupted task. Unmap Canceled/DeadlineExceeded codes back to the standard context sentinels. - Pin the error-mapping contract in staterpc: context-identity preservation, sentinel round-trip fidelity for every store sentinel, and the sanitisation rule (an Internal error must not leak raw detail or map to a sentinel). * test(state-store): assert single-owner SQLite composition (daemon/gateway require target) The single-owner invariant (docs/prds/state-store-contract.md §12 step 7/8) is that only archie-state-store opens archie.db; the daemon and gateway resolve their task-store surface from openStateStoreAdapter, which requires [services.state].target and opens no local store. This test pins that composition root so a future change cannot silently reintroduce a local *store.Store in the daemon or gateway path (dual-store ownership). * test(state-store): skip real-process smoke tests under -short The smoke tests build and exec the real archie-state-store binary, which is slow enough that a -short run (e.g. a focused edit loop) should skip them. task check (go test ./... -count=1) does not use -short, so they still run in the full gate. * test(state-store): fix subprocess log race under -race The smoke test captured the subprocess's stdout/stderr in a strings.Builder that the os/exec writer goroutine and the test reader goroutine touched concurrently, racing under -race. Replace it with a mutex-guarded buffer, the standard pattern for reading a running subprocess's output. * test(state-store): satisfy lint (noctx, errorlint) on new tests - exec.Command -> exec.CommandContext, http.Get -> NewRequestWithContext + Client.Do, so the subprocess and readiness polling honour the test context (noctx). - errorlint: compare the raw non-status error via errors.Is rather than != on a wrapped error. * docs(deployments): document State Store readiness verification and single-owner invariant The deployment runbook described how to launch archie-state-store but not how to verify it is healthy before starting archied. Document the -ready-addr readiness surface (/healthz, /health/detailed state_db component) and the single-owner invariant: one task SQLite file owned by archie-state-store, with archied/archie-gateway never opening archie.db directly (state-store-contract §12 step 7/8). * fix(state-store): preserve context deadline/cancel identity in mapError The symmetric server-side gap to the unmapError fix: if a store query fails because the caller's context expires mid-call (§6 StoreTimeout), the handler's mapError folded context.Canceled/DeadlineExceeded into codes.Internal. The client would receive a sanitised internal error with no deadline identity, so the agent's workflow consumer (errors.Is(err, context.DeadlineExceeded)) could not distinguish an interrupted stage from a failed one. Map a context cancel/deadline error to its matching gRPC code so unmapError rehydrates it back to the standard context sentinel, completing the §6/§7 round-trip. Pin both directions with TestMapErrorPreservesContextIdentity. |
||
|
|
e6a2e5b3b4 | feat(state-store): stand up archie-state-store binary and own archie.db (archie-core-8cda.4.3) (#771) |