archie-state-store gains the offline recovery commands
docs/prds/runtime-control-plane.md ("Bootstrap, migration, and recovery")
asks for. They work on the task database file directly -- <db_path>-tasks.sqlite,
not db_path -- and need neither archied nor the Web UI, which matters because
the failure they exist for is the one that stops archied starting:
backup -db F -out F consistent snapshot (VACUUM INTO), verified as a
usable store before it is published; safe against
a serving store
restore -db F -from F replace the database with a snapshot, dropping the
replaced file's -wal/-shm
validate -db F readable, not corrupt, schema no newer than this
binary, every stored resource decoding under the
definition that owns it
rollback -db F -kind K replay an earlier revision of a stored resource
[-revision N] through the ordinary replace, recorded as a new
revision
No rollback RPC is added and none is needed: a revision carries its own value,
so restoring one is the same replace the dashboard and the channels already
use, by hand, with the store stopped.
"The State Store is stopped" is checked rather than assumed. The serving
process now takes an exclusive flock on <db>.lock for its whole life
(store.AcquireOwnership), and restore/rollback refuse with
`task database is owned by another process` while it holds it. backup is the
deliberate exception: the update installer runs inside the process it is
replacing and cannot stop the State Store.
scripts/archie-update-install takes its pre-flight snapshot through
`go run ./cmd/archie-state-store backup` from the release being installed
instead of its own sqlite3 shell, so the installer and an operator run one
implementation. The watchdog's refusal semantics are unchanged: an update that
cannot back up still must not start.
Docs: safe-change-and-recovery.md's "Recovery: not yet built" now describes
what exists.
printf '%s\\n' emitted the literal characters backslash-n, which
archie-update-check's release-version check rejects, so every managed-agent
update left the installed agent version reported as unknown.
Service.Check matched deferred[component.ID] == component.Available, and a
never-deferred recipient's absent entry is the empty string -- equal to an
empty Available emitted for a component with no update. Guard on a real
(non-empty) available version before applying the deferral, so an
up-to-date recipient is not told their update is deferred. No WIRE/DELETE
decision applies: this is a correctness fix on a live consumer, not
unreachable code.
Components are independently versioned (RELEASING.md), so the newest
archied/v* and archie/v* tags usually sit on different commits --
gateway-only releases are the norm. The installer required both tags to
resolve to the same commit whenever both components changed, which
refused every update once the two newest releases diverged
(archied/v1.24.0 at 037326a, archie/v1.23.0 at 43bc29f).
Build each changed component from its own tag's commit instead: daemon
binaries and the adapter refresh from the daemon release, the managed
worker image from the agent release.
Found by a full-repo 'go test -race ./...' sweep. internal/releaseupdate was
the only package that failed it.
The race: blockingInstallerStub.Install assigned s.gotCancelled lazily --
if s.gotCancelled == nil { s.gotCancelled = make(chan struct{}) }
Install runs on the caller's goroutine (Service.Install calls it), while the
test goroutine reads s.gotCancelled. Two goroutines, one unsynchronised field.
The channel is now built by a newBlockingInstallerStub constructor, so no
goroutine ever writes it after the stub is handed to the test. Fixing it at
the producer rather than guarding the read.
The sleep: the test asserted 'the installer did not observe cancellation' with
select { case <-s.gotCancelled: ... case <-time.After(50*time.Millisecond): }
which is a race against the clock to prove a negative, and is weaker than the
thing it stands in for -- a channel nothing has closed is simply not ready.
It is now a select/default, asserted synchronously.
The assertion was also unfalsifiable in one direction: if the channel were
never closed at all, the old test passed for the wrong reason. It now also
asserts the positive half -- after Install returns, the detached context is
released, so the observer MUST have fired -- which is what makes the negative
check meaningful.
A second constructor of the same stub is updated too; it relied on the lazy
assignment and would have closed a nil channel.
Reproduced before the fix (race detector, same test):
WARNING: DATA RACE ... service_test.go:201
--- FAIL: TestServiceInstallSurvivesCallerContextCancellation
After: 3/3 clean runs under -race.
configured_task_db_path() read the top-level db_path and handed that literal
file to the watchdog as ARCHIE_TASK_DB_PATH (scripts/archie-update-install).
The daemon's task store is not that file: taskDBPath() in
internal/app/archied/main.go derives `<db_path>-tasks.sqlite`, and the
configured path is commonly a zero-byte placeholder beside it. The updater
therefore backed up an empty file, and a rollback would have restored that
empty file over the real database and deleted its -wal/-shm -- destroying the
task store the backup exists to protect.
That awk also matched db_path in any section. [indexing] owns its own
database, so a config listing it before the top-level key handed the watchdog
an unrelated database as ARCHIE_TASK_DB_PATH and backed that up instead.
Resolve the top-level key only, append the sibling suffix the daemon uses, and
refuse the update when the backup cannot be taken. The backup is the only
route back from a forward schema migration, so an update that cannot take one
must not start; it now runs as pre-flight, ahead of the adapter refresh, so a
refusal changes nothing on the host.
The tests drive the real script with a config file the harness controls; all
three fail against the previous adapter.
The check adapter resolved available releases with
`git -C "$REPO" ls-remote`, where REPO defaulted to $HOME/projects/archie-core.
With that checkout absent, moved or renamed, the swallowed error left Available
empty for every component -- which FormatSnapshot renders as "Archie is up to
date." A failed lookup therefore read as parity: the exact inverse of what an
operator needs to know.
Resolve releases from the forge API and `git ls-remote` against the remote
directly, with no working-tree state and no dependence on a clone existing. When
neither source answers, exit non-zero so the /update command reports that it
could not check instead of fabricating an up-to-date result.
A tagged release is announced whether or not it has a published artifact, and
Reference now records which source answered and the artifact state, so an
operator is not sent to a tag with nothing to download without being told.
This is what hid archied/v1.24.0 from the live host: tagged, never published.
The failure tests run the real script through a fake git and curl, so they
assert the source-selection logic rather than a stub. Against the previous
script all four fail, including the checkout dependency itself.
A self-update to 1.23.0 failed its health check and rolled back. The
adapter built and installed ./cmd/archied and nothing else, and the
watchdog restarted archied.service and nothing else, so the new daemon
came up against a State Store, Gateway and UI still on the previous
release. Since 1.23.0 archied cannot boot without a reachable State
Store (openStateStoreAdapter), which on a host that had never run one is
an immediate exit, a failed health check, and a rollback that looks like
a bad release rather than an incomplete deployment.
Install every binary the reference deployment runs, versioned off the
archied tag, and back each up so the rollback restores all of them
together. A binary with no .prev was introduced by the failed release, so
rollback removes it instead of leaving one behind that points at
processes the restored release does not run.
Restart in dependency order: the State Store owns archie.db and every
other process dials it, the Gateway serves the contract archied and the
UI dial, so each comes up before its consumers and stops after them.
archied itself is cycled unconditionally and a failed stop still aborts
the rollback, because a presence probe that guesses wrong must never
skip the stop that keeps a live process off the binary and the database
being restored. The siblings are best-effort, which is safe because the
adapter now refuses the update outright, before touching the host, when
a unit for one of them is missing, naming what to create.
install.sh had the same single-binary assumption, so a fresh install
produced a host that could not start either.
Closes archie-core-hbqk.
The dashboard's Run read the watchdog's phase-2 report off local disk and
put it on the activity stream. That is a host-local file read in the process
that is moving off the host: the watchdog restarts archied and leaves its
verdict beside archied, so archied is what can read it.
The daemon now polls and relays it, publishing the same event through the
same bus, so it persists with every other event and reaches whichever
dashboard is watching. The event builder moves to releaseupdate, which owns
Report and its verification; the dashboard keeps only what it needs to tell
an installer where to write.
The startup poll stays: the watchdog writes its result only once the process
answers /healthz, so a single read at boot can always miss the report
describing that very boot.
persistAndBroadcastEvents takes the daemon context and detaches it for the
insert, which is what it always did with context.Background() -- the last
events of a run are the ones an operator reads afterwards.
The update watchdog verifies archied's restart, so archied has to be what
answers. It was probing the dashboard's /healthz, which belongs to the
dashboard: switch [web] off and nothing answers, and in the extracted UI
topology that endpoint reports a different process entirely, which would
confirm a release while the daemon it was meant to verify is down.
[health].listen (127.0.0.1:8485, defaulted, always on) serves /healthz,
/health and /health/detailed for the life of the process. Liveness answers
503 until the daemon reaches its run loop: the listener binds before the
subsystems do, so an unconditional 200 would tell the watchdog a half-booted
release came up. Failing to bind is fatal, because a daemon nothing can probe
gets every update rolled back. The State Store's readiness surface now shares
the listener plumbing, keeping its log and error text verbatim.
Closes archie-core-exbz, which asked which of two bad options to take when
the dashboard is off. Neither: the daemon owns its own liveness.
Also deletes TestInstallUpdateDetachesInstallerFromCallbackContext. It has
failed intermittently under load more than once, racing its fake bot server's
multipart parse against the cancellation it induces.
The watchdog polled http://127.0.0.1:8420/healthz. Nothing binds 8420: the
daemon serves /healthz on [web].listen (127.0.0.1:8484 by default), so every
dashboard- or Telegram-initiated update waited out the 60s health timeout and
rolled back a release that had come up fine.
The daemon is the only process that knows which address it binds, so it now
derives the probe URL from its own config and hands it to the installer;
CommandInstaller exports ARCHIE_HEALTH_URL and the install script forwards it
into the watchdog unit, whose remaining default matches the dashboard's.
The suite never dialled the URL: every watchdog test faked curl as an
immediate failure, so only the unhealthy branch existed. One test now runs
the real curl against a real listener, and another pins the script default.
Baseline `task check` failed on a clean main checkout, blocking unrelated
in-flight work. Fixes only those pre-existing failures:
- bootstrap.go: extract wireWebStoreSurfaces from setupObservability
(cyclop 16 > 15) and startEmbeddedNATS from connectNATS (nestif).
- telegram_setup_test.go: os.Setenv -> t.Setenv (usetesting).
- install_script_test.go: exec.Command -> exec.CommandContext with
t.Context() (noctx).
- Taskfile: add a ui:deps install step shared by `ui` and `test:ui`.
ui/node_modules is gitignored and nothing in the gate installed it, so
`task test:ui` died on a missing jsdom rather than a real test failure.
- shutdown_watchdog_test.go: `go fix` output adopted verbatim.
Refs bead archie-core-14cs.
-X main.gatewayVersion/main.runtimeVersion targeted package main, but
those vars live in internal/app/archied (cmd/archied is a thin
wrapper). An -X target that doesn't exist is silently ignored by the
linker, so every archied binary this shipped -- task build, install.sh,
Dockerfile.archied, and the carina self-update script -- reported
"dev" instead of its real version. Point every one of those ldflags at
the actual package.
Also, while chasing the carina incident this surfaced:
- restart.go: /restart didn't check for an in-flight /update install.
The install runs as archied's own child process, so a restart killed
it mid-build (observed: "signal: terminated") instead of letting it
finish or fail cleanly. Refuse the restart while updateInProgress.
- update.go / update_report.go: the "no version could be confirmed
running" branch still opened with "Update complete", contradicting
its own next clause. Reworded so an unverified outcome isn't
reported as completion.
Dashboard-initiated updates also retain a configured Telegram restart-report route, while falling back to the web UI report when Telegram is unavailable.
Autonomous workflows now always hand a complete task to a scoped archie-agent container. Embedded NATS is the default and external NATS changes only broker placement; legacy agent and containers.enabled inputs decode for migration but cannot select an execution path. The daemon flushes task-event subscriptions before a worker can publish, and runtime updates snapshot and restore the configured worker image.
Build selected components from a temporary worktree at the exact approved tag commit instead of the moving branch tip. Validate release version inputs and require combined component tags to resolve to the same source commit.\n\nRefs #606
Carry the revalidated update snapshot through the installer contract so the shell adapter receives an explicit per-component plan. Build, back up, install, and roll back only selected components, and pass the runtime release version into agent image builds.\n\nCovers gateway-only, runtime-only, combined, and no-op updates.\n\nRefs #606\nBead: archie-core-462t
Component.Installed/Available come from an operator-owned external
check command, which reports what version exists but has no way to
know how THIS process/deployment actually installed it -- native
binary, container, or embedded process. Two sources for that already
exist and were previously unused for this: internal/installtype
(this process's own build-time stamp) and daemon.AgentStatus (an
archie-agent version observed at runtime, landed in the prior commit).
Add Component.InstallType/Reference and a Service.Enrich hook that
fills them in per component ID after the check command runs, without
overwriting whatever a check command already reported for an ID it
doesn't recognise. Wire the actual sources in cmd/archied via
componentInstallTypeEnricher: the daemon's own installtype.Type(),
the agent's most recently observed install type, and -- for NATS, a
third-party dependency this repo doesn't build -- "embedded" or
"external" (with its URL) read straight from [nats] configuration,
which is already the deployment profile's own record of how NATS is
run. This is wired into the single Service construction shared by
Telegram and the dashboard, so both surfaces get it for free.
Part 2/3 of archie-core-1786751948782-3-837a8fd0 point 4. Closes
archie-core-1786751948782-3-837a8fd0.2.
A pending update report used to be relayed verbatim. It is entirely
self-reported: scripts/archie-update-watchdog decides "passed" from
`curl -sf $HEALTH_URL/healthz`, which only proves that *a* daemon is
answering, then copies the version the installer *intended*
(ARCHIE_UPDATE_INSTALLED_GATEWAY) into the report. If the new binary
never made it into service, the previous daemon answers the probe just
as happily, so the message read "Update complete: daemon 1.9.8 -> 1.9.0.
Health check passed - archie-core is healthy on the new version" about a
version that was not running. That shipped: /update reported success and
the next task hit the exact bug the update was meant to fix (#468).
Report.Verify now checks the claims against the versions components
report for *themselves* -- archied's compiled-in gatewayVersion, which no
installer can talk it out of -- and partitions every claimed component
into Confirmed, Drift or Unverified.
What changed in MEANING:
- A passed health check used to mean "the new version is running". It
now means only "something answered"; the version claim is a separate,
separately-checked assertion.
- "Update complete ... healthy on the new version" used to be the only
success wording. There are now three outcomes, because collapsing them
is how the false success happened: confirmed, unconfirmed ("I could
not confirm which version is actually running"), and contradicted
("Update did NOT take effect").
- An unverifiable component is NOT a failing one. Absent, empty,
"unknown" (the watchdog's own default) and "dev" (an unstamped build)
all mean "we could not check", never "it is wrong" -- treating them as
drift would turn healthy updates whose bookkeeping went missing into
reported failures.
Verification covers the daemon only, deliberately. archied's compiled-in
runtimeVersion records what its own pipeline stamped, not what the agent
container runs, so vouching for the agent would be the same kind of
guess this change exists to remove. The agent is therefore Unverified on
every successful update, and the success message names it as unchecked
rather than letting the confirmed daemon silently vouch for it.
Drift detail is attached per component: one component can sit on the
version the update meant to replace while another is on some third
version, and a single explanatory sentence would state a falsehood about
one of them.
Regression tests were confirmed to fail against the pre-fix code,
reproducing the exact false-success string, and each new guard was
mutation-tested. The drift-ordering test's component count is
load-bearing and documented: Go rotates small maps rather than permuting
them, so 8 components let an unsorted implementation pass ~12% of the
time while 9 fails it outright.
Refs #522, #468
The /update command went silent between "starting" and one final message,
gave no version summary, and required manual intervention on failure.
CommandInstaller now streams the installer script's stdout live instead of
buffering it, and parses a structured result (previous/installed versions,
per-component status) from a sentinel line. Because the reference install
script hands the daemon restart off to a detached systemd unit specifically
so restarting archied.service doesn't kill the process that requested the
update, the outcome is reported in two phases: the synchronous build/install
result comes back through Install() itself, and the post-restart health
check/rollback outcome is written by a new watchdog script and relayed by
whichever archied process boots next, via a small JSON handoff file cleared
once relayed.
Also fixes three defects an adversarial review pass found in the first
draft: the caller's request/handler context was passed straight into the
install child process, so a concurrent /restart or a browser disconnect
could SIGKILL it mid binary-copy; Telegram and the web dashboard each
tracked "install in progress" with their own private flag despite sharing
one Service, allowing two installs to race the same .prev backups; and the
watchdog reported rolled_back:true even when no backup existed to restore,
which would have told an operator to stand down while the daemon was still
broken.
internal/webui gains an unauthenticated /healthz the watchdog polls after
restarting archied.
/update could report success at the correct target version while the
archie-agent container running it never actually changed: the update
mechanism trusted the configured installer's exit code and reported
whatever version number the release claimed, with nothing verifying
the two agreed.
The first piece of the fix: a new internal/installtype package
reports how a binary was built for distribution (Type(), defaulting
to "unknown"), stamped once per release artifact via
"-ldflags -X .../installtype.buildType=<value>" rather than guessed
at runtime by probing the environment -- restic and Tailscale both
treat "which install shape is this" as a build-time decision for the
same reason: a probed signal is guessable, a compiled-in one is not.
releaseupdate.Service.Install now refuses to run the configured
Installer at all when InstallType is unset or installtype.Unknown, so
an unstamped or misconfigured build can no longer silently attempt an
update it has no way to know is even the right kind for how this
instance was deployed. Wired through every real archied build path so
the gate doesn't just exist in principle: Taskfile.yml's build task
(binary), Dockerfile.archied (container), and install.sh (binary) all
now stamp it; cmd/archied/telegram_setup.go wires
Service.InstallType from installtype.Type().
Configuration-page display of install type/version and post-update
verification against the actually-running reference (not just the
installer's reported version) remain open -- this lands the
correctness gate, not the full observability surface.
Refs #522
Brings the chat surface to the Web UI: /api/chat with its own handlers
and tests, a self-contained ui/src/chat feature folder, and the shared
plumbing to reach the same router Telegram uses.
Includes a dangerous-command approval path (internal/gateway/dangerous.go,
internal/webui/dangerous.go, and the Telegram counterpart) so /rollback
and /stop are gated the same way on both surfaces, persona handling for
web sessions, and release-update formatting split out of the Telegram
service so both can render it.
web_parity_test.go pins that the two surfaces expose the same commands.
Authored in a parallel session; committed here so it lands with the rest
rather than sitting uncommitted while other work builds on top of it.
task check passes with it in place.
Screenshots used while building are gitignored (*.png) rather than
committed, matching the call made on the earlier design reference.