Self-update: artifact install, topology migration, journaled transaction #1361

Closed
opened 2026-10-27 13:40:05 +00:00 by sam · 0 comments
Owner

Problem

The self-update contract (internal/releaseupdate: catalog / defer / install / watchdog / verify) is sound. Behind it, the reference adapter scripts carry three structural assumptions that do not hold for any host already in the field.

  1. Discovery required a source checkout. scripts/archie-update-check resolved available releases via git -C "$REPO" ls-remote, with REPO defaulting to an operator checkout path, guarded by 2>/dev/null || true. A missing, moved or renamed checkout produced an empty Available for every component — which FormatSnapshot renders as "Archie is up to date." An unreachable remote read as parity. Fixed in #793.

  2. Install builds from source on the operator host. scripts/archie-update-install clones the repo and runs go build for every binary in the release, so the host needs a Go toolchain, network access to the git remote, and enough disk and CPU to compile the whole tree.

  3. The release is assumed to fit the running topology. A host deployed before the process decomposition has no [services.state] config, no State Store / Gateway / UI units, and no sibling binaries. Crossing that boundary is not a binary swap, and nothing in the product performs or documents the crossing.

Two further gaps in the same area:

  • No release artifacts exist to consume. The CI dist-zip job builds a complete distribution archive, but it is an Actions artifact with 90-day retention and is never attached to a release — so it cannot be an install source for anything older than the retention window.
  • A fresh install reproduces the broken shape. install.sh writes only archied.service, while three deployment docs describe four units. deployment_contract_test.go still asserts the single-unit topology.

Scope

1. Artifact-based install. Publish per-platform archive, SHA256SUMS and a release.json describing components/versions/supported deployment profiles, as release assets on the tag. The updater downloads the artifact for the approved tag, verifies it against the manifest, stages it and installs from the staged tree. Removes the toolchain requirement, the checkout dependency, and the dependency on the remote being reachable from the host. Source build stays as an explicit opt-in fallback.

2. Topology migration. An observed topology (which units exist, which process owns the database), a per-release declaration of the topology it requires, and an approved migration plan that creates missing units, materialises required config additions, installs binaries and starts them in dependency order — with the rollback covering units and config as well as binaries. Refuse with a precise plan when the target topology cannot be derived.

3. Journaled transaction. Replace the current copy-and-trap install with: pre-flight (required units, disk space, artifact verifiable, database backed up at the resolved path, schema migration reversible or explicitly accepted), a durable intent journal outside the paths being replaced, stage-then-switch (never overwrite a live binary with a non-atomic copy — a mid-copy kill is the unrecoverable case), health verification against the address the candidate actually serves, bounded post-promotion observation, transaction-wide rollback, and resume-on-boot from an unfinished journal. A single-writer lock that lives outside the process being replaced.

4. Channel, pinning, automation. stable / next / exact-pin channel selection (today's "newest tag by sort -V" is next semantics and cannot be pinned). Runtime image pinned by digest in every shipped profile, instead of a moving :latest, with the Runtime component's version identity taken from the image execution actually uses. Opt-in auto-update that refuses to start with work in flight.

Known trap

The reference install script refreshes its own adapters from the approved release source partway through the run, and bash keeps executing the copy it opened at startup. A newly added pre-flight check therefore first protects the next update, not the one installing it — so a host already in the field crosses one release unprotected. Any topology requirement must live in the daemon, which knows the deployment and holds the approved snapshot, and be passed to the installer; the shell adapter's copy is defence in depth only.

Acceptance criteria

  • A host with no Go toolchain can install an approved release from its published artifacts.
  • Discovery and install never require or mutate an operator/agent checkout.
  • A host on a pre-decomposition topology can cross it via an approved, reversible migration plan.
  • An update that cannot be rolled back is refused before any host mutation.
  • An interrupted update is resumable and leaves the host in a known state.
  • Rollback restores binaries, units, config additions and the database, in stop -> restore -> restart order.
  • Every shipped profile pins the runtime image by digest.

Related

  • #641 — install without a source checkout (install side landed previously; the check side landed in #793).
  • #644 — relay dashboard update completion through Telegram.
  • #645 — Web UI session UUID compatibility and update-status synchronisation.

Merged from the other tracker

The remainder of the self-update work, after #793 landed the discovery and DB-backup fixes.

Context

The update contract (internal/releaseupdate: catalog/defer/install/watchdog/verify) is sound. Behind it, the reference adapter scripts/archie-update-{check,install,watchdog} carry assumptions that do not hold for any host in the field.

Work

  1. Artifact install. Publish per-platform archive + SHA256SUMS + release.json as release ASSETS (currently the CI dist-zip is an Actions artifact with 90-day retention and is not attached to any release, so it cannot be an install source). Install from the staged artifact instead of cloning and compiling on the operator host. Keep the source build as an explicit opt-in fallback.

  2. Topology migration. A host deployed before the process decomposition has no [services.state] config, no State Store/Gateway/UI units and no sibling binaries. Crossing that boundary is not a binary swap, and nothing in the product performs or documents it. Needs: an observed topology (which units exist, which process owns the DB), a per-release declaration of the topology it requires, and an approved migration plan that creates units and config additions and starts them in dependency order. Refuse, with a precise plan, when it cannot be derived.

  3. Journaled transaction. Replace the copy-and-trap install with: pre-flight, a durable intent journal outside the replaced paths, stage-then-switch (never overwrite a live binary with a non-atomic copy), health verification against the candidate rather than a fixed default, bounded post-promotion observation, transaction-wide rollback covering units and config as well as binaries, and resume-on-boot. Single-writer lock outside the process being replaced.

  4. Channel / pinning / automation. stable|next|exact-pin channel selection; runtime image pinned by digest in every shipped profile (currently :latest everywhere); opt-in auto-update that refuses to start with work in flight.

Known trap (do not repeat)

The reference install script refreshes its own adapters from the release source partway through the run, and bash keeps executing the copy it opened at startup. A newly added pre-flight check therefore first protects the NEXT update, not the one installing it -- so a host already in the field crosses one release unprotected. Any topology requirement must live in the daemon, which knows the deployment and holds the approved snapshot, and be passed to the installer; the shell adapter's copy is defence in depth only.

Also relevant

An install.sh on a fresh host writes only archied.service, while three deployment docs describe four units -- so a new install reproduces the same broken shape. deployment_contract_test.go still asserts the single-unit topology.

Design rationale (not for a public PR): the private tracker and session notes hold the full analysis.

Close reason

Closed


Imported from beads archie-core-hj1t · GitHub #804 · body merged from the fuller GitHub #794 · type epic · priority P2 · created 2026-09-12 · closed 2026-10-05

## Problem The self-update contract (`internal/releaseupdate`: catalog / defer / install / watchdog / verify) is sound. Behind it, the reference adapter scripts carry three structural assumptions that do not hold for any host already in the field. 1. **Discovery required a source checkout.** `scripts/archie-update-check` resolved available releases via `git -C "$REPO" ls-remote`, with `REPO` defaulting to an operator checkout path, guarded by `2>/dev/null || true`. A missing, moved or renamed checkout produced an empty `Available` for every component — which `FormatSnapshot` renders as *"Archie is up to date."* An unreachable remote read as parity. **Fixed in #793.** 2. **Install builds from source on the operator host.** `scripts/archie-update-install` clones the repo and runs `go build` for every binary in the release, so the host needs a Go toolchain, network access to the git remote, and enough disk and CPU to compile the whole tree. 3. **The release is assumed to fit the running topology.** A host deployed before the process decomposition has no `[services.state]` config, no State Store / Gateway / UI units, and no sibling binaries. Crossing that boundary is not a binary swap, and nothing in the product performs or documents the crossing. Two further gaps in the same area: - **No release artifacts exist to consume.** The CI `dist-zip` job builds a complete distribution archive, but it is an Actions artifact with 90-day retention and is never attached to a release — so it cannot be an install source for anything older than the retention window. - **A fresh install reproduces the broken shape.** `install.sh` writes only `archied.service`, while three deployment docs describe four units. `deployment_contract_test.go` still asserts the single-unit topology. ## Scope **1. Artifact-based install.** Publish per-platform archive, `SHA256SUMS` and a `release.json` describing components/versions/supported deployment profiles, as **release assets** on the tag. The updater downloads the artifact for the approved tag, verifies it against the manifest, stages it and installs from the staged tree. Removes the toolchain requirement, the checkout dependency, and the dependency on the remote being reachable from the host. Source build stays as an explicit opt-in fallback. **2. Topology migration.** An observed topology (which units exist, which process owns the database), a per-release declaration of the topology it requires, and an approved migration plan that creates missing units, materialises required config additions, installs binaries and starts them in dependency order — with the rollback covering units and config as well as binaries. Refuse with a precise plan when the target topology cannot be derived. **3. Journaled transaction.** Replace the current copy-and-trap install with: pre-flight (required units, disk space, artifact verifiable, database backed up at the resolved path, schema migration reversible or explicitly accepted), a durable intent journal outside the paths being replaced, stage-then-switch (never overwrite a live binary with a non-atomic copy — a mid-copy kill is the unrecoverable case), health verification against the address the candidate actually serves, bounded post-promotion observation, transaction-wide rollback, and resume-on-boot from an unfinished journal. A single-writer lock that lives outside the process being replaced. **4. Channel, pinning, automation.** `stable` / `next` / exact-pin channel selection (today's "newest tag by `sort -V`" is `next` semantics and cannot be pinned). Runtime image pinned by digest in every shipped profile, instead of a moving `:latest`, with the Runtime component's version identity taken from the image execution actually uses. Opt-in auto-update that refuses to start with work in flight. ## Known trap The reference install script refreshes its own adapters from the approved release source partway through the run, and bash keeps executing the copy it opened at startup. A newly added pre-flight check therefore first protects the **next** update, not the one installing it — so a host already in the field crosses one release unprotected. Any topology requirement must live in the daemon, which knows the deployment and holds the approved snapshot, and be passed to the installer; the shell adapter's copy is defence in depth only. ## Acceptance criteria - [ ] A host with no Go toolchain can install an approved release from its published artifacts. - [ ] Discovery and install never require or mutate an operator/agent checkout. - [ ] A host on a pre-decomposition topology can cross it via an approved, reversible migration plan. - [ ] An update that cannot be rolled back is refused before any host mutation. - [ ] An interrupted update is resumable and leaves the host in a known state. - [ ] Rollback restores binaries, units, config additions and the database, in `stop -> restore -> restart` order. - [ ] Every shipped profile pins the runtime image by digest. ## Related - #641 — install without a source checkout (install side landed previously; the check side landed in #793). - #644 — relay dashboard update completion through Telegram. - #645 — Web UI session UUID compatibility and update-status synchronisation. ### Merged from the other tracker The remainder of the self-update work, after #793 landed the discovery and DB-backup fixes. ## Context The update contract (internal/releaseupdate: catalog/defer/install/watchdog/verify) is sound. Behind it, the reference adapter scripts/archie-update-{check,install,watchdog} carry assumptions that do not hold for any host in the field. ## Work 1. **Artifact install.** Publish per-platform archive + SHA256SUMS + release.json as release ASSETS (currently the CI dist-zip is an Actions artifact with 90-day retention and is not attached to any release, so it cannot be an install source). Install from the staged artifact instead of cloning and compiling on the operator host. Keep the source build as an explicit opt-in fallback. 2. **Topology migration.** A host deployed before the process decomposition has no [services.state] config, no State Store/Gateway/UI units and no sibling binaries. Crossing that boundary is not a binary swap, and nothing in the product performs or documents it. Needs: an observed topology (which units exist, which process owns the DB), a per-release declaration of the topology it requires, and an approved migration plan that creates units and config additions and starts them in dependency order. Refuse, with a precise plan, when it cannot be derived. 3. **Journaled transaction.** Replace the copy-and-trap install with: pre-flight, a durable intent journal outside the replaced paths, stage-then-switch (never overwrite a live binary with a non-atomic copy), health verification against the candidate rather than a fixed default, bounded post-promotion observation, transaction-wide rollback covering units and config as well as binaries, and resume-on-boot. Single-writer lock outside the process being replaced. 4. **Channel / pinning / automation.** stable|next|exact-pin channel selection; runtime image pinned by digest in every shipped profile (currently :latest everywhere); opt-in auto-update that refuses to start with work in flight. ## Known trap (do not repeat) The reference install script refreshes its own adapters from the release source partway through the run, and bash keeps executing the copy it opened at startup. A newly added pre-flight check therefore first protects the NEXT update, not the one installing it -- so a host already in the field crosses one release unprotected. Any topology requirement must live in the daemon, which knows the deployment and holds the approved snapshot, and be passed to the installer; the shell adapter's copy is defence in depth only. ## Also relevant An install.sh on a fresh host writes only archied.service, while three deployment docs describe four units -- so a new install reproduces the same broken shape. deployment_contract_test.go still asserts the single-unit topology. Design rationale (not for a public PR): the private tracker and session notes hold the full analysis. ### Close reason Closed --- _Imported from beads `archie-core-hj1t` · GitHub #804 · body merged from the fuller GitHub #794 · type epic · priority P2 · created 2026-09-12 · closed 2026-10-05_
sam added the mediumepic labels 2026-10-27 13:40:05 +00:00
sam self-assigned this 2026-10-27 13:40:05 +00:00
sam closed this issue 2026-10-27 13:40:05 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1361