Release by CI dispatch and promote to stable only after a canary evaluation #1716

Open
opened 2026-10-31 10:28:49 +00:00 by sam · 2 comments
Owner

Releases are cut by hand and every non-prerelease moves next and stable together, so a bad release reaches every host on stable at once.

Wanted

  • A manual release workflow (version or bump level) moves [Unreleased] in CHANGELOG.md to the new version, runs task news, commits, tags and pushes atomically. It refuses an empty [Unreleased]. The subject-dump generator (tools/release.sh, release:preview, release:prepare) is deleted; notes are written with each change.
  • A release publishes v<version> and next only.
  • The canary is the production host on next (decided). The deploy run points it at next and installs through its dashboard API.
  • A scheduled evaluator compares the canary's 24h after the release was issued with its 24h before (decided: one production host, so the control is the canary's own earlier window):
    • hard fail at any check: the installed version is not the release (the watchdog rolled it back), readiness is degraded, or a new action alert is open;
    • relative fail at 24h: error-level log lines over 2 × baseline + 10; task park+dead rate over baseline + 10 points, needing at least 5 finished tasks in the window, otherwise that signal is skipped.
  • Promote (automatic, decided): stable is retagged to the release's signed manifest, refused if that would move stable to an older issued_at. No re-signing: the signature covers the statement.
  • Roll back: stable stays put; the canary is pinned to the stable version and reinstalled. If the release migrated the database, the binaries cannot go back without restoring the pre-update snapshot (losing writes since), so the evaluator holds the canary and reports instead.
  • RELEASING.md describes the new flow.

Done when

  • Dispatching the workflow with an [Unreleased] section produces the tag, the signed artifact on v<version> and next, and the release-notes page with no local steps.
  • The evaluator returns promote or roll back from those signals over that window; stable moves only on promote.
  • task check passes on each commit; a local registry:2 rehearsal of the pipeline is recorded here.
Releases are cut by hand and every non-prerelease moves `next` and `stable` together, so a bad release reaches every host on `stable` at once. ## Wanted - A manual `release` workflow (version or bump level) moves `[Unreleased]` in CHANGELOG.md to the new version, runs `task news`, commits, tags and pushes atomically. It refuses an empty `[Unreleased]`. The subject-dump generator (`tools/release.sh`, `release:preview`, `release:prepare`) is deleted; notes are written with each change. - A release publishes `v<version>` and `next` only. - The canary is the production host on `next` (decided). The deploy run points it at `next` and installs through its dashboard API. - A scheduled evaluator compares the canary's 24h after the release was issued with its 24h before (decided: one production host, so the control is the canary's own earlier window): - hard fail at any check: the installed version is not the release (the watchdog rolled it back), readiness is degraded, or a new `action` alert is open; - relative fail at 24h: error-level log lines over `2 × baseline + 10`; task park+dead rate over `baseline + 10` points, needing at least 5 finished tasks in the window, otherwise that signal is skipped. - Promote (automatic, decided): `stable` is retagged to the release's signed manifest, refused if that would move `stable` to an older `issued_at`. No re-signing: the signature covers the statement. - Roll back: `stable` stays put; the canary is pinned to the `stable` version and reinstalled. If the release migrated the database, the binaries cannot go back without restoring the pre-update snapshot (losing writes since), so the evaluator holds the canary and reports instead. - RELEASING.md describes the new flow. ## Done when - Dispatching the workflow with an `[Unreleased]` section produces the tag, the signed artifact on `v<version>` and `next`, and the release-notes page with no local steps. - The evaluator returns promote or roll back from those signals over that window; `stable` moves only on promote. - `task check` passes on each commit; a local `registry:2` rehearsal of the pipeline is recorded here.
Author
Owner

Built in a7f6e517a (release workflow, task release, generator deleted) and a4a7c96da (publish to next only, archied release canary start|check, archied release promote, canary.yml, RELEASING.md).

Rehearsal

Local registry:2 on loopback, a throwaway signing key, a stub canary dashboard serving the same endpoints (/api/chat/update, /health/detailed, /api/alerts, /api/tasks, /api/logs, update-settings replace, install), and a rehearsal-only archied built with go build -overlay to backdate issued_at (nothing in the tree).

  • task release RELEASE=minor in a clone: refused with an empty [Unreleased]; with an entry it wrote ## [1.57.0] - 2026-10-31 under a new [Unreleased], ran newsgen, committed and tagged v1.57.0.
  • 1.0.0 issued 25h ago, first release: canary start set the channel to next (source and key kept) and installed it; canary check → promote; stable resolves to 1.0.0.
  • 1.1.0 issued 2h ago, readiness degraded, no migration: → roll back; canary pinned to 1.0.0 and reinstalled; stable unchanged; exit 1. The next check: "held on exact-pin 1.0.0", no action.
  • 1.2.0 issued 1h ago, migrations changed, degraded: → roll back; canary held on 1.2.0; stable unchanged.
  • 1.3.0-rc.1 on next: nothing to judge. release promote refuses it (prerelease) and refuses 1.0.0 once stable is 1.1.0 (issued before).
  • task check passed on both commits. TestPromote and TestJudgeCanary cover the promotion refusals and the verdict table; reverting the issued-before or prerelease check fails TestPromote.

Not yet exercised

The workflows on the live runner, and the dashboard API on the real canary. They need the CANARY_URL variable and the CANARY_TOKEN secret (an instance-admin API token on the canary), and the runner must reach that URL.

Built in a7f6e517a (release workflow, `task release`, generator deleted) and a4a7c96da (publish to `next` only, `archied release canary start|check`, `archied release promote`, `canary.yml`, RELEASING.md). ## Rehearsal Local `registry:2` on loopback, a throwaway signing key, a stub canary dashboard serving the same endpoints (`/api/chat/update`, `/health/detailed`, `/api/alerts`, `/api/tasks`, `/api/logs`, update-settings replace, install), and a rehearsal-only archied built with `go build -overlay` to backdate `issued_at` (nothing in the tree). - `task release RELEASE=minor` in a clone: refused with an empty `[Unreleased]`; with an entry it wrote `## [1.57.0] - 2026-10-31` under a new `[Unreleased]`, ran newsgen, committed and tagged `v1.57.0`. - 1.0.0 issued 25h ago, first release: `canary start` set the channel to next (source and key kept) and installed it; `canary check` → promote; `stable` resolves to 1.0.0. - 1.1.0 issued 2h ago, readiness degraded, no migration: → roll back; canary pinned to 1.0.0 and reinstalled; `stable` unchanged; exit 1. The next check: "held on exact-pin 1.0.0", no action. - 1.2.0 issued 1h ago, migrations changed, degraded: → roll back; canary held on 1.2.0; `stable` unchanged. - 1.3.0-rc.1 on next: nothing to judge. `release promote` refuses it (prerelease) and refuses 1.0.0 once `stable` is 1.1.0 (issued before). - `task check` passed on both commits. `TestPromote` and `TestJudgeCanary` cover the promotion refusals and the verdict table; reverting the issued-before or prerelease check fails `TestPromote`. ## Not yet exercised The workflows on the live runner, and the dashboard API on the real canary. They need the `CANARY_URL` variable and the `CANARY_TOKEN` secret (an instance-admin API token on the canary), and the runner must reach that URL.
Author
Owner

2181b5fd0: release images are published as archied:v<version> and archie-agent:v<version>; promotion (canary check -latest, release promote -latest) moves each image's :latest to its :v<version> before stable, refusing an agent image whose digest is not the one the statement pins.

Rehearsed on a local registry:2: promoting 1.0.0 moved both :latest tags to the :v1.0.0 digest; promoting 1.1.0 whose archie-agent:v1.1.0 differed from the pinned digest was refused and stable stayed on 1.0.0. task check passed.

2181b5fd0: release images are published as `archied:v<version>` and `archie-agent:v<version>`; promotion (`canary check -latest`, `release promote -latest`) moves each image's `:latest` to its `:v<version>` before `stable`, refusing an agent image whose digest is not the one the statement pins. Rehearsed on a local `registry:2`: promoting 1.0.0 moved both `:latest` tags to the `:v1.0.0` digest; promoting 1.1.0 whose `archie-agent:v1.1.0` differed from the pinned digest was refused and `stable` stayed on 1.0.0. `task check` passed.
sam added the featuremedium labels 2026-11-14 14:45:44 +00:00
sam added the deferred label 2026-11-15 17:26:14 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1716