feat(releaseupdate): judge the installed release, window from install (#1725)
Quality / check (push) Successful in 4m2s
deploy / gate (push) Successful in 4m31s
deploy / archied-image (push) Successful in 3m14s
deploy / agent-image (push) Successful in 11s
deploy / dist-zip (push) Skipped

The canary window now counts from when a release is installed on the host
(installed.json / installed_at in the update snapshot), not from the
statement's issued_at, and 'release canary check' judges the release the
canary actually runs via its exact-pin tag instead of the moving next tag.
A newer release on next queues behind the one being judged instead of
restarting the clock, so stable moves once per release.

New subcommands: canary status (read-only state, signals, verdict), canary
promote (promote now when hard checks pass), canary unpin (back to next),
and start -force. A held/pinned canary is verified installable every check
and a stranded pin fails the run red instead of looking green. Rollback
pre-flight checks stable's zip and agent image digests before pinning.
Relative signals are judged whenever both windows carry minimum data, so a
breach rolls back mid-window; promote still needs the full window unless
the operator promotes early (new promote.yml workflow_dispatch). The window
is configurable via -window / CANARY_WINDOW.

Also: dashboard canary status card (ui), and task canary:rehearse, a local
end-to-end rehearsal of publish -> canary start -> status -> check ->
promote against a throwaway registry and host.

task check passes; rehearsal run green (REHEARSE_EXIT=0).
This commit is contained in:
sam committed 2026-11-02 01:15:29 +11:00
1 parent bca1f3b412
commit 5b5059615b
18 files changed
+1487 -180

No files matched your search

+12 -4
View File
@@ -1,7 +1,9 @@
# Every 15 minutes, checks how the newest release on the `next` channel is
# running on the canary host. If a hard check fails, the canary goes back to
# the stable release straight away. If the release stays healthy for 24 hours,
# it's promoted to stable. A rollback fails the run, so it shows up red.
# Every 15 minutes, checks how the release installed on the canary host is
# running, judged over the window counted from its install (-window, default
# 24h; CANARY_WINDOW overrides it). If a hard check fails, the canary goes
# back to the stable release straight away. If the release stays healthy for
# the window, it's promoted to stable. A rollback fails the run, so it shows
# up red.
name: canary
on:
@@ -48,6 +50,12 @@ jobs:
- name: Judge the canary
env:
ARCHIE_CANARY_TOKEN: ${{ secrets.CANARY_TOKEN }}
CANARY_WINDOW: ${{ vars.CANARY_WINDOW }}
run: |
window_args=()
if [ -n "$CANARY_WINDOW" ]; then
window_args=(-window "$CANARY_WINDOW")
fi
"$RUNNER_TEMP/archied" release canary check -dashboard "${{ vars.CANARY_URL }}" \
"${window_args[@]}" \
-latest git.catlow.cloud/sam/archied,git.catlow.cloud/sam/archie-agent
+6 -2
View File
@@ -418,8 +418,12 @@ jobs:
"$api/repos/$repo/releases" >/dev/null
echo "published release notes for $tag"
# The canary host follows the `next` channel. canary.yml then decides
# whether this release is promoted to stable or the canary is rolled back.
# The canary host follows the `next` channel. canary.yml then judges the
# release installed on the canary and decides whether it is promoted to
# stable or the canary is rolled back. The start is queue-safe: a release
# published while the canary still judges a newer one is queued behind
# it and picked up by the next check, so the running window isn't
# restarted under a different release.
- name: Install the release on the canary
if: vars.CANARY_URL != ''
env:
+54
View File
@@ -0,0 +1,54 @@
# Promotes the release the canary is running right now, without waiting out
# the judging window: the canary's signals must show no hard failure and no
# relative breach, otherwise the run refuses and fails. Dispatch it with the
# version the canary runs (`archied release canary status` shows it).
name: promote
on:
workflow_dispatch:
inputs:
version:
description: "release the canary is expected to run; must match what it runs"
required: true
type: string
concurrency:
group: promote
permissions:
contents: read
packages: write
jobs:
promote:
if: vars.CANARY_URL != ''
runs-on: self-hosted
steps:
- name: Checkout
uses: actions/checkout@v4
with:
ref: main
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version-file: go.mod
# The self-hosted runner keeps the Go caches on disk between jobs, and
# archiving them for the Actions cache takes minutes.
cache: false
- name: Build archied
# The promote never serves the dashboard, so the build leaves it out.
run: go build -tags no_ui -o "$RUNNER_TEMP/archied" ./cmd/archied
# Promotion retags stable and the images' latest in the registry.
- name: Log in to the Gitea registry
run: echo "${{ secrets.DEPLOY_TOKEN }}" | docker login git.catlow.cloud -u "${{ github.actor }}" --password-stdin
- name: Promote the canary's release
env:
ARCHIE_CANARY_TOKEN: ${{ secrets.CANARY_TOKEN }}
run: |
"$RUNNER_TEMP/archied" release canary promote -dashboard "${{ vars.CANARY_URL }}" \
-version "${{ inputs.version }}" \
-latest git.catlow.cloud/sam/archied,git.catlow.cloud/sam/archie-agent