Files
archie/RELEASING.md
T
sam 85b21578d8
Quality / check (push) Successful in 3m46s
deploy / gate (push) Successful in 4m29s
deploy / archied-image (push) Successful in 2m53s
deploy / agent-image (push) Successful in 12s
deploy / dist-zip (push) Skipped
ci(release): name the missing DEPLOY_TOKEN before checkout, and list what a mirror needs
Refs #1788
2026-11-05 00:33:18 +11:00

14 KiB
Raw Blame History

Releasing

One release stream. CHANGELOG.md carries every release section, one generated page per version describes it, and one v* tag names the release. One version covers every process in the host bundle (archied, archie-gateway, archie-state-store, archie-ui, archie-messaging) and the archie-agent image; per-component detail is a labelled section inside the version (### archied, ### archie-agent).

A release is cut by CI, published to next, run on the canary for a day, and promoted to stable only if the canary's signals pass. Nothing is run locally.

Notes are written with the change

A commit that changes what a reader experiences adds its bullet under ## [Unreleased] in CHANGELOG.md, in the same commit. The release takes that section as it stands, so write the bullet as the release note, not as a commit subject.

Release notes describe the reader's experience of what shipped. The test is the reader, not whether a sentence is negative:

  • A capability that never existed leaves no expectation to correct, so a note about it reads as a feature to someone who never had the thing: out.
  • The implementation's journey (what was rebuilt, what a guard finally covers, what a browser API refuses to surface) isn't the reader's experience either: out.
  • A capability that shipped but isn't yet wired is the reader's experience, so it belongs, worded as a caveat on the capability and paired with the behaviour that compensates for it.

Design constraints and withheld capabilities are not release news. Deleting a caveat a reader needs misleads them, which is the failure this rule exists to prevent.

Cutting a release

Dispatch the release workflow (.github/workflows/release.yml) with major, minor, patch or an exact version. It runs task release on main:

  1. refuses a dirty tree, an existing tag, or an [Unreleased] with no entries;
  2. turns [Unreleased] into ## [x.y.z] - date under a new empty [Unreleased], and runs task news;
  3. commits chore(release): vx.y.z and tags vx.y.z;

then pushes main and the tag atomically with DEPLOY_TOKEN. A push made with the job token wouldn't start other workflows, so the user token is what lets the tag start the deploy.

Every workflow that reaches the registry or the canary reads its credentials from Actions secrets, so a mirror of this repository carries its own set:

Name Kind Used by
DEPLOY_TOKEN secret release, deploy, canary, promote — the registry push, the tag push, and the package links
RELEASE_SIGNING_KEY secret deploy — signs the release statement
CANARY_TOKEN secret deploy, canary, promote — the canary's dashboard API
CANARY_URL variable deploy, canary, promote — where that dashboard is
CANARY_WINDOW variable canary — the judged window; empty means the default

A mirror missing them fails at its first login or push; release.yml names the missing DEPLOY_TOKEN before it checks out, so the run doesn't read as a broken release.

After a release, pull main before committing on it again: CI has added the release commit.

What a release publishes

The tag push runs deploy.yml:

  • Gate. task check TEST_FLAGS=-count=1 (uncached, including the Docker-backed tests -short skips) and govulncheck, then a check that the tag has a [version] section in CHANGELOG.md.
  • Images. archied:v<version> and archie-agent:v<version> on a release tag; archied:edge and archied:sha-<commit> on any other push to main. :latest moves only on promotion, so it names the newest promoted release.
  • The signed release. archied release publish signs a statement (version, issued_at, zip digest, agent image by digest) with RELEASE_SIGNING_KEY and pushes it with the distribution zip to git.catlow.cloud/sam/archie as v<version> and next. It never moves stable.
  • The release-notes page on Gitea, from that version's CHANGELOG.md section, which the statement links to.
  • The canary install: archied release canary start points the canary at next and installs the release through its dashboard.

GitHub-style Actions run the workflow from the tagged commit, so a fix to deploy.yml reaches a release only if it is an ancestor of the tag. Re-cut the release rather than committing the fix on top of a tagged one.

The canary

The canary is the production host, following next. Every quarter hour canary.yml runs archied release canary check, which reads the canary through its dashboard API and judges the release the canary actually runs, not the moving next tag:

Signal Source Fails when
Install outcome GET /api/chat/update the canary doesn't run the judged release (the watchdog rolled it back)
Readiness GET /health/detailed the rollup isn't ok
Alerts GET /api/alerts an open action alert was first raised after the release was installed
Error log GET /api/logs?level=error error lines per window exceed 2 × baseline + 10
Task outcomes GET /api/tasks the share of finished tasks that parked or died rose more than 10 points
  • Control. The canary's own window before the release was installed. With one production host there's no simultaneous control.
  • Window. 24 hours counted from the install on the canary, not from the release's issue time: the host records it in installed.json (committed_at) and reports it as installed_at in GET /api/chat/update. One cycle of schedules and traffic. The first 30 minutes are for the install and restart. canary.yml passes -window when the CANARY_WINDOW Actions variable is set; the flag defaults to 24h. A host that doesn't report an install time degrades to the release statement's issue time.
  • Hard signals (install, readiness, alerts) roll back at any check after that. Relative signals (errors, tasks) are judged against the baseline whenever both windows carry enough data, so a breach rolls the release back even mid-window; a full window with no failure promotes it.
  • Too little data skips a signal rather than failing it: fewer than 5 finished tasks in the window, or an error log that reaches back less than an hour into the baseline. A log read that stops short of a window is scaled to a full window from what it covers.

Queueing

deploy.yml installs each new release on the canary with archied release canary start -version V, which first points the canary back at next (clearing any rollback pin) and then installs V — unless the canary is already judging a release newer than stable that isn't V, in which case V is queued behind it and the canary keeps running what it judges. -force installs regardless. A check that promotes (or finds nothing to judge) installs the newer next in the same run, so a queue drains one release per promotion.

Subcommands

archied release canary status    read-only: channel/pin, installed release and
                                 installed_at, stable/next, window progress, and
                                 the verdict the current signals would give
archied release canary start     point the canary at next and install -version,
                                 queued behind a release being judged (-force to
                                 override)
archied release canary check     judge the installed release over -window
                                 (default 24h): promote, pick up a newer next,
                                 or roll back; a held canary is reported, and a
                                 held pin that is no longer installable fails
                                 the run red
archied release canary promote   promote the installed release now if its
                                 signals pass; refuses with exit 1 otherwise
archied release canary unpin     return the canary to next from any pin

Promote

A release that passes the window is promoted: stable is retagged to the v<version> manifest. The signature covers the statement, not the tag, so the retag needs no signing key. The archied and archie-agent images' :latest move to their :v<version> first, and the agent image must be the digest the statement pins. Promotion refuses a prerelease and a release issued before the one stable names, the same replay rule hosts enforce. The decision and its reasons are in the canary run log.

Early promotion is a dispatch away: the promote workflow (.github/workflows/promote.yml) takes a version input and runs archied release canary promote -version V on the canary. It judges the release the canary runs — the window doesn't matter for the decision — and refuses (red run) on any hard failure or relative breach with enough data.

Roll back

stable stays put, and the run turns red. Before the canary is pinned to stable, check verifies stable is actually installable — its statement's zip blob must still be in the release repository and its pinned agent image must still resolve (releaseupdate.Installable). If it isn't, the run fails loudly and the canary stays on next rather than being stranded on a pin. Otherwise the canary is pinned to the stable version through its update settings and reinstalled through its dashboard.

A release whose migrations differ from stable's can't go back without restoring the pre-update database snapshot, which loses every write since. The canary is held on the failed release instead (pinned to it) and the fix goes forward in the next release. The next release's deploy points the canary back at next.

A pinned canary isn't judged, so a rollback is decided once — but check verifies the pin is still installable every run, so a stranded pin shows red instead of green. archied release canary unpin returns the canary to next by hand.

Configuration

Name Kind Holds
CANARY_URL Actions variable the canary's dashboard URL; unset disables the canary steps
CANARY_TOKEN Actions secret an API token bound to a dedicated service_account identity on the canary, carrying only the canary grant policy (see below) — never instance admin
CANARY_WINDOW Actions variable optional -window override for canary check; unset is 24h
DEPLOY_TOKEN Actions secret registry and repository write, for publishing, promoting and the release push; stays CI-only, promotion runs in CI and never on the host
RELEASE_SIGNING_KEY Actions secret the release signing seed

Setting up CANARY_TOKEN once

CANARY_TOKEN is a bearer token for a dedicated service-account identity in the system org whose only grant is the canary policy (access.CanaryGrantPolicy): read the signal endpoints (GET /api/chat/update, GET /api/alerts, GET /api/logs, GET /api/tasks), read and write the update-settings resource, and trigger an update install (POST /api/chat/update/install). It carries no other rights, and instance admin is never needed. With an instance-admin token (for example the operator session), run against the canary host:

# 1. Create the service-account identity in the system org.
curl -X POST "$CANARY_URL/api/identities" -H "Authorization: Bearer $ADMIN" \
  -H 'Content-Type: application/json' \
  -d '{"kind":"service_account","display_name":"ci-canary"}'
# -> {"id":"<canary-identity-id>", ...}

# 2. Issue its token (shown exactly once).
curl -X POST "$CANARY_URL/api/identities/<canary-identity-id>/tokens" \
  -H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' \
  -d '{"name":"ci-canary","days":90}'
# -> {"id":"<token-hash>","token":"archie_pat_..."}  -- store the token value as the CANARY_TOKEN secret

# 3. Install the grant pair, scoped to that identity. The policies' org is
#    pinned to the caller's org (org-sys for the instance admin), so the
#    bodies need no org field. The permit alone is not enough: service
#    accounts created through the API are assigned the developer role, so a
#    matching forbid (installed second) negates everything outside the
#    permit's surface for that identity.
curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \
  -H 'Content-Type: application/json' \
  -d '{"id":"org-canary-grant","level":"org","text":"permit(principal, action, resource) when {\n    resource.org == principal.org &&\n    principal == Archie::Identity::\"<canary-identity-id>\" &&\n    (\n        (action == Archie::Action::\"read\" && [\"dashboard\", \"task\"].contains(resource.kind)) ||\n        (action == Archie::Action::\"read\" && resource == Archie::Object::\"update-settings\") ||\n        (action == Archie::Action::\"update\" && resource == Archie::Object::\"update\") ||\n        (action == Archie::Action::\"manage_identities\" && resource == Archie::Object::\"update-settings\")\n    )\n};"}'

The second policy forbids the canary identity from everything the permit does not name, so its developer role cannot widen the grant:

curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \
  -H 'Content-Type: application/json' \
  -d '{"id":"org-canary-grant-forbid","level":"org","text":"forbid(principal, action, resource) when {\\n    resource.org == principal.org &&\\n    principal == Archie::Identity::\\\"<canary-identity-id>\\\" &&\\n    !(\\n        (action == Archie::Action::\\\"read\\\" && [\\\"dashboard\\\", \\\"task\\\"].contains(resource.kind)) ||\\n        (action == Archie::Action::\\\"read\\\" && resource == Archie::Object::\\\"update-settings\\\") ||\\n        (action == Archie::Action::\\\"update\\\" && resource == Archie::Object::\\\"update\\\") ||\\n        (action == Archie::Action::\\\"manage_identities\\\" && resource == Archie::Object::\\\"update-settings\\\")\\n    )\\n};"}'

The policy texts are exactly access.CanaryGrantPolicy and access.CanaryGrantForbidPolicy from internal/domain/access/canary.go with the identity id filled in. The canary needs no other rights: GET /health/detailed is unauthenticated, and every other call it makes is covered by the grant pair.

The runner must reach CANARY_URL. Without a canary a release stays on next until promoted by hand, under the same refusals:

archied release promote -version 1.57.0 \
  -latest git.catlow.cloud/sam/archied,git.catlow.cloud/sam/archie-agent