14 KiB
Releasing
One release stream. CHANGELOG.md carries every release section, one generated
page per version describes it, and one v* tag names the release. One version
covers every process in the host bundle (archied, archie-gateway,
archie-state-store, archie-ui, archie-messaging) and the archie-agent
image; per-component detail is a labelled section inside the version
(### archied, ### archie-agent).
A release is cut by CI, published to next, run on the canary for a day, and
promoted to stable only if the canary's signals pass. Nothing is run locally.
Notes are written with the change
A commit that changes what a reader experiences adds its bullet under
## [Unreleased] in CHANGELOG.md, in the same commit. The release takes that
section as it stands, so write the bullet as the release note, not as a commit
subject.
Release notes describe the reader's experience of what shipped. The test is the reader, not whether a sentence is negative:
- A capability that never existed leaves no expectation to correct, so a note about it reads as a feature to someone who never had the thing: out.
- The implementation's journey (what was rebuilt, what a guard finally covers, what a browser API refuses to surface) isn't the reader's experience either: out.
- A capability that shipped but isn't yet wired is the reader's experience, so it belongs, worded as a caveat on the capability and paired with the behaviour that compensates for it.
Design constraints and withheld capabilities are not release news. Deleting a caveat a reader needs misleads them, which is the failure this rule exists to prevent.
Cutting a release
Dispatch the release workflow (.github/workflows/release.yml) with
major, minor, patch or an exact version. It runs task release on main:
- refuses a dirty tree, an existing tag, or an
[Unreleased]with no entries; - turns
[Unreleased]into## [x.y.z] - dateunder a new empty[Unreleased], and runstask news; - commits
chore(release): vx.y.zand tagsvx.y.z;
then pushes main and the tag atomically with DEPLOY_TOKEN. A push made with
the job token wouldn't start other workflows, so the user token is what lets
the tag start the deploy.
Every workflow that reaches the registry or the canary reads its credentials from Actions secrets, so a mirror of this repository carries its own set:
| Name | Kind | Used by |
|---|---|---|
DEPLOY_TOKEN |
secret | release, deploy, canary, promote — the registry push, the tag push, and the package links |
RELEASE_SIGNING_KEY |
secret | deploy — signs the release statement |
CANARY_TOKEN |
secret | deploy, canary, promote — the canary's dashboard API |
CANARY_URL |
variable | deploy, canary, promote — where that dashboard is |
CANARY_WINDOW |
variable | canary — the judged window; empty means the default |
A mirror missing them fails at its first login or push; release.yml names the
missing DEPLOY_TOKEN before it checks out, so the run doesn't read as a broken
release.
After a release, pull main before committing on it again: CI has added the
release commit.
What a release publishes
The tag push runs deploy.yml:
- Gate.
task check TEST_FLAGS=-count=1(uncached, including the Docker-backed tests-shortskips) andgovulncheck, then a check that the tag has a[version]section inCHANGELOG.md. - Images.
archied:v<version>andarchie-agent:v<version>on a release tag;archied:edgeandarchied:sha-<commit>on any other push tomain.:latestmoves only on promotion, so it names the newest promoted release. - The signed release.
archied release publishsigns a statement (version, issued_at, zip digest, agent image by digest) withRELEASE_SIGNING_KEYand pushes it with the distribution zip togit.catlow.cloud/sam/archieasv<version>andnext. It never movesstable. - The release-notes page on Gitea, from that version's
CHANGELOG.mdsection, which the statement links to. - The canary install:
archied release canary startpoints the canary atnextand installs the release through its dashboard.
GitHub-style Actions run the workflow from the tagged commit, so a fix to
deploy.yml reaches a release only if it is an ancestor of the tag. Re-cut the
release rather than committing the fix on top of a tagged one.
The canary
The canary is the production host, following next. Every quarter hour
canary.yml runs archied release canary check, which reads the canary
through its dashboard API and judges the release the canary actually runs,
not the moving next tag:
| Signal | Source | Fails when |
|---|---|---|
| Install outcome | GET /api/chat/update |
the canary doesn't run the judged release (the watchdog rolled it back) |
| Readiness | GET /health/detailed |
the rollup isn't ok |
| Alerts | GET /api/alerts |
an open action alert was first raised after the release was installed |
| Error log | GET /api/logs?level=error |
error lines per window exceed 2 × baseline + 10 |
| Task outcomes | GET /api/tasks |
the share of finished tasks that parked or died rose more than 10 points |
- Control. The canary's own window before the release was installed. With one production host there's no simultaneous control.
- Window. 24 hours counted from the install on the canary, not from
the release's issue time: the host records it in
installed.json(committed_at) and reports it asinstalled_atinGET /api/chat/update. One cycle of schedules and traffic. The first 30 minutes are for the install and restart.canary.ymlpasses-windowwhen theCANARY_WINDOWActions variable is set; the flag defaults to 24h. A host that doesn't report an install time degrades to the release statement's issue time. - Hard signals (install, readiness, alerts) roll back at any check after that. Relative signals (errors, tasks) are judged against the baseline whenever both windows carry enough data, so a breach rolls the release back even mid-window; a full window with no failure promotes it.
- Too little data skips a signal rather than failing it: fewer than 5 finished tasks in the window, or an error log that reaches back less than an hour into the baseline. A log read that stops short of a window is scaled to a full window from what it covers.
Queueing
deploy.yml installs each new release on the canary with
archied release canary start -version V, which first points the canary back
at next (clearing any rollback pin) and then installs V — unless the
canary is already judging a release newer than stable that isn't V, in which
case V is queued behind it and the canary keeps running what it judges.
-force installs regardless. A check that promotes (or finds nothing to
judge) installs the newer next in the same run, so a queue drains one
release per promotion.
Subcommands
archied release canary status read-only: channel/pin, installed release and
installed_at, stable/next, window progress, and
the verdict the current signals would give
archied release canary start point the canary at next and install -version,
queued behind a release being judged (-force to
override)
archied release canary check judge the installed release over -window
(default 24h): promote, pick up a newer next,
or roll back; a held canary is reported, and a
held pin that is no longer installable fails
the run red
archied release canary promote promote the installed release now if its
signals pass; refuses with exit 1 otherwise
archied release canary unpin return the canary to next from any pin
Promote
A release that passes the window is promoted: stable is retagged to the
v<version> manifest. The signature covers the statement, not the tag, so the
retag needs no signing key. The archied and archie-agent images' :latest
move to their :v<version> first, and the agent image must be the digest the
statement pins. Promotion refuses a prerelease and a release issued before the
one stable names, the same replay rule hosts enforce. The decision and its
reasons are in the canary run log.
Early promotion is a dispatch away: the promote workflow
(.github/workflows/promote.yml) takes a version input and runs
archied release canary promote -version V on the canary. It judges the
release the canary runs — the window doesn't matter for the decision — and
refuses (red run) on any hard failure or relative breach with enough data.
Roll back
stable stays put, and the run turns red. Before the canary is pinned to
stable, check verifies stable is actually installable — its statement's
zip blob must still be in the release repository and its pinned agent image
must still resolve (releaseupdate.Installable). If it isn't, the run fails
loudly and the canary stays on next rather than being stranded on a pin.
Otherwise the canary is pinned to the stable version through its update
settings and reinstalled through its dashboard.
A release whose migrations differ from stable's can't go back without
restoring the pre-update database snapshot, which loses every write since. The
canary is held on the failed release instead (pinned to it) and the fix goes
forward in the next release. The next release's deploy points the canary back
at next.
A pinned canary isn't judged, so a rollback is decided once — but check
verifies the pin is still installable every run, so a stranded pin shows red
instead of green. archied release canary unpin returns the canary to next
by hand.
Configuration
| Name | Kind | Holds |
|---|---|---|
CANARY_URL |
Actions variable | the canary's dashboard URL; unset disables the canary steps |
CANARY_TOKEN |
Actions secret | an API token bound to a dedicated service_account identity on the canary, carrying only the canary grant policy (see below) — never instance admin |
CANARY_WINDOW |
Actions variable | optional -window override for canary check; unset is 24h |
DEPLOY_TOKEN |
Actions secret | registry and repository write, for publishing, promoting and the release push; stays CI-only, promotion runs in CI and never on the host |
RELEASE_SIGNING_KEY |
Actions secret | the release signing seed |
Setting up CANARY_TOKEN once
CANARY_TOKEN is a bearer token for a dedicated service-account identity in
the system org whose only grant is the canary policy
(access.CanaryGrantPolicy): read the signal endpoints (GET /api/chat/update,
GET /api/alerts, GET /api/logs, GET /api/tasks), read and write the
update-settings resource, and trigger an update install
(POST /api/chat/update/install). It carries no other rights, and instance
admin is never needed. With an instance-admin token (for example the operator
session), run against the canary host:
# 1. Create the service-account identity in the system org.
curl -X POST "$CANARY_URL/api/identities" -H "Authorization: Bearer $ADMIN" \
-H 'Content-Type: application/json' \
-d '{"kind":"service_account","display_name":"ci-canary"}'
# -> {"id":"<canary-identity-id>", ...}
# 2. Issue its token (shown exactly once).
curl -X POST "$CANARY_URL/api/identities/<canary-identity-id>/tokens" \
-H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' \
-d '{"name":"ci-canary","days":90}'
# -> {"id":"<token-hash>","token":"archie_pat_..."} -- store the token value as the CANARY_TOKEN secret
# 3. Install the grant pair, scoped to that identity. The policies' org is
# pinned to the caller's org (org-sys for the instance admin), so the
# bodies need no org field. The permit alone is not enough: service
# accounts created through the API are assigned the developer role, so a
# matching forbid (installed second) negates everything outside the
# permit's surface for that identity.
curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \
-H 'Content-Type: application/json' \
-d '{"id":"org-canary-grant","level":"org","text":"permit(principal, action, resource) when {\n resource.org == principal.org &&\n principal == Archie::Identity::\"<canary-identity-id>\" &&\n (\n (action == Archie::Action::\"read\" && [\"dashboard\", \"task\"].contains(resource.kind)) ||\n (action == Archie::Action::\"read\" && resource == Archie::Object::\"update-settings\") ||\n (action == Archie::Action::\"update\" && resource == Archie::Object::\"update\") ||\n (action == Archie::Action::\"manage_identities\" && resource == Archie::Object::\"update-settings\")\n )\n};"}'
The second policy forbids the canary identity from everything the permit does not name, so its developer role cannot widen the grant:
curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \
-H 'Content-Type: application/json' \
-d '{"id":"org-canary-grant-forbid","level":"org","text":"forbid(principal, action, resource) when {\\n resource.org == principal.org &&\\n principal == Archie::Identity::\\\"<canary-identity-id>\\\" &&\\n !(\\n (action == Archie::Action::\\\"read\\\" && [\\\"dashboard\\\", \\\"task\\\"].contains(resource.kind)) ||\\n (action == Archie::Action::\\\"read\\\" && resource == Archie::Object::\\\"update-settings\\\") ||\\n (action == Archie::Action::\\\"update\\\" && resource == Archie::Object::\\\"update\\\") ||\\n (action == Archie::Action::\\\"manage_identities\\\" && resource == Archie::Object::\\\"update-settings\\\")\\n )\\n};"}'
The policy texts are exactly access.CanaryGrantPolicy and
access.CanaryGrantForbidPolicy from internal/domain/access/canary.go with
the identity id filled in. The canary needs no other rights: GET /health/detailed is unauthenticated, and every other call it makes is covered
by the grant pair.
The runner must reach CANARY_URL. Without a canary a release stays on next
until promoted by hand, under the same refusals:
archied release promote -version 1.57.0 \
-latest git.catlow.cloud/sam/archied,git.catlow.cloud/sam/archie-agent