Files
archie/RELEASING.md
T
sam 85b21578d8
Quality / check (push) Successful in 3m46s
deploy / gate (push) Successful in 4m29s
deploy / archied-image (push) Successful in 2m53s
deploy / agent-image (push) Successful in 12s
deploy / dist-zip (push) Skipped
ci(release): name the missing DEPLOY_TOKEN before checkout, and list what a mirror needs
Refs #1788
2026-11-05 00:33:18 +11:00

259 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Releasing
One release stream. `CHANGELOG.md` carries every release section, one generated
page per version describes it, and one `v*` tag names the release. One version
covers every process in the host bundle (`archied`, `archie-gateway`,
`archie-state-store`, `archie-ui`, `archie-messaging`) and the `archie-agent`
image; per-component detail is a labelled section inside the version
(`### archied`, `### archie-agent`).
A release is cut by CI, published to `next`, run on the canary for a day, and
promoted to `stable` only if the canary's signals pass. Nothing is run locally.
## Notes are written with the change
A commit that changes what a reader experiences adds its bullet under
`## [Unreleased]` in `CHANGELOG.md`, in the same commit. The release takes that
section as it stands, so write the bullet as the release note, not as a commit
subject.
**Release notes describe the reader's experience of what shipped.** The test is
the reader, not whether a sentence is negative:
- A capability that **never existed** leaves no expectation to correct, so a
note about it reads as a feature to someone who never had the thing: out.
- **The implementation's journey** (what was rebuilt, what a guard finally
covers, what a browser API refuses to surface) isn't the reader's experience
either: out.
- A capability that **shipped but isn't yet wired** _is_ the reader's
experience, so it belongs, worded as a caveat on the capability and paired
with the behaviour that compensates for it.
Design constraints and withheld capabilities are not release news. Deleting a
caveat a reader needs misleads them, which is the failure this rule exists to
prevent.
## Cutting a release
Dispatch the **release** workflow (`.github/workflows/release.yml`) with
`major`, `minor`, `patch` or an exact version. It runs `task release` on `main`:
1. refuses a dirty tree, an existing tag, or an `[Unreleased]` with no entries;
2. turns `[Unreleased]` into `## [x.y.z] - date` under a new empty
`[Unreleased]`, and runs `task news`;
3. commits `chore(release): vx.y.z` and tags `vx.y.z`;
then pushes `main` and the tag atomically with `DEPLOY_TOKEN`. A push made with
the job token wouldn't start other workflows, so the user token is what lets
the tag start the deploy.
Every workflow that reaches the registry or the canary reads its credentials
from Actions secrets, so a mirror of this repository carries its own set:
| Name | Kind | Used by |
|---|---|---|
| `DEPLOY_TOKEN` | secret | release, deploy, canary, promote — the registry push, the tag push, and the package links |
| `RELEASE_SIGNING_KEY` | secret | deploy — signs the release statement |
| `CANARY_TOKEN` | secret | deploy, canary, promote — the canary's dashboard API |
| `CANARY_URL` | variable | deploy, canary, promote — where that dashboard is |
| `CANARY_WINDOW` | variable | canary — the judged window; empty means the default |
A mirror missing them fails at its first login or push; `release.yml` names the
missing `DEPLOY_TOKEN` before it checks out, so the run doesn't read as a broken
release.
After a release, pull `main` before committing on it again: CI has added the
release commit.
## What a release publishes
The tag push runs `deploy.yml`:
- **Gate.** `task check TEST_FLAGS=-count=1` (uncached, including the
Docker-backed tests `-short` skips) and `govulncheck`, then a check that the
tag has a `[version]` section in `CHANGELOG.md`.
- **Images.** `archied:v<version>` and `archie-agent:v<version>` on a release
tag; `archied:edge` and `archied:sha-<commit>` on any other push to `main`.
`:latest` moves only on promotion, so it names the newest promoted release.
- **The signed release.** `archied release publish` signs a statement
(version, issued_at, zip digest, agent image by digest) with
`RELEASE_SIGNING_KEY` and pushes it with the distribution zip to
`git.catlow.cloud/sam/archie` as `v<version>` and `next`. It never moves
`stable`.
- **The release-notes page** on Gitea, from that version's `CHANGELOG.md`
section, which the statement links to.
- **The canary install**: `archied release canary start` points the canary at
`next` and installs the release through its dashboard.
GitHub-style Actions run the workflow from the tagged commit, so a fix to
`deploy.yml` reaches a release only if it is an ancestor of the tag. Re-cut the
release rather than committing the fix on top of a tagged one.
## The canary
The canary is the production host, following `next`. Every quarter hour
`canary.yml` runs `archied release canary check`, which reads the canary
through its dashboard API and judges **the release the canary actually runs**,
not the moving `next` tag:
| Signal | Source | Fails when |
| --- | --- | --- |
| Install outcome | `GET /api/chat/update` | the canary doesn't run the judged release (the watchdog rolled it back) |
| Readiness | `GET /health/detailed` | the rollup isn't `ok` |
| Alerts | `GET /api/alerts` | an open `action` alert was first raised after the release was installed |
| Error log | `GET /api/logs?level=error` | error lines per window exceed `2 × baseline + 10` |
| Task outcomes | `GET /api/tasks` | the share of finished tasks that parked or died rose more than 10 points |
- **Control.** The canary's own window before the release was installed. With
one production host there's no simultaneous control.
- **Window.** 24 hours counted from the **install** on the canary, not from
the release's issue time: the host records it in `installed.json`
(`committed_at`) and reports it as `installed_at` in `GET /api/chat/update`.
One cycle of schedules and traffic. The first 30 minutes are for the install
and restart. `canary.yml` passes `-window` when the `CANARY_WINDOW` Actions
variable is set; the flag defaults to 24h. A host that doesn't report an
install time degrades to the release statement's issue time.
- **Hard signals** (install, readiness, alerts) roll back at any check after
that. **Relative signals** (errors, tasks) are judged against the baseline
whenever both windows carry enough data, so a breach rolls the release back
even mid-window; a full window with no failure promotes it.
- **Too little data skips a signal** rather than failing it: fewer than 5
finished tasks in the window, or an error log that reaches back less than an
hour into the baseline. A log read that stops short of a window is scaled to
a full window from what it covers.
### Queueing
`deploy.yml` installs each new release on the canary with
`archied release canary start -version V`, which first points the canary back
at `next` (clearing any rollback pin) and then installs `V` — unless the
canary is already judging a release newer than stable that isn't `V`, in which
case `V` is **queued** behind it and the canary keeps running what it judges.
`-force` installs regardless. A `check` that promotes (or finds nothing to
judge) installs the newer `next` in the same run, so a queue drains one
release per promotion.
### Subcommands
```
archied release canary status read-only: channel/pin, installed release and
installed_at, stable/next, window progress, and
the verdict the current signals would give
archied release canary start point the canary at next and install -version,
queued behind a release being judged (-force to
override)
archied release canary check judge the installed release over -window
(default 24h): promote, pick up a newer next,
or roll back; a held canary is reported, and a
held pin that is no longer installable fails
the run red
archied release canary promote promote the installed release now if its
signals pass; refuses with exit 1 otherwise
archied release canary unpin return the canary to next from any pin
```
### Promote
A release that passes the window is promoted: `stable` is retagged to the
`v<version>` manifest. The signature covers the statement, not the tag, so the
retag needs no signing key. The `archied` and `archie-agent` images' `:latest`
move to their `:v<version>` first, and the agent image must be the digest the
statement pins. Promotion refuses a prerelease and a release issued before the
one `stable` names, the same replay rule hosts enforce. The decision and its
reasons are in the `canary` run log.
Early promotion is a dispatch away: the **promote** workflow
(`.github/workflows/promote.yml`) takes a `version` input and runs
`archied release canary promote -version V` on the canary. It judges the
release the canary runs — the window doesn't matter for the decision — and
refuses (red run) on any hard failure or relative breach with enough data.
### Roll back
`stable` stays put, and the run turns red. Before the canary is pinned to
`stable`, `check` verifies `stable` is actually installable — its statement's
zip blob must still be in the release repository and its pinned agent image
must still resolve (`releaseupdate.Installable`). If it isn't, the run fails
loudly and the canary stays on `next` rather than being stranded on a pin.
Otherwise the canary is pinned to the `stable` version through its update
settings and reinstalled through its dashboard.
A release whose migrations differ from `stable`'s can't go back without
restoring the pre-update database snapshot, which loses every write since. The
canary is held on the failed release instead (pinned to it) and the fix goes
forward in the next release. The next release's deploy points the canary back
at `next`.
A pinned canary isn't judged, so a rollback is decided once — but `check`
verifies the pin is still installable every run, so a stranded pin shows red
instead of green. `archied release canary unpin` returns the canary to `next`
by hand.
### Configuration
| Name | Kind | Holds |
| --- | --- | --- |
| `CANARY_URL` | Actions variable | the canary's dashboard URL; unset disables the canary steps |
| `CANARY_TOKEN` | Actions secret | an API token bound to a dedicated `service_account` identity on the canary, carrying only the canary grant policy (see below) — never instance admin |
| `CANARY_WINDOW` | Actions variable | optional `-window` override for `canary check`; unset is 24h |
| `DEPLOY_TOKEN` | Actions secret | registry and repository write, for publishing, promoting and the release push; stays CI-only, promotion runs in CI and never on the host |
| `RELEASE_SIGNING_KEY` | Actions secret | the release signing seed |
#### Setting up `CANARY_TOKEN` once
`CANARY_TOKEN` is a bearer token for a dedicated service-account identity in
the system org whose only grant is the canary policy
(`access.CanaryGrantPolicy`): read the signal endpoints (`GET /api/chat/update`,
`GET /api/alerts`, `GET /api/logs`, `GET /api/tasks`), read and write the
`update-settings` resource, and trigger an update install
(`POST /api/chat/update/install`). It carries no other rights, and instance
admin is never needed. With an instance-admin token (for example the operator
session), run against the canary host:
```bash
# 1. Create the service-account identity in the system org.
curl -X POST "$CANARY_URL/api/identities" -H "Authorization: Bearer $ADMIN" \
-H 'Content-Type: application/json' \
-d '{"kind":"service_account","display_name":"ci-canary"}'
# -> {"id":"<canary-identity-id>", ...}
# 2. Issue its token (shown exactly once).
curl -X POST "$CANARY_URL/api/identities/<canary-identity-id>/tokens" \
-H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' \
-d '{"name":"ci-canary","days":90}'
# -> {"id":"<token-hash>","token":"archie_pat_..."} -- store the token value as the CANARY_TOKEN secret
# 3. Install the grant pair, scoped to that identity. The policies' org is
# pinned to the caller's org (org-sys for the instance admin), so the
# bodies need no org field. The permit alone is not enough: service
# accounts created through the API are assigned the developer role, so a
# matching forbid (installed second) negates everything outside the
# permit's surface for that identity.
curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \
-H 'Content-Type: application/json' \
-d '{"id":"org-canary-grant","level":"org","text":"permit(principal, action, resource) when {\n resource.org == principal.org &&\n principal == Archie::Identity::\"<canary-identity-id>\" &&\n (\n (action == Archie::Action::\"read\" && [\"dashboard\", \"task\"].contains(resource.kind)) ||\n (action == Archie::Action::\"read\" && resource == Archie::Object::\"update-settings\") ||\n (action == Archie::Action::\"update\" && resource == Archie::Object::\"update\") ||\n (action == Archie::Action::\"manage_identities\" && resource == Archie::Object::\"update-settings\")\n )\n};"}'
```
The second policy forbids the canary identity from everything the permit does
not name, so its developer role cannot widen the grant:
```bash
curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \
-H 'Content-Type: application/json' \
-d '{"id":"org-canary-grant-forbid","level":"org","text":"forbid(principal, action, resource) when {\\n resource.org == principal.org &&\\n principal == Archie::Identity::\\\"<canary-identity-id>\\\" &&\\n !(\\n (action == Archie::Action::\\\"read\\\" && [\\\"dashboard\\\", \\\"task\\\"].contains(resource.kind)) ||\\n (action == Archie::Action::\\\"read\\\" && resource == Archie::Object::\\\"update-settings\\\") ||\\n (action == Archie::Action::\\\"update\\\" && resource == Archie::Object::\\\"update\\\") ||\\n (action == Archie::Action::\\\"manage_identities\\\" && resource == Archie::Object::\\\"update-settings\\\")\\n )\\n};"}'
```
The policy texts are exactly `access.CanaryGrantPolicy` and
`access.CanaryGrantForbidPolicy` from `internal/domain/access/canary.go` with
the identity id filled in. The canary needs no other rights: `GET
/health/detailed` is unauthenticated, and every other call it makes is covered
by the grant pair.
The runner must reach `CANARY_URL`. Without a canary a release stays on `next`
until promoted by hand, under the same refusals:
```bash
archied release promote -version 1.57.0 \
-latest git.catlow.cloud/sam/archied,git.catlow.cloud/sam/archie-agent
```