# Releasing One release stream. `CHANGELOG.md` carries every release section, one generated page per version describes it, and one `v*` tag names the release. One version covers every process in the host bundle (`archied`, `archie-gateway`, `archie-state-store`, `archie-ui`, `archie-messaging`) and the `archie-agent` image; per-component detail is a labelled section inside the version (`### archied`, `### archie-agent`). A release is cut by CI, published to `next`, run on the canary for a day, and promoted to `stable` only if the canary's signals pass. Nothing is run locally. ## Notes are written with the change A commit that changes what a reader experiences adds its bullet under `## [Unreleased]` in `CHANGELOG.md`, in the same commit. The release takes that section as it stands, so write the bullet as the release note, not as a commit subject. **Release notes describe the reader's experience of what shipped.** The test is the reader, not whether a sentence is negative: - A capability that **never existed** leaves no expectation to correct, so a note about it reads as a feature to someone who never had the thing: out. - **The implementation's journey** (what was rebuilt, what a guard finally covers, what a browser API refuses to surface) isn't the reader's experience either: out. - A capability that **shipped but isn't yet wired** _is_ the reader's experience, so it belongs, worded as a caveat on the capability and paired with the behaviour that compensates for it. Design constraints and withheld capabilities are not release news. Deleting a caveat a reader needs misleads them, which is the failure this rule exists to prevent. ## Cutting a release Dispatch the **release** workflow (`.github/workflows/release.yml`) with `major`, `minor`, `patch` or an exact version. It runs `task release` on `main`: 1. refuses a dirty tree, an existing tag, or an `[Unreleased]` with no entries; 2. turns `[Unreleased]` into `## [x.y.z] - date` under a new empty `[Unreleased]`, and runs `task news`; 3. commits `chore(release): vx.y.z` and tags `vx.y.z`; then pushes `main` and the tag atomically with `DEPLOY_TOKEN`. A push made with the job token wouldn't start other workflows, so the user token is what lets the tag start the deploy. Every workflow that reaches the registry or the canary reads its credentials from Actions secrets, so a mirror of this repository carries its own set: | Name | Kind | Used by | |---|---|---| | `DEPLOY_TOKEN` | secret | release, deploy, canary, promote — the registry push, the tag push, and the package links | | `RELEASE_SIGNING_KEY` | secret | deploy — signs the release statement | | `CANARY_TOKEN` | secret | deploy, canary, promote — the canary's dashboard API | | `CANARY_URL` | variable | deploy, canary, promote — where that dashboard is | | `CANARY_WINDOW` | variable | canary — the judged window; empty means the default | A mirror missing them fails at its first login or push; `release.yml` names the missing `DEPLOY_TOKEN` before it checks out, so the run doesn't read as a broken release. After a release, pull `main` before committing on it again: CI has added the release commit. ## What a release publishes The tag push runs `deploy.yml`: - **Gate.** `task check TEST_FLAGS=-count=1` (uncached, including the Docker-backed tests `-short` skips) and `govulncheck`, then a check that the tag has a `[version]` section in `CHANGELOG.md`. - **Images.** `archied:v` and `archie-agent:v` on a release tag; `archied:edge` and `archied:sha-` on any other push to `main`. `:latest` moves only on promotion, so it names the newest promoted release. - **The signed release.** `archied release publish` signs a statement (version, issued_at, zip digest, agent image by digest) with `RELEASE_SIGNING_KEY` and pushes it with the distribution zip to `git.catlow.cloud/sam/archie` as `v` and `next`. It never moves `stable`. - **The release-notes page** on Gitea, from that version's `CHANGELOG.md` section, which the statement links to. - **The canary install**: `archied release canary start` points the canary at `next` and installs the release through its dashboard. GitHub-style Actions run the workflow from the tagged commit, so a fix to `deploy.yml` reaches a release only if it is an ancestor of the tag. Re-cut the release rather than committing the fix on top of a tagged one. ## The canary The canary is the production host, following `next`. Every quarter hour `canary.yml` runs `archied release canary check`, which reads the canary through its dashboard API and judges **the release the canary actually runs**, not the moving `next` tag: | Signal | Source | Fails when | | --- | --- | --- | | Install outcome | `GET /api/chat/update` | the canary doesn't run the judged release (the watchdog rolled it back) | | Readiness | `GET /health/detailed` | the rollup isn't `ok` | | Alerts | `GET /api/alerts` | an open `action` alert was first raised after the release was installed | | Error log | `GET /api/logs?level=error` | error lines per window exceed `2 × baseline + 10` | | Task outcomes | `GET /api/tasks` | the share of finished tasks that parked or died rose more than 10 points | - **Control.** The canary's own window before the release was installed. With one production host there's no simultaneous control. - **Window.** 24 hours counted from the **install** on the canary, not from the release's issue time: the host records it in `installed.json` (`committed_at`) and reports it as `installed_at` in `GET /api/chat/update`. One cycle of schedules and traffic. The first 30 minutes are for the install and restart. `canary.yml` passes `-window` when the `CANARY_WINDOW` Actions variable is set; the flag defaults to 24h. A host that doesn't report an install time degrades to the release statement's issue time. - **Hard signals** (install, readiness, alerts) roll back at any check after that. **Relative signals** (errors, tasks) are judged against the baseline whenever both windows carry enough data, so a breach rolls the release back even mid-window; a full window with no failure promotes it. - **Too little data skips a signal** rather than failing it: fewer than 5 finished tasks in the window, or an error log that reaches back less than an hour into the baseline. A log read that stops short of a window is scaled to a full window from what it covers. ### Queueing `deploy.yml` installs each new release on the canary with `archied release canary start -version V`, which first points the canary back at `next` (clearing any rollback pin) and then installs `V` — unless the canary is already judging a release newer than stable that isn't `V`, in which case `V` is **queued** behind it and the canary keeps running what it judges. `-force` installs regardless. A `check` that promotes (or finds nothing to judge) installs the newer `next` in the same run, so a queue drains one release per promotion. ### Subcommands ``` archied release canary status read-only: channel/pin, installed release and installed_at, stable/next, window progress, and the verdict the current signals would give archied release canary start point the canary at next and install -version, queued behind a release being judged (-force to override) archied release canary check judge the installed release over -window (default 24h): promote, pick up a newer next, or roll back; a held canary is reported, and a held pin that is no longer installable fails the run red archied release canary promote promote the installed release now if its signals pass; refuses with exit 1 otherwise archied release canary unpin return the canary to next from any pin ``` ### Promote A release that passes the window is promoted: `stable` is retagged to the `v` manifest. The signature covers the statement, not the tag, so the retag needs no signing key. The `archied` and `archie-agent` images' `:latest` move to their `:v` first, and the agent image must be the digest the statement pins. Promotion refuses a prerelease and a release issued before the one `stable` names, the same replay rule hosts enforce. The decision and its reasons are in the `canary` run log. Early promotion is a dispatch away: the **promote** workflow (`.github/workflows/promote.yml`) takes a `version` input and runs `archied release canary promote -version V` on the canary. It judges the release the canary runs — the window doesn't matter for the decision — and refuses (red run) on any hard failure or relative breach with enough data. ### Roll back `stable` stays put, and the run turns red. Before the canary is pinned to `stable`, `check` verifies `stable` is actually installable — its statement's zip blob must still be in the release repository and its pinned agent image must still resolve (`releaseupdate.Installable`). If it isn't, the run fails loudly and the canary stays on `next` rather than being stranded on a pin. Otherwise the canary is pinned to the `stable` version through its update settings and reinstalled through its dashboard. A release whose migrations differ from `stable`'s can't go back without restoring the pre-update database snapshot, which loses every write since. The canary is held on the failed release instead (pinned to it) and the fix goes forward in the next release. The next release's deploy points the canary back at `next`. A pinned canary isn't judged, so a rollback is decided once — but `check` verifies the pin is still installable every run, so a stranded pin shows red instead of green. `archied release canary unpin` returns the canary to `next` by hand. ### Configuration | Name | Kind | Holds | | --- | --- | --- | | `CANARY_URL` | Actions variable | the canary's dashboard URL; unset disables the canary steps | | `CANARY_TOKEN` | Actions secret | an API token bound to a dedicated `service_account` identity on the canary, carrying only the canary grant policy (see below) — never instance admin | | `CANARY_WINDOW` | Actions variable | optional `-window` override for `canary check`; unset is 24h | | `DEPLOY_TOKEN` | Actions secret | registry and repository write, for publishing, promoting and the release push; stays CI-only, promotion runs in CI and never on the host | | `RELEASE_SIGNING_KEY` | Actions secret | the release signing seed | #### Setting up `CANARY_TOKEN` once `CANARY_TOKEN` is a bearer token for a dedicated service-account identity in the system org whose only grant is the canary policy (`access.CanaryGrantPolicy`): read the signal endpoints (`GET /api/chat/update`, `GET /api/alerts`, `GET /api/logs`, `GET /api/tasks`), read and write the `update-settings` resource, and trigger an update install (`POST /api/chat/update/install`). It carries no other rights, and instance admin is never needed. With an instance-admin token (for example the operator session), run against the canary host: ```bash # 1. Create the service-account identity in the system org. curl -X POST "$CANARY_URL/api/identities" -H "Authorization: Bearer $ADMIN" \ -H 'Content-Type: application/json' \ -d '{"kind":"service_account","display_name":"ci-canary"}' # -> {"id":"", ...} # 2. Issue its token (shown exactly once). curl -X POST "$CANARY_URL/api/identities//tokens" \ -H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' \ -d '{"name":"ci-canary","days":90}' # -> {"id":"","token":"archie_pat_..."} -- store the token value as the CANARY_TOKEN secret # 3. Install the grant pair, scoped to that identity. The policies' org is # pinned to the caller's org (org-sys for the instance admin), so the # bodies need no org field. The permit alone is not enough: service # accounts created through the API are assigned the developer role, so a # matching forbid (installed second) negates everything outside the # permit's surface for that identity. curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \ -H 'Content-Type: application/json' \ -d '{"id":"org-canary-grant","level":"org","text":"permit(principal, action, resource) when {\n resource.org == principal.org &&\n principal == Archie::Identity::\"\" &&\n (\n (action == Archie::Action::\"read\" && [\"dashboard\", \"task\"].contains(resource.kind)) ||\n (action == Archie::Action::\"read\" && resource == Archie::Object::\"update-settings\") ||\n (action == Archie::Action::\"update\" && resource == Archie::Object::\"update\") ||\n (action == Archie::Action::\"manage_identities\" && resource == Archie::Object::\"update-settings\")\n )\n};"}' ``` The second policy forbids the canary identity from everything the permit does not name, so its developer role cannot widen the grant: ```bash curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \ -H 'Content-Type: application/json' \ -d '{"id":"org-canary-grant-forbid","level":"org","text":"forbid(principal, action, resource) when {\\n resource.org == principal.org &&\\n principal == Archie::Identity::\\\"\\\" &&\\n !(\\n (action == Archie::Action::\\\"read\\\" && [\\\"dashboard\\\", \\\"task\\\"].contains(resource.kind)) ||\\n (action == Archie::Action::\\\"read\\\" && resource == Archie::Object::\\\"update-settings\\\") ||\\n (action == Archie::Action::\\\"update\\\" && resource == Archie::Object::\\\"update\\\") ||\\n (action == Archie::Action::\\\"manage_identities\\\" && resource == Archie::Object::\\\"update-settings\\\")\\n )\\n};"}' ``` The policy texts are exactly `access.CanaryGrantPolicy` and `access.CanaryGrantForbidPolicy` from `internal/domain/access/canary.go` with the identity id filled in. The canary needs no other rights: `GET /health/detailed` is unauthenticated, and every other call it makes is covered by the grant pair. The runner must reach `CANARY_URL`. Without a canary a release stays on `next` until promoted by hand, under the same refusals: ```bash archied release promote -version 1.57.0 \ -latest git.catlow.cloud/sam/archied,git.catlow.cloud/sam/archie-agent ```