259 lines
14 KiB
Markdown
259 lines
14 KiB
Markdown
# Releasing
|
||
|
||
One release stream. `CHANGELOG.md` carries every release section, one generated
|
||
page per version describes it, and one `v*` tag names the release. One version
|
||
covers every process in the host bundle (`archied`, `archie-gateway`,
|
||
`archie-state-store`, `archie-ui`, `archie-messaging`) and the `archie-agent`
|
||
image; per-component detail is a labelled section inside the version
|
||
(`### archied`, `### archie-agent`).
|
||
|
||
A release is cut by CI, published to `next`, run on the canary for a day, and
|
||
promoted to `stable` only if the canary's signals pass. Nothing is run locally.
|
||
|
||
## Notes are written with the change
|
||
|
||
A commit that changes what a reader experiences adds its bullet under
|
||
`## [Unreleased]` in `CHANGELOG.md`, in the same commit. The release takes that
|
||
section as it stands, so write the bullet as the release note, not as a commit
|
||
subject.
|
||
|
||
**Release notes describe the reader's experience of what shipped.** The test is
|
||
the reader, not whether a sentence is negative:
|
||
|
||
- A capability that **never existed** leaves no expectation to correct, so a
|
||
note about it reads as a feature to someone who never had the thing: out.
|
||
- **The implementation's journey** (what was rebuilt, what a guard finally
|
||
covers, what a browser API refuses to surface) isn't the reader's experience
|
||
either: out.
|
||
- A capability that **shipped but isn't yet wired** _is_ the reader's
|
||
experience, so it belongs, worded as a caveat on the capability and paired
|
||
with the behaviour that compensates for it.
|
||
|
||
Design constraints and withheld capabilities are not release news. Deleting a
|
||
caveat a reader needs misleads them, which is the failure this rule exists to
|
||
prevent.
|
||
|
||
## Cutting a release
|
||
|
||
Dispatch the **release** workflow (`.github/workflows/release.yml`) with
|
||
`major`, `minor`, `patch` or an exact version. It runs `task release` on `main`:
|
||
|
||
1. refuses a dirty tree, an existing tag, or an `[Unreleased]` with no entries;
|
||
2. turns `[Unreleased]` into `## [x.y.z] - date` under a new empty
|
||
`[Unreleased]`, and runs `task news`;
|
||
3. commits `chore(release): vx.y.z` and tags `vx.y.z`;
|
||
|
||
then pushes `main` and the tag atomically with `DEPLOY_TOKEN`. A push made with
|
||
the job token wouldn't start other workflows, so the user token is what lets
|
||
the tag start the deploy.
|
||
|
||
Every workflow that reaches the registry or the canary reads its credentials
|
||
from Actions secrets, so a mirror of this repository carries its own set:
|
||
|
||
| Name | Kind | Used by |
|
||
|---|---|---|
|
||
| `DEPLOY_TOKEN` | secret | release, deploy, canary, promote — the registry push, the tag push, and the package links |
|
||
| `RELEASE_SIGNING_KEY` | secret | deploy — signs the release statement |
|
||
| `CANARY_TOKEN` | secret | deploy, canary, promote — the canary's dashboard API |
|
||
| `CANARY_URL` | variable | deploy, canary, promote — where that dashboard is |
|
||
| `CANARY_WINDOW` | variable | canary — the judged window; empty means the default |
|
||
|
||
A mirror missing them fails at its first login or push; `release.yml` names the
|
||
missing `DEPLOY_TOKEN` before it checks out, so the run doesn't read as a broken
|
||
release.
|
||
|
||
After a release, pull `main` before committing on it again: CI has added the
|
||
release commit.
|
||
|
||
## What a release publishes
|
||
|
||
The tag push runs `deploy.yml`:
|
||
|
||
- **Gate.** `task check TEST_FLAGS=-count=1` (uncached, including the
|
||
Docker-backed tests `-short` skips) and `govulncheck`, then a check that the
|
||
tag has a `[version]` section in `CHANGELOG.md`.
|
||
- **Images.** `archied:v<version>` and `archie-agent:v<version>` on a release
|
||
tag; `archied:edge` and `archied:sha-<commit>` on any other push to `main`.
|
||
`:latest` moves only on promotion, so it names the newest promoted release.
|
||
- **The signed release.** `archied release publish` signs a statement
|
||
(version, issued_at, zip digest, agent image by digest) with
|
||
`RELEASE_SIGNING_KEY` and pushes it with the distribution zip to
|
||
`git.catlow.cloud/sam/archie` as `v<version>` and `next`. It never moves
|
||
`stable`.
|
||
- **The release-notes page** on Gitea, from that version's `CHANGELOG.md`
|
||
section, which the statement links to.
|
||
- **The canary install**: `archied release canary start` points the canary at
|
||
`next` and installs the release through its dashboard.
|
||
|
||
GitHub-style Actions run the workflow from the tagged commit, so a fix to
|
||
`deploy.yml` reaches a release only if it is an ancestor of the tag. Re-cut the
|
||
release rather than committing the fix on top of a tagged one.
|
||
|
||
## The canary
|
||
|
||
The canary is the production host, following `next`. Every quarter hour
|
||
`canary.yml` runs `archied release canary check`, which reads the canary
|
||
through its dashboard API and judges **the release the canary actually runs**,
|
||
not the moving `next` tag:
|
||
|
||
| Signal | Source | Fails when |
|
||
| --- | --- | --- |
|
||
| Install outcome | `GET /api/chat/update` | the canary doesn't run the judged release (the watchdog rolled it back) |
|
||
| Readiness | `GET /health/detailed` | the rollup isn't `ok` |
|
||
| Alerts | `GET /api/alerts` | an open `action` alert was first raised after the release was installed |
|
||
| Error log | `GET /api/logs?level=error` | error lines per window exceed `2 × baseline + 10` |
|
||
| Task outcomes | `GET /api/tasks` | the share of finished tasks that parked or died rose more than 10 points |
|
||
|
||
- **Control.** The canary's own window before the release was installed. With
|
||
one production host there's no simultaneous control.
|
||
- **Window.** 24 hours counted from the **install** on the canary, not from
|
||
the release's issue time: the host records it in `installed.json`
|
||
(`committed_at`) and reports it as `installed_at` in `GET /api/chat/update`.
|
||
One cycle of schedules and traffic. The first 30 minutes are for the install
|
||
and restart. `canary.yml` passes `-window` when the `CANARY_WINDOW` Actions
|
||
variable is set; the flag defaults to 24h. A host that doesn't report an
|
||
install time degrades to the release statement's issue time.
|
||
- **Hard signals** (install, readiness, alerts) roll back at any check after
|
||
that. **Relative signals** (errors, tasks) are judged against the baseline
|
||
whenever both windows carry enough data, so a breach rolls the release back
|
||
even mid-window; a full window with no failure promotes it.
|
||
- **Too little data skips a signal** rather than failing it: fewer than 5
|
||
finished tasks in the window, or an error log that reaches back less than an
|
||
hour into the baseline. A log read that stops short of a window is scaled to
|
||
a full window from what it covers.
|
||
|
||
### Queueing
|
||
|
||
`deploy.yml` installs each new release on the canary with
|
||
`archied release canary start -version V`, which first points the canary back
|
||
at `next` (clearing any rollback pin) and then installs `V` — unless the
|
||
canary is already judging a release newer than stable that isn't `V`, in which
|
||
case `V` is **queued** behind it and the canary keeps running what it judges.
|
||
`-force` installs regardless. A `check` that promotes (or finds nothing to
|
||
judge) installs the newer `next` in the same run, so a queue drains one
|
||
release per promotion.
|
||
|
||
### Subcommands
|
||
|
||
```
|
||
archied release canary status read-only: channel/pin, installed release and
|
||
installed_at, stable/next, window progress, and
|
||
the verdict the current signals would give
|
||
archied release canary start point the canary at next and install -version,
|
||
queued behind a release being judged (-force to
|
||
override)
|
||
archied release canary check judge the installed release over -window
|
||
(default 24h): promote, pick up a newer next,
|
||
or roll back; a held canary is reported, and a
|
||
held pin that is no longer installable fails
|
||
the run red
|
||
archied release canary promote promote the installed release now if its
|
||
signals pass; refuses with exit 1 otherwise
|
||
archied release canary unpin return the canary to next from any pin
|
||
```
|
||
|
||
### Promote
|
||
|
||
A release that passes the window is promoted: `stable` is retagged to the
|
||
`v<version>` manifest. The signature covers the statement, not the tag, so the
|
||
retag needs no signing key. The `archied` and `archie-agent` images' `:latest`
|
||
move to their `:v<version>` first, and the agent image must be the digest the
|
||
statement pins. Promotion refuses a prerelease and a release issued before the
|
||
one `stable` names, the same replay rule hosts enforce. The decision and its
|
||
reasons are in the `canary` run log.
|
||
|
||
Early promotion is a dispatch away: the **promote** workflow
|
||
(`.github/workflows/promote.yml`) takes a `version` input and runs
|
||
`archied release canary promote -version V` on the canary. It judges the
|
||
release the canary runs — the window doesn't matter for the decision — and
|
||
refuses (red run) on any hard failure or relative breach with enough data.
|
||
|
||
### Roll back
|
||
|
||
`stable` stays put, and the run turns red. Before the canary is pinned to
|
||
`stable`, `check` verifies `stable` is actually installable — its statement's
|
||
zip blob must still be in the release repository and its pinned agent image
|
||
must still resolve (`releaseupdate.Installable`). If it isn't, the run fails
|
||
loudly and the canary stays on `next` rather than being stranded on a pin.
|
||
Otherwise the canary is pinned to the `stable` version through its update
|
||
settings and reinstalled through its dashboard.
|
||
|
||
A release whose migrations differ from `stable`'s can't go back without
|
||
restoring the pre-update database snapshot, which loses every write since. The
|
||
canary is held on the failed release instead (pinned to it) and the fix goes
|
||
forward in the next release. The next release's deploy points the canary back
|
||
at `next`.
|
||
|
||
A pinned canary isn't judged, so a rollback is decided once — but `check`
|
||
verifies the pin is still installable every run, so a stranded pin shows red
|
||
instead of green. `archied release canary unpin` returns the canary to `next`
|
||
by hand.
|
||
|
||
### Configuration
|
||
|
||
| Name | Kind | Holds |
|
||
| --- | --- | --- |
|
||
| `CANARY_URL` | Actions variable | the canary's dashboard URL; unset disables the canary steps |
|
||
| `CANARY_TOKEN` | Actions secret | an API token bound to a dedicated `service_account` identity on the canary, carrying only the canary grant policy (see below) — never instance admin |
|
||
| `CANARY_WINDOW` | Actions variable | optional `-window` override for `canary check`; unset is 24h |
|
||
| `DEPLOY_TOKEN` | Actions secret | registry and repository write, for publishing, promoting and the release push; stays CI-only, promotion runs in CI and never on the host |
|
||
| `RELEASE_SIGNING_KEY` | Actions secret | the release signing seed |
|
||
|
||
#### Setting up `CANARY_TOKEN` once
|
||
|
||
`CANARY_TOKEN` is a bearer token for a dedicated service-account identity in
|
||
the system org whose only grant is the canary policy
|
||
(`access.CanaryGrantPolicy`): read the signal endpoints (`GET /api/chat/update`,
|
||
`GET /api/alerts`, `GET /api/logs`, `GET /api/tasks`), read and write the
|
||
`update-settings` resource, and trigger an update install
|
||
(`POST /api/chat/update/install`). It carries no other rights, and instance
|
||
admin is never needed. With an instance-admin token (for example the operator
|
||
session), run against the canary host:
|
||
|
||
```bash
|
||
# 1. Create the service-account identity in the system org.
|
||
curl -X POST "$CANARY_URL/api/identities" -H "Authorization: Bearer $ADMIN" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"kind":"service_account","display_name":"ci-canary"}'
|
||
# -> {"id":"<canary-identity-id>", ...}
|
||
|
||
# 2. Issue its token (shown exactly once).
|
||
curl -X POST "$CANARY_URL/api/identities/<canary-identity-id>/tokens" \
|
||
-H "Authorization: Bearer $ADMIN" -H 'Content-Type: application/json' \
|
||
-d '{"name":"ci-canary","days":90}'
|
||
# -> {"id":"<token-hash>","token":"archie_pat_..."} -- store the token value as the CANARY_TOKEN secret
|
||
|
||
# 3. Install the grant pair, scoped to that identity. The policies' org is
|
||
# pinned to the caller's org (org-sys for the instance admin), so the
|
||
# bodies need no org field. The permit alone is not enough: service
|
||
# accounts created through the API are assigned the developer role, so a
|
||
# matching forbid (installed second) negates everything outside the
|
||
# permit's surface for that identity.
|
||
curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"id":"org-canary-grant","level":"org","text":"permit(principal, action, resource) when {\n resource.org == principal.org &&\n principal == Archie::Identity::\"<canary-identity-id>\" &&\n (\n (action == Archie::Action::\"read\" && [\"dashboard\", \"task\"].contains(resource.kind)) ||\n (action == Archie::Action::\"read\" && resource == Archie::Object::\"update-settings\") ||\n (action == Archie::Action::\"update\" && resource == Archie::Object::\"update\") ||\n (action == Archie::Action::\"manage_identities\" && resource == Archie::Object::\"update-settings\")\n )\n};"}'
|
||
```
|
||
|
||
The second policy forbids the canary identity from everything the permit does
|
||
not name, so its developer role cannot widen the grant:
|
||
|
||
```bash
|
||
curl -X PUT "$CANARY_URL/api/access/policies" -H "Authorization: Bearer $ADMIN" \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"id":"org-canary-grant-forbid","level":"org","text":"forbid(principal, action, resource) when {\\n resource.org == principal.org &&\\n principal == Archie::Identity::\\\"<canary-identity-id>\\\" &&\\n !(\\n (action == Archie::Action::\\\"read\\\" && [\\\"dashboard\\\", \\\"task\\\"].contains(resource.kind)) ||\\n (action == Archie::Action::\\\"read\\\" && resource == Archie::Object::\\\"update-settings\\\") ||\\n (action == Archie::Action::\\\"update\\\" && resource == Archie::Object::\\\"update\\\") ||\\n (action == Archie::Action::\\\"manage_identities\\\" && resource == Archie::Object::\\\"update-settings\\\")\\n )\\n};"}'
|
||
```
|
||
|
||
The policy texts are exactly `access.CanaryGrantPolicy` and
|
||
`access.CanaryGrantForbidPolicy` from `internal/domain/access/canary.go` with
|
||
the identity id filled in. The canary needs no other rights: `GET
|
||
/health/detailed` is unauthenticated, and every other call it makes is covered
|
||
by the grant pair.
|
||
|
||
The runner must reach `CANARY_URL`. Without a canary a release stays on `next`
|
||
until promoted by hand, under the same refusals:
|
||
|
||
```bash
|
||
archied release promote -version 1.57.0 \
|
||
-latest git.catlow.cloud/sam/archied,git.catlow.cloud/sam/archie-agent
|
||
```
|