The canary (see RELEASING.md) judges a release for a fixed 24 hours measured from the statement's issued_at. Three things follow from that:
Promotion happens 24 hours after whenever the release was cut, so the maintainer's testing window is set by the clock, not by when they're available.
The canary is the only production host, so a new release can't be tested there without replacing the one being judged. Cutting a release mid-window abandons the pending promotion and restarts the clock.
Releasing more often than daily means stable never moves.
Done when
The maintainer can put a new release on the canary and test it at any time without losing a release that's already passed its hard checks.
Promotion doesn't depend on waiting out a fixed wall-clock window from publish. Options to weigh: count the window from install rather than issue, shorten it with relative signals judged on whatever data exists, and let the operator promote from the dashboard once satisfied.
A newer release on next doesn't starve stable.
Plan (approved)
Design around the maintainer's loop: window from install on the host, judge the
installed release (not the moving next), new releases queue instead of
restarting the clock, promotion possible as soon as the maintainer is satisfied,
rollback that can't strand the host, held state that is loud. Full plan in the
session notes; staging:
1. Install-time window (committed_at on the update journal, exposed via the update snapshot) + judge-installed via exact-pin tag + queue-safe canary start + archied release canary status
2. Loud held/pinned state in canary check + rollback pre-flight (stable statement, zip digest, agent image digest verified before pinning) + canary unpin for CLI recovery
3. Early promote (promote workflow_dispatch + canary promote) + configurable window/baseline + progressive relative-signal judging with symmetric minimum data
## What's wrong
The canary (see RELEASING.md) judges a release for a fixed 24 hours measured from the statement's `issued_at`. Three things follow from that:
- Promotion happens 24 hours after whenever the release was cut, so the maintainer's testing window is set by the clock, not by when they're available.
- The canary is the only production host, so a new release can't be tested there without replacing the one being judged. Cutting a release mid-window abandons the pending promotion and restarts the clock.
- Releasing more often than daily means `stable` never moves.
## Done when
- The maintainer can put a new release on the canary and test it at any time without losing a release that's already passed its hard checks.
- Promotion doesn't depend on waiting out a fixed wall-clock window from publish. Options to weigh: count the window from install rather than issue, shorten it with relative signals judged on whatever data exists, and let the operator promote from the dashboard once satisfied.
- A newer release on `next` doesn't starve `stable`.
## Plan (approved)
Design around the maintainer's loop: window from install on the host, judge the
installed release (not the moving `next`), new releases queue instead of
restarting the clock, promotion possible as soon as the maintainer is satisfied,
rollback that can't strand the host, held state that is loud. Full plan in the
session notes; staging:
- [x] 1. Install-time window (`committed_at` on the update journal, exposed via the update snapshot) + judge-installed via exact-pin tag + queue-safe `canary start` + `archied release canary status`
- [x] 2. Loud held/pinned state in `canary check` + rollback pre-flight (stable statement, zip digest, agent image digest verified before pinning) + `canary unpin` for CLI recovery
- [x] 3. Early promote (`promote` workflow_dispatch + `canary promote`) + configurable window/baseline + progressive relative-signal judging with symmetric minimum data
- [x] 4. Dashboard canary card + `task canary:rehearse` local rehearsal (registry:2 + e2e host)
- [x] 5. Follow-up issues filed: #1726 (host install script restarts), #1727 (scoped canary grant)
A rollback pins the canary to the stable version, and nothing unpins it. If stable can't be installed (no zip for it, as with 1.56.0), the host stays pinned and every later check skips it as held, so it looks green while judging nothing.
The rollback decision and the pin are made before checking that the stable release is installable at all.
The window judges whatever is running, not what was installed: 1.57.0's window ran mostly on 1.55 processes because install.sh didn't restart the units, and its error and task baselines were mostly measuring 1.55.
The control is the same host's previous 24 hours, so a release cut after a quiet day (a weekend, say) is compared against a skewed baseline.
The canary token needs instance admin on the production host, and that token sits in CI.
Recovering any of this needs the maintainer in the dashboard (unpinning, setting the channel back to next). There's no command or workflow to do it.
More problems found today, beyond the timing:
- A rollback pins the canary to the stable version, and nothing unpins it. If stable can't be installed (no zip for it, as with 1.56.0), the host stays pinned and every later check skips it as held, so it looks green while judging nothing.
- The rollback decision and the pin are made before checking that the stable release is installable at all.
- The window judges whatever is running, not what was installed: 1.57.0's window ran mostly on 1.55 processes because install.sh didn't restart the units, and its error and task baselines were mostly measuring 1.55.
- The control is the same host's previous 24 hours, so a release cut after a quiet day (a weekend, say) is compared against a skewed baseline.
- The canary token needs instance admin on the production host, and that token sits in CI.
- Recovering any of this needs the maintainer in the dashboard (unpinning, setting the channel back to next). There's no command or workflow to do it.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What's wrong
The canary (see RELEASING.md) judges a release for a fixed 24 hours measured from the statement's
issued_at. Three things follow from that:stablenever moves.Done when
nextdoesn't starvestable.Plan (approved)
Design around the maintainer's loop: window from install on the host, judge the
installed release (not the moving
next), new releases queue instead ofrestarting the clock, promotion possible as soon as the maintainer is satisfied,
rollback that can't strand the host, held state that is loud. Full plan in the
session notes; staging:
committed_aton the update journal, exposed via the update snapshot) + judge-installed via exact-pin tag + queue-safecanary start+archied release canary statuscanary check+ rollback pre-flight (stable statement, zip digest, agent image digest verified before pinning) +canary unpinfor CLI recoverypromoteworkflow_dispatch +canary promote) + configurable window/baseline + progressive relative-signal judging with symmetric minimum datatask canary:rehearselocal rehearsal (registry:2 + e2e host)More problems found today, beyond the timing: