Update watchdog hangs forever when the new release changes the service transport #1455

Closed
opened 2026-10-27 13:40:28 +00:00 by sam · 0 comments
Owner

The watchdog is a frozen copy of the installing (old) archied, so it speaks the old release's protocol to the new State Store. Upgrading 1.53 to 1.54.1 hung in phase switching: the 1.53 watchdog dialed the TLS-only State Store in plaintext with wait-for-ready, and the image-policy call never returned. The call also carries no per-attempt deadline, so HealthTimeout is only checked between attempts and rollback never fires. Done when the watchdog runs the staged release's own archied (so it speaks the new protocol) and every control-plane call in switchOver and rollback has a bounded deadline, so a stuck switch rolls back.

Close reason

Watchdog runs staged archied; bounded deadlines; a973bd65


Imported from beads archie-core-mub3d · GitHub #1695 · type bug · priority P1 · created 2026-10-15 · closed 2026-10-17

The watchdog is a frozen copy of the installing (old) archied, so it speaks the old release's protocol to the new State Store. Upgrading 1.53 to 1.54.1 hung in phase switching: the 1.53 watchdog dialed the TLS-only State Store in plaintext with wait-for-ready, and the image-policy call never returned. The call also carries no per-attempt deadline, so HealthTimeout is only checked between attempts and rollback never fires. Done when the watchdog runs the staged release's own archied (so it speaks the new protocol) and every control-plane call in switchOver and rollback has a bounded deadline, so a stuck switch rolls back. ### Close reason Watchdog runs staged archied; bounded deadlines; a973bd65 --- _Imported from beads `archie-core-mub3d` · GitHub #1695 · type bug · priority P1 · created 2026-10-15 · closed 2026-10-17_
sam added the highbug labels 2026-10-27 13:40:28 +00:00
sam self-assigned this 2026-10-27 13:40:28 +00:00
sam closed this issue 2026-10-27 13:40:28 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1455