an update can restart archied before the State Store accepts connections, failing the first boot closed #1575

Closed
opened 2026-10-27 13:40:56 +00:00 by sam · 0 comments
Owner

An update can restart archied before the State Store accepts connections, so the new binary's first boot fails closed.

Observed on a production host during a 1.37.1 -> 1.38.0 upgrade:

Stopping archied.service ... Stopped (old release)
Started archied.service ... health listener up (new release)
+2s ERROR runtime settings unavailable: control-plane unavailable:
connection refused to the State Store target
+2s archied.service: Main process exited, code=exited, status=1/FAILURE
+7s Scheduled restart job, restart counter is at 1 -> Started -> healthy

The update succeeded -- every installed binary reports the release and the daemon is up -- but the FIRST boot of the new binary failed closed because the State Store was not yet accepting connections. Only systemd's Restart=on-failure rescued it seven seconds later.

Blast radius: a host whose unit lacks Restart=on-failure, or where the State Store binds slower than the restart budget, gets a failed update behind a healthy-looking "build and install succeeded" line.

Cause: scripts/archie-update-watchdog start_units orders the units (the State Store, then the Gateway, then archied and the UI), but ordering only sequences systemctl --user start calls. For a Type=simple unit systemctl returns once the process is forked, not once its port accepts, so the order guarantees nothing about readiness. archied correctly fails closed on an unreachable State Store; nothing waits for it.

Fix direction: poll the State Store's listener between the ordered start calls, with a bounded budget, so archied is only started once its dependency can answer. Do not weaken archied's fail-closed boot.

The host, its addresses and the timestamps are recorded in the operator's memos instance, not here: beads must not carry production instance detail.

Close reason

Resolved in current source: installer passes the configured State Store target to the updater watchdog, which polls its listener after restarting the store. Additionally 7fd092ac makes service State Store calls wait for the peer through WaitForReady, including daemon runtime-settings boot reads. Temporary connection refusal no longer follows the reported immediate-exit path. Invalid configuration/authentication still fails closed; this review did not perform a live upgrade.


Imported from beads archie-core-sq5u · GitHub #1091 · type bug · priority P2 · created 2026-09-23 · closed 2026-10-04

An update can restart archied before the State Store accepts connections, so the new binary's first boot fails closed. Observed on a production host during a 1.37.1 -> 1.38.0 upgrade: Stopping archied.service ... Stopped (old release) Started archied.service ... health listener up (new release) +2s ERROR runtime settings unavailable: control-plane unavailable: connection refused to the State Store target +2s archied.service: Main process exited, code=exited, status=1/FAILURE +7s Scheduled restart job, restart counter is at 1 -> Started -> healthy The update succeeded -- every installed binary reports the release and the daemon is up -- but the FIRST boot of the new binary failed closed because the State Store was not yet accepting connections. Only systemd's Restart=on-failure rescued it seven seconds later. Blast radius: a host whose unit lacks Restart=on-failure, or where the State Store binds slower than the restart budget, gets a failed update behind a healthy-looking "build and install succeeded" line. Cause: scripts/archie-update-watchdog start_units orders the units (the State Store, then the Gateway, then archied and the UI), but ordering only sequences `systemctl --user start` calls. For a Type=simple unit systemctl returns once the process is forked, not once its port accepts, so the order guarantees nothing about readiness. archied correctly fails closed on an unreachable State Store; nothing waits for it. Fix direction: poll the State Store's listener between the ordered start calls, with a bounded budget, so archied is only started once its dependency can answer. Do not weaken archied's fail-closed boot. The host, its addresses and the timestamps are recorded in the operator's memos instance, not here: beads must not carry production instance detail. ### Close reason Resolved in current source: installer passes the configured State Store target to the updater watchdog, which polls its listener after restarting the store. Additionally 7fd092ac makes service State Store calls wait for the peer through WaitForReady, including daemon runtime-settings boot reads. Temporary connection refusal no longer follows the reported immediate-exit path. Invalid configuration/authentication still fails closed; this review did not perform a live upgrade. --- _Imported from beads `archie-core-sq5u` · GitHub #1091 · type bug · priority P2 · created 2026-09-23 · closed 2026-10-04_
sam added the mediumbug labels 2026-10-27 13:40:56 +00:00
sam self-assigned this 2026-10-27 13:40:56 +00:00
sam closed this issue 2026-10-27 13:40:57 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1575