An update can restart archied before the State Store accepts connections, so the new binary's first boot fails closed.
Observed on a production host during a 1.37.1 -> 1.38.0 upgrade:
Stopping archied.service ... Stopped (old release)
Started archied.service ... health listener up (new release)
+2s ERROR runtime settings unavailable: control-plane unavailable:
connection refused to the State Store target
+2s archied.service: Main process exited, code=exited, status=1/FAILURE
+7s Scheduled restart job, restart counter is at 1 -> Started -> healthy
The update succeeded -- every installed binary reports the release and the daemon is up -- but the FIRST boot of the new binary failed closed because the State Store was not yet accepting connections. Only systemd's Restart=on-failure rescued it seven seconds later.
Blast radius: a host whose unit lacks Restart=on-failure, or where the State Store binds slower than the restart budget, gets a failed update behind a healthy-looking "build and install succeeded" line.
Cause: scripts/archie-update-watchdog start_units orders the units (the State Store, then the Gateway, then archied and the UI), but ordering only sequences systemctl --user start calls. For a Type=simple unit systemctl returns once the process is forked, not once its port accepts, so the order guarantees nothing about readiness. archied correctly fails closed on an unreachable State Store; nothing waits for it.
Fix direction: poll the State Store's listener between the ordered start calls, with a bounded budget, so archied is only started once its dependency can answer. Do not weaken archied's fail-closed boot.
The host, its addresses and the timestamps are recorded in the operator's memos instance, not here: beads must not carry production instance detail.
Close reason
Resolved in current source: installer passes the configured State Store target to the updater watchdog, which polls its listener after restarting the store. Additionally 7fd092ac makes service State Store calls wait for the peer through WaitForReady, including daemon runtime-settings boot reads. Temporary connection refusal no longer follows the reported immediate-exit path. Invalid configuration/authentication still fails closed; this review did not perform a live upgrade.
Imported from beads archie-core-sq5u · GitHub #1091 · type bug · priority P2 · created 2026-09-23 · closed 2026-10-04
An update can restart archied before the State Store accepts connections, so the new binary's first boot fails closed.
Observed on a production host during a 1.37.1 -> 1.38.0 upgrade:
Stopping archied.service ... Stopped (old release)
Started archied.service ... health listener up (new release)
+2s ERROR runtime settings unavailable: control-plane unavailable:
connection refused to the State Store target
+2s archied.service: Main process exited, code=exited, status=1/FAILURE
+7s Scheduled restart job, restart counter is at 1 -> Started -> healthy
The update succeeded -- every installed binary reports the release and the daemon is up -- but the FIRST boot of the new binary failed closed because the State Store was not yet accepting connections. Only systemd's Restart=on-failure rescued it seven seconds later.
Blast radius: a host whose unit lacks Restart=on-failure, or where the State Store binds slower than the restart budget, gets a failed update behind a healthy-looking "build and install succeeded" line.
Cause: scripts/archie-update-watchdog start_units orders the units (the State Store, then the Gateway, then archied and the UI), but ordering only sequences `systemctl --user start` calls. For a Type=simple unit systemctl returns once the process is forked, not once its port accepts, so the order guarantees nothing about readiness. archied correctly fails closed on an unreachable State Store; nothing waits for it.
Fix direction: poll the State Store's listener between the ordered start calls, with a bounded budget, so archied is only started once its dependency can answer. Do not weaken archied's fail-closed boot.
The host, its addresses and the timestamps are recorded in the operator's memos instance, not here: beads must not carry production instance detail.
### Close reason
Resolved in current source: installer passes the configured State Store target to the updater watchdog, which polls its listener after restarting the store. Additionally 7fd092ac makes service State Store calls wait for the peer through WaitForReady, including daemon runtime-settings boot reads. Temporary connection refusal no longer follows the reported immediate-exit path. Invalid configuration/authentication still fails closed; this review did not perform a live upgrade.
---
_Imported from beads `archie-core-sq5u` · GitHub #1091 · type bug · priority P2 · created 2026-09-23 · closed 2026-10-04_
sam
added the mediumbug labels 2026-10-27 13:40:56 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
An update can restart archied before the State Store accepts connections, so the new binary's first boot fails closed.
Observed on a production host during a 1.37.1 -> 1.38.0 upgrade:
Stopping archied.service ... Stopped (old release)
Started archied.service ... health listener up (new release)
+2s ERROR runtime settings unavailable: control-plane unavailable:
connection refused to the State Store target
+2s archied.service: Main process exited, code=exited, status=1/FAILURE
+7s Scheduled restart job, restart counter is at 1 -> Started -> healthy
The update succeeded -- every installed binary reports the release and the daemon is up -- but the FIRST boot of the new binary failed closed because the State Store was not yet accepting connections. Only systemd's Restart=on-failure rescued it seven seconds later.
Blast radius: a host whose unit lacks Restart=on-failure, or where the State Store binds slower than the restart budget, gets a failed update behind a healthy-looking "build and install succeeded" line.
Cause: scripts/archie-update-watchdog start_units orders the units (the State Store, then the Gateway, then archied and the UI), but ordering only sequences
systemctl --user startcalls. For a Type=simple unit systemctl returns once the process is forked, not once its port accepts, so the order guarantees nothing about readiness. archied correctly fails closed on an unreachable State Store; nothing waits for it.Fix direction: poll the State Store's listener between the ordered start calls, with a bounded budget, so archied is only started once its dependency can answer. Do not weaken archied's fail-closed boot.
The host, its addresses and the timestamps are recorded in the operator's memos instance, not here: beads must not carry production instance detail.
Close reason
Resolved in current source: installer passes the configured State Store target to the updater watchdog, which polls its listener after restarting the store. Additionally
7fd092acmakes service State Store calls wait for the peer through WaitForReady, including daemon runtime-settings boot reads. Temporary connection refusal no longer follows the reported immediate-exit path. Invalid configuration/authentication still fails closed; this review did not perform a live upgrade.Imported from beads
archie-core-sq5u· GitHub #1091 · type bug · priority P2 · created 2026-09-23 · closed 2026-10-04