staterpc: no keepalive, so a half-open connection freezes a control-plane watch silently #522

Closed
opened 2026-10-27 13:36:39 +00:00 by sam · 0 comments
Owner

Found by the archie-core-yrmr review (operator angle). The yrmr fix reconnects when a stream ENDS, but nothing detects a stream that stops without ending: internal/infrastructure/staterpc/dial.go:37-49 installs no keepalive, grpc's defaultClientKeepaliveTime is infinity, and controlplane.Server.Watch polls only its local store and writes nothing while resource.Version is unchanged (internal/app/controlplane/controlplane.go:130-147). Neither peer ever notices a silently dropped half-open connection, so the watch is frozen for the life of the process with no error, no log, and the last clean apply-status record continuing to re-stamp as current. This is the yrmr failure mode in a narrower window; yrmr's reconnect does not cover it. Fix direction: client-side keepalive on the staterpc dial, or a server-side heartbeat write on the Watch stream so an idle-but-dead connection is detected.

Close reason

Already shipped in d033c3ac (2026-09-22, confirmed ancestor of main) — the bead's premise is stale. Client half: internal/infrastructure/staterpc/dial.go:59 installs grpc.WithKeepaliveParams(ClientKeepaliveParams()) with Time 10s / Timeout 5s, and the option comment names exactly this defect (the control-plane Watch stream is silent by design while the resource version is unchanged, so with neither peer writing neither notices a half-open connection). Server half: ServerKeepaliveOption (dial.go:137) and ServerKeepalivePolicy (:147, MinTime 5s = half the dialer's clamped 10s interval) exist so the pings are not answered with GOAWAY too_many_pings, and they are installed in production on every State Store listener (internal/app/archied/state_store.go:483 stateStoreServerOpts -> :488). staterpc.Dial is the production path for the daemon (state_store_client.go:19), archie-ui (archieui/run.go:68), archie-messaging (archiemessaging/run.go:47) and the NATS agent transport (agenttransport/nats/transport.go:98). Regression-guarded and verified on this tree: TestDialInstallsClientKeepalive passes; TestStateStoreServerOptsHoldTheDialersKeepaliveMargin passes; and TestStateStoreServerOptionsTolerateTheDialersKeepalive proves both directions over a real gRPC connection — gRPC's default enforcement policy tears the dialer's idle watch down at 41s, while the State Store's server options keep it up for the full 50s. No code change needed.


Imported from beads archie-core-0q92 · GitHub #1136 · type bug · priority P2 · created 2026-09-21 · closed 2026-10-02

Found by the archie-core-yrmr review (operator angle). The yrmr fix reconnects when a stream ENDS, but nothing detects a stream that stops without ending: internal/infrastructure/staterpc/dial.go:37-49 installs no keepalive, grpc's defaultClientKeepaliveTime is infinity, and controlplane.Server.Watch polls only its local store and writes nothing while resource.Version is unchanged (internal/app/controlplane/controlplane.go:130-147). Neither peer ever notices a silently dropped half-open connection, so the watch is frozen for the life of the process with no error, no log, and the last clean apply-status record continuing to re-stamp as current. This is the yrmr failure mode in a narrower window; yrmr's reconnect does not cover it. Fix direction: client-side keepalive on the staterpc dial, or a server-side heartbeat write on the Watch stream so an idle-but-dead connection is detected. ### Close reason Already shipped in d033c3ac (2026-09-22, confirmed ancestor of main) — the bead's premise is stale. Client half: internal/infrastructure/staterpc/dial.go:59 installs grpc.WithKeepaliveParams(ClientKeepaliveParams()) with Time 10s / Timeout 5s, and the option comment names exactly this defect (the control-plane Watch stream is silent by design while the resource version is unchanged, so with neither peer writing neither notices a half-open connection). Server half: ServerKeepaliveOption (dial.go:137) and ServerKeepalivePolicy (:147, MinTime 5s = half the dialer's clamped 10s interval) exist so the pings are not answered with GOAWAY too_many_pings, and they are installed in production on every State Store listener (internal/app/archied/state_store.go:483 stateStoreServerOpts -> :488). staterpc.Dial is the production path for the daemon (state_store_client.go:19), archie-ui (archieui/run.go:68), archie-messaging (archiemessaging/run.go:47) and the NATS agent transport (agenttransport/nats/transport.go:98). Regression-guarded and verified on this tree: TestDialInstallsClientKeepalive passes; TestStateStoreServerOptsHoldTheDialersKeepaliveMargin passes; and TestStateStoreServerOptionsTolerateTheDialersKeepalive proves both directions over a real gRPC connection — gRPC's default enforcement policy tears the dialer's idle watch down at 41s, while the State Store's server options keep it up for the full 50s. No code change needed. --- _Imported from beads `archie-core-0q92` · GitHub #1136 · type bug · priority P2 · created 2026-09-21 · closed 2026-10-02_
sam added the mediumbug labels 2026-10-27 13:36:39 +00:00
sam self-assigned this 2026-10-27 13:36:39 +00:00
sam closed this issue 2026-10-27 13:36:39 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#522