Cancelled contexts and reconnects are logged as warnings and errors #1809

Closed
opened 2026-11-08 06:15:13 +00:00 by sam · 1 comment
Owner

What is wrong

Context cancellation from a shutdown or a caller that went away is logged as an error or warning. Counts over 7 days: State Store 318 staterpc call failed ERROR lines (mostly context canceled: PutPresence 112, PutApplyStatus 59, ListSteps 37, StatusCounts 32), daemon 107 and gateway 59 WARN apply status not reported, 41 State Store identity store failed, UI 832 control-plane watch ended; reconnecting WARN (mostly connection refused during a peer restart), 96 container network auto-detect: could not inspect self by hostname WARN (once per boot on a native, non-container install). These bury the real faults in the same journals.

Done when

  • A cancelled or deadline-exceeded context caused by shutdown or a departing caller is not logged at WARN or ERROR (presence.Run already does this; apply it to the others).
  • A reconnecting watch logs once per outage, not once per attempt.
  • The container-network auto-detect line is logged only when the daemon is actually in a container.
  • Verified by the journal counts for these messages on a normal restart being zero.
## What is wrong Context cancellation from a shutdown or a caller that went away is logged as an error or warning. Counts over 7 days: State Store 318 `staterpc call failed` ERROR lines (mostly `context canceled`: PutPresence 112, PutApplyStatus 59, ListSteps 37, StatusCounts 32), daemon 107 and gateway 59 WARN `apply status not reported`, 41 State Store `identity store failed`, UI 832 `control-plane watch ended; reconnecting` WARN (mostly connection refused during a peer restart), 96 `container network auto-detect: could not inspect self by hostname` WARN (once per boot on a native, non-container install). These bury the real faults in the same journals. ## Done when - A cancelled or deadline-exceeded context caused by shutdown or a departing caller is not logged at WARN or ERROR (`presence.Run` already does this; apply it to the others). - A reconnecting watch logs once per outage, not once per attempt. - The container-network auto-detect line is logged only when the daemon is actually in a container. - Verified by the journal counts for these messages on a normal restart being zero.
Author
Owner

Fixed in 1d9eff65: context.Canceled is logged at debug in staterpc, identity store and apply status; the control-plane watch logs once per outage; the container network auto-detect runs only inside a container. Deadline-exceeded stays at its level since it can be a real database timeout.

Fixed in 1d9eff65: context.Canceled is logged at debug in staterpc, identity store and apply status; the control-plane watch logs once per outage; the container network auto-detect runs only inside a container. Deadline-exceeded stays at its level since it can be a real database timeout.
sam closed this issue 2026-11-08 06:28:42 +00:00
sam added the bug label 2026-11-12 15:52:54 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1809