Woodpecker agent wedges silently after a server recreate, and its healthcheck cannot detect it #1276

Open
opened 2026-10-27 13:39:44 +00:00 by sam · 0 comments
Owner

Found 2026-09-21 while diagnosing why every catlow-tech pipeline reported 'error' with zero steps on a config that lints clean.

MECHANISM (from Woodpecker v3.18.0 source + live logs on carina)

  • On a server recreate the server sends gRPC GOAWAY (graceful_stop); the agent retries report_health, then logs 'backoff: maximum elapsed time exceeded' -> 'runner error, retrying...' and then goes SILENT for 5+ minutes. Not even the 10s health-report loop logs, so it is a wedge, not slow backoff.
  • The one path that guarantees a fresh process never fires: on clean loss the runner logs 'connection to server lost, shutting down agent' and exits (so restart:unless-stopped recovers it), but that only happens when retryRPC returns ErrConnectionLost, which requires the last error to be the synthetic errNotConnected. With a real RPC Unavailable it falls through to the generic retry branch. So the shutdown path is effectively unreachable for a real transport failure.
  • The container healthcheck (woodpecker-agent ping -> /healthz -> State.Healthy()) only inspects in-flight workflow timeouts with a 1h buffer. With no workflow running it is ALWAYS true, so it stayed green through the whole wedge. Docker also does not restart on unhealthy.
  • Consequence: pipelines are accepted by the server, assigned to nobody, and recorded as error with zero steps and correct metadata - indistinguishable from a broken pipeline config. This cost a lane hours of wrong-hypothesis hunting.

FIX (role change, applied but not yet run)

  1. Handler recreates server AND agent: docker compose up -d --force-recreate woodpecker-server woodpecker-agent (drop the --no-deps/server-only scoping). Sparing the agent buys nothing because a server recreate already drops its workflow leases (log shows 'queue: task expired').
  2. Role-side post-recreate assertion that the agent re-registered/connected, failing the play if not - converts green-but-orphaned into a visible failure.
  3. WOODPECKER_RETRY_TIMEOUT=0 as supporting config only; it does not close the wedge alone.

RUNBOOK: 'pipeline reported error with zero steps' => check agent connectivity FIRST, not the pipeline config.

OPEN HYPOTHESIS (not proven, not pursued - would need a controlled outage): every RPC goes through an auth interceptor that attaches a token and refreshes it over the auth connection on Unauthenticated; if that refresh hangs, RPCs hang instead of failing fast and retrying, which would explain the total silence better than the backoff story.


Imported from beads archie-core-cyoq · GitHub #1112 · type bug · priority P2 · created 2026-09-21

Found 2026-09-21 while diagnosing why every catlow-tech pipeline reported 'error' with zero steps on a config that lints clean. MECHANISM (from Woodpecker v3.18.0 source + live logs on carina) - On a server recreate the server sends gRPC GOAWAY (graceful_stop); the agent retries report_health, then logs 'backoff: maximum elapsed time exceeded' -> 'runner error, retrying...' and then goes SILENT for 5+ minutes. Not even the 10s health-report loop logs, so it is a wedge, not slow backoff. - The one path that guarantees a fresh process never fires: on clean loss the runner logs 'connection to server lost, shutting down agent' and exits (so restart:unless-stopped recovers it), but that only happens when retryRPC returns ErrConnectionLost, which requires the last error to be the synthetic errNotConnected. With a real RPC Unavailable it falls through to the generic retry branch. So the shutdown path is effectively unreachable for a real transport failure. - The container healthcheck (woodpecker-agent ping -> /healthz -> State.Healthy()) only inspects in-flight workflow timeouts with a 1h buffer. With no workflow running it is ALWAYS true, so it stayed green through the whole wedge. Docker also does not restart on unhealthy. - Consequence: pipelines are accepted by the server, assigned to nobody, and recorded as error with zero steps and correct metadata - indistinguishable from a broken pipeline config. This cost a lane hours of wrong-hypothesis hunting. FIX (role change, applied but not yet run) 1. Handler recreates server AND agent: docker compose up -d --force-recreate woodpecker-server woodpecker-agent (drop the --no-deps/server-only scoping). Sparing the agent buys nothing because a server recreate already drops its workflow leases (log shows 'queue: task expired'). 2. Role-side post-recreate assertion that the agent re-registered/connected, failing the play if not - converts green-but-orphaned into a visible failure. 3. WOODPECKER_RETRY_TIMEOUT=0 as supporting config only; it does not close the wedge alone. RUNBOOK: 'pipeline reported error with zero steps' => check agent connectivity FIRST, not the pipeline config. OPEN HYPOTHESIS (not proven, not pursued - would need a controlled outage): every RPC goes through an auth interceptor that attaches a token and refreshes it over the auth connection on Unauthenticated; if that refresh hangs, RPCs hang instead of failing fast and retrying, which would explain the total silence better than the backoff story. --- _Imported from beads `archie-core-cyoq` · GitHub #1112 · type bug · priority P2 · created 2026-09-21_
sam added the mediumbug labels 2026-10-27 13:39:44 +00:00
sam self-assigned this 2026-10-27 13:39:44 +00:00
sam added the deferred label 2026-11-15 17:26:20 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1276