Found 2026-09-21 while diagnosing why every catlow-tech pipeline reported 'error' with zero steps on a config that lints clean.
MECHANISM (from Woodpecker v3.18.0 source + live logs on carina)
On a server recreate the server sends gRPC GOAWAY (graceful_stop); the agent retries report_health, then logs 'backoff: maximum elapsed time exceeded' -> 'runner error, retrying...' and then goes SILENT for 5+ minutes. Not even the 10s health-report loop logs, so it is a wedge, not slow backoff.
The one path that guarantees a fresh process never fires: on clean loss the runner logs 'connection to server lost, shutting down agent' and exits (so restart:unless-stopped recovers it), but that only happens when retryRPC returns ErrConnectionLost, which requires the last error to be the synthetic errNotConnected. With a real RPC Unavailable it falls through to the generic retry branch. So the shutdown path is effectively unreachable for a real transport failure.
The container healthcheck (woodpecker-agent ping -> /healthz -> State.Healthy()) only inspects in-flight workflow timeouts with a 1h buffer. With no workflow running it is ALWAYS true, so it stayed green through the whole wedge. Docker also does not restart on unhealthy.
Consequence: pipelines are accepted by the server, assigned to nobody, and recorded as error with zero steps and correct metadata - indistinguishable from a broken pipeline config. This cost a lane hours of wrong-hypothesis hunting.
FIX (role change, applied but not yet run)
Handler recreates server AND agent: docker compose up -d --force-recreate woodpecker-server woodpecker-agent (drop the --no-deps/server-only scoping). Sparing the agent buys nothing because a server recreate already drops its workflow leases (log shows 'queue: task expired').
Role-side post-recreate assertion that the agent re-registered/connected, failing the play if not - converts green-but-orphaned into a visible failure.
WOODPECKER_RETRY_TIMEOUT=0 as supporting config only; it does not close the wedge alone.
RUNBOOK: 'pipeline reported error with zero steps' => check agent connectivity FIRST, not the pipeline config.
OPEN HYPOTHESIS (not proven, not pursued - would need a controlled outage): every RPC goes through an auth interceptor that attaches a token and refreshes it over the auth connection on Unauthenticated; if that refresh hangs, RPCs hang instead of failing fast and retrying, which would explain the total silence better than the backoff story.
Imported from beads archie-core-cyoq · GitHub #1112 · type bug · priority P2 · created 2026-09-21
Found 2026-09-21 while diagnosing why every catlow-tech pipeline reported 'error' with zero steps on a config that lints clean.
MECHANISM (from Woodpecker v3.18.0 source + live logs on carina)
- On a server recreate the server sends gRPC GOAWAY (graceful_stop); the agent retries report_health, then logs 'backoff: maximum elapsed time exceeded' -> 'runner error, retrying...' and then goes SILENT for 5+ minutes. Not even the 10s health-report loop logs, so it is a wedge, not slow backoff.
- The one path that guarantees a fresh process never fires: on clean loss the runner logs 'connection to server lost, shutting down agent' and exits (so restart:unless-stopped recovers it), but that only happens when retryRPC returns ErrConnectionLost, which requires the last error to be the synthetic errNotConnected. With a real RPC Unavailable it falls through to the generic retry branch. So the shutdown path is effectively unreachable for a real transport failure.
- The container healthcheck (woodpecker-agent ping -> /healthz -> State.Healthy()) only inspects in-flight workflow timeouts with a 1h buffer. With no workflow running it is ALWAYS true, so it stayed green through the whole wedge. Docker also does not restart on unhealthy.
- Consequence: pipelines are accepted by the server, assigned to nobody, and recorded as error with zero steps and correct metadata - indistinguishable from a broken pipeline config. This cost a lane hours of wrong-hypothesis hunting.
FIX (role change, applied but not yet run)
1. Handler recreates server AND agent: docker compose up -d --force-recreate woodpecker-server woodpecker-agent (drop the --no-deps/server-only scoping). Sparing the agent buys nothing because a server recreate already drops its workflow leases (log shows 'queue: task expired').
2. Role-side post-recreate assertion that the agent re-registered/connected, failing the play if not - converts green-but-orphaned into a visible failure.
3. WOODPECKER_RETRY_TIMEOUT=0 as supporting config only; it does not close the wedge alone.
RUNBOOK: 'pipeline reported error with zero steps' => check agent connectivity FIRST, not the pipeline config.
OPEN HYPOTHESIS (not proven, not pursued - would need a controlled outage): every RPC goes through an auth interceptor that attaches a token and refreshes it over the auth connection on Unauthenticated; if that refresh hangs, RPCs hang instead of failing fast and retrying, which would explain the total silence better than the backoff story.
---
_Imported from beads `archie-core-cyoq` · GitHub #1112 · type bug · priority P2 · created 2026-09-21_
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found 2026-09-21 while diagnosing why every catlow-tech pipeline reported 'error' with zero steps on a config that lints clean.
MECHANISM (from Woodpecker v3.18.0 source + live logs on carina)
FIX (role change, applied but not yet run)
RUNBOOK: 'pipeline reported error with zero steps' => check agent connectivity FIRST, not the pipeline config.
OPEN HYPOTHESIS (not proven, not pursued - would need a controlled outage): every RPC goes through an auth interceptor that attaches a token and refreshes it over the auth connection on Unauthenticated; if that refresh hangs, RPCs hang instead of failing fast and retrying, which would explain the total silence better than the backoff story.
Imported from beads
archie-core-cyoq· GitHub #1112 · type bug · priority P2 · created 2026-09-21