agentworker: stage handle-failure path has no backoff, causing rapid NATS redelivery storms #977

Closed
opened 2026-10-27 13:38:32 +00:00 by sam · 0 comments
Owner

Problem

runMainLoopWithDelay (internal/app/agentworker/stage_loop.go:23-52) sleeps 1 second before retrying when fetching a message fails:

case err != nil:
    ...
    log.Error("fetch failed", "err", err)
    delay(time.Second)
    continue

But when handling a fetched message fails, it Naks and loops straight back to the next fetch with no delay at all:

if err := handleSingleStageRequest(ctx, message, log, productionSingleStageDependencies()); err != nil {
    log.Error("handle failed", "err", err)
    if err := message.Nak(); err != nil {
        log.Warn("nak failed", "err", err)
    }
    continue   // <-- no delay before the next FetchStage
}

If a stage message fails deterministically (bad request payload, unreachable dependency, permanently broken environment), Nak → immediate redelivery → immediate re-fail repeats at machine speed until MaxDeliver (default 3, internal/infrastructure/eventbus/nats/config.go:16) is exhausted for that message — but with multiple stage messages queued for a stuck task, or a MaxDeliver that doesn't apply the way expected in the stage-consumer path, this produces an uncontrolled burst.

Evidence

Task #1000000000000001, 2026-08-15: 996 identical handle failed err={} log lines within a 3-second window (08:51:28–08:51:31), confirmed live via the dashboard /logs page. Related: archie-core-9c985f28 style Sam messages describe this independently as "unbounded rapid message redelivery."

Fix

Add a backoff (e.g. the same delay(time.Second), or better, exponential with jitter) before continue in the handle-failure branch, and confirm MaxDeliver is actually enforced for the stage consumer's Nak path (internal/infrastructure/agenttransport/nats/transport.go) so a permanently-failing message stops being redelivered rather than burning the log and the NATS server.

Close reason

Fixed in 3eab6a2 (archied/archie v1.9.3): added the same time.Second backoff after Nak that the sibling fetch-error branch already had, stopping the immediate redelivery-storm loop. Regression test in stage_loop_test.go. Not done: exponential backoff+jitter (used the existing constant-delay pattern instead, for consistency) and explicit verification that MaxDeliver is enforced end-to-end for the stage consumer's Nak path -- worth a follow-up if the storm recurs at a smaller scale.


Imported from beads archie-core-1786749700516-3-f1298573 · GitHub #517 · type bug · priority P1 · created 2026-08-14 · closed 2026-08-15

## Problem `runMainLoopWithDelay` (`internal/app/agentworker/stage_loop.go:23-52`) sleeps 1 second before retrying when *fetching* a message fails: ```go case err != nil: ... log.Error("fetch failed", "err", err) delay(time.Second) continue ``` But when *handling* a fetched message fails, it Naks and loops straight back to the next fetch with **no delay at all**: ```go if err := handleSingleStageRequest(ctx, message, log, productionSingleStageDependencies()); err != nil { log.Error("handle failed", "err", err) if err := message.Nak(); err != nil { log.Warn("nak failed", "err", err) } continue // <-- no delay before the next FetchStage } ``` If a stage message fails deterministically (bad request payload, unreachable dependency, permanently broken environment), Nak → immediate redelivery → immediate re-fail repeats at machine speed until `MaxDeliver` (default 3, `internal/infrastructure/eventbus/nats/config.go:16`) is exhausted for that message — but with multiple stage messages queued for a stuck task, or a `MaxDeliver` that doesn't apply the way expected in the stage-consumer path, this produces an uncontrolled burst. ## Evidence Task `#1000000000000001`, 2026-08-15: **996 identical `handle failed err={}` log lines within a 3-second window** (08:51:28–08:51:31), confirmed live via the dashboard `/logs` page. Related: archie-core-9c985f28 style Sam messages describe this independently as "unbounded rapid message redelivery." ## Fix Add a backoff (e.g. the same `delay(time.Second)`, or better, exponential with jitter) before `continue` in the handle-failure branch, and confirm `MaxDeliver` is actually enforced for the stage consumer's Nak path (`internal/infrastructure/agenttransport/nats/transport.go`) so a permanently-failing message stops being redelivered rather than burning the log and the NATS server. ### Close reason Fixed in 3eab6a2 (archied/archie v1.9.3): added the same time.Second backoff after Nak that the sibling fetch-error branch already had, stopping the immediate redelivery-storm loop. Regression test in stage_loop_test.go. Not done: exponential backoff+jitter (used the existing constant-delay pattern instead, for consistency) and explicit verification that MaxDeliver is enforced end-to-end for the stage consumer's Nak path -- worth a follow-up if the storm recurs at a smaller scale. --- _Imported from beads `archie-core-1786749700516-3-f1298573` · GitHub #517 · type bug · priority P1 · created 2026-08-14 · closed 2026-08-15_
sam added the highbug labels 2026-10-27 13:38:32 +00:00
sam self-assigned this 2026-10-27 13:38:32 +00:00
sam closed this issue 2026-10-27 13:38:32 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#977