But when handling a fetched message fails, it Naks and loops straight back to the next fetch with no delay at all:
iferr:=handleSingleStageRequest(ctx,message,log,productionSingleStageDependencies());err!=nil{log.Error("handle failed","err",err)iferr:=message.Nak();err!=nil{log.Warn("nak failed","err",err)}continue// <-- no delay before the next FetchStage}
If a stage message fails deterministically (bad request payload, unreachable dependency, permanently broken environment), Nak → immediate redelivery → immediate re-fail repeats at machine speed until MaxDeliver (default 3, internal/infrastructure/eventbus/nats/config.go:16) is exhausted for that message — but with multiple stage messages queued for a stuck task, or a MaxDeliver that doesn't apply the way expected in the stage-consumer path, this produces an uncontrolled burst.
Evidence
Task #1000000000000001, 2026-08-15: 996 identical handle failed err={} log lines within a 3-second window (08:51:28–08:51:31), confirmed live via the dashboard /logs page. Related: archie-core-9c985f28 style Sam messages describe this independently as "unbounded rapid message redelivery."
Fix
Add a backoff (e.g. the same delay(time.Second), or better, exponential with jitter) before continue in the handle-failure branch, and confirm MaxDeliver is actually enforced for the stage consumer's Nak path (internal/infrastructure/agenttransport/nats/transport.go) so a permanently-failing message stops being redelivered rather than burning the log and the NATS server.
Close reason
Fixed in 3eab6a2 (archied/archie v1.9.3): added the same time.Second backoff after Nak that the sibling fetch-error branch already had, stopping the immediate redelivery-storm loop. Regression test in stage_loop_test.go. Not done: exponential backoff+jitter (used the existing constant-delay pattern instead, for consistency) and explicit verification that MaxDeliver is enforced end-to-end for the stage consumer's Nak path -- worth a follow-up if the storm recurs at a smaller scale.
Imported from beads archie-core-1786749700516-3-f1298573 · GitHub #517 · type bug · priority P1 · created 2026-08-14 · closed 2026-08-15
## Problem
`runMainLoopWithDelay` (`internal/app/agentworker/stage_loop.go:23-52`) sleeps 1 second before retrying when *fetching* a message fails:
```go
case err != nil:
...
log.Error("fetch failed", "err", err)
delay(time.Second)
continue
```
But when *handling* a fetched message fails, it Naks and loops straight back to the next fetch with **no delay at all**:
```go
if err := handleSingleStageRequest(ctx, message, log, productionSingleStageDependencies()); err != nil {
log.Error("handle failed", "err", err)
if err := message.Nak(); err != nil {
log.Warn("nak failed", "err", err)
}
continue // <-- no delay before the next FetchStage
}
```
If a stage message fails deterministically (bad request payload, unreachable dependency, permanently broken environment), Nak → immediate redelivery → immediate re-fail repeats at machine speed until `MaxDeliver` (default 3, `internal/infrastructure/eventbus/nats/config.go:16`) is exhausted for that message — but with multiple stage messages queued for a stuck task, or a `MaxDeliver` that doesn't apply the way expected in the stage-consumer path, this produces an uncontrolled burst.
## Evidence
Task `#1000000000000001`, 2026-08-15: **996 identical `handle failed err={}` log lines within a 3-second window** (08:51:28–08:51:31), confirmed live via the dashboard `/logs` page. Related: archie-core-9c985f28 style Sam messages describe this independently as "unbounded rapid message redelivery."
## Fix
Add a backoff (e.g. the same `delay(time.Second)`, or better, exponential with jitter) before `continue` in the handle-failure branch, and confirm `MaxDeliver` is actually enforced for the stage consumer's Nak path (`internal/infrastructure/agenttransport/nats/transport.go`) so a permanently-failing message stops being redelivered rather than burning the log and the NATS server.
### Close reason
Fixed in 3eab6a2 (archied/archie v1.9.3): added the same time.Second backoff after Nak that the sibling fetch-error branch already had, stopping the immediate redelivery-storm loop. Regression test in stage_loop_test.go. Not done: exponential backoff+jitter (used the existing constant-delay pattern instead, for consistency) and explicit verification that MaxDeliver is enforced end-to-end for the stage consumer's Nak path -- worth a follow-up if the storm recurs at a smaller scale.
---
_Imported from beads `archie-core-1786749700516-3-f1298573` · GitHub #517 · type bug · priority P1 · created 2026-08-14 · closed 2026-08-15_
sam
added the highbug labels 2026-10-27 13:38:32 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Problem
runMainLoopWithDelay(internal/app/agentworker/stage_loop.go:23-52) sleeps 1 second before retrying when fetching a message fails:But when handling a fetched message fails, it Naks and loops straight back to the next fetch with no delay at all:
If a stage message fails deterministically (bad request payload, unreachable dependency, permanently broken environment), Nak → immediate redelivery → immediate re-fail repeats at machine speed until
MaxDeliver(default 3,internal/infrastructure/eventbus/nats/config.go:16) is exhausted for that message — but with multiple stage messages queued for a stuck task, or aMaxDeliverthat doesn't apply the way expected in the stage-consumer path, this produces an uncontrolled burst.Evidence
Task
#1000000000000001, 2026-08-15: 996 identicalhandle failed err={}log lines within a 3-second window (08:51:28–08:51:31), confirmed live via the dashboard/logspage. Related: archie-core-9c985f28 style Sam messages describe this independently as "unbounded rapid message redelivery."Fix
Add a backoff (e.g. the same
delay(time.Second), or better, exponential with jitter) beforecontinuein the handle-failure branch, and confirmMaxDeliveris actually enforced for the stage consumer's Nak path (internal/infrastructure/agenttransport/nats/transport.go) so a permanently-failing message stops being redelivered rather than burning the log and the NATS server.Close reason
Fixed in 3eab6a2 (archied/archie v1.9.3): added the same time.Second backoff after Nak that the sibling fetch-error branch already had, stopping the immediate redelivery-storm loop. Regression test in stage_loop_test.go. Not done: exponential backoff+jitter (used the existing constant-delay pattern instead, for consistency) and explicit verification that MaxDeliver is enforced end-to-end for the stage consumer's Nak path -- worth a follow-up if the storm recurs at a smaller scale.
Imported from beads
archie-core-1786749700516-3-f1298573· GitHub #517 · type bug · priority P1 · created 2026-08-14 · closed 2026-08-15