Since 49d7619 (fix(daemon): claim a requeued backlog without a poll wait per task, shipped after v1.61.1), archied claims every requeued task in the store in a single drain pass and submits them all to the dispatcher. Observed effect: every task appears to run concurrently (self-improvement work included), and /home (8 GB) filled up within ~24 hours.
What changed
Before, drainNATS fell back to d.Store.ClaimNext once per pass, then returned to the stream. Each empty Fetch waited a full poll timeout, which slowed the backlog to one claim per poll. That was an accident, but it worked as a throttle.
This loops until the store has nothing left to claim, with no limit tied to dispatcher capacity. The watchdog problem it set out to fix (14 resumed tasks outlasting the 30 s systemd watchdog) was real, and that fix should stay. What's missing is a capacity limit.
Why the dispatcher cap doesn't save us
Daemon.dispatcher is documented as bounding execution globally via containers.max_concurrency (daemon.go ~L184). But the drainLoop doc comment (~L384) says:
Submit never blocks, so a full dispatcher just leaves tasks waiting on slots
So claimRequeued marks every task as claimed in SQLite and parks one goroutine per task waiting for a slot. The store and UI then report the whole backlog as running, even though only max_concurrency should be executing.
Not yet verified (check these first)
Where the disk goes. Does process (or the dispatcher wrapper around it) create the worktree, storage volume or container before acquiring a dispatcher slot? If so, every claimed task allocates disk immediately, which would explain the 8 GB in 24 h. If allocation only happens after acquiring the slot, the disk growth has another cause (e.g. volumes not being cleaned up after runs; see cleanupExpiredStorage, which runs only at Startup).
Effective containers.max_concurrency. Confirm the deployed value. Zero or unset may mean unbounded, depending on how taskDispatcher treats it.
Where taskDispatcher is defined.internal/daemon/dispatcher.go is an empty stub (package daemon, 15 bytes), as is identity.go. The type presumably lives in daemon.go (2,740 lines, ~100 KB) after a refactor. If the stubs are leftovers, delete them; either way, consider splitting daemon.go.
Proposed fix
Make claimRequeued capacity-aware: claim only while the dispatcher has free slots, e.g. a dispatcher.Free() / TryReserve() check before ClaimNext, and leave the rest in the store for later passes. That keeps the watchdog fix (no poll wait per claim) without pulling in the whole backlog.
Alternatively, have Submit take a slot synchronously and claimRequeued stop when none are available. Don't put back a sleep or a per-claim stream wait.
Make sure no per-task resource (worktree, volume, container) is allocated before a slot is held.
Add a test next to TestDrainPassClaimsABacklogWithoutWaitingPerTask: with max_concurrency = 2 and 10 queued tasks, one pass claims exactly 2 and leaves 8 unclaimed in the store.
Cleanup after deploying the fix
Requeue or unclaim the tasks currently marked claimed but waiting on slots.
Reclaim disk: prune orphaned worktrees and storage volumes. Consider running cleanupExpiredStorage periodically, not only at startup.
Related
The event storm from this backlog is also driving the task-list frontend performance issue (filed separately).
## Summary
Since 49d7619 (`fix(daemon): claim a requeued backlog without a poll wait per task`, shipped after v1.61.1), archied claims every requeued task in the store in a single drain pass and submits them all to the dispatcher. Observed effect: every task appears to run concurrently (self-improvement work included), and `/home` (8 GB) filled up within ~24 hours.
## What changed
Before, `drainNATS` fell back to `d.Store.ClaimNext` once per pass, then returned to the stream. Each empty `Fetch` waited a full poll timeout, which slowed the backlog to one claim per poll. That was an accident, but it worked as a throttle.
The fix moved the fallback into `claimRequeued`:
```go
func (d *Daemon) claimRequeued(ctx context.Context, dispatcher *taskDispatcher) {
for ctx.Err() == nil {
task, err := d.Store.ClaimNext(ctx)
...
if task == nil {
return
}
dispatcher.Submit(ctx, task, d.process)
d.markPass("drain")
}
}
```
This loops until the store has nothing left to claim, with no limit tied to dispatcher capacity. The watchdog problem it set out to fix (14 resumed tasks outlasting the 30 s systemd watchdog) was real, and that fix should stay. What's missing is a capacity limit.
## Why the dispatcher cap doesn't save us
`Daemon.dispatcher` is documented as bounding execution globally via `containers.max_concurrency` (`daemon.go` ~L184). But the `drainLoop` doc comment (~L384) says:
> Submit never blocks, so a full dispatcher just leaves tasks waiting on slots
So `claimRequeued` marks every task as claimed in SQLite and parks one goroutine per task waiting for a slot. The store and UI then report the whole backlog as running, even though only `max_concurrency` should be executing.
## Not yet verified (check these first)
1. **Where the disk goes.** Does `process` (or the dispatcher wrapper around it) create the worktree, storage volume or container *before* acquiring a dispatcher slot? If so, every claimed task allocates disk immediately, which would explain the 8 GB in 24 h. If allocation only happens after acquiring the slot, the disk growth has another cause (e.g. volumes not being cleaned up after runs; see `cleanupExpiredStorage`, which runs only at `Startup`).
2. **Effective `containers.max_concurrency`.** Confirm the deployed value. Zero or unset may mean unbounded, depending on how `taskDispatcher` treats it.
3. **Where `taskDispatcher` is defined.** `internal/daemon/dispatcher.go` is an empty stub (`package daemon`, 15 bytes), as is `identity.go`. The type presumably lives in `daemon.go` (2,740 lines, ~100 KB) after a refactor. If the stubs are leftovers, delete them; either way, consider splitting `daemon.go`.
## Proposed fix
- Make `claimRequeued` capacity-aware: claim only while the dispatcher has free slots, e.g. a `dispatcher.Free()` / `TryReserve()` check before `ClaimNext`, and leave the rest in the store for later passes. That keeps the watchdog fix (no poll wait per claim) without pulling in the whole backlog.
- Alternatively, have `Submit` take a slot synchronously and `claimRequeued` stop when none are available. Don't put back a sleep or a per-claim stream wait.
- Make sure no per-task resource (worktree, volume, container) is allocated before a slot is held.
- Add a test next to `TestDrainPassClaimsABacklogWithoutWaitingPerTask`: with `max_concurrency = 2` and 10 queued tasks, one pass claims exactly 2 and leaves 8 unclaimed in the store.
## Cleanup after deploying the fix
- Requeue or unclaim the tasks currently marked claimed but waiting on slots.
- Reclaim disk: prune orphaned worktrees and storage volumes. Consider running `cleanupExpiredStorage` periodically, not only at startup.
## Related
The event storm from this backlog is also driving the task-list frontend performance issue (filed separately).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Summary
Since
49d7619(fix(daemon): claim a requeued backlog without a poll wait per task, shipped after v1.61.1), archied claims every requeued task in the store in a single drain pass and submits them all to the dispatcher. Observed effect: every task appears to run concurrently (self-improvement work included), and/home(8 GB) filled up within ~24 hours.What changed
Before,
drainNATSfell back tod.Store.ClaimNextonce per pass, then returned to the stream. Each emptyFetchwaited a full poll timeout, which slowed the backlog to one claim per poll. That was an accident, but it worked as a throttle.The fix moved the fallback into
claimRequeued:This loops until the store has nothing left to claim, with no limit tied to dispatcher capacity. The watchdog problem it set out to fix (14 resumed tasks outlasting the 30 s systemd watchdog) was real, and that fix should stay. What's missing is a capacity limit.
Why the dispatcher cap doesn't save us
Daemon.dispatcheris documented as bounding execution globally viacontainers.max_concurrency(daemon.go~L184). But thedrainLoopdoc comment (~L384) says:So
claimRequeuedmarks every task as claimed in SQLite and parks one goroutine per task waiting for a slot. The store and UI then report the whole backlog as running, even though onlymax_concurrencyshould be executing.Not yet verified (check these first)
process(or the dispatcher wrapper around it) create the worktree, storage volume or container before acquiring a dispatcher slot? If so, every claimed task allocates disk immediately, which would explain the 8 GB in 24 h. If allocation only happens after acquiring the slot, the disk growth has another cause (e.g. volumes not being cleaned up after runs; seecleanupExpiredStorage, which runs only atStartup).containers.max_concurrency. Confirm the deployed value. Zero or unset may mean unbounded, depending on howtaskDispatchertreats it.taskDispatcheris defined.internal/daemon/dispatcher.gois an empty stub (package daemon, 15 bytes), as isidentity.go. The type presumably lives indaemon.go(2,740 lines, ~100 KB) after a refactor. If the stubs are leftovers, delete them; either way, consider splittingdaemon.go.Proposed fix
claimRequeuedcapacity-aware: claim only while the dispatcher has free slots, e.g. adispatcher.Free()/TryReserve()check beforeClaimNext, and leave the rest in the store for later passes. That keeps the watchdog fix (no poll wait per claim) without pulling in the whole backlog.Submittake a slot synchronously andclaimRequeuedstop when none are available. Don't put back a sleep or a per-claim stream wait.TestDrainPassClaimsABacklogWithoutWaitingPerTask: withmax_concurrency = 2and 10 queued tasks, one pass claims exactly 2 and leaves 8 unclaimed in the store.Cleanup after deploying the fix
cleanupExpiredStorageperiodically, not only at startup.Related
The event storm from this backlog is also driving the task-list frontend performance issue (filed separately).