/stop does not kill an in-flight archie-agent NATS worker or its container — task keeps running #979

Closed
opened 2026-10-27 13:38:32 +00:00 by sam · 0 comments
Owner

This is the most important finding from today's investigation.

/stop is meant to be an emergency brake. It is not one: it reported success ("🛑 Stopped task(s) #5") while the task kept running, kept burning tokens, and kept flooding the logs with errors for at least the next 20+ minutes.

Root cause

/stop → Gateway.stopRunningTasks → router.Controller.StopRunning (internal/gateway/tasks.go:227) → Daemon.CancelRunningTasks (internal/daemon/running.go:114) → runningTasks.stopAll, which cancels a context.CancelFunc registered in an in-process map (internal/daemon/running.go:36-52, populated by Daemon.process at internal/daemon/daemon.go:800).

That context governs only archied's own orchestration goroutine for the task. In the NATS-dispatched execution model (internal/app/agentworker), the actual work happens in a separate process — the archie-agent worker — that pulled a stage request off NATS and is running it inside handleSingleStageRequest / runMainLoopWithDelay (internal/app/agentworker/stage_loop.go), driven by its own top-level context that has no connection whatsoever to the daemon's per-task cancel registry. Cancelling the daemon-side context does not reach it. There is currently no signal, subject, or mechanism by which archied can tell an already-dispatched archie-agent worker to stop.

This matches what was observed live: task #1000000000000001 was reported stopped, remained Working in the dashboard 20+ minutes later, and its log showed 996 handle failed err={} entries in the minutes after the stop request — the worker was still very much alive and looping.

Expected behavior

/stop (and task Cancel generally) should be a hard kill for both halves of the execution:

  1. archied's own orchestration (already works, via the existing context cancellation).
  2. The archie-agent worker actually doing the work for that task.

Suggested fix

Containers are per-task and Docker-managed by the daemon regardless of deployment shape (internal/container/pool.go). The most reliable "dead in its tracks" mechanism is for /stop/Cancel to kill the task's container directly via the Docker client the pool already holds, rather than relying on context propagation across a process boundary that current code has no way to cross. A NATS-side alternative (publish a cancellation event on a per-task subject that stage_loop.go's loop checks between fetch and ack, or have the daemon revoke the task lease so the worker's next Ack/Respond fails) is a secondary/defense-in-depth improvement, but killing the container is what actually guarantees the work stops.

Related

The redelivery-storm behavior after a stop/nak (#517) and the swallowed error detail (#516) both make this worse in practice — an operator watching the logs after /stop sees hundreds of contentless error lines with no way to tell whether the stop is taking effect.

Close reason

Root cause found after Sam confirmed live testing still showed the task's in-flight tool calls/generation continuing tens of seconds to ~2 minutes after /stop reported success. internal/container/pool.go Release() always slept the configured GracePeriod and always used Docker's default ~10s graceful SIGTERM-then-SIGKILL timeout, even when the release was happening because the task's context had just been cancelled by /stop -- so a cancelled task's container (and the in-flight work inside it) could sit doing nothing for up to GracePeriod+~10s before actually dying. Fixed with a new releaseDecision(ctx, grace) helper: a cancelled ctx now skips the grace sleep entirely and requests an immediate SIGKILL (ContainerStopOptions.Timeout=0) instead of a graceful wait; a live ctx (normal completion) keeps the original behaviour byte-for-byte. Red-then-green with 3 new unit tests (internal/container/pool_test.go), task check clean, adversarially reviewed by a fresh subagent with zero blocking findings -- context-chain trace, Docker Timeout=0 semantics, and struct-equivalence for the unchanged path were all independently verified against the moby client source.


Imported from beads archie-core-1786749700580-5-e3878224 · GitHub #519 · type bug · priority P1 · created 2026-08-14 · closed 2026-08-17

## This is the most important finding from today's investigation. `/stop` is meant to be an emergency brake. It is not one: it reported success ("🛑 Stopped task(s) #5") while the task kept running, kept burning tokens, and kept flooding the logs with errors for at least the next 20+ minutes. ## Root cause `/stop` → `Gateway.stopRunningTasks` → `router.Controller.StopRunning` (`internal/gateway/tasks.go:227`) → `Daemon.CancelRunningTasks` (`internal/daemon/running.go:114`) → `runningTasks.stopAll`, which cancels a `context.CancelFunc` registered in an in-process map (`internal/daemon/running.go:36-52`, populated by `Daemon.process` at `internal/daemon/daemon.go:800`). That context governs only `archied`'s own orchestration goroutine for the task. In the NATS-dispatched execution model (`internal/app/agentworker`), the actual work happens in a **separate process** — the archie-agent worker — that pulled a stage request off NATS and is running it inside `handleSingleStageRequest` / `runMainLoopWithDelay` (`internal/app/agentworker/stage_loop.go`), driven by its own top-level context that has no connection whatsoever to the daemon's per-task cancel registry. Cancelling the daemon-side context does not reach it. There is currently no signal, subject, or mechanism by which `archied` can tell an already-dispatched archie-agent worker to stop. This matches what was observed live: task `#1000000000000001` was reported stopped, remained `Working` in the dashboard 20+ minutes later, and its log showed 996 `handle failed err={}` entries in the minutes after the stop request — the worker was still very much alive and looping. ## Expected behavior `/stop` (and task Cancel generally) should be a hard kill for both halves of the execution: 1. `archied`'s own orchestration (already works, via the existing context cancellation). 2. The archie-agent worker actually doing the work for that task. ## Suggested fix Containers are per-task and Docker-managed by the daemon regardless of deployment shape (`internal/container/pool.go`). The most reliable "dead in its tracks" mechanism is for `/stop`/`Cancel` to kill the task's container directly via the Docker client the pool already holds, rather than relying on context propagation across a process boundary that current code has no way to cross. A NATS-side alternative (publish a cancellation event on a per-task subject that `stage_loop.go`'s loop checks between fetch and ack, or have the daemon revoke the task lease so the worker's next `Ack`/`Respond` fails) is a secondary/defense-in-depth improvement, but killing the container is what actually guarantees the work stops. ## Related The redelivery-storm behavior after a stop/nak (#517) and the swallowed error detail (#516) both make this worse in practice — an operator watching the logs after `/stop` sees hundreds of contentless error lines with no way to tell whether the stop is taking effect. ### Close reason Root cause found after Sam confirmed live testing still showed the task's in-flight tool calls/generation continuing tens of seconds to ~2 minutes after /stop reported success. internal/container/pool.go Release() always slept the configured GracePeriod and always used Docker's default ~10s graceful SIGTERM-then-SIGKILL timeout, even when the release was happening because the task's context had just been cancelled by /stop -- so a cancelled task's container (and the in-flight work inside it) could sit doing nothing for up to GracePeriod+~10s before actually dying. Fixed with a new releaseDecision(ctx, grace) helper: a cancelled ctx now skips the grace sleep entirely and requests an immediate SIGKILL (ContainerStopOptions.Timeout=0) instead of a graceful wait; a live ctx (normal completion) keeps the original behaviour byte-for-byte. Red-then-green with 3 new unit tests (internal/container/pool_test.go), task check clean, adversarially reviewed by a fresh subagent with zero blocking findings -- context-chain trace, Docker Timeout=0 semantics, and struct-equivalence for the unchanged path were all independently verified against the moby client source. --- _Imported from beads `archie-core-1786749700580-5-e3878224` · GitHub #519 · type bug · priority P1 · created 2026-08-14 · closed 2026-08-17_
sam added the highbug labels 2026-10-27 13:38:32 +00:00
sam self-assigned this 2026-10-27 13:38:32 +00:00
sam closed this issue 2026-10-27 13:38:32 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#979