Transient task parks are never retried and raise no alert #1804

Closed
opened 2026-11-08 06:15:12 +00:00 by sam · 1 comment
Owner

What is wrong

Tasks parked with class transient (storage full, worktree prepare failed, run credential timeout, container exit, models unresolved) are never retried and raise no alert. On the production instance 189 tasks were parked this way in one 5-second burst and stayed parked for over 30 hours. The only alerts raised were task.needs_decision for the 24 gate-red tasks. The 189 environmental parks are visible only by filtering the task list. The class doc in internal/taskstate/taskstate.go says "Nothing retries them automatically yet".

Evidence: 194 parked/transient rows, 192 of them parked within seconds of creation, none with a later attempt; alerts table has no row for any of them.

Done when

  • A task parked transient is requeued automatically with bounded backoff, or raises one grouped alert naming the cause and count (one alert for "189 tasks stopped by a full disk", not 189).
  • A task that exhausts the retries parks as needs_decision with the last cause.
  • Verified by a test that a transient park is retried and a repeated one escalates.
## What is wrong Tasks parked with class `transient` (storage full, worktree prepare failed, run credential timeout, container exit, models unresolved) are never retried and raise no alert. On the production instance 189 tasks were parked this way in one 5-second burst and stayed parked for over 30 hours. The only alerts raised were `task.needs_decision` for the 24 gate-red tasks. The 189 environmental parks are visible only by filtering the task list. The class doc in `internal/taskstate/taskstate.go` says "Nothing retries them automatically yet". Evidence: 194 parked/transient rows, 192 of them parked within seconds of creation, none with a later attempt; alerts table has no row for any of them. ## Done when - A task parked `transient` is requeued automatically with bounded backoff, or raises one grouped alert naming the cause and count (one alert for "189 tasks stopped by a full disk", not 189). - A task that exhausts the retries parks as `needs_decision` with the last cause. - Verified by a test that a transient park is retried and a repeated one escalates.
Author
Owner

Fixed in 860b2c85. The daemon cycle now requeues transient parks after a doubling backoff (30s, 1m, 2m, 4m, 8m) on the task's stored retry mode. After 5 retries the State Store parks the task needs_decision when it is parked, which raises the existing task.needs_decision alert. Alerts are per task, not grouped. Tests: taskstate backoff and escalation table, daemon retry selection with a fake store. task check passes.

Fixed in 860b2c85. The daemon cycle now requeues transient parks after a doubling backoff (30s, 1m, 2m, 4m, 8m) on the task's stored retry mode. After 5 retries the State Store parks the task needs_decision when it is parked, which raises the existing task.needs_decision alert. Alerts are per task, not grouped. Tests: taskstate backoff and escalation table, daemon retry selection with a fake store. task check passes.
sam closed this issue 2026-11-08 06:22:43 +00:00
sam added the bug label 2026-11-12 15:52:54 +00:00
sam added the task label 2026-11-14 14:45:22 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1804