Runs keep the protections the task lifecycle had #2036

Open
opened 2026-11-16 10:39:49 +00:00 by sam · 1 comment
Owner

The old task lifecycle (internal/taskstate, internal/daemon/park.go, the State Store's guarded transitions) handled cases the single binary's run engine (modules/runs) doesn't. Cross-checked 2026-11-16 against the code and the fix history; MIGRATION.md row "Task lifecycle and states".

Missing for every run:

  1. Restart cap. A task interrupted by 3 restarts in a row parked for a person (taskstate.PlanRecovery, e75d5c8ad). A run whose step crashes the process is reclaimed after every lease expiry, forever: attempts counts claims and nothing caps it.
  2. A state for "needs a person". Old parks carried a class (needs a decision, transient, terminal) and a blocker naming what was missing, so the dashboard linked to the fix (internal/webui/task_recovery.go). A run can only fail.
  3. Transient retry with backoff. An environmental park retried itself after 30s, doubling, 5 times, then asked a person (taskstate.TransientRetryDue, 860b2c85e). Runs retry only model calls, inside a step.
  4. Retry a failed run, optionally resuming at a step with the results before it (RetryTask, resumeResults). The runs API has no retry.
  5. Stop at once, keep the work. Stop cancelled a running task immediately and parked it, resumable (daemon/running.go, 7d6f22d22). A run checks for cancellation only between steps, so a long agent step runs to its end, and a cancelled run can't be resumed.
  6. State-change history. Every status change wrote a row with from, to and detail. A run records its steps but not its state changes or who asked for them (overlaps #2019).
  7. One table of what each state allows. The dashboard read which operator actions each status offered, and stale or illegal actions were refused (taskstate.Actions, 5734e194c).
  8. Ask to continue at the budget. A stage that hit its step budget waited for approval to continue, including inside a parallel branch (bcbd09fc0). A run has no ask-to-continue path.
  9. Progress watchdog. The systemd watchdog beat only while every loop had started a pass recently (StalestLoop). Readiness checks the database and connectors, not that workers are moving (#2010).

Done when: each of the nine either works for runs, with a test where it touches a listed seam, or has a recorded decision to drop it.

The old task lifecycle (`internal/taskstate`, `internal/daemon/park.go`, the State Store's guarded transitions) handled cases the single binary's run engine (`modules/runs`) doesn't. Cross-checked 2026-11-16 against the code and the fix history; MIGRATION.md row "Task lifecycle and states". Missing for every run: 1. **Restart cap.** A task interrupted by 3 restarts in a row parked for a person (`taskstate.PlanRecovery`, e75d5c8ad). A run whose step crashes the process is reclaimed after every lease expiry, forever: `attempts` counts claims and nothing caps it. 2. **A state for "needs a person".** Old parks carried a class (needs a decision, transient, terminal) and a blocker naming what was missing, so the dashboard linked to the fix (`internal/webui/task_recovery.go`). A run can only fail. 3. **Transient retry with backoff.** An environmental park retried itself after 30s, doubling, 5 times, then asked a person (`taskstate.TransientRetryDue`, 860b2c85e). Runs retry only model calls, inside a step. 4. **Retry a failed run**, optionally resuming at a step with the results before it (`RetryTask`, `resumeResults`). The runs API has no retry. 5. **Stop at once, keep the work.** Stop cancelled a running task immediately and parked it, resumable (`daemon/running.go`, 7d6f22d22). A run checks for cancellation only between steps, so a long agent step runs to its end, and a cancelled run can't be resumed. 6. **State-change history.** Every status change wrote a row with from, to and detail. A run records its steps but not its state changes or who asked for them (overlaps #2019). 7. **One table of what each state allows.** The dashboard read which operator actions each status offered, and stale or illegal actions were refused (`taskstate.Actions`, 5734e194c). 8. **Ask to continue at the budget.** A stage that hit its step budget waited for approval to continue, including inside a parallel branch (bcbd09fc0). A run has no ask-to-continue path. 9. **Progress watchdog.** The systemd watchdog beat only while every loop had started a pass recently (`StalestLoop`). Readiness checks the database and connectors, not that workers are moving (#2010). Done when: each of the nine either works for runs, with a test where it touches a listed seam, or has a recorded decision to drop it.
sam added the featurehigh labels 2026-11-16 10:39:49 +00:00
sam added a new dependency 2026-11-16 10:39:56 +00:00
Author
Owner

Maintainer requirement: each protection here must be available to whoever writes the workflow, Archie or a person, as a property or parameter of the document, with defaults that match today's behaviour. Today's vocabulary already carried several of them (retry with attempts and backoff, on_failure park or continue, workflow.finish with a parked or waiting-for-approval status, retry from a top-level step), and #2037 brings that vocabulary onto the durable engine. The rest (the restart cap, park classes and blockers, transient backoff, ask to continue at the budget) get document properties alongside them.

Maintainer requirement: each protection here must be available to whoever writes the workflow, Archie or a person, as a property or parameter of the document, with defaults that match today's behaviour. Today's vocabulary already carried several of them (`retry` with attempts and backoff, `on_failure` park or continue, `workflow.finish` with a parked or waiting-for-approval status, retry from a top-level step), and #2037 brings that vocabulary onto the durable engine. The rest (the restart cap, park classes and blockers, transient backoff, ask to continue at the budget) get document properties alongside them.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Reference: sam/archie#2036