Task clones of parked and finished tasks are never reclaimed, so the work disk fills and all task work stops #1814

Closed
opened 2026-11-08 12:00:50 +00:00 by sam · 1 comment
Owner

Wrong: a task clone is removed only when its PR merges or is rejected, or when the operator archives the task. Tasks that park, complete or close without a PR keep their clone (about 570 MB each, mostly dependency stores) indefinitely, and volume_ttl does not cover them. Dozens of parked tasks filled the work volume to 94%; the daemon then refused to clone below its 5 GiB free-space floor, so every task stalled and nothing in the dashboard said why disk was the cause.

Done when: a clone with nothing unpushed or uncommitted is reclaimed after a bounded idle time, or when free space falls below the floor (oldest idle first); clones holding unpushed or uncommitted work are never removed automatically and are surfaced to the operator; the stall caused by low disk is visible as an alert.

Wrong: a task clone is removed only when its PR merges or is rejected, or when the operator archives the task. Tasks that park, complete or close without a PR keep their clone (about 570 MB each, mostly dependency stores) indefinitely, and volume_ttl does not cover them. Dozens of parked tasks filled the work volume to 94%; the daemon then refused to clone below its 5 GiB free-space floor, so every task stalled and nothing in the dashboard said why disk was the cause. Done when: a clone with nothing unpushed or uncommitted is reclaimed after a bounded idle time, or when free space falls below the floor (oldest idle first); clones holding unpushed or uncommitted work are never removed automatically and are surfaced to the operator; the stall caused by low disk is visible as an alert.
Author
Owner

Fixed in 1d329bad3, deployed to carina. New hourly built-in job clone-reclaim removes clones idle past 72h regardless of free space; clones with uncommitted or unpushed work are kept and logged as a warning naming the directory. Kept clones are not yet a dashboard alert. task check passed on the merged tree.

Fixed in 1d329bad3, deployed to carina. New hourly built-in job clone-reclaim removes clones idle past 72h regardless of free space; clones with uncommitted or unpushed work are kept and logged as a warning naming the directory. Kept clones are not yet a dashboard alert. task check passed on the merged tree.
sam closed this issue 2026-11-14 04:09:22 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1814