Self-hosted Gitea runner fills host /tmp and all CI jobs fail with exit 128 #1828

Closed
opened 2026-11-08 15:50:02 +00:00 by sam · 1 comment
Owner

The self-hosted Gitea Actions runner bind-mounts the host /tmp into every job container (the runner config's container options include -v /tmp:/tmp). /tmp is a small tmpfs, so each job's toolchain downloads (~70M temp files per job), the node compile cache and leaked per-job temp dirs fill it within hours. Once full, the container runtime cannot exec (write /tmp/runc-process*: no space left on device, exit 128) and every job on the runner fails, leaking more temp files - a death spiral. Seen 2026-11-08: release workflow runs failed repeatedly, push-triggered deploy runs failed, /tmp at 100% with ~20 UUID-named 55-69M temp files plus a 554M node-compile-cache. Worked around by deleting the leaked files and remounting /tmp to 32G; a second host reportedly shows the same fill. Proper fix: drop the host /tmp bind from the runner's container options (or point the runner's temp/work dir at disk-backed storage), and enlarge the tmpfs or clean it on a schedule.

The self-hosted Gitea Actions runner bind-mounts the host /tmp into every job container (the runner config's container options include -v /tmp:/tmp). /tmp is a small tmpfs, so each job's toolchain downloads (~70M temp files per job), the node compile cache and leaked per-job temp dirs fill it within hours. Once full, the container runtime cannot exec (write /tmp/runc-process*: no space left on device, exit 128) and every job on the runner fails, leaking more temp files - a death spiral. Seen 2026-11-08: release workflow runs failed repeatedly, push-triggered deploy runs failed, /tmp at 100% with ~20 UUID-named 55-69M temp files plus a 554M node-compile-cache. Worked around by deleting the leaked files and remounting /tmp to 32G; a second host reportedly shows the same fill. Proper fix: drop the host /tmp bind from the runner's container options (or point the runner's temp/work dir at disk-backed storage), and enlarge the tmpfs or clean it on a schedule.
Author
Owner

Mitigated on the carina runner host: an hourly root timer removes the UUID-named tool downloads older than 2h that job containers leave in the shared /tmp (the runners bind-mount /tmp for TestKitRuntimeDiscovery). Kept the mount by decision.

Mitigated on the carina runner host: an hourly root timer removes the UUID-named tool downloads older than 2h that job containers leave in the shared /tmp (the runners bind-mount /tmp for TestKitRuntimeDiscovery). Kept the mount by decision.
sam closed this issue 2026-11-09 03:42:32 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1828