Disk readiness probe watches only work_dir, so a full root filesystem goes unreported #1676

Closed
opened 2026-10-27 13:41:21 +00:00 by sam · 0 comments
Owner

The disk probe (internal/infrastructure/readiness/probes.go, NewDiskProbe/DefaultDiskThreshold 0.90) is constructed with a single path. On carina that is work_dir, which lives on /home.

Measured 2026-09-17: /home was 95% used and correctly reported degraded, while / was 59% and /var 47%. The inverse case is the dangerous one and it is invisible: a runaway cache under / would fill a 30G root with 12G free and nothing in /health/detailed would say so until writes started failing.

This is not hypothetical for the Go toolchain as configured: GOCACHE and GOMODCACHE now point at /mnt/app_caches (a ZFS dataset). Go does NOT error when a cache directory is missing -- it creates it. So a failed zpool import at boot silently redirects every build cache write onto /, which the probe would not report.

Two parts:

  1. Probe / (and ideally /var) as well as work_dir, reporting per-filesystem results rather than one path.
  2. Consider a mount guard for /mnt/app_caches in the build path, since Go's fail-open behaviour turns a missing mount into writes on the wrong disk.

Confirmed the fail-open behaviour directly: GOCACHE=/tmp/does-not-exist-xyz/build on 'go build' exited 0 and created the directory.

Close reason

Landed in 8378c92; GitHub #835 already closed.


Imported from beads archie-core-yijm · GitHub #835 · type bug · priority P2 · created 2026-09-17 · closed 2026-09-17

The disk probe (internal/infrastructure/readiness/probes.go, NewDiskProbe/DefaultDiskThreshold 0.90) is constructed with a single path. On carina that is work_dir, which lives on /home. Measured 2026-09-17: /home was 95% used and correctly reported degraded, while / was 59% and /var 47%. The inverse case is the dangerous one and it is invisible: a runaway cache under / would fill a 30G root with 12G free and nothing in /health/detailed would say so until writes started failing. This is not hypothetical for the Go toolchain as configured: GOCACHE and GOMODCACHE now point at /mnt/app_caches (a ZFS dataset). Go does NOT error when a cache directory is missing -- it creates it. So a failed zpool import at boot silently redirects every build cache write onto /, which the probe would not report. Two parts: 1. Probe / (and ideally /var) as well as work_dir, reporting per-filesystem results rather than one path. 2. Consider a mount guard for /mnt/app_caches in the build path, since Go's fail-open behaviour turns a missing mount into writes on the wrong disk. Confirmed the fail-open behaviour directly: GOCACHE=/tmp/does-not-exist-xyz/build on 'go build' exited 0 and created the directory. ### Close reason Landed in 8378c92; GitHub #835 already closed. --- _Imported from beads `archie-core-yijm` · GitHub #835 · type bug · priority P2 · created 2026-09-17 · closed 2026-09-17_
sam added the bugmedium labels 2026-10-27 13:41:21 +00:00
sam self-assigned this 2026-10-27 13:41:21 +00:00
sam closed this issue 2026-10-27 13:41:21 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sam/archie#1676