1. Priority
Medium - when /mnt/local fills, every shard on the reactor hard-fails. Manual cleanup of the unreferenced directories is a reliable workaround, which holds this below High.
2. Scope and prevalence
- One reactor on a private Azure data plane held 24,435 state directories under
/mnt/local/reactor against 148 live shards, totalling 111 GiB of a 125 GB disk. Only 142 directories were referenced by any running process (verified via open file descriptors); 24,293 were unreferenced.
- Plane-wide, not confined to the alerting reactor. Two peers on the same plane held 23,476 and 14,283 directories against 148 live shards each. The reactor that alerted was simply the first to cross the threshold.
- Directory-creation rate measured on the same reactor: 4-5/hour when healthy, ~650/hour during sustained shard-restart churn. At that rate a 125 GB disk is exhausted in well under a day.
- Two independent prior occurrences on other tenants' planes, both requiring manual deletion (see References).
3. Problem description
- Expected: when a shard restarts or is reassigned away, its superseded local state directory is removed. Per data-plane eng in prior discussion, these directories are ephemeral, fully recoverable from recovery logs, and "if a shard is disabled, or un-assigned from this reactor, we simply delete its directory".
- Observed: superseded directories persist indefinitely. Each restart creates a new directory for the same shard ID with a new trailing suffix, and prior generations remain. Individual shards accumulated up to 62 generations. Directories older than the current host boot were absent, so nothing reaps them on any longer cycle.
- Size distribution is bimodal, tracking how far each restart got before failing: 1,057 directories at >=64 MB (78 GiB) and 23,123 at 1-16 MB (33 GiB).
- The accumulation is not reclaimable by restarting the process or rebooting the host. Deleted-but-open files totalled ~0 bytes on the affected reactor (
du and df agreed at 114 G), and on this plane /mnt/local is a Premium_LRS managed data disk, so it survives reboot and deallocation.
- Observed consequences once the disk was near full, raw
ts from reactor logs:
2026-07-31T01:51:35Z servePrimary failed: completeRecovery: app.NewStore: fetching DB: storage: object doesn't exist
2026-07-31T02:12:20Z servePrimary failed: completeRecovery: app.NewStore: fetching DB: storage: object doesn't exist
2026-07-31T03:40:47Z runTransactions: txnStartCommit: store.StartCommit: h2 protocol error: error reading a body from connection
- Steps to reproduce: induce sustained shard-restart churn on a reactor (any repeated task-level failure will do), then compare
find /mnt/local/reactor -maxdepth 1 -mindepth 1 -type d | wc -l against that reactor's live shard count from gazette_shard_up.
4. References
1. Priority
Medium - when
/mnt/localfills, every shard on the reactor hard-fails. Manual cleanup of the unreferenced directories is a reliable workaround, which holds this below High.2. Scope and prevalence
/mnt/local/reactoragainst 148 live shards, totalling 111 GiB of a 125 GB disk. Only 142 directories were referenced by any running process (verified via open file descriptors); 24,293 were unreferenced.3. Problem description
duanddfagreed at 114 G), and on this plane/mnt/localis a Premium_LRS managed data disk, so it survives reboot and deallocation.tsfrom reactor logs:find /mnt/local/reactor -maxdepth 1 -mindepth 1 -type d | wc -lagainst that reactor's live shard count fromgazette_shard_up.4. References