Skip to content

data-plane: reactor accumulates superseded per-shard state directories until /mnt/local is exhausted #3287

Description

@jwhartley

1. Priority

Medium - when /mnt/local fills, every shard on the reactor hard-fails. Manual cleanup of the unreferenced directories is a reliable workaround, which holds this below High.

2. Scope and prevalence

  • One reactor on a private Azure data plane held 24,435 state directories under /mnt/local/reactor against 148 live shards, totalling 111 GiB of a 125 GB disk. Only 142 directories were referenced by any running process (verified via open file descriptors); 24,293 were unreferenced.
  • Plane-wide, not confined to the alerting reactor. Two peers on the same plane held 23,476 and 14,283 directories against 148 live shards each. The reactor that alerted was simply the first to cross the threshold.
  • Directory-creation rate measured on the same reactor: 4-5/hour when healthy, ~650/hour during sustained shard-restart churn. At that rate a 125 GB disk is exhausted in well under a day.
  • Two independent prior occurrences on other tenants' planes, both requiring manual deletion (see References).

3. Problem description

  • Expected: when a shard restarts or is reassigned away, its superseded local state directory is removed. Per data-plane eng in prior discussion, these directories are ephemeral, fully recoverable from recovery logs, and "if a shard is disabled, or un-assigned from this reactor, we simply delete its directory".
  • Observed: superseded directories persist indefinitely. Each restart creates a new directory for the same shard ID with a new trailing suffix, and prior generations remain. Individual shards accumulated up to 62 generations. Directories older than the current host boot were absent, so nothing reaps them on any longer cycle.
  • Size distribution is bimodal, tracking how far each restart got before failing: 1,057 directories at >=64 MB (78 GiB) and 23,123 at 1-16 MB (33 GiB).
  • The accumulation is not reclaimable by restarting the process or rebooting the host. Deleted-but-open files totalled ~0 bytes on the affected reactor (du and df agreed at 114 G), and on this plane /mnt/local is a Premium_LRS managed data disk, so it survives reboot and deallocation.
  • Observed consequences once the disk was near full, raw ts from reactor logs:
2026-07-31T01:51:35Z  servePrimary failed: completeRecovery: app.NewStore: fetching DB: storage: object doesn't exist
2026-07-31T02:12:20Z  servePrimary failed: completeRecovery: app.NewStore: fetching DB: storage: object doesn't exist
2026-07-31T03:40:47Z  runTransactions: txnStartCommit: store.StartCommit: h2 protocol error: error reading a body from connection
  • Steps to reproduce: induce sustained shard-restart churn on a reactor (any repeated task-level failure will do), then compare find /mnt/local/reactor -maxdepth 1 -mindepth 1 -type d | wc -l against that reactor's live shard count from gazette_shard_up.

4. References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions