Skip to content

Bound native FDs and prevent text merge livelock - #433

Merged
ajroetker merged 30 commits into
mainfrom
codex/fix-fd-budget-text-merge-backpressure
Aug 4, 2026
Merged

Bound native FDs and prevent text merge livelock#433
ajroetker merged 30 commits into
mainfrom
codex/fix-fd-budget-text-merge-backpressure

Conversation

@ajroetker

@ajroetker ajroetker commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • enforce one process-wide, RLIMIT-derived native storage FD budget across every BackendRuntime, default NativeStorage helper, LSM store, cached read, WAL/file create, atomic writer, directory traversal, persistent lock, and transient full-text mmap open
  • admit transient descriptors before opening with weighted FIFO progress, cancellation-aware waits, release wakeups, and aggregate capacity/admitted/waiter/wait metrics
  • reserve descriptor headroom for backend-lifetime path locks and fail explicitly on true persistent exhaustion, preventing an impossible-to-drain admission wait during table open or restore
  • stop full-text merge livelock by reconciling concurrent deletion-bitmap changes into replacement segments atomically instead of discarding completed merges
  • keep text merge maintenance and producer admission active during dense bulk ingestion, with event-driven waits and a bounded five-second terminal deadline
  • give text publication admission weighted FIFO ordering so older catch-up batches cannot starve behind younger small writes, and propagate runtime cancellation independently from capacity timeouts
  • export total active text segments and maximum per-index segment fan-out separately from pending merge debt
  • invalidate each NativeStorage cache namespace before its runtime closes, preventing restore or reopen from reusing a cached descriptor for a replaced inode

Correctness

Late text deletes are reconciled by stable document ordinal, with stored document ID as fallback. Replacement deletion bitmaps are persisted in the same transaction as the segment catalog update and installed in the same volatile snapshot publication. Source segment replacement and non-monotonic deletion state remain stale and are still rejected.

Native path locks remain backed by owned std.Io.Threaded runtimes because their filesystem and locking operations are blocking. Text producer admission uses the owning BackendRuntime threaded I/O so cancellation and shutdown propagate through the runtime.

This branch is based on merged PR #430 HEAD. The four previously observed algebraic-filter mismatches were not a source-verified defect and this PR does not claim an unproven fix for them. A frozen repeat should still retain acknowledged write sequence/version, query identity generation, index applied sequence, direct source/hash, posting membership, restart result, and the first mismatching store.

Validation

  • zig build: passed
  • text merge focused suite: 17 passed, 0 failed, 0 leaked
  • admission sweep: 44 passed, 0 failed, 0 leaked
  • native FD focused suite: 9 passed, including aggregate pinning, reserved persistent headroom, explicit lifetime exhaustion, weighted FIFO progress, and cancellation
  • data runtime FD and text fan-out metric regression: passed
  • cluster backup/restore round trip, restore modes, and partial-backup retry/restore: 3 passed in 43.85s
  • full-text bulk maintenance, permit release wakeup, concurrent delete publication, persisted replacement tombstone, FIFO producer ordering, and canceled-wait cleanup regressions: passed

@ajroetker
ajroetker force-pushed the codex/fix-fd-budget-text-merge-backpressure branch from 3279baa to b298451 Compare August 3, 2026 04:52
…ext-merge-backpressure

# Conflicts:
#	zig/pkg/antfly/src/storage/db/catalog/index_manager.zig
@ajroetker
ajroetker merged commit e34b130 into main Aug 4, 2026
9 checks passed
@ajroetker
ajroetker deleted the codex/fix-fd-budget-text-merge-backpressure branch August 4, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant