Per-thread LMDB reader slots are never released in long-lived processes → MDB_READERS_FULL (maxreaders left at heed default 126, no MDB_NOTLS)

[ChaoXu1997]

Summary

FFF's LMDB environments are opened without max_readers and without MDB_NOTLS, so LMDB runs in default TLS mode: a reader slot is pinned per thread and only released when the process exits. In long-lived embedding processes (Neovim, Node-based agents via fff-node, …) that accumulate more than 126 reader threads over days, the reader table fills up and every subsequent env open fails:

FFF init failed: Failed to init frecency db: Failed to start read transaction
for frecency database: MDB_READERS_FULL: Environment maxreaders limit reached

This is the complementary case to #460: clear_stale_readers() (added in #468) reclaims slots of dead PIDs, but in our incident all slot holders were alive — 7 long-lived processes holding 15–19 slots each, 126/126 total:

pid 3118389  19 slots  pi        (up 5 days)
pid 648283   19 slots  pi        (up 5 days)
pid 1033540  19 slots  pi        (up 1 day)
...                                   (7 processes, 126 slots total)

So the fix from #468 cannot help here — nothing is stale.

Environment

• @ff-labs/fff-node 0.10.3 (libfff_c.so from @ff-labs/fff-bin-linux-x64-gnu 0.10.3)
• Host: Node v22 embedding via the pi coding agent + pi-pretty 0.6.21 (frecency/history at ~/.pi/agent/pi-pretty/fff/)
• Linux x86_64 (glibc), kernel threads: Node main + libuv threadpool + fff native threads
• Same behavior reproduced with fff-node 0.9.6 before upgrading

Evidence

Reader-table forensics on frecency.mdb/lock.mdb (8 KiB page, 64-byte slots → 126 slots + header, matching the LMDB default):

# lock.mdb page 0: header at +0, slots at 64-byte stride, pid at slot+8
import os, struct
lock = open("frecency.mdb/lock.mdb", "rb").read()
live = {int(d) for d in os.listdir("/proc") if d.isdigit()}
for off in range(64, len(lock) - 16, 64):
    pid = struct.unpack_from("<I", lock, off + 8)[0]
    if pid:
        print(pid, "ALIVE" if pid in live else "DEAD")

Controlled probes (throwaway DBs, small tree):

• Short-lived process, 10 create/destroy/destroy cycles of FileFinder → exactly 3 slots (main + 2 native threads), stable across cycles — slots are per-thread, not per-finder or per-open. Env pooling from fix(pi-fff): stop reopening main LMDB envs in aux finders (#700) #701 works.
• Long-lived agent processes → 15–19 slots each and growing with uptime; nothing frees them until the process exits.

Current code

On main, EnvOpenOptions only sets map_size (+ max_dbs):

• crates/fff-core/src/dbs/env_pool.rs (~L95–99)
• crates/fff-core/src/dbs/lmdb.rs (open_env, ~L141–147)

No max_readers(...), no NOTLS flag. clear_stale_readers() on open (env_pool.rs) only helps dead PIDs.

Suggestions

1. Cheap mitigation: raise max_readers explicitly (e.g. 256–512). Reader slots are tiny; the cost is negligible, and it buys embedding hosts a lot of headroom.
2. Structural fix: open the envs with MDB_NOTLS so read transactions borrow a slot from the pool only for the txn lifetime instead of pinning one per thread. This is the standard recommendation for hosts with transient/many threads (libuv threadpool, LSP-style worker churn). Caveat: in NOTLS mode a read txn must begin and end on the same thread — since fff owns its txn lifecycles inside fff-core, this should be enforceable there.
3. Optionally expose the value to embedders (e.g. InitOptions field or env var like FFF_LMDB_MAX_READERS) so hosts can tune without a recompile.

Workaround for affected users

Exit all processes holding slots (they are alive, so only process exit frees them), then optionally delete frecency.mdb/lock.mdb while nothing is running (the lock file is rebuilt on next open; frecency data in data.mdb is untouched). Starting the newest fff first also helps: it reclaims dead-PID slots on open, but it can not reclaim slots of the still-running old processes.

Related

• Environment maxreaders limit reached error #460 — original MDB_READERS_FULL (dead-reader case), fixed by fix: LMDB stale readers and automatic compactions  #468
• fix: LMDB stale readers and automatic compactions  #468 — clear_stale_readers + automatic compactions
• [Bug]: pi fff auxiliary finder reopens the main LMDB databases and fails #700 / fix(pi-fff): stop reopening main LMDB envs in aux finders (#700) #701 — aux finders reopening LMDB envs (env pooling)
• scripts/diagnose_lockmdb.py — diagnoses mutex holders/waiters; the slot-count snippet above complements it for reader-slot accounting

[gustav-fff]

[triage-bot] REPRODUCED: Reproduced on origin/main. Branch: triage-bot/issue-783.

Confirmed root cause: envs are opened with only map_size (+max_dbs) in crates/fff-core/src/dbs/env_pool.rs:94-101, so heed's default max_readers=126 and default TLS mode apply. >126 live reader threads each pin a slot and exhaust the table.

Direct heed repro matching fff's open options (200 long-lived threads, each holding a read txn):

running 1 test
thread 'default_open_exhausts_reader_table_past_126' panicked at
crates/fff-core/tests/lmdb_readers_full_repro.rs:81:5:
MDB_READERS_FULL hit: only 126/200 live reader threads got a slot

Expected: all 200 get a slot. Actual: plateaus at 126, rest fail with MDB_READERS_FULL.

Scope note: shipping the cheap mitigation (suggestion 1) + config exposure (suggestion 3) only. MDB_NOTLS (suggestion 2) is the structural fix but touches read-txn thread discipline and heed's RoTxn Send bounds — leaving that call to @dmtrKovalenko.

Working on fix on triage-bot/issue-783.
Honk-Honk 🪿