[Bug]: pi-fff external-path grep can permanently block the Pi process

[felixkang]

Frontend

Other / multiple — @ff-labs/pi-fff using @ff-labs/fff-node.

Environment

• @ff-labs/pi-fff: reproduced from the 0.10.5 code path; the relevant external-finder path still appears present in 0.10.6
• Pi: 0.84.4
• Node: 26.0.0
• macOS 27.0

Reproduction

Run an external absolute-path grep against a package under an otherwise broad indexed tree:

ffgrep /allowRanges/ in ~/.pi/agent/npm/node_modules/pi-web-access/ limit 20

Intermittently, Pi remains on Working... indefinitely. The tool-level 10-second grep budget and the auxiliary finder startup timeout do not recover the process.

The target itself is small and the same search normally completes quickly, so this is not simply a slow full-tree scan.

Expected behavior

The search should return matches, No matches found, or a bounded actionable timeout without blocking Pi's main JavaScript thread.

Actual behavior

A synchronous native create/index/grep call can stop returning. Because the event loop is blocked inside FFI, an outer Promise.race/timer cannot fire and the whole Pi process remains stuck.

Investigation

Instrumentation of the pi-fff auxiliary finder lifecycle showed this sequence in the reproduced case:

1. A broader auxiliary finder exceeded its scan/startup wait.
2. Native scan/warmup continued after the JavaScript-side wait timed out.
3. The finder remained reusable while the global creation queue was released.
4. A later external-root grep entered native FFI while the earlier native work was still active.
5. Once the synchronous FFI call stopped returning, the JavaScript timeout could not interrupt it.

There may also be an interaction with:

• [Bug]: pi-fff reuses broader aux finder that excludes explicit absolute-path target #711 / fix(pi-fff): use exact aux finder for external paths #713 — reusing a broader finder can exclude an explicitly requested node_modules subtree
• [Bug]: grep timeBudgetMs is ignored for zero-match searches #826 / fix(grep): enforce_time_budget option for zero-match searches (#826) #827 — zero-match searches can skip the configured time budget
• [Bug]: broad absolute home-path ffgrep can appear to hang without returning a result #746 / fix(pi-fff): dedup concurrent aux finders and bound grep time (#746) #750 — related broad-path initialization hangs and same-root deduplication

Those fixes improve routing/budget behavior, but a JavaScript timer still cannot terminate a native synchronous call that has already wedged.

Locally verified mitigation

As a local diagnostic/fix, we moved external-path grep only behind a bounded child-process lifecycle:

• exact-root workers
• same-root startup deduplication
• serialized native finder creation across roots
• bounded hot-worker pool
• startup/request/abort/shutdown all terminate the child process when necessary
• workspace grep and fffind remain unchanged

Regression coverage includes cold/hot search, matches and zero matches, explicit node_modules roots, startup timeout, grep timeout, abort, concurrent roots, and recovery after termination.

With that isolation, the original target returns 38 matches in about 1.6 seconds, and a deliberately wedged native worker can be killed without freezing Pi.

I am filing the issue first because the preferred upstream fix may be either:

1. native cancellation/time-budget enforcement in the SDK/Rust owner, or
2. process/thread isolation in pi-fff for external-path operations.

I can submit the isolated pi-fff implementation if that direction is useful.

Logs

No useful log is emitted once the synchronous FFI call blocks; the JavaScript process cannot progress far enough to report the timeout.

[gustav-fff]

[triage-bot] REPRODUCED: mechanism confirmed on origin/main (f3647b7). Not the permanent wedge — the unbounded synchronous block that causes it.

Repro does not need pi. FileFinder.grep() is a plain synchronous FFI call (packages/fff-node/src/finder.ts:368 -> ffiLiveGrep), so nothing on the JS side can bound it.

cargo build --release -p fff-c
cd packages/fff-node && bun install && cd ../..
bun /tmp/repro843.ts   # broad basePath, waitForScan(5000), then grep(timeBudgetMs: 10_000)
                       # with a setTimeout(1000) watchdog armed before the grep call

create() returned sync after 304ms
waitForScan(5000) -> {"ok":true,"value":true} at 3160ms
grep() blocked the thread for 20169ms (budget 10000ms)
watchdog fired during grep: false
grep ok=true matches=0

Two separate defects, both required for your hang:

1. timeBudgetMs is not honoured — 20169ms observed against a 10000ms budget, zero matches. Same root cause as [Bug]: grep timeBudgetMs is ignored for zero-match searches #826, PR fix(grep): enforce_time_budget option for zero-match searches (#826) #827 open.
2. The 1000ms JS watchdog never fired during a 20s call. Confirms Promise.race / setTimeout is structurally incapable of bounding grep, find or FileFinder.create(). crates/fff-c/src/lib.rs:671 hardcodes abort_signal: None, and fff-c exports no cancel entry point at all, so GrepSearchOptions.abort_signal (crates/fff-core/src/grep/types.rs:112) is unreachable from Node.

No PR. #827 fixes (1). (2) needs either a new fff_abort-style C export wired through fff-node, or your child-process isolation — both are API-surface decisions, not a triage-sized patch. @dmtrKovalenko your call on direction; #827 alone will not make the process interruptible.

Honk-Honk 🪿

[dmtrKovalenko]

This doesn't describe "indefinetely" hold @gustav-fff do one more try or give me a way to reproduce it

[gustav-fff]

[triage-bot] DIRECTED: you are right — I cannot produce an unbounded hold. 20s is not "indefinite". Below: the deterministic recipe for the long block, three "indefinite" theories I killed, and the one genuinely unbounded path left.

Deterministic repro of the long block (not infinite)

The block length depends entirely on whether the bigram index is installed yet. Same corpus, same query, same budget, two runs on origin/main (f3647b7), M-series macOS, /Users/x/dev/chromium = 498548 files:

cargo build --release -p fff-c
cd packages/fff-node && bun install && cd ../..

// /tmp/repro843.ts
import { FileFinder } from "./packages/fff-node/src/index.ts";
const f = FileFinder.create({ basePath: "/Users/x/dev/chromium", aiMode: true }).value;
await f.waitForScan(Number(process.argv[3])); // 5000 = index NOT installed yet
await new Promise(r => setTimeout(r, Number(process.argv[4]))); // settle ms
let fired = false; const t = setTimeout(() => { fired = true }, 1000);
const t0 = Date.now();
const r = f.grep("zzz_no_such_token_qqq", { mode: "plain", pageSize: 20, timeBudgetMs: 10_000 });
console.log(`grep blocked ${Date.now() - t0}ms, watchdog fired: ${fired}, matches=${r.value.items.length}, filesSearched=${r.value.totalFilesSearched}/${r.value.totalFiles}`);

bun /tmp/repro843.ts x 5000 0       -> grep blocked 20169ms, watchdog fired: false, matches=0, filesSearched=498548/498548
bun /tmp/repro843.ts x 600000 15000 -> grep blocked   295ms, watchdog fired: false, matches=0, filesSearched=6720/498548

Two defects stack in the first row:

1. No prefilter yet -> all 498548 files get opened/mmapped/memmem'd.
2. crates/fff-core/src/grep/grep.rs:610 — && all_matches.len() > 1 gates the budget check, so a zero-match query never evaluates search_start.elapsed() > budget. The 10000ms budget is dead. Same root cause as [Bug]: grep timeBudgetMs is ignored for zero-match searches #826.

So the duration is O(corpus content), not O(timeBudgetMs). Broad root + cold page cache + a zero-match query = minutes of a fully blocked JS thread with no log line. That reads as "indefinite" to a user but it does terminate. abort_signal is still hardcoded None at crates/fff-c/src/lib.rs:671 and :753, and no cancel symbol is exported, so nothing can shorten it.

Ruled out (so you don't redo these)

• Blocking read on a special file (FIFO/device wedging a rayon worker forever). Walker rejects non-regular entries (crates/fff-core/src/walk/zlob.rs:54, walk/ripgrep.rs:51) and crates/fff-core/src/types.rs:753 bails on size == 0, which every special file reports. Verified: mkfifo inside an indexed dir is never indexed, grep returns in 1ms.
• watch callback vs dispatcher.drain() (crates/fff-core/src/watcher/watch.rs:344, recv() with no timeout, reached from fff_destroy and from rebase_watches). Would deadlock iff the fff-watch-callback thread blocks inside the C callback waiting for the JS loop. It does not — ffi-rs delivers the trampoline via a non-blocking threadsafe fn. Verified: queued watch event + JS thread spun for 500ms, then destroy() returned in 1ms.
• LMDB reader-slot exhaustion. clear_stale_readers() runs on every env open (crates/fff-core/src/dbs/env_pool.rs), and the env is pooled per canonical path so a second finder joins instead of reopening.

The one unbounded path I did find — unproven, needs your call

crates/fff-core/src/dbs/lmdb.rs:181-187 — you already wrote // unfortunately this CAN be deadlocking. env.write_txn() there has no timeout, and it is reached synchronously from FileFinder.create(): crates/fff-c/src/lib.rs:219 FrecencyTracker::open (same for QueryTracker). pi-fff opens every picker — cwd and each aux root — on the same two dbs (packages/pi-fff/src/file-picker.ts:60-61), and fff.nvim shares them too.

lmdb-master-sys 0.2.5 builds with default features, so MDB_USE_POSIX_SEM is off and the writer lock is a process-shared pthread mutex. macOS has no PTHREAD_MUTEX_ROBUST, so MDB_ROBUST_SUPPORTED is undefined there (mdb.c:338-346): a process that dies holding the writer mutex leaves it locked forever, and every later write_txn() in every process blocks in pthread_mutex_lock with no recovery. Matches the report exactly — macOS only, intermittent, no log line, survives the JS timeout, unaffected by #827.

Only hit when the db must be created (first run / after erase_if_oversized), which is why it is rare.

@felixkang to confirm which of the two it is, next time Pi wedges, before killing it:

sample $(pgrep -f 'pi ' | head -1) 10 -file /tmp/pi-wedge.txt

If the main thread sits in pthread_mutex_lock under mdb_txn_begin it is the LMDB path and it is truly permanent. If it sits in fff_live_grep/perform_grep/memmem it is the budget bypass above and #827 plus a real abort export fixes it. Attach the file.

@dmtrKovalenko no PR from me — the grep half is #827's, and the LMDB half is an API/ownership decision (bounded write_txn, or drop the create-time write txn, or process isolation as the reporter proposes).

Honk-Honk 🪿