Redundant full-root filesystem scans across concurrent mappers

[Tanmay-008]

Problem

Currently, when clawpatch orchestrates the mapping process, it launches all 17 feature mappers concurrently. Three of these mappers — Go, C/C++, and Dotnet — perform full-root filesystem scans (walk(root, [""])) to discover files without any early exit conditions (like checking for .sln or CMakeLists.txt), because they support mapping loose source files.

Because these mappers run in parallel and there is no shared file state, they trigger 3 independent, overlapping filesystem traversals. In large polyglot monorepos (e.g., 50,000+ files), this results in a massive 3x redundant disk I/O overhead (150,000+ lstat calls) just to build the directory tree, causing a significant performance bottleneck during initialization.

Proposed Solution

Introduce a Virtual File System (VFS) Cache to share filesystem I/O across all concurrent mappers:

Memoize raw node:fs calls (readdir, lstat, realpath) using Promise-based Maps inside the MapperContext.
Inject this VFS cache into the shared walk() helper.
Update the Go, C/C++, and Dotnet mappers to pass the VFS cache down to their respective walk calls.
This ensures that the first mapper to traverse a directory pays the actual disk I/O cost, while the subsequent mappers instantly receive the cached memory representation. Language-specific filters (e.g., shouldSkipCOrCppPath) will continue to run normally, but entirely in memory

[clawsweeper]

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve. Reviewed July 16, 2026, 9:19 PM ET / July 17, 2026, 01:19 UTC.

Summary
Keep this issue open because current main still has the reported independent full-root traversals, and the explicitly linked pull request remains the active unmerged implementation candidate.

Reproducibility: yes. at the source level: concurrent orchestration reaches three independent full-root walker calls without shared filesystem state. A measured real-repository performance reproduction is still missing.

Root-cause cluster
Relationship: canonical
Canonical: #158
Summary: This issue is the canonical performance report, and the linked pull request is its active candidate fix.

Members:

• fixed_by_candidate: perf(mapper): cache full-root filesystem scans across concurrent mappers #159 - The open pull request explicitly targets the three redundant traversal paths and links this issue with closing syntax.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Next step
The open linked pull request already owns this bounded implementation, so dispatching another fix branch would duplicate contributor work.

Review details

Best possible solution:

Land a narrowly scoped shared per-run filesystem cache after the linked pull request demonstrates reduced filesystem calls or initialization time on a representative polyglot tree and confirms unchanged traversal and error behavior.

Do we have a high-confidence way to reproduce the issue?

Yes at the source level: concurrent orchestration reaches three independent full-root walker calls without shared filesystem state. A measured real-repository performance reproduction is still missing.

Is this the best way to solve the issue?

Unclear until validated in a real setup; a per-run shared cache is a focused approach, but it should prove actual I/O savings and preserve traversal, memory, and error semantics.

AGENTS.md: found and applied where relevant.

Remaining risk / open question:

• The reported syscall count and approximately 66% reduction have not yet been demonstrated through a real before-and-after run.
• Shared Promise caching must preserve filesystem error semantics and avoid excessive retained directory metadata during a mapping run.

Codex review notes: model internal, reasoning high; reviewed against 041bf4bfd670.

Label changes

Label justifications:

• P2: The redundant scans are a bounded mapper performance problem that can materially slow large repositories without breaking correctness or core availability.
• impact:other: The issue concerns avoidable filesystem and initialization overhead, which is meaningful maintainer-visible impact outside the more specific owned categories.

Evidence reviewed

What I checked:

• Current-main behavior: The supplied current-main review context confirms concurrent mapper orchestration still reaches independent uncached full-root walks for Go, C/C++, and .NET. (src/mappers/shared.ts, 041bf4bfd670)
• Affected callers: The reported source paths identify three loose-file mapper fallbacks that independently invoke the shared walker without shared filesystem state. (src/mappers/go.ts, 041bf4bfd670)
• Open implementation candidate: The clean, open perf(mapper): cache full-root filesystem scans across concurrent mappers #159 explicitly links this issue and introduces a shared Promise-based filesystem cache across the affected mapper paths; it has not merged yet. (src/mappers/vfs-cache.ts, 20e6eeaea8e6)
• Release status: The latest supplied release is v0.7.0 at 0cd24d0, while the reported behavior remains on current main, so no shipped implementation supports closing this issue. (0cd24d07a262)
• Linked PR proof state: The linked pull request is labeled as needing proof; its unit regression coverage is useful, but the supplied discussion does not contain a measured after-fix run demonstrating syscall or initialization-time reduction in a real polyglot repository. (20e6eeaea8e6)

Likely related people:

• Tanmay-008: Authored the active pull request that adds the shared VFS cache and updates the affected mapper paths and regression coverage. (role: linked implementation author; confidence: high; commits: 20e6eeaea8e6; files: src/mappers/vfs-cache.ts, src/mappers/shared.ts, src/mappers/go.ts)

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

[clawsweeper]

ClawSweeper status: review started.

I am starting a fresh review of this issue: Redundant full-root filesystem scans across concurrent mappers This is item 1/1 in the current shard. Shard 0/1.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.