perf(mapper): cache full-root filesystem scans across concurrent mappers

[Tanmay-008]

Description

Fixes the redundant full-root filesystem traversals reported in #158 while preserving each mapper's existing directory-pruning behavior.

The original Promise-based VFS cache reduced raw calls, but real built-CLI proof showed that retaining Stats and directory state made mapping slower and more memory-intensive. This revision instead performs one policy-aware traversal per mapping run. Go fallback, C/C++, and .NET receive separate sorted file views from the same physical walk, so their existing skip policies still prune directories before readdir and lstat work.

Changes

• Add a mapping-run-scoped, memoized root-file inventory to MapperContext.
• Traverse the root once while carrying the Go fallback, C/C++, and .NET skip policies independently.
• Keep the ordinary walk() API and error semantics intact for other consumers.
• Add regression coverage for concurrent inventory sharing and ancestor-aware per-policy pruning.
• Credit @Tanmay-008 in the unreleased changelog and final maintainer rewrite.

Real behavior proof

Built with pnpm build, then ran the packaged CLI three times against the same disposable 18,022-file polyglot repository containing Go fallback, C++, and C# inputs:

• Current main (041bf4bfd670): 1.27s, 1.28s, 2.43s; peak memory about 157-166 MB.
• Original VFS-cache head (20e6eeaea8e6): 4.42s, 6.04s, 5.83s; peak memory about 176-190 MB.
• Revised policy-aware head (b4d0c256fa61): 1.12s, 1.11s, 1.09s; peak memory about 157-158 MB.

All runs mapped the same four features with Go fallback and the C/C++ and .NET mappers active.

Validation

• pnpm typecheck
• pnpm lint
• pnpm format:check
• pnpm test
• pnpm build
• pnpm pack:smoke
• AutoReview local repair: clean, no accepted/actionable findings (correctness 0.94).
• AutoReview complete branch against origin/main: clean, no accepted/actionable findings (correctness 0.92).

Fixes #158

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed July 17, 2026, 10:38 PM ET / July 18, 2026, 02:38 UTC.

Summary
The PR adds a mapping-run-scoped Promise cache for directory reads, stats, and realpath calls, then shares it among Go fallback, C/C++, and .NET full-root walks.

Reproducibility: no. high-confidence runtime reproduction is provided. Source and the PR describe the three concurrent full-root traversal paths, but there is no observed current-main syscall trace or representative live benchmark.

Review metrics: 2 noteworthy metrics.

• Patch surface: 2 added, 8 changed files; 155 additions, 29 deletions. The change spans the shared traversal helper, mapper context, and three consumers, so live behavior proof is more informative than unit coverage alone.
• Targeted consumers: 3 full-root mapper paths. Go fallback, C/C++, and .NET are the intended concurrent scan consumers of the shared cache.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #158
Summary: This PR is the candidate implementation for the open report about redundant concurrent full-root scans; the report should remain open until this branch merges.

Members:

• canonical: Redundant full-root filesystem scans across concurrent mappers #158 - The issue describes the same Go, C/C++, and .NET concurrent traversal problem and is explicitly referenced as the target of this PR.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

• [P1] Add redacted terminal evidence comparing current behavior with the branch on a representative polyglot workspace, preferably including syscall counts or equivalent traversal metrics.
• Show the Go fallback path is exercised alongside C/C++ and .NET mapping, then update the PR body with the observed result.

Proof guidance:

• [P1] Needs real behavior proof before merge: Focused mocked regression tests and green checks are present, but this external PR needs a redacted after-fix live CLI benchmark, syscall trace, or runtime transcript showing the shared traversal result before merge; update the PR body afterward, or ask a maintainer to comment @clawsweeper re-review if a fresh review does not trigger.

Risk before merge

• [P1] The claimed reduction in filesystem calls and behavior on a real large polyglot workspace remain unverified; mocked tests cannot establish the runtime performance result or catch environment-specific traversal effects.

Maintainer options:

1. Decide the mitigation before merge
   Keep the cache scoped to one mapFeatures run, then add a redacted live terminal benchmark or syscall-count comparison that exercises Go fallback together with C/C++ and .NET discovery on a representative workspace.
2. Pause or close
   Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

• [P1] The remaining merge blocker is contributor-supplied real behavior proof, not a concrete code or test repair that an automated fix lane can perform.

Security
Cleared: The patch only wraps existing Node filesystem promises in in-memory per-run maps and adds no dependency, workflow, secret, permission, or code-execution surface.

Review details

Best possible solution:

Keep the cache scoped to one mapFeatures run, then add a redacted live terminal benchmark or syscall-count comparison that exercises Go fallback together with C/C++ and .NET discovery on a representative workspace.

Do we have a high-confidence way to reproduce the issue?

No high-confidence runtime reproduction is provided. Source and the PR describe the three concurrent full-root traversal paths, but there is no observed current-main syscall trace or representative live benchmark.

Is this the best way to solve the issue?

Likely yes: a run-scoped cache below the shared traversal helper is a narrow way to remove duplicate raw filesystem work while retaining mapper-specific filters. A live comparison must still confirm that the intended performance gain and traversal behavior hold outside mocks.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 041bf4bfd670.

Label changes

Label justifications:

• P2: This is a bounded mapper-performance improvement with no demonstrated urgent user-facing regression.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: Focused mocked regression tests and green checks are present, but this external PR needs a redacted after-fix live CLI benchmark, syscall trace, or runtime transcript showing the shared traversal result before merge; update the PR body afterward, or ask a maintainer to comment @clawsweeper re-review if a fresh review does not trigger.

Evidence reviewed

What I checked:

• Current-main gap: The PR adds createVfsCache() to the collectSeeds mapper context and threads context.vfs through the full-root walks; this capability is not part of the current default-branch implementation identified by the supplied main SHA. (src/mapper.ts:295, 041bf4bfd670)
• Focused implementation: The exact PR head introduces a per-run cache and uses it only for raw filesystem operations beneath existing walk() traversal and mapper-specific filtering, preserving the existing mapper orchestration boundary. (src/mappers/vfs-cache.ts:10, 20e6eeaea8e6)
• Regression coverage: The branch adds focused cache tests, including preservation of rejected readdir calls, and the provided check set reports successful TypeScript analysis, tests, CodeQL, and secret scanning. (src/mappers/vfs-cache.test.ts:16, 20e6eeaea8e6)
• Proof gate remains unmet: The PR body describes expected syscall reduction and mocked regression coverage, but does not include an after-fix CLI run, benchmark, syscall trace, or redacted runtime transcript from a representative polyglot workspace. The prior review also requested this exact proof. (20e6eeaea8e6)
• Related canonical report: The open performance report remains the canonical user-problem tracker; this PR is its candidate implementation and should not cause the report to close until the branch lands. (20e6eeaea8e6)

Likely related people:

• Tanmay-008: The related mapper-context performance work and this exact cache branch both center on the context and mapper traversal surface. (role: recent mapper-context contributor; confidence: medium; commits: 481ee4a5e27c, 20e6eeaea8e6; files: src/mappers/context.ts, src/mappers/types.ts, src/mappers/shared.ts)

What the crustacean ranks mean

• 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
• 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
• 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
• 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
• 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
• 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
• 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (3 earlier review cycles)

• reviewed 2026-07-14T12:05:43.423Z sha 7096744 :: needs real behavior proof before merge. :: [P2] Preserve directory-read failures in the cache
• reviewed 2026-07-14T18:03:21.915Z sha 20e6eea :: needs real behavior proof before merge. :: none
• reviewed 2026-07-15T23:10:08.826Z sha 20e6eea :: needs real behavior proof before merge. :: none

[Tanmay-008]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

• State: Complete
• Detail: The targeted re-review published an exact durable review comment and queued an executing target-wide serialized router scan.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/29355879250
• Updated: 2026-07-14T18:05:54.751Z

[steipete]

Landed on main as 03c2556a26b3d57cbaac673550730e69b70827d7 via rebase merge. The landed tree exactly matches the reviewed PR head tree.

Tested on exact PR head b4d0c256fa6170b11cf893f47f3e4fa896be0741:

• pnpm typecheck
• pnpm lint
• pnpm format:check
• pnpm test
• pnpm build
• pnpm pack:smoke
• Built CLI against an 18,022-file polyglot fixture with Go fallback, C++, and C#: revised head 1.12s / 1.11s / 1.09s versus current-main baseline 1.27s / 1.28s / 2.43s. The rejected VFS-cache design measured 4.42s / 6.04s / 5.83s and was rewritten.
• AutoReview of the local repair: clean, no accepted/actionable findings (0.94 correctness).
• AutoReview of the complete branch diff: clean, no accepted/actionable findings (0.92 correctness).
• CI run 29794209287: success.
• CodeQL run 29794209221: success.
• Secret-scanning run 29794209227: success.

The implementation preserves mapper-specific traversal pruning and keeps @Tanmay-008's authored commits, changelog credit, and co-author credit on the final maintainer rewrite.