feat(mapper): support incremental mapping via --since and --include-d…

[Tanmay-008]

Description

This PR introduces incremental mapping support for the clawpatch map command. By using git diffs to identify modified files, the mapper now only re-evaluates and updates features that have actually changed.

This addresses a major bottleneck in CI and local workflows on large codebases where scanning 10,000+ files for a 2-file change was previously required.

Changes Made

• CLI Arguments: Added --since and --include-dirty flags to src/cli.ts for the map command.
• Seed Filtering: Updated MapOptions and mapFeatureSeeds in src/mapper.ts to filter out seeds whose entryPath or ownedFiles do not intersect with the changed files.
• Smart Write Phase: Modified src/app.ts to ensure that unchanged features are preserved exactly as-is (including their locks and status). Only features that were part of the diff but are no longer present in the new seeds are marked as skipped.

Testing & Validation

• All 877 automated tests passing.
• Typecheck and oxlint run without warnings.
• Verified locally that unchanged features are not overwritten or marked as skipped when running clawpatch map --since HEAD~1.

📈 Impact

Expect up to a 10x reduction in mapping time for iterative changes!

Closes #171

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed August 1, 2026, 9:42 PM ET / August 2, 2026, 01:42 UTC.

ClawSweeper review

What this changes

The branch adds clawpatch map --since and --include-dirty, threads Git-selected paths into mapping reconciliation, and preserves untouched stored feature records.

Merge readiness

⛔ Blocked until real behavior proof is added - 10 items remain

This PR remains the active candidate for the paired incremental-mapping feature request, so it should stay open. It is not ready to merge: changed-file filtering occurs after heuristic discovery and does not constrain the agent mapper’s inventory or provider prompt, while no inspectable real-run proof demonstrates the claimed scope reduction.

Priority: P2
Reviewed head: 2378c6e074bd46ea03eafaabf67f64ab5e0b9626
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🧂 unranked krab (1/6)	The patch has a focused intent and clean security surface, but core scope-reduction defects and absent real-run proof keep it from merge readiness.
Proof confidence	🧂 unranked krab (1/6)	Needs real behavior proof before merge: The PR claims local verification but provides no inspectable after-fix terminal transcript, recording, live output, or redacted log showing reduced heuristic/provider scope and preserved untouched records. Add redacted evidence to the PR body; after updating it, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.
Patch quality	🦪 silver shellfish (2/6)	3 actionable review findings remain.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs real behavior proof before merge: The PR claims local verification but provides no inspectable after-fix terminal transcript, recording, live output, or redacted log showing reduced heuristic/provider scope and preserved untouched records. Add redacted evidence to the PR body; after updating it, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.
Evidence reviewed	6 items	Repository policy applied: The full repository policy requires focused mapper/workflow coverage and user-facing documentation for behavior changes; that guidance informed the findings. Heuristic selection is too late: At the PR head, mapFeatures calls collectSeeds before mapFeatureSeeds; the newly added changed-file check cannot prevent repository-wide heuristic seed discovery. Agent provider input remains unscoped: The PR threads changed paths into reconciliation, but agentMap constructs the provider prompt from the complete inventory first, leaving unchanged repository context in provider work.
Findings	3 actionable findings	[P1] Filter before running every heuristic mapper [P1] Scope the agent inventory and prompt to changed paths [P2] Document the incremental map flags
Security	None	None.

How this fits together

clawpatch map detects a project, runs heuristic mappers and an optional provider-backed agent mapper, then reconciles their feature records with persisted .clawpatch state. The proposed flags obtain changed paths from Git and are intended to constrain this pipeline without modifying untouched records.

flowchart LR
  A[Map CLI flags] --> B[Git changed paths]
  B --> C[Mapping workflow]
  C --> D[Heuristic discovery]
  C --> E[Agent inventory and prompt]
  D --> F[Feature reconciliation]
  E --> F
  F --> G[Stored feature records]

Loading

Decision needed

Question	Recommendation
Should clawpatch map support incremental mapping as a first-class CLI mode with an explicit performance and compatibility contract?	Sponsor and repair the incremental mode: Keep the opt-in CLI mode, require pre-discovery scoping, compatibility tests, documentation, and real-run proof before merge.

Why: The repair is mechanically bounded, but accepting these new flags commits the project to a user-facing workflow and stated performance behavior that need maintainer sponsorship.

Before merge

• Add real behavior proof - Needs real behavior proof before merge: The PR claims local verification but provides no inspectable after-fix terminal transcript, recording, live output, or redacted log showing reduced heuristic/provider scope and preserved untouched records. Add redacted evidence to the PR body; after updating it, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.
• Filter before running every heuristic mapper (P1) - mapFeatures still performs collectSeeds(root, options) before the new changed-file check in mapFeatureSeeds, so --since can traverse and run every mapper across the repository before discarding untouched seeds. Restrict discovery itself to the changed paths before collecting seeds.
• Scope the agent inventory and prompt to changed paths (P1) - The new changedFiles value reaches only reconciliation; agentMap builds the provider prompt from the full inventory, so a small incremental run still sends unchanged repository context to the provider. Filter the inventory before buildAgentMapPrompt and cover that boundary with a regression test.
• Document the incremental map flags (P2) - The branch adds public map --since and --include-dirty behavior but changes no user-facing map documentation. Document their Git-base and dirty-worktree semantics, plus that they preserve untouched feature records, so users can safely adopt the new mode.
• Resolve merge risk (P1) - Merging would advertise an incremental mode intended to avoid full scans while heuristic seed discovery and agent provider input can still cover the full repository.
• Resolve merge risk (P1) - The reconciliation changes alter which persisted feature records are refreshed or marked skipped, so changed, deleted, and untouched record behavior needs explicit upgrade-safe coverage.
• Improve patch quality - Move changed-file selection ahead of heuristic discovery and agent prompt creation.
• Improve patch quality - Add regressions for scoped heuristic discovery, scoped agent inventory, and untouched-record preservation.
• Improve patch quality - Post a redacted terminal or live-output incremental-run proof; remove private paths, endpoints, and credentials.

Findings

• [P1] Filter before running every heuristic mapper — src/mapper.ts:77-78
• [P1] Scope the agent inventory and prompt to changed paths — src/agent-mapper.ts:227-231
• [P2] Document the incremental map flags — src/cli.ts:134-143

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Changed surface	6 files; 106 additions, 5 deletions	The branch changes command parsing, map orchestration, both mapper paths, stored-state reconciliation, one workflow test, and a release note.
New regression coverage	1 workflow test added	It proves record preservation but does not prove reduced heuristic or provider scope.

Merge-risk options

Maintainer options:

1. Repair before merge (recommended)
   Apply changed-path selection before heuristic discovery and agent inventory/prompt construction, then add upgrade-safe regressions and real-run proof.
2. Pause for product direction
   Do not merge the new CLI mode until a maintainer confirms the performance and compatibility contract users should rely on.

Copy recommended automerge instruction

@clawsweeper automerge

Special instructions:
Apply changed-file selection before mapper discovery and agent inventory/prompt construction; add focused scope and untouched-record regression coverage; document the flags; attach a redacted real-run transcript.

Technical review

Best possible solution:

If maintainers sponsor the CLI mode, apply changed-path selection before heuristic discovery and agent inventory/prompt construction; preserve the full-map default; document both flags; and prove reduced scope plus unchanged-record safety with focused tests and a redacted real run.

Do we have a high-confidence way to reproduce the issue?

Yes for the source-level defect: mapFeatures collects seeds before the new changed-file filter can act, and the provider path builds its inventory before reconciliation. A real fixture run is still needed to demonstrate the end-to-end performance impact.

Is this the best way to solve the issue?

No: filtering generated feature seeds after discovery cannot fulfill a feature whose stated purpose is avoiding repository-wide mapper and provider work; selection must happen before those stages.

Full review comments:

• [P1] Filter before running every heuristic mapper — src/mapper.ts:77-78
mapFeatures still performs collectSeeds(root, options) before the new changed-file check in mapFeatureSeeds, so --since can traverse and run every mapper across the repository before discarding untouched seeds. Restrict discovery itself to the changed paths before collecting seeds.
  Confidence: 0.99
• [P1] Scope the agent inventory and prompt to changed paths — src/agent-mapper.ts:227-231
  The new changedFiles value reaches only reconciliation; agentMap builds the provider prompt from the full inventory, so a small incremental run still sends unchanged repository context to the provider. Filter the inventory before buildAgentMapPrompt and cover that boundary with a regression test.
  Confidence: 0.98
• [P2] Document the incremental map flags — src/cli.ts:134-143
  The branch adds public map --since and --include-dirty behavior but changes no user-facing map documentation. Document their Git-base and dirty-worktree semantics, plus that they preserve untouched feature records, so users can safely adopt the new mode.
  Confidence: 0.96

Overall correctness: patch is incorrect
Overall confidence: 0.98

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 4a5028c27f01.

Labels

Label changes:

• add merge-risk: 🚨 compatibility: The branch changes when existing persisted feature records are refreshed or transitioned to skipped during map runs.

Label justifications:

• P2: This is a bounded developer-workflow feature with incomplete promised behavior, not an urgent production regression.
• merge-risk: 🚨 compatibility: The branch changes when existing persisted feature records are refreshed or transitioned to skipped during map runs.
• rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦪 silver shellfish.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR claims local verification but provides no inspectable after-fix terminal transcript, recording, live output, or redacted log showing reduced heuristic/provider scope and preserved untouched records. Add redacted evidence to the PR body; after updating it, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.

Evidence

What I checked:

• Repository policy applied: The full repository policy requires focused mapper/workflow coverage and user-facing documentation for behavior changes; that guidance informed the findings. (AGENTS.md:1, 4a5028c27f01)
• Heuristic selection is too late: At the PR head, mapFeatures calls collectSeeds before mapFeatureSeeds; the newly added changed-file check cannot prevent repository-wide heuristic seed discovery. (src/mapper.ts:77, 2378c6e074bd)
• Agent provider input remains unscoped: The PR threads changed paths into reconciliation, but agentMap constructs the provider prompt from the complete inventory first, leaving unchanged repository context in provider work. (src/agent-mapper.ts:212, 2378c6e074bd)
• Added test covers only preservation: The new workflow test checks that an untouched agent-mapped record is not skipped, but does not assert that heuristic discovery or agent inventory/prompt input excludes unchanged paths. (src/workflow.test.ts:1836, 2378c6e074bd)
• Current-main provenance: Current main is 4a5028c, while the proposed implementation is only on open branch commit 2378c6e; the central feature is not already implemented on the default branch. (src/app.ts:72, 2378c6e074bd)
• Documentation gap: Current user documentation describes clawpatch map but has no map-command documentation for --since, --include-dirty, or incremental mapping semantics. (docs/feature-mapping.md:12, 4a5028c27f01)

Likely related people:

• Peter Steinberger: The available current-main blame attributes the central mapping, agent-prompt, and workflow orchestration paths to the v0.7.1 source boundary commit. (role: current-main mapping-path author; confidence: medium; commits: f6e00010cc2a; files: src/mapper.ts, src/agent-mapper.ts, src/app.ts)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (12 earlier review cycles; latest 8 shown)

• reviewed 2026-07-31T23:34:45.818Z sha 2378c6e :: needs real behavior proof before merge. :: [P1] Short-circuit empty incremental agent maps | [P1] Apply selection before mapper discovery | [P2] Document the new map flags
• reviewed 2026-08-01T07:11:08.604Z sha 2378c6e :: needs real behavior proof before merge. :: [P1] Filter before running every heuristic mapper | [P1] Scope agent inventory and prompts to changed paths | [P2] Document the new map flags
• reviewed 2026-08-01T09:35:26.637Z sha 2378c6e :: needs real behavior proof before merge. :: [P1] Filter before running every heuristic mapper | [P1] Scope agent inventory and prompts to changed paths | [P2] Document the new map flags
• reviewed 2026-08-01T12:45:08.005Z sha 2378c6e :: needs real behavior proof before merge. :: [P1] Filter before collecting all mapper seeds | [P1] Scope agent inventory and prompts to selected paths | [P2] Document the new map flags
• reviewed 2026-08-01T14:27:49.124Z sha 2378c6e :: needs real behavior proof before merge. :: [P1] Scope discovery and agent inventory to changed paths | [P2] Document the new map flags
• reviewed 2026-08-01T20:13:55.381Z sha 2378c6e :: needs real behavior proof before merge. :: [P1] Filter before collecting all mapper seeds | [P1] Scope agent inventory and prompts to changed paths | [P2] Document the new map flags
• reviewed 2026-08-01T21:09:17.330Z sha 2378c6e :: needs real behavior proof before merge. :: [P1] Apply changed-file selection before mapper discovery | [P1] Scope the agent inventory before building its prompt | [P2] Document the new map flags
• reviewed 2026-08-01T22:37:40.915Z sha 2378c6e :: needs real behavior proof before merge. :: [P1] Filter before running every heuristic mapper | [P1] Scope the agent inventory and prompt to changed paths | [P2] Document the incremental map flags

[steipete]

I built and live-tested this branch. Triage outcome: recommend CLOSE.

The focused checks pass (121 workflow tests, formatting, type-checking, linting, and production build), but the real incremental path violates both core promises:

1. A built-CLI map --source heuristic --since origin/main --dry-run --json with six changed files still emitted all 17 mapper-start events. The implementation runs every heuristic mapper and filters seeds afterward, so it does not avoid the discovery work it claims to optimize.
2. More importantly, after a baseline map produced 36 features, that incremental dry run returned five filtered features and planned stale: 31. Untouched durable features are being reconciled as absent from a complete map and would be marked stale by a non-dry run.

This needs a persisted incremental-reconciliation contract and mapper-specific scoped discovery, not output filtering after a full map. Landing the current branch would risk corrupting feature status on ordinary incremental runs.