feat(indexing): include symlinked reference repositories in parent graphs

.trail/learnings/2026-11-11-explicit-repository-mounts-need-boundary-checks-atif.md

@@ -0,0 +1,25 @@
+---
+title: Explicit repository mounts need boundary checks and retarget detection
+author: Atif
+date: 2026-11-11
+branch: atif_symlink
+id: l-467468cd
+scope: repo
+status: active
+src: agent
+touches:
+  - src/ingest/fs.ts
+  - src/graph/fingerprint.ts
+  - test/graph-follow-nested-repos.test.ts
+---
+
+## Decided
+Review of atif_symlink at a18b1bc found two reproducible gaps before merging. Preserve main's independent submodule and nested-repository opt-ins when admitting explicit mounts. Capture physical repository targets during the existing file walk and persist them in the fingerprint. A changed target makes freshness unknown, forcing one rebuild that hashes source bytes and saves the new identity. This avoids trusting identical file stats from a different checkout without adding another enumeration or hashing every file on every query. Keep older fingerprints readable so their saved filters survive the one-time identity-aware rebuild.
+
+## Tried and ruled out
+The 37 existing ingest, nested-repo, only-dir and context-only-dir tests all passed, but do not establish these edge cases. A temporary Git fixture with a mode-160000 entry returned no files with followNestedRepos=true and followSubmodules=false; adding that path to includeIgnoredRepos returned the child's source. A second fixture built a linked repo then retargeted the link to another repo with different equal-length source and equal mtimes. Default probeDrift reported clean; GRAFT_REFRESH=hash detected the changed file. Therefore file-list parity and ordinary edit-refresh tests alone are insufficient.
+
+## Watch out
+Keep logical mount paths for graph IDs and physical checkout paths for cycle/target identity. Do not replace recursion-stack cycle detection with global realpath deduplication without deciding how multiple mount aliases should behave. The stat shortcut predates this branch; symlink retargeting gives it a new trigger. Explicit mount admission must not change a gitlink into a nested clone: classify first, then apply the independent traversal gate. Record target identity before invoking Git so filesystem fallback also retains it.
+
+In the code: src/ingest/fs.ts explicit entries and followChild; src/graph/fingerprint.ts probeDrift and statUnchanged; test/graph-follow-nested-repos.test.ts. Run the four targeted test files with node --import tsx --test and test-local Git signing disabled.

README.md

@@ -699,9 +699,16 @@ trail build -e .ts .py               # only index these extensions (also limits
 trail build --no-reuse               # re-parse every file instead of replaying unchanged ones from cache
 trail build --follow-submodules      # include initialized submodules; persist the choice for builds + MCP refresh
 trail build --no-follow-submodules   # exclude submodules again and persist that choice (the default)
-trail build --follow-nested-repos    # include nested git clones the index doesn't track; persist the choice
+trail build --follow-nested-repos    # include nested git clones and symlinked repos; persist the choice
 trail build --no-follow-nested-repos # exclude nested clones again and persist that choice (the default)
 
+# Index normal project source plus selected repo mounts (child ignores still apply).
+trail build --follow-nested-repos \
+  --include-repo source-material/historical-repos/widget \
+  --include-repo source-material/historical-repos/databraid-api
+trail build --all-dirs                # clear a saved --only-dir restriction; keep selected repo mounts
+trail build --no-include-ignored-repos # clear the saved ignored-repository mounts
+
 trail status [dir]                   # the graft map, learnings and tokens saved, on one screen
 trail upgrade                        # npm install -g the latest published version
                                      # a new version is announced automatically (checked once a day);
@@ -791,12 +798,17 @@ The graft map supports these layouts:
   refreshes behave the same way. Run `trail build --no-follow-submodules` to
   restore and persist the default boundary.
 - **A git repo with other repos cloned inside it** (no gitlink, no index entry): the shape multi-repo manifest tools like `west`, `repo`, `gclient` and `tsrc` check dependencies out into, and the shape you get by cloning an upstream into the tree to patch it locally. `--follow-submodules` cannot reach these: they have no `160000` index entry to follow. Run `trail build --follow-nested-repos` to fold them into one graph, prefixing child paths (for example, `external/parser/src/index.ts`) while honoring each clone's own Git ignore rules. A clone at a git-ignored path stays absent, since Git never reports it. The choice is saved in `.graft/config.json` and is independent of `--follow-submodules`; neither flag implies the other. Run `trail build --no-follow-nested-repos` to restore and persist the default boundary. Prefer this over the multi-repo split below when the nested repos import from each other and you want those edges in one graph; prefer the split when you want each repo scored and refreshed on its own.
+- **A Git repo containing symlinks to other repos** (for example, a review folder with `repos/api` and `repos/frontend` pointing to separate checkouts) — run `trail build --follow-nested-repos` at the review folder's root. Git-visible links to directories with their own `.git` are followed, including linked worktrees. The single graph stays at the review folder's `graft/`, with paths such as `repos/api/src/index.ts`, so agents and queries opened at that root use the same index. Each linked repo keeps its own ignore rules; broken links, cycles, file links, and links to ordinary directories are skipped. The saved option also applies to later builds and automatic refreshes.
 - **A folder of separate git repos** (no `.git` at the top): `trail build`
   auto-splits: each child gets its own (git-ignored) `graft/`, and the parent
   gets a `graft/workspace.json` index. Queries from the parent federate across
   every child, always labeled `<child>/`. Run `trail build` inside a child to
   work on just that repo.
 
+To index the normal project source alongside selected linked repositories, use `--follow-nested-repos` and repeat `--include-repo <path>` for each mount, including mounts the parent deliberately Git-ignores. Every selected path must be a repository under the build root. Graft saves these explicit mounts in local `.graft/config.json`, preserves the parent's `.gitignore`, and continues to honor each child's ignore rules. Plain rebuilds and automatic refreshes retain this setup. `.ignore` controls ripgrep and does not override Git's visibility for Graft.
+
+`--only-dir` restricts the entire index to its paths, including project source. If an earlier build saved such a restriction, pass `--all-dirs` once to restore the normal file scope while retaining selected repository mounts. The older `--include-ignored-repos --only-dir <repo> ...` form remains available for an index limited to those repositories.
+
 In every layout, narrow to one sub-project with `trail ask "<task>" --in <scope>/`
 once you know where you're working.
 

src/cli.ts

@@ -12,7 +12,7 @@ import { spawnSync } from "node:child_process";
 import "dotenv/config";
 import { Command, Option } from "commander";
 import { existsSync, readdirSync, readFileSync, rmSync, statSync } from "node:fs";
-import { basename, dirname, join, relative, resolve, sep } from "node:path";
+import { basename, dirname, join, relative, resolve, isAbsolute, sep } from "node:path";
 import { fileURLToPath } from "node:url";
 import { Graft } from "./engine.js";
 import { resolveConfig, type EngineConfig } from "./ai/providers.js";
@@ -58,7 +58,7 @@ import { createInterface } from "node:readline";
 import { createHash } from "node:crypto";
 import { formatUpgradeReport, formatVersionReport, getNpmViewVersion, readCurrentVersion, readPackageName, runUpgrade } from "./cli-meta.js";
 import { nativeFallbackNotice } from "./graph/native.js";
-import { lockHeldByOther, patchBuildConfig, type BuildConfig } from "./util/state.js";
+import { lockHeldByOther, patchBuildConfig, readFollowNestedRepos, type BuildConfig } from "./util/state.js";
 import { NEVER_INCLUDE_DIRS } from "./ingest/fs.js";
 import { normalizePathPrefix } from "./util/paths.js";
 import { latestSession, formatSessionStats, sessionInputRate } from "./claude/session-metrics.js";
@@ -551,12 +551,26 @@ grouped(program.command("build"), GROUP.setup)
   )
   .option(
     "--follow-nested-repos",
-    "include nested Git clones the index does not track (a multi-repo manifest checkout, or any repo cloned into the tree) " +
+    "include nested Git clones and directory symlinks to Git repos " +
       "as ONE graph, so imports across them resolve; persisted for later builds and automatic refreshes",
   )
   .option(
     "--no-follow-nested-repos",
-    "exclude untracked nested Git clones; persisted for later builds and automatic refreshes (default)",
+    "exclude nested Git clones and directory symlinks to Git repos; persisted for later builds and automatic refreshes (default)",
+  )
+  .option(
+    "--include-repo <path>",
+    "with --follow-nested-repos: add this repository to the normal project index, even if its mount is Git-ignored; repeatable and persisted; child ignores still apply",
+    (val: string, prev: string[]) => [...prev, val],
+    [] as string[],
+  )
+  .option(
+    "--include-ignored-repos",
+    "with --follow-nested-repos and --only-dir: include explicitly selected Git repos even if the parent ignores their mounts; child ignores still apply; persisted",
+  )
+  .option(
+    "--no-include-ignored-repos",
+    "clear the saved ignored-repository mounts",
   )
   .option(
     "--include-dir <name>",
@@ -574,6 +588,7 @@ grouped(program.command("build"), GROUP.setup)
     (val: string, prev: string[]) => [...prev, val],
     [] as string[],
   )
+  .option("--all-dirs", "clear any saved --only-dir restriction and restore the normal project file scope")
   .option("--no-gitignore", "skip writing graft/ into .gitignore (same as GRAFT_NO_GITIGNORE=1)")
   .option("--no-ignore", "skip writing .ignore for ripgrep re-admit (same as GRAFT_NO_IGNORE=1)")
   .action(async (
@@ -591,9 +606,12 @@ grouped(program.command("build"), GROUP.setup)
       synthConcurrency?: string;
       synthBatchChars?: string;
       includeDir?: string[];
+      includeRepo?: string[];
       onlyDir?: string[];
+      allDirs?: boolean;
       followSubmodules?: boolean;
       followNestedRepos?: boolean;
+      includeIgnoredRepos?: boolean;
       gitignore?: boolean;
       ignore?: boolean;
     },
@@ -657,7 +675,11 @@ grouped(program.command("build"), GROUP.setup)
     // The whitelist is NOT persisted to `.graft/config.json`: it belongs with the
     // graph (the fingerprint records it at build time), never in the source repo,
     // so a `--only-dir` build leaves no trace under the repo being indexed.
-    let onlyDirs: string[] | undefined;
+    if (opts.allDirs && opts.onlyDir?.length) {
+      console.error("✗ --all-dirs and --only-dir cannot be used together");
+      process.exit(1);
+    }
+    let onlyDirs: string[] | undefined = opts.allDirs ? [] : undefined;
     if (opts.onlyDir && opts.onlyDir.length > 0) {
       // --only-dir takes a repo-relative path prefix, normalized to the same
       // posix, no-`./`, no-trailing-slash form `--in` uses, so the prefix match
@@ -679,6 +701,40 @@ grouped(program.command("build"), GROUP.setup)
     if (followNestedReposWasExplicit && typeof opts.followNestedRepos === "boolean") {
       buildConfigPatch.followNestedRepos = opts.followNestedRepos;
     }
+    const explicitIgnoredRepos = command.getOptionValueSource("includeIgnoredRepos") === "cli";
+    if (opts.includeRepo?.length && explicitIgnoredRepos) {
+      console.error("✗ --include-repo cannot be combined with --include-ignored-repos or --no-include-ignored-repos");
+      process.exit(1);
+    }
+    if (opts.includeRepo?.length || explicitIgnoredRepos) {
+      if (explicitIgnoredRepos && opts.includeIgnoredRepos === false) {
+        buildConfigPatch.includeIgnoredRepos = [];
+      } else {
+        const flag = opts.includeRepo?.length ? "--include-repo" : "--include-ignored-repos";
+        const selectedRepos = opts.includeRepo?.length
+          ? [...new Set(opts.includeRepo.map((path) => normalizePathPrefix(path)))]
+          : onlyDirs;
+        const root = resolve(dir);
+        const followsNested = buildConfigPatch.followNestedRepos ?? readFollowNestedRepos(root);
+        if (!followsNested || !selectedRepos?.length || !existsSync(join(root, ".git"))) {
+          console.error(`✗ ${flag} requires --follow-nested-repos and ${flag === "--include-repo" ? "repository paths" : "--only-dir paths to repositories"} under a Git root`);
+          process.exit(1);
+        }
+        for (const path of selectedRepos) {
+          const abs = resolve(root, path);
+          const rel = relative(root, abs);
+          let valid = rel !== "" && rel !== ".." && !rel.startsWith(`..${sep}`) && !isAbsolute(rel);
+          try {
+            valid = valid && statSync(abs).isDirectory() && existsSync(join(abs, ".git"));
+          } catch { valid = false; }
+          if (!valid) {
+            console.error(`✗ ${flag}: "${path}" must name a Git repository inside the build root`);
+            process.exit(1);
+          }
+        }
+        buildConfigPatch.includeIgnoredRepos = selectedRepos;
+      }
+    }
     if (Object.keys(buildConfigPatch).length > 0) {
       patchBuildConfig(resolve(dir), buildConfigPatch);
     }

src/context/build.ts

@@ -21,7 +21,7 @@ import { readFingerprint } from "../graph/fingerprint.js";
 import { contentHash } from "../util/id.js";
 import { relPosix } from "../util/paths.js";
 import { readSourceFile } from "../util/source.js";
-import { readFollowNestedRepos, readFollowSubmodules, readIncludeDirs } from "../util/state.js";
+import { readFollowNestedRepos, readFollowSubmodules, readIncludeDirs, readIncludeIgnoredRepos } from "../util/state.js";
 import type { Summarizer } from "../ai/summarize.js";
 import { LlmFailureGate } from "../ai/failure.js";
 import type { FileSummary, SynthNode, Synthesizer } from "../ai/synthesize.js";
@@ -122,7 +122,7 @@ export interface BuildResult {
  * fingerprint — the same source `checkGraph` / `probeDrift` use, so a later
  * `--deep` without the flag still skips files the wiring pass excluded. */
 function resolveOnlyDirs(outDir: string, explicit?: readonly string[]): Set<string> | undefined {
-  const list = explicit && explicit.length > 0 ? explicit : (readFingerprint(outDir)?.onlyDirs ?? []);
+  const list = explicit ?? readFingerprint(outDir)?.onlyDirs ?? [];
   return list.length > 0 ? new Set(list) : undefined;
 }
 
@@ -141,6 +141,7 @@ export function listContextFiles(
   const walked = walkDir(root, readIncludeDirs(root), {
     followSubmodules: readFollowSubmodules(root),
     followNestedRepos: readFollowNestedRepos(root),
+    includeIgnoredRepos: readIncludeIgnoredRepos(root),
   })
     .filter((f) => exts.some((e) => f.toLowerCase().endsWith(e)))
     .filter((f) => !f.startsWith(outDir));

src/graph/build.ts

@@ -25,14 +25,14 @@ import type { NativeFallback } from "./native.js";
 import { contentHash } from "../util/id.js";
 import { relPosix } from "../util/paths.js";
 import { readSourceFile } from "../util/source.js";
-import { readFollowNestedRepos, readFollowSubmodules, readIncludeDirs } from "../util/state.js";
+import { readFollowNestedRepos, readFollowSubmodules, readIncludeDirs, readIncludeIgnoredRepos } from "../util/state.js";
 import {
   emptyExtractCache,
   readExtractCache,
   writeExtractCache,
   type ExtractEntry,
 } from "./extract-cache.js";
-import { writeFingerprint } from "./fingerprint.js";
+import { readFingerprint, writeFingerprint } from "./fingerprint.js";
 import { seedGraph, type SeedResult } from "./seed.js";
 import { filterByOnlyDirs, listSourceStats, normalizeExtensions } from "./source-files.js";
 import { resolveEdges, type GoModule } from "./resolve.js";
@@ -223,11 +223,15 @@ export async function buildGraph(
   // Enumerate once: source extraction, scope discovery, and Go module
   // resolution must agree on the same Git-ignore-aware working-tree view —
   // including the repo's persisted directory and submodule choices.
+  const repositoryTargets: Record<string, string> = Object.create(null);
   const walked = walkDir(root, readIncludeDirs(root), {
     followSubmodules: readFollowSubmodules(root),
     followNestedRepos: readFollowNestedRepos(root),
+    includeIgnoredRepos: readIncludeIgnoredRepos(root),
+    onRepository: (mount, target) => { repositoryTargets[mount] = target; },
   });
-  const onlyDirs = opts.onlyDirs && opts.onlyDirs.length > 0 ? new Set(opts.onlyDirs) : undefined;
+  const selectedDirs = opts.onlyDirs ?? readFingerprint(outDir)?.onlyDirs;
+  const onlyDirs = selectedDirs?.length ? new Set(selectedDirs) : undefined;
   const repoFiles = filterByOnlyDirs(walked, root, onlyDirs);
   const extensions = normalizeExtensions(opts.extensions);
   const files = listSourceStats(root, outDir, repoFiles, undefined, extensions);
@@ -436,7 +440,7 @@ export async function buildGraph(
   // these source bytes." Nothing about the projections below — which is why it is
   // safe to write here, and why `graphOnly` builds (the query path, which stops
   // right after this line) are still recorded as fresh.
-  writeFingerprint(outDir, entries, opts.onlyDirs, extensions);
+  writeFingerprint(outDir, entries, selectedDirs, extensions, repositoryTargets);
 
   // Tier-2 passive surface: project the nodes into per-file markdown cards, and
   // refresh the INDEX roster. Pure projection — no LLM, no network.