From 24f18377cf04825bf71325831668a6bf6baa932b Mon Sep 17 00:00:00 2001 From: Yashasvi Date: Fri, 25 Sep 2026 01:35:14 +0530 Subject: [PATCH 1/2] fix(graph): read impact's knowledge links from the committed scaffold (#224) impact answered from the _mex_grounded_source cache in graph.db, which only setup, sync and graph ground fill. A graph-only build left it empty, so a fresh clone returned no knowledge links (0 of 12 on this repository), and a checkout whose Markdown moved on kept returning dropped links. impact now reads grounds_to at query time through extractGroundings, so root, mex and mixed shapes all count. It walks the scaffold with the Wiki index's own contained, bounded discovery and reader, honouring wiki.exclude, so it agrees with wiki for-code. node_aliases still maps an old id to the current node. Files that cannot be read within bounds, a walk stopped by a corpus ceiling, and a scaffold that changes during the call each produce a grounding-omitted record instead of a silently partial answer. The cache table stays for check's baselines. --- CHANGELOG.md | 1 + src/config.ts | 11 + src/graph/cli-agent.ts | 123 ++++++-- src/graph/committed-groundings.ts | 210 ++++++++++++++ test/graph-cli-agent.test.ts | 35 ++- .../graph-impact-committed-groundings.test.ts | 263 ++++++++++++++++++ 6 files changed, 621 insertions(+), 22 deletions(-) create mode 100644 src/graph/committed-groundings.ts create mode 100644 test/graph-impact-committed-groundings.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index c8d5cb33..cb7298e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,7 @@ All notable changes to this project will be documented in this file. - `mex check` no longer switches grounding off after a source edit. When the only reason the graph is stale is that source files changed, and the complete list of changed files is known, groundings in unchanged files are checked against the last snapshot. A node in an edited tree-sitter file (Python, Go and the other non-TypeScript languages) is re-extracted from that file with the same extractor and body hash a refresh uses, so a changed body is a real `GROUNDING_DRIFT` without a refresh. What cannot be settled that way — a deleted file, an edited TypeScript or JavaScript file (its spans come from a whole compiler program), or a node no longer found in its file under the same identity — is the new `GROUNDING_UNVERIFIED` warning, counted in the score, with a message to run `mex graph refresh`. Nothing is reported `GROUNDING_GONE` or moved from a stale snapshot, and the freshness warning stays. Config, semantic-input, branch, grammar, schema or parse-health staleness still skips grounding exactly as before. Previously the score did not move after the very edit grounding exists to catch until a full refresh. The accepted cost: every grounding in an edited TypeScript or JavaScript file reads unverified rather than clean or drifted until the next refresh (#228). ### Fixed +- `mex impact` now returns knowledge links on a fresh clone. It read groundings only from a cache inside `.mex/graph.db` that setup, `mex sync` and `mex graph ground` fill and `mex graph`, `rebuild` and `refresh` never do, so a teammate who cloned and built got none: on the MEX repository it returned 0 of 12 committed groundings at any budget, where `wiki for-code` returned 12. It now reads `grounds_to` from the committed scaffold at query time, both the root key and `mex.grounds_to`, through the same contained, bounded walk the Wiki index uses, honouring `wiki.exclude`; a grounding recorded under a renamed node's old id still attaches to the current node. The same checkout now returns 11 of 12; the twelfth names a node that no longer exists, which `mex check` reports. With no copy there is nothing to go stale, so a cached link the Markdown no longer declares is not returned. A scaffold file that cannot be read within bounds, or a walk stopped by a scaffold-wide ceiling, is named in a new `grounding-omitted` record; if the scaffold changes during the call, every grounding record is withheld and that record names the changed files. The cache table stays for `mex check`'s baselines. The accepted costs: each `impact` call reads the scaffold, about 0.65 s on the MEX repository, most of it loading the YAML and Markdown parser; and a link that exists only as an inline `mex://` anchor is no longer returned, because the Wiki does not index anchors as groundings and `impact` now agrees with `wiki for-code` (#224). - Groundings kept in a root `grounds_to` are no longer invisible in a file that also has a `mex:` map, the shape setup itself produced on every multi-entity context file it populated. `mex check` now reads both keys, deduplicated by node, so drift in code grounded only at the root is reported; on a setup-populated Hono scaffold it had skipped 4 of 24 groundings. The Wiki attaches a root `grounds_to` to the file's file-level entity, never to a section entity, so `wiki for-code` returns it; a file with only section entities still keeps its root groundings unattributed and reported. `wiki migrate` now moves root groundings into the file-level entity whatever the number of sections, including files an earlier run already adopted, and any `mex ground`, sync or setup write that touches such a file moves them under `mex.grounds_to` and removes the root key, leaving the rest of the file byte-identical. `mex check` and `wiki validate` report the split as the new info-level `GROUNDING_MIXED_SHAPE`, raised to a warning when the two keys ground one node differently. In that case the `mex.grounds_to` entry is the one checked and the root one is kept at the root until someone resolves it. Setup's population and `mex graph ground` prompts now tell agents to write under `mex.grounds_to` once a file has a `mex:` map. The accepted cost: the first write to a split file moves its root groundings, and on an adopted file bumps its entity revision, so that file shows a frontmatter diff; a file-level entity whose section children each had their own claim now also carries the file's root groundings (#226). - Closing a read-only grounding runtime now releases the descriptor that binds `graph.db`, not only its SQLite reader. On Windows the open handle made the next refresh in the same process, after any check that opened grounding readers, fail with "The live graph changed before candidate publication" (#228). diff --git a/src/config.ts b/src/config.ts index ea997fb1..601e9c80 100644 --- a/src/config.ts +++ b/src/config.ts @@ -146,6 +146,17 @@ export function loadConfiguredSetupMode(scaffoldRoot: string): "code-repo" | "ag return loadPersistedConfig(scaffoldRoot)?.setupMode === "agent-memory" ? "agent-memory" : "code-repo"; } +/** + * Read the checkout's `wiki` settings without requiring a complete scaffold. + * + * For read-only callers outside the Wiki's own commands that must walk the + * scaffold the way the Wiki index does, so `wiki.exclude` hides a file from + * both or from neither. + */ +export function loadConfiguredWikiConfig(scaffoldRoot: string): WikiConfig { + return loadWikiConfig(loadPersistedConfig(scaffoldRoot)); +} + /** Persist setup intent using the same atomic, key-preserving config writer. */ export function saveConfiguredSetupMode(scaffoldRoot: string, mode: "code-repo" | "agent-memory"): void { mergeIntoConfig(scaffoldRoot, { setupMode: mode }); diff --git a/src/graph/cli-agent.ts b/src/graph/cli-agent.ts index 9548871a..dae252e9 100644 --- a/src/graph/cli-agent.ts +++ b/src/graph/cli-agent.ts @@ -14,6 +14,10 @@ import { compactFact, groupByFile, planFileSource, readNodeSource, selectScope, sourceHash, type CompactFact, type DetailLevel, type RankedScopeFile, type ScopedCandidate, type SourceRange, } from "./scope.js"; +import { + committedGroundingsChangedSince, observeCommittedGroundings, + type CommittedGrounding, type CommittedGroundingObservation, +} from "./committed-groundings.js"; import { FingerprintStore } from "./fingerprint-store.js"; import { serializeFingerprint } from "./fingerprint.js"; import { @@ -140,9 +144,16 @@ export function runImpact( // Knowledge links are admitted before callers and source (#225): they are // tiny and no other command returns them, so budget pressure cuts the // callers other commands can reproduce instead. Output order is unchanged. - const groundingRecords: Rec[] = []; - for (const grounding of groundedFiles(session.db, affectedIds)) { - const record: Rec = { type: "grounding", node: grounding.node_id, file: grounding.scaffold_file }; + // They are read from the committed scaffold, not the graph.db cache (#224), + // and anything that scaffold read could not cover is said, not skipped. + const scaffold = observeCommittedGroundings(rootDir); + (deps as AgentCommandInternalDeps).__internal?.afterCommittedGroundingRead?.(); + let groundingRecords: Rec[] = []; + for (const record of groundingOmissionRecords(scaffold)) { + if (ledger.tryAdd(record)) groundingRecords.push(record); else truncated = true; + } + for (const grounding of groundedFiles(session.db, affectedIds, scaffold.groundings)) { + const record: Rec = { type: "grounding", node: grounding.node, file: grounding.file }; if (ledger.tryAdd(record)) groundingRecords.push(record); else truncated = true; } @@ -161,6 +172,16 @@ export function runImpact( const sourceRecords = planSource(session, ledger, emittedNodes, rootDir, opts); + // The graph snapshot is revalidated after this task; the scaffold is not + // part of it, so it is observed again here. Links read from bytes that + // have since changed are withheld rather than returned as current. + const changed = committedGroundingsChangedSince(rootDir, scaffold); + if (changed.length > 0) { + groundingRecords = []; + const record = groundingOmissionRecord("scaffold-changed", changed); + if (ledger.tryAdd(record)) groundingRecords.push(record); else truncated = true; + } + emitAll(write, meta, [ ...configDriftRecords(session), ...headRecords, ...factRecords, ...sourceRecords, ...groundingRecords, @@ -2000,6 +2021,8 @@ type AgentSessionTask = (session: AgentGraphSession, write: (line: string) => vo interface AgentCommandInternalHooks { freshRead?: Omit; beforeFinalFreshnessValidation?: () => void | Promise; + /** Runs after `impact` reads the scaffold and before it observes it again. */ + afterCommittedGroundingRead?: () => void; } type AgentCommandInternalDeps = AgentCommandDeps & { @@ -2606,19 +2629,87 @@ function unresolvedCallSites( ).all(name, Math.max(0, limit)) as UnresolvedCallSite[]; } -function groundedFiles(db: SqliteDatabase, nodeIds: string[]): Array<{ scaffold_file: string; node_id: string }> { - if (nodeIds.length === 0) return []; - const placeholders = nodeIds.map(() => "?").join(","); - return db.prepare( - `SELECT DISTINCT grounded.scaffold_file, - COALESCE(aliases.canonical_node_id, grounded.node_id) AS node_id - FROM _mex_grounded_source grounded - LEFT JOIN node_aliases aliases ON aliases.alias_id = grounded.node_id - WHERE grounded.scaffold_file IS NOT NULL - AND (grounded.node_id IN (${placeholders}) - OR aliases.canonical_node_id IN (${placeholders})) - ORDER BY grounded.scaffold_file, node_id`, - ).all(...nodeIds, ...nodeIds) as Array<{ scaffold_file: string; node_id: string }>; +/** + * The committed groundings that name one of `nodeIds`, deduplicated and sorted + * by file then node. + * + * A grounding recorded under an older id still attaches through + * `node_aliases`, and is reported under the current id, as it was when this + * read the `_mex_grounded_source` cache. That cache is no longer consulted + * (#224): it was filled only by capture, so a fresh clone had none of it and a + * moved-on checkout kept links the Markdown had dropped. + */ +function groundedFiles( + db: SqliteDatabase, + nodeIds: readonly string[], + committed: readonly CommittedGrounding[], +): CommittedGrounding[] { + if (nodeIds.length === 0 || committed.length === 0) return []; + const affected = new Set(nodeIds); + const canonical = nodeAliases(db, [...new Set(committed.map((entry) => entry.node))]); + const found = new Map(); + for (const entry of committed) { + const aliased = canonical.get(entry.node); + const node = aliased !== undefined && affected.has(aliased) ? aliased + : affected.has(entry.node) ? entry.node : undefined; + if (node === undefined) continue; + found.set(`${entry.file}\0${node}`, { file: entry.file, node }); + } + return [...found.values()].sort((left, right) => + left.file < right.file ? -1 : left.file > right.file ? 1 + : left.node < right.node ? -1 : left.node > right.node ? 1 : 0); +} + +/** `alias → canonical` for the given ids, read in bounded batches. */ +function nodeAliases(db: SqliteDatabase, ids: readonly string[]): Map { + const aliases = new Map(); + for (let start = 0; start < ids.length; start += ALIAS_LOOKUP_BATCH) { + const batch = ids.slice(start, start + ALIAS_LOOKUP_BATCH); + const rows = db.prepare( + `SELECT alias_id, canonical_node_id FROM node_aliases + WHERE alias_id IN (${batch.map(() => "?").join(",")})`, + ).all(...batch) as Array<{ alias_id: string; canonical_node_id: string }>; + for (const row of rows) aliases.set(row.alias_id, row.canonical_node_id); + } + return aliases; +} + +const ALIAS_LOOKUP_BATCH = 500; +/** Files named by one omission record; the rest are counted, not listed. */ +const GROUNDING_OMISSION_MAX_FILES = 20; + +type GroundingOmissionReason = "scaffold-unreadable" | "scaffold-limit" | "scaffold-changed"; + +/** + * Say which knowledge an answer could not include. + * + * - `scaffold-unreadable`: the listed files could not be read within bounds, + * so their groundings are absent; every other file's are present. + * - `scaffold-limit`: a scaffold-wide ceiling (`limit`) stopped the walk, so + * files after it were not read. + * - `scaffold-changed`: the scaffold changed during the call. No grounding + * record is returned, because none can be said to be current. + */ +function groundingOmissionRecord( + reason: GroundingOmissionReason, + files: readonly string[], + limit?: string, +): Rec { + const listed = files.slice(0, GROUNDING_OMISSION_MAX_FILES); + return { + type: "grounding-omitted", + reason, + files: listed, + ...(files.length > listed.length ? { moreFiles: files.length - listed.length } : {}), + ...(limit === undefined ? {} : { limit }), + }; +} + +function groundingOmissionRecords(scaffold: CommittedGroundingObservation): Rec[] { + return [ + ...(scaffold.unreadable.length > 0 ? [groundingOmissionRecord("scaffold-unreadable", scaffold.unreadable)] : []), + ...(scaffold.limit !== undefined ? [groundingOmissionRecord("scaffold-limit", [], scaffold.limit)] : []), + ]; } function liveUnindexedFiles(indexedFiles: IndexedFileInfo[], rootDir: string): string[] { diff --git a/src/graph/committed-groundings.ts b/src/graph/committed-groundings.ts new file mode 100644 index 00000000..b5ce06dd --- /dev/null +++ b/src/graph/committed-groundings.ts @@ -0,0 +1,210 @@ +/** + * The groundings the committed scaffold declares — what `impact` reports as + * knowledge (#224). + * + * `impact` used to answer from `_mex_grounded_source` in `.mex/graph.db`. That + * table is a cache of baselines, filled only by capture (setup, `mex sync`, + * `mex graph ground`) and never by `mex graph`, `rebuild` or `refresh`. So a + * teammate who cloned and built had an empty table and `impact` returned no + * knowledge at all, and a checkout whose Markdown moved on kept returning links + * the scaffold no longer made. The answer lived only in a disposable index — + * the failure `patterns/durable-change-signal.md` describes for baselines. + * + * The committed Markdown is now the source of truth, read at query time. There + * is no copy, so there is nothing to go stale. The cache table stays: `check` + * still reads baselines from it. + * + * ## One walk, one reader, shared with the Wiki + * + * The walk is the Wiki index's own `discoverMarkdownFiles`, honouring the + * checkout's `wiki.exclude`, and every file goes through `readContainedSource` + * under `WIKI_CORPUS_LIMITS`. That is deliberate: `wiki for-code` answers the + * same question from the index that walk builds, and two walks would disagree + * about which files the scaffold contains. It also means symlink containment, + * descriptor-bound reads and the byte ceilings are the ones already reviewed, + * not a second set. + * + * Groundings are read through `extractGroundings`, which takes the union of the + * root `grounds_to` and `mex.grounds_to` (#226). Inline `mex://` anchors are + * not read: they are links in prose, the Wiki does not index them as + * groundings, and reading them would make `impact` and `wiki for-code` disagree + * about the same node. The cache did hold them, because capture records + * anchors too, so a link that exists only as an anchor is no longer returned. + * `mex check` still verifies anchors. + * + * Two known differences from `wiki for-code` remain, both on the Wiki's side + * of the join: a pre-wiki file with only a root `grounds_to` has no Wiki + * entity, so `for-code` cannot return it and this does; and groundings inside + * a section entity's `` block are not frontmatter, so + * `for-code` returns them and this, like `mex check`, does not read them. + * + * ## Nothing is dropped silently + * + * Scaffold Markdown sits outside the graph snapshot `impact` binds, so it is + * read once per invocation and observed again before output. A file that could + * not be read within those bounds, a walk stopped by a scaffold-wide ceiling, + * or a file that changed during the call is returned to the caller as an + * omission, never as an answer that merely looks complete. + */ + +import { lstatSync } from "node:fs"; +import { relative, resolve } from "node:path"; +import { loadConfiguredWikiConfig } from "../config.js"; +import { extractGroundings } from "../markdown.js"; +import { toPosix } from "../paths.js"; +import { addWikiCorpusBytes, WikiCorpusLimitError, type WikiCorpusLimit } from "../wiki/index/corpus-policy.js"; +import { discoverMarkdownFiles, type DiscoveredFile } from "../wiki/index/discover.js"; +import { readContainedSource } from "../wiki/index/source-read.js"; + +/** One declared grounding: a scaffold file, relative to the project root, and the node it names. */ +export interface CommittedGrounding { + file: string; + node: string; +} + +export interface CommittedGroundingObservation { + /** Declared entries, deduplicated per file, in walk order. */ + groundings: CommittedGrounding[]; + /** Files that could not be read within bounds; their groundings are absent. Sorted. */ + unreadable: string[]; + /** The scaffold-wide ceiling that stopped the walk, if one did. */ + limit?: WikiCorpusLimit; + /** Per-file identity taken before each read, so the same call can prove nothing moved. */ + readonly observed: ReadonlyMap; +} + +interface ScaffoldWalk { + root: string; + scaffoldRoot: string; + files: DiscoveredFile[]; + unreadable: Set; + limit?: WikiCorpusLimit; +} + +/** Walk and read the scaffold under `/.mex` once. Never throws for scaffold content. */ +export function observeCommittedGroundings(projectRoot: string): CommittedGroundingObservation { + const walk = walkScaffold(projectRoot); + const groundings: CommittedGrounding[] = []; + const observed = new Map(); + if (walk === null) return { groundings, unreadable: [], observed }; + + let limit = walk.limit; + let corpusBytes = 0; + for (const file of walk.files) { + const path = projectPath(walk, file.path); + // Taken before the read: an edit that lands during it moves the identity, + // and the second observation sees that. + observed.set(path, fileIdentity(file.absolutePath)); + let text: string; + try { + text = readContainedSource(walk.scaffoldRoot, file.absolutePath); + } catch { + // Too large, not a regular file, not valid UTF-8, or retargeted under us. + walk.unreadable.add(path); + continue; + } + try { + corpusBytes = addWikiCorpusBytes(corpusBytes, Buffer.byteLength(text, "utf8")); + } catch (error) { + limit = error instanceof WikiCorpusLimitError ? error.limit : "maxCorpusBytes"; + break; + } + const seen = new Set(); + for (const grounding of declaredGroundings(text)) { + if (seen.has(grounding.node)) continue; + seen.add(grounding.node); + groundings.push({ file: path, node: grounding.node }); + } + } + + return { + groundings, + unreadable: [...walk.unreadable].sort(), + ...(limit === undefined ? {} : { limit }), + observed, + }; +} + +/** + * Files that differ from an earlier observation, sorted. + * + * Walks again and compares each file's identity — size, inode, modification + * and change time — as `readBoundedText` does for grounding documents. A write + * moves the change time, which no caller can set back, so a file whose + * identity matches still holds the bytes the answer was built from. Empty + * means the scaffold is as it was read. + */ +export function committedGroundingsChangedSince( + projectRoot: string, + previous: CommittedGroundingObservation, +): string[] { + const walk = walkScaffold(projectRoot); + const current = new Map(); + for (const file of walk?.files ?? []) { + current.set(projectPath(walk!, file.path), fileIdentity(file.absolutePath)); + } + const changed = new Set(); + for (const [path, identity] of previous.observed) { + if (current.get(path) !== identity) changed.add(path); + } + for (const path of current.keys()) if (!previous.observed.has(path)) changed.add(path); + for (const path of walk?.unreadable ?? []) { + if (!previous.unreadable.includes(path) && !current.has(path)) changed.add(path); + } + return [...changed].sort(); +} + +/** Discover the scaffold's Markdown the way the Wiki index does; null when there is no scaffold. */ +function walkScaffold(projectRoot: string): ScaffoldWalk | null { + const root = resolve(projectRoot); + const scaffoldRoot = resolve(root, ".mex"); + // No scaffold is an ordinary state — nothing is committed, so nothing is missing. + try { + if (!lstatSync(scaffoldRoot).isDirectory()) return null; + } catch { + return null; + } + const walk: ScaffoldWalk = { root, scaffoldRoot, files: [], unreadable: new Set() }; + try { + const exclude = loadConfiguredWikiConfig(scaffoldRoot).exclude; + const discovery = discoverMarkdownFiles({ root: scaffoldRoot, exclude }); + walk.files = discovery.files; + // Discovery names what it skipped: an escaping or broken symlink, a + // directory it could not open. Any of them may hold groundings. + for (const entry of discovery.diagnostics) { + if (typeof entry.file === "string") walk.unreadable.add(projectPath(walk, entry.file)); + } + } catch (error) { + // A scaffold problem must cost the knowledge links, not the whole answer. + if (error instanceof WikiCorpusLimitError) walk.limit = error.limit; + else walk.unreadable.add(projectPath(walk, ".")); + } + return walk; +} + +function projectPath(walk: ScaffoldWalk, scaffoldRelative: string): string { + return toPosix(relative(walk.root, resolve(walk.scaffoldRoot, scaffoldRelative))); +} + +function fileIdentity(absolutePath: string): string { + try { + const stats = lstatSync(absolutePath, { bigint: true }); + return `${stats.size}:${stats.ino}:${stats.mtimeNs}:${stats.ctimeNs}`; + } catch { + return "absent"; + } +} + +/** + * `extractGroundings` over the frontmatter block alone. + * + * Groundings live only in frontmatter, and parsing a whole document to reach + * it cost most of this read. Frontmatter opens on the first line and closes at + * the first line that is exactly `---`, so the text up to that line holds the + * same block and parses to the same entries. A document with no such line is + * parsed whole, as before. + */ +function declaredGroundings(text: string): ReturnType { + const close = /\r?\n---\r?(?:\n|$)/.exec(text); + return extractGroundings(close === null ? text : text.slice(0, close.index + close[0].length)); +} diff --git a/test/graph-cli-agent.test.ts b/test/graph-cli-agent.test.ts index 9a26e95b..9023e485 100644 --- a/test/graph-cli-agent.test.ts +++ b/test/graph-cli-agent.test.ts @@ -1,4 +1,4 @@ -import { describe, expect, it, vi } from "vitest"; +import { afterEach, describe, expect, it, vi } from "vitest"; import { runGraphQuery, runGraphScope, runImpact, type AgentCommandDeps } from "../src/graph/cli-agent.js"; import type { GraphEngine } from "../src/graph/engine.js"; import type { GraphEdge, GraphNode } from "../src/graph/types.js"; @@ -6,7 +6,7 @@ import { __resetTelemetryForTest, __setTelemetryEndpointForTest, __setTransport, import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; import { createHash } from "node:crypto"; import { tmpdir } from "node:os"; -import { join } from "node:path"; +import { dirname, join } from "node:path"; import { createServer } from "node:http"; function node(id: string, name: string, file = "src/a.ts", line = 1): GraphNode { @@ -67,10 +67,31 @@ function deps(indexedSources: Record = {}): { return { deps: { open: () => ({ graph, db, close }), write: (line) => output.push(line) }, output, close }; } +const scaffoldRoots: string[] = []; + +afterEach(() => { + for (const root of scaffoldRoots.splice(0)) rmSync(root, { recursive: true, force: true }); +}); + +/** A project root whose committed scaffold declares these groundings; impact reads them from here (#224). */ +function scaffold(groundings: ReadonlyArray<{ file: string; node: string }>): string { + const root = mkdtempSync(join(tmpdir(), "mex-impact-scaffold-")); + scaffoldRoots.push(root); + const byFile = new Map(); + for (const { file, node } of groundings) byFile.set(file, [...(byFile.get(file) ?? []), node]); + for (const [file, nodes] of byFile) { + mkdirSync(dirname(join(root, file)), { recursive: true }); + const entries = nodes.map((node) => ` - node: ${node}\n fingerprint: mh:64:00\n`).join(""); + writeFileSync(join(root, file), `---\ngrounds_to:\n${entries}---\n`); + } + return root; +} + describe("agent graph commands", () => { it("impact emits deterministic JSONL with transitive callers and grounded memory", () => { const fixture = deps(); - runImpact("leaf", "/repo", fixture.deps); + const root = scaffold([{ file: ".mex/context/architecture.md", node: "function:leaf" }]); + runImpact("leaf", root, fixture.deps); const rows = fixture.output.map((line) => JSON.parse(line)); expect(rows.map((row) => row.type)).toEqual(["meta", "target", "defines", "caller", "caller", "grounding", "summary"]); expect(rows.filter((row) => row.type === "caller").map((row) => row.depth)).toEqual([1, 2]); @@ -98,12 +119,13 @@ describe("agent graph commands", () => { const db = { prepare: (sql: string) => ({ run: vi.fn(), get: vi.fn(), iterate: vi.fn(), - all: () => sql.includes("FROM nodes") ? [] : groundings, + all: () => [], }), exec: vi.fn(), pragma: vi.fn(), transaction: (fn: () => T) => fn(), close: vi.fn(), open: true, }; const output: string[] = []; - runImpact("leaf", "/repo", { open: () => ({ graph, db, close: vi.fn() }), write: (line) => output.push(line) }, { maxNodes: 100 }); + const root = scaffold(groundings.map((g) => ({ file: g.scaffold_file, node: g.node_id }))); + runImpact("leaf", root, { open: () => ({ graph, db, close: vi.fn() }), write: (line) => output.push(line) }, { maxNodes: 100 }); const rows = output.map((line) => JSON.parse(line)); expect(rows.at(-1)).toMatchObject({ type: "summary", truncated: true }); expect(rows.filter((row) => row.type === "caller").length).toBeLessThan(callers.length); @@ -121,7 +143,8 @@ describe("agent graph commands", () => { expect(floor).toBeGreaterThan(1); const fixture = deps(); - runImpact("leaf", "/repo", fixture.deps, { maxOutputTokens: floor }); + const root = scaffold([{ file: ".mex/context/architecture.md", node: "function:leaf" }]); + runImpact("leaf", root, fixture.deps, { maxOutputTokens: floor }); const rows = fixture.output.map((line) => JSON.parse(line)); expect(rows[0]).toMatchObject({ type: "meta" }); expect(rows.at(-1)).toMatchObject({ type: "summary", truncated: true }); diff --git a/test/graph-impact-committed-groundings.test.ts b/test/graph-impact-committed-groundings.test.ts new file mode 100644 index 00000000..de6b0e81 --- /dev/null +++ b/test/graph-impact-committed-groundings.test.ts @@ -0,0 +1,263 @@ +import { + existsSync, + mkdirSync, + mkdtempSync, + readFileSync, + rmSync, + statSync, + writeFileSync, +} from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, describe, expect, it } from "vitest"; +import { runImpact, type AgentCommandDeps } from "../src/graph/cli-agent.js"; +import { openGraphDatabase } from "../src/graph/db/database.js"; +import { createGraphEngine } from "../src/graph/engine-impl.js"; +import { FingerprintStore } from "../src/graph/fingerprint-store.js"; +import { MinHashReconciler } from "../src/graph/reconcile-engine.js"; +import { WIKI_CORPUS_LIMITS } from "../src/wiki/index/corpus-policy.js"; +import { createGroundingGraph, deriveGrounding } from "../src/wiki/grounding/adapter.js"; +import { resolveGrounding } from "../src/wiki/grounding/resolve.js"; +import { rebuildWikiIndex } from "../src/wiki/index/rebuild.js"; +import { knowledgeRecordsFor } from "../src/wiki/cli/for-code.js"; + +// `impact` reads knowledge links from the committed scaffold (#224). Every +// graph here is built by `mex graph` alone: no setup, sync or ground, so the +// `_mex_grounded_source` cache starts empty, as it does on a fresh clone. + +const roots: string[] = []; + +afterEach(() => { + for (const root of roots.splice(0)) { + try { + rmSync(root, { recursive: true, force: true }); + } catch { + // Windows can hold a SQLite handle briefly after close. + } + } +}); + +interface Built { + root: string; + dbPath: string; + leaf: string; + parent: string; +} + +async function built(prefix: string): Promise { + const root = mkdtempSync(join(tmpdir(), prefix)); + roots.push(root); + mkdirSync(join(root, "src"), { recursive: true }); + mkdirSync(join(root, ".mex", "context"), { recursive: true }); + mkdirSync(join(root, ".mex", "patterns"), { recursive: true }); + writeFileSync(join(root, ".mex", "ROUTER.md"), "# Router\n"); + writeFileSync(join(root, "src", "leaf.ts"), "export function leaf(value: number): number {\n return value + 1;\n}\n"); + writeFileSync( + join(root, "src", "parent.ts"), + "import { leaf } from \"./leaf\";\nexport function parent(): number {\n return leaf(41);\n}\n", + ); + const engine = createGraphEngine({ rootDir: root }); + await engine.build(); + const leaf = engine.searchNodes("leaf").find((node) => node.name === "leaf" && node.kind === "function"); + const parent = engine.searchNodes("parent").find((node) => node.name === "parent" && node.kind === "function"); + engine.close(); + if (!leaf || !parent) throw new Error("fixture nodes missing"); + const dbPath = join(root, ".mex", "graph.db"); + for (const suffix of ["-wal", "-shm"]) { + const path = `${dbPath}${suffix}`; + if (existsSync(path) && statSync(path).size === 0) rmSync(path, { force: true }); + } + return { root, dbPath, leaf: leaf.id, parent: parent.id }; +} + +function entry(node: string): string { + return ` - node: ${node}\n fingerprint: mh:64:00\n`; +} + +/** A pre-wiki file: `grounds_to` at the frontmatter root. */ +function rootShape(name: string, nodes: readonly string[]): string { + return `---\nname: ${name}\ngrounds_to:\n${nodes.map(entry).join("")}---\n\n# ${name}\n`; +} + +/** A file-level wiki entity: groundings under `mex.grounds_to`, and optionally also at the root (#226). */ +function mexShape(name: string, id: string, mexNodes: readonly string[], rootNodes: readonly string[] = []): string { + const mex = mexNodes.map((node) => ` - node: ${node}\n fingerprint: mh:64:00\n`).join(""); + return `---\nname: ${name}\n` + + (rootNodes.length > 0 ? `grounds_to:\n${rootNodes.map(entry).join("")}` : "") + + `mex:\n id: ${id}\n type: pattern\n status: promoted\n revision: 1\n title: ${name}\n` + + (mexNodes.length > 0 ? ` grounds_to:\n${mex}` : "") + + `---\n\n# ${name}\n\nBody.\n`; +} + +async function impact( + target: string, + root: string, + internal?: Record, + options: Record = {}, +): Promise[]> { + const output: string[] = []; + const deps = { write: (line: string) => output.push(line), ...(internal ? { __internal: internal } : {}) }; + await runImpact(target, root, deps as AgentCommandDeps, options); + return output.map((line) => JSON.parse(line) as Record); +} + +function groundings(records: Record[]): Array<{ node: unknown; file: unknown }> { + return records.filter((record) => record.type === "grounding").map(({ node, file }) => ({ node, file })); +} + +describe("impact reads committed groundings (#224)", () => { + it("returns every committed grounding on a graph-only build with an empty cache", async () => { + const fixture = await built("mex-impact-fresh-clone-"); + writeFileSync(join(fixture.root, ".mex", "context", "architecture.md"), rootShape("architecture", [fixture.leaf])); + writeFileSync(join(fixture.root, ".mex", "patterns", "callers.md"), rootShape("callers", [fixture.parent])); + + const records = await impact("leaf", fixture.root); + + expect(records.some((record) => record.type === "error")).toBe(false); + expect(groundings(records)).toEqual([ + { node: fixture.leaf, file: ".mex/context/architecture.md" }, + { node: fixture.parent, file: ".mex/patterns/callers.md" }, + ]); + expect(records.some((record) => record.type === "grounding-omitted")).toBe(false); + }); + + it("does not return a cached grounding the Markdown no longer declares", async () => { + const fixture = await built("mex-impact-stale-cache-"); + writeFileSync(join(fixture.root, ".mex", "context", "architecture.md"), rootShape("architecture", [fixture.leaf])); + // What capture once recorded, before the author removed the grounding. + const db = openGraphDatabase(fixture.dbPath); + try { + new FingerprintStore(db).saveGroundedSource({ + scaffoldFile: ".mex/patterns/removed.md", + nodeId: fixture.leaf, + source: "export function leaf() {}", + bodyHash: "0".repeat(64), + fingerprint: "mh:64:00", + }); + } finally { + db.close(); + } + writeFileSync(join(fixture.root, ".mex", "patterns", "removed.md"), "---\nname: removed\n---\n\n# Removed\n"); + + const records = await impact("leaf", fixture.root); + + expect(groundings(records)).toEqual([{ node: fixture.leaf, file: ".mex/context/architecture.md" }]); + }); + + it("attaches a grounding recorded under a renamed node's old id to the current node", async () => { + const fixture = await built("mex-impact-alias-"); + const oldId = "function:00000000000000000000000000000001"; + const db = openGraphDatabase(fixture.dbPath); + try { + db.prepare( + `INSERT INTO node_aliases (alias_id, canonical_node_id, match_method, confidence, created_at) + VALUES (?, ?, 'test', 1, 0)`, + ).run(oldId, fixture.leaf); + } finally { + db.close(); + } + writeFileSync(join(fixture.root, ".mex", "context", "architecture.md"), rootShape("architecture", [oldId])); + + const records = await impact("leaf", fixture.root); + + expect(groundings(records)).toEqual([{ node: fixture.leaf, file: ".mex/context/architecture.md" }]); + }); + + it("returns root-shape, mex-shape and mixed-shape groundings alike", async () => { + const fixture = await built("mex-impact-shapes-"); + const scaffold = join(fixture.root, ".mex"); + writeFileSync(join(scaffold, "context", "root.md"), rootShape("root", [fixture.leaf])); + writeFileSync(join(scaffold, "patterns", "migrated.md"), mexShape("migrated", "mx_01KR2E4K002H3ZYA9G0C4XV531", [fixture.leaf])); + // Setup's own leftover shape: a `mex` map, with the root key beside it. + writeFileSync( + join(scaffold, "patterns", "mixed.md"), + mexShape("mixed", "mx_01KRMEXM00JAAVJPQVVRX8N56V", [fixture.parent], [fixture.leaf]), + ); + + const records = await impact("leaf", fixture.root); + + expect(groundings(records)).toEqual([ + { node: fixture.leaf, file: ".mex/context/root.md" }, + { node: fixture.leaf, file: ".mex/patterns/migrated.md" }, + { node: fixture.leaf, file: ".mex/patterns/mixed.md" }, + { node: fixture.parent, file: ".mex/patterns/mixed.md" }, + ]); + }); + + it("names a scaffold file over the read bound instead of dropping its links silently", async () => { + const fixture = await built("mex-impact-oversized-"); + writeFileSync(join(fixture.root, ".mex", "context", "architecture.md"), rootShape("architecture", [fixture.leaf])); + const oversized = rootShape("oversized", [fixture.leaf]) + "x".repeat(WIKI_CORPUS_LIMITS.maxFileBytes); + writeFileSync(join(fixture.root, ".mex", "patterns", "oversized.md"), oversized); + + const records = await impact("leaf", fixture.root); + + expect(groundings(records)).toEqual([{ node: fixture.leaf, file: ".mex/context/architecture.md" }]); + expect(records.filter((record) => record.type === "grounding-omitted")).toEqual([{ + type: "grounding-omitted", + reason: "scaffold-unreadable", + files: [".mex/patterns/oversized.md"], + }]); + // The omission is a record like any other, admitted before the summary. + expect(records.at(-1)).toMatchObject({ type: "summary" }); + }); + + it("withholds every link when the scaffold changes during the call, and says so", async () => { + const fixture = await built("mex-impact-scaffold-race-"); + const architecture = join(fixture.root, ".mex", "context", "architecture.md"); + writeFileSync(architecture, rootShape("architecture", [fixture.leaf])); + + const records = await impact("leaf", fixture.root, { + afterCommittedGroundingRead: () => { + writeFileSync(architecture, `${readFileSync(architecture, "utf-8")}\nEdited mid-call.\n`); + }, + }); + + expect(groundings(records)).toEqual([]); + expect(records.filter((record) => record.type === "grounding-omitted")).toEqual([{ + type: "grounding-omitted", + reason: "scaffold-changed", + files: [".mex/context/architecture.md"], + }]); + // The graph half of the answer is unaffected. + expect(records.some((record) => record.type === "defines")).toBe(true); + }); + + it("returns the same scaffold files as wiki for-code for each grounded node", async () => { + const fixture = await built("mex-impact-for-code-"); + const scaffold = join(fixture.root, ".mex"); + const db = openGraphDatabase(fixture.dbPath); + const engine = createGraphEngine({ rootDir: fixture.root }); + try { + const seam = createGroundingGraph(engine, new MinHashReconciler(new FingerprintStore(db)), db); + const leaf = deriveGrounding(seam, fixture.leaf)!; + const parent = deriveGrounding(seam, fixture.parent)!; + const real = (node: string): string => node === fixture.leaf ? leaf.fingerprint : parent.fingerprint; + const withFingerprints = (text: string): string => + text.replace(/node: (\S+)\n(\s+)fingerprint: mh:64:00/g, (_all, node: string, indent: string) => + `node: ${node}\n${indent}fingerprint: ${real(node)}`); + writeFileSync(join(scaffold, "patterns", "migrated.md"), + withFingerprints(mexShape("migrated", "mx_01KR2E4K002H3ZYA9G0C4XV531", [fixture.leaf]))); + writeFileSync(join(scaffold, "patterns", "mixed.md"), + withFingerprints(mexShape("mixed", "mx_01KRMEXM00JAAVJPQVVRX8N56V", [fixture.parent], [fixture.leaf]))); + writeFileSync(join(scaffold, "patterns", "callers.md"), + withFingerprints(mexShape("callers", "mx_01M1M0CJJD2AQZ6XKHV4VKYTGJ", [fixture.parent]))); + rebuildWikiIndex({ + scaffoldRoot: scaffold, + indexPath: join(scaffold, "wiki.db"), + resolveGrounding: (grounding) => resolveGrounding(grounding, seam), + }); + } finally { + engine.close(); + db.close(); + } + + const records = await impact("leaf", fixture.root); + for (const node of [fixture.leaf, fixture.parent]) { + const fromImpact = groundings(records).filter((record) => record.node === node).map((record) => record.file); + const fromWiki = knowledgeRecordsFor([node], {}, fixture.root).map((record) => `.mex/${record.file}`).sort(); + expect(fromImpact.length).toBeGreaterThan(0); + expect(fromImpact).toEqual(fromWiki); + } + }); +}); From f7bf180e3cbde10c1d8c00537abbcf6bc199c65a Mon Sep 17 00:00:00 2001 From: Yashasvi Date: Fri, 25 Sep 2026 02:32:41 +0530 Subject: [PATCH 2/2] refactor: move the committed-grounding reader out of src/graph (#224) No module under src/graph imports the Wiki. The reader uses the Wiki index's discovery and contained reader, so it now sits beside markdown.ts, which the graph already uses the same way. The Wiki modules it imports reference nothing in the graph, so there is no cycle. --- src/{graph => }/committed-groundings.ts | 16 ++++++++++------ src/graph/cli-agent.ts | 2 +- 2 files changed, 11 insertions(+), 7 deletions(-) rename src/{graph => }/committed-groundings.ts (93%) diff --git a/src/graph/committed-groundings.ts b/src/committed-groundings.ts similarity index 93% rename from src/graph/committed-groundings.ts rename to src/committed-groundings.ts index b5ce06dd..c8978ffb 100644 --- a/src/graph/committed-groundings.ts +++ b/src/committed-groundings.ts @@ -24,6 +24,10 @@ * descriptor-bound reads and the byte ceilings are the ones already reviewed, * not a second set. * + * It lives beside `markdown.ts` rather than under `src/graph/` for the same + * reason that module does: no graph module imports the Wiki, and the Wiki + * modules used here import nothing from the graph, so no cycle is formed. + * * Groundings are read through `extractGroundings`, which takes the union of the * root `grounds_to` and `mex.grounds_to` (#226). Inline `mex://` anchors are * not read: they are links in prose, the Wiki does not index them as @@ -49,12 +53,12 @@ import { lstatSync } from "node:fs"; import { relative, resolve } from "node:path"; -import { loadConfiguredWikiConfig } from "../config.js"; -import { extractGroundings } from "../markdown.js"; -import { toPosix } from "../paths.js"; -import { addWikiCorpusBytes, WikiCorpusLimitError, type WikiCorpusLimit } from "../wiki/index/corpus-policy.js"; -import { discoverMarkdownFiles, type DiscoveredFile } from "../wiki/index/discover.js"; -import { readContainedSource } from "../wiki/index/source-read.js"; +import { loadConfiguredWikiConfig } from "./config.js"; +import { extractGroundings } from "./markdown.js"; +import { toPosix } from "./paths.js"; +import { addWikiCorpusBytes, WikiCorpusLimitError, type WikiCorpusLimit } from "./wiki/index/corpus-policy.js"; +import { discoverMarkdownFiles, type DiscoveredFile } from "./wiki/index/discover.js"; +import { readContainedSource } from "./wiki/index/source-read.js"; /** One declared grounding: a scaffold file, relative to the project root, and the node it names. */ export interface CommittedGrounding { diff --git a/src/graph/cli-agent.ts b/src/graph/cli-agent.ts index dae252e9..c159e7b2 100644 --- a/src/graph/cli-agent.ts +++ b/src/graph/cli-agent.ts @@ -17,7 +17,7 @@ import { import { committedGroundingsChangedSince, observeCommittedGroundings, type CommittedGrounding, type CommittedGroundingObservation, -} from "./committed-groundings.js"; +} from "../committed-groundings.js"; import { FingerprintStore } from "./fingerprint-store.js"; import { serializeFingerprint } from "./fingerprint.js"; import {