diff --git a/.claude/skills/tri-comparison-ledger-sweep/SKILL.md b/.claude/skills/tri-comparison-ledger-sweep/SKILL.md index 2e83a64e2..e1336e376 100644 --- a/.claude/skills/tri-comparison-ledger-sweep/SKILL.md +++ b/.claude/skills/tri-comparison-ledger-sweep/SKILL.md @@ -370,6 +370,95 @@ lines that broke `func_start` confirmed the identical false-positive, requiring same PR). A shared root cause fixed in only one of two structurally-identical rule keys is a half-fixed bug, not a complete one. +## 2.6. The recall audit -- MANDATORY, every function GitGalaxy does not detect gets individually assessed + +**This is a validation GATE, not an optional deep-dive. A language's tri-comparison work is not +"done" -- step 8's capstone must not be written -- until every function tree-sitter OR ctags +reports that GitGalaxy does not, across the WHOLE corpus, has been individually looked at and +sorted into exactly one of the two buckets below.** The point is a hard guarantee: *we know +whether GitGalaxy is missing anything real, per language, and it is written down.* + +Why this needs to be its own step and can't be left to per-shape investigation: the ledger's +shapes are keyed by discrepancy *pattern*. A shape like `/function/existence/ +agree[tree_sitter]_vs[ctags,gitgalaxy]` can be marked `status: validated` with a verdict of +"tree-sitter over-detects here" **without anyone having confirmed that every occurrence in that +bucket is a tree-sitter error rather than a GitGalaxy miss.** A blanket verdict silently hides a +real recall gap sitting in the same bucket. Confirmed, not hypothetical (2026-08-29, cpp): the +`agree[tree_sitter]_vs[ctags,gitgalaxy]` bucket had ~164 occurrences; ~96 were `OPCODE(...)` +bytecode-dispatch-macro hallucinations, ~60 were `_FORCE_INLINE_`-macro mis-parses and bodyless +`= delete`/`= default` special members -- all tree-sitter's fault -- but **~2 were genuine +GitGalaxy recall gaps** (`STDAPI DllCanUnloadNow()` macro-return-type form; K&R-style +`PRIVATE void PlinkPrint(out,plp,tag)` with untyped param declarations on the following lines) +that the "tree-sitter over-detects" framing would have buried forever. + +### How to run it + +Do it per language, against BOTH tools, at the OCCURRENCE level (not just name-set diff -- a name +GitGalaxy finds 3 times and tree-sitter finds 5 times is 2 unassessed non-detections). Use +`tests/tools/recall_audit.py` (`python tests/tools/recall_audit.py [ ...]`, or no +args for the full set) -- it enumerates every `tree-sitter-finds / GitGalaxy-misses` and +`ctags-only-finds / GitGalaxy-misses` occurrence with its source line, so the only thing left is +reading source and classifying. It also cross-checks `measure()`'s own `real_functions - +found_functions` (the number the published accuracy table's recall is computed from) so a +divergence between the raw name-diff and the audit's counted misses is itself surfaced. + +### The two buckets -- every non-detection lands in exactly one + +1. **A real GitGalaxy recall gap.** A body-bearing definition, in live (non-dead-code) source, + that GitGalaxy's `func_start` genuinely does not match or that `detector.py`'s slicing drops. + -> **GitHub issue with an isolated repro** (the exact source form + the minimal failing case), + same standing rule as step 4.3's bucket 1. Group multiple occurrences of the *same* form into + one issue. If the fix is small and well-scoped (a regex widening with a regression check), + ship it in the same pass per bucket 1's extension. + +2. **A comparison-tool or audit-tool artifact -- GitGalaxy is correct.** Name the specific + mechanism; do not accept a vague "tree-sitter over-detects." The mechanisms seen so far, each + with a different correct resolution: + - **Macro-invocation hallucination** (`OPCODE(X) {`, `ENTRY(X): {`, `IFACEMETHOD_(...)`) -- + tree-sitter/ctags read a function-like macro call as a definition. Resolution: the audit's + tree-sitter walk must filter it. C already has `_C_KNOWN_MACRO_HALLUCINATIONS`; add/extend + the per-language set (`_CPP_KNOWN_MACRO_HALLUCINATIONS`, ...) rather than leaving the audit + to count it against GitGalaxy's recall forever. + - **Dead preprocessor code** (`#if 0 ... #endif`, `#if FALSE`) -- tree-sitter has no + preprocessor model and parses the dead branch. Resolution: detect the guard and drop those + tree-sitter functions from ground truth (this is `docs/why_gitgalaxy_beats_ast_here.md` + Claim 8's exact shape -- add the concrete citations there too). + - **Bodyless declaration** -- a forward declaration / prototype / `= default` / `= delete` + special member / abstract signature with no body. GitGalaxy deliberately only counts + body-bearing definitions. Resolution: the audit already drops these for perl/typescript; + extend the drop to the language in hand. + - **Deliberate scope boundary** -- a nested closure GitGalaxy does not record as a peer node + (shell `zgrep` inside an `if` guard, a 2-levels-deep lambda). Already an accepted design + choice; confirm the verdict says so and move on. + - **Naming-convention mismatch** -- both tools found it, they just name it differently + (`operator String() const` vs `operator String`; ctags' full ObjC selector `linkTo:` vs + GitGalaxy's `linkTo`; tree-sitter naming `- unsigned char foo` by the return type + `unsigned`). Resolution: canonicalize the tree-sitter/ctags name to GitGalaxy's convention + in the shared reader (`_get_node_name` / `ctags_reader.py`), OR document the mapping. NOT a + recall gap -- the function is found. + - **Occurrence-alignment fuzz** -- a name defined many times where the rank/line pairing in + `_align_occurrences_by_line` leaves a phantom "unmatched real" even though GitGalaxy's own + count for that name is correct. Resolution: confirm GitGalaxy's occurrence count matches the + real source count for that name, note it, and (if it's inflating a published number) widen + the pairing tolerance. + +### Log the cleared result + +The language's capstone §9 (step 8) must state, explicitly: the recall number, that **every** +non-detection was individually assessed, the count in each bucket, and -- for bucket 2 -- the +per-mechanism breakdown with at least one real `file:line` per mechanism. A capstone that just +says "recall 98.6%" without the assessment breakdown is not finished. If any audit-tool fix from +bucket 2 changed the published recall/precision number, the capstone and the summary table must +both reflect the post-fix number, and the baseline is regenerated. + +### Cross-language, once + +After the per-language passes, run `recall_audit.py` with no args one time and confirm the only +remaining `tree-sitter-finds / GitGalaxy-misses` occurrences anywhere are the ones already filed +as issues. That list -- ideally short, every entry pointing at an open issue -- is the standing +answer to "is GitGalaxy missing anything," and belongs in `docs/self_scan/ +tri_comparison_README.md` so it doesn't have to be re-derived. + ## 3. Dispatch to Gemini -- read-only, no worktree, self-contained prompt Use `Agent` with `subagent_type: gemini-analyzer`, `run_in_background: true`. One dispatch per @@ -590,8 +679,12 @@ PR sit open and rot into a conflict is now proactive by default. ## 8. Capstone: when a language's backlog clears, write it up before moving on **Trigger:** every currently-reproducing shape for a language is `status: "validated"` (check with -the same query step 1 uses, filtered to that language, confirming zero results). Do this BEFORE -starting the next language, not as a someday follow-up -- the whole reason it's cheap right now is +the same query step 1 uses, filtered to that language, confirming zero results) **AND step 2.6's +recall audit for that language is complete -- every function GitGalaxy does not detect has been +individually assessed and sorted, with the breakdown ready to write into §9.** Both conditions, +not just the first: a fully-`validated` ledger with an unaudited recall gap is exactly the +silent-miss failure step 2.6 exists to prevent. Do this BEFORE starting the next language, not as +a someday follow-up -- the whole reason it's cheap right now is that every file:line citation, every confirmed mechanism, and every "is this GitGalaxy's fault" verdict is still loaded in this session's context. Reconstructing that same picture from a cold read of the ledger later costs real tokens and real judgment a fresh session doesn't have for @@ -624,6 +717,10 @@ Two things to produce, both while the context is cheap: - Summary stats: shapes investigated, occurrences covered, confirmed GitGalaxy engine defects found (compare across languages -- `rust` found 2 real ones, `c` found zero, and that contrast IS the finding, not a gap in one or the other). + - **The step 2.6 recall-audit result**: the recall number, "every non-detection individually + assessed", the bucket-1 count (real gaps, each linked to its issue) and the bucket-2 + per-mechanism breakdown with a real `file:line` each. This is the part that lets a reader + trust the recall number instead of taking it on faith. - "Where GitGalaxy wins outright" -- the confirmed cases, with real file:line citations, not hand-wavy summaries. - "Where the other tools have real, documented gaps" -- same standard. diff --git a/docs/self_scan/tri_comparison_README.md b/docs/self_scan/tri_comparison_README.md index 123b4b83c..407a55ffb 100644 --- a/docs/self_scan/tri_comparison_README.md +++ b/docs/self_scan/tri_comparison_README.md @@ -69,6 +69,29 @@ same repo: `credit_tools` only makes sense for a shape where ONE tool's claim wa crediting a tool that's already in a 2-of-3 agreeing pair double-counts, since that pair already gets precision credit from the base reconciliation). +## The recall audit: is GitGalaxy missing anything? + +A validated shape verdict says what's true about a *pattern* — it does not, on its own, prove +that every occurrence in that pattern's bucket is a tree-sitter/ctags error and not a real +GitGalaxy miss hiding among them (cpp, 2026-08-29: ~164 "misses" in one bucket, ~162 tree-sitter +macro/parse artifacts, 2 genuine GitGalaxy recall gaps the "tree-sitter over-detects" verdict +would have buried). So the `tri-comparison-ledger-sweep` skill's **step 2.6** requires a +separate, occurrence-level pass: every function tree-sitter OR ctags reports that GitGalaxy does +not is individually read and sorted into either a filed GitHub issue (a real recall gap) or a +named tool/audit artifact (macro hallucination, `#if 0` dead code, bodyless declaration, +deliberate scope boundary, naming-convention mismatch, occurrence-alignment fuzz). Run +`python tests/tools/recall_audit.py` (no args = every language) to enumerate them; a language's +step 8 capstone is not done until its list is empty of unassessed entries. + +**Standing answer — the only real GitGalaxy function-recall gaps across the whole corpus:** + + +_(populated by the sweep — each entry is `: -> #`)_ + + +Everything else `recall_audit.py` prints is a comparison-tool artifact, catalogued per language +in that language's `docs/language_status/.md` §9. + ## The catalog: confirmed, evidenced differences between the three tools This is the part `docs/self_scan/README.md` was missing — not a live list of open questions (that's diff --git a/tests/tools/recall_audit.py b/tests/tools/recall_audit.py new file mode 100644 index 000000000..b063ea941 --- /dev/null +++ b/tests/tools/recall_audit.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +""" +recall_audit.py + +Enumerates every function tree-sitter OR universal-ctags reports that GitGalaxy does NOT, across +the whole language-crucible corpus, per language, at the OCCURRENCE level (a name GitGalaxy finds +3x and tree-sitter finds 5x is 2 non-detections, not 0). For each, prints the source line so the +only work left is reading it and sorting it into one of the two buckets in the +`tri-comparison-ledger-sweep` skill's step 2.6: + + 1. a real GitGalaxy recall gap -> file a GitHub issue with an isolated repro + 2. a comparison/audit-tool artifact -> name the mechanism, fix the audit or document it + +This is the standing answer to "is GitGalaxy missing anything?" -- run it with no arguments for +the full sweep, or name languages to scope it. + + python tests/tools/recall_audit.py # every language + python tests/tools/recall_audit.py cpp shell # just these + +It is verification-only tooling (needs tree-sitter-language-pack, optionally universal-ctags), +never imported by anything under gitgalaxy/. Same optional-dependency contract as +tri_comparison_gatherer.py. +""" + +from __future__ import annotations + +import collections +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent)) + +import tree_sitter_accuracy_audit as tsaa +from tri_comparison_gatherer import gather_language + +# The languages with a tree-sitter baseline (NODE_MAPS) plus the ctags-comparable gg-only set -- +# the exact universe where a "GitGalaxy misses X" question is answerable. +_TS_LANGS = sorted(tsaa.NODE_MAPS) + + +def _src_line(corpus_dir: Path, relpath: str, lineno: int) -> str: + p = corpus_dir / relpath + try: + lines = p.read_text(encoding="utf-8", errors="replace").splitlines() + except OSError: + return "" + if 1 <= lineno <= len(lines): + return lines[lineno - 1].strip()[:120] + return "" + + +def audit_language(lang: str) -> dict: + readings = gather_language(lang) + corpus_dir = tsaa.ensure_corpus(lang) + + ts_misses: list[tuple[str, str, int]] = [] # (file, name, line) + ct_misses: list[tuple[str, str, int]] = [] + + for fr in readings: + gg_by = collections.Counter(o.name for o in fr.gg_funcs) + + ts_lines: dict[str, list[int]] = collections.defaultdict(list) + for o in fr.ts_funcs: + ts_lines[o.name].append(o.line) + ct_lines: dict[str, list[int]] = collections.defaultdict(list) + for o in fr.ctags_funcs: + ct_lines[o.name].append(o.line) + + for name, lines in ts_lines.items(): + surplus = sorted(lines)[gg_by.get(name, 0) :] + ts_misses.extend((fr.file_path, name, ln) for ln in surplus) + + ts_names = set(ts_lines) + for name, lines in ct_lines.items(): + if name in ts_names: + continue # already covered by the tree-sitter column + surplus = sorted(lines)[gg_by.get(name, 0) :] + ct_misses.extend((fr.file_path, name, ln) for ln in surplus) + + return { + "lang": lang, + "corpus_dir": corpus_dir, + "ts_misses": sorted(ts_misses), + "ct_misses": sorted(ct_misses), + } + + +def _measure_counted_misses(lang: str) -> int | None: + """The number the published accuracy table's recall is computed from: + real_functions - found_functions from measure().""" + if lang not in tsaa.NODE_MAPS: + return None + try: + m = tsaa.measure(lang) + except SystemExit: + return None + return m["real_functions"] - m["found_functions"] + + +def main(argv: list[str]) -> int: + langs = argv[1:] or _TS_LANGS + grand_ts = 0 + for lang in langs: + try: + res = audit_language(lang) + except Exception as exc: + print(f"\n{'=' * 78}\n{lang}: SKIPPED -- {exc}\n{'=' * 78}") + continue + + counted = _measure_counted_misses(lang) + n_ts = len(res["ts_misses"]) + n_ct = len(res["ct_misses"]) + grand_ts += n_ts + print(f"\n{'=' * 78}\n{lang}\n{'=' * 78}") + if counted is not None: + note = "" + if counted != n_ts: + note = f" <-- differs from raw name-diff ({n_ts}); check drop-rules / alignment" + print(f" measure() counted misses (recall denominator gap): {counted}{note}") + + print(f"\n tree-sitter finds / GitGalaxy misses ({n_ts}):") + for f, name, ln in res["ts_misses"]: + print(f" {f}:{ln} {name!r}") + print(f" | {_src_line(res['corpus_dir'], f, ln)}") + + if n_ct: + print(f"\n ctags-only finds / GitGalaxy misses ({n_ct}):") + per_file = collections.Counter(f for f, _, _ in res["ct_misses"]) + shown = collections.Counter() + for f, name, ln in res["ct_misses"]: + shown[f] += 1 + if shown[f] <= 8: + print(f" {f}:{ln} {name!r} | {_src_line(res['corpus_dir'], f, ln)}") + for f, c in per_file.items(): + if c > 8: + print(f" ... {f}: +{c - 8} more") + + print(f"\n{'=' * 78}\nTOTAL tree-sitter-finds / GitGalaxy-misses across {len(langs)} language(s): {grand_ts}") + print("Every one must be sorted into skill step 2.6's bucket 1 (issue) or bucket 2 (artifact).") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv))