test(recall-audit): stop counting tree-sitter's mistakes as GitGalaxy misses — c/cpp/fortran/objc → ~100% recall - #2463
Merged
Merged
Conversation
squid-protocol
added a commit
that referenced
this pull request
Aug 29, 2026
#2459 is the shell keyword-as-argument issue this audit work surfaced, not the audit correction itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S
…laxy misses
Step 2.6's recall audit run against c / cpp / fortran / objective-c. Every
function tree-sitter reported that GitGalaxy did not was individually read;
the large majority were tree-sitter parse artifacts the accuracy audit was
wrongly folding into GitGalaxy's recall denominator. Corrections, all in
tree_sitter_accuracy_audit.py's ground-truth walk (real_funcs), never a
gated metric:
- `#if 0` / `#if false` dead-block detection (c + cpp) -- tree-sitter has
no preprocessor model and parses the dead branch. `_PyObject_Managed
DictValidityCheck`, `print_stack`, sqlite/lemon.c's K&R `PlinkPrint` /
`SetPrint` are all inside `#if 0`. docs Claim 8's exact shape.
- `_CPP_KNOWN_MACRO_HALLUCINATIONS` (mirror of `_C_...`) -- `OPCODE(X) {`
(~96, godot's bytecode computed-goto table), `IFACEMETHOD_`, and the
bare control-flow keywords (`if`/`for`/...) error recovery emits as a
"function name" after a macro in statement position.
- cpp `= default` / `= delete` special members -- not body-bearing, so
GitGalaxy correctly doesn't count them (same rule as perl bodyless).
- cpp function_definition with an ERROR / class_specifier child -- a
corrupted `_FORCE_INLINE_`-mangled parse or a field-with-initializer
(`ptr_type _value = ptr_type();`) read as a definition. Name unreliable.
- cpp / fortran: a function_definition tree-sitter built INSIDE an
already-identified blind spot is not ground truth -- #1849 Phase 2 already
promotes GitGalaxy's correct reading there; this is the symmetric half.
- fortran: a WRF-style module file's `#ifdef VERT_UNIT` unit-test driver
`program X ... end program X` blocks are dead when built as a module;
mark them blind spots.
- objective-c `_get_node_name`: name a NeXT-era `- unsigned char foo` /
`- void bar` method (no parenthesised return type) by its real selector,
not by the return-type token tree-sitter picks. objc-only (the leading
`-`/`+` marker gates it away from JS/TS `method_definition`).
Measured (recall / precision):
c 99.7% -> 100.0% / 99.5% (unchanged)
cpp 89.3% -> 99.9% / 100.0% (unchanged)
fortran 98.6% -> 100.0% / 100.0% (unchanged)
objc 98.7% -> 100.0% / 98.1% -> 99.4%
`tree_sitter_accuracy_audit --all --ci` 31/31 OK; no engine change, no
golden-master re-bless. Baselines + summary table regenerated.
The genuine GitGalaxy recall gaps this surfaced are filed separately
(cpp macro-return-type `STDAPI Foo()`; shell keyword-as-argument;
lua multi-statement `local function` line). dart's remaining 9 stay in
#2072.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S
- tri_comparison_README.md's RECALL_AUDIT block: the complete list of genuine GitGalaxy function-recall gaps across the whole corpus (4 forms, each linked to #2459/#2460/#2461/#2462), plus a one-line note that everything else recall_audit.py prints is a catalogued tool artifact. - c.md / cpp.md / fortran.md / lua.md §9: a "Recall audit" subsection per skill step 2.6 -- the before/after recall number, "every non-detection individually assessed", and the per-mechanism bucket breakdown with file:line citations. - lua.md §5: corrected -- `constructs.lua:f` is a real gap (`local a; local function f`), not alignment fuzz; both lua misses are now #2461. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S
#2459 is the shell keyword-as-argument issue this audit work surfaced, not the audit correction itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S
squid-protocol
force-pushed
the
func-recall-fixes
branch
from
August 29, 2026 22:37
2155c98 to
fe24c7a
Compare
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Depends on #2458 (the step 2.6 process change +
recall_audit.py— this branch includes itscommit; merge #2458 first). This PR executes step 2.6 for the 4 languages whose func recall
was < 100% and dominated by measurement noise.
Every function tree-sitter reported that GitGalaxy did not was individually read. The corrections
are all in
tree_sitter_accuracy_audit.py's ground-truth walk (real_funcs) — never a gatedmetric.
Corrections (bucket 2 — tree-sitter artifacts)
#if 0/#if falsedead blocks (c + cpp)_find_dead_preproc_ranges(Claim 8)OPCODE(X) {(~96),IFACEMETHOD_, bareif/forafter a macro_CPP_KNOWN_MACRO_HALLUCINATIONS= default/= deletespecial members (cpp)_FORCE_INLINE_-mangled parses (ERROR child,class_specifieras return type, field-with-initializer)#ifdef VERT_UNITunit-testprogram X … end program Xdrivers- unsigned char fooobjc method by its return type_get_node_nameuses the real selector (objc-only)Result
tree_sitter_accuracy_audit --all --ci31/31 OK. No engine change, no golden-masterre-bless. Baselines + summary table regenerated.
Genuine GitGalaxy recall gaps this surfaced (bucket 1 — filed)
local functionnot first on its lineextension, generic-method prefixes (supersedes closed dart func_start: remaining recall/precision gaps from the switch-expression-arm sweep (#2071 follow-up) #2072)Standing list:
tri_comparison_README.md'sRECALL_AUDITblock; per-language detail in eachdocs/language_status/<lang>.md§9.🤖 Generated with Claude Code
https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S