From def65306a1290bcc1b916fa4e72f8c8a10f3e0df Mon Sep 17 00:00:00 2001 From: squid-protocol Date: Sat, 29 Aug 2026 22:27:06 -0400 Subject: [PATCH 1/2] docs(readme): function-detection target closed; refresh tri-comparison stats MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The "Structural validation" and "Evidence" sections carried stale numbers (87/180 discrepancies validated, 16 languages on a 2-tool comparison) from early in the tri-comparison sweep. Current: 200/201 shapes validated, 13 on a 2-tool comparison, and the manual-verification set is abap/agc_assembly/ dockerfile/jcl/livecode (yaml gained a comparator; agc_assembly needs manual verification for `args` since no tool emits a signature there). Adds the first closed target: every tree-sitter-comparable language (31) now measures 100.0% function recall and 100.0% validated function precision on the pinned corpus. Framed narrowly on purpose — one structural target (existence), one fixed corpus, small-n languages where the percentage means little, not an "as accurate as an AST" claim — with a pointer to the mandatory recall audit as the process for the next real-world gap. Class and args targets still open. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S --- README.md | 45 +++++++++++++++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 67ce8917b..2f9e35199 100644 --- a/README.md +++ b/README.md @@ -182,18 +182,33 @@ When tools disagree: 6. genuine tool limitations are documented; 7. the result is re-measured. -**24 of 45 languages get all three tools compared, 16 more get two, and -5 GitGalaxy-only languages** (`abap`, `dockerfile`, `jcl`, `livecode`, -`yaml`) get hand-reviewed manual verification instead of cross-tool -agreement. Of the 180 discrepancy shapes logged so far, **87 are -validated (48%)** — read, investigated, and recorded with a verdict, -not just counted. - -The goal is to finish the audit, close remaining GitGalaxy defects, -establish independent ground truth where necessary, and then publish -final precision/recall measurements. See +**24 of 45 languages get all three tools compared, 13 more get two, and +5** (`abap`, `agc_assembly`, `dockerfile`, `jcl`, `livecode`) **use +committed manual verification** — in whole, or for the `args` metric +where no tool emits a signature — instead of cross-tool agreement. Of +the 201 discrepancy shapes logged, **200 are validated (99.5%)** — read +against real source, investigated, and recorded with a verdict, not just +counted. + +**First target closed: function detection (2026-08-30).** Every one of +the 31 tree-sitter-comparable languages now measures **100.0% function +recall and 100.0% validated function precision** on the pinned corpus — +GitGalaxy finds every function tree-sitter finds, and every function it +reports is real, once each three-way disagreement has been read and +verdicted. This is a narrow claim: one structural target (existence, not +argument counts, not scope nesting), one fixed corpus, and several +languages whose function counts are small enough that the percentage +carries little weight — it is not "GitGalaxy parses as accurately as an +AST" in general. Real code outside the corpus will surface shapes it +doesn't cover; the mandatory occurrence-level +[recall audit](docs/self_scan/tri_comparison_README.md#the-recall-audit-is-gitgalaxy-missing-anything) +is the standing process for the next one. Class and argument targets are +still in progress. + +See [the tri-comparison methodology doc](docs/self_scan/tri_comparison_README.md) -for how matching, the ledger lifecycle, and CI enforcement work. +for how matching, the ledger lifecycle, the recall audit, and CI +enforcement work. ![Tri-comparison](docs/self_scan/tri_comparison_chart.svg) @@ -413,11 +428,13 @@ never hand-edited. ### Tri-comparison The same corpus is analyzed against GitGalaxy, Tree-sitter and Ctags -where coverage exists — 24 of 45 languages get all three tools, 87 of -180 logged discrepancies validated so far. See +where coverage exists — 24 of 45 languages get all three tools, 200 of +201 logged discrepancies validated. Function detection is the first +target closed: 100.0% recall and 100.0% validated precision across all +31 tree-sitter-comparable languages, on the pinned corpus. See [the methodology](docs/self_scan/tri_comparison_README.md) and the ["structural validation" section above](#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags) -for the full picture. +for the full picture and its limits. ### Raw repository output From 403f80d63a9a67b6608669f2031b3ca97a6d20c4 Mon Sep 17 00:00:00 2001 From: squid-protocol Date: Sat, 29 Aug 2026 23:49:32 -0400 Subject: [PATCH 2/2] =?UTF-8?q?docs(readme):=20correct=20the=20accuracy=20?= =?UTF-8?q?claim=20=E2=80=94=20classes/args=20are=20done=20too,=20shell=20?= =?UTF-8?q?recall=20is=2030/31?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first draft said "class and argument targets still in progress" and "all 31 at 100% function recall" — both overstated in one direction and understated in the other. Corrected against the ledger: - Function precision: 100% validated, all 31 (accurate). - Function recall: 100% for 30 of 31. Shell is 99.8% — one function defined inside an `if` guard that the top-level-only Mode-D shell extractor skips by design (validated as an intentional limitation, ledger shape `shell/function/existence/agree[ctags,tree_sitter]_vs[gitgalaxy]`). The c and haskell `_vs[gitgalaxy]` shapes are validated comparison-tool artifacts (shared macro hallucination; multi-clause tagging) with 0 counted misses. - Classes: GitGalaxy is never the debited tool in any class shape — 100% validated recall and precision wherever a class ground truth exists. - Args: GitGalaxy is never debited in any *validated* args shape; every one is tree-sitter/ctags miscounting or a no-formal-params language. One objc shape (n=2) remains unvalidated. Reframed as "current state" bullets rather than "first target closed". Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_019Zm1uVBpVEZJ9SB7bWFR8S --- README.md | 48 ++++++++++++++++++++++++++++++++---------------- 1 file changed, 32 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 2f9e35199..d7eba9ac5 100644 --- a/README.md +++ b/README.md @@ -190,20 +190,35 @@ the 201 discrepancy shapes logged, **200 are validated (99.5%)** — read against real source, investigated, and recorded with a verdict, not just counted. -**First target closed: function detection (2026-08-30).** Every one of -the 31 tree-sitter-comparable languages now measures **100.0% function -recall and 100.0% validated function precision** on the pinned corpus — -GitGalaxy finds every function tree-sitter finds, and every function it -reports is real, once each three-way disagreement has been read and -verdicted. This is a narrow claim: one structural target (existence, not -argument counts, not scope nesting), one fixed corpus, and several -languages whose function counts are small enough that the percentage -carries little weight — it is not "GitGalaxy parses as accurately as an -AST" in general. Real code outside the corpus will surface shapes it -doesn't cover; the mandatory occurrence-level +**Current state (2026-08-30), on the pinned corpus:** + +- **Functions — precision:** 100.0% validated across all 31 + tree-sitter-comparable languages. Once every three-way disagreement is + read and verdicted, every function GitGalaxy reports is a real + function. +- **Functions — recall:** 100.0% for 30 of 31. Shell measures 99.8% — a + single nested function definition (inside an `if` guard) that + GitGalaxy's top-level-only shell extractor doesn't reach by design. + Every other apparent miss is a validated comparison-tool artifact + (both tools independently hallucinating the same macro, per-clause + tagging of one Haskell function, and so on). +- **Classes:** GitGalaxy is never the tool found wrong in any class + disagreement — 100.0% validated recall and precision wherever a class + ground truth exists. +- **Arguments:** GitGalaxy is never debited in any *validated* argument + disagreement either; each one resolves to tree-sitter or ctags + miscounting, or a language with no formal parameter list. One small + Objective-C shape is still unverified. + +This is a narrow benchmark: three structural targets, one fixed corpus, +and several languages whose entity counts are small enough that a +percentage means little. It is not "GitGalaxy parses as accurately as an +AST" in general — it is "for the entities GitGalaxy's graph needs, +targeted extraction recovers them as completely as established parsers, +here." Real code outside the corpus will surface shapes it doesn't cover; +the mandatory occurrence-level [recall audit](docs/self_scan/tri_comparison_README.md#the-recall-audit-is-gitgalaxy-missing-anything) -is the standing process for the next one. Class and argument targets are -still in progress. +is the standing process for the next one. See [the tri-comparison methodology doc](docs/self_scan/tri_comparison_README.md) @@ -429,9 +444,10 @@ never hand-edited. The same corpus is analyzed against GitGalaxy, Tree-sitter and Ctags where coverage exists — 24 of 45 languages get all three tools, 200 of -201 logged discrepancies validated. Function detection is the first -target closed: 100.0% recall and 100.0% validated precision across all -31 tree-sitter-comparable languages, on the pinned corpus. See +201 logged discrepancies validated. On the pinned corpus, GitGalaxy is +never the tool found wrong in a validated function-precision, class, or +argument disagreement, and matches tree-sitter's function recall +everywhere except one nested shell definition it skips by design. See [the methodology](docs/self_scan/tri_comparison_README.md) and the ["structural validation" section above](#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags) for the full picture and its limits.