diff --git a/README.md b/README.md index 67ce8917b..d7eba9ac5 100644 --- a/README.md +++ b/README.md @@ -182,18 +182,48 @@ When tools disagree: 6. genuine tool limitations are documented; 7. the result is re-measured. -**24 of 45 languages get all three tools compared, 16 more get two, and -5 GitGalaxy-only languages** (`abap`, `dockerfile`, `jcl`, `livecode`, -`yaml`) get hand-reviewed manual verification instead of cross-tool -agreement. Of the 180 discrepancy shapes logged so far, **87 are -validated (48%)** — read, investigated, and recorded with a verdict, -not just counted. - -The goal is to finish the audit, close remaining GitGalaxy defects, -establish independent ground truth where necessary, and then publish -final precision/recall measurements. See +**24 of 45 languages get all three tools compared, 13 more get two, and +5** (`abap`, `agc_assembly`, `dockerfile`, `jcl`, `livecode`) **use +committed manual verification** — in whole, or for the `args` metric +where no tool emits a signature — instead of cross-tool agreement. Of +the 201 discrepancy shapes logged, **200 are validated (99.5%)** — read +against real source, investigated, and recorded with a verdict, not just +counted. + +**Current state (2026-08-30), on the pinned corpus:** + +- **Functions — precision:** 100.0% validated across all 31 + tree-sitter-comparable languages. Once every three-way disagreement is + read and verdicted, every function GitGalaxy reports is a real + function. +- **Functions — recall:** 100.0% for 30 of 31. Shell measures 99.8% — a + single nested function definition (inside an `if` guard) that + GitGalaxy's top-level-only shell extractor doesn't reach by design. + Every other apparent miss is a validated comparison-tool artifact + (both tools independently hallucinating the same macro, per-clause + tagging of one Haskell function, and so on). +- **Classes:** GitGalaxy is never the tool found wrong in any class + disagreement — 100.0% validated recall and precision wherever a class + ground truth exists. +- **Arguments:** GitGalaxy is never debited in any *validated* argument + disagreement either; each one resolves to tree-sitter or ctags + miscounting, or a language with no formal parameter list. One small + Objective-C shape is still unverified. + +This is a narrow benchmark: three structural targets, one fixed corpus, +and several languages whose entity counts are small enough that a +percentage means little. It is not "GitGalaxy parses as accurately as an +AST" in general — it is "for the entities GitGalaxy's graph needs, +targeted extraction recovers them as completely as established parsers, +here." Real code outside the corpus will surface shapes it doesn't cover; +the mandatory occurrence-level +[recall audit](docs/self_scan/tri_comparison_README.md#the-recall-audit-is-gitgalaxy-missing-anything) +is the standing process for the next one. + +See [the tri-comparison methodology doc](docs/self_scan/tri_comparison_README.md) -for how matching, the ledger lifecycle, and CI enforcement work. +for how matching, the ledger lifecycle, the recall audit, and CI +enforcement work. ![Tri-comparison](docs/self_scan/tri_comparison_chart.svg) @@ -413,11 +443,14 @@ never hand-edited. ### Tri-comparison The same corpus is analyzed against GitGalaxy, Tree-sitter and Ctags -where coverage exists — 24 of 45 languages get all three tools, 87 of -180 logged discrepancies validated so far. See +where coverage exists — 24 of 45 languages get all three tools, 200 of +201 logged discrepancies validated. On the pinned corpus, GitGalaxy is +never the tool found wrong in a validated function-precision, class, or +argument disagreement, and matches tree-sitter's function recall +everywhere except one nested shell definition it skips by design. See [the methodology](docs/self_scan/tri_comparison_README.md) and the ["structural validation" section above](#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags) -for the full picture. +for the full picture and its limits. ### Raw repository output