Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 47 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -182,18 +182,48 @@ When tools disagree:
6. genuine tool limitations are documented;
7. the result is re-measured.

**24 of 45 languages get all three tools compared, 16 more get two, and
5 GitGalaxy-only languages** (`abap`, `dockerfile`, `jcl`, `livecode`,
`yaml`) get hand-reviewed manual verification instead of cross-tool
agreement. Of the 180 discrepancy shapes logged so far, **87 are
validated (48%)** — read, investigated, and recorded with a verdict,
not just counted.

The goal is to finish the audit, close remaining GitGalaxy defects,
establish independent ground truth where necessary, and then publish
final precision/recall measurements. See
**24 of 45 languages get all three tools compared, 13 more get two, and
5** (`abap`, `agc_assembly`, `dockerfile`, `jcl`, `livecode`) **use
committed manual verification** — in whole, or for the `args` metric
where no tool emits a signature — instead of cross-tool agreement. Of
the 201 discrepancy shapes logged, **200 are validated (99.5%)** — read
against real source, investigated, and recorded with a verdict, not just
counted.

**Current state (2026-08-30), on the pinned corpus:**

- **Functions — precision:** 100.0% validated across all 31
tree-sitter-comparable languages. Once every three-way disagreement is
read and verdicted, every function GitGalaxy reports is a real
function.
- **Functions — recall:** 100.0% for 30 of 31. Shell measures 99.8% — a
single nested function definition (inside an `if` guard) that
GitGalaxy's top-level-only shell extractor doesn't reach by design.
Every other apparent miss is a validated comparison-tool artifact
(both tools independently hallucinating the same macro, per-clause
tagging of one Haskell function, and so on).
- **Classes:** GitGalaxy is never the tool found wrong in any class
disagreement — 100.0% validated recall and precision wherever a class
ground truth exists.
- **Arguments:** GitGalaxy is never debited in any *validated* argument
disagreement either; each one resolves to tree-sitter or ctags
miscounting, or a language with no formal parameter list. One small
Objective-C shape is still unverified.

This is a narrow benchmark: three structural targets, one fixed corpus,
and several languages whose entity counts are small enough that a
percentage means little. It is not "GitGalaxy parses as accurately as an
AST" in general — it is "for the entities GitGalaxy's graph needs,
targeted extraction recovers them as completely as established parsers,
here." Real code outside the corpus will surface shapes it doesn't cover;
the mandatory occurrence-level
[recall audit](docs/self_scan/tri_comparison_README.md#the-recall-audit-is-gitgalaxy-missing-anything)
is the standing process for the next one.

See
[the tri-comparison methodology doc](docs/self_scan/tri_comparison_README.md)
for how matching, the ledger lifecycle, and CI enforcement work.
for how matching, the ledger lifecycle, the recall audit, and CI
enforcement work.

![Tri-comparison](docs/self_scan/tri_comparison_chart.svg)

Expand Down Expand Up @@ -413,11 +443,14 @@ never hand-edited.
### Tri-comparison

The same corpus is analyzed against GitGalaxy, Tree-sitter and Ctags
where coverage exists — 24 of 45 languages get all three tools, 87 of
180 logged discrepancies validated so far. See
where coverage exists — 24 of 45 languages get all three tools, 200 of
201 logged discrepancies validated. On the pinned corpus, GitGalaxy is
never the tool found wrong in a validated function-precision, class, or
argument disagreement, and matches tree-sitter's function recall
everywhere except one nested shell definition it skips by design. See
[the methodology](docs/self_scan/tri_comparison_README.md) and the
["structural validation" section above](#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags)
for the full picture.
for the full picture and its limits.

### Raw repository output

Expand Down
Loading