From 25040d0115e16dc4f345839a8a3c30aa429b6247 Mon Sep 17 00:00:00 2001 From: squid-protocol Date: Mon, 31 Aug 2026 12:39:36 -0400 Subject: [PATCH] docs: product-first README restructure + keyword-rosetta consistency proof README.md 687 -> 387 lines. The long proof narrative (structural-extraction thesis, full tri-comparison section, validity ladder, risk-over-git-history experiment, research direction) moves intact to the new docs/validation.md; the README keeps a condensed "Accuracy, measured" summary holding the old #structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags anchor stable. New product section "What a scan gives you" leads: the six outputs, the LLM architecture brief showcased with two dated, reproducible scan excerpts from the current engine (curl: cross-language pillar/function ranking, 16.4% scanned caveat; cics-genapp: COBOL/DB2 risk hitlist, flat-graph pillar degeneration caveat). New proof: keyword-rosetta 46-language control corpus embedded via its raw main-branch chart URL (self-updating), shipped limitations-first (3/33 strict passes, avg 75% in-band), with a new "Measurement consistency" rung in the validity ladder. Kubernetes bullet gains its long-missing rule-4 limitation clause (fitted two-regime speed model). Pandoc-style tables converted to GFM pipe tables so they actually render on GitHub. how_to_maintain_the_readme.md section-order target updated to the new structure; readme_evidence_roadmap.md gains the shipped keyword-rosetta row. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01M11wL1mSgk1fghxz1PAWGY --- README.md | 705 +++++++++-------------------- docs/how_to_maintain_the_readme.md | 131 +++--- docs/readme_evidence_roadmap.md | 1 + docs/validation.md | 334 ++++++++++++++ 4 files changed, 604 insertions(+), 567 deletions(-) create mode 100644 docs/validation.md diff --git a/README.md b/README.md index d7eba9ac5..cd3bcaeea 100644 --- a/README.md +++ b/README.md @@ -3,9 +3,10 @@ **Repository-scale structural intelligence without compilation.** [Docs](https://squid-protocol.github.io/gitgalaxy/) · -[Visualizer](https://gitgalaxy.io/) · [Language -Crucible](https://github.com/squid-protocol/language-crucible) · [Raw -Output](https://github.com/squid-protocol/gitgalaxy-raw-output) +[Visualizer](https://gitgalaxy.io/) · +[Language Crucible](https://github.com/squid-protocol/language-crucible) · +[Keyword Rosetta](https://github.com/squid-protocol/keyword-rosetta) · +[Raw Output](https://github.com/squid-protocol/gitgalaxy-raw-output) **1 scan · 97 structural signals · 50+ languages · no compilation · 19 risk-exposure categories · 6 outputs** @@ -13,37 +14,10 @@ risk-exposure categories · 6 outputs** ## The short version GitGalaxy builds a **language-agnostic structural graph of an entire -repository** directly from source text. - -It is designed for repositories that are polyglot, partially broken, -legacy, vendor-heavy, or otherwise difficult to analyze through a -build-first workflow. - -Instead of requiring a successful build and a separate parser/toolchain -for every language, GitGalaxy extracts a common vocabulary of -**structural signatures**---functions, classes, arguments, control flow, -state mutation, I/O, APIs, dependencies, and other signals---and -normalizes those observations into one repository model. - -The same graph can then feed: - -- architecture analysis -- risk-exposure prioritization -- dependency/SBOM analysis -- refactoring and ownership analysis -- legacy-code analysis -- AI-oriented codebase context -- CI/CD workflows -- historical risk analysis - -> **Central thesis:** complete language parsing is not always necessary -> to recover highly useful structural information at repository scale. - ------------------------------------------------------------------------- +repository** directly from source text — no build, no per-language toolchain. -## The problem - -Large repositories routinely contain: +It is designed for repositories that are polyglot, partially broken, legacy, +vendor-heavy, or otherwise difficult to analyze through a build-first workflow: ``` text Go + C++ + Python + Java + Bash + YAML @@ -51,292 +25,174 @@ Go + C++ + Python + Java + Bash + YAML + half-migrated modules + broken dependencies ``` -Traditional language tooling can be excellent within its intended scope -while still leaving the repository fragmented across language-specific -representations. - -GitGalaxy makes a different trade: - -``` text -Source repository - | - v -Structural signatures - | - v -Normalized entities + risk signals - | - v -Deterministic repository graph - | - +---- Architecture - +---- Risk exposure - +---- Dependencies / SBOM - +---- AI context - +---- Refactoring - +---- Git-history analysis -``` +Instead of a separate parser per language, GitGalaxy extracts a common +vocabulary of **structural signatures** — functions, classes, arguments, +control flow, state mutation, I/O, APIs, dependencies — and normalizes them +into one deterministic repository model that feeds architecture analysis, +risk-exposure prioritization, SBOM generation, refactoring and ownership +analysis, AI-oriented codebase context, and CI/CD gates. -The objective is **not** to reproduce every syntactic detail of every -language. +> **Central thesis:** complete language parsing is not always necessary to +> recover highly useful structural information at repository scale. -The objective is to recover the structural information downstream -repository intelligence actually needs. +How that thesis is tested — against Tree-sitter and Ctags, against a planted +control corpus, and next against Git history — is summarized in +[Accuracy, measured](#accuracy-measured) below and laid out in full in +[the validation program](docs/validation.md). ------------------------------------------------------------------------ -## One graph, many consumers - -GitGalaxy's core output is a deterministic structural representation of -the repository. - - ----------------------------------------------------------------------- - Consumer Question - ----------------------------------- ----------------------------------- - Architecture What is this repository made of? - - Structural analysis Where are the functions, classes, - APIs, dependencies and control - structures? +## What a scan gives you - Risk exposure Where are potentially important - risk patterns concentrated? +One command: - Refactoring Which files are complex, high-churn - or load-bearing? - - Supply chain What dependencies physically exist - on disk? - - AI context What architecture and relationships - should an agent know? - - Legacy migration Where are the structural units to - transform? +``` bash +pip install gitgalaxy +galaxyscope path/to/repo +``` - Historical analysis How does measured exposure change - as the repository evolves? - ----------------------------------------------------------------------- +Six coordinated views of the same deterministic scan: + +| Output | Purpose | +|---|---| +| **LLM architecture brief** | Compact machine/agent-oriented context (below) | +| **SARIF** | CI/security dashboard integration | +| **CycloneDX SBOM** | Dependency inventory/compliance | +| **SQLite** | Queryable repository knowledge graph | +| **JSON audit data** | Forensic/automation workflows | +| **3D visualization data** | Interactive repository topology | + +### The architecture brief + +The flagship report is a single Markdown brief built to hand an engineer — or +an AI agent — a working mental model of a repository they've never seen. It is +a self-contained package: the risk equations are printed in the report itself, +and an embedded interpretation prompt lets any LLM narrate it without +hallucinating what the numbers mean. Sections cover macro state and language +composition, network topology (modularity, articulation points, cyclic +density), dependency choke points, the heaviest functions and files, per-file +structural signatures with PageRank blast radius, targeted and cumulative risk +hitlists, supply-chain audits, and refactoring targets ranked by volatility and +authorship centralization — plus an itemized list of every file it *refused* to +scan, and why. + +Two examples, scanned 2026-08-31 with the current engine. Both repositories +are public — clone either and run `galaxyscope --llm-only ` to reproduce +the full brief: + +**curl** — 4,250 artifacts, 696 scanned, 112,653 LOC across C, Perl, Python, +Shell, M4 and Makefile. The brief ranks `src/tool_setup.h` as the top +structural pillar (80 inbound connections) and puts a *Perl* function — +`APPEND_imap` in `tests/ftpserver.pl`, Impact 2135, 1,672 LOC — at the top of +the repo-wide function hitlist, in the same ranking as the C code. That +cross-language graph is the product: one comparable signal set across every +language in the repo. The honest caveat in the same brief: only 16.4% of +artifacts were scanned — the ingestion filter drops binaries, generated code +and test data aggressively, and §5 of the brief itemizes every exclusion by +extension and reason. + +**cics-genapp** (IBM's CICS COBOL/DB2 sample) — 92.1% scanned: 44 COBOL +programs, 29 JCL jobs. The cumulative-risk hitlist leads with +`base/src/lgupdb01.cbl` (state flux ~100%, cognitive load 92%), and the +heaviest paragraph in the repo is `UPDATE-POLICY-DB2-INFO` — the `SELECT FOR +UPDATE` row-locking logic, which is exactly where a maintainer of that program +would want to look first. The same brief also shows a limitation plainly: on a +flat architecture with no real import graph, the "structural pillars" list +degenerates to zero-connection files, and the report says to check the +connection counts before trusting it. + +Hundreds of unedited briefs for independently selected repositories are +committed at +[gitgalaxy-raw-output](https://github.com/squid-protocol/gitgalaxy-raw-output); +this repo's own always-current self-scan brief is at +[`docs/gitgalaxy_architecture_brief.md`](docs/gitgalaxy_architecture_brief.md). + +### One graph, many consumers + +| Consumer | Question | +|---|---| +| Architecture | What is this repository made of? | +| Structural analysis | Where are the functions, classes, APIs, dependencies and control structures? | +| Risk exposure | Where are potentially important risk patterns concentrated? | +| Refactoring | Which files are complex, high-churn or load-bearing? | +| Supply chain | What dependencies physically exist on disk? | +| AI context | What architecture and relationships should an agent know? | +| Legacy migration | Where are the structural units to transform? | +| Historical analysis | How does measured exposure change as the repository evolves? | ![GitGalaxy architecture pipeline](docs/wiki/assets/sankey_v4.3.1.png) ------------------------------------------------------------------------ -# The structural-extraction thesis +## Accuracy, measured -GitGalaxy deliberately does **not** begin by constructing a complete AST -for every language. +Two standing measurement programs back the claims above. The full narrative — +methodology, verdicts, limits, and what comes next — lives in +[the validation program](docs/validation.md); this is the summary. -It uses approximately 97 structural-signal categories to identify things -such as: +### Structural validation: GitGalaxy vs Tree-sitter vs Ctags -- function and method boundaries -- classes and declarations -- arguments -- branches and control flow -- state mutation -- I/O -- APIs and routes -- imports and dependencies -- unsafe operations -- reflection and dynamic execution -- concurrency -- closures -- globals -- entropy and physical-file anomalies - -This creates a specific, testable hypothesis: - -> **For repository-scale intelligence, targeted structural extraction -> can recover the entities required for useful code intelligence without -> requiring a complete language parser for every file.** - -That hypothesis is being tested empirically. - ------------------------------------------------------------------------- - -# Structural validation: GitGalaxy vs Tree-sitter vs Ctags - -This is currently one of the most important validation programs in the -project. - -GitGalaxy is being evaluated against **Tree-sitter and Universal Ctags** -on the same [Language Crucible](https://github.com/squid-protocol/language-crucible) corpus. - -The first structural targets are: - -- functions -- classes -- arguments - -The benchmark is deliberately **not** treated as a three-tool popularity -contest. - -When tools disagree: - -1. the disagreement is recorded; -2. the source is inspected; -3. each tool's behavior is investigated; -4. GitGalaxy is fixed when GitGalaxy is wrong; -5. comparator/adaptor code is fixed when the comparator is wrong; -6. genuine tool limitations are documented; -7. the result is re-measured. - -**24 of 45 languages get all three tools compared, 13 more get two, and -5** (`abap`, `agc_assembly`, `dockerfile`, `jcl`, `livecode`) **use -committed manual verification** — in whole, or for the `args` metric -where no tool emits a signature — instead of cross-tool agreement. Of -the 201 discrepancy shapes logged, **200 are validated (99.5%)** — read -against real source, investigated, and recorded with a verdict, not just -counted. - -**Current state (2026-08-30), on the pinned corpus:** - -- **Functions — precision:** 100.0% validated across all 31 - tree-sitter-comparable languages. Once every three-way disagreement is - read and verdicted, every function GitGalaxy reports is a real - function. -- **Functions — recall:** 100.0% for 30 of 31. Shell measures 99.8% — a - single nested function definition (inside an `if` guard) that - GitGalaxy's top-level-only shell extractor doesn't reach by design. - Every other apparent miss is a validated comparison-tool artifact - (both tools independently hallucinating the same macro, per-clause - tagging of one Haskell function, and so on). -- **Classes:** GitGalaxy is never the tool found wrong in any class - disagreement — 100.0% validated recall and precision wherever a class - ground truth exists. -- **Arguments:** GitGalaxy is never debited in any *validated* argument - disagreement either; each one resolves to tree-sitter or ctags - miscounting, or a language with no formal parameter list. One small - Objective-C shape is still unverified. - -This is a narrow benchmark: three structural targets, one fixed corpus, -and several languages whose entity counts are small enough that a -percentage means little. It is not "GitGalaxy parses as accurately as an -AST" in general — it is "for the entities GitGalaxy's graph needs, -targeted extraction recovers them as completely as established parsers, -here." Real code outside the corpus will surface shapes it doesn't cover; -the mandatory occurrence-level -[recall audit](docs/self_scan/tri_comparison_README.md#the-recall-audit-is-gitgalaxy-missing-anything) -is the standing process for the next one. - -See -[the tri-comparison methodology doc](docs/self_scan/tri_comparison_README.md) -for how matching, the ledger lifecycle, the recall audit, and CI -enforcement work. +GitGalaxy is benchmarked against **Tree-sitter and Universal Ctags** on the +pinned [Language Crucible](https://github.com/squid-protocol/language-crucible) +corpus — 24 of 45 languages get all three tools, 13 more get two, and every +disagreement is investigated against real source and recorded with a verdict +(200 of 201 logged discrepancy shapes validated). On that corpus, GitGalaxy's +validated function precision is 100% across all 31 tree-sitter-comparable +languages, and it is never the tool found wrong in a validated class or +argument disagreement. The limit: three structural targets, one fixed corpus — +not "parses as accurately as an AST" in general. ![Tri-comparison](docs/self_scan/tri_comparison_chart.svg) -See: - -- [`tests/tools/tri_comparison_chart.py`](tests/tools/tri_comparison_chart.py) -- [`docs/self_scan/tri_comparison_ledger.json`](docs/self_scan/tri_comparison_ledger.json) - — the full, per-shape validated record -- [`docs/self_scan/tri_comparison_points_of_interest.md`](docs/self_scan/tri_comparison_points_of_interest.md) - — the same ledger, rendered and ranked by signal strength -- [`docs/self_scan/how_to_investigate_a_discrepancy.md`](docs/self_scan/how_to_investigate_a_discrepancy.md) -- [`docs/self_scan/manual_verification.json`](docs/self_scan/manual_verification.json) - -### What the benchmark is actually asking +### Cross-language consistency: the Keyword Rosetta control corpus -Not: +The newer program asks the opposite question: **does GitGalaxy measure +identical intent identically in every language?** The +[keyword-rosetta](https://github.com/squid-protocol/keyword-rosetta) corpus +plants the same 12-probe program in all 46 supported languages with exact known +signal counts — so any divergence is measured language bias, by construction. +Current answer: not yet. On average 75% of languages land within ±25% of the +cross-language median per metric, but only 3 of 33 metrics pass the strict +cross-language gate, and the weakest (`risk_cognitive_load`, at 15% of +languages in band) are named in the chart rather than hidden. Each deviation is +recorded in a validated ledger and the defect classes found this way are +[filed as GitGalaxy issues](https://github.com/squid-protocol/keyword-rosetta/blob/main/docs/findings_by_language.md). -> "Is GitGalaxy a better parser than Tree-sitter?" +![Cross-language variance chart](https://raw.githubusercontent.com/squid-protocol/keyword-rosetta/main/docs/bias_variance_chart.svg) -But: - -> **"For the structural entities GitGalaxy needs to build its repository -> graph, how accurately can targeted structural extraction recover them -> compared with established parsing and indexing systems?"** - -That is the narrower claim the experiment can support. - -### Languages without suitable comparator coverage - -Some languages do not currently have a suitable independent -Tree-sitter/Ctags comparison path. - -Those are kept in a separate evidentiary category and use committed -manual verification rather than pretending cross-tool agreement exists. - -This currently includes languages such as: - -- ABAP -- Dockerfile -- JCL -- LiveCode -- YAML - -Where practical, the next step is to add independent lexical, -grammar-based, or domain-specific comparators. Where no credible -independent comparator exists, human-verified ground truth remains the -appropriate category. +A control corpus proves measurement inequality on identical intent; it says +nothing about accuracy on real code (the tri-comparison's job above) — and a +constant per-file bias still preserves ranking *within* a language. The +[incidence report](https://github.com/squid-protocol/keyword-rosetta/blob/main/docs/incidence_report.md) +sizes each confirmed shape against real licensed code. ------------------------------------------------------------------------ -# Validation is a ladder - -GitGalaxy's evidence is being organized around progressively stronger -questions. - -### 1. Structural validity - -**Does GitGalaxy correctly identify code structures?** - -Tree-sitter + Ctags + independently investigated disagreements. See -["Structural validation" above](#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags). - -### 2. Regression validity - -**Does the implementation remain stable on real code?** - -[Golden-master testing](tests/tools/update_golden_master.py) against -[Language Crucible](https://github.com/squid-protocol/language-crucible). - -### 3. Scale validity - -**Does it work on real repositories?** +## Real-world scale -Unedited raw scan output from -[hundreds of repositories](https://github.com/squid-protocol/gitgalaxy-raw-output). +Example: **Kubernetes** — ~1.39M lines across Go, YAML, JSON, Shell and Proto. +End-to-end scan: **50.83 seconds**. Scan time follows a fitted two-regime model +(flat ~0.11s below ~4.3K LOC, then `time(s) ≈ 3.36e-05 × LOC^0.969`, R²=0.88 +across 599 repos) — so a handful of 20M+ LOC outliers still take minutes, and a +fast scan proves throughput, not insight quality; the accuracy question is +handled separately below. -### 4. Model validity - -**Do structural signatures correspond to the exposure categories they -are intended to represent?** - -Statistical analysis against independently observable outcomes---not -merely against GitGalaxy's own equations. - -### 5. Temporal validity - -**Does exposure behave sensibly as software changes?** - -Git-history analysis comparing repository states before and after real -changes. - -### 6. External validity - -**Do exposure changes correspond to independently documented security or -maintenance outcomes?** - -Future work: security fixes, regressions, advisories, defects and other -external event datasets. +![GitGalaxy scan +speed](https://raw.githubusercontent.com/squid-protocol/gitgalaxy-raw-output/main/speed_charts/latest/loc_vs_time.png) -This distinction matters: a score can be internally consistent without -necessarily being externally meaningful. +See the [raw output +repository](https://github.com/squid-protocol/gitgalaxy-raw-output) for +unedited artifacts. ------------------------------------------------------------------------ # Risk exposure: what GitGalaxy claims -GitGalaxy produces **risk-exposure measurements**, not vulnerability -verdicts. +GitGalaxy produces **risk-exposure measurements**, not vulnerability verdicts. A high exposure means: -> **This location deserves attention relative to the rest of the -> repository.** +> **This location deserves attention relative to the rest of the repository.** It does not mean: @@ -361,108 +217,62 @@ The underlying signatures cover patterns involving areas such as: - entropy - other structural/security characteristics -The important research question is whether these signatures are -**empirically associated with meaningful classes of software risk**, -rather than merely correlated with a score that GitGalaxy itself -mathematically constructed. +The important research question is whether these signatures are **empirically +associated with meaningful classes of software risk**, rather than merely +correlated with a score that GitGalaxy itself mathematically constructed. That distinction drives the next phase. ------------------------------------------------------------------------ -# The next validation: risk over Git history - -Once structural validation is sufficiently mature, GitGalaxy can test -its exposure model longitudinally. - -``` text -Git history - | - v -security-relevant event - | - +-------------------+ - | | - v v -parent state changed state - | | - v v -GitGalaxy scan GitGalaxy scan - | | - +---------+---------+ - | - v - exposure delta - | - v - independent event class -``` - -The central experiment is: - -> **Do commits independently identified as security fixes typically -> reduce the corresponding GitGalaxy exposure?** - -Negative controls are equally important: - -> Do ordinary development commits show the same behavior? - -Eventually: - -> Do security regressions increase exposure? - -The planned harness will preserve commit SHA, parent state, changed -files/functions, exposure before/after, exposure deltas, structural -changes and event classification. - -That tests: - -**structure → exposure → real software evolution** - -rather than merely testing the internal mathematics of the exposure -model. - ------------------------------------------------------------------------- - # Evidence, not just claims ### Language Crucible -A pinned corpus of real-world source including projects such as Godot, -Roslyn, curl, Kubernetes and Apollo 11 flight software. +A pinned corpus of real-world source including projects such as Godot, Roslyn, +curl, Kubernetes and Apollo 11 flight software. [Language Crucible](https://github.com/squid-protocol/language-crucible) ### Golden-master regression -Real source is rescanned and compared against checked-in expected output -so parser changes have an observable diff. Regenerated with +Real source is rescanned and compared against checked-in expected output so +parser changes have an observable diff. Regenerated with [`tests/tools/update_golden_master.py`](tests/tools/update_golden_master.py), never hand-edited. ### Tri-comparison -The same corpus is analyzed against GitGalaxy, Tree-sitter and Ctags -where coverage exists — 24 of 45 languages get all three tools, 200 of -201 logged discrepancies validated. On the pinned corpus, GitGalaxy is -never the tool found wrong in a validated function-precision, class, or -argument disagreement, and matches tree-sitter's function recall -everywhere except one nested shell definition it skips by design. See -[the methodology](docs/self_scan/tri_comparison_README.md) and the -["structural validation" section above](#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags) -for the full picture and its limits. +The same corpus is analyzed against GitGalaxy, Tree-sitter and Ctags where +coverage exists — 24 of 45 languages get all three tools, 200 of 201 logged +discrepancies validated. On the pinned corpus, GitGalaxy is never the tool +found wrong in a validated function-precision, class, or argument disagreement, +and matches tree-sitter's function recall everywhere except one nested shell +definition it skips by design. See +[the methodology](docs/self_scan/tri_comparison_README.md) and +[the validation program](docs/validation.md) for the full picture and its +limits. + +### Keyword Rosetta control corpus + +The same 12-probe program planted in all 46 supported languages with exact +known signal counts, measuring cross-language consistency of every metric — +with a validated deviation ledger and the resulting engine defects +[filed as issues](https://github.com/squid-protocol/keyword-rosetta/blob/main/docs/findings_by_language.md). + +[Keyword Rosetta](https://github.com/squid-protocol/keyword-rosetta) ### Raw repository output -Unedited GitGalaxy output is retained for hundreds of independently -selected repositories. +Unedited GitGalaxy output is retained for hundreds of independently selected +repositories. [Raw Output](https://github.com/squid-protocol/gitgalaxy-raw-output) ### Regression suite -**7,043 tests** in the default suite (`python -m pytest tests/`), of -which **6,165** are per-signature tests across all 45 structurally-signatured +**7,043 tests** in the default suite (`python -m pytest tests/`), of which +**6,165** are per-signature tests across all 45 structurally-signatured languages — positive matches, explicit exclusions, and adversarial/ReDoS inputs. See [`tests/README.md`](tests/README.md) for the breakdown, and [`docs/why_gitgalaxy_beats_ast_here.md`](docs/why_gitgalaxy_beats_ast_here.md) @@ -470,8 +280,9 @@ for specific, evidenced cases where this extraction beats an AST read. ### Historical validation -The next research layer will test whether exposure measurements -correspond to real security and maintenance events over Git history. +The next research layer will test whether exposure measurements correspond to +real security and maintenance events over Git history — see +[the validation program](docs/validation.md#the-next-validation-risk-over-git-history). ------------------------------------------------------------------------ @@ -498,81 +309,25 @@ correspond to real security and maintenance events over Git history. - a complete language parser - a guarantee that a high exposure is a vulnerability - ----------------------------------------------------------------------- - Tool Primary question - ----------------------------------- ----------------------------------- - **GitGalaxy** What does this entire repository - look like, structurally, and where - should attention go first? - - Tree-sitter What syntactic structure does this - source contain? - - Ctags Where are the navigable code - entities? - - Semgrep Does this code match a specified - pattern? - - CodeQL What data/control relationships can - deeper analysis establish? - - SCA/CVE tools Is this dependency/version - associated with a known advisory? - ----------------------------------------------------------------------- - ------------------------------------------------------------------------- - -# Real-world scale - -GitGalaxy is intended for repositories too heterogeneous or broken for a -traditional single-language build-first workflow. - -Example: **Kubernetes** - -\~1.39M lines across Go, YAML, JSON, Shell and Proto. - -End-to-end scan: **50.83 seconds**. - -![GitGalaxy scan -speed](https://raw.githubusercontent.com/squid-protocol/gitgalaxy-raw-output/main/speed_charts/latest/loc_vs_time.png) - -See the [raw output -repository](https://github.com/squid-protocol/gitgalaxy-raw-output) for -unedited artifacts. - ------------------------------------------------------------------------- - -# Outputs - - Output Purpose - ---------------------------- ---------------------------------------- - **SARIF** CI/security dashboard integration - **CycloneDX SBOM** Dependency inventory/compliance - **SQLite** Queryable repository knowledge graph - **LLM architecture brief** Compact machine/agent-oriented context - **JSON audit data** Forensic/automation workflows - **3D visualization data** Interactive repository topology - -These are different views of the same deterministic scan, rather than -independent analysis engines. +| Tool | Primary question | +|---|---| +| **GitGalaxy** | What does this entire repository look like, structurally, and where should attention go first? | +| Tree-sitter | What syntactic structure does this source contain? | +| Ctags | Where are the navigable code entities? | +| Semgrep | Does this code match a specified pattern? | +| CodeQL | What data/control relationships can deeper analysis establish? | +| SCA/CVE tools | Is this dependency/version associated with a known advisory? | ------------------------------------------------------------------------ # Git history and architecture -GitGalaxy already incorporates Git history into signals such as: - -- churn -- contributor concentration -- bus-factor exposure -- refactoring hotspots -- file ownership -- temporal activity - -The research direction is to extend this from **history as a contextual -signal** to **history as an external validation source for the exposure -model**. +GitGalaxy already incorporates Git history into signals such as churn, +contributor concentration, bus-factor exposure, refactoring hotspots, file +ownership and temporal activity. The research direction is to extend this from +**history as a contextual signal** to **history as an external validation +source for the exposure model** — the experiment design is in +[the validation program](docs/validation.md#the-next-validation-risk-over-git-history). ------------------------------------------------------------------------ @@ -599,82 +354,27 @@ current commands and configuration. ### CI/CD -Templates are provided for: - -- GitHub Actions -- GitLab CI -- Bitbucket Pipelines -- Azure Pipelines -- generic shell-invocable CI environments - -See [`templates/`](templates/) and the [CI integration +Templates are provided for GitHub Actions, GitLab CI, Bitbucket Pipelines, +Azure Pipelines, and generic shell-invocable CI environments. See +[`templates/`](templates/) and the [CI integration guide](github-action-readme.md). ------------------------------------------------------------------------ # Explore the evidence - --------------------------------------------------------------------------------------------------------------------------------- - Resource What it contains - --------------------------------------------------------------------------------------------- ----------------------------------- - [Documentation](https://squid-protocol.github.io/gitgalaxy/) Architecture, claims and - methodology - - [Language Crucible](https://github.com/squid-protocol/language-crucible) Cross-language benchmark and golden - corpus - - [Raw Output](https://github.com/squid-protocol/gitgalaxy-raw-output) Unedited scans of real repositories - - [`tests/README.md`](tests/README.md) Regression and golden-master - methodology - - [`tri_comparison_ledger.json`](docs/self_scan/tri_comparison_ledger.json) Disagreement-by-disagreement - validation record - - [`manual_verification.json`](docs/self_scan/manual_verification.json) Reviewed cases where comparator - coverage is unavailable - - [`how_to_investigate_a_discrepancy.md`](docs/self_scan/how_to_investigate_a_discrepancy.md) Comparator-disagreement methodology - - [Visualizer](https://gitgalaxy.io/) Local browser-based repository - visualization - --------------------------------------------------------------------------------------------------------------------------------- - ------------------------------------------------------------------------- - -# Current research direction - -GitGalaxy is moving through a sequence of increasingly difficult -questions: - -> **Can we scan heterogeneous source without compiling it?** - -↓ - -> **Can we reliably recover the structural entities needed to understand -> it?** - -↓ - -> **Do those structural measurements correspond to meaningful risk -> exposure?** - -↓ - -> **Does measured exposure behave correctly as real software evolves?** - -The Tree-sitter/Ctags validation is currently about halfway complete. -The immediate priority is to finish that audit before turning -preliminary measurements into stronger claims. - -The next major experiment is: - -**Git history → independently identified change/fix events → GitGalaxy -before/after scans → exposure deltas → statistical analysis.** - -That is where GitGalaxy can begin testing not only whether it *sees* -structure, but whether its structural model **tracks meaningful changes -in real software**. +| Resource | What it contains | +|---|---| +| [Documentation](https://squid-protocol.github.io/gitgalaxy/) | Architecture, claims and methodology | +| [The validation program](docs/validation.md) | The full proof narrative: thesis, benchmarks, validity ladder, next experiments | +| [Language Crucible](https://github.com/squid-protocol/language-crucible) | Cross-language benchmark and golden corpus | +| [Keyword Rosetta](https://github.com/squid-protocol/keyword-rosetta) | 46-language planted control corpus and bias reports | +| [Raw Output](https://github.com/squid-protocol/gitgalaxy-raw-output) | Unedited scans of real repositories | +| [`tests/README.md`](tests/README.md) | Regression and golden-master methodology | +| [`tri_comparison_ledger.json`](docs/self_scan/tri_comparison_ledger.json) | Disagreement-by-disagreement validation record | +| [`manual_verification.json`](docs/self_scan/manual_verification.json) | Reviewed cases where comparator coverage is unavailable | +| [`how_to_investigate_a_discrepancy.md`](docs/self_scan/how_to_investigate_a_discrepancy.md) | Comparator-disagreement methodology | +| [Visualizer](https://gitgalaxy.io/) | Local browser-based repository visualization | ------------------------------------------------------------------------ @@ -682,7 +382,6 @@ in real software**. Copyright (c) 2026 Joe Esquibel -GitGalaxy is distributed under the **PolyForm Noncommercial License -1.0.0**. +GitGalaxy is distributed under the **PolyForm Noncommercial License 1.0.0**. -See the repository license for full terms. \ No newline at end of file +See the repository license for full terms. diff --git a/docs/how_to_maintain_the_readme.md b/docs/how_to_maintain_the_readme.md index 80283d42c..e3d6c0e9b 100644 --- a/docs/how_to_maintain_the_readme.md +++ b/docs/how_to_maintain_the_readme.md @@ -44,71 +44,74 @@ Use this doc every time you touch README.md, not just on a dedicated cleanup pas adoption numbers before enterprise/licensing. A skeptical engineer reads in that order — the README should match. -## Section order (current target state, as of the 2026-08-22 rewrite) +## Section order (current target state, as of the 2026-08-31 product-first restructure) -1. Title + one-line tagline ("Repository-scale structural intelligence without - compilation.") + the Docs/Visualizer/Language Crucible/Raw Output link line. **No badges - currently** — License, PyPI version, Python version, and Dependencies badges all existed - before this rewrite and were dropped along with it; restoring any of them (each still has - a real link behind it, so each would pass rule 1) is an open gap, not a considered removal. -2. Proof strip — currently `1 scan · 97 structural signals · 50+ languages · no compilation · - 19 risk-exposure categories · 6 outputs`. This dropped the dependency-count claim - entirely; cross-check against `docs/readme_evidence_roadmap.md`'s PyYAML row before - deciding whether to add "0 dependencies" back — that claim was shipped and true as of - #1104, so its absence here is a regression, not a status change. -3. **The short version** — plain-terms description, the "same graph feeds N consumers" - bullet list, and a blockquoted central-thesis line. No BLAST/genomics analogy anywhere in - the current draft (the whole brand-metaphor vocabulary rule-2 exists for is currently - unused) — reintroducing it is fine as long as rule 2 still holds once it's back. -4. **The problem** — ASCII-diagrammed statement of the trade GitGalaxy makes. This replaced - "What Pain Point Does This Solve?"'s bulleted, externally-motivated framing (see that - section's git history for why it existed — a first-time visitor couldn't tell within a - minute whether this competes with CodeQL/Semgrep/SonarQube). The explicit "this is not X" - disambiguation that used to live here now only exists later, folded into "What GitGalaxy is - — and isn't" (item 13) — confirm that's still reachable early enough for a skeptical - first-time reader before treating this as settled. -5. **One graph, many consumers** — consumer/question table + the architecture pipeline - diagram (`docs/wiki/assets/sankey_v4.3.1.png`). -6. **The structural-extraction thesis** — the ~97 structural-signal-category list + the - testable-hypothesis blockquote. -7. **Structural validation: GitGalaxy vs Tree-sitter vs Ctags** — the tri-comparison - methodology, current coverage numbers (languages with 3/2/0 comparator tools, ledger - validation count), and the "what the benchmark is/isn't asking" + "languages without - comparator coverage" subsections. Anchor slug - `#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags` is now a link target from later - sections — keep it stable. -8. **Validation is a ladder** — new in this rewrite, no prior equivalent. Six escalating - validity levels (structural → regression → scale → model → temporal → external). This is a - genuinely stronger evidence framework than what existed before the rewrite; keep it even if - other sections get walked back toward the pre-rewrite structure. -9. **Risk exposure: what GitGalaxy claims** (tone-bar section #1 — do not rewrite its prose, - only relocate it if needed; see the intro note above on why the name changed). -10. **The next validation: risk over Git history** — forward-looking research section - (git-history-as-validation-source experiment design). New in this rewrite. -11. **Evidence, not just claims** (tone-bar section #2). Every bullet under it should name a - real, current number — "Regression suite" in particular must cite the actual collected - test count (`python -m pytest tests/ --collect-only -q`), not a vague "thousands"; numbers - drift as the suite grows, so re-verify rather than copying forward. -12. **What GitGalaxy is — and isn't** — is/is-not bullet lists + the tool-comparison table. - This is the section that now carries what "How This Compares, Architecturally" used to - carry (comparator table vs. Tree-sitter/Ctags/Semgrep/CodeQL/SCA tools); no longer at a - stable, separately-named anchor, so re-check any old link pointing at - `#how-this-compares-architecturally` — it no longer resolves here. -13. **Real-world scale** — the Kubernetes benchmark bullet. Needs a rule-4 limitation clause - (currently missing — see rule 4 above). -14. **Outputs** — SARIF/SBOM/SQLite/LLM-brief/JSON/3D-viz table. -15. **Git history and architecture** — churn/bus-factor/hotspot signals, framed as feeding the - Git-history validation direction (item 10). -16. **Privacy and deployment** — one section, not two. If a future edit reintroduces a second - privacy section (it has happened once already, pre-rewrite — "Data Privacy & On-Premise - Deployment" and "Zero-Trust Data Security" said the same three things in two places), - merge them back. -17. **Installation** (+ CI/CD subsection). -18. **Explore the evidence** — link table to Docs/Language Crucible/Raw Output/tests/README.md - /ledger/manual-verification/how-to-investigate/Visualizer. This is currently the only place - the Visualizer gets a dedicated mention — it no longer has its own section. -19. **Current research direction** — closing sequence-of-questions narrative + what's next. -20. **License.** +The 2026-08-31 restructure split the README in two: `README.md` leads with the product +(what a scan gives you, shown with real report excerpts), and the long proof narrative — +the structural-extraction thesis, the full tri-comparison section with its subsections, +the validity ladder, the risk-over-Git-history experiment, and the closing research +direction — moved intact to `docs/validation.md`. Edit proof prose there, not by +reintroducing it into README.md. The README keeps a condensed "Accuracy, measured" +summary with both charts and per-claim limitation clauses. + +1. Title + one-line tagline + the Docs/Visualizer/Language Crucible/Keyword + Rosetta/Raw Output link line. **No badges currently** — License, PyPI version, Python + version, and Dependencies badges all existed before the 2026-08-22 rewrite and were + dropped with it; restoring any of them (each still has a real link behind it, so each + would pass rule 1) is an open gap, not a considered removal. +2. Proof strip — `1 scan · 97 structural signals · 50+ languages · no compilation · + 19 risk-exposure categories · 6 outputs`. Still missing the "0 dependencies" claim + (shipped and true as of #1104) — its absence remains a regression, not a status change. +3. **The short version** — plain-terms description folding in the old "The problem" + polyglot code block, the consumer list as prose, the blockquoted central-thesis line, + and a forward link to `docs/validation.md`. +4. **What a scan gives you** — the product section (new 2026-08-31): install/run command, + the six-outputs table (absorbing the old standalone "Outputs" section), and "The + architecture brief" showcase with two dated, reproducible scan excerpts (curl, + cics-genapp), each carrying its own rule-4 limitation clause. Excerpt numbers are + point-in-time by design — they name their scan date and the engine that produced them; + refresh the excerpts (rerun `galaxyscope --llm-only` on the same public repos) rather + than letting undated numbers drift. Ends with the "One graph, many consumers" + consumer/question table + the sankey pipeline diagram. +5. **Accuracy, measured** — condensed two-part proof summary. Subsection headings + "Structural validation: GitGalaxy vs Tree-sitter vs Ctags" (keeps the old + `#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags` anchor resolving — do not + rename it) and "Cross-language consistency: the Keyword Rosetta control corpus". Chart + order is deliberate: tri-comparison SVG (accuracy on real code) before the + keyword-rosetta bias-variance SVG (consistency on planted code, embedded from that + repo's raw main-branch URL so it self-updates). Each subsection states one limitation + and links to `docs/validation.md` for the full treatment. Proof sits before the + speed benchmark on purpose (rule 5) — don't float "Real-world scale" back above it. +6. **Real-world scale** — Kubernetes benchmark + the fitted two-regime speed model as its + rule-4 limitation clause (the clause rule 4 previously flagged as missing — now + present; keep it) + the self-updating loc_vs_time chart. +7. **Risk exposure: what GitGalaxy claims** (tone-bar section #1 — do not rewrite its + prose, only relocate it if needed; see the intro note above on why the name changed). +8. **Evidence, not just claims** (tone-bar section #2). Every bullet under it should name + a real, current number — "Regression suite" in particular must cite the actual + collected test count (`python -m pytest tests/ --collect-only -q`), not a vague + "thousands"; numbers drift as the suite grows, so re-verify rather than copying + forward. Now includes a "Keyword Rosetta control corpus" bullet alongside the + tri-comparison one. +9. **What GitGalaxy is — and isn't** — is/is-not bullet lists + the tool-comparison table + (carries what "How This Compares, Architecturally" used to; the old + `#how-this-compares-architecturally` anchor still doesn't resolve anywhere). +10. **Git history and architecture** — churn/bus-factor/hotspot signals as a short + paragraph, linking to the validation doc's Git-history experiment. +11. **Privacy and deployment** — one section, not two. If a future edit reintroduces a + second privacy section (it has happened once already, pre-rewrite), merge them back. +12. **Installation** (+ CI/CD subsection). +13. **Explore the evidence** — link table; now includes `docs/validation.md` and Keyword + Rosetta rows. Still the only place the Visualizer gets a dedicated mention. +14. **License.** + +**Sections moved to `docs/validation.md` on 2026-08-31 (not cut):** "The +structural-extraction thesis," the full "Structural validation" section with its "what +the benchmark is asking" and "languages without comparator coverage" subsections, +"Validation is a ladder" (now seven rungs — a "Measurement consistency" rung for the +control corpus was added between structural and regression validity), "The next +validation: risk over Git history," and "Current research direction." The five rules +apply to that file exactly as they do to README.md. **Sections present before the 2026-08-22 rewrite and currently absent:** badges (item 1 above), "Weakness Classes, Not Just CVEs," "Real-World Adoption" (stars/downloads, deliberately diff --git a/docs/readme_evidence_roadmap.md b/docs/readme_evidence_roadmap.md index 5bda2de2f..a8b9fd68b 100644 --- a/docs/readme_evidence_roadmap.md +++ b/docs/readme_evidence_roadmap.md @@ -47,4 +47,5 @@ before its evidence exists (add a row here instead of writing it into README.md | "What Pain Point Does This Solve?" section, addressing external feedback that a first-time visitor couldn't tell within a minute whether GitGalaxy competes with CodeQL/Semgrep/SonarQube or does something else — explicit "this is not X" disambiguation linking to the comparison table, plus a real cited example (Kubernetes, 1.39M LOC, 50.83s scan) instead of an abstract claim | 2026-08-06 | README.md, right after the proof strip; also retitled "Whole-Repository Intelligence with a Security Layer" to lead with the "one graph, three consumers" framing instead of security-first branding | | Restore the "0 dependencies" claim (stat line + badge) | 2026-08-06, [#1104](https://github.com/squid-protocol/gitgalaxy/issues/1104) — `PyYAML` moved from `pyproject.toml`'s `dependencies` into an optional `gitgalaxy[yaml]` extra; the 3 lazy-import call sites now raise/warn with an actionable install message instead of a silent no-op or bare `ImportError` | README.md stat line + badge | | Fitted two-regime scan-speed model, replacing the "a handful of 20M+ LOC outliers take a few minutes" hand-wave: flat ~0.11s below ~4,258 LOC (fixed overhead), `time(s) ≈ 3.36e-05 × LOC^0.969` above it (R²=0.88, 599 repos) — both the knee and the exponent are derived from `gitgalaxy-raw-output`'s own batch data (`tools/generate_speed_telemetry.py`'s `compute_rate_model()`), not eyeballed. LOC-vs-time chart embedded from that repo's self-updating `speed_charts/latest/loc_vs_time.png`, so it never goes stale here | 2026-08-06 | README.md "What Pain Point Does This Solve?" (speed bullet) and "Proof, Not Just Claims" (item 3, plus the embedded chart) | +| Cross-language measurement-consistency claim, backed by a planted control corpus: [keyword-rosetta](https://github.com/squid-protocol/keyword-rosetta) plants the same 12-probe program in all 46 supported languages with exact known signal counts, so every cross-language divergence is measured bias, not opinion. Shipped limitations-first: the README states that only 3 of 33 metrics currently pass the strict cross-language gate (avg. 75% of languages within ±25% of median), names the weakest metrics, and links the validated deviation ledger + the findings-by-language index of filed GitGalaxy issues. Chart embedded from that repo's raw main-branch URL (same self-updating pattern as the speed chart) | 2026-08-31 | README.md "Accuracy, measured" (Keyword Rosetta subsection) + "Evidence, not just claims" bullet; full treatment in `docs/validation.md` | | Measured structural-extraction accuracy cross-checked against Tree-sitter AND ctags (not just one, and with no reader treated as privileged ground truth): 24 of 45 languages get all three tools compared, 16 more get two, and the remaining 5 gg-only languages (`abap`, `dockerfile`, `jcl`, `livecode`, `yaml`) get a hand-reviewed manual-verification record instead of cross-tool agreement. Every disagreement is logged to a reviewed ledger (`docs/self_scan/tri_comparison_ledger.json`) and investigated by hand — not asserted. Backed by `tests/tools/tri_comparison_chart.py`, re-measured and re-charted automatically on every push touching the parsing engine — the same "always current, embedded from the repo" pattern as the speed chart above, not a one-time number. Superseded the earlier bi-comparison-only (Tree-sitter alone) version of this item; `python.md` under `docs/language_status/` already has its own §10 tri-comparison section built the same way, `javascript.md` still only has the earlier bi-comparison §9 | 2026-08-12, updated 2026-08-21 when the bi-comparison chart was retired in favor of this tri-comparison one | README.md "Proof, Not Just Claims" (item 4, embedding `docs/self_scan/tri_comparison_chart.svg`), and "One Graph, Not Five Separate Tools" (the "why not Tree-sitter" paragraph, which links to this measurement instead of asserting the tradeoff unbacked) | diff --git a/docs/validation.md b/docs/validation.md new file mode 100644 index 000000000..fedeea2c2 --- /dev/null +++ b/docs/validation.md @@ -0,0 +1,334 @@ +# The GitGalaxy Validation Program + +This is the full narrative of how GitGalaxy's claims get tested — the thesis, the +three-tool structural benchmark, the cross-language control corpus, the validity +ladder, and the next experiment. [`README.md`](../README.md) carries the summary; +this document carries the whole argument, with every number linked to the artifact +behind it. + +------------------------------------------------------------------------ + +## The structural-extraction thesis + +GitGalaxy deliberately does **not** begin by constructing a complete AST for every +language. + +It uses approximately 97 structural-signal categories to identify things such as: + +- function and method boundaries +- classes and declarations +- arguments +- branches and control flow +- state mutation +- I/O +- APIs and routes +- imports and dependencies +- unsafe operations +- reflection and dynamic execution +- concurrency +- closures +- globals +- entropy and physical-file anomalies + +This creates a specific, testable hypothesis: + +> **For repository-scale intelligence, targeted structural extraction can recover +> the entities required for useful code intelligence without requiring a complete +> language parser for every file.** + +That hypothesis is being tested empirically, on three independent axes: accuracy +against established parsers (below), measurement consistency across languages +(the control corpus), and — next — correspondence with real software evolution +(risk over Git history). + +------------------------------------------------------------------------ + +## Structural validation: GitGalaxy vs Tree-sitter vs Ctags + +This is currently one of the most important validation programs in the project. + +GitGalaxy is being evaluated against **Tree-sitter and Universal Ctags** on the +same [Language Crucible](https://github.com/squid-protocol/language-crucible) +corpus. + +The first structural targets are: + +- functions +- classes +- arguments + +The benchmark is deliberately **not** treated as a three-tool popularity contest. + +When tools disagree: + +1. the disagreement is recorded; +2. the source is inspected; +3. each tool's behavior is investigated; +4. GitGalaxy is fixed when GitGalaxy is wrong; +5. comparator/adaptor code is fixed when the comparator is wrong; +6. genuine tool limitations are documented; +7. the result is re-measured. + +**24 of 45 languages get all three tools compared, 13 more get two, and 5** +(`abap`, `agc_assembly`, `dockerfile`, `jcl`, `livecode`) **use committed manual +verification** — in whole, or for the `args` metric where no tool emits a +signature — instead of cross-tool agreement. Of the 201 discrepancy shapes +logged, **200 are validated (99.5%)** — read against real source, investigated, +and recorded with a verdict, not just counted. + +**Current state (2026-08-30), on the pinned corpus:** + +- **Functions — precision:** 100.0% validated across all 31 + tree-sitter-comparable languages. Once every three-way disagreement is read and + verdicted, every function GitGalaxy reports is a real function. +- **Functions — recall:** 100.0% for 30 of 31. Shell measures 99.8% — a single + nested function definition (inside an `if` guard) that GitGalaxy's + top-level-only shell extractor doesn't reach by design. Every other apparent + miss is a validated comparison-tool artifact (both tools independently + hallucinating the same macro, per-clause tagging of one Haskell function, and + so on). +- **Classes:** GitGalaxy is never the tool found wrong in any class disagreement + — 100.0% validated recall and precision wherever a class ground truth exists. +- **Arguments:** GitGalaxy is never debited in any *validated* argument + disagreement either; each one resolves to tree-sitter or ctags miscounting, or + a language with no formal parameter list. One small Objective-C shape is still + unverified. + +This is a narrow benchmark: three structural targets, one fixed corpus, and +several languages whose entity counts are small enough that a percentage means +little. It is not "GitGalaxy parses as accurately as an AST" in general — it is +"for the entities GitGalaxy's graph needs, targeted extraction recovers them as +completely as established parsers, here." Real code outside the corpus will +surface shapes it doesn't cover; the mandatory occurrence-level +[recall audit](self_scan/tri_comparison_README.md#the-recall-audit-is-gitgalaxy-missing-anything) +is the standing process for the next one. + +![Tri-comparison](self_scan/tri_comparison_chart.svg) + +See: + +- [the tri-comparison methodology doc](self_scan/tri_comparison_README.md) — + how matching, the ledger lifecycle, the recall audit, and CI enforcement work +- [`tests/tools/tri_comparison_chart.py`](../tests/tools/tri_comparison_chart.py) +- [`docs/self_scan/tri_comparison_ledger.json`](self_scan/tri_comparison_ledger.json) + — the full, per-shape validated record +- [`docs/self_scan/tri_comparison_points_of_interest.md`](self_scan/tri_comparison_points_of_interest.md) + — the same ledger, rendered and ranked by signal strength +- [`docs/self_scan/how_to_investigate_a_discrepancy.md`](self_scan/how_to_investigate_a_discrepancy.md) +- [`docs/self_scan/manual_verification.json`](self_scan/manual_verification.json) + +### What the benchmark is actually asking + +Not: + +> "Is GitGalaxy a better parser than Tree-sitter?" + +But: + +> **"For the structural entities GitGalaxy needs to build its repository graph, +> how accurately can targeted structural extraction recover them compared with +> established parsing and indexing systems?"** + +That is the narrower claim the experiment can support. + +### Languages without suitable comparator coverage + +Some languages do not currently have a suitable independent Tree-sitter/Ctags +comparison path. + +Those are kept in a separate evidentiary category and use committed manual +verification rather than pretending cross-tool agreement exists. + +This currently includes languages such as: + +- ABAP +- Dockerfile +- JCL +- LiveCode +- YAML + +Where practical, the next step is to add independent lexical, grammar-based, or +domain-specific comparators. Where no credible independent comparator exists, +human-verified ground truth remains the appropriate category. + +------------------------------------------------------------------------ + +## Cross-language consistency: the Keyword Rosetta control corpus + +The tri-comparison above asks *"does GitGalaxy find real structure?"* A separate +control corpus, [keyword-rosetta](https://github.com/squid-protocol/keyword-rosetta), +asks the orthogonal question: *"does GitGalaxy measure identical intent +identically across languages?"* + +The same 12-probe, functions-only program shell is hand-planted in **all 46 +supported languages**, with the exact count of every signal keyword known in +advance. Any column-to-column divergence on that corpus is measured language +bias — an extraction inequality or a scoring inequality — because the planted +intent is identical by construction. + +Current results +([bias report](https://github.com/squid-protocol/keyword-rosetta/blob/main/docs/bias_report.md)): +across 33 comparable metrics, on average **75% of languages land within ±25% of +the cross-language median** — but under the strict gate (no language beyond ±50% +of the median), only **3 of 33 metrics pass** cross-language validation today. +The weakest metrics are named, not hidden: `risk_cognitive_load` holds only 15% +of languages in the ±25% band, `risk_api_exposure` 43%, `state_mutation` 46%. Every known +deviation is recorded in a validated +[deviation ledger](https://github.com/squid-protocol/keyword-rosetta/blob/main/deviation_ledger.json), +and the defect classes found this way are filed as GitGalaxy issues — see the +[findings-by-language index](https://github.com/squid-protocol/keyword-rosetta/blob/main/docs/findings_by_language.md). +A companion +[incidence report](https://github.com/squid-protocol/keyword-rosetta/blob/main/docs/incidence_report.md) +sizes each confirmed shape against real licensed code, separating constant +per-file biases (e.g. PHP's open tag counting as a branch in 100% of files) from +sparse per-repo hazards. + +What this proves and doesn't: a control corpus demonstrates measurement +*inequality* on identical intent with zero ambiguity, but says nothing about +accuracy on real code (that's the tri-comparison's job) and nothing about whether +a biased metric is still *usefully ranked* within a single language (a constant +per-file offset preserves ordering inside that language). + +------------------------------------------------------------------------ + +## Validation is a ladder + +GitGalaxy's evidence is being organized around progressively stronger questions. + +### 1. Structural validity + +**Does GitGalaxy correctly identify code structures?** + +Tree-sitter + Ctags + independently investigated disagreements. See +["Structural validation" above](#structural-validation-gitgalaxy-vs-tree-sitter-vs-ctags). + +### 2. Measurement consistency + +**Does GitGalaxy measure the same intent the same way in every language?** + +The [Keyword Rosetta control corpus above](#cross-language-consistency-the-keyword-rosetta-control-corpus) +— exact planted signal counts, 46 languages, validated deviation ledger. + +### 3. Regression validity + +**Does the implementation remain stable on real code?** + +[Golden-master testing](../tests/tools/update_golden_master.py) against +[Language Crucible](https://github.com/squid-protocol/language-crucible). + +### 4. Scale validity + +**Does it work on real repositories?** + +Unedited raw scan output from +[hundreds of repositories](https://github.com/squid-protocol/gitgalaxy-raw-output). + +### 5. Model validity + +**Do structural signatures correspond to the exposure categories they are +intended to represent?** + +Statistical analysis against independently observable outcomes---not merely +against GitGalaxy's own equations. + +### 6. Temporal validity + +**Does exposure behave sensibly as software changes?** + +Git-history analysis comparing repository states before and after real changes. + +### 7. External validity + +**Do exposure changes correspond to independently documented security or +maintenance outcomes?** + +Future work: security fixes, regressions, advisories, defects and other external +event datasets. + +This distinction matters: a score can be internally consistent without +necessarily being externally meaningful. + +------------------------------------------------------------------------ + +## The next validation: risk over Git history + +Once structural validation is sufficiently mature, GitGalaxy can test its +exposure model longitudinally. + +``` text +Git history + | + v +security-relevant event + | + +-------------------+ + | | + v v +parent state changed state + | | + v v +GitGalaxy scan GitGalaxy scan + | | + +---------+---------+ + | + v + exposure delta + | + v + independent event class +``` + +The central experiment is: + +> **Do commits independently identified as security fixes typically reduce the +> corresponding GitGalaxy exposure?** + +Negative controls are equally important: + +> Do ordinary development commits show the same behavior? + +Eventually: + +> Do security regressions increase exposure? + +The planned harness will preserve commit SHA, parent state, changed +files/functions, exposure before/after, exposure deltas, structural changes and +event classification. + +That tests: + +**structure → exposure → real software evolution** + +rather than merely testing the internal mathematics of the exposure model. + +------------------------------------------------------------------------ + +## Current research direction + +GitGalaxy is moving through a sequence of increasingly difficult questions: + +> **Can we scan heterogeneous source without compiling it?** + +↓ + +> **Can we reliably recover the structural entities needed to understand it?** + +↓ + +> **Do those structural measurements correspond to meaningful risk exposure?** + +↓ + +> **Does measured exposure behave correctly as real software evolves?** + +The Tree-sitter/Ctags validation is currently about halfway complete. The +immediate priority is to finish that audit before turning preliminary +measurements into stronger claims. + +The next major experiment is: + +**Git history → independently identified change/fix events → GitGalaxy +before/after scans → exposure deltas → statistical analysis.** + +That is where GitGalaxy can begin testing not only whether it *sees* structure, +but whether its structural model **tracks meaningful changes in real software**.