Skip to content

refactor(standards): split language_standards.py into a per-language package - #2527

Merged
squid-protocol merged 2 commits into
mainfrom
refactor/language-standards-package
Aug 31, 2026
Merged

refactor(standards): split language_standards.py into a per-language package#2527
squid-protocol merged 2 commits into
mainfrom
refactor/language-standards-package

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Summary

gitgalaxy/standards/language_standards.py had grown to 14,403 lines / 942KB across 59
languages -- the largest file in the repo, and the one place that hadn't adopted the
per-language-file convention already used by tests/extraction/languages/ and
docs/language_status/. Every consumer (detector.py, language_lens.py, prism.py,
galaxyscope.py, the supply-chain tools, ~150+ tests) only ever accesses
LANGUAGE_DEFINITIONS as an already-merged, in-memory dict via constructor injection or
whole-dict import -- nothing depends on the file being physically monolithic, which is what
makes this safe as a pure structural refactor.

gitgalaxy/standards/language_standards.py is now a package:
gitgalaxy/standards/language_standards/__init__.py assembles the exact same merged dict
(same 59 keys, same order) from one file per language under languages/, plus sibling modules
for the genuinely cross-language pieces (_lens_config.py, _prism_config.py,
_shared_patterns.py's GLOBAL_* debt/AI-SDK detectors, _overrides.py's
PROJECT_OVERRIDES). Every existing import path
(from gitgalaxy.standards.language_standards import LANGUAGE_DEFINITIONS, etc.) is
unchanged, so no consumer -- including language_lens.py (the language-identity classifier)
-- needed any code change.

The split was done as a source-text slice (ast.get_source_segment per top-level dict entry
against the git-committed source), not a load-then-repr() round-trip, specifically to
preserve every in-block # comment (real documentation throughout this registry) and avoid
any risk of altering a regex string literal's escaping.

Companion changes in this PR (all mechanically required, not optional):

  • gitgalaxy_config.py's XRAY_BYPASS_PATHS exact-file entry -> a directory prefix (the
    binary-anomaly-detector's bypass check is a substring match, so this still covers every new
    file underneath).
  • tree_sitter_accuracy_audit.py's _LANGUAGE_STANDARDS_PATH repointed to the new
    __init__.py (where the auto-generated accuracy summary table now lives, byte-identical).
  • Four GitHub workflow files' paths: triggers and exact-file diff/commit checks updated
    (tree-sitter-accuracy-audit.yml/-history.yml, tri-comparison-audit.yml/-history.yml)
    since they gate on or auto-commit to this exact path.
  • how_to_add_a_language.md's registration steps, the standards README.md's module
    breakdown, and the harden-language-extraction skill + class-start-scout/
    strict-signature-scout subagents updated to describe the new per-language file layout
    (a handful of purely-prose mentions elsewhere are left for a follow-up sweep, since the
    dotted import path itself didn't change).

Type of change

  • Bug fix
  • New feature / language support
  • Parsing or engine logic (gitgalaxy/core/detector.py, language_standards.py, prism.py, or a per-language rule)
  • Docs, tooling, or CI only
  • Other (describe above)

CI checklist

  • python tests/tools/audit_check.py passes (bundles the ruff/mypy/dead-key baseline-gated audits + ruff format --check)
  • python -m pytest tests/ passes
  • python tests/tools/crucible_check.py -- see Verification below, drift is pre-existing/environmental, not caused by this PR
  • python tests/tools/tree_sitter_accuracy_audit.py --all --ci -- see Verification below, same environmental caveat for 4 of 35 checked languages

Differential Scan target

Not applicable -- pure structural refactor, no regex/parsing behavior changed (see the
byte-for-byte equivalence check below).

Verification

$ python /tmp/verify_split_equivalence.py   -- deep-compares every language's rules dict
  (patterns + flags) plus LENS_CONFIG/PRISM_CONFIG/PROJECT_OVERRIDES/
  HTML_NONEXECUTABLE_SCRIPT_TAG against `git show origin/main:...language_standards.py`
  OK: 59 languages, all byte-for-byte equivalent, key order preserved

$ python /tmp/verify_comments.py            -- every in-block comment survived the split
  OK: every language's comment set matches exactly

$ ruff format --check .                     -- clean
$ python tests/tools/audit_check.py         -- [ruff] OK, [mypy] OK, [dead-key] OK, [ast-accuracy] OK
$ python -m pytest tests/                   -- 7138 passed, 2 skipped, 9 xfailed, 3 xpassed, 0 test files touched
$ python tests/tools/tree_sitter_accuracy_audit.py --summary-table   -- zero diff (confirms the repointed path)
$ python tests/tools/tri_comparison_chart.py --all --ci              -- all OK (3 languages with committed precision baselines)
$ fresh `pip install -e ".[yaml]"` + import sanity check              -- 59 languages load correctly
  (catches a missing package __init__.py that a stale editable install would mask)
$ galaxyscope self-scan of the new package --fail-on-secrets          -- zero secrets-risk findings,
  confirming the split didn't need to propagate the galaxyscope:ignore marker beyond __init__.py

crucible_check.py and tree_sitter_accuracy_audit.py --all --ci both show drift in this
environment (4061/4054 golden-master diffs; 4 of 35 languages failing a "ground-truth corpus
changed" check). Both were verified as pre-existing and unrelated to this PR by stashing all
changes and re-running against pristine, unmodified origin/main: the golden-master diff count
matched exactly (4061/4054), and the same 4 languages failed with the identical
"files_scanned: baseline=X current=Y ... This means the corpus changed" message (a stale local
language-crucible checkout, not a regex regression -- the tool itself distinguishes this
failure mode from an actual accuracy regression). Combined with the byte-for-byte equivalence
check above, this confirms zero behavioral change from the split.

Worktree rebase note: 8 active local worktrees carry small, isolated single-language diffs
to the old language_standards.py (fix-python-2238, fix-csharp-2237, fix-ruby-2241,
fix-jcl-2482, fix-cpp-2010, fix-cpp-2009, fix-cpp-2012, fix-perl-2239) and will hit a
real delete/modify conflict rebasing onto this -- expected and loud, not silent; each one maps
cleanly onto its language's new file under languages/.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

…package

The single-file LANGUAGE_DEFINITIONS registry had grown to 14,403 lines /
942KB across 59 languages, the largest file in the repo and the one place
that hadn't adopted the per-language-file convention already used by
tests/extraction/languages/ and docs/language_status/. Every consumer
(detector.py, language_lens.py, prism.py, galaxyscope.py, the supply-chain
tools, ~150+ tests) only ever accesses LANGUAGE_DEFINITIONS as an
already-merged, in-memory dict via constructor injection or whole-dict
import -- nothing depends on the file being physically monolithic.

gitgalaxy/standards/language_standards.py is now a package:
gitgalaxy/standards/language_standards/__init__.py assembles the exact same
merged dict (same 59 keys, same order) from one file per language under
languages/, plus sibling modules for the genuinely cross-language pieces
(_lens_config.py, _prism_config.py, _shared_patterns.py's GLOBAL_* debt/AI-SDK
detectors, _overrides.py's PROJECT_OVERRIDES). Every existing import path
(`from gitgalaxy.standards.language_standards import LANGUAGE_DEFINITIONS`,
etc.) is unchanged.

The split was done as a source-text slice (ast.get_source_segment per
top-level dict entry against the git-committed source), not a
load-then-repr() round-trip, to preserve every in-block comment and avoid
any risk of altering regex literals. Verified via:
- a deep-compare script confirming every language's rules dict (patterns +
  flags) and LENS_CONFIG/PRISM_CONFIG/PROJECT_OVERRIDES/
  HTML_NONEXECUTABLE_SCRIPT_TAG are byte-for-byte equivalent to the
  pre-split file, plus key order preserved
- a comment-fidelity check confirming every in-block comment survived
- full pytest suite (7138 passed, 0 test files touched)
- ruff format/check clean, audit_check.py baselines unchanged
- a fresh `pip install -e` sanity check (catches a missing package
  __init__.py that a stale editable install would mask)
- a self-scan confirming no new sec_hardcoded_secrets/secrets_risk findings
- tree_sitter_accuracy_audit.py --summary-table and
  tri_comparison_chart.py --all --ci both clean against the repointed path

Companion changes: XRAY_BYPASS_PATHS' exact-file entry -> a directory
prefix (binary_anomaly_detector.py's bypass check is substring-based, so
this still covers every new file); tree_sitter_accuracy_audit.py's
summary-table path repointed to the new __init__.py; four GitHub workflow
files' path triggers/exact-file checks updated (tree-sitter-accuracy-audit/
-history, tri-comparison-audit/-history) since they gate on or auto-commit
to this exact path; how_to_add_a_language.md's registration steps, the
standards README's module breakdown, and the harden-language-extraction
skill + two scout subagents updated to describe the new per-language file
layout.

crucible_check.py and tree_sitter_accuracy_audit.py --all --ci both show
pre-existing drift in this environment (confirmed via control runs against
unmodified origin/main producing the identical drift) -- local
golden-master graph-layout non-determinism and a stale local
language-crucible corpus checkout, respectively. Neither is caused by or
related to this change; both are orthogonal environment issues.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Comment thread gitgalaxy/standards/language_standards/languages/makefile.py Fixed
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

CodeQL flagged [;|&|(] as a duplicate-character character class. Confirmed
pre-existing in the original monolithic language_standards.py (unchanged by
the split, just relocated) -- [;|&|(] and [;|&(] match the exact same set
of characters, so this is a no-op for matching behavior, verified with a
side-by-side regex comparison across realistic Makefile conditional forms
plus the full makefile test suite.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@squid-protocol
squid-protocol merged commit 495f5b8 into main Aug 31, 2026
30 checks passed
@squid-protocol
squid-protocol deleted the refactor/language-standards-package branch August 31, 2026 01:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants