refactor(standards): split language_standards.py into a per-language package - #2527
Merged
Merged
Conversation
…package The single-file LANGUAGE_DEFINITIONS registry had grown to 14,403 lines / 942KB across 59 languages, the largest file in the repo and the one place that hadn't adopted the per-language-file convention already used by tests/extraction/languages/ and docs/language_status/. Every consumer (detector.py, language_lens.py, prism.py, galaxyscope.py, the supply-chain tools, ~150+ tests) only ever accesses LANGUAGE_DEFINITIONS as an already-merged, in-memory dict via constructor injection or whole-dict import -- nothing depends on the file being physically monolithic. gitgalaxy/standards/language_standards.py is now a package: gitgalaxy/standards/language_standards/__init__.py assembles the exact same merged dict (same 59 keys, same order) from one file per language under languages/, plus sibling modules for the genuinely cross-language pieces (_lens_config.py, _prism_config.py, _shared_patterns.py's GLOBAL_* debt/AI-SDK detectors, _overrides.py's PROJECT_OVERRIDES). Every existing import path (`from gitgalaxy.standards.language_standards import LANGUAGE_DEFINITIONS`, etc.) is unchanged. The split was done as a source-text slice (ast.get_source_segment per top-level dict entry against the git-committed source), not a load-then-repr() round-trip, to preserve every in-block comment and avoid any risk of altering regex literals. Verified via: - a deep-compare script confirming every language's rules dict (patterns + flags) and LENS_CONFIG/PRISM_CONFIG/PROJECT_OVERRIDES/ HTML_NONEXECUTABLE_SCRIPT_TAG are byte-for-byte equivalent to the pre-split file, plus key order preserved - a comment-fidelity check confirming every in-block comment survived - full pytest suite (7138 passed, 0 test files touched) - ruff format/check clean, audit_check.py baselines unchanged - a fresh `pip install -e` sanity check (catches a missing package __init__.py that a stale editable install would mask) - a self-scan confirming no new sec_hardcoded_secrets/secrets_risk findings - tree_sitter_accuracy_audit.py --summary-table and tri_comparison_chart.py --all --ci both clean against the repointed path Companion changes: XRAY_BYPASS_PATHS' exact-file entry -> a directory prefix (binary_anomaly_detector.py's bypass check is substring-based, so this still covers every new file); tree_sitter_accuracy_audit.py's summary-table path repointed to the new __init__.py; four GitHub workflow files' path triggers/exact-file checks updated (tree-sitter-accuracy-audit/ -history, tri-comparison-audit/-history) since they gate on or auto-commit to this exact path; how_to_add_a_language.md's registration steps, the standards README's module breakdown, and the harden-language-extraction skill + two scout subagents updated to describe the new per-language file layout. crucible_check.py and tree_sitter_accuracy_audit.py --all --ci both show pre-existing drift in this environment (confirmed via control runs against unmodified origin/main producing the identical drift) -- local golden-master graph-layout non-determinism and a stale local language-crucible corpus checkout, respectively. Neither is caused by or related to this change; both are orthogonal environment issues. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
CodeQL flagged [;|&|(] as a duplicate-character character class. Confirmed pre-existing in the original monolithic language_standards.py (unchanged by the split, just relocated) -- [;|&|(] and [;|&(] match the exact same set of characters, so this is a no-op for matching behavior, verified with a side-by-side regex comparison across realistic Makefile conditional forms plus the full makefile test suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
gitgalaxy/standards/language_standards.pyhad grown to 14,403 lines / 942KB across 59languages -- the largest file in the repo, and the one place that hadn't adopted the
per-language-file convention already used by
tests/extraction/languages/anddocs/language_status/. Every consumer (detector.py,language_lens.py,prism.py,galaxyscope.py, the supply-chain tools, ~150+ tests) only ever accessesLANGUAGE_DEFINITIONSas an already-merged, in-memory dict via constructor injection orwhole-dict import -- nothing depends on the file being physically monolithic, which is what
makes this safe as a pure structural refactor.
gitgalaxy/standards/language_standards.pyis now a package:gitgalaxy/standards/language_standards/__init__.pyassembles the exact same merged dict(same 59 keys, same order) from one file per language under
languages/, plus sibling modulesfor the genuinely cross-language pieces (
_lens_config.py,_prism_config.py,_shared_patterns.py'sGLOBAL_*debt/AI-SDK detectors,_overrides.py'sPROJECT_OVERRIDES). Every existing import path(
from gitgalaxy.standards.language_standards import LANGUAGE_DEFINITIONS, etc.) isunchanged, so no consumer -- including
language_lens.py(the language-identity classifier)-- needed any code change.
The split was done as a source-text slice (
ast.get_source_segmentper top-level dict entryagainst the git-committed source), not a load-then-
repr()round-trip, specifically topreserve every in-block
#comment (real documentation throughout this registry) and avoidany risk of altering a regex string literal's escaping.
Companion changes in this PR (all mechanically required, not optional):
gitgalaxy_config.py'sXRAY_BYPASS_PATHSexact-file entry -> a directory prefix (thebinary-anomaly-detector's bypass check is a substring match, so this still covers every new
file underneath).
tree_sitter_accuracy_audit.py's_LANGUAGE_STANDARDS_PATHrepointed to the new__init__.py(where the auto-generated accuracy summary table now lives, byte-identical).paths:triggers and exact-file diff/commit checks updated(
tree-sitter-accuracy-audit.yml/-history.yml,tri-comparison-audit.yml/-history.yml)since they gate on or auto-commit to this exact path.
how_to_add_a_language.md's registration steps, the standardsREADME.md's modulebreakdown, and the
harden-language-extractionskill +class-start-scout/strict-signature-scoutsubagents updated to describe the new per-language file layout(a handful of purely-prose mentions elsewhere are left for a follow-up sweep, since the
dotted import path itself didn't change).
Type of change
gitgalaxy/core/detector.py,language_standards.py,prism.py, or a per-language rule)CI checklist
python tests/tools/audit_check.pypasses (bundles the ruff/mypy/dead-key baseline-gated audits +ruff format --check)python -m pytest tests/passespython tests/tools/crucible_check.py-- see Verification below, drift is pre-existing/environmental, not caused by this PRpython tests/tools/tree_sitter_accuracy_audit.py --all --ci-- see Verification below, same environmental caveat for 4 of 35 checked languagesDifferential Scan target
Not applicable -- pure structural refactor, no regex/parsing behavior changed (see the
byte-for-byte equivalence check below).
Verification
crucible_check.pyandtree_sitter_accuracy_audit.py --all --ciboth show drift in thisenvironment (4061/4054 golden-master diffs; 4 of 35 languages failing a "ground-truth corpus
changed" check). Both were verified as pre-existing and unrelated to this PR by stashing all
changes and re-running against pristine, unmodified
origin/main: the golden-master diff countmatched exactly (4061/4054), and the same 4 languages failed with the identical
"files_scanned: baseline=X current=Y ... This means the corpus changed" message (a stale local
language-cruciblecheckout, not a regex regression -- the tool itself distinguishes thisfailure mode from an actual accuracy regression). Combined with the byte-for-byte equivalence
check above, this confirms zero behavioral change from the split.
Worktree rebase note: 8 active local worktrees carry small, isolated single-language diffs
to the old
language_standards.py(fix-python-2238,fix-csharp-2237,fix-ruby-2241,fix-jcl-2482,fix-cpp-2010,fix-cpp-2009,fix-cpp-2012,fix-perl-2239) and will hit areal delete/modify conflict rebasing onto this -- expected and loud, not silent; each one maps
cleanly onto its language's new file under
languages/.Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com