fix(core): yacc .y grammar files silently excluded from output (#1926) - #2324
Merged
Conversation
`.y` is a collision extension (both `c` and `yacc` claim it). The classifier
had no `internal_discriminator` for yacc, so a real grammar whose embedded
C-action blocks out-mass the grammar resolved to `c`, or -- when the lexical
winner was `yacc` -- to a Tier-4 "Collision Resolved" lock. Either way,
`statistical_auditor.py`'s Heuristic Extension Consensus triage
(`tier >= 4 or "Collision" in proof` -> ambiguous) then banished the file:
the ecosystem loop-back can only rescue an ambiguous file via a vote from
*confidently-parsed siblings of the same extension*, which for `.y` can never
exist, and no `.h/.hpp/.inc` fallback applies. Both corpus grammars, plus
`cobol/gnucobol_internals/parser.y`, were deleted at
`statistical_auditor.py:218` ("Unresolved Ambiguity") despite extracting
real functions -- the MAD/density gates the issue hypothesised never ran.
(`c/sqlite/parse.y` stays excluded for an unrelated reason: the
machine-generated-source radar.)
Two coordinated fixes:
1. `language_standards.py` -- yacc gets an `internal_discriminator` matching
definitive yacc/bison/lex syntax no plain C file contains (`%{`/`%}`
prologue delimiters, the `%%` section separator, `%token`/`%union`/`%type`/
... declarations). `_tier_2_fingerprint_check` now resolves the `.y`
collision to `yacc` deterministically -- Tier 2, extension matched, no
"Collision" proof string -- so the grammar passes the consensus triage as
a confident artifact. Same collision-resolution role objective-c/matlab
already use for `.m`. Plain `.c` files are untouched (the discriminator
is only consulted for `.y`/`.yy`/`.ypp`/`.l`/`.ll`/`.lpp`).
2. `statistical_auditor.py` -- the consensus loop-back's final "banish"
branch now KEEPS a decisively-resolved collision file (came in via a
genuine extension collision, identity confidence >= 0.85, and actually
produced extracted structure) instead of silently deleting it. Safety net
for the general class of collision-only extensions; a bare Tier-4 lexical
guess with no extension support is still banished unchanged.
Result: `jailparse.y` (7 grammar rules), `config.y` (11), and
`gnucobol_internals/parser.y` (781) now appear in `file_data` as `yacc`.
Golden masters re-blessed -- the only content-level change is those 3 files
going from Excluded -> parsed yacc; the rest of the 398-diff churn is
downstream spatial-mapper coordinate / directory-mass / global-health
recomputation from adding ~799 nodes to the graph. The tri-comparison-history
companion will pick up chart/ledger coverage on the next main push.
Verification: test_yacc.py + test_yacc_strict.py (77) green; full
tests/core_engine + tests/extraction (6791) green; ruff/mypy/dead-key/
ast-accuracy clean (ruff baseline: one line-shift + 7 pre-existing stale
galaxyscope entries auto-pruned by audit_check --regenerate); crucible_check
both modes PASS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Aug 27, 2026
Contributor
squid-protocol
added a commit
that referenced
this pull request
Aug 27, 2026
… (#2333) The Heuristic Extension Consensus loop-back in `statistical_auditor.py` sent every `tier >= 4 or "Collision" in proof` artifact to `ambiguous_artifacts` and then DELETED any it couldn't confirm via an 80% ecosystem vote (or the `.h`/`.hpp`/`.inc` C-family fallback, or #2324's decisive-collision keep) -- even when the file classified to a real language and GitGalaxy had extracted real named functions/classes from it. #1926 was one instance (every `.y` grammar); this is the general case. Replaces the "banish unless decisively-resolved collision" tail with a Structure Retention rule: - ambiguous + real language + >=1 named symbol (function/class), OR a decisively-resolved collision (conf >= 0.85, >= 5 signal hits) -> KEEP. - a decisively-resolved collision keeps the stronger "Lexically Decisive" Tier-3 label (unchanged from #2324); anything else is kept as an explicitly PROVISIONAL identity: `identity_lock_tier = 4`, `telemetry["provisional_identity"] = True`, source proof "Provisional Identity (Ambiguous, N symbol(s) retained: <lang>)", and a WARNING log so nothing downstream silently over-trusts the label. - only genuine noise -- a degenerate `lang_id`, or zero extracted structure -- is still banished, now as "Unresolved Ambiguity (No Retainable Structure)". Zero corpus impact: after #2324, no file in the ~80-repo crucible corpus hits this path, so both golden masters are unchanged (crucible_check PASS both modes). Defensive hardening for arbitrary real-world repos with rare or collision-prone extensions. Verification: test_statistical_auditor.py (10, +1 retention case) and the full tests/security_auditing suite (161) green; ruff/mypy --ci clean (ruff baseline: one PERF203 line-shift); crucible_check both modes PASS. Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1926.
Root cause (the issue's own "MAD branch on n=2" hypothesis was wrong)
.yis a collision extension — bothcandyaccclaim it. The language classifier had nointernal_discriminatorfor yacc, so a real grammar file whose embedded C-action blocks out-mass the grammar itself resolves toc, or — when the lexical winner isyacc— to a Tier-4"Collision Resolved"lock.statistical_auditor.py's Heuristic Extension Consensus triage (statistical_auditor.py:127-219) then sends everytier >= 4 or "Collision" in prooffile toambiguous_artifacts. The loop-back can only rescue those via an ecosystem vote from confidently-parsed siblings of the same extension — which for.ycan never exist (every.yis a collision). No.h/.hpp/.incfallback applies either, so both grammars hit line 218:"Unresolved Ambiguity (Tier 4 Fallback failed Ecosystem Consensus)"→ deleted, despite having extracted real functions. The MAD / density gates the issue points at never run.Not corpus-specific:
cobol/gnucobol_internals/parser.ywas also silently excluded. (c/sqlite/parse.ystays out for an unrelated reason — the machine-generated-source radar.)Fix — two coordinated changes
1.
language_standards.py— yaccinternal_discriminatorMatches definitive yacc/bison/lex syntax no plain C file contains:
%{/%}prologue delimiters, the%%section separator,%token/%union/%type/… declarations._tier_2_fingerprint_checknow resolves the.ycollision toyaccdeterministically (Tier 2, extension matched, no"Collision"proof string), so the grammar passes the consensus triage as a confident artifact — the same collision-resolution mechanism objective-c/matlab already use for.m. Only consulted for.y/.yy/.ypp/.l/.ll/.lpp; plain.cis untouched.2.
statistical_auditor.py— decisive-collision keepThe consensus loop-back's final "banish" branch now keeps a decisively-resolved collision file (came in via a genuine extension collision + identity confidence ≥ 0.85 + real extracted structure) instead of silently deleting it. Safety net for the general class of collision-only extensions. A bare Tier-4 lexical guess with no extension support is still banished, unchanged.
Result
yacc/freebsd/jailparse.yyacc, 7 grammar rulesyacc/freebsd/config.yyacc, 11 rulescobol/gnucobol_internals/parser.yyacc, 781 rulesGolden masters
Re-blessed both modes. The only content-level change is those 3 files going Excluded → parsed
yacc(verified: no other file changed language or appeared/disappeared). The rest of the ~398-diff churn is downstreamspatial_mappercoordinate + directory-mass + global-health recomputation from adding ~799 nodes to the dependency graph.crucible_check.pyPASS both modes.Incidental findings (filing as separate issues, not fixed here)
StatisticalAuditoris constructed withoutlang_defs(galaxyscope.py:722) →self.lang_defs = {}→ every language hitsis_inert = Trueinaudit()'s per-language loop → GATE C (orphan threshold), GATE D (MAD / density floors), and dead-code detection are all dead code in production scans. Only the pre-loop consensus triage runs. Needs a decision (wire it up vs. delete the dead gates) + a careful regression pass.Verification
test_yacc.py+test_yacc_strict.py— 77 passtests/core_engine+tests/extraction— 6791 passgalaxyscope.pyentries auto-pruned byaudit_check --regenerate)crucible_check.py— PASS both modes🤖 Generated with Claude Code