yacc corpus: both real .y files silently excluded from file_data by statistical_auditor.py despite successful extraction

[squid-protocol]

Description

Found investigating why the tri-comparison chart reports "yacc -- corpus has no real yacc files, treating as awaiting data" despite language-crucible/data/yacc/freebsd/ containing two real, non-trivial .y grammar files (jailparse.y, 276 lines; config.y, 503 lines).

Confirmed this is NOT a language-routing/ingestion problem -- running galaxyscope directly against this corpus shows the extraction pipeline DOES process both files and DOES find real functions in them (debug output: GG SPLICE FUNCS: ['newopt', 'newdev', 'newenvvar', 'rmdev_schedule', 'rmopt_schedule', 'finddev', 'findopt', 'yywrap', 'newfile', 'newoptionsfile', 'devopt', 'yyerror'] for one file, ['conf', 'jail_name', 'name', 'string', 'value', 'param', 'jail'] for the other -- real yacc/bison C-action function names). But the run's own summary log says:

Anomaly sweep concluded | Stable Files Mapped: 1 | Collapsed to Exclusion Queue: 2

and the final file_data table (queried directly from the produced sqlite DB) contains only freebsd/COPYRIGHT (correctly classified plaintext) -- both .y files are missing entirely, not misclassified as some other language.

Root cause (partially isolated, not fully pinned down)

gitgalaxy/metrics/statistical_auditor.py runs a per-language-cohort anomaly sweep (README.md's "Quality Gate" section: the Impossible Density Law at rho > 3.0 for loc > 30, a Zero-Density hard floor, and a robust Z-score/MAD check for everything else -- see lines 382-405). The yacc corpus has exactly 2 real code files total in this pinned snapshot. A per-cohort statistical check (median/MAD-based Z-score, or any relative-density comparison) computed over a population of 2 is close to a degenerate case -- either file being denser or sparser than the other by any margin can produce an extreme Z-score purely from having almost no comparison population, independent of whether the file is actually anomalous.

I was not able to fully isolate which of the three specific checks (Zero-Density Threshold, Packed Payload Guard/Impossible Density Law, or the Z-score/MAD branch) is the one actually firing for these two files without deeper instrumentation than this pass covered -- the log line that would say (relegation_reason, set at statistical_auditor.py:385/393/405) isn't surfaced at the log level galaxyscope --db-only runs at by default.

Suggested next step

Add debug-level logging of relegation_reason (or run with a debug logger) against this exact corpus to confirm which check fires, then decide the real fix: most likely the Z-score/MAD branch needs a minimum-cohort-size guard (skip the statistical check entirely below some N, same spirit as the file already bypassing checks for "heavy polyglot" files) rather than applying a population-relative statistic to an n=2 population.

Impact

This is why yacc has never had any tri-comparison coverage at all (tri_comparison_chart.py reports "no real yacc files" and skips it) despite ctags_reader.py's own documented intent that GitGalaxy has real functions to find here that ctags structurally cannot see ("yacc": set() in CTAGS_FUNC_KINDS, with the comment "GitGalaxy has functions to find here, ctags structurally cannot see them"). The corpus and the extraction both work; only this exclusion step is blocking any comparison at all.

Evidence

Reproduced directly: galaxyscope language-crucible/data/yacc --db-only --output /tmp/yacc_test_scan.json, then querying the resulting sqlite DB's file_data table.

[squid-protocol]

Root-caused and fixed in #2324 (merged).

The earlier "MAD branch on an n=2 population" hypothesis was wrong — that branch never runs for these files. Actual cause:

.y is a collision extension (c + yacc), and the classifier had no internal_discriminator for yacc, so an action-heavy grammar resolved either to c or to a Tier-4 "Collision Resolved" lock. statistical_auditor.py's Heuristic Extension Consensus triage (tier >= 4 or "Collision" in proof → ambiguous_artifacts) then banished it at statistical_auditor.py:218 — the ecosystem loop-back can only rescue an ambiguous file via a vote from confidently-parsed siblings of the same extension, which for .y can never exist. Confirmed to have hit cobol/gnucobol_internals/parser.y too, not just the yacc corpus.

Fix (#2324):

1. language_standards.py — yacc gets an internal_discriminator (%{/%}/%%/%token/%union/…). .y grammars now classify as yacc at Tier 2 with the extension matched, passing the consensus triage as confident artifacts.
2. statistical_auditor.py — the consensus loop-back's "banish" branch now keeps a decisively-resolved collision file (extension collision + confidence ≥ 0.85 + real extracted structure) instead of deleting it.

jailparse.y (7 rules), config.y (11), gnucobol parser.y (781) now appear in file_data as yacc. Golden masters re-blessed. The tri-comparison-history companion will add yacc chart/ledger coverage on the next main push (GitGalaxy-only there — ctags' YACC parser sees only labels/tokens, no tree-sitter grammar).

Two incidental findings spun out (see #2324's description): the consensus engine silently deleting cleanly-extracted files, and StatisticalAuditor being constructed without lang_defs at galaxyscope.py:722 (which makes GATE C / GATE D / dead-code detection dead code in production).