Skip to content

fix(groovy): tri-comparison manual verification -- 3 real engine defects fixed - #2532

Merged
squid-protocol merged 1 commit into
mainfrom
tri-comparison/groovy-validation
Aug 31, 2026
Merged

fix(groovy): tri-comparison manual verification -- 3 real engine defects fixed#2532
squid-protocol merged 1 commit into
mainfrom
tri-comparison/groovy-validation

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Summary

tree_sitter_language_pack's "groovy" grammar loads without error but can't parse real
Groovy declarations -- confirmed via direct inspection (30-file/13K-node sample, 3.2%
ERROR rate, no class_declaration/method_declaration node type at all). The two node
types this repo's NODE_MAPS used to map groovy to ("func", "generics_class") turned
out to be semantically wrong, not just imprecise: "func" nodes are call expressions
(getLogger(JiraService.class)), not definitions; "generics_class" nodes are generic
type usage
(Map<String, X>), not class declarations. ctags has zero Groovy support at
all. Groovy is now routed through the gg_only manual-verification treatment (same category
abap/dockerfile/jcl/livecode/yaml are already in) instead of a broken tree-sitter comparison.

With no automated ground truth available, ran a manual verification per the
tri-comparison-ledger-sweep skill's fallback procedure: an independent grep-based
ground-truth scan across the full 188-file language-crucible v1.2.0 groovy corpus,
cross-checked against the real pipeline's DB output. Found and fixed three real,
independent GitGalaxy engine defects
:

  1. class_start had zero capture groups -- class existence was detected correctly,
    but every one of 286 named classes was literally called "Anonymous_Class" instead of
    its real name. The only language in _CLASS_START_NAMED_EXTRACTION_LANGS with this
    defect (checked all 36).
  2. Prism never shielded groovy's """/''' multi-line strings -- code-shaped fixture
    text embedded in one (Gradle integration-test build-script snippets, Spock's own
    compiler-smoke-test source samples) leaked into the real code stream and corrupted
    extraction for the rest of the file via desynced brace-counting. New
    _strip_groovy_triple_quoted_strings in prism.py, same idiom as the existing
    PowerShell here-string / Lua long-bracket precedents.
  3. func_start's zero-prefix branch matched the bare def keyword as a function name
    once a Spock quoted feature-method name got correctly blanked for brace-safety. Fixed two
    ways (excluded def from the exclusion lookahead; preserved a quoted name following
    def in the brace-safety shield instead of blanking it), plus a related _extract_name
    fix (it silently truncated a quoted multi-word name to its last word) -- confirmed
    zero-blast-radius elsewhere since groovy is currently the only language whose
    func_start captures a quoted name at all.

Post-fix: zero unexplained recall gaps across the whole corpus for either functions or
classes. One confirmed, unfixed false-positive class remains -- MarkupBuilder/
Jenkins-Pipeline-DSL trailing-closure calls (button(...) { }) are syntactically
indistinguishable from a real zero-prefix declaration -- filed as
#2530 rather than patched with a
fragile denylist (needs its own design pass). Verified: 661/718 (92.1%) named functions,
222/222 (100%) named classes.

Full write-up: docs/language_status/groovy.md §9.

Type of change

  • Bug fix
  • New feature / language support
  • Parsing or engine logic (gitgalaxy/core/detector.py, language_standards.py, prism.py, or a per-language rule)
  • Docs, tooling, or CI only
  • Other (describe above)

CI checklist

  • python tests/tools/audit_check.py passes
  • python -m pytest tests/ passes -- 7145 passed
  • python tests/tools/crucible_check.py passes -- both golden masters re-blessed after confirming the diff (2109 changes) traced entirely to the groovy fix plus expected small ripple (see Verification below)
  • python tests/tools/tree_sitter_accuracy_audit.py --all --ci passes -- groovy no longer measured this way (see Summary)

Differential Scan target

Not applicable to a single repo -- verified against the full ~80-repo crucible corpus (see Verification).

Verification

$ python tests/tools/audit_check.py                                    -- clean
$ python -m pytest tests/                                              -- 7145 passed
$ python tests/tools/tree_sitter_accuracy_audit.py --summary-table     -- groovy removed, zero diff otherwise
$ python tests/tools/tri_comparison_chart.py --all --write             -- groovy now renders as gg_only (1-bar)

Full-corpus differential scan, done twice for control:

  1. Ran a standalone golden_diff.deep_compare() (uncapped, unlike the 50-line-truncated
    pytest failure message) against the committed golden master with the 3 fixes applied:
    2109 diffs. 1587 directly groovy-attributable (directory_groups/groovy/* mass/exposure
    recomputing from corrected counts). The remaining 522 are all tiny (<1% relative, one
    5.5% outlier that's a repo-wide aggregate directly downstream of the groovy change) --
    topological-coordinate ripple from the shared force-directed graph layout recomputing
    when any node's structural mass changes, and global risk-normalization shifting slightly
    -- explicitly called out as an acceptable ripple class in CLAUDE.md's Differential Scan
    protocol. Zero off-target anomalies in any unrelated file.
  2. Control run: stashed all code changes, ran the identical scan against pristine
    origin/main -- zero diffs, confirming the 2109 diffs are entirely attributable to
    this PR's fixes, not environment noise.
  3. Re-blessed both golden masters (crucible_check.py --update --yes), then re-ran
    crucible_check.py plain -- both full_precision and zero_dependency now PASS clean.

Other changes in this PR

  • tests/tools/tree_sitter_accuracy_audit.py: removed groovy's broken NODE_MAPS entry,
    documented the exclusion (same style as the existing cobol/dockerfile/scheme/yaml bullets),
    deleted the now-dead tests/tree_sitter_accuracy_baseline_groovy.json.
  • tests/tools/tri_comparison_chart.py: added groovy to _GG_ONLY_LANGS so it renders
    correctly as a GitGalaxy-only bar instead of silently disappearing from --all; fixed
    stale language-count comments this move affected (45 total unchanged, groovy just moved
    buckets -- was mis-stated as 46 in an earlier draft of this PR, corrected).
  • tests/tools/tri_comparison_reconcile.py, tests/tools/ctags_reader.py: matching
    comment-accuracy fixes for the same bucket move.
  • docs/self_scan/tri_comparison_ledger.json: validated the pre-existing
    groovy/function/existence/agree[gitgalaxy]_vs[tree_sitter] shape
    (credit_tools: [gitgalaxy]) -- verified only this one entry's status/verdict changed,
    everything else is the normal chart-regen timestamp/count refresh.
  • docs/self_scan/manual_verification.json: new groovy entry (function/class), same
    schema as abap/dockerfile/jcl/livecode/yaml.
  • docs/language_status/groovy.md: new, sections 1-8 built via the language-status skill
    from primary sources (rules dict, test counts, closed issues, gitgalaxy-raw-output
    evidence), §9 the full tri-comparison write-up.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

…cts fixed

tree-sitter has no usable grammar for Groovy despite tree_sitter_language_pack
nominally loading a "groovy" grammar: it has no class_declaration/method_declaration
node type at all (class Foo { parses into generic command/unit/block soup, 3.2%
ERROR rate on a 30-file sample), and the two node types this repo's NODE_MAPS used
to map it to ("func", "generics_class") are semantically wrong, not just imprecise
-- "func" nodes are call expressions, not definitions, and "generics_class" nodes
are generic type usage, not class declarations. ctags has zero Groovy support at
all. Removed groovy from tree_sitter_accuracy_audit.py's NODE_MAPS (documented
alongside the existing cobol/dockerfile/scheme/yaml exclusions) and routed it
through the gg_only manual-verification treatment instead, same category
abap/dockerfile/jcl/livecode/yaml are already in.

With no automated ground truth available, ran a manual verification instead: an
independent grep-based ground-truth scan across the full 188-file language-crucible
v1.2.0 groovy corpus (sharing no implementation with GitGalaxy's own regexes),
cross-checked against the real pipeline's actual DB output. Found and fixed three
real, independent GitGalaxy engine defects along the way:

- class_start had zero capture groups (pattern.groups() == 0) -- class existence
  was detected correctly but every one of 286 named classes was literally called
  "Anonymous_Class" instead of its real name. The only language in
  _CLASS_START_NAMED_EXTRACTION_LANGS with this defect (checked all 36).
- Prism never shielded groovy's """/''' multi-line strings -- code-shaped fixture
  text embedded in one (Gradle integration-test build-script snippets, Spock's own
  compiler-smoke-test source samples) leaked into the real code stream and
  corrupted extraction for the rest of the file via desynced brace-counting. Fixed
  with a new _strip_groovy_triple_quoted_strings in prism.py, same idiom as the
  existing PowerShell here-string / Lua long-bracket precedents.
- func_start's zero-prefix branch matched the bare "def" keyword as a function name
  once a Spock quoted feature-method name got correctly blanked for brace-safety.
  Fixed two ways: excluded "def" from both branches' own exclusion lookahead, and
  preserved a quoted name immediately following "def" in detector.py's brace-safety
  shield instead of blanking it (mirrors the existing zig `@"..."` exception in the
  same function). A related _extract_name defect (silently truncating a quoted
  multi-word name down to its last word) was also fixed with an early-return fast
  path -- confirmed zero-blast-radius elsewhere since groovy is currently the only
  language whose func_start captures a quoted name at all.

Post-fix: zero unexplained recall gaps found across the whole corpus for either
functions or classes. One confirmed, unfixed false-positive class remains --
MarkupBuilder/Jenkins-Pipeline-DSL trailing-closure calls (`button(...) { }`) are
syntactically indistinguishable from a real zero-prefix declaration -- filed as
#2530 rather than patched with a fragile denylist (needs its own design pass).
Verified: 661/718 (92.1%) named functions, 222/222 (100%) named classes.

Full verification chain: standalone regex re-tests, extraction gauntlet + strict
suite, ruff/mypy/dead-key audits, full pytest suite (7145 passed), and the full
~80-repo crucible differential scan (2109 diffs -- 1587 directly groovy-
attributable, the remaining 522 all tiny ripple <1% relative from the shared
force-directed graph layout and global risk-normalization recomputing, confirmed
via a pristine-origin/main control run showing zero diffs). Both golden master
fixtures re-blessed. Ledger shape
groovy/function/existence/agree[gitgalaxy]_vs[tree_sitter] validated with
credit_tools=[gitgalaxy] (tree-sitter's non-corroboration is a confirmed tool
limitation, not evidence against GitGalaxy). Full write-up in
docs/language_status/groovy.md §9; verified counts recorded in
docs/self_scan/manual_verification.json.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

@squid-protocol
squid-protocol merged commit 7b7cf31 into main Aug 31, 2026
30 checks passed
@squid-protocol
squid-protocol deleted the tri-comparison/groovy-validation branch August 31, 2026 12:25
squid-protocol added a commit that referenced this pull request Aug 31, 2026
…r declarations (#2597)

* fix(groovy): func_start no longer mistakes MarkupBuilder/DSL calls for declarations

Closes #2530. func_start's zero-prefix branch (needed to match real bare
constructors, `MyClass(String arg) {`) was syntactically indistinguishable
from Groovy's MarkupBuilder/NodeBuilder DSL idiom -- a method call with a
named-argument map plus a trailing closure (`div(class: "x") { ... }`,
`button(name: "clear", type: "submit") { ... }`). Pervasive in Jenkins's
own `.groovy` view templates and Jenkins Pipeline DSL steps.

Fix: a real Groovy declaration's parameter list can never contain a `key:`
token (that syntax is call-site named-argument sugar only), so branch 2
now rejects a candidate whose parenthesized argument list contains an
identifier immediately followed by `:` with no intervening whitespace --
distinguishing it from a ternary's `cond ? a : b`, which conventionally
has a space before the colon.

Verified via a full corpus differential scan (galaxyscope --db-only, 201
files): 44 of the 57 misdetections in jenkins_view_groovy/ resolved, zero
removals anywhere else in the corpus -- no legitimate bare constructor
lost recall, including edge cases like a Map-literal default (`= [:]`)
or a ternary default value. ReDoS-scaling probed (linear to n=16000).

13 residual FPs remain, out of scope per the issue: single-positional-arg
DSL calls (`stage('Build') { }`, `node('x') { }`, `dir(x) { }`) have no
`key:` token to key off of -- same class the issue explicitly deferred
(a curated builder-tag-name denylist is fragile/incomplete by
construction). Fixing the button(...) false positive also unmasked one
new, unrelated func_start defect in a different branch (a paren-less
call, `raw _("Dismiss")`, previously silently absorbed as "nested inside
button") -- filed separately as #2558 rather than bundled in here.

Golden masters re-blessed (both precision modes): 43 net diff (44 direct
removals - 1 new #2558 surfacing), rest is the standard topological/
aggregate ripple class already documented in CLAUDE.md's Differential
Scan protocol. Verified with a full uncapped diff categorization: zero
unexplained/off-target changes anywhere in the 201-file corpus.

Also updates docs/language_status/groovy.md, manual_verification.json,
and regenerates the tri-comparison chart/ledger to reflect the corrected
groovy function precision (661/718 -> 661/675, 92.1% -> 97.9%).

Verification: full groovy extraction gauntlet + strict suite (150
passed, new regression test added), cross-language strict registry (21
passed), full pytest suite (7154 passed), audit_check.py clean,
tree_sitter_accuracy_audit.py --all --ci clean (groovy correctly
excluded as gg_only per #2532), crucible_check.py both modes PASS after
re-bless.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* chore: regenerate tri-comparison chart/ledger after merging main

Resolves the merge conflict in tri_comparison_ledger.json from main's
own auto-update-tri-comparison-data bot commit (#2553) landing while
this branch's crucible verification was running. Took main's version
as the base and regenerated via tri_comparison_chart.py --all --write
on top of the merged code state, so the output reflects both main's
other-language refreshes and this branch's groovy fix together.
Re-verified clean: audit_check.py, full pytest suite (7154 passed),
crucible_check.py both modes PASS.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Joe Esquibel <squid-protocol@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant