Skip to content

core: shield string literals in positional comment stripper (#259) - #2506

Merged
squid-protocol merged 1 commit into
mainfrom
fix/259-positional-comment-literal-shield
Aug 30, 2026
Merged

core: shield string literals in positional comment stripper (#259)#2506
squid-protocol merged 1 commit into
mainfrom
fix/259-positional-comment-literal-shield

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

What

Prism._strip_positional_comments (the positional_anchored / positional_abap families — cobol, fortran, abap) split each line directly on *> / ! / " with no string-literal shielding, unlike every other stripping path in prism.py (_strip_single_line_comments, _strip_segment_comments, _strip_nested_comments all mask via LITERAL_MASK_PATTERN first).

A delimiter-shaped character inside a string literal was misread as a real inline-comment marker:

_strip_positional_comments('       DISPLAY "Rate *> 5%" TO CONSOLE.\n       MOVE X TO Y.', cobol_mode=True)
  # before:  code = '       DISPLAY "Rate \n       MOVE X TO Y.'   ← "TO CONSOLE." lost, dangling unterminated "
  # after:   code = '       DISPLAY "Rate *> 5%" TO CONSOLE.\n       MOVE X TO Y.'

Same failure for the ! branch (DISPLAY "Warning!") and the ABAP " branch (x = 'he said "hi"'). The tail of the literal + rest of the statement moved to the comment stream, deflating coding_loc and silently dropping real structural signals.

The prior closure ("resolved by #273") referenced the wrong PR — #273 only touched network_risk_sensor.py. #1745 fixed a different bug in the same function (line-count geometry). No regression test guarded this.

Fix

Golden masters (re-blessed, both modes)

python tests/tools/update_golden_master.py run once per mode. 5 diffs, all on one file: cobol/che-che4z_lsp_project_fixtures/special_schema.cbl, whose DATA ... VALUE clauses hold XML-ish string literals containing ! (' <itemName>Not here!</itemName>'). With the literal no longer truncated at the !:

  • Type/Safety Bypasses: 0 → 1 (a real signal previously hidden in the mis-stripped comment portion)
  • Error & Exception Exposure: 62.98% → 65.52% on that file, + 3 aggregate rollups

No line-shift noise anywhere else across the corpus.

Verification

  • tests/core_engine/ (572 passed) + new test_prism.py cases (delimiter-inside-literal for cobol *>/!, fortran !, abap ", plus real-inline-comment control cases)
  • ✅ cobol / fortran / abap extraction gauntlets + _strict siblings (412 passed)
  • tests/tools/crucible_check.py — both modes PASS after re-bless
  • tri_comparison_chart.py --all --ci, tree_sitter_accuracy_audit.py --all --ci (pre-existing unrelated css corpus-drift failure reproduces on clean main with these changes stashed — not introduced here)
  • ruff_audit.py --ci, mypy_audit.py --ci, dead_key_audit.py --ci, ruff format --check on prism.py

Closes #259

🤖 Generated with Claude Code

`Prism._strip_positional_comments` (the `positional_anchored` /
`positional_abap` families -- cobol, fortran, abap) split each line
directly on `*>` / `!` / `"` with no string-literal shielding, unlike
every other stripping path in prism.py (`_strip_single_line_comments`,
`_strip_segment_comments`, `_strip_nested_comments` all mask via
`LITERAL_MASK_PATTERN` first).

A delimiter-shaped character inside a literal -- `DISPLAY "Rate *> 5%"`,
`PRINT *, "Warning!"`, ABAP `x = 'he said "hi"'` -- was misread as a real
inline-comment marker: the tail of the literal plus the rest of the
statement moved to the comment stream and `code_stream` was left with a
dangling unterminated quote, deflating `coding_loc` and dropping real
structural signals.

Fix: mask literals per line (the same bounded discipline
`_strip_single_line_comments` adopted for #1184) before the inline
`*>` / `"` / `!` search, restoring them into both halves after the split.
The column-1 / column-7 anchor check stays on the raw line so column
positions are exact. ABAP masks single-quote / backtick only -- its `"`
is the comment delimiter, not a string quote (new
`ABAP_LITERAL_MASK_PATTERN`). `_mask_line_literals` gains an optional
`pattern` override. Column-7 `-` continued literals remain out of scope
(masking is deliberately line-bounded).

Golden masters re-blessed (both modes): 5 diffs, all on
`cobol/che-che4z_lsp_project_fixtures/special_schema.cbl`, whose DATA
VALUE clauses hold XML-ish string literals containing `!`
(`'<itemName>Not here!</itemName>'`). With the literal no longer
truncated at the `!`, one previously-hidden `safety_bypasses` signal is
now correctly retained (0 -> 1) and its downstream risk-exposure
percentages shift. No line-shift noise anywhere else across the ~80-repo
corpus.

Closes #259

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ni57RP5ATiqfN3XzJvRDPY
@squid-protocol squid-protocol added legacy-modernization COBOL refractor, dead-code extraction, and JCL forging priority: high Core feature broken, but workarounds exist threat: false-positive Accuracy tuning: Engine incorrectly flagged safe code bug Unintended behavior or logic failure in the engine core-engine Modifications to the central physics and parsing engine labels Aug 30, 2026
@github-actions

Copy link
Copy Markdown
Contributor

🐦‍⬛ Muninn Security Scan

✅ No security issues found.

🐦‍⬛ Powered by Muninn · Skald Lab

@squid-protocol
squid-protocol merged commit e792b85 into main Aug 30, 2026
89 of 90 checks passed
@squid-protocol
squid-protocol deleted the fix/259-positional-comment-literal-shield branch August 30, 2026 20:48
@squid-protocol

Copy link
Copy Markdown
Owner Author

The failing tree-sitter-accuracy-audit / css check is pre-existing and unrelated to this PR — real_functions 24→25 drift from a stale css baseline (last regenerated in #2478; #2500's audit-reader changes weren't re-blessed). Verified it reproduces on main (4cc5001f) with this branch's changes stashed, and #2500 merged with the identical failure. This PR only touches _strip_positional_comments (cobol/fortran/abap), which cannot affect css.

Tracked separately in #2507 (also covers the @layer/@media/@supports-as-function FP the same audit surfaces).

All other checks green: crucible-audit both modes, full-suite-matrix ×40, tri-comparison, ruff/mypy/dead-key.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Unintended behavior or logic failure in the engine core-engine Modifications to the central physics and parsing engine legacy-modernization COBOL refractor, dead-code extraction, and JCL forging priority: high Core feature broken, but workarounds exist threat: false-positive Accuracy tuning: Engine incorrectly flagged safe code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fixed-form COBOL/Fortran comment stripping doesn't shield string literals

1 participant