Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/self_scan/tri_comparison_chart.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
444 changes: 219 additions & 225 deletions docs/self_scan/tri_comparison_ledger.json

Large diffs are not rendered by default.

227 changes: 115 additions & 112 deletions docs/self_scan/tri_comparison_points_of_interest.md

Large diffs are not rendered by default.

28 changes: 28 additions & 0 deletions docs/why_gitgalaxy_beats_ast_here.md
Original file line number Diff line number Diff line change
Expand Up @@ -870,6 +870,34 @@ Together these two shapes are the full, sole explanation for the ledger entry's
+ 1, zero unexplained residual) — see `tests/tools/ctags_reader.py`'s cobol notes for the
cross-reference.

### Fourth instance (2026-08-30): COBOL segment-numbered SECTION headers ctags' Cobol parser can't parse

The same "generic parser's grammar assumption is narrower than the real dialect" mechanism,
function-side this time. COBOL-68/74 **program segmentation** lets a section header carry a
segment-priority number between `SECTION` and the terminating period —
`NUMBER1 SECTION 18.`, `SECT-IC219-0001 SECTION 30.` — an overlay-structuring feature for early
mainframes' limited memory, still accepted by modern compilers for legacy support. Confirmed
against `language-crucible` v1.2.0's NIST COBOL-85 CCVS content, ledger shape
`cobol/function/existence/agree[gitgalaxy]_vs[ctags]` (`credit_tools=[gitgalaxy]`).

`ctags -x --language-force=Cobol --kinds-Cobol='*'` emits **zero** tags — neither section (`s`)
nor paragraph (`p`) — for a segment-numbered section header, while tagging a plain
`X SECTION.` on the same run without trouble:
```
$ ctags -x --language-force=Cobol --kinds-Cobol='*' che-che4z_nist_ccvs85/SG4014.2.cbl
CONFIGURATION section 7 ... 000700 CONFIGURATION SECTION.
...
SG401M-DUMMY2 paragraph 31 ... 003100 SG401M-DUMMY2.
# NUMBER1/NUMBER2/NUMBER3 SECTION 18/19. -> no tag at all
```
GitGalaxy's `func_start` matches all three via its `SECTION(?:[ \t\n]+[0-9]{1,2})?` allowance
(added for exactly this feature, epic #813/#854). Confirmed on all 9 real occurrences
(`SG3034.2.cbl`, `SG4014.2.cbl` — `NUMBER1/2/3`; `OBIC24.2.cbl` — `SECT-IC219-0001/0002/0003`),
a whole-corpus check. This is **not** a `CTAGS_FUNC_KINDS` filter issue — that map has carried
section kind `s` since #1891 was fixed (#2121, 2026-08-22); ctags genuinely produces nothing to
filter. Sole explanation for the shape's residual after #2480's mangle fix landed. See
`tests/tools/ctags_reader.py`'s cobol notes.

## Where Claim 12 does NOT apply

- This is specific to a dialect whose real lexical grammar (what counts as a valid identifier)
Expand Down
19 changes: 16 additions & 3 deletions gitgalaxy/standards/language_standards.py
Original file line number Diff line number Diff line change
Expand Up @@ -8905,7 +8905,7 @@ class PrismConfigSchema(TypedDict):
# =====================================================================
# 1. THE HORIZONTAL ANCHOR & FORMAT SHIELD
# Safely handles strict 80-column punched card formats (6-char sequence)
# and modern free-format code. Upgraded to `[ \t\n]*` to allow vertical gaps.
# and modern free-format code.
# The column-7 indicator slot accepts a blank/`-` (continuation), and a
# `D`/`d` debug flag ONLY when a name-char immediately follows it
# (`064100D` + `DEBUG-LINE-TEST-03-A`, otherwise captured as
Expand All @@ -8915,9 +8915,22 @@ class PrismConfigSchema(TypedDict):
# this fix is about a mangled name, not about widening what counts.
# `*`/`/` (comment, page-eject) are excluded -- prism.py strips those
# first, and a commented-out paragraph is dead_code, not a func_start.
# The post-anchor slot is `[ \t]*` -- HORIZONTAL whitespace only, no
# `\n`. #2480: the old `[ \t\n]*` let `^` (re.M) match on a blank line
# that prism.py leaves behind after stripping the `*` comment lines that
# bracket a debug paragraph, then skip forward across the newline INTO
# the real content line -- PAST the col-1-6 sequence-area shield -- so the
# sequence number + col-7 `D` were swept into the captured name
# (`066600DDEBUG-LINE-TEST-05-A` in DB1034.2.cbl:665). Dropping `\n` here
# fixes it with no loss: `^` already re-anchors on the real line under
# re.M, and the one genuine vertical gap (name and its `SECTION` keyword
# on separate physical lines) is consumed by the section-6 lookahead's
# own `[ \t\n]+SECTION`, not here. Re-adding a blank-line consumer was
# tried and rejected -- every variant either regressed corpus output or
# was measurably catastrophic on a whitespace-heavy pathological input.
# Confirmed against language-crucible v1.2.0
# (che-che4z_nist_ccvs85/DB1024.2.cbl:640, DB1034.2.cbl).
r"^(?:[0-9a-zA-Z \t]{6}(?:[ \-]|(?<=[0-9])[Dd](?=[A-Za-z]))?)?[ \t\n]*"
# (che-che4z_nist_ccvs85/DB1024.2.cbl:640, DB1034.2.cbl:665).
r"^(?:[0-9a-zA-Z \t]{6}(?:[ \-]|(?<=[0-9])[Dd](?=[A-Za-z]))?)?[ \t]*"
# 2. THE DATA DIVISION SHIELD
# Explicitly bans data level indicators (01 through 88).
# Prevents massive "01 POLICY." data structures from being hallucinated as paragraphs.
Expand Down
9 changes: 9 additions & 0 deletions tests/core_engine/test_galaxyscope.py
Original file line number Diff line number Diff line change
Expand Up @@ -733,6 +733,15 @@ def test_worker_strips_leading_utf8_bom(self, mock_is_file, MockSecurity, MockDe
self.mock_config["LANGUAGE_DEFINITIONS"] = {"python": {"extensions": [".py"], "rules": {}}}

with tempfile.TemporaryDirectory() as td:
# Resolve the tempdir the way the real orchestrator resolves its
# scan target (`Path(args.target).resolve()`): on macOS `td` is
# under `/var/...` but resolves to `/private/var/...`, and on
# Windows it can come back as an `8.3` short name (`RUNNER~1`) that
# resolves to the long form -- GuideStarLens / ApertureFilter both
# `.resolve()` their own root internally, so an unresolved root_str
# here makes their `path.relative_to(self.root)` raise ValueError
# before the BOM assertions below are ever reached.
td = str(Path(td).resolve())
src = Path(td) / "leading_bom.py"
src.write_bytes(b"\xef\xbb\xbfimport os\nprint('hi')\n")

Expand Down
42 changes: 41 additions & 1 deletion tests/extraction/languages/test_cobol.py
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,38 @@ def test_cobol_func_start_segment_number_regression():
assert m2 and m2.group(1) == "TargetFunc", "plain (non-segmented) section must still match"


def test_cobol_func_start_debug_line_bracketed_by_comments_regression():
"""
Regression test for #2480. In DB1034.2.cbl a debug-line paragraph
(`066600DDEBUG-LINE-TEST-05-A.` -- col-1-6 sequence number, col-7 `D`
debug indicator, name glued on) is bracketed by `*` comment lines.
prism.py blanks those comment lines to EMPTY lines before detector.py
runs func_start, leaving a run of `\\n` in front of the real line.

The old anchor `^(?:<seq-area>)?[ \\t\\n]*` let `^` (re.M) match on one
of those blank lines: the 6-char sequence-area group matched empty
(a `\\n` is not `[0-9a-zA-Z \\t]`), then `[ \\t\\n]*` skipped forward
across the newline onto the real line -- PAST the sequence-area shield
-- so `066600` + `D` were swept into the captured name
(`066600DDEBUG-LINE-TEST-05-A`). Fixed by consuming blank lines with a
leading `(?:[ \\t]*\\n)*` so the shield always re-anchors on the real
content line.
"""
func_start = COBOL_RULES["func_start"]
# leading blank lines == prism's output after it strips the bracketing `*` comments
bracketed = "\n\n\n066600DDEBUG-LINE-TEST-05-A.\n066700D PERFORM FAIL."
m = func_start.search(bracketed)
assert m and m.group(1) == "DEBUG-LINE-TEST-05-A", (
f"#2480: sequence number must not glue onto the name, got {m.group(1) if m else None!r}"
)
# the inline-body sibling shape (never regressed) must still be clean
inline = "063900DDEBUG-LINE-TEST-03A. PERFORM PASS."
m2 = func_start.search(inline)
assert m2 and m2.group(1) == "DEBUG-LINE-TEST-03A", (
f"col-7 D debug indicator regressed, got {m2.group(1) if m2 else None!r}"
)


def test_cobol_func_start_known_limitation_multiline_perform_target_still_matches():
"""
Documents a known, NOT-fixed limitation: a multi-line PERFORM statement
Expand All @@ -126,9 +158,17 @@ def test_cobol_func_start_known_limitation_multiline_perform_target_still_matche


def test_cobol_func_start_redos_immunity():
"""ReDoS sweep for the widened SECTION segment-number lookahead."""
"""ReDoS sweep for the widened SECTION segment-number lookahead and the
#2480 anchor change (`[ \\t\\n]*` -> `[ \\t]*` after the sequence-area
group). Candidates that re-added a blank-line consumer -- a second
`[ \\t]*` after the original `[ \\t\\n]*`, or a leading `(?:[ \\t]*\\n)*`
-- were both measurably catastrophic on a whitespace-heavy input; these
payloads pin that down."""
func_start = COBOL_RULES["func_start"]
assert_redos_immune(func_start, "a" * 200000 + " SECTION 1", timeout_sec=3.0)
assert_redos_immune(func_start, (" " + " " * 40 + "\n") * 5000, timeout_sec=3.0)
assert_redos_immune(func_start, "\n" * 100000 + " " * 100000 + "!", timeout_sec=3.0)
assert_redos_immune(func_start, " " * 200000, timeout_sec=3.0)
assert func_start.search("TargetFunc.")


Expand Down
Loading
Loading