fix(assembly): strip /* */ block comments; re-validate 16-folder tri-comparison corpus

docs/self_scan/tri_comparison_points_of_interest.md

@@ -52,8 +52,8 @@ Sorted 2-vs-1 splits before 3-way splits, unvalidated before validated, biggest
 
 *2-vs-1 -- 1790 occurrences as of 2026-08-31T00:23:46Z*
 
-**Verdict** (by Claude (Sonnet 5), direct investigation via tri-comparison-ledger-sweep, 2026-08-27T00:00:00Z):
-> Re-validated 2026-08-27 (same PR as the mirror shape). All 26 occurrences are ctags' generic Asm parser tagging line-start labels that are not subroutine entries; GitGalaxy correctly excludes every one, confirmed by reading corpus source for each. (1) Data-emission labels: `intro`/`error_message`/`commands`/`int_0x20` in bootos/os.asm (`db`/`dw` string and jump-table data), `A`/`B`/`C` in hellosilicon/matrixmultneon.s (`.short`/`.fill` matrices), `prtstr`/`getcreditcards`/`instr` (`.asciz`/`.ascii` strings). (2) Section / object markers in cosmopolitan/ape.S: `__ro`/`cstr`/`_gdt_end`/`sconf` (`.endobj`), `ape_loader` (`.incbin`), `ape_phdrs`/`ape_macho`/`ape_grub`/`ape_mz`/`apesh` (ELF/Mach-O/Multiboot/shell header data emitted via `.long`/`.ascii`), `_gdtr`/`_gdtrlo` (GDT register values), `ape_idata_idtend`/`ape_idata_iatend` (`.byte` terminators). (3) `.Lenv0`/`.Largv0` -- `.L`-prefixed GCC compiler-local labels `func_start`'s negative lookahead deliberately excludes. The four NASM local-label name-normalization cases from the earlier sample (`load_vec`/`loop`/`empty`/`find`) are gone: the reconciler now pairs them with GitGalaxy's dot-prefixed form. No GitGalaxy recall miss remains in this shape -- the #1949 `_slice_by_labels` cases were fixed and that issue is closed. ctags already pays for these in its own precision denominator (it alone claims them); no credit/debit adjustment applies -- there is no shared-consensus mistake to debit and nothing of GitGalaxy's to credit.
+**Verdict** (by Claude (Sonnet 5), direct investigation via tri-comparison-ledger-sweep, 2026-08-31):
+> Re-validated 2026-08-31 after the corpus grew 3->16 folders (253 new files, PR #13). Same confirmed mechanism as the prior 2026-08-27 verdict (ctags' generic Asm parser tagging line-start tokens that are not subroutine entries; GitGalaxy correctly excludes every one), now confirmed at ~70x the sample size (1791 occurrences vs. 26) across every new folder, plus two NEW manifestations of the SAME underlying weakness the smaller corpus never exercised: (1) MACRO-INVOCATION-KEYWORD mistagging -- ctags tags the LITERAL macro name as if it were a label, not the real subroutine name inside it. Confirmed directly: NASM's own instruction-set self-test suite (nasm_testsuite/avx005.asm and siblings) uses a `%macro x 1+.nolist` wrapper invoked as bare `x OPCODE ...` on hundreds of lines with no colon anywhere -- `ctags -x --language-force=Asm` tags the word `x` as a 'label' up to 192 times in one file (confirmed via raw ctags run); cpm65_6502's `zproc NAME`/`zendproc` procedure- definition macro convention gets the identical treatment (`zproc`/`zendproc` themselves tagged as literal 'label' names, cpm65_6502/apps_devices.asm, 5 occurrences each) as does the `.label NAME` forward-declaration pseudo-op (tags the NAME as if `.label` itself were the definition, duplicating the real `zproc`-declared one). (2) BARE INSTRUCTION MNEMONIC mistagging on FASM-dialect files -- raspberrypi_baremetal's `format`/`include`/`code64` FASM-assembler directives desync ctags' line-start heuristic entirely, tagging ordinary indented instruction lines (`mov`, `add`, `ldr`, `ands`, `mrc`, ...) as repeated 'labels' (confirmed: x86_bare_metal/apm_shutdown.S tags `mov`x6/`int`x3/`xor`x2 with zero real labels present). GitGalaxy correctly excludes all of these -- its func_start regex requires an actual `name:` colon-terminated declaration head, which none of these forms have. No credit/debit: ctags already pays for these in its own precision denominator (it alone claims them); there is no shared-consensus mistake to debit and nothing of GitGalaxy's to credit.
 
 | file | name | GitGalaxy | tree-sitter | ctags |
 |---|---|---|---|---|
@@ -72,8 +72,8 @@ Sorted 2-vs-1 splits before 3-way splits, unvalidated before validated, biggest
 
 *2-vs-1 -- 26 occurrences as of 2026-08-31T00:23:46Z*
 
-**Verdict** (by Claude (Sonnet 5), direct investigation via tri-comparison-ledger-sweep, 2026-08-27T00:00:00Z):
-> Re-validated 2026-08-27; down from 7 occurrences to 2 after two coordinated fixes in the same PR. Four of the original seven (`.load_vec`/`.loop`/`.empty`/`.find`, bootos/os.asm) were never a real disagreement: both tools found the identical label at the identical line, ctags' Asm parser just strips the leading `.` from NASM/GAS local labels while GitGalaxy keeps it verbatim -- `tri_comparison_reconcile.py` now normalizes a single leading dot before pairing, so these register as agreements. One (`ape.mbrpad`, cosmopolitan/ape.S:525) was a genuine GitGalaxy false positive -- a `.org`/`.endobj` MBR-padding object, not a subroutine -- now fixed by a `func_start` negative lookahead that rejects a label followed only by a pure data-emission / location-counter directive (the generic-assembly counterpart to agc_assembly's positive 'opcode must follow' lookahead). The remaining 2 (`.1`, `.2` in bootos/counter.asm:52,67) are real NASM numeric local CODE labels -- `.1:` opens a routine (`int 0x22` ...), `.2:` likewise (`mov [di],ax` ...) -- that Universal Ctags' Asm parser structurally cannot tag (a tag name must start with a letter). GitGalaxy is correct here and ctags has a confirmed structural limitation, not an open question -> credit_tools: [gitgalaxy].
+**Verdict** (by Claude (Sonnet 5), direct investigation via tri-comparison-ledger-sweep, 2026-08-31):
+> Re-validated 2026-08-31 after the corpus grew 3->16 folders; down from 26 to 21 occurrences -- 5 were a real, confirmed GitGalaxy engine defect, now FIXED (not just credited): assembly's own lexical_family ('line_exclusive') only ever recognized `;`/`#` line comments, but `.S` files are routed through the C preprocessor and routinely carry genuine `/* ... */` block comments (BSD/FreeBSD license headers, Emacs modelines, register-usage doc comments) that were never stripped at all -- func_start matched label-shaped text INSIDE the unstripped comment: `Result:` (linux_1_0_kernel/drivers_FPU-emu_reg_u_div.S, a stack-layout doc comment) and `r9:`/`r10:`/`r11:` (freebsd_kernel_arch/amd64_amd64_kexec_tramp.S, a register-usage doc comment). Fixed by adding a new `_strip_asm_block_comments` pre-processing pass in prism.py, run BEFORE `;`/`#` line-stripping (confirmed via direct corpus measurement, not assumed: 121 real `/* ... */` blocks in this corpus contain a bare `;`/`#` internally -- copyright prose, URLs, Emacs modelines -- so line-stripping first would truncate every one of those blocks at its first internal `;`/`#`, corrupting the search for the block's real closing `*/`; the reverse risk, a `;`/`#` comment containing an unclosed `/*`, was checked and found to occur zero times in this corpus). Verified: exactly 5 fewer func_start matches corpus-wide post-fix (1144 -> 1139), matching the confirmed count precisely; both golden masters re-blessed. The remaining 21 are all confirmed GitGalaxy-correct, ctags-structurally-can't, via two established mechanisms plus one newly-confirmed variant of the first: (1) NASM leading-dot local labels (`.loop`/`.find`/`.empty`/`.load_vec`/`.getgot`/`.setcs`/`.no_error`/`.disconnect_error`, bootos/os.asm + nasm_testsuite/aoutso.asm + x86_bare_metal/*, 8 occ) and numeric local labels (`.1`/`.2`, bootos/counter.asm, 2 occ) -- ctags strips the leading dot (normalized by the reconciler) or can't tag a name starting with a digit at all, same precedent as the prior verdict. (2) NEW: GAS's `funcname.localname:` dot-SCOPED local-label convention (NOT leading-dot -- the dot sits in the middle, scoping a jump target to its enclosing routine) -- `do_e820.jmpin`/`do_e820.e820lp`/`do_e820.e820f`/`do_e820.skipent`/`do_e820.notext`/`do_e820.failed` (x86_bare_metal/bios_detect_memory.S, 6 occ) and `seta20.1`/`seta20.2` (xv6_x86_kernel/bootasm.S, 2 occ) -- confirmed via raw ctags run: it tags the real top-level `do_e820` fine but zero of its six dotted sub-labels, a structural inability to parse a `.` inside a label name in this position, not a GitGalaxy over-match (GitGalaxy's own inclusion of these as separate satellites, rather than folding them into the enclosing routine, matches its existing, established policy for NASM leading-dot locals in bucket 1 -- consistent, not a new design question). (3) Real subroutines a `zproc`/custom-macro-heavy file declares via a genuine `name:` label that ctags' 'zproc'/'.label'-mistagging (see the sibling shape's verdict) causes it to miss entirely -- `skip`/`syntax_error`/`write_char`/`io_error`/`justprint` (cpm65_6502/apps_bedit.asm, 6 occ), `longjmp` (cosmopolitan_runtime/longjmp.S -- confirmed real via its `_longjmp:` sibling and `#ifdef __x86_64__` body), `end` (os_tutorial_x86 + x86_bare_metal/bios_pixel*.S, 3 occ). One residual, ACKNOWLEDGED false positive, not chased (single occurrence, narrow root cause): xv6_x86_kernel/usys.S's `name` -- `#define SYSCALL(name) \\n .globl name; \\n name: \\n ...`, a C-preprocessor macro TEMPLATE where `name` is a parameter, not a real label; func_start matches the literal `name:` inside the macro's own body text (a textual regex has no concept of preprocessor parameter substitution). Left in the credited shape's count since it's a single, narrow, low-value edge case (one occurrence in the whole corpus) against 20 confirmed-correct siblings -- documented here rather than silently absorbed into the credit, or chased for a fix disproportionate to its impact.
 
 | file | name | GitGalaxy | tree-sitter | ctags |
 |---|---|---|---|---|

gitgalaxy/core/prism.py

@@ -337,6 +337,9 @@
         elif lang_id == "powershell":
             text, ps_lits = self._strip_powershell_herestrings(text)
             lits.extend(ps_lits)
+        elif lang_id == "assembly":
+            text, asm_lits = self._strip_asm_block_comments(text)
+            lits.extend(asm_lits)
 
         # 2. SPECIALIZED LEXICAL FAMILY ROUTING
         # #386: these three used to check "recursive_c_style"/"column_sensitive"/
@@ -693,6 +696,47 @@
 
         return self._PS_HERESTRING_RE.sub(_repl, text), lits
 
+    def _strip_asm_block_comments(self, text: str) -> tuple[str, list[str]]:
+        """Strips C-style `/* ... */` block comments before assembly's own `line_exclusive`
+        family (`;`/`#` line comments only, per its own docstring: "The language possesses no
+        native multi-line block syntax") ever runs. Real gap, not theoretical: `.S` files are
+        routed through the C preprocessor before assembling, so both GAS and NASM `.S`/`.asm`
+        sources routinely carry genuine `/* */` blocks (BSD/FreeBSD kernel license headers, Emacs
+        modelines, register-usage doc comments) that `line_exclusive` never recognized at all --
+        confirmed real corpus false positives (`func_start` matching label-shaped text INSIDE an
+        unstripped block comment): `Result:` (linux_1_0_kernel/drivers_FPU-emu_reg_u_div.S, a
+        stack-layout doc comment) and `r9:`/`r10:`/`r11:` (freebsd_kernel_arch/amd64_amd64_
+        kexec_tramp.S, a register-usage doc comment), 5 occurrences total across the corpus.
+
+        Runs BEFORE `;`/`#` line-stripping, not after -- deliberately, confirmed by direct corpus
+        measurement rather than assumed either way. 121 real `/* ... */` blocks in this same
+        corpus contain a bare `;` or `#` (copyright-header prose, URLs, Emacs modelines like
+        `/*-*- mode:unix-assembly; indent-tabs-mode:t; ... -*-*/`, and `/* #define ... */`-style
+        commented-out-code notes) -- stripping line comments FIRST would truncate every one of
+        those blocks at its first internal `;`/`#`, corrupting the search for the block's real
+        closing `*/` (silently swallowing everything up to the next UNRELATED `*/` later in the
+        file, or the whole rest of the file if none exists). The reverse direction was also
+        checked and found clean: zero lines in this corpus have a `;`/`#` line comment containing
+        an unclosed `/*` that could similarly mis-pair with a later real `*/`. Same shielded-
+        alternation idiom as the generic REGEX_MATRIX stripper (LITERAL_MASK_PATTERN tried first
+        so a `/*`-shaped byte sequence inside a real string literal, e.g. `.ascii "a /* b"`,
+        passes through unharmed), same non-greedy-bounded-by-two-fixed-delimiters shape already
+        accepted as ReDoS-safe for every "standard_block" C-family language's own `/\\*.*?\\*/`.
+        """
+        lits: list[str] = []
+        pattern = re.compile(rf"{self.LITERAL_MASK_PATTERN}|(/\*.*?\*/)", re.DOTALL)
+
+        def _repl(m: "re.Match[str]") -> str:
+            if m.group(1) is not None:
+                return m.group(0)
+            comment = m.group(2)
+            lits.append(comment.strip())
+            return "\n" * comment.count("\n")
+
+        return pattern.sub(_repl, text), lits
+
+        return self._PS_HERESTRING_RE.sub(_repl, text), lits
+
     _LUA_LONG_BRACKET_RE = re.compile(r"(?:--)?\[(=*)\[.*?\]\1\]", re.DOTALL)
 
     @staticmethod