Skip to content

fix(neutrality): span-based allow-phrase exemption (from PR #3) - #9

Merged
StevenMih merged 1 commit into
mainfrom
fix/neutrality-scan-span
Jul 29, 2026
Merged

fix(neutrality): span-based allow-phrase exemption (from PR #3)#9
StevenMih merged 1 commit into
mainfrom
fix/neutrality-scan-span

Conversation

@StevenMih

Copy link
Copy Markdown
Contributor

Summary

  • The `allow_phrases` exemption in `neutrality_scan.py` was over-exempting: any occurrence of a reserved token on a line was exempt if ANY allow-phrase carrying that token also appeared on the same line.
  • Fix: track allow-phrase character spans `[start, end)` over the lowercased line; a vocabulary hit is exempt ONLY when its span falls inside an allow-phrase span.
  • Two occurrences of the same token on one line are now handled correctly: the one inside the phrase is exempt, the one outside is flagged.

Changes

  • `neutrality_scan.py`: span-based `_line_offenders`, new `_git_tracked_files` helper (scan only git-tracked files), `_run_self_tests` / `--self-test` flag with 3-case test, `.html` added to SCAN_SUFFIXES
  • `neutrality.yml`: adds "Neutrality scanner self-test" step before the vocabulary scan

Test plan

  • `python .github/neutrality_scan.py --self-test` → `neutrality self-test: OK (span-based allow-phrase exemption)` (all 3 test cases pass)
  • CI neutrality job passes on merge

Propagated from scitt-payload-binding PR #3 Fix 4.

🤖 Generated with Claude Code

The allow-phrase exemption was over-exempting: any occurrence of a
token on a line was exempt if ANY allow-phrase carrying that token
also appeared on the line.

Fix: track allow-phrase character spans over the lowercased line; a
vocabulary hit is exempt ONLY when its span falls inside an allow-phrase
span.  Two occurrences of the same token on one line are now handled
correctly: the one inside the phrase is exempt, the one outside is
flagged.

Also adds: _run_self_tests / --self-test flag with three-case test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: stevenmih <stevenmih88-3@gmail.com>
@StevenMih
StevenMih force-pushed the fix/neutrality-scan-span branch from 6a98220 to 6ec1f45 Compare July 29, 2026 04:15
@StevenMih
StevenMih merged commit 0e0d3a7 into main Jul 29, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant