Fix extension grammar: multi-char operators (=>, ->, …) highlight as one token#31
Merged
Merged
Conversation
TextMate applies, at each position, the first pattern in a rule list that matches there — ties at the same start are decided by list order, not match length. Make every multi-character operator (=> -> := |> <- :: == != <= >= && ||) win over its single-character prefixes by ordering the multi-char operator rules ahead of the single-char ones, and split the logical rule so the two-char `&&`/`||` sit in the multi-char tier while single-char `!` stays in the single-char tier. Each multi-char operator keeps exactly one scope; `<<`/`>>` remain single import/export tokens. Add grammar tokenization tests (src/grammar.test.ts) backed by a tiny dependency-free TextMate match-engine (src/grammar.ts) that reproduces the ordered first-match-wins rule, so this can't regress. The tests assert each multi-char operator tokenizes to a single consistent scope (spaced and tight), plus regression guards for `<`/`>`, `=`, `$`, comments, strings, and numbers. Document the single-token behavior and the grammar tests in the extension README. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Post-merge fixup after integrating the pnpm migration and CodeLens work: the test list now names all three Node-testable modules (diagnostics, entry-point detector, grammar tokenization) and uses `pnpm test`. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
In the Quilon VS Code grammar, multi-character operators risked being highlighted as two tokens (e.g.
=>could color the=in one scope and the>in another), the same for->,:=,|>,<-,==,!=,<=,>=,&&,||,::. TextMate applies, at each position, the first pattern in a rule list that matches there — ties at the same start position are decided by list order, not match length — so a single-character operator rule ordered ahead of a multi-character one would split the operator.Fix
In
syntaxes/quilon.tmLanguage.json, make every multi-character operator win as a single token:|>/||before|,<=before<,=>/==before=, etc.). Each multi-char operator keeps exactly one scope.&&|\|\||!rule so the two-char&&/||sit in the multi-char tier while the single-char!stays in the single-char tier (keeps!=correct).<</>>remain single import/export tokens (unchanged module-line rules).$(unit),~comments,</>(comparison + block delimiters), strings, or numbers.Tests (mandatory)
Added
src/grammar.test.ts, backed by a tiny dependency-free TextMate match-engine (src/grammar.ts) that reproduces the ordered first-match-wins rule (no nativevscode-textmate/vscode-onigurumadependency, so it runs under the existingnode --testgate). It asserts:=>,->,:=,|>,<-,::,==,!=,<=,>=,&&,||) tokenizes to a single, consistent scope — both spaced (a => b) and tight (a=>b);<</>>stay single tokens; adjacent operators (a==b!=c<=d>=e) each stay one token; a representative lambda + arrow-type line highlights each operator once;</>,=,$, comments, strings, and numbers.Verified the test is meaningful: with the buggy (single-char-first) ordering, 19 of these assertions fail; with the fix they all pass.
Updated the extension README to document the single-token behavior and the new grammar tests.
Verification
pnpm install --frozen-lockfile, thenpnpm run lint,pnpm run fmt:check,pnpm test(51 passing), andpnpm run package(.vsix builds) — all green. Scope is limited to the grammar JSON, the grammar tests, and the README; nopackage.json/lockfile/.npmrc/workflow/Rust changes.🤖 Generated with Claude Code