Skip to content

Give tokens and token exceptions a line and column - #154

Merged
chrisjenkinson merged 2 commits into
masterfrom
claude/token-positions
Sep 25, 2026
Merged

chrisjenkinson merged 2 commits into
masterfrom
claude/token-positions

Conversation

@chrisjenkinson

@chrisjenkinson chrisjenkinson commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Token::getPosition() always threw a TypeError because nothing set a position. This implements it and uses the position in the lexer's exceptions:

  • Cursor tracks the line and column as it advances, both 1-based, with columns counted in characters. Token takes its TokenPosition in the constructor, and setPosition() is gone.
  • NoTokenFoundException and AmbiguousTokenFoundException messages show the line, column and the rest of the line (capped at 100 characters) instead of all the remaining text. The full text is still available through the getters.

Breaking changes: the Token, NoTokenFoundException and AmbiguousTokenFoundException constructors now require a TokenPosition, and guardAgainstWrongNumberOfMatches() takes one too.

Base automatically changed from claude/matching-and-tokens to master September 25, 2026 09:33
@chrisjenkinson
chrisjenkinson merged commit 0805780 into master Sep 25, 2026
8 checks passed
@chrisjenkinson
chrisjenkinson deleted the claude/token-positions branch September 25, 2026 09:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant