Repository navigation
Conversation
…ole (hypit-ai#391) The number alternative of LEXICAL_UNIT only took numeric hyphenated suffixes, and its lookahead rejects stopping before -letters, so a comma group followed by a word fell back to the bare prefix and split at the comma: 1,000-word became 1, and 000-word, which also let a Cue break between them through (SCRIPT_CAPTION_BREAK_TOKEN_BOUNDARY never fired). After its numeric suffixes the number alternative now takes one hyphenated word suffix, with the ' ’ . - joins the plain-word alternative allows. The joins matter: the number alternative comes first, so it now reaches compounds the word alternative used to own, and without them 90-year-old's or 2020-word.com would be cut at the apostrophe or dot. 1,000-word, 123,456,789-point and 1,000.5-word each stay one unit, every un-grouped compound tokenizes as before, and the regression test pins both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SanHsien
added a commit
to SanHsien/hypit
that referenced
this pull request
Oct 25, 2026
…advance the PR watermark Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ole (#391)
The number alternative of LEXICAL_UNIT only took numeric hyphenated suffixes, and its lookahead rejects stopping before -letters, so a comma group followed by a word fell back to the bare prefix and split at the comma: 1,000-word became 1, and 000-word, which also let a Cue break between them through (SCRIPT_CAPTION_BREAK_TOKEN_BOUNDARY never fired).
After its numeric suffixes the number alternative now takes one hyphenated word suffix, with the ' ’ . - joins the plain-word alternative allows. The joins matter: the number alternative comes first, so it now reaches compounds the word alternative used to own, and without them 90-year-old's or 2020-word.com would be cut at the apostrophe or dot. 1,000-word, 123,456,789-point and 1,000.5-word each stay one unit, every un-grouped compound tokenizes as before, and the regression test pins both.