Repository navigation
Akejzman/various speedups - #74
Merged
Merged
Conversation
AlonKejzman
force-pushed
the
akejzman/various-speedups
branch
from
September 30, 2026 13:45
84bca93 to
0325d99
Compare
AlonKejzman
added a commit
to GuyStone/fastokens
that referenced
this pull request
Oct 8, 2026
Rebasing onto main brings in `Normalizer::is_identity_on` (crusoecloud#74), whose match did not cover the new `Lowercase` variant, so the crate no longer compiled. `Lowercase` is the identity exactly when `first_change` finds nothing to lower, which also holds for every substring, so expose that scan as `Lowercase::is_normalized` (as `Nfc::is_normalized`) and use it: already lowercase text keeps the in-place encode path instead of being copied through normalization. The HF parity test now also asserts that `is_identity_on` agrees with whether HF's Lowercase changed the text, since the encode path skips normalization when it says so. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
AlonKejzman
added a commit
that referenced
this pull request
Oct 8, 2026
* normalizers: implement the Lowercase normalizer
`NormalizerConfig::Lowercase` is already parsed but has no runtime
implementation, so tokenizers declaring `{"type": "Lowercase"}` fail to load
with "unsupported normalizer type: Lowercase". Add the Lowercase normalizer
and wire it into `Normalizer`. This unlocks standalone `Lowercase` and the
CLIP / sentence-transformers shape `Sequence[NFC, Replace, Lowercase]`;
uncased BERT still needs `NFD` + `StripAccents`.
Lowercasing is done per character with `char::to_lowercase`, mirroring HF's
`NormalizedString::lowercase`. This is deliberately not `str::to_lowercase`:
the std method additionally applies the Greek final-sigma rule, rendering
`ὈΔΥΣΣΕΎΣ` as `ὀδυσσεύς` where HF produces `ὀδυσσεύσ`. U+03A3 is the only
character the two disagree on. The "would this character change?" predicate
compares against the mapping itself rather than `char::is_uppercase()`, which
is false for titlecase letters such as `Dž` U+01C5 and would produce wrong
output.
Normalization runs single-threaded over the whole document before rayon
parallelism starts at BPE, so both the scan and the transform take a SWAR fast
path over ASCII, mirroring `ascii_lower_run_end` in `pre_tokenizers::scan`. The
transform is branch-free: `0x80 >> 2 == 0x20` is exactly the ASCII case bit.
Measured against a straightforward `chars().flat_map(char::to_lowercase)` on
~100 KiB inputs: 68x faster on already-lowercase ASCII, 45x on mixed case and
on ALL CAPS, and 0.95x on CJK (both dominated by std's LOWERCASE_TABLE binary
search, which HF pays too).
Tests cover the Cow::Borrowed guarantee, a capital at every one of 28 byte
offsets to pin the SWAR lane indexing and window boundary, titlecase letters,
the 1->2 expansion of `İ` U+0130, and final sigma — the last asserting
inequality with `str::to_lowercase` so a refactor towards it fails loudly.
`local_tests::lowercase_normalizer_matches_huggingface` diffs against the
`tokenizers` crate directly.
Also re-exports `Prepend` at the crate root, which was missed when it landed
in edccd31 (`Nfc` and `Replace` are both there).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* normalizers: cover Lowercase in is_identity_on
Rebasing onto main brings in `Normalizer::is_identity_on` (#74), whose match
did not cover the new `Lowercase` variant, so the crate no longer compiled.
`Lowercase` is the identity exactly when `first_change` finds nothing to
lower, which also holds for every substring, so expose that scan as
`Lowercase::is_normalized` (as `Nfc::is_normalized`) and use it: already
lowercase text keeps the in-place encode path instead of being copied
through normalization.
The HF parity test now also asserts that `is_identity_on` agrees with
whether HF's Lowercase changed the text, since the encode path skips
normalization when it says so.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Alon Kejzman <alonkeizman@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.