Repository navigation
Conversation
The encode JSDoc said special tokens are added according to the tokenizer.json post_processor, but the native binding calls encode(text, 0, 0), so the post_processor template is never applied. Correct the comment and note in the text embeddings custom model tip that a model expecting special tokens must add them inside its exported graph.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
…ith encode The Tokenizers page, the native comment in tokenizer.cpp, the tokenization API test and the NLP demo app all described encode as adding the tokenizer.json post_processor special tokens. The binding passes bos/eos 0, so none are added. Update them to match, and note in the encode JSDoc that truncation and padding from tokenizer.json still apply.
|
I pushed a second commit after a review pass. The Tokenizers page, the native comment in Two questions:
|
|
Thanks for this PR. Regarding your questions:
Yes, for now, let's keep that behavior of
Yes, the versioned docs need this update as well. |
…x docs Both release branches call encode(text, 0, 0) like main, so the versioned Tokenizers page, text embeddings tip and generated Tokenizer API page get the same correction.
Done in f7710ce. I checked that release/0.10 and release/0.11 both call encode(text, 0, 0) so the same fix applies to both. |
## Description `createSdxsTextToImage` sends the prompt to `encode` without CLIP's `<|startoftext|>` (49406). `Tokenizer.encode` adds no special tokens (see #1633), and the exported `encode` method feeds the ids straight into the CLIP token embedding, so nothing adds it. Prompts longer than 77 tokens were also cut without `<|endoftext|>`. This PR builds the ids the way `CLIPTokenizer` does: `<|startoftext|>`, at most 75 prompt ids, then `<|endoftext|>` padding up to 77. Two API tests pin the ids passed to `encode`. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [x] Bug fix (change which fixes an issue) - [ ] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [ ] iOS - [ ] Android ### Testing instructions 1. `yarn workspace react-native-executorch test __tests__/tasks/remainingTasks.test.ts -t createSdxsTextToImage` ### Related issues ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes I checked this against the published `sdxs_512_dreamshaper_xnnpack_int8.pte` (v0.11.0) with the Python ExecuTorch runtime: - In `encode`, the token embedding reads the input ids directly (same memory as the method input, no concat), so the graph does not add the start token. - Compared to the reference CLIP text encoder of `IDKiro/sdxs-512-dreamshaper`, the `encode` output has cos 0.92–0.93 with the current ids and cos 1.000 with the fixed ids (4 prompts). - The fixed ids are identical to `CLIPTokenizer(padding="max_length", max_length=77, truncation=True)`, including a 120-token prompt. - On 16 images (8 prompts × 2 seeds, CLIP ViT-B/32 like the model card), the mean CLIP score goes from 35.03 to 35.46. That's within noise at this sample size, but some images follow the prompt more closely, for example one cat instead of two for "a cat sitting on a red sofa" (top row today, bottom row with the fix): <img width="1024" height="512" alt="image" src="https://github.com/user-attachments/assets/331b21ee-10cd-4b3c-8eba-4c74b84a1fdf" />
Description
The docs say
Tokenizer.encodeadds the special tokens defined by thetokenizer.jsonpost_processor. The native binding callsencode(text, 0, 0), so the post_processor adds none: no[CLS]/[SEP], no BOS/EOS. Thetruncationandpaddingsettings still apply.This PR updates every place that described the old contract:
encodeJSDoc intokenizer.tsencodesection and example ids, custom tokenizer note)kNumAddedBosTokensincpp/extensions/nlp/tokenizer.cpp(comment only, no behavior change)I ran into this with a custom embedding export that expected
<bos>/<eos>and got neither.Introduces a breaking change?
Type of change
Tested on
Testing instructions
yarn workspace react-native-executorch test __tests__/tasks/tokenization.test.tscd docs && yarn build, then open the Tokenizers page, the text embeddings page and the generatedTokenizerAPI page and check the new wording.Related issues
Checklist
Additional notes
I first thought
encodeshould add the special tokens, but the shipped models expect them absent. Running the publishedall_minilm_l6_v2_xnnpack_int8.pteandall_mpnet_base_v2_xnnpack_int8.ptewith the Python ExecuTorch runtime, ids without special tokens match the sentence-transformers reference (cos ≥ 0.998), while ids with them drift (cos down to 0.87). The MiniLM and LFM2.5-Embedding graphs add the tokens themselves (input length + 2 and + 1 before the embedding). So the behavior is right and only the docs needed fixing.The versioned docs for 0.10.x and 0.11.x get the same fix:
release/0.10andrelease/0.11both callencode(text, 0, 0).