Skip to content

docs(nlp): state that Tokenizer.encode adds no special tokens - #1633

Merged
barhanc merged 3 commits into
software-mansion:mainfrom
Rasaboun:docs/tokenizer-encode-special-tokens
Nov 12, 2026
Merged

barhanc merged 3 commits into
software-mansion:mainfrom
Rasaboun:docs/tokenizer-encode-special-tokens

Conversation

@Rasaboun

@Rasaboun Rasaboun commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Description

The docs say Tokenizer.encode adds the special tokens defined by the tokenizer.json post_processor. The native binding calls encode(text, 0, 0), so the post_processor adds none: no [CLS]/[SEP], no BOS/EOS. The truncation and padding settings still apply.

This PR updates every place that described the old contract:

  • the encode JSDoc in tokenizer.ts
  • the Tokenizers page (intro, encode section and example ids, custom tokenizer note)
  • the "Using Custom Models" tip of the text embeddings page: a model that expects special tokens has to add them inside its exported graph
  • the comment above kNumAddedBosTokens in cpp/extensions/nlp/tokenizer.cpp (comment only, no behavior change)
  • the tokenization API test, which registered a fake tokenizer that added BOS/EOS and asserted it
  • the comment and sample index in the NLP demo app's tokenizer screen

I ran into this with a custom embedding export that expected <bos>/<eos> and got neither.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

  1. yarn workspace react-native-executorch test __tests__/tasks/tokenization.test.ts
  2. cd docs && yarn build, then open the Tokenizers page, the text embeddings page and the generated Tokenizer API page and check the new wording.

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

I first thought encode should add the special tokens, but the shipped models expect them absent. Running the published all_minilm_l6_v2_xnnpack_int8.pte and all_mpnet_base_v2_xnnpack_int8.pte with the Python ExecuTorch runtime, ids without special tokens match the sentence-transformers reference (cos ≥ 0.998), while ids with them drift (cos down to 0.87). The MiniLM and LFM2.5-Embedding graphs add the tokens themselves (input length + 2 and + 1 before the embedding). So the behavior is right and only the docs needed fixing.

The versioned docs for 0.10.x and 0.11.x get the same fix: release/0.10 and release/0.11 both call encode(text, 0, 0).

The encode JSDoc said special tokens are added according to the tokenizer.json post_processor, but the native binding calls encode(text, 0, 0), so the post_processor template is never applied. Correct the comment and note in the text embeddings custom model tip that a model expecting special tokens must add them inside its exported graph.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 14:14

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

…ith encode

The Tokenizers page, the native comment in tokenizer.cpp, the tokenization API test and the NLP demo app all described encode as adding the tokenizer.json post_processor special tokens. The binding passes bos/eos 0, so none are added. Update them to match, and note in the encode JSDoc that truncation and padding from tokenizer.json still apply.
@Rasaboun

Copy link
Copy Markdown
Contributor Author

I pushed a second commit after a review pass. The Tokenizers page, the native comment in tokenizer.cpp, the tokenization API test and the NLP demo app all described encode as adding the post_processor special tokens, so I updated them too.

Two questions:

  • The old comment in tokenizer.cpp said that passing 0 lets tokenizer.json add the special tokens. In practice 0 turns them off, and the shipped models depend on that: their graphs add [CLS]/[SEP] (or BOS) themselves. So I documented that encode adds none. Can you confirm that's the intended behavior, rather than making encode add them?
  • The 0.10.x and 0.11.x versioned docs have the same wrong sentence, and 0.11.x is the default version on the site. Should I update them in this PR too?

@barhanc barhanc added the documentation Improvements or additions to documentation label Nov 12, 2026
@barhanc
barhanc self-requested a review November 12, 2026 15:23
@barhanc

barhanc commented Nov 12, 2026

Copy link
Copy Markdown
Member

Thanks for this PR. Regarding your questions:

  • The old comment in tokenizer.cpp said that passing 0 lets tokenizer.json add the special tokens. In practice 0 turns them off, and the shipped models depend on that: their graphs add [CLS]/[SEP] (or BOS) themselves. So I documented that encode adds none. Can you confirm that's the intended behavior, rather than making encode add them?

Yes, for now, let's keep that behavior of encode.

  • The 0.10.x and 0.11.x versioned docs have the same wrong sentence, and 0.11.x is the default version on the site. Should I update them in this PR too?

Yes, the versioned docs need this update as well.

…x docs

Both release branches call encode(text, 0, 0) like main, so the versioned Tokenizers page, text embeddings tip and generated Tokenizer API page get the same correction.
@Rasaboun

Copy link
Copy Markdown
Contributor Author

Thanks for this PR. Regarding your questions:

  • The old comment in tokenizer.cpp said that passing 0 lets tokenizer.json add the special tokens. In practice 0 turns them off, and the shipped models depend on that: their graphs add [CLS]/[SEP] (or BOS) themselves. So I documented that encode adds none. Can you confirm that's the intended behavior, rather than making encode add them?

Yes, for now, let's keep that behavior of encode.

  • The 0.10.x and 0.11.x versioned docs have the same wrong sentence, and 0.11.x is the default version on the site. Should I update them in this PR too?

Yes, the versioned docs need this update as well.

Done in f7710ce. I checked that release/0.10 and release/0.11 both call encode(text, 0, 0) so the same fix applies to both.

@barhanc
barhanc merged commit 3937bbc into software-mansion:main Nov 12, 2026
8 checks passed
barhanc pushed a commit that referenced this pull request Nov 12, 2026
## Description

`createSdxsTextToImage` sends the prompt to `encode` without CLIP's
`<|startoftext|>` (49406). `Tokenizer.encode` adds no special tokens
(see #1633), and the exported `encode` method feeds the ids straight
into the CLIP token embedding, so nothing adds it. Prompts longer than
77 tokens were also cut without `<|endoftext|>`.

This PR builds the ids the way `CLIPTokenizer` does: `<|startoftext|>`,
at most 75 prompt ids, then `<|endoftext|>` padding up to 77. Two API
tests pin the ids passed to `encode`.

### Introduces a breaking change?

- [ ] Yes
- [x] No

### Type of change

- [x] Bug fix (change which fixes an issue)
- [ ] New feature (change which adds functionality)
- [ ] Documentation update (improves or adds clarity to existing
documentation)
- [ ] Other (chores, tests, code style improvements etc.)

### Tested on

- [ ] iOS
- [ ] Android

### Testing instructions

1. `yarn workspace react-native-executorch test
__tests__/tasks/remainingTasks.test.ts -t createSdxsTextToImage`

### Related issues

### Checklist

- [x] I have performed a self-review of my code
- [x] I have commented my code, particularly in hard-to-understand areas
- [ ] I have updated the documentation accordingly
- [x] My changes generate no new warnings

### Additional notes

I checked this against the published
`sdxs_512_dreamshaper_xnnpack_int8.pte` (v0.11.0) with the Python
ExecuTorch runtime:
- In `encode`, the token embedding reads the input ids directly (same
memory as the method input, no concat), so the graph does not add the
start token.
- Compared to the reference CLIP text encoder of
`IDKiro/sdxs-512-dreamshaper`, the `encode` output has cos 0.92–0.93
with the current ids and cos 1.000 with the fixed ids (4 prompts).
- The fixed ids are identical to `CLIPTokenizer(padding="max_length",
max_length=77, truncation=True)`, including a 120-token prompt.
- On 16 images (8 prompts × 2 seeds, CLIP ViT-B/32 like the model card),
the mean CLIP score goes from 35.03 to 35.46. That's within noise at
this sample size, but some images follow the prompt more closely, for
example one cat instead of two for "a cat sitting on a red sofa" (top
row today, bottom row with the fix):

<img width="1024" height="512" alt="image"
src="https://github.com/user-attachments/assets/331b21ee-10cd-4b3c-8eba-4c74b84a1fdf"
/>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants