Skip to content

fix(cv): frame the SDXS prompt with CLIP's start and end tokens - #1635

Merged
barhanc merged 1 commit into
software-mansion:mainfrom
Rasaboun:fix/sdxs-clip-start-token
Nov 12, 2026
Merged

barhanc merged 1 commit into
software-mansion:mainfrom
Rasaboun:fix/sdxs-clip-start-token

Conversation

@Rasaboun

@Rasaboun Rasaboun commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Description

createSdxsTextToImage sends the prompt to encode without CLIP's <|startoftext|> (49406). Tokenizer.encode adds no special tokens (see #1633), and the exported encode method feeds the ids straight into the CLIP token embedding, so nothing adds it. Prompts longer than 77 tokens were also cut without <|endoftext|>.

This PR builds the ids the way CLIPTokenizer does: <|startoftext|>, at most 75 prompt ids, then <|endoftext|> padding up to 77. Two API tests pin the ids passed to encode.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

  1. yarn workspace react-native-executorch test __tests__/tasks/remainingTasks.test.ts -t createSdxsTextToImage

Related issues

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Additional notes

I checked this against the published sdxs_512_dreamshaper_xnnpack_int8.pte (v0.11.0) with the Python ExecuTorch runtime:

  • In encode, the token embedding reads the input ids directly (same memory as the method input, no concat), so the graph does not add the start token.
  • Compared to the reference CLIP text encoder of IDKiro/sdxs-512-dreamshaper, the encode output has cos 0.92–0.93 with the current ids and cos 1.000 with the fixed ids (4 prompts).
  • The fixed ids are identical to CLIPTokenizer(padding="max_length", max_length=77, truncation=True), including a 120-token prompt.
  • On 16 images (8 prompts × 2 seeds, CLIP ViT-B/32 like the model card), the mean CLIP score goes from 35.03 to 35.46. That's within noise at this sample size, but some images follow the prompt more closely, for example one cat instead of two for "a cat sitting on a red sofa" (top row today, bottom row with the fix):
image

Tokenizer.encode adds no special tokens and the exported encode method feeds the ids straight into CLIP's token embedding, so the text encoder never saw <|startoftext|>, and prompts longer than 77 tokens lost <|endoftext|>. Build the ids like CLIPTokenizer: <|startoftext|>, at most 75 prompt ids, then <|endoftext|> padding up to 77.
Copilot AI balanced review requested due to automatic review settings November 12, 2026 16:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

@barhanc barhanc added the bug fix PRs that are fixing bugs label Nov 12, 2026
@barhanc
barhanc self-requested a review November 12, 2026 16:14

@barhanc barhanc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this fix. LGTM

@barhanc
barhanc merged commit 35458b8 into software-mansion:main Nov 12, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug fix PRs that are fixing bugs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants