Skip to content

Decode byte-fallback pieces in metaspace BPE tokenizers - #810

Open
ayushmk7 wants to merge 1 commit into
cactus-compute:mainfrom
ayushmk7:fix/bpe-byte-fallback-decode
Open

ayushmk7 wants to merge 1 commit into
cactus-compute:mainfrom
ayushmk7:fix/bpe-byte-fallback-decode

Conversation

@ayushmk7

Copy link
Copy Markdown
Contributor

Fixes #684

Gemma-style BPE vocabularies with byte_fallback emit <0xHH> pieces for codepoints missing from the vocab. The replace_metaspace path in BPETokenizer::decode appended them verbatim, so output looked like 잠 <0xEA><0xB9><0xB0> 수 있는 instead of 잠 깰 수 있는.

<0xHH> pieces are now mapped back to raw bytes when byte_fallback is enabled, matching what SPTokenizer::decode already does.

Testing

  • Built cactus-engine on macOS arm64.
  • Local harness decoding ▁잠 <0xEA> <0xB9> <0xB0>: returns " 잠깰" with this change (literal <0xEA>… on main); with byte_fallback=false the piece is left as is.

Gemma-style BPE vocabularies with byte_fallback emit <0xHH> pieces for
codepoints missing from the vocab. The replace_metaspace decode path
appended them verbatim, so Korean and Polish output contained literal
<0xEA><0xB9><0xB0> instead of the decoded character.

Map <0xHH> pieces back to raw bytes when byte_fallback is enabled, as
SPTokenizer::decode already does.

Fixes cactus-compute#684

Signed-off-by: ayushmk7 <contactayushmadhav@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gemma byte-fallback tokens are not reassembled into UTF-8

1 participant