Skip to content

Fix SP vocab loading for tokens containing tabs - #809

Open
ayushmk7 wants to merge 1 commit into
cactus-compute:mainfrom
ayushmk7:fix/sp-vocab-tab-tokens
Open

ayushmk7 wants to merge 1 commit into
cactus-compute:mainfrom
ayushmk7:fix/sp-vocab-tab-tokens

Conversation

@ayushmk7

Copy link
Copy Markdown
Contributor

Fixes #577

vocab.txt is written as ID\ttoken or ID\ttoken\tscore. The loader split on the first tab inside the token, so Gemma tokens like "\t\t\t" were read as a score: either std::stof throws during init, or the token is silently truncated.

The scored format is now detected from the first line, and the score is split off the last tab only when scores are present. Unscored files never parse a score, so tokens such as "\t1" stay intact.

Testing

  • Built cactus-engine on macOS arm64.
  • Local harness loading scored and unscored vocab files with tab-containing tokens: fails on main, passes with this change.

vocab.txt is written as ID<TAB>token or ID<TAB>token<TAB>score. The loader
split on the first tab inside the token, so Gemma tokens such as "\t\t\t"
were parsed as a score and init failed with a std::stof exception.

Detect the scored format from the first line and split the score on the
last tab only when scores are present.

Fixes cactus-compute#577

Signed-off-by: ayushmk7 <contactayushmadhav@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

f8afc46 regresses Gemma tokenizer loading: tab tokens in vocab.txt cause std::stof exception during init

1 participant