I implement a custom Byte-Pair Encoding (BPE) tokenizer that relies on two core asset files: vocab.json and merges.txt. Together, these files allow my tokenizer to break arbitrary, out-of-vocabulary strings into meaningful sub-word tokens.
vocab.json(The Token Dictionary): Maps every recognized sub-word token to a unique integer ID.merges.txt(The Merge Rulebook): Contains an ordered list of rank-prioritized character pair merges. It teaches the tokenizer how to construct sub-words from individual characters step-by-step.
Without merges.txt, my tokenizer would fail to parse unseen words and fall back to unknown (<unk>) tokens. With merges.txt, it iteratively combines character pairs until every segment maps to a valid entry in vocab.json.
When my custom bpe_tokenize function encounters an unseen word like "tokenization", it executes the following sequence:
- Initial Character Splitting:
The raw string is split into individual character tokens:
['t', 'o', 'k', 'e', 'n', 'i', 'z', 'a', 't', 'i', 'o', 'n'] - Sequential Merge Application:
The tokenizer reads
merges.txtand applies matching rules in strict order of their priority rank:
- Rule
#1(t+o$\rightarrow$ to):['to', 'k', 'e', 'n', 'i', 'z', 'a', 't', 'i', 'o', 'n'] - Rule
#50(a+t$\rightarrow$ at):['to', 'k', 'e', 'n', 'i', 'z', 'at', 'i', 'o', 'n'] - Higher-ranked rules merge remaining pairs (
i+on$\rightarrow$ ion,at+ion$\rightarrow$ ation).
- Final Sub-word Output:
The process stops when no further merges from
merges.txtcan be applied, yielding known sub-word tokens:['token', 'ization'] - Vocabulary Lookup:
My implementation maps these final sub-words directly to integer IDs via
vocab.jsonfor model inference.
vocab.jsondefines the valid terminal states (the final set of available IDs).merges.txtdefines the deterministic path used to reach those terminal states from raw character inputs.