feat: add TokenChunker for token-level chunking via tiktoken (closes #44)

[prashantshukla01]

Description

Adds TokenChunker to split text into chunks measured in tokens using tiktoken. This ensures chunks align directly with embedding-model and LLM token limits instead of relying on character counts.

Closes #44

Changes

• Implemented TokenChunker subclassing TextChunker in ragframework/document/chunkers.py with sliding window and token overlap.
• Added guarded import for tiktoken with installation instructions.
• Stored token_count and chunk_index in chunk metadata following existing conventions.
• Added tokens = ["tiktoken>=0.7"] optional dependency and included it in the all extra in pyproject.toml.
• Exported TokenChunker in ragframework/document/__init__.py.
• Added unit tests in tests/test_document/test_chunkers.py using a fake tiktoken module to avoid network downloads in CI.
• Updated CHANGELOG.md under [Unreleased].

Type of change

• Bug fix
• New feature / integration
• Documentation update
• Refactor / code quality

Checklist

• pytest tests/ -v passes locally
• ruff check ragframework/ passes
• mypy ragframework/ passes
• New or updated tests cover the changes
• Optional dependencies are guarded with a helpful import error
• Docstrings updated where applicable
• CHANGELOG.md updated under [Unreleased]

[adaumsilva]

Thanks for adding TokenChunker @prashantshukla01 . The existing tests and Ruff pass, but testing with real tiktoken revealed two issues:

1. Unicode text can be corrupted. Decoding an arbitrary token window can split a character’s UTF-8 bytes. With chunk_tokens=1 and overlap_tokens=0, "🙂" becomes chunks containing "�" and "��". Please preserve complete characters at chunk boundaries while respecting the token limit. If a character cannot fit, raise a clear error rather than silently corrupting it. Add emoji and CJK regression tests.

2. Special-token strings cause valid document text to fail. Encoding "Hello <|endoftext|> world" raises ValueError. Please treat these strings as ordinary text, for example with encode(document.content, disallowed_special=()), and add a regression test.

Please ensure the tests cover these tokenizer behaviors; the current ASCII-only fake-tokenizer cases miss them.

Also resolve the changelog conflict while preserving existing entries. I’ll handle the broader changelog cleanup afterward.

Please request another review once the fixes are pushed and CI passes.

[adaumsilva]

Thanks for contributing @prashantshukla01 I requestes a few changes before merging the PR. Please, address these changes.

[prashantshukla01]

Hi @adaumsilva, I've pushed the fixes addressing your feedback:

1. UTF-8 Byte Boundaries: TokenChunker now decodes raw bytes and steps backward to ensure chunk boundaries never split multi-byte characters (emojis, CJK). If a single character requires more tokens than chunk_tokens, it raises a clear ValueError.
2. Special Tokens: Added disallowed_special=() so strings like <|endoftext|> are treated as ordinary text.
3. Tests & Fake Tokenizer: Updated FakeEncoding to simulate byte-level tokens, special token checks, and decode_bytes(). Added regression tests for emojis, CJK text, special tokens, and token limit validation.
4. Changelog: Resolved the merge conflict in CHANGELOG.md while keeping all upstream entries.

All unit tests (41/41), ruff, and mypy pass cleanly locally. Ready for another review!

[adaumsilva]

Thanks for addressing the requested changes! I verified with real tiktoken that Unicode characters remain intact, special-token strings are treated as ordinary text, and an insufficient token budget raises a clear error.

All 41 chunker tests pass locally, Ruff and mypy for the chunker module pass, and all GitHub checks are green. The changelog update also preserves existing entries.

No blocking issues remain. Approved—thanks for contributing TokenChunker!

[adaumsilva]

Great work! Thanks for contributing @prashantshukla01 PR Merged! Remember to star the repo if you didn't yet.