Search results

Add support for actively maintained sentence tokenizers (spaCy, Stanza, wtpsplit)

enhancementNew feature or requestNew feature or request

Status: Open.

#15 In 429er/afterthoughts;

· 429er opened on Feb 2, 2026

Add EntityEncoder for entity-aware late chunking

enhancementNew feature or requestNew feature or request

Status: Open.

#14 In 429er/afterthoughts;

· 429er opened on Feb 2, 2026

Replace BlingFire as default sentence tokenizer due to lack of maintenance

enhancementNew feature or requestNew feature or request

Status: Open.

#13 In 429er/afterthoughts;

· 429er opened on Feb 2, 2026

Set up ReadTheDocs documentation site

documentationImprovements or additions to documentationImprovements or additions to documentation

Status: Open.

#12 In 429er/afterthoughts;

· 429er opened on Feb 1, 2026

Add get_chunks() method for dry-run chunk preview without encoding

enhancementNew feature or requestNew feature or request

Status: Open.

#10 In 429er/afterthoughts;

· 429er opened on Feb 1, 2026

Add structure-aware sentence tokenizers (markdown, Python code, HTML)

enhancementNew feature or requestNew feature or request

Status: Open.

#9 In 429er/afterthoughts;

· 429er opened on Feb 1, 2026