Add custom callable and newline sentence tokenizers

[429er]

Summary

This PR adds support for custom callable sentence tokenizers and a built-in newline tokenizer, along with improvements to the pysbd implementation.

Changes

New Features

• Custom callable sentence tokenizers - Users can now pass a custom function to sent_tokenizer parameter that returns sentence offsets as torch.Tensor. This enables domain-specific segmentation (e.g., speaker turns in transcripts, code blocks, etc.)

• Built-in newline tokenizer - New sent_tokenizer="newline" option for line-oriented text like code, transcripts, or structured data. Splits on \n and skips empty lines.

Improvements

• pysbd native char_span - Refactored get_sentence_offsets_pysbd() to use pysbd's native char_span=True feature instead of manually searching for sentences. More reliable, cleaner code, and more accurate offset detection.

• Documentation - Added explicit documentation that sentence offsets are half-open intervals [start, end) that work with Python slicing (text[start:end]).

Tests

• Added comprehensive tests for newline tokenizer (basic usage and empty line handling)
• Added tests for custom callable tokenizers (basic usage and error handling)
• All 219 tests pass

Examples

Custom callable tokenizer:

def speaker_turns(text: str) -> torch.Tensor:
    '''Split transcript by speaker turns.'''
    import re
    pattern = r'(?:^|\n)(?:Speaker [A-Z]:|[\w+]:)'
    matches = list(re.finditer(pattern, text))
    if not matches:
        return torch.tensor([[0, len(text)]]).reshape(-1, 2)
    offsets = []
    for i in range(len(matches)):
        start = matches[i].start()
        end = matches[i+1].start() if i+1 < len(matches) else len(text)
        offsets.append([start, end])
    return torch.tensor(offsets).reshape(-1, 2)

df, embeds = encoder.encode(transcripts, sent_tokenizer=speaker_turns)

Newline tokenizer:

# For code or line-oriented text
df, embeds = encoder.encode(code_snippets, sent_tokenizer="newline")

Related

• Closes issue about custom sentence tokenizers (if one exists)
• Enables use cases like speaker-based chunking, code chunking, and custom domain segmentation