Skip to content

Improve Opensearch standard tokenizer accent handling#41

Open
Dtenwolde wants to merge 3 commits into
duckdb:mainfrom
Dtenwolde:opensearch-compat
Open

Improve Opensearch standard tokenizer accent handling#41
Dtenwolde wants to merge 3 commits into
duckdb:mainfrom
Dtenwolde:opensearch-compat

Conversation

@Dtenwolde

Copy link
Copy Markdown
Member

Part of: https://github.com/duckdblabs/duckdb-internal/issues/9855

Couple of points in this PR:

  • Reject strip_accents when the tokenizer mode is set to opensearch_standard
  • Decomposed combining marks are now preserved inside tokens (these would previously cause a split)

A follow-up should work on:

  • Improved word-boundary implementation using the ICU BreakIterator, but at the moment this is not compiled in the ICU extension.
  • Opensearch-style ASCII folding (e.g. Straße --> strasse)
  • Support for max_token_length

If we want even more compatibility, we should also look at the stemmer and stopword list to make it match with OpenSearch/Lucene

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant