Skip to content

Support FTS and bulk loading indices#1

Merged
gkennos merged 8 commits intomainfrom
fts
Mar 23, 2026
Merged

Support FTS and bulk loading indices#1
gkennos merged 8 commits intomainfrom
fts

Conversation

@nicoloesch
Copy link
Copy Markdown
Collaborator

Changes in this commit

  • Create FTS indices
  • Support other SQLAlchemy indices
  • Support bulk loading of indices after table creation.

Rationale
The changes include optimisations to support a wide variety of indices. Particularly, the indices can be set in the SQL ORM table, but the respective staging table will not include them as they are computed.
This has several advantages:

  1. Loading the tables from .csv doesn't cause warnings as the newly created index column isn't populated
  2. Computing the indices after bulk ingestion has the advantage that each index is not computed per row that is ingested. This results in a detrimental increase in performance, particularly for FTS indices.

@nicoloesch nicoloesch requested a review from gkennos March 20, 2026 05:08
Copy link
Copy Markdown
Contributor

@gkennos gkennos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed

@gkennos gkennos merged commit 1c1df20 into main Mar 23, 2026
3 checks passed
@gkennos gkennos deleted the fts branch March 23, 2026 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants