Skip to content

Feat/metadata search with postgres fts - #310

Merged
aaronjae22 merged 10 commits into
mainfrom
feat/metadata-search-with-postgres-fts
Jul 22, 2026
Merged

Feat/metadata search with postgres fts#310
aaronjae22 merged 10 commits into
mainfrom
feat/metadata-search-with-postgres-fts

Conversation

@aaronjae22

@aaronjae22 aaronjae22 commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

This PR adds Metadata search with Postgres FTS.

Closes #295

The new search method rank schemas by full-text relevance against query_text, with name weighted above description. When query_text is blank the queryset is returned unchanged, so the caller's existing ordering (e.g. alphabetical on the index page) is preserved for plain browsing.

class Schema(BaseModel):
    ...
    ...
    ...
    search_vector = models.GeneratedField(
        expression=(
            SearchVector("name", weight="A", config="english")
            + SearchVector("description", weight="B", config="english")
        ),
        output_field=SearchVectorField(),
        db_persist=True,
    )

    class Meta:
        indexes = [
            models.Index(fields=["published_at"]),
            GinIndex(fields=["search_vector"], name="schema_search_vec_gin"),
        ]

GeneratedField is a database-computed column. Postgres itself calculates the value from the expression. It makes no use of triggers, no manual .update(), no overriding save(). The database keeps it in sync automatically whenever name or description changes.

expression is the full-text search document. In this case just concatenates two SearchVectors, name with weight='A' and description with weight=B. So a match in the name ranks higher than a match in the description.

config="english" appliies English stemming/stop-words rules.

output_field=SearchVectorField() tells Django the generated column's type is a tsvector.

db_persist=True means the value is stored on disk (Postgres STORED generated column), not recomputed on every read. This is required for Postgres, and it’s what lets the GIN index work.


Row name / description Stored search_vector
A Invoice Schema / A directory of Schemas 'directori':4B 'invoic':1A 'schema':2A,6B
B ACME Format / None 'acm':1A 'format':2A
  • ’invoic':1A — "Invoice" stemmed, position 1, weight A (from name).
  • 'schema':2A,6B — this one entry is the whole feature. The stem schema appears at position 2 weight A (from name "Invoice Schema") and position 6 weight B (from description "...Schemas"). Singular and plural collapsed into one lexeme, with both origins retained.
  • 'directori':4B — "directory" stemmed, weight B.
  • Stop-words (A, of) are dropped, which is why positions skip.
  • Row B proves NULL is safe: a None description contributed nothing and the vector is still valid.

@aaronjae22 aaronjae22 self-assigned this Jul 21, 2026
@aaronjae22

Copy link
Copy Markdown
Collaborator Author

@alexbainter can you check this out when you have some time

@alexbainter alexbainter left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Had one comment about a test that looks like it might be missing a couple assertions, but not a big deal. Looking forward to using this for #293!

Comment thread core/models.py


class PublicSchemaManager(models.Manager):
class SchemaQuerySet(models.QuerySet):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nice! I introduced a SchemaQuerySet on my branch for #293 too.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

Comment thread tests/test_search.py Outdated
Comment on lines +42 to +48
@pytest.mark.django_db
def test_blank_query_returns_all_without_ranking():
SchemaFactory(name="One")
SchemaFactory(name="Two")
# Blank / None must not filter or reorder — plain browsing is preserved.
assert Schema.public_objects.search("").count() == 2
assert Schema.public_objects.search(None).count() == 2

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test title and comment seem to imply this should also verify the order of the results. Did you mean to do that too?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I completely agree with you. I modified the test in order to check the plain browsing order in which they should get returned.

Thanks for checking that out.

@aaronjae22

aaronjae22 commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator Author

I updated the test as you suggested and I agreed with it.

This update should mirror what happens in our stg/prod in which the index view builds its base queryset with .order_by("name") before calling .search(), so an empty search box on the homepage browses schemas alphabetically, and only when we type something does .search() override that with .order_by("-rank", "name").

Thanks for noticing it. I'll move forward and merge this.

@aaronjae22
aaronjae22 merged commit 01cbbec into main Jul 22, 2026
11 checks passed
@aaronjae22
aaronjae22 deleted the feat/metadata-search-with-postgres-fts branch July 22, 2026 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Metadata Search Implementation with Postgres FTS

2 participants