diff --git a/docs-website/reference_versioned_docs/version-3.2-unstable/integrations-api/pgvector.md b/docs-website/reference_versioned_docs/version-3.2-unstable/integrations-api/pgvector.md new file mode 100644 index 0000000000..cbad50a83e --- /dev/null +++ b/docs-website/reference_versioned_docs/version-3.2-unstable/integrations-api/pgvector.md @@ -0,0 +1,962 @@ +--- +title: "Pgvector" +id: integrations-pgvector +description: "Pgvector integration for Haystack" +slug: "/integrations-pgvector" +--- + + +## haystack_integrations.components.retrievers.pgvector.embedding_retriever + +### PgvectorEmbeddingRetriever + +Retrieves documents from the `PgvectorDocumentStore`, based on their dense embeddings. + +Example usage: + +```python +from haystack.document_stores import DuplicatePolicy +from haystack import Document, Pipeline +# Requires: pip install sentence-transformers-haystack +from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersTextEmbedder +from haystack_integrations.components.embedders.sentence_transformers import SentenceTransformersDocumentEmbedder + +from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore +from haystack_integrations.components.retrievers.pgvector import PgvectorEmbeddingRetriever + +# Set an environment variable `PG_CONN_STR` with the connection string to your PostgreSQL database. +# e.g., "postgresql://USER:PASSWORD@HOST:PORT/DB_NAME" + +document_store = PgvectorDocumentStore( + embedding_dimension=768, + vector_function="cosine_similarity", + recreate_table=True, +) + +documents = [Document(content="There are over 7,000 languages spoken around the world today."), + Document(content="Elephants have been observed to behave in a way that indicates..."), + Document(content="In certain places, you can witness the phenomenon of bioluminescent waves.")] + +document_embedder = SentenceTransformersDocumentEmbedder() +documents_with_embeddings = document_embedder.run(documents) + +document_store.write_documents(documents_with_embeddings.get("documents"), policy=DuplicatePolicy.OVERWRITE) + +query_pipeline = Pipeline() +query_pipeline.add_component("text_embedder", SentenceTransformersTextEmbedder()) +query_pipeline.add_component("retriever", PgvectorEmbeddingRetriever(document_store=document_store)) +query_pipeline.connect("text_embedder.embedding", "retriever.query_embedding") + +query = "How many languages are there?" + +res = query_pipeline.run({"text_embedder": {"text": query}}) + +assert res['retriever']['documents'][0].content == "There are over 7,000 languages spoken around the world today." +``` + +#### __init__ + +```python +__init__( + *, + document_store: PgvectorDocumentStore, + filters: dict[str, Any] | None = None, + top_k: int = 10, + vector_function: ( + Literal["cosine_similarity", "inner_product", "l2_distance"] | None + ) = None, + filter_policy: str | FilterPolicy = FilterPolicy.REPLACE +) -> None +``` + +Initialize the PgvectorEmbeddingRetriever. + +**Parameters:** + +- **document_store** (PgvectorDocumentStore) – An instance of `PgvectorDocumentStore`. +- **filters** (dict\[str, Any\] | None) – Filters applied to the retrieved Documents. +- **top_k** (int) – Maximum number of Documents to return. +- **vector_function** (Literal['cosine_similarity', 'inner_product', 'l2_distance'] | None) – The similarity function to use when searching for similar embeddings. + Defaults to the one set in the `document_store` instance. + `"cosine_similarity"` and `"inner_product"` are similarity functions and + higher scores indicate greater similarity between the documents. + `"l2_distance"` returns the straight-line distance between vectors, + and the most similar documents are the ones with the smallest score. + **Important**: if the document store is using the `"hnsw"` search strategy, the vector function + should match the one utilized during index creation to take advantage of the index. +- **filter_policy** (str | FilterPolicy) – Policy to determine how filters are applied. + +**Raises:** + +- ValueError – If `document_store` is not an instance of `PgvectorDocumentStore` or if `vector_function` + is not one of the valid options. + +#### to_dict + +```python +to_dict() -> dict[str, Any] +``` + +Serializes the component to a dictionary. + +**Returns:** + +- dict\[str, Any\] – Dictionary with serialized data. + +#### from_dict + +```python +from_dict(data: dict[str, Any]) -> PgvectorEmbeddingRetriever +``` + +Deserializes the component from a dictionary. + +**Parameters:** + +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. + +**Returns:** + +- PgvectorEmbeddingRetriever – Deserialized component. + +#### close + +```python +close() -> None +``` + +Release the synchronous resources of the underlying Document Store. + +#### close_async + +```python +close_async() -> None +``` + +Release the asynchronous resources of the underlying Document Store. + +#### run + +```python +run( + query_embedding: list[float], + filters: dict[str, Any] | None = None, + top_k: int | None = None, + vector_function: ( + Literal["cosine_similarity", "inner_product", "l2_distance"] | None + ) = None, +) -> dict[str, list[Document]] +``` + +Retrieve documents from the `PgvectorDocumentStore`, based on their embeddings. + +**Parameters:** + +- **query_embedding** (list\[float\]) – Embedding of the query. +- **filters** (dict\[str, Any\] | None) – Filters applied to the retrieved Documents. The way runtime filters are applied depends on + the `filter_policy` chosen at retriever initialization. See init method docstring for more + details. +- **top_k** (int | None) – Maximum number of Documents to return. +- **vector_function** (Literal['cosine_similarity', 'inner_product', 'l2_distance'] | None) – The similarity function to use when searching for similar embeddings. + +**Returns:** + +- dict\[str, list\[Document\]\] – A dictionary with the following keys: +- `documents`: List of `Document`s that are similar to `query_embedding`. + +#### run_async + +```python +run_async( + query_embedding: list[float], + filters: dict[str, Any] | None = None, + top_k: int | None = None, + vector_function: ( + Literal["cosine_similarity", "inner_product", "l2_distance"] | None + ) = None, +) -> dict[str, list[Document]] +``` + +Asynchronously retrieve documents from the `PgvectorDocumentStore`, based on their embeddings. + +**Parameters:** + +- **query_embedding** (list\[float\]) – Embedding of the query. +- **filters** (dict\[str, Any\] | None) – Filters applied to the retrieved Documents. The way runtime filters are applied depends on + the `filter_policy` chosen at retriever initialization. See init method docstring for more + details. +- **top_k** (int | None) – Maximum number of Documents to return. +- **vector_function** (Literal['cosine_similarity', 'inner_product', 'l2_distance'] | None) – The similarity function to use when searching for similar embeddings. + +**Returns:** + +- dict\[str, list\[Document\]\] – A dictionary with the following keys: +- `documents`: List of `Document`s that are similar to `query_embedding`. + +## haystack_integrations.components.retrievers.pgvector.keyword_retriever + +### PgvectorKeywordRetriever + +Retrieve documents from the `PgvectorDocumentStore`, based on keywords. + +To rank the documents, the `ts_rank_cd` function of PostgreSQL is used. +It considers how often the query terms appear in the document, how close together the terms are in the document, +and how important is the part of the document where they occur. +For more details, see +[Postgres documentation](https://www.postgresql.org/docs/current/textsearch-controls.html#TEXTSEARCH-RANKING). + +Usage example: + +````python +from haystack.document_stores import DuplicatePolicy +from haystack import Document + +from haystack_integrations.document_stores.pgvector import PgvectorDocumentStore +from haystack_integrations.components.retrievers.pgvector import PgvectorKeywordRetriever + +# Set an environment variable `PG_CONN_STR` with the connection string to your PostgreSQL database. +# e.g., "postgresql://USER:PASSWORD@HOST:PORT/DB_NAME" + +document_store = PgvectorDocumentStore(language="english", recreate_table=True) + +documents = [Document(content="There are over 7,000 languages spoken around the world today."), + Document(content="Elephants have been observed to behave in a way that indicates..."), + Document(content="In certain places, you can witness the phenomenon of bioluminescent waves.")] + +document_store.write_documents(documents_with_embeddings.get("documents"), policy=DuplicatePolicy.OVERWRITE) + +retriever = PgvectorKeywordRetriever(document_store=document_store) + +result = retriever.run(query="languages") + +assert res['retriever']['documents'][0].content == "There are over 7,000 languages spoken around the world today." + +#### __init__ + +```python +__init__( + *, + document_store: PgvectorDocumentStore, + filters: dict[str, Any] | None = None, + top_k: int = 10, + filter_policy: str | FilterPolicy = FilterPolicy.REPLACE +) -> None +```` + +Initialize the PgvectorKeywordRetriever. + +**Parameters:** + +- **document_store** (PgvectorDocumentStore) – An instance of `PgvectorDocumentStore`. +- **filters** (dict\[str, Any\] | None) – Filters applied to the retrieved Documents. +- **top_k** (int) – Maximum number of Documents to return. +- **filter_policy** (str | FilterPolicy) – Policy to determine how filters are applied. + +**Raises:** + +- ValueError – If `document_store` is not an instance of `PgvectorDocumentStore`. + +#### to_dict + +```python +to_dict() -> dict[str, Any] +``` + +Serializes the component to a dictionary. + +**Returns:** + +- dict\[str, Any\] – Dictionary with serialized data. + +#### from_dict + +```python +from_dict(data: dict[str, Any]) -> PgvectorKeywordRetriever +``` + +Deserializes the component from a dictionary. + +**Parameters:** + +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. + +**Returns:** + +- PgvectorKeywordRetriever – Deserialized component. + +#### close + +```python +close() -> None +``` + +Release the synchronous resources of the underlying Document Store. + +#### close_async + +```python +close_async() -> None +``` + +Release the asynchronous resources of the underlying Document Store. + +#### run + +```python +run( + query: str, filters: dict[str, Any] | None = None, top_k: int | None = None +) -> dict[str, list[Document]] +``` + +Retrieve documents from the `PgvectorDocumentStore`, based on keywords. + +**Parameters:** + +- **query** (str) – String to search in `Document`s' content. +- **filters** (dict\[str, Any\] | None) – Filters applied to the retrieved Documents. The way runtime filters are applied depends on + the `filter_policy` chosen at retriever initialization. See init method docstring for more + details. +- **top_k** (int | None) – Maximum number of Documents to return. + +**Returns:** + +- dict\[str, list\[Document\]\] – A dictionary with the following keys: +- `documents`: List of `Document`s that match the query. + +#### run_async + +```python +run_async( + query: str, filters: dict[str, Any] | None = None, top_k: int | None = None +) -> dict[str, list[Document]] +``` + +Asynchronously retrieve documents from the `PgvectorDocumentStore`, based on keywords. + +**Parameters:** + +- **query** (str) – String to search in `Document`s' content. +- **filters** (dict\[str, Any\] | None) – Filters applied to the retrieved Documents. The way runtime filters are applied depends on + the `filter_policy` chosen at retriever initialization. See init method docstring for more + details. +- **top_k** (int | None) – Maximum number of Documents to return. + +**Returns:** + +- dict\[str, list\[Document\]\] – A dictionary with the following keys: +- `documents`: List of `Document`s that match the query. + +## haystack_integrations.document_stores.pgvector.document_store + +### PgvectorDocumentStore + +A Document Store using PostgreSQL with the [pgvector extension](https://github.com/pgvector/pgvector) installed. + +#### __init__ + +```python +__init__( + *, + connection_string: Secret = Secret.from_env_var("PG_CONN_STR"), + create_extension: bool = True, + schema_name: str = "public", + table_name: str = "haystack_documents", + language: str = "english", + embedding_dimension: int = 768, + vector_type: Literal["vector", "halfvec"] = "vector", + vector_function: Literal[ + "cosine_similarity", "inner_product", "l2_distance" + ] = "cosine_similarity", + recreate_table: bool = False, + search_strategy: Literal[ + "exact_nearest_neighbor", "hnsw" + ] = "exact_nearest_neighbor", + hnsw_recreate_index_if_exists: bool = False, + hnsw_index_creation_kwargs: dict[str, int] | None = None, + hnsw_index_name: str = "haystack_hnsw_index", + hnsw_ef_search: int | None = None, + keyword_index_name: str = "haystack_keyword_index" +) -> None +``` + +Creates a new PgvectorDocumentStore instance. + +It is meant to be connected to a PostgreSQL database with the pgvector extension installed. +A specific table to store Haystack documents will be created if it doesn't exist yet. + +**Parameters:** + +- **connection_string** (Secret) – The connection string to use to connect to the PostgreSQL database, defined as an + environment variable. Supported formats: +- URI, e.g. `PG_CONN_STR="postgresql://USER:PASSWORD@HOST:PORT/DB_NAME"` (use percent-encoding for special + characters) +- keyword/value format, e.g. `PG_CONN_STR="host=HOST port=PORT dbname=DBNAME user=USER password=PASSWORD"` + See [PostgreSQL Documentation](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING) + for more details. +- **create_extension** (bool) – Whether to create the pgvector extension if it doesn't exist. + Set this to `True` (default) to automatically create the extension if it is missing. + Creating the extension may require superuser privileges. + If set to `False`, ensure the extension is already installed; otherwise, an error will be raised. +- **schema_name** (str) – The name of the schema the table is created in. The schema must already exist. +- **table_name** (str) – The name of the table to use to store Haystack documents. +- **language** (str) – The language to be used to parse query and document content in keyword retrieval. + To see the list of available languages, you can run the following SQL query in your PostgreSQL database: + `SELECT cfgname FROM pg_ts_config;`. + More information can be found in this [StackOverflow answer](https://stackoverflow.com/a/39752553). +- **embedding_dimension** (int) – The dimension of the embedding. +- **vector_type** (Literal['vector', 'halfvec']) – The type of vector used for embedding storage. + "vector" is the default. + "halfvec" stores embeddings in half-precision, which is particularly useful for high-dimensional embeddings + (dimension greater than 2,000 and up to 4,000). Requires pgvector versions 0.7.0 or later. For more + information, see the [pgvector documentation](https://github.com/pgvector/pgvector?tab=readme-ov-file). +- **vector_function** (Literal['cosine_similarity', 'inner_product', 'l2_distance']) – The similarity function to use when searching for similar embeddings. + `"cosine_similarity"` and `"inner_product"` are similarity functions and + higher scores indicate greater similarity between the documents. + `"l2_distance"` returns the straight-line distance between vectors, + and the most similar documents are the ones with the smallest score. + **Important**: when using the `"hnsw"` search strategy, an index will be created that depends on the + `vector_function` passed here. Make sure subsequent queries will keep using the same + vector similarity function in order to take advantage of the index. +- **recreate_table** (bool) – Whether to recreate the table if it already exists. +- **search_strategy** (Literal['exact_nearest_neighbor', 'hnsw']) – The search strategy to use when searching for similar embeddings. + `"exact_nearest_neighbor"` provides perfect recall but can be slow for large numbers of documents. + `"hnsw"` is an approximate nearest neighbor search strategy, + which trades off some accuracy for speed; it is recommended for large numbers of documents. + **Important**: when using the `"hnsw"` search strategy, an index will be created that depends on the + `vector_function` passed here. Make sure subsequent queries will keep using the same + vector similarity function in order to take advantage of the index. +- **hnsw_recreate_index_if_exists** (bool) – Whether to recreate the HNSW index if it already exists. + Only used if search_strategy is set to `"hnsw"`. +- **hnsw_index_creation_kwargs** (dict\[str, int\] | None) – Additional keyword arguments to pass to the HNSW index creation. + Only used if search_strategy is set to `"hnsw"`. You can find the list of valid arguments in the + [pgvector documentation](https://github.com/pgvector/pgvector?tab=readme-ov-file#hnsw) +- **hnsw_index_name** (str) – Index name for the HNSW index. +- **hnsw_ef_search** (int | None) – The `ef_search` parameter to use at query time. Only used if search_strategy is set to + `"hnsw"`. You can find more information about this parameter in the + [pgvector documentation](https://github.com/pgvector/pgvector?tab=readme-ov-file#hnsw). +- **keyword_index_name** (str) – Index name for the Keyword index. + +#### to_dict + +```python +to_dict() -> dict[str, Any] +``` + +Serializes the component to a dictionary. + +**Returns:** + +- dict\[str, Any\] – Dictionary with serialized data. + +#### from_dict + +```python +from_dict(data: dict[str, Any]) -> PgvectorDocumentStore +``` + +Deserializes the component from a dictionary. + +**Parameters:** + +- **data** (dict\[str, Any\]) – Dictionary to deserialize from. + +**Returns:** + +- PgvectorDocumentStore – Deserialized component. + +#### close + +```python +close() -> None +``` + +Release the associated synchronous resources. + +#### close_async + +```python +close_async() -> None +``` + +Release the associated asynchronous resources. + +#### delete_table + +```python +delete_table() -> None +``` + +Deletes the table used to store Haystack documents. + +The name of the schema (`schema_name`) and the name of the table (`table_name`) +are defined when initializing the `PgvectorDocumentStore`. + +#### delete_table_async + +```python +delete_table_async() -> None +``` + +Async method to delete the table used to store Haystack documents. + +#### count_documents + +```python +count_documents() -> int +``` + +Returns how many documents are present in the document store. + +**Returns:** + +- int – Number of documents in the document store. + +#### count_documents_async + +```python +count_documents_async() -> int +``` + +Returns how many documents are present in the document store. + +**Returns:** + +- int – Number of documents in the document store. + +#### filter_documents + +```python +filter_documents(filters: dict[str, Any] | None = None) -> list[Document] +``` + +Returns the documents that match the filters provided. + +For a detailed specification of the filters, +refer to the [documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering) + +**Parameters:** + +- **filters** (dict\[str, Any\] | None) – The filters to apply to the document list. + +**Returns:** + +- list\[Document\] – A list of Documents that match the given filters. + +**Raises:** + +- TypeError – If `filters` is not a dictionary. +- ValueError – If `filters` syntax is invalid. + +#### filter_documents_async + +```python +filter_documents_async(filters: dict[str, Any] | None = None) -> list[Document] +``` + +Asynchronously returns the documents that match the filters provided. + +For a detailed specification of the filters, +refer to the [documentation](https://docs.haystack.deepset.ai/docs/metadata-filtering) + +**Parameters:** + +- **filters** (dict\[str, Any\] | None) – The filters to apply to the document list. + +**Returns:** + +- list\[Document\] – A list of Documents that match the given filters. + +**Raises:** + +- TypeError – If `filters` is not a dictionary. +- ValueError – If `filters` syntax is invalid. + +#### write_documents + +```python +write_documents( + documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE +) -> int +``` + +Writes documents to the document store. + +**Parameters:** + +- **documents** (list\[Document\]) – A list of Documents to write to the document store. +- **policy** (DuplicatePolicy) – The duplicate policy to use when writing documents. + +**Returns:** + +- int – The number of documents written to the document store. + +**Raises:** + +- ValueError – If `documents` contains objects that are not of type `Document`. +- DuplicateDocumentError – If a document with the same id already exists in the document store + and the policy is set to `DuplicatePolicy.FAIL` (or not specified). +- DocumentStoreError – If the write operation fails for any other reason. + +#### write_documents_async + +```python +write_documents_async( + documents: list[Document], policy: DuplicatePolicy = DuplicatePolicy.NONE +) -> int +``` + +Asynchronously writes documents to the document store. + +**Parameters:** + +- **documents** (list\[Document\]) – A list of Documents to write to the document store. +- **policy** (DuplicatePolicy) – The duplicate policy to use when writing documents. + +**Returns:** + +- int – The number of documents written to the document store. + +**Raises:** + +- ValueError – If `documents` contains objects that are not of type `Document`. +- DuplicateDocumentError – If a document with the same id already exists in the document store + and the policy is set to `DuplicatePolicy.FAIL` (or not specified). +- DocumentStoreError – If the write operation fails for any other reason. + +#### delete_documents + +```python +delete_documents(document_ids: list[str]) -> None +``` + +Deletes documents that match the provided `document_ids` from the document store. + +**Parameters:** + +- **document_ids** (list\[str\]) – the document ids to delete + +#### delete_documents_async + +```python +delete_documents_async(document_ids: list[str]) -> None +``` + +Asynchronously deletes documents that match the provided `document_ids` from the document store. + +**Parameters:** + +- **document_ids** (list\[str\]) – the document ids to delete + +#### delete_all_documents + +```python +delete_all_documents() -> None +``` + +Deletes all documents in the document store. + +#### delete_all_documents_async + +```python +delete_all_documents_async() -> None +``` + +Asynchronously deletes all documents in the document store. + +#### delete_by_filter + +```python +delete_by_filter(filters: dict[str, Any]) -> int +``` + +Deletes all documents that match the provided filters. + +**Parameters:** + +- **filters** (dict\[str, Any\]) – The filters to apply to select documents for deletion. + For filter syntax, see [Haystack metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering) + +**Returns:** + +- int – The number of documents deleted. + +#### delete_by_filter_async + +```python +delete_by_filter_async(filters: dict[str, Any]) -> int +``` + +Asynchronously deletes all documents that match the provided filters. + +**Parameters:** + +- **filters** (dict\[str, Any\]) – The filters to apply to select documents for deletion. + For filter syntax, see [Haystack metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering) + +**Returns:** + +- int – The number of documents deleted. + +#### update_by_filter + +```python +update_by_filter(filters: dict[str, Any], meta: dict[str, Any]) -> int +``` + +Updates the metadata of all documents that match the provided filters. + +**Parameters:** + +- **filters** (dict\[str, Any\]) – The filters to apply to select documents for updating. + For filter syntax, see [Haystack metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering) +- **meta** (dict\[str, Any\]) – The metadata fields to update. + +**Returns:** + +- int – The number of documents updated. + +#### update_by_filter_async + +```python +update_by_filter_async(filters: dict[str, Any], meta: dict[str, Any]) -> int +``` + +Asynchronously updates the metadata of all documents that match the provided filters. + +**Parameters:** + +- **filters** (dict\[str, Any\]) – The filters to apply to select documents for updating. + For filter syntax, see [Haystack metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering) +- **meta** (dict\[str, Any\]) – The metadata fields to update. + +**Returns:** + +- int – The number of documents updated. + +#### count_documents_by_filter + +```python +count_documents_by_filter(filters: dict[str, Any]) -> int +``` + +Returns the number of documents that match the provided filters. + +**Parameters:** + +- **filters** (dict\[str, Any\]) – The filters to apply to count documents. + For filter syntax, see [Haystack metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering) + +**Returns:** + +- int – The number of documents that match the filters. + +#### count_documents_by_filter_async + +```python +count_documents_by_filter_async(filters: dict[str, Any]) -> int +``` + +Asynchronously returns the number of documents that match the provided filters. + +**Parameters:** + +- **filters** (dict\[str, Any\]) – The filters to apply to count documents. + For filter syntax, see [Haystack metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering) + +**Returns:** + +- int – The number of documents that match the filters. + +#### count_unique_metadata_by_filter + +```python +count_unique_metadata_by_filter( + filters: dict[str, Any], metadata_fields: list[str] +) -> dict[str, int] +``` + +Returns the count of unique values for each specified metadata field. + +Considers only documents that match the provided filters. + +**Parameters:** + +- **filters** (dict\[str, Any\]) – The filters to apply to select documents. + For filter syntax, see [Haystack metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering) +- **metadata_fields** (list\[str\]) – List of metadata field names to count unique values for. + Field names can include or omit the "meta." prefix. + +**Returns:** + +- dict\[str, int\] – A dictionary mapping field names to their unique value counts. + +#### count_unique_metadata_by_filter_async + +```python +count_unique_metadata_by_filter_async( + filters: dict[str, Any], metadata_fields: list[str] +) -> dict[str, int] +``` + +Asynchronously returns the count of unique values for each specified metadata field. + +Considers only documents that match the provided filters. + +**Parameters:** + +- **filters** (dict\[str, Any\]) – The filters to apply to select documents. + For filter syntax, see [Haystack metadata filtering](https://docs.haystack.deepset.ai/docs/metadata-filtering) +- **metadata_fields** (list\[str\]) – List of metadata field names to count unique values for. + Field names can include or omit the "meta." prefix. + +**Returns:** + +- dict\[str, int\] – A dictionary mapping field names to their unique value counts. + +#### get_metadata_fields_info + +```python +get_metadata_fields_info() -> dict[str, dict[str, str]] +``` + +Returns the information about the metadata fields in the document store. + +Since metadata is stored in a JSONB field, this method analyzes actual data +to infer field types. + +Example return: + +```python +{ + 'category': {'type': 'text'}, + 'status': {'type': 'text'}, + 'priority': {'type': 'integer'}, +} +``` + +**Returns:** + +- dict\[str, dict\[str, str\]\] – A dictionary mapping field names to their type information. + +#### get_metadata_fields_info_async + +```python +get_metadata_fields_info_async() -> dict[str, dict[str, str]] +``` + +Asynchronously returns the information about the metadata fields in the document store. + +Since metadata is stored in a JSONB field, this method analyzes actual data +to infer field types. + +**Returns:** + +- dict\[str, dict\[str, str\]\] – A dictionary mapping field names to their type information. + +#### get_metadata_field_min_max + +```python +get_metadata_field_min_max(metadata_field: str) -> dict[str, Any] +``` + +Returns the minimum and maximum values for a given metadata field. + +**Parameters:** + +- **metadata_field** (str) – The name of the metadata field. Can include or omit the "meta." prefix. + +**Returns:** + +- dict\[str, Any\] – A dictionary with 'min' and 'max' keys containing the minimum and maximum values. + For numeric fields (integer, real), returns numeric min/max. + For text fields, returns lexicographic min/max based on database collation. + Returns `{"min": None, "max": None}` when the field has no values or the store is empty. + +#### get_metadata_field_min_max_async + +```python +get_metadata_field_min_max_async(metadata_field: str) -> dict[str, Any] +``` + +Asynchronously returns the minimum and maximum values for a given metadata field. + +**Parameters:** + +- **metadata_field** (str) – The name of the metadata field. Can include or omit the "meta." prefix. + +**Returns:** + +- dict\[str, Any\] – A dictionary with 'min' and 'max' keys containing the minimum and maximum values. + For numeric fields (integer, real), returns numeric min/max. + For text fields, returns lexicographic min/max based on database collation. + Returns `{"min": None, "max": None}` when the field has no values or the store is empty. + +#### get_metadata_field_unique_values + +```python +get_metadata_field_unique_values( + metadata_field: str, + search_term: str | None = None, + from_: int = 0, + size: int = 10, + filters: dict[str, Any] | None = None, +) -> tuple[list[Any], int] +``` + +Returns unique values for a given metadata field, optionally filtered by a search term. + +**Note**: values of different JSON type categories are kept distinct - a string, a number and +a boolean never collapse into each other, even when they share a textual form (e.g. the string +`"1"` and the number `1`). One exception: the `meta` column is JSONB, whose equality treats a +whole-number float (`1.0`) as identical to a numerically equal int (`1`), so those two collapse +into a single value. Floats with a fractional part (e.g. `1.5`) are unaffected. + +**Parameters:** + +- **metadata_field** (str) – The name of the metadata field. Can include or omit the "meta." prefix. +- **search_term** (str | None) – Optional search term to filter unique values by a case-insensitive substring + match against the metadata field's own value. If None, all values are considered. +- **from\_** (int) – The offset for pagination (0-based). +- **size** (int) – The number of unique values to return. +- **filters** (dict\[str, Any\] | None) – Optional filters to restrict the documents considered. + +**Returns:** + +- tuple\[list\[Any\], int\] – A tuple containing: +- A list of unique values in their original type +- The total count of unique values + +#### get_metadata_field_unique_values_async + +```python +get_metadata_field_unique_values_async( + metadata_field: str, + search_term: str | None = None, + from_: int = 0, + size: int = 10, + filters: dict[str, Any] | None = None, +) -> tuple[list[Any], int] +``` + +Asynchronously returns unique values for a given metadata field, optionally filtered by a search term. + +**Note**: values of different JSON type categories are kept distinct - a string, a number and +a boolean never collapse into each other, even when they share a textual form (e.g. the string +`"1"` and the number `1`). One exception: the `meta` column is JSONB, whose equality treats a +whole-number float (`1.0`) as identical to a numerically equal int (`1`), so those two collapse +into a single value. Floats with a fractional part (e.g. `1.5`) are unaffected. + +**Parameters:** + +- **metadata_field** (str) – The name of the metadata field. Can include or omit the "meta." prefix. +- **search_term** (str | None) – Optional search term to filter unique values by a case-insensitive substring + match against the metadata field's own value. If None, all values are considered. +- **from\_** (int) – The offset for pagination (0-based). +- **size** (int) – The number of unique values to return. +- **filters** (dict\[str, Any\] | None) – Optional filters to restrict the documents considered. + +**Returns:** + +- tuple\[list\[Any\], int\] – A tuple containing: +- A list of unique values in their original type +- The total count of unique values