diff --git a/README.md b/README.md index c91f556..2b5563a 100644 --- a/README.md +++ b/README.md @@ -1,19 +1,37 @@ -# 🐍 The official async Python client for Montycat β€” the Rust-powered NoSQL database built for the Data Mesh era. +# 🐍 Montycat for Python β€” The AI-Native NoSQL Database with Semantic Search for RAG & Agents + +### Abolish the two-database stack. + +The official async Python client for [Montycat](https://montygovernance.com) β€” a self-hosted **NoSQL + vector database** with AI **semantic search** forged into the core, built for **RAG and AI-agent memory**. One Rust engine, not a sprawl of services. **Your hardware. Your data. Your meaning.** -[![PyPI Downloads](https://static.pepy.tech/personalized-badge/montycat?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/montycat) -[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE) [![PyPI Version](https://img.shields.io/pypi/v/montycat.svg)](https://pypi.org/project/montycat/) +[![PyPI Downloads](https://static.pepy.tech/personalized-badge/montycat?period=total&units=INTERNATIONAL_SYSTEM&left_color=BLACK&right_color=GREEN&left_text=downloads)](https://pepy.tech/projects/montycat) +[![Docker Pulls](https://img.shields.io/docker/pulls/montygovernance/montycat)](https://hub.docker.com/r/montygovernance/montycat) [![Python Version](https://img.shields.io/pypi/pyversions/montycat)](https://www.python.org/) -[![Maintenance](https://img.shields.io/badge/maintained-yes-brightgreen.svg)]() +[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/MontyGovernance/montycat_python/blob/master/LICENSE) + +```python +# Search your data by MEANING β€” no external APIs, no separate vector database. +# (already ON by default in the montycat-semantic server edition) +hits = await Sales.semantic_search_get_values("something to listen to music without wires", limit=5) +# β†’ [{key, score, value: {"name": "Wireless Headphones"}}, ...] (matched by meaning, not keywords) +``` + +> ### 🧩 All-in-one. AI-native. **Zero external dependencies.** +> The vector-embedding engine runs **inside** the database β€” **no** separate vector DB, **no** embedding API, **no** API keys, **no** sidecar service. One engine, one binary, your hardware. ## What is Montycat? -Montycat is a Rust-powered NoSQL engine designed for the future of data β€” decentralized by nature, ultra-fast, and natively async. +For a generation we were told the price of intelligence was two systems: a database for your records, and a separate vector store β€” with its per-query bill β€” for their meaning. Montycat rejects that tax. It is a **self-hosted NoSQL + vector database**: one Rust-powered engine with semantic search built in, so **RAG, AI-agent memory, and vector search** live where your data already lives. No cloud lock-in. No ops headache. Decentralized by nature, ultra-fast, and natively async. + +Think of it as an **open-source, self-hosted alternative to Pinecone, Weaviate, Chroma, Qdrant, and Redis** β€” a **vector database _and_ a NoSQL store in a single engine**, so your records and their embeddings live together instead of in two systems you have to keep in sync. Montycat is not an incremental improvement on the databases you know. It is a break with them. + ## 🧠 Why Montycat? - ⚑ Blazing Speed β€” Powered by the Montycat Engine written in Rust, built for microsecond-level read/write performance. - πŸŒ€ Async-First Design β€” Fully asynchronous, built on asyncio. Perfect for APIs, pipelines, and real-time apps. - πŸ’Ύ Hybrid Storage β€” In-memory for raw speed or persistent for durability β€” or mix both in one engine. +- πŸ”Ž AI Semantic & Vector Search β€” Rank data by *meaning* with on-device embeddings. Built-in kNN vector search for **RAG, AI agents & LLM apps** β€” no external API, no separate vector database. *(requires the `montycat-semantic` server edition β€” Docker image, package, or apt; see below.)* - 🧩 Schema-Aware β€” Define data schemas in Python, enforce them at runtime β€” with zero ceremony. - πŸ—‚οΈ True Data Mesh Architecture β€” Each keyspace is a self-owned, domain-oriented data product. - πŸ“‘ Reactive Subscriptions β€” Subscribe to live updates in real-time β€” per key or per keyspace. @@ -25,25 +43,38 @@ Montycat is a Rust-powered NoSQL engine designed for the future of data β€” dece Montycat is not a database wrapper. It’s a new way to think about data β€” composable, fast by design. No legacy baggage. Just clean async functions and pure data. Montycat isn’t inspired by NoSQL. It redefines it β€” with elegance, concurrency, and memory safety. -## πŸ‘‰ Install the Engine: https://montygovernance.com - ## Montycat for Python? -This is the official Python client, built to bring Montycat’s next-generation Data Mesh architecture directly into your Python applications. This client empowers developers to seamlessly manage and query their data while leveraging the unparalleled flexibility and scalability offered by NoSQL databases within a decentralized data ownership paradigm -Forget ORM hell, clunky SQL syntax, or blocking I/O. -With Montycat, data feels alive β€” reactive, structured, and fast enough to keep up with your imagination. +This is the official Python client, built to bring Montycat’s next-generation Data Mesh architecture directly into your Python applications β€” manage and query your data with the flexibility of NoSQL and true decentralized data ownership. Forget ORM hell, clunky SQL syntax, and blocking I/O. With Montycat, data feels alive β€” reactive, structured, and fast enough to keep up with your imagination. ## πŸ” Example Use Cases +- **RAG pipelines & semantic retrieval** for LLM-powered apps +- **AI agent / chatbot long-term memory** that survives restarts +- **Semantic product search & recommendations** β€” match intent, not keywords - Real-time dashboards and analytics - Async ETL pipelines with real-time awareness and processing -- Microservice data stores -- Event-driven data systems +- Microservice data stores and event-driven systems - Collaborative data products in a Mesh architecture +## πŸš€ Get the Engine (30 seconds) + +The client talks to a Montycat server. Fastest way β€” Docker, with AI semantic search built in: + +```bash +docker run -d --name montycat \ + -p 21210:21210 -p 21211:21211 \ + -e MONTYCAT_SUPEROWNER="admin" \ + -e MONTYCAT_PASSWORD="change-me" \ + -v montycat_data:/app/.montycat \ + montygovernance/montycat:semantic +``` + +Prefer the lean edition without the embedding engine? Use the `latest` tag. Prebuilt packages (apt, macOS, Windows) at **https://montygovernance.com**. + ## Installation -You can install Python client for Montycat using `pip`: +Install the Python client with `pip`: ```bash pip install montycat @@ -52,8 +83,8 @@ pip install montycat ## Quick Start ```python -from montycat import Engine, Keyspace, Schema import asyncio +from montycat import Engine, Keyspace, Schema # setup connection @@ -65,6 +96,8 @@ connection = Engine( store="Departments", ) +# keyspaces: persistent or in-memory β€” mix freely in one engine + class Sales(Keyspace.Persistent): keyspace = "Sales" @@ -74,15 +107,7 @@ class Production(Keyspace.InMemory): Sales.connect_engine(connection) Production.connect_engine(connection) -# create store and keyspaces using runtime migration - -async def setup_keyspaces(): - await Production.create_keyspace() - await Sales.create_keyspace() - -asyncio.run(setup_keyspaces()) - -# create schemas and enforce them on the database side (optional) +# schemas, enforced on the database side (optional) class SalesSchema(Schema): product: str @@ -92,32 +117,86 @@ class ProductionSchema(Schema): items: list work_order: str | None -async def migrate_schemas(): - await Production.enforce_schema() - await Sales.enforce_schema() +async def main(): + # create store and keyspaces using runtime migration + await Sales.create_keyspace() + await Production.create_keyspace() + + await Sales.enforce_schema(SalesSchema) + await Production.enforce_schema(ProductionSchema) + + # write + sale = SalesSchema(product="Product1", amount=12).serialize() + await Sales.insert_value(sale) -asyncio.run(migrate_schemas()) + order = ProductionSchema(items=["Product1"], work_order="WO 000012").serialize() + await Production.insert_value(order) -# run first queries + # query + print(await Sales.lookup_values_where(schema=SalesSchema, key_included=True)) + print(await Production.lookup_keys_where(work_order="WO 000012")) -sales = SalesSchema( - product = "Product1", - amount = 12 -).serialize() +asyncio.run(main()) +``` -asyncio.run(Sales.insert_value(sales)) +## 🧠 AI-Native Semantic Search β€” Vector Search Built Into Your Database -items_ordered = ProductionSchema( - items = ["Product1"], - work_order = "WO 000012" -).serialize() +**Stop bolting a separate vector database onto your stack.** Montycat ranks your data by +*meaning*, not keywords β€” an embedded, on-device vector-embedding engine turns every write +into a searchable vector automatically. It's the retrieval layer for **RAG pipelines, AI +agents, semantic search, recommendation engines, and LLM-powered apps** β€” with **zero +external APIs, zero API keys, and zero extra infrastructure.** -asyncio.run(Production.insert_value(items_ordered)) +- πŸ”Ž **Semantic / vector search** β€” kNN similarity over on-device embeddings, not brittle keyword matches. +- πŸ€– **Built for AI** β€” RAG, semantic retrieval, AI agents, recommendations, dedup, clustering. +- πŸ”’ **Private & free** β€” embeddings never leave your machine. No OpenAI/Cohere bill, no data egress. +- ⚑ **One system, not two** β€” your data *and* its vectors live in the same database. No sync jobs, no drift, no second service to run. +- πŸš€ **Zero setup** β€” no index tuning, no pipeline: `enable_semantic_search()` and you're ranking by meaning. -# verify +> **⚠️ Requires the semantic edition of the server β€” nothing to compile.** Semantic +> search runs an embedded ONNX vector-embedding engine that ships only in the +> **`montycat-semantic`** edition; the default lean `montycat` server does not include it. +> Get it the way that suits you β€” pull the **Docker image** +> (`montygovernance/montycat:semantic`), download the prebuilt **package**, or install +> `montycat-semantic` from the **apt repository**. The Python client API is identical +> either way; just point it at a semantic-edition server (semantic search is enabled by +> default there, using the `bge-small` model). -asyncio.run(Sales.lookup_values_where(schema=SalesSchema, key_included=True)) -asyncio.run(Production.lookup_keys_where(work_order="WO 000012")) +The switch is DB-wide and already on in the semantic edition. The embedding model is +downloaded on demand, and every keyspace is embedded in the background as data is written. +```python +# Semantic search is ON by default in the montycat-semantic edition β€” just search. +# Rank stored items by meaning β€” two flavors: +# get_values β†’ each hit is {key, score, value} +# get_keys β†’ each hit is {key, score} (lighter; fetch a page later with get_bulk) +hits = await Sales.semantic_search_get_values("something to listen to music without wires", limit=5) +keys = await Sales.semantic_search_get_keys("something to listen to music without wires", limit=5) + +# Optionally drop weak matches by cosine similarity (range [-1, 1]). +strong = await Sales.semantic_search_get_keys("something to listen to music without wires", limit=5, min_score=0.35) + +# Control the DB-wide switch (optional β€” it's already on): +# switch the embedding model: 'minilm' | 'bge-small' (default) | 'bge-base' | 'e5-small' +await connection.enable_semantic_search(model="bge-base") + +# turn it off (vectors are kept so re-enabling resumes instantly; +# pass drop_vectors=True to also clear stored vectors) +await connection.disable_semantic_search() ``` +## πŸ”— Links + +- 🌐 **Website & Docs** β€” https://montygovernance.com +- πŸ“¦ **PyPI** β€” https://pypi.org/project/montycat/ +- 🐳 **Docker Hub** β€” https://hub.docker.com/r/montygovernance/montycat +- πŸ’» **Source** β€” https://github.com/MontyGovernance/montycat_python + +## ❓ FAQ + +- **Is Montycat a vector database or a NoSQL database?** Both β€” one engine. Store records and query them by *meaning* (vector / semantic search) or by key/schema, without running two systems. +- **Do I need OpenAI or an embedding API?** No. Embeddings run on-device in the `montycat-semantic` server. No API keys, no per-query bill, no data egress. +- **Is it a Pinecone / Weaviate / Chroma / Qdrant alternative?** Yes β€” self-hosted and open-source, with a NoSQL store built in. +- **Which Python versions?** 3.9+ β€” fully async (`asyncio`). + + diff --git a/build_and_push.sh b/build_and_push.sh index d02c831..f7e5813 100755 --- a/build_and_push.sh +++ b/build_and_push.sh @@ -2,7 +2,10 @@ # Configuration PACKAGE_NAME="montycat" -VERSION="1.0.4" +# setup.py is the SINGLE SOURCE OF TRUTH for all metadata (description, +# keywords, classifiers, project_urls). Derive the version from it β€” never +# regenerate the file, or a build would silently downgrade the PyPI SEO. +VERSION="$(grep -oE "version='[^']+'" setup.py | head -1 | cut -d\' -f2)" PYPI_TOKEN="${PYPI_TOKEN:-}" # Exit on any error @@ -34,37 +37,6 @@ check_prerequisites() { fi } -# Update setup.py without package_data -update_setup() { - echo "Updating setup.py..." - cat > setup.py < Any: """ return await self._execute_query_with_credentials(['list-owners']) + async def enable_semantic_search(self, model: Union[str, None] = None, field: Union[str, None] = None, store: Union[str, None] = None) -> Any: + """ + Enable semantic (vector similarity) search. + + Without `store`, this is DB-wide: it flips the whole database on, sets the default + embedding model and field, and enrolls every existing keyspace that has no semantic + config yet (each gets a background backfill so its existing items become searchable). + The chosen model is downloaded on demand on first enable, so this call may take a + while the first time. + + With `store`, it is scoped: only that store's un-enrolled keyspaces are enrolled and + backfilled; the DB-wide switch and default model/field are left untouched. Use this + to (re-)enable one store without re-embedding the entire database. + + Args: + model (str, optional): The embedding model key to use by default. One of + 'minilm', 'bge-small', 'bge-base', 'e5-small'. Default + None, which uses the server default ('bge-small'). + field (str, optional): The JSON field of each value to embed. Default None, + which embeds the whole value. + store (str, optional): Restrict enrollment/backfill to this store only. Default + None (DB-wide). If the DB-wide switch is off, a scoped + enable enrolls but nothing embeds until a DB-wide enable. + + Returns: + Any: The server's response describing the enabled model and enrolled keyspaces. + """ + command = ['enable-semantic-search'] + if model: + command.extend(["model", model]) + if field: + command.extend(["field", field]) + if store: + command.extend(["store", store]) + + return await self._execute_query_with_credentials(command) + + async def disable_semantic_search(self, drop_vectors: bool = False, store: Union[str, None] = None) -> Any: + """ + Disable semantic search. + + Without `store`, this is DB-wide: embedding and semantic queries stop across the + whole database; stored vectors are kept by default so re-enabling resumes without + a full re-embed. + + With `store`, it is scoped: only that store's keyspaces are unenrolled (their + configs and resident graphs dropped); the DB-wide switch and all other stores are + left untouched. This is the surgical way to reset one store's semantic state + instead of nuking and re-backfilling the whole database. + + Args: + drop_vectors (bool, optional): If True, also clear stored vectors β€” every + keyspace's DB-wide, or the scoped store's when + `store` is set. Required before switching to a + different embedding model. Default False. + store (str, optional): Restrict the disable to this store only. Default None + (DB-wide). + + Returns: + Any: The server's response confirming the disable. + """ + command = ['disable-semantic-search'] + if drop_vectors: + command.append("drop-vectors") + if store: + command.extend(["store", store]) + + return await self._execute_query_with_credentials(command) + async def get_structure_available(self) -> Any: """ Retrieves the structure of the current store. @@ -222,3 +291,105 @@ async def get_structure_available(self) -> Any: command = ['get-structure-available', "store", self.store] if self.store else ['get-structure-available'] return await self._execute_query_with_credentials(command) + + async def enable_wait_for_index(self) -> Any: + """ + Enable the DB-wide "wait for index" default: writes block until their + secondary indexes are updated before returning, so a write is + immediately visible to index-backed reads (e.g. lookup_*_where) at the + cost of higher write latency. + + Requires superowner credentials. + + Returns: + Any: The server's response confirming the change. + """ + return await self._execute_query_with_credentials(['enable-wait-for-index']) + + async def disable_wait_for_index(self) -> Any: + """ + Disable the DB-wide "wait for index" default: writes return as soon as + the data is committed and indexing happens asynchronously in the + background (lower write latency; index-backed reads may briefly lag). + This is the default behavior. + + Requires superowner credentials. + + Returns: + Any: The server's response confirming the change. + """ + return await self._execute_query_with_credentials(['disable-wait-for-index']) + + async def enable_reports(self) -> Any: + """ + Enable server-side operation reporting (logging). Requires superowner credentials. + + Returns: + Any: The server's response confirming the change. + """ + return await self._execute_query_with_credentials(['enable-reports']) + + async def disable_reports(self) -> Any: + """ + Disable server-side operation reporting (logging). Requires superowner credentials. + + Returns: + Any: The server's response confirming the change. + """ + return await self._execute_query_with_credentials(['disable-reports']) + + async def allow_subscriptions(self) -> Any: + """ + Allow clients to open keyspace subscriptions DB-wide. Requires superowner credentials. + + Returns: + Any: The server's response confirming the change. + """ + return await self._execute_query_with_credentials(['allow-subscriptions']) + + async def restrict_subscriptions(self) -> Any: + """ + Restrict (disallow) keyspace subscriptions DB-wide. Requires superowner credentials. + + Returns: + Any: The server's response confirming the change. + """ + return await self._execute_query_with_credentials(['restrict-subscriptions']) + + async def queue_depths(self) -> Any: + """ + Sample the current depth of every background task queue (index, timer, + counting) β€” an observability probe for whether the background runners + are keeping up with the write rate. Requires superowner credentials. + + Returns: + Any: The server's response whose payload maps + "index" | "timer" | "counting" to per-queue depth maps. + """ + return await self._execute_query_with_credentials(['queue-depths']) + + async def set_snapshot_rate(self, rate: int) -> Any: + """ + Set the server-wide snapshot rate. Requires superowner credentials. + + Args: + rate (int): The snapshot rate value (server-defined units). + + Returns: + Any: The server's response confirming the change. + """ + return await self._execute_query_with_credentials(['snapshot-rate', str(rate)]) + + async def set_expiration_check_rate(self, rate: int) -> Any: + """ + Set how often the server scans for expired keys. Requires superowner credentials. + + Args: + rate (int): The check period in whole seconds (e.g. rate=10 β†’ a scan + every 10 seconds). Stored as-is, like the snapshot rate. + Defaults to 1 second server-side. + + Returns: + Any: The server's response confirming the change. + """ + return await self._execute_query_with_credentials(['expiration-check', str(rate)]) diff --git a/montycat/store_classes/inmemory.py b/montycat/store_classes/inmemory.py index c669ceb..b644578 100644 --- a/montycat/store_classes/inmemory.py +++ b/montycat/store_classes/inmemory.py @@ -69,11 +69,14 @@ async def stop_snapshots_for_keyspace(cls): return await cls._run_query(query) @classmethod - async def insert_custom_key(cls, custom_key: str, expire_sec: int = 0): + async def insert_custom_key(cls, custom_key: str, expire_sec: int = 0, wait_for_index: Union[bool, None] = None): """ Args: custom_key: A custom key to insert into the store. This key can be used to retrieve the value later. expire_sec: The number of seconds before the inserted value expires. + wait_for_index: Per-request synchronous-index override. No-op for in-memory + keyspaces (snapshot-driven indexing); present for API symmetry + with the persistent variant, where it takes effect. Returns: True if the insert operation was successful. Class 'str' if the insert operation failed. """ @@ -82,16 +85,17 @@ async def insert_custom_key(cls, custom_key: str, expire_sec: int = 0): custom_key_converted = convert_custom_key(custom_key) - query = convert_to_binary_query(cls, command="insert_custom_key", key=custom_key_converted, expire_sec=expire_sec) + query = convert_to_binary_query(cls, command="insert_custom_key", key=custom_key_converted, expire_sec=expire_sec, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def insert_custom_key_value(cls, custom_key: str, value: dict, expire_sec: int = 0): + async def insert_custom_key_value(cls, custom_key: str, value: dict, expire_sec: int = 0, wait_for_index: Union[bool, None] = None): """ Args: custom_key: A custom key to insert into the store. This key can be used to retrieve the value later. value: A Python class / dict to insert into the store. expire_sec: The number of seconds before the inserted value expires. + wait_for_index: Per-request synchronous-index override; no-op for in-memory. Returns: True if the insert operation was successful. Class 'str' if the insert operation failed. @@ -103,26 +107,27 @@ async def insert_custom_key_value(cls, custom_key: str, value: dict, expire_sec: custom_key_converted = convert_custom_key(custom_key) - query = convert_to_binary_query(cls, command="insert_custom_key_value", key=custom_key_converted, value=value, expire_sec=expire_sec) + query = convert_to_binary_query(cls, command="insert_custom_key_value", key=custom_key_converted, value=value, expire_sec=expire_sec, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def insert_value(cls, value: dict, expire_sec: int = 0): + async def insert_value(cls, value: dict, expire_sec: int = 0, wait_for_index: Union[bool, None] = None): """ Args: value: A Python class / dict to insert into the store. expire_sec: The number of seconds before the inserted value expires. + wait_for_index: Per-request synchronous-index override; no-op for in-memory. Returns: Key number if the insert operation was successful. Class 'str' if the insert operation failed. """ if not value: raise ValueError("No value provided for insertion.") - query = convert_to_binary_query(cls, command="insert_value", value=value, expire_sec=expire_sec) + query = convert_to_binary_query(cls, command="insert_value", value=value, expire_sec=expire_sec, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def update_value(cls, key: Union[str, None] = None, custom_key: Union[str, None] = None, expire_sec: int = 0, **filters): + async def update_value(cls, key: Union[str, None] = None, custom_key: Union[str, None] = None, expire_sec: int = 0, wait_for_index: Union[bool, None] = None, **filters): """ Update the value associated with a given key in the store. If a custom key is provided, it will be converted to the appropriate format before updating. @@ -151,15 +156,16 @@ async def update_value(cls, key: Union[str, None] = None, custom_key: Union[str, if not key: raise ValueError("No key provided") - query = convert_to_binary_query(cls, command="update_value", key=key, value=filters, expire_sec=expire_sec) + query = convert_to_binary_query(cls, command="update_value", key=key, value=filters, expire_sec=expire_sec, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def insert_bulk(cls, bulk_values: list, expire_sec: int = 0): + async def insert_bulk(cls, bulk_values: list, expire_sec: int = 0, wait_for_index: Union[bool, None] = None): """ Args: bulk_values: A list of Python objects to insert into the store. expire_sec: The number of seconds before the inserted values expire. + wait_for_index: Per-request synchronous-index override; no-op for in-memory. Returns: True if the bulk insert operation was successful. @@ -169,7 +175,7 @@ async def insert_bulk(cls, bulk_values: list, expire_sec: int = 0): if not bulk_values: raise ValueError("No values provided for bulk insertion.") - query = convert_to_binary_query(cls, command="insert_bulk", bulk_values=bulk_values, expire_sec=expire_sec) + query = convert_to_binary_query(cls, command="insert_bulk", bulk_values=bulk_values, expire_sec=expire_sec, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod diff --git a/montycat/store_classes/kv.py b/montycat/store_classes/kv.py index e5a3d07..716100c 100644 --- a/montycat/store_classes/kv.py +++ b/montycat/store_classes/kv.py @@ -139,13 +139,15 @@ async def remove_enforced_schema(cls, schema): if not schema: raise ValueError("No schema provided for removal") + # Server parses positionally: it expects "schema_name" at raw index 7 + # (get_schema_name). Do NOT insert a "distributed" pair here β€” it would + # shift schema_name and the server rejects the request as invalid. query = orjson.dumps({ "raw": [ "remove-enforced-schema", "store", cls.store, "keyspace", cls.keyspace, "persistent", "y" if cls.persistent else "n", - "distributed", "y" if cls.distributed else "n", "schema_name", str(schema) ], "credentials": [cls.username, cls.password] @@ -177,7 +179,7 @@ async def get_value(cls, key: Union[str, None] = None, custom_key: Union[str, No return await cls._run_query(query) @classmethod - async def delete_key(cls, key: Union[str, None] = None, custom_key: Union[str, None] = None): + async def delete_key(cls, key: Union[str, None] = None, custom_key: Union[str, None] = None, wait_for_index: Union[bool, None] = None): """ Delete a key from the store. If a custom key is provided, it will be converted to the appropriate format before deletion. @@ -187,6 +189,8 @@ async def delete_key(cls, key: Union[str, None] = None, custom_key: Union[str, N Default is an empty string, which will be ignored if custom_key is provided. custom_key (str, optional): The custom key to delete. This is used if the key provided is a custom key string. Default is an empty string. + wait_for_index: Per-request synchronous-index override; None (default) uses the DB-wide default. + Applies to persistent keyspaces; in-memory already indexes synchronously. Returns: bool | str: Returns a boolean indicating success (True) or failure (False), @@ -201,11 +205,11 @@ async def delete_key(cls, key: Union[str, None] = None, custom_key: Union[str, N if not key: raise ValueError("No key provided") - query = convert_to_binary_query(cls, command="delete_key", key=key) + query = convert_to_binary_query(cls, command="delete_key", key=key, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def delete_bulk(cls, bulk_keys: list = [], bulk_custom_keys: list = []): + async def delete_bulk(cls, bulk_keys: list = [], bulk_custom_keys: list = [], wait_for_index: Union[bool, None] = None): """ Delete multiple keys in bulk. If custom keys are provided, they are first converted to the appropriate format and then appended to the list of keys to be deleted. @@ -231,7 +235,7 @@ async def delete_bulk(cls, bulk_keys: list = [], bulk_custom_keys: list = []): if not bulk_keys: raise ValueError("No keys provided for deletion.") - query = convert_to_binary_query(cls, command="delete_bulk", bulk_keys=bulk_keys) + query = convert_to_binary_query(cls, command="delete_bulk", bulk_keys=bulk_keys, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod @@ -282,7 +286,7 @@ async def get_bulk( return await cls._run_query(query) @classmethod - async def update_bulk(cls, bulk_keys_values: dict = {}, bulk_custom_keys_values: dict = {}): + async def update_bulk(cls, bulk_keys_values: dict = {}, bulk_custom_keys_values: dict = {}, wait_for_index: Union[bool, None] = None): """ Update multiple keys in bulk with the provided new values. If custom keys are provided, they will be converted before being applied to the bulk update. @@ -309,7 +313,7 @@ async def update_bulk(cls, bulk_keys_values: dict = {}, bulk_custom_keys_values: bulk_custom_keys_values = convert_custom_keys_values(bulk_custom_keys_values) bulk_keys_values = {**bulk_keys_values, **bulk_custom_keys_values} - query = convert_to_binary_query(cls, command="update_bulk", bulk_keys_values=bulk_keys_values) + query = convert_to_binary_query(cls, command="update_bulk", bulk_keys_values=bulk_keys_values, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod @@ -353,6 +357,98 @@ async def lookup_values_where(cls, limit: Union[int, list] = 0, with_pointers: b query = convert_to_binary_query(cls, command="lookup_values", limit_output=handle_limit(limit), search_criteria=filters, with_pointers=with_pointers, key_included=key_included, pointers_metadata=pointers_metadata, schema=str(schema) if schema else None) return await cls._run_query(query) + @classmethod + async def _semantic_search(cls, query: str, limit: Union[int, list], min_score: Union[float, None], with_pointers: bool, key_included: bool, pointers_metadata: bool): + """Shared core for `semantic_search_get_keys` / `semantic_search_get_values`. + + The server command is the same either way (`semantic_search`); the two + public methods differ only in which value-inclusion flags they pass, so + the wire call lives here once. + """ + if not query or not query.strip(): + raise ValueError("No query text provided for semantic search.") + + query_binary = convert_to_binary_query( + cls, + command="semantic_search", + semantic_query=query, + limit_output=handle_limit(limit), + min_score=min_score, + with_pointers=with_pointers, + key_included=key_included, + pointers_metadata=pointers_metadata, + ) + return await cls._run_query(query_binary) + + @classmethod + async def semantic_search_get_keys(cls, query: str, limit: Union[int, list] = 0, min_score: Union[float, None] = None): + """ + Semantic (vector similarity) search returning ranked keys only. + + Ranks stored items by how close their embeddings are to the embedding of + `query` and returns just the matched key and score for each hit β€” the + lightweight variant when you only need identity + ranking (e.g. to then + `get_bulk` a page, or to test membership). Use `semantic_search_get_values` + when you want the values inline. + + Semantic search must be enabled DB-wide first (see `Engine.enable_semantic_search`). + The keyspace is embedded in the background as items are written, so results + reflect whatever has been embedded so far. + + Args: + query (str): The natural-language query text to embed and search for. + limit (int | list, optional): The maximum number of ranked results to return. + An int is treated as the top-k; a two-item list + [start, stop] paginates the ranked hits. Default 0, + which lets the server apply its default top-k (10). + min_score (float, optional): Drop hits whose cosine similarity (in [-1, 1]) is + below this value. Default None (no score filter). + + Returns: + list | str: A list of ranked hits, each `{'key': ..., 'score': ...}`. + Returns a string error message if the query fails. + + Raises: + ValueError: If no query text is provided. + """ + return await cls._semantic_search(query, limit, min_score, False, False, False) + + @classmethod + async def semantic_search_get_values(cls, query: str, limit: Union[int, list] = 0, min_score: Union[float, None] = None, with_pointers: bool = False, pointers_metadata: bool = False): + """ + Semantic (vector similarity) search returning ranked hits with their values. + + Ranks stored items by how close their embeddings are to the embedding of + `query` and returns the value inline with each hit β€” the key is always + included so every value is tagged with its key. Use + `semantic_search_get_keys` when you only need keys + scores. + + Semantic search must be enabled DB-wide first (see `Engine.enable_semantic_search`). + The keyspace is embedded in the background as items are written, so results + reflect whatever has been embedded so far. + + Args: + query (str): The natural-language query text to embed and search for. + limit (int | list, optional): The maximum number of ranked results to return. + An int is treated as the top-k; a two-item list + [start, stop] paginates the ranked hits. Default 0, + which lets the server apply its default top-k (10). + min_score (float, optional): Drop hits whose cosine similarity (in [-1, 1]) is + below this value. Default None (no score filter). + with_pointers (bool, optional): If True, include pointers (foreign values) in each + returned value. Default False. + pointers_metadata (bool, optional): If True, include pointer metadata in each + returned value. Default False. + + Returns: + list | str: A list of ranked hits, each `{'key': ..., 'score': ..., 'value': ...}`. + Returns a string error message if the query fails. + + Raises: + ValueError: If no query text is provided. + """ + return await cls._semantic_search(query, limit, min_score, with_pointers, True, pointers_metadata) + @classmethod async def list_all_depending_keys(cls, key: Union[str, None] = None, custom_key: Union[str, None] = None): """ diff --git a/montycat/store_classes/persistent.py b/montycat/store_classes/persistent.py index 2f1bf14..75fc608 100644 --- a/montycat/store_classes/persistent.py +++ b/montycat/store_classes/persistent.py @@ -7,10 +7,15 @@ class persistent_kv: persistent: bool = True @classmethod - async def insert_custom_key(cls, custom_key: str): + async def insert_custom_key(cls, custom_key: str, wait_for_index: Union[bool, None] = None): """ Args: custom_key: A custom key to insert into the store. This key can be used to retrieve the value later. + wait_for_index: Per-request override for synchronous index waiting. True β†’ the + write returns only after its secondary indexes are updated + (read-your-writes); False β†’ fire-and-forget indexing; None (default) + β†’ use the DB-wide default (see Engine.enable/disable_wait_for_index). + Applies to persistent writes; in-memory already indexes synchronously. Returns: True if the insert operation was successful. Class 'str' if the insert operation failed. """ @@ -18,15 +23,16 @@ async def insert_custom_key(cls, custom_key: str): raise ValueError("No custom key provided for insertion.") custom_key_converted = convert_custom_key(custom_key) - query = convert_to_binary_query(cls, command="insert_custom_key", key=custom_key_converted) + query = convert_to_binary_query(cls, command="insert_custom_key", key=custom_key_converted, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def insert_custom_key_value(cls, custom_key: str, value: dict): + async def insert_custom_key_value(cls, custom_key: str, value: dict, wait_for_index: Union[bool, None] = None): """ Args: custom_key: A custom key to insert into the store. This key can be used to retrieve the value later. value: A Python class / dict to insert into the store. + wait_for_index: Per-request synchronous-index override; None (default) uses the DB-wide default. Returns: True if the insert operation was successful. Class 'str' if the insert operation failed. @@ -37,25 +43,26 @@ async def insert_custom_key_value(cls, custom_key: str, value: dict): raise ValueError("No custom key provided for insertion.") custom_key_converted = convert_custom_key(custom_key) - query = convert_to_binary_query(cls, command="insert_custom_key_value", key=custom_key_converted, value=value) + query = convert_to_binary_query(cls, command="insert_custom_key_value", key=custom_key_converted, value=value, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def insert_value(cls, value: dict): + async def insert_value(cls, value: dict, wait_for_index: Union[bool, None] = None): """ Args: value: A Python class / dict to insert into the store. + wait_for_index: Per-request synchronous-index override; None (default) uses the DB-wide default. Returns: Key number if the insert operation was successful. Class 'str' if the insert operation failed. """ if not value: raise ValueError("No value provided for insertion.") - query = convert_to_binary_query(cls, command="insert_value", value=value) + query = convert_to_binary_query(cls, command="insert_value", value=value, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def update_value(cls, key: Union[str, None] = None, custom_key: Union[str, None] = None, **filters): + async def update_value(cls, key: Union[str, None] = None, custom_key: Union[str, None] = None, wait_for_index: Union[bool, None] = None, **filters): """ Update the value associated with a given key in the store. If a custom key is provided, it will be converted to the appropriate format before updating. @@ -84,14 +91,15 @@ async def update_value(cls, key: Union[str, None] = None, custom_key: Union[str, if not key: raise ValueError("No key provided") - query = convert_to_binary_query(cls, command="update_value", key=key, value=filters) + query = convert_to_binary_query(cls, command="update_value", key=key, value=filters, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod - async def insert_bulk(cls, bulk_values: list): + async def insert_bulk(cls, bulk_values: list, wait_for_index: Union[bool, None] = None): """ Args: bulk_values: A list of Python objects to insert into the store. + wait_for_index: Per-request synchronous-index override; None (default) uses the DB-wide default. Returns: True if the bulk insert operation was successful. List of values that were not inserted. @@ -100,7 +108,7 @@ async def insert_bulk(cls, bulk_values: list): if not bulk_values: raise ValueError("No values provided for bulk insertion.") - query = convert_to_binary_query(cls, command="insert_bulk", bulk_values=bulk_values) + query = convert_to_binary_query(cls, command="insert_bulk", bulk_values=bulk_values, wait_for_index=wait_for_index) return await cls._run_query(query) @classmethod @@ -141,7 +149,7 @@ async def create_keyspace(cls, cache: Union[int, None] = None, compression: bool "keyspace", cls.keyspace, "persistent", "y", "distributed", "y" if cls.distributed else "n", - "cache", cache if cache else "0", + "cache", str(cache) if cache else "0", "compression", "y" if compression else "n" ], "credentials": [cls.username, cls.password] @@ -166,7 +174,7 @@ async def update_cache_and_compression(cls, cache: Union[int, None] = None, comp 'update-cache-compression', "store", cls.store, "keyspace", cls.keyspace, - "cache", cache if cache else "0", + "cache", str(cache) if cache else "0", "compression", "y" if compression else "n" ], "credentials": [cls.username, cls.password] diff --git a/montycat/store_functions/store_generic_functions.py b/montycat/store_functions/store_generic_functions.py index 8764cf1..9810bbb 100644 --- a/montycat/store_functions/store_generic_functions.py +++ b/montycat/store_functions/store_generic_functions.py @@ -118,6 +118,9 @@ def convert_to_binary_query( pointers_metadata: bool = False, limit_output: dict = {}, schema: Union[str, None] = None, + semantic_query: Union[str, None] = None, + min_score: Union[float, None] = None, + wait_for_index: Union[bool, None] = None, ) -> bytes: """ Converts parameters into a binary query format suitable for transmission. @@ -198,7 +201,9 @@ def convert_to_binary_query( "bulk_values": [normalize_bools(v) for v in bulk_values], "bulk_keys": bulk_keys, "bulk_keys_values": {k: normalize_bools(v) for k, v in bulk_keys_values.items()}, - "search_criteria": normalize_bools(search_criteria), + # `semantic_search` sends the raw query text (the engine trims it as a + # plain string); every other command sends a JSON-encoded filter map. + "search_criteria": semantic_query if semantic_query is not None else normalize_bools(search_criteria), "with_pointers": with_pointers, "volumes": volumes, "latest_volume": latest_volume, @@ -206,6 +211,16 @@ def convert_to_binary_query( "pointers_metadata": pointers_metadata, } + # Only `semantic_search` honors min_score; omit it otherwise so the wire is + # unchanged for existing commands (the engine defaults the field to None). + if min_score is not None: + query_dict["min_score"] = min_score + + # Per-request wait_for_index override for persistent writes; omit when None + # so the server falls back to its DB-wide default (existing wire unchanged). + if wait_for_index is not None: + query_dict["wait_for_index"] = wait_for_index + return orjson.dumps(query_dict) def handle_timestamps_and_pointers(search_criteria: dict) -> dict: diff --git a/setup.py b/setup.py index e6a3b00..8c876e7 100644 --- a/setup.py +++ b/setup.py @@ -2,22 +2,58 @@ setup( name='montycat', - version='1.0.4', - description='A Python client for Montycat, NoSQL database utilizing Data Mesh architecture.', + version='1.0.5', + description=( + 'Self-hosted vector database + NoSQL with built-in AI semantic search β€” the async ' + 'Python client for Montycat. A Rust-powered, AI-native Pinecone / Weaviate / Chroma ' + 'alternative for RAG, AI agents & LLM memory.' + ), packages=find_packages(), zip_safe=False, long_description=open('README.md', encoding='utf-8').read(), long_description_content_type='text/markdown', author='MontyGovernance', author_email='eugene.and.monty@gmail.com', + url='https://montygovernance.com', + license='MIT', install_requires=['orjson', 'xxhash'], + project_urls={ + 'Homepage': 'https://montygovernance.com', + 'Documentation': 'https://montygovernance.com', + 'Source': 'https://github.com/MontyGovernance/montycat_python', + 'Issues': 'https://github.com/MontyGovernance/montycat_python/issues', + 'Changelog': 'https://github.com/MontyGovernance/montycat_python/releases', + 'Docker Hub': 'https://hub.docker.com/r/montygovernance/montycat', + }, classifiers=[ + "Development Status :: 5 - Production/Stable", + "Intended Audience :: Developers", + "Intended Audience :: Science/Research", + "Intended Audience :: Information Technology", + "License :: OSI Approved :: MIT License", + "Operating System :: OS Independent", + "Natural Language :: English", "Programming Language :: Python :: 3", + "Programming Language :: Python :: 3 :: Only", + "Programming Language :: Python :: 3.9", + "Programming Language :: Python :: 3.10", + "Programming Language :: Python :: 3.11", + "Programming Language :: Python :: 3.12", + "Programming Language :: Python :: 3.13", + "Framework :: AsyncIO", "Topic :: Database", + "Topic :: Database :: Database Engines/Servers", + "Topic :: Database :: Front-Ends", + "Topic :: Scientific/Engineering :: Artificial Intelligence", "Topic :: Software Development :: Libraries", - "Intended Audience :: Developers", - "License :: OSI Approved :: MIT License", + "Topic :: Software Development :: Libraries :: Python Modules", ], - keywords="database nosql sql data-mesh cache key-value realtime montycat", + keywords=( + "vector-database vector-search semantic-search embeddings similarity-search knn hnsw " + "nosql database database-client rag retrieval-augmented-generation ai ai-agents " + "agent-memory ai-memory llm mcp self-hosted embedded-database " + "pinecone-alternative weaviate-alternative chroma-alternative qdrant-alternative " + "redis-alternative data-mesh async asyncio rust realtime key-value cache montycat" + ), python_requires='>=3.9', -) \ No newline at end of file +)