Fix client compatibility and add filtered ANN support - #9
Draft
dovinmu wants to merge 40 commits into
Draft
Conversation
* enhance: Migrate PyMilvus orm to MilvusClient * chore: add .worktrees/ to .gitignore Signed-off-by: yangxuan <xuan.yang@zilliz.com>
- add PolarDB vector search client with FAISS_HNSW_FLAT, FAISS_HNSW_PQ, and FAISS_HNSW_SQ index types - add CLI integration with hnswflat, hnswpq, and hnswsq benchmark commands - add frontend (Streamlit) UI support with index type selection, HNSW/PQ/SQ parameter configuration
1. Fix concurrent insert memory and process cleanup 2. Add configurable load concurrency for performance cases 3. Make CLI Ctrl+C work by polling has_running() instead of blocking on concurrent.futures.wait(), which swallows SIGINT. 4. Remove perf-case insert from SerialInsertRunner 5. Ignore S608 lint rule and fix formatting Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* fix: Add back ujson in the requirements * fix the coding style Signed-off-by: yangxuan <xuan.yang@zilliz.com>
…client - Add --region CLI parameter (required) for region-based API routing - Change --api-base-url to optional override for private networking - Rename write(columns=...) to write(upsert_columns=...) per current SDK - Fix docstring referencing wrong database name Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
…ults - Defer tpuf.Turbopuffer client creation to init() to avoid pickle errors with ProcessPoolExecutor(spawn) - Cast search result IDs to int for ground truth recall comparison - Update leaderboard_v2.json with 20 TurboPuffer filter performance entries Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
…rboard Merge 22 individual TurboPuffer result files into single consolidated result file. Add streaming benchmark entries (500/1000 rows/s) to leaderboard_v2_streaming.json. Normalize TurboPuffer db_name and label in both leaderboard files. Signed-off-by: jamesgao-jpg <james.gao@zilliz.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Milvus results (16c64g, force_merge, v2.6.14): - 1M Cohere: SQ4U+FP16 (sweep refine_k) + SQ8 (sweep ef), 8 points each - 10M Cohere: SQ4U+FP16 + SQ8 (sweep ef), 8 points each - Total 32 benchmark configurations ElasticCloud and ZillizCloud results from standard benchmark runs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rename ElasticCloud and ZillizCloud result files from 20260209 to 20260403 and update task_label to standard_20260403 for consistency with Milvus results. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update db_name/label in leaderboard_v2_streaming.json to match leaderboard_v2.json after force_merge became the default: - Milvus: 16c64g-sq8 -> 16c64g-sq8-force_merge - ElasticCloud: 8c60g -> 8c60g-force_merge This fixes the website failing to associate streaming and vector search results due to mismatched db_name keys. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace ZillizCloud-8cu-perf case_id=4/5 data with new Cardinal backend benchmark results (level 1-9, 1M and 10M datasets, v2026.4). Remove force_merge entries as Cardinal uses unified 4-segment architecture for 10M. New results show significant QPS improvement: - 1M: 13,316 QPS (was 9,704) at recall 0.938 - 10M: 7,385 QPS (was 3,957) at recall 0.938 Sort all leaderboard entries by (db_name, dataset, filter_ratio, qps DESC) to fix line chart rendering. Remove one SQ4U 1M outlier (recall=0.84). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1. Upgrade pydantic to 2.x 2. Remove results/ from .gitignore, those files need to track 3. fix the coding styles in the results Signed-off-by: yangxuan <xuan.yang@zilliz.com>
…illiztech#751) Populate insert_duration, optimize_duration, load_duration for all entries in result_20260403 files. Previously only the first entry per index had values while the rest were 0.0. Milvus (re-measured on 2.6-opt-v2): - 1M SQ4U: insert=129.8s, optimize=152.2s, load=282.0s - 1M SQ8: insert=119.5s, optimize=235.9s, load=355.4s - 10M SQ4U/SQ8: copied from existing first-entry values ZillizCloud (from prior build runs): - 1M: insert=246.7s, optimize=101.2s, load=347.9s - 10M: insert=2450.8s, optimize=136.9s, load=2587.8s Co-authored-by: Ubuntu <ubuntu@ip-10-15-14-123.us-west-2.compute.internal> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
…lliztech#754) Update result_20260403_standard_zillizcloud.json to use the latest validated build timings from recent reruns for case_id=5 (1M) and case_id=4 (10M), including insert_duration, optimize_duration, and load_duration. Co-authored-by: Ubuntu <ubuntu@ip-10-15-14-123.us-west-2.compute.internal> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: add VectorChord support and VCHORDRQ index type * feat: add VectorChordRQ command to CLI * feat: add VectorChord support to README * feat: add VectorChordGraph support and configuration * feat: add max_scan_tuples parameter to VectorChordGraph * feat: enhance VectorChord with improved type safety and search functionality * feat: add vectorchord extension creation on connection Co-authored-by: edgar-p <edgar.p@kakaocorp.com>
zilliztech#760) * fix(pgvector): normalize index_type to lowercase in _create_index to match PostgreSQL access method names PostgreSQL pgvector extension registers index access methods in lowercase (e.g. "hnsw", "ivfflat"), but the frontend passes IndexType.HNSW.value which is uppercase "HNSW", causing "access method HNSW does not exist" error. * Fix index type usage in pgvector.py Replaced index_param['index_type'] with index_type_lower for consistency. * add comment sign '#' I have added the # before [FIX]
Adds a complete Apache Pinot client for VectorDBBench. Index types: HNSW (Lucene), IVF_FLAT, IVF_PQ, IVF_ON_DISK Metrics: L2, IP, COSINE Filters: NumGE, StrEqual Optional dep: pip install "vectordb-bench[pinot]" Parallel loading: thread_safe=True — each worker thread maintains its own row buffer and flushes to Pinot via a fresh HTTP session. Since Pinot's ingestFromFile is synchronous (blocks until HNSW index is built, ~6 min per 100K×768D segment), concurrent flushes across threads reduce load time significantly vs sequential flushing. Benchmark results: Small dataset (OpenAI 50K, 768D, L2): HNSW: 798 QPS, recall=1.000 IVF_FLAT: 800 QPS, recall=1.000 IVF_PQ: 795 QPS, recall=1.000 IVF_ON_DISK: 691 QPS, recall=1.000 Large dataset (Cohere 1M, 768D, COSINE): HNSW m=16: 74 QPS, recall=0.982 Filter benchmark (Cohere 1M, COSINE, HNSW m=32): 1% NumGE: 71 QPS, recall=0.977 99% NumGE: 97 QPS, recall=0.649 Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…oud commands (zilliztech#761) * fix: support self-hosted Elasticsearch via --host/--port in elasticcloud commands ElasticCloudConfig previously required cloud_id, so the elasticcloudhnsw* subcommands could only target Elastic Cloud. Users benchmarking self-hosted stock Elasticsearch had no working path: tencentelasticsearch accepts host/port but forces Tencent's vsearch index_options type, which stock ES rejects with "Unknown vector index options type [vsearch]". Extend ElasticCloudConfig with scheme/host/port/user fields (mutually exclusive with cloud_id) and expose them on all four ElasticCloudHNSW* CLI subcommands. Existing cloud_id callers are unchanged. Refs zilliztech#758 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * style: apply black formatting to elastic_cloud/config.py Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…lliztech#763) * Fix: Map "ivf_flat" to "ivfflat" for pgvector index access method - IndexType.IVFFlat.value="IVF_FLAT" → .lower()="ivf_flat" caused SQL to fail with "access method 'ivf_flat' does not exist" - pgvector PostgreSQL extension expects "ivfflat" (no underscore), not "ivf_flat" - Added explicit mapping after lowercase normalization: if index_type_lower == "ivf_flat": index_type_lower = "ivfflat" * style(pgvector): fix comment wrapping and remove commented code --------- Co-authored-by: rnagaraju <rnagaraju@zeomega.com>
…lliztech#764) For non-thread-safe DBs (e.g. PgVector), ConcurrentInsertRunner clamps max_workers to 1, so there is always exactly one worker thread. There is no need to deepcopy self.db per thread — the single worker can use self.db directly via the connection already opened by task()'s `with self.db.init():`. The original code called deepcopy(self.db) inside _get_thread_db() after task() had already opened a live psycopg C-extension Connection on self.db. C-extension objects cannot be deep-copied, causing: TypeError: no default __reduce__ due to non-trivial __cinit__ Fix: remove the deepcopy branch entirely. All workers (thread-safe or not) now use self.db directly; thread-safety is guaranteed for non-thread-safe DBs by the max_workers=1 clamp. Also clean up stale comments in pgvector.py left over from zilliztech#760/zilliztech#763. Adds tests/test_pgvector.py with: - unit test that reproduces the bug (fails on original, passes on fix) - e2e regression test via ConcurrentInsertRunner + OpenAI 50K dataset See also: zilliztech#756 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Add label filtering support to pgdiskann client * Refactor pgdiskann filtering logic * Refactor: remove unrelated function * style: apply black formatting to pgdiskann.py * fix: remove trailing whitespace and fix import sorting * docs: add comments for label naming and vector storage optimization * Revert "docs: add comments for label naming and vector storage optimization" This reverts commit d10b296. --------- Co-authored-by: Eesha Faisal <eesha.faisal@emumba.com>
…ech#766) - Migrate DB config validators to pydantic v2; list all empty fields instead of raising on first; consolidate via `_extra_empty_skip`. - Surface missing client modules at config render time as `{DB} needs `{module}` but it is not installed.` - Replace streamlit-autorefresh with native `@st.fragment(run_every)` so live progress does not block UI. - Bump streamlit to 1.47+ (picks up streamlit#11890 fragment fix); switch to native `st.switch_page`, drop `streamlit_extras`. - Migrate deprecated `use_container_width=True` to `width="stretch"`. - Patch tornado `write_message` to consume expected `WebSocketClosedError` on tab-close races (streamlit#9787). - Add contract test: each DB enum resolves config_cls/init_cls or raises ModuleNotFoundError. See also: zilliztech#446 Signed-off-by: yangxuan <xuan.yang@zilliz.com>
ConcurrentInsertRunner previously defaulted to mp.cpu_count(), spawning one worker per CPU when load_concurrency was unset. On high-core hosts this opens many parallel client connections, saturating modest DBs / network paths and yielding worse load throughput than a smaller, steadier worker count. Cap the unset default to min(cpu_count, 4). Explicit load_concurrency from CLI / config / submitTask still wins. Signed-off-by: yangxuan <xuan.yang@zilliz.com>
) Add a new vector database backend for SeekDB, connecting via mysql-connector-python over the MySQL wire protocol. Key components: - seekdb.py: VectorDB implementation with heap-organized table, HNSW vector index, and version-aware optimize() that calls dbms_index_manager.refresh() on SeekDB >= 1.3.0 - config.py: DBConfig with host/port/user/password/database and SeekDBHNSWConfig with m/ef_construction/ef_search parameters - cli.py: Click command `SeekDBHNSW` for command-line benchmarks Registration: - Add SeekDB to the DB enum in backend/clients/__init__.py with lazy imports for init_cls, config_cls, and case_config_cls - Register SeekDBHNSW CLI command in cli/vectordbbench.py - Add seekdb optional dependency in pyproject.toml (pip install vectordb-bench[seekdb]) Filter support: - NonFilter and NumGE (id >= N) filters are supported - StrEqual (label filter) is intentionally excluded since the table schema only has id and embedding columns Thread safety: - mysql.connector is not thread-safe (thread_safe = False). ConcurrentInsertRunner uses max_workers=1 accordingly - rate_runner.py handles SeekDB specially: copies the db object, resets the connection, and calls init() per worker thread Co-authored-by: liuhao6741 <liuhaobupt@foxmail.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
… cosine support (zilliztech#776) * feat(oceanbase): configurable index params, KEY partitioning, HNSW_BQ cosine support - Add --create-index-parallel CLI option (default 16) - Add --extra-info-max-size CLI option (default 32, set 0 to omit) - Add --partitions CLI option for KEY partitioning (default 0, no partition) - HNSW_BQ: remove forced L2 for cosine, now supports cosine natively - need_normalize_cosine returns False for all index types - pyproject.toml: add pyyaml dependency, fix packages.find to include all subpackages * fix(oceanbase): declare thread_safe=False to prevent cursor sharing across threads * fix: restore seekdb dependency accidentally removed
…lliz-upstream-with-legacy-fallback
Two fixes for invalid 1M benchmark rows: - The write-readiness probe no longer deletes its doc. The tombstone permanently disabled the all-docs-visible fast path, so every dense query materialized the full live-doc set as a positive filter (~40ms/query at 1M docs, linear in table size). - TEMPORARY: pace inserts on the server's catch-up backlog (catch_up_target_sequence - applied) when using async sync levels. Unthrottled write-sync loads outrun dense catch-up + LSM compaction and wedge the write path mid-load. Env-tunable via ANTFLY_VDBBENCH_MAX_LAG_SEQ / RESUME_LAG_SEQ / PACE_EVERY; remove once the server applies its own ingest backpressure.
dovinmu
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.