Repository navigation
chore(vendor): track zvec v0.8.1; feat: adapt to grouped search and IVF PQ - #25
Merged
Merged
Conversation
8397071 (v0.8.0-beta2) -> 30d662d (v0.8.1), 30 upstream commits. c_api.h is not byte-identical: 4470 -> 4540 lines, +4 functions, +2 types, +2 constants, nothing removed and no breaking signature change. - feat(turbo): support product quantization end to end (#833) adds zvec_index_params_set_quantizer_pq / _get_quantizer_pq, zvec_pq_quantizer_params_t, ZVEC_PQ_QUANTIZER_PARAMS_DEFAULT and ZVEC_QUANTIZE_TYPE_PQ, and rewords the enable_rotate doc: with PQ the same flag now selects a learned OPQ rotation. - feat(c-api): expose grouped vector search (#845) adds zvec_collection_group_by_query, zvec_group_results_free and zvec_group_result_t. zvec_group_by_vector_query_* was already bound, but until now a grouped query could only be built, never executed. - zvec_collection_multi_query's doc now spells out the NULL / zero-result contract. Comment only, no signature change. Beyond the header: DiskANN is now built for Intel macOS and Windows ARM64 (#874), and RaBitQ for Linux/macOS/Android/iOS ARM64 plus Intel macOS (#832), which leaves Windows ARM64 the only prebuilt target without RaBitQ. zvec/NOTICE is unchanged, so no SDK NOTICE needs regenerating. The Rust adaptation follows in the next commit.
Grouped search. GroupBySearchQuery could be built but never executed, because the C API only gained zvec_collection_group_by_query in v0.8.1. Collection::group_by_query now runs it and returns Vec<SearchGroup>, each group carrying its grouping value and its documents in similarity order. zvec_group_results_free is the only release path for a zvec_group_result_t array, and that array owns everything inside it: the grouping values, each group's document pointer array, and the documents themselves. collect_group_results therefore moves every document into an owning Doc, releases the pointer array with zvec_free, and nulls docs/doc_count before handing the shell back, so the engine is left with only the strings and the outer array to free and cannot destroy a document twice. grouped_results_outlive_the_query_and_the_collection pins that contract down: the groups stay readable after both handles are gone. Product quantization. PqQuantizerParams mirrors zvec_pq_quantizer_params_t, its Default reproduces ZVEC_PQ_QUANTIZER_PARAMS_DEFAULT, and IndexParams::set_quantizer_pq / quantizer_pq wrap the new setter and getter. QuantizeType::Pq (8) joins the enum mirror, its From mapping and the roundtrip test. The C API accepts PQ on IVF params only, which the wrapper documents and the tests assert along with the num_chunk / num_bits / fast_scan / OPQ validation rules. set_quantizer_enable_rotate's docs said rotation "has no effect with other quantize types"; since #833 it selects learned OPQ with PQ, and it writes the same QuantizerParam field that set_quantizer_pq writes, so both docs now say which call wins. Platform gates moved without the header (#874, #832). The DiskANN and IVF RaBitQ end-to-end tests already skip on NotSupported, so they widen their own coverage with no code change -- on Apple Silicon the RaBitQ test now runs instead of skipping. Their comments and both READMEs' availability prose are updated to match, and the Windows ARM64 note now says RaBitQ is the only thing missing there. Also adds the group_by_search example (make example-group-by), an IVF PQ section to schema_builder, and grouped-search plus PQ docs to README.md and README_CN.md. Verified against the zvec v0.8.1 osx-arm64 SDK, whose c_api.h is byte-identical to the submodule's: cargo fmt --check, clippy --workspace --all-targets -D warnings, build --workspace --all-targets and make test-all (unit + integration + doc) all pass.
Bump the workspace from 0.8.0 to 0.8.1 to match the v0.8.1 tag the vendor submodule now tracks, along with the zvec-rust-sys path dependency requirement, Cargo.lock and the dependency snippets in both READMEs. Additive release: grouped vector search (Collection::group_by_query, SearchGroup) and IVF product quantization (PqQuantizerParams, IndexParams::set_quantizer_pq / quantizer_pq) are new surface, no C-API symbol was dropped and no wrapper was removed, so consumers keep compiling unchanged. One caveat for the semver purists: QuantizeType gained a Pq variant and the enum is not #[non_exhaustive], so a caller matching it exhaustively without a wildcard arm has to add one. The version still follows the upstream zvec release it tracks, as 0.8.0 did. release.yml and publish-crates.yml re-check these strings against the tag, so this commit has to be on main before v0.8.1 is tagged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bumps the
vendor/zvecsubmodule to the upstream v0.8.1 release and adapts the Rust bindings to it. Also bumps the workspace to 0.8.1 so the SDK version matches the tag the submodule tracks.Submodule move
8397071(v0.8.0-beta2) →30d662d(v0.8.1) — 30 upstream commits.Upstream commits
Upstream also notes an upgrade caveat of its own: collections with pending WAL recovery must first be opened and closed in writable mode before being opened read-only (#840). That is engine behaviour, not an SDK surface change.
C-API delta
c_api.his not byte-identical: 4470 → 4540 lines.Added by #845 (grouped vector search in the C API):
zvec_collection_group_by_queryzvec_group_results_freezvec_group_result_tAdded by #833 (product quantization end to end):
zvec_index_params_set_quantizer_pqzvec_index_params_get_quantizer_pqzvec_pq_quantizer_params_tZVEC_PQ_QUANTIZER_PARAMS_DEFAULTZVEC_QUANTIZE_TYPE_PQDoc-only changes, no ABI impact:
zvec_index_params_set_quantizer_enable_rotate(rotation now means learned OPQ under PQ) andzvec_collection_multi_query(the NULL / zero-result contract is now spelled out).Nothing was removed, so no binding had to be deleted.
Breaking?
Additive. No wrapper was removed and no signature changed, so consumers keep compiling unchanged.
One caveat for the semver purists:
QuantizeTypegained aPqvariant and the enum is not#[non_exhaustive], so a caller matching it exhaustively without a wildcard arm has to add one. The SDK version still follows the upstream zvec release it tracks, as 0.8.0 did.What this adds
Grouped search.
GroupBySearchQuerycould already be built, but the C API only gained the execution call in v0.8.1 — so until now a grouped query could never be run.Collection::group_by_querycloses that gap and returnsVec<SearchGroup>, each group carrying its grouping value and its documents in similarity order.zvec_group_results_freeis the only release path for azvec_group_result_tarray, and that array owns everything inside it: the grouping values, each group's document pointer array, and the documents themselves.collect_group_resultstherefore moves every document into an owningDoc, releases the pointer array withzvec_free, and nullsdocs/doc_countbefore handing the shell back — so the engine is left with only the strings and the outer array to free and cannot destroy a document twice.grouped_results_outlive_the_query_and_the_collectionpins that contract down: the groups stay readable after both handles are gone.Product quantization.
PqQuantizerParamsmirrorszvec_pq_quantizer_params_t, itsDefaultreproducesZVEC_PQ_QUANTIZER_PARAMS_DEFAULT, andIndexParams::set_quantizer_pq/quantizer_pqwrap the new setter and getter. The C API accepts PQ on IVF params only, which the wrapper documents and the tests assert along with thenum_chunk/num_bits/fast_scan/ OPQ validation rules.set_quantizer_enable_rotate's docs also said rotation "has no effect with other quantize types", which is no longer true under PQ and writes the sameQuantizerParamfield, so both docs now say which call wins.Platform gates moved without the header (#874, #832). DiskANN is now built for Intel macOS and Windows ARM64, and RaBitQ for Linux/macOS/Android/iOS ARM64 plus Intel macOS, leaving Windows ARM64 the only prebuilt target without RaBitQ. The DiskANN and IVF RaBitQ end-to-end tests already skip on
NotSupported, so they widen their own coverage with no code change; their comments and both READMEs' availability prose are updated to match, and the Windows ARM64 note now says RaBitQ is the only thing missing there.zvec/NOTICEis unchanged, so no SDK NOTICE needs regenerating.Also: the
group_by_searchexample (make example-group-by), an IVF PQ section inschema_builder, and grouped-search plus PQ docs inREADME.mdandREADME_CN.md.Verification
Against a from-source build of v0.8.1 (
make zvec-build, 581s,[100%] Built target zvec_c_api, no errors), not a downloaded prebuilt:RABITQ_SUPPORTED: ON(it was OFF on macOS ARM64 before v0.8.1),DISKANN_SUPPORTED: ON,BUILD_C_BINDINGS: ONnmconfirms all four new symbols are exported by the built libraryv0.8.1— the submodule is pinned at the tag, not a random commit, sogit describeyields no suffixcargo fmt --check,cargo clippy --workspace --all-targets -- -D warnings,cargo build --workspace --all-targets: passmake test-all(unit + integration + doc): pass — 125 lib unit tests plus every integration and doc testSymbol coverage: all four new
zvec_*functions are bound; nothing is bound that the new header no longer declares; and the not-bound list is identical to before the bump (its one real entry,zvec_doc_add_field_by_struct, is pre-existing and out of scope here).Commits
chore(vendor): track zvec v0.8.1— gitlink onlyfeat: adapt to zvec v0.8.1 grouped search and IVF product quantizationchore(release): 0.8.1— version strings onlyEach commit is independently buildable: the version bump is isolated in the last one, so the workspace version and the
zvec-rust-syspath-dependency requirement stay consistent at every step.Before tagging
release.ymlandpublish-crates.ymlre-verify the version strings against the tag; those are committed here. Runner labels inbuild-prebuilt.ymlare all current (macos-15,macos-15-intel,ubuntu-24.04,ubuntu-24.04-arm,windows-2025,windows-11-arm), so all eight prebuilt legs should produce assets. crates.io publication stays a manual step.