Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions assets/ruvector/packages/ruvector-pq-search.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions assets/ruvector/packages/ruvector-rairs.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions crates/ruvector-core/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,10 @@ authors.workspace = true
repository.workspace = true
readme = "README.md"
description = "High-performance Rust vector database core with HNSW indexing"
homepage = "https://ruv.io"
documentation = "https://docs.rs/ruvector-core"
keywords = ["vector-database", "hnsw", "ann", "embeddings", "quantization"]
categories = ["database-implementations", "algorithms", "data-structures"]

[dependencies]
# Core functionality
Expand Down
2 changes: 2 additions & 0 deletions crates/ruvector-core/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# Ruvector Core

<p align="center"><img src="https://raw.githubusercontent.com/ruvnet/RuVector/main/assets/ruvector/packages/core.svg" alt="ruvector-core: HNSW vector database core" width="100%"></p>

[![Crates.io](https://img.shields.io/crates/v/ruvector-core.svg)](https://crates.io/crates/ruvector-core)
[![Documentation](https://docs.rs/ruvector-core/badge.svg)](https://docs.rs/ruvector-core)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
Expand Down
2 changes: 2 additions & 0 deletions crates/ruvector-pq-search/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@ authors.workspace = true
repository.workspace = true
description = "Product Quantization with Asymmetric Distance Computation (ADC) for compressed approximate nearest-neighbor search — Flat PQ, IVF+PQ, and Residual-corrected PQ variants in safe Rust"
readme = "README.md"
homepage = "https://ruv.io"
documentation = "https://docs.rs/ruvector-pq-search"
keywords = ["vector-search", "ann", "product-quantization", "compression", "approximate-search"]
categories = ["algorithms", "data-structures", "science"]

Expand Down
43 changes: 24 additions & 19 deletions crates/ruvector-pq-search/README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
# ruvector-pq-search

<p align="center"><img src="https://raw.githubusercontent.com/ruvnet/RuVector/main/assets/ruvector/packages/ruvector-pq-search.svg" alt="ruvector-pq-search: product quantization with ADC search" width="100%"></p>

**Product Quantization with Asymmetric Distance Computation (ADC) for compressed approximate nearest-neighbor search — 64× compression in safe Rust.**

[![crates.io](https://img.shields.io/crates/v/ruvector-pq-search)](https://crates.io/crates/ruvector-pq-search)
Expand All @@ -19,35 +21,38 @@ a fraction of the storage and compute cost.

| Variant | Description | Recall | Memory |
|---------|-------------|--------|--------|
| `FlatPq` | Flat scan over PQ codes with ADC | ≥0.85 | **64× less** |
| `IvfPq` | IVF coarse quantizer + PQ codes | ≥0.80 | **64× less** |
| `ResidualPq` | Residual-corrected PQ for higher recall | ≥0.90 | **64× less** |
| `FlatPqIndex` | Flat scan over PQ codes with ADC | ≥0.85 | **64× less** |
| `IvfPqIndex` | IVF coarse quantizer + PQ codes | ≥0.80 | **64× less** |
| `ResidualPqIndex` | Residual-corrected PQ for higher recall | ≥0.90 | **64× less** |

## Quick start

```rust
use ruvector_pq_search::{FlatPq, PqConfig, PqIndex};

let cfg = PqConfig {
dims: 128,
m_subvecs: 8, // 8 sub-vectors of 16 dims each
k_centroids: 256, // 256 centroids per sub-vector (8-bit codes)
n_train: 50_000, // training set size for k-means
};
use ruvector_pq_search::{FlatPqIndex, PqCodebook, PqConfig, PqSearch};

let mut idx = FlatPq::new(cfg);
let dim = 32;
// 1,000 vectors of 32 dims, flat row-major (use your embeddings here).
let data: Vec<f32> = (0..1_000 * dim).map(|i| ((i * 7919) % 1000) as f32 / 1000.0).collect();

// Train on representative data
let train_data: Vec<Vec<f32>> = /* ... */;
idx.train(&train_data);
// 8 sub-spaces x 16 centroids per sub-space (one u8 code each), Lloyd training.
let codebook = PqCodebook::train(PqConfig { m: 8, k: 16, iterations: 10, seed: 42 }, &data, dim);

// Insert (stored as 8-byte PQ codes)
idx.insert(0, &[0.1_f32; 128]);
// Or train the same codebook layout with e-means (ruvector-e-means):
let options = ruvector_e_means::Config { clusters: 16, epochs: 3, seed: 42, ..Default::default() };
let e_means_codebook = PqCodebook::train_e_means(8, &data, dim, &options).unwrap();

// Search with ADC (query stays at full precision)
let results: Vec<(u64, f32)> = idx.search(&[0.1_f32; 128], 10);
let mut index = FlatPqIndex::new(codebook);
for row in data.chunks_exact(dim) {
index.insert(row);
}
let hits = index.search(&data[..dim], 5); // ADC: the query stays full precision
println!("{:?}", hits.iter().map(|h| h.id).collect::<Vec<_>>());
```

`IvfPqIndex::new_e_means` trains the coarse lists with e-means too, and
`try_insert` / `try_search` validate shape and finiteness first. Lloyd training
stays the default.

## Compression tradeoffs

| Method | Compression | Recall@10 | Build time |
Expand Down
3 changes: 3 additions & 0 deletions crates/ruvector-rairs/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@ description = "RAIRS IVF: Redundant Assignment with Amplified Inverse Residual
authors = ["ruvnet", "claude-flow"]
license = "MIT OR Apache-2.0"
repository = "https://github.com/ruvnet/ruvector"
homepage = "https://ruv.io"
documentation = "https://docs.rs/ruvector-rairs"
readme = "README.md"
keywords = ["ann", "ivf", "vector-search", "nearest-neighbor", "ruvector"]
categories = ["algorithms", "data-structures"]

Expand Down
45 changes: 45 additions & 0 deletions crates/ruvector-rairs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# ruvector-rairs

<p align="center"><img src="https://raw.githubusercontent.com/ruvnet/RuVector/main/assets/ruvector/packages/ruvector-rairs.svg" alt="ruvector-rairs: IVF with redundant assignment" width="100%"></p>

Inverted-file (IVF) indexes that keep recall high at low `nprobe`. Each vector
goes to its primary list and to a residual-amplified secondary list, so
neighbours near a cell boundary are still found when few lists are probed.

| Index | Assignment | Layout |
|---|---|---|
| `IvfFlat` | single | flat lists (baseline) |
| `RairsStrict` | primary + secondary | flat lists |
| `RairsSeil` | primary + secondary | shared 32-vector blocks, deduplicated at query time |

All three implement `AnnIndex`. Train with Lloyd's k-means (`train`) or with the
shared e-means optimizer (`train_e_means`, via `ruvector-e-means`).

## Quick start

```rust
use ruvector_rairs::{AnnIndex, RairsSeil};

let dim = 16;
// 2,000 vectors of 16 dims (use your embeddings here).
let corpus: Vec<Vec<f32>> = (0..2_000)
.map(|i| (0..dim).map(|d| ((i * 31 + d * 7) % 97) as f32 / 97.0).collect())
.collect();

// 32 lists, 10 Lloyd iterations, seed 42, RAIR lambda 1.0.
let mut index = RairsSeil::new(dim, 32, 10, 42, 1.0);
index.train(&corpus).unwrap();
// Or: index.train_e_means(&corpus, &ruvector_e_means::Config { clusters: 32, ..Default::default() })
index.add(&corpus).unwrap();

let hits = index.search(&corpus[0], 5, 4).unwrap(); // k = 5, nprobe = 4
println!("{:?}", hits.iter().map(|h| h.id).collect::<Vec<_>>());
```

The "RAIRS / SEIL" names follow the design notes in ADR-193; treat this crate as
an original implementation of redundant assignment and judge it on the
benchmarks in `src/main.rs`.

## License

MIT. Part of [RuVector](https://github.com/ruvnet/ruvector), see [ruv.io](https://ruv.io).
Loading