Skip to content

feat: add shared e-means clustering and ecosystem integrations - #1489

Merged
ruvnet merged 2 commits into
mainfrom
codex/e-means-ecosystem-integrations
Nov 11, 2026
Merged

ruvnet merged 2 commits into
mainfrom
codex/e-means-ecosystem-integrations

Conversation

@ruvnet

@ruvnet ruvnet commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Adds a shared implementation of e-means (arXiv:2611.09501v1) for centroid/codebook training, with one Rust numerical kernel compiled to portable WASM. Existing Lloyd and neural-training defaults remain available.

Changes

  • Add ruvector-e-means 0.1.0 with scaling-law defaults, EMA updates, cosine scheduling, weighted donor revival, both main finalizers, checked PQ/IVF modules, and exact distance pruning.
  • Add @ruvector/e-means 0.1.0 with typed-array result transfer, explicit runtime/quantization/IVF/MetaHarness/Cloudflare subpaths, and installed-archive qualification.
  • Wire explicit training into core PQ, PQ-search, IVF+PQ, and the IVF/RAIRS/SEIL family; expose clustering subpaths from the core, main, WASM, and umbrella npm packages.
  • Add bounded, held-out evaluation/tuning with actual Darwin 0.13.0 and authenticated Workers using static WASM, centroid-only R2 persistence, and explicit Vectorize export helpers.
  • Record ADR-359/360, benchmark data, and qualification receipts. The author measured 1.84× (clustered native fit) and 2.41× (npm embedding fixture) against an earlier build that is not in the tree. Review reproduction against a pruning-off build of this tree: 1.58× and 2.26×, 1.00× at 768D and 8D PQ, 2.54× on the revival fixture; outputs are bit-identical to the pruning-off build and match the digests recorded in the artifact.

Validation

  • 50 targeted Rust tests across the shared crate, PQ-search, and RAIRS; strict Clippy across all targets.
  • 44 shared npm tests, strict TypeScript, installed consumers with and without the optional SDK, and actual Darwin evolution.
  • Installed consumer subpaths for @ruvector/core, ruvector, @ruvector/all, and the current @ruvector/wasm source.
  • Real local workerd execution of the packaged static WASM and R2 roundtrip; Vectorize binding-contract fixtures.
  • Crate archive verification, packaged native Rust 1.81 check, and isolated integration-tool lockfile audit with zero reported advisories.

Qualification is targeted: unrelated consumer root/prepack release builds and the whole monorepo matrix are not certified. The existing Darwin-only npm workspace blocks a whole-workspace Linux install; the integration tooling has its own reviewed lockfile. Browser Chromium qualification remains pending after independent startup failures.

Release order

The shared crate and npm package are not published by this PR. Publish and verify both 0.1.0 dependencies before releasing dependent consumer versions. Hosted Cloudflare deployment and live Vectorize writes remain pending.

Companion consumer change: ruvnet/ruv-FANN#227 (draft until the shared registry dependencies are published).

Review (lane 2026-11-11aq)

Fixed on the branch (commit 9864a65):

  • ruvector-core ProductQuantized::encode had moved from f32 to f64 ordering, which changes codes for existing users at f32 near-ties (1 in 80,000 random 256-entry lookups flipped). It now keeps the f32 ordering for finite distances and uses f64 only when an f32 distance overflows. A regression test pins a tie case.
  • New 0.x edges pinned exactly (rand_chacha =0.3.1, rand_core =0.6.4, ruvector-e-means =0.1.0).
  • @ruvector/core lists e-means as a dependency rather than optional, since clustering.mjs re-exports it statically.
  • tuneClustering threw through ruvector/clustering because ruvector resolves Darwin ^0.13.0 to 0.13.1 and e-means required exactly 0.13.0. Both versions run; the peer and guard now accept exactly 0.13.0 or 0.13.1.
  • cloudflare/build loads esbuild lazily, with a clear error when it is missing.
  • ADR-356 metadata, animated SVG, and README text that no longer presents unreleased ruv-FANN or consumer facades as available.
  • ADR-359 labels the author numbers and adds the reproduction.

Verified: the shipped .wasm rebuilds byte-for-byte from source (Rust 1.99.1, wasm-bindgen 0.2.125). Rust tests, strict clippy, cargo package, 44 npm tests, typecheck, packed-consumer and ecosystem-consumer tests pass. No secrets in fixtures or receipts.

CI note: Build, npm-publish-pipeline (npm/packages/ruvector) and optional-deps-resolvable-on-npm fail with E404 because consumers depend on the unpublished @ruvector/e-means@0.1.0. These failures come from this PR (they are green on main), so they block merge until the leaf package is on npm. cargo deny fails identically on main (#1487).

🤖 Generated with claude-flow

…add release metadata

- ruvector-core ProductQuantized::encode keeps the existing f32 ordering for
  finite distances; f64 is used only when an f32 squared distance overflows.
  Adds a regression test with a pair that ties in f32 but not in f64.
- Pin new 0.x edges exactly: rand_chacha =0.3.1, rand_core =0.6.4 and
  ruvector-e-means =0.1.0 in core, pq-search and rairs.
- @ruvector/core: e-means moves from optionalDependencies to dependencies,
  since clustering.mjs re-exports it statically and has no fallback.
- e-means MetaHarness accepts Darwin 0.13.0 and 0.13.1 (both run). ruvector
  resolves ^0.13.0 to 0.13.1, which made tuneClustering throw through
  ruvector/clustering. The receipt reports the version actually loaded.
- cloudflare/build imports esbuild lazily and explains a missing install.
- ADR-356 metadata for ruvector-e-means and @ruvector/e-means (homepage, bugs,
  keywords, description, animated SVG). READMEs no longer state unreleased
  ruv-FANN or consumer facades as available; crate README links are absolute.
- ADR-359 labels the author's speedups as measured against an earlier build
  that is not in the tree, and adds an independent reproduction of the
  pruning change with matching output digests.

Co-Authored-By: claude-flow <ruv@ruv.net>
@ruvnet

ruvnet commented Nov 11, 2026

Copy link
Copy Markdown
Owner Author

Main-red evidence before merge (head 9864a65): 62 pass, 4 skipped, 1 fail. The only red job is cargo deny (license + source + ban policy) in supply-chain. It fails with the same single error[unsound] (xattr zero-length buffer) as main's latest supply-chain run 43031735623 at 39a104e, tracked in #1487. Build and npm-publish-pipeline (npm/packages/ruvector) failed earlier only with E404 on @ruvector/e-means@0.1.0. They passed on re-run after ruvector-e-means 0.1.0 (crates.io) and @ruvector/e-means 0.1.0 (npm) were published from this head.

🤖 Generated with claude-flow

@ruvnet
ruvnet merged commit ea6620f into main Nov 11, 2026
91 of 95 checks passed
ruvnet added a commit that referenced this pull request Nov 11, 2026
… ruvector-core 2.5.2, ruvector-pq-search 2.3.1, ruvector-rairs 0.1.1 (#1491)

Ships the e-means clustering subpaths and training APIs from #1489, now that
ruvector-e-means 0.1.0 and @ruvector/e-means 0.1.0 are published.

@ruvector/all: #1489 added @ruvector/e-means as a dependency and new files
without updating the umbrella's manifest test, so two tests failed. e-means
is now an ESM member (all.loadEMeans()), and the test accepts exact pins for
new 0.x edges alongside caret ranges.

ruvector: test/rabitq-optional.js guards the hard-dependency list; it now lists
@ruvector/e-means, the deliberate edge #1489 added for ruvector/clustering.mjs.

Unreleased changes also shipped by the crate bumps:
- ruvector-core: generated-ID reservation, live HNSW identity retention,
  memory batch validation (each with tests), fuzz lockfile refresh.
- ruvector-pq-search: Rust 1.88 MSRV fallback in codebook.rs.
- ruvector-rairs: clippy allowance, crates.io keyword length fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant