Add PyLucene integration and CPU/GPU end-to-end tests - #174
Conversation
and finish updating cuvs version
Convert the PyLucene smoke runner to a pytest-backed case matrix covering GPU CAGRA search, CAGRA-built HNSW, and forced CPU HNSW fallback across segment and force-merge topologies. Add named one-layer and three-layer CAGRA-to-HNSW codecs plus writer-path telemetry so the e2e suite can assert which CPU/GPU path was exercised. Tighten sidecar and Lucene delegate-codec validation to avoid misleading expected probe warnings while still failing on unsupported delegate codecs.
nvzm123
changed the title
| @Override | ||
| public KnnVectorsWriter fieldsWriter(SegmentWriteState state) throws IOException { | ||
| var flatWriter = FLAT_VECTORS_FORMAT.fieldsWriter(state); | ||
| System.setProperty( |
There was a problem hiding this comment.
These System.setProperty methods make sense in the context of this PR as a way to communicate between the Java and Python layers. However, they introduce unnecessary overhead for most other applications. Moreover, this approach would not work reliably in a multithreaded environment, since the same property is reused across requests. I think it would be better to separate the telemetry concerns and extract them into a dedicated method that is invoked only when telemetry data is actually needed.
There was a problem hiding this comment.
Good point! Removed the JVM-global system properties from the indexing path. Telemetry is now computed on demand from the vectors format only when the PyLucene test requests it.
cjnolet
added
improvement
| To run the PyLucene pytest smoke suite against a local PyLucene environment: | ||
|
|
||
| ```sh | ||
| ./ci/test_pylucene_smoke.sh |
There was a problem hiding this comment.
I like that we have ci scripts for this , but a user should not have to invoke ci scripts in order to run tests. Rather, we should document how to run these pytests without the need to call scripts inside the CI directory while also providing the scripts in the CI directory for GitHub actions to run automatically.
There was a problem hiding this comment.
@cjnolet I moved the user-facing logic to test_pylucene.sh and kept a thin wrapper under ci/. The current GitHub Actions workflows do not invoke it yet. Did you intend for this PR to add a PyLucene Actions job as well, or is providing the CI entry-point sufficient for now?
| ./ci/test_pylucene_smoke.sh --gpu-e2e | ||
| ``` | ||
|
|
||
| The expanded suite runs the `gpu-basic`, `gpu-segments`, `cpu-hnsw`, and |
There was a problem hiding this comment.
Great description here. How to build and run tests should really be in a separate build and install guide. I think this is okay for now, especially since we are moving cuVS-Lucene to cuVS, but it's something to consider.
| .withCagraGraphBuildAlgo(CagraGraphBuildAlgo.NN_DESCENT) | ||
| .withGraphDegree(CAGRA_GRAPH_DEGREE) | ||
| .withIntermediateGraphDegree(CAGRA_INTERMEDIATE_GRAPH_DEGREE) | ||
| .withHNSWLayer(1) |
There was a problem hiding this comment.
Wasn't this an issue that you resolved? Why layer 1?
There was a problem hiding this comment.
Yes, that issue was resolved. Layer 1 is intentional test coverage, paired with the three-layer case -- that said, I am not sure that these two particular tests will provide much utility
Move the PyLucene suite into a pytest-owned test tree and add explicit coverage for CPU HNSW, CAGRA-built HNSW, and CAGRA search. Cover segment and force-merge topologies, one- and three-layer HNSW, CAGRA search widths, deletions, vectorless documents, filtering, and single-document behavior. Verify execution paths, persisted graph configuration, and brute-force recall with deterministic vectors. Keep the shell runner focused on environment and classpath setup, and document direct pytest and full GPU execution.
Remove redundant edge cases and reuse canonical topology cases for one-layer HNSW and searchWidth=1 coverage. Add selective brute-force-validated filters across CPU HNSW, CAGRA-built HNSW, and ten-segment CAGRA search, then make cpu-hnsw-1-segment the default documented smoke case.
imotov
left a comment
imotov
left a comment
There was a problem hiding this comment.
I'm not sure how comprehensive we want these cases to be, but they all follow the same pattern - index records under some configuration, close the writer, then search. These are solid, deterministic tests, and I don't want to hold up the PR over this, but there are several important use cases where I think the next round of coverage could pay off.
Everything here is single-threaded: SerialMergeScheduler runs merges inline, IndexSearcher is built without an executor, and numMergeWorkers defaults to 1. So CuVSResources, which is managed per-thread, is only ever exercised in its simplest configuration - one thread, one resource.
In production, indexing and searching frequently overlap: new documents arriving, existing documents being updated, deletes landing, merges running (sometimes on several threads), and queries served against the index the whole time. In my experience that's where the tricky bugs live. A couple of other specific gaps:
- No updateDocument/softUpdateDocument at all, so the delete-then-add path is untested even on a single thread.
- No near-real-time search — the reader always opens after the writer closes, so a segment is never read while it's still being written, and deletes are always committed before anything searches. That should be covered by Lucene, but it would be nice to ensure that Readers still play by the book, don't leak any resources and do what Lucene expects them to do.
- No search concurrent with a merge, so a segment is never dropped from under a live searcher. Same here, Lucene should take care of it, but we should cooperate.
Create flat vector writers only when the accelerated writer consumes them. CPU fallback paths construct their own writers, so eager allocation leaked the unused flat writer for both binary and scalar quantization.
Keep the existing Lucene 102 binary-format initialization while preserving the reminder to replace version-specific partial providers in a separate initiative.
nvzm123
changed the title
3 participants
Summary
This PR adds PyLucene integration support and a pytest-owned CPU/GPU end-to-end suite for cuVS-Lucene.
The standard
cuvs-lucenejar contains only cuVS-Lucene classes and service descriptors. PyLucene supplies Lucene classes, while the basecuvs-javajar remains a separate classpath dependency. The custom assembly descriptor preserves merged service descriptors in the optional dependency jar.The PR also adds:
PyLucene test coverage
The suite explicitly covers these execution paths:
Coverage includes:
searchWidthvalues 1, 16, and 32Vectors and queries are deterministic. Expected neighbors are computed by brute force. Assertions verify:
The selective-filter cases accept roughly one quarter of each segment and retain more than
topKaccepted vectors per segment. This prevents Lucene from substituting exact scoring and ensures the approximate/native prefilter path is exercised. Filtered recall is calculated only against accepted vectors.GPU-required cases fail if cuVS is unavailable or silently falls back to CPU. CPU cases force and report Lucene’s CPU HNSW path. HNSW cases independently verify persisted graph structure.
CAGRA configurations use
graphDegree=32andintermediateGraphDegree=64. Each constructed CAGRA graph contains at least 97 vectors, and the three-layer case starts with 24,832 vectors so its third layer also retains at least 97. This avoids cuVS graph-parameter clamping warnings.Test layout
src/test/python/test_pylucene_end_to_end.pyowns pytest cases, parametrization, assertions, and reporting.src/test/python/pylucene_test_support.pycontains reusable PyLucene index and search helpers.src/test/python/conftest.pyhandles case/group selection.src/test/java/com/nvidia/cuvs/lucene/PyLuceneTestSupport.javaprovides test-only Java adapters.ci/run_pylucene_pytests.shhandles Maven artifacts, classpath setup, JVM/native-library inputs, and pytest invocation.test_pylucene.shis the repository-root convenience entry point.The default check runs jar-packaging assertions and
cpu-hnsw-1-segment:Run the complete CPU/GPU suite with:
Run the filter group with:
Pytest can also run directly once the documented PyLucene classpath environment variables are available:
Validation
Validated on an NVIDIA A10G instance:
The PyLucene runs emitted no cuVS graph-clamping or CPU-fallback warnings. The only warning was the JVM notice for the incubating vector module.
After the branch was synchronized with the current 26.10
main, static validation still passed. A fresh Maven rebuild is temporarily blocked becausecom.nvidia.cuvs:cuvs-java:26.10.0is not yet available from the configured Maven repositories.