New Dataset API Clarifying Ownership #2 - #2391
Open
HowardHuang1 wants to merge 84 commits into
Open
Conversation
… vs padded datasets) Introduces the new Dataset API at the C++ API layer and migrates all consumers (C API, Python, Go, Java, benchmarks, examples, faiss patch) to the new dataset API. Now downstream functions including build, search, extend, merge, serialize/deserialize, etc. all take in new dataset API rather than mdspan at the C++ API layer and DLPack at the C layer. Old inheritance hierarchy for Dataset API is replaced with composition with templates."
…th only 26.08 ancestry.
Add dispatch on Layout and MemType params within one single centralized build function rather than adding build function overloads with different names at C API layer.
Add lifetime-tracked dataset views and keep deserialized storage coupled to its index handle. This aligns Rust with the new C dataset API and prevents dangling dataset references.
Split the `cagra::index_params::from_hnsw_params` into a core `cagra::index_params::from_dataset` and a thin wrapper ``cagra::index_params::from_hnsw_params`. This solves two problems: 1. Avoid using the HNSW parameters that do not make sense for CAGRA 2. Allow HNSW parameter heuristics change without affecting CAGRA in future The new parameter defaults are selected to yield exactly the same configurations at old call sites of `from_hnsw_params` Authors: - Artem M. Chirkin (https://github.com/achirkin) Approvers: - Corey J. Nolet (https://github.com/cjnolet) - Tamas Bela Feher (https://github.com/tfeher) URL: NVIDIA#2345
…unction that took mdspan. Restore code that calls make_device_padded_dataset() factory within build_ace() and call build_from_device_matrix() instead. This fixes build issue.
This reverts commit 21481e0.
…issing update_dataset() calls on host build
… to rename. Now updated to use new factory names
…I. Host vs device is no longer specified in the factory itself. Rather, host vs device is dispatched at the C API layer. If input data is on host, factory returns a host dataset. If input data is on device, factory returns a device dataset. This allowed us to consolidate several dispatch branches in the language wrappers.
…originates on host. Merge() can no longer merge host data. Merge() only takes data on device
… OOM. However, we should re-evaluate this because I don't think this is good for the new DtasetAPI as the index return type can now be host index when GPU hits OOM but the user still expects device index. With the new templating on the datasets and indexes, this can cause user/caller confusion.
…earch. TODO: need to revisit this later. This is a temporary patch to fix failing java tests and avoid exposing a public tiered index update dataset function for users. Instead we use CAGRA's existing update dataset functions and keep it tiered index's responsibility to call this internally rather than leaving the responsibility to the users
Update C Tests to Use Getters
… padded datasets onto host or standard indexes prior to search within Java tests.
…ndexImpl.java Co-authored-by: Igor Motov <igor@motovs.org>
…ndexImpl.java Co-authored-by: Igor Motov <igor@motovs.org>
cjnolet
added
improvement
Contributor
|
/ok to test 5b1bd3c |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds upon PR #1846 which has already been merged in.