Repository navigation
feat(core,llm): support ArrayBuffer, custom ModelDataLoader, and external .ptd weights in loadModel - #1630
feat(core,llm): support ArrayBuffer, custom ModelDataLoader, and external .ptd weights in loadModel#1630Rasaboun wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.
a6a894c to
27fcf37
Compare
27fcf37 to
fc4d9b6
Compare
barhanc
left a comment
There was a problem hiding this comment.
Hi, I went over the proposed changes, and it would be a pretty big change for the codebase. I need to discuss how we want to approach this with @msluszniak, who unfortunately won’t be available until next Monday. In the meantime, I’ve put together a modified version of your implementation to make it simpler for us to adopt, especially the model.ts contract and native core changes. When you have a moment, please take a look at draft #1631 and let me know what you think and whether it covers your use case.
Thanks for putting #1631 together, it covers our use case. A few things I noticed:
You can't mix a path and a loader, for example a plain The I can close #1630 in favour of #1631 and test it on iOS once you've decided. |
|
Thanks for pointing these out, I will resolve those problems. And yes, if you don't mind, let's proceed with #1631. |
Sure, I'll close #1630. Would you mind adding me as co-author on the merge commit ? Adding this line at the end of the #1631 description should carry it into the squash commit. Thanks ! Co-authored-by: Rayane 40967731+Rasaboun@users.noreply.github.com |
|
Sure, no problem! Added you as co-author in the PR description. |
Description
Extends
loadModel()and the LLM runner/task APIs to load.pteprograms and separated.ptdexternal weight archives (data_map_loader) from local file paths, in-memoryArrayBuffers, or custom random-accessModelDataLoaderimplementations ({ size(): number; load(offset: number, size: number): ArrayBuffer }):src/core/model.ts&cpp/core/model.*:ModelDataLoaderandModelSource = string | ArrayBuffer | ModelDataLoader, plusLoadModelOptions.dataMapPath?: ModelSource.MmapDataLoader(NoMlock).ArrayBuffersources use a 64-byte-aligned owningBufferDataLoader(model.path === '<buffer>').ModelDataLoadersources useJsiDataLoader(model.path === '<loader>'), returning 64-byte-alignedFreeableBuffers with a deallocation callback so ExecuTorch/XNNPACK releases each external weight segment immediately after unpacking.loadModel()returns so nojsi::Runtime*or JS closure outlives the load call, and captures JS callback exceptions via a stack-ownedstd::shared_ptr<std::exception_ptr>rather than unwinding through ExecuTorch C++ frames.ModelSourceandLoadModelOptions(dataMapPath) throughcreateLLMRunner,createLLMChatSession,createLLMTranslator, anduseLLMChatSession.LLMModel<TSource extends ModelSource = string>with a default ofstringsomodels.llm.*registry entries remain strictly typed asstringURLs.ArrayBufferandModelDataLoaderreference identity inuseModelanduseResourceDownloadconfig memoization, and skipsModelDataLoaderobjects duringdownload()URL traversal.Introduces a breaking change?
Type of change
Tested on
Testing instructions
cd packages/react-native-executorch scripts/run-native-tests.sh -R ModelTestyarn typecheck yarn workspace react-native-executorch testRelated issues
Closes #1629
Checklist