Skip to content

feat(core,llm): support ArrayBuffer, custom ModelDataLoader, and external .ptd weights in loadModel - #1630

Closed
Rasaboun wants to merge 3 commits into
software-mansion:mainfrom
Rasaboun:feat/llm-memory-loading
Closed

Rasaboun wants to merge 3 commits into
software-mansion:mainfrom
Rasaboun:feat/llm-memory-loading

Conversation

@Rasaboun

@Rasaboun Rasaboun commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

Extends loadModel() and the LLM runner/task APIs to load .pte programs and separated .ptd external weight archives (data_map_loader) from local file paths, in-memory ArrayBuffers, or custom random-access ModelDataLoader implementations ({ size(): number; load(offset: number, size: number): ArrayBuffer }):

  • src/core/model.ts & cpp/core/model.*:
    • Adds ModelDataLoader and ModelSource = string | ArrayBuffer | ModelDataLoader, plus LoadModelOptions.dataMapPath?: ModelSource.
    • File paths continue to use MmapDataLoader (NoMlock).
    • ArrayBuffer sources use a 64-byte-aligned owning BufferDataLoader (model.path === '<buffer>').
    • ModelDataLoader sources use JsiDataLoader (model.path === '<loader>'), returning 64-byte-aligned FreeableBuffers with a deallocation callback so ExecuTorch/XNNPACK releases each external weight segment immediately after unpacking.
    • Detaches JS function/object handles before loadModel() returns so no jsi::Runtime* or JS closure outlives the load call, and captures JS callback exceptions via a stack-owned std::shared_ptr<std::exception_ptr> rather than unwinding through ExecuTorch C++ frames.
  • LLM runner, tasks & hooks:
    • Threads ModelSource and LoadModelOptions (dataMapPath) through createLLMRunner, createLLMChatSession, createLLMTranslator, and useLLMChatSession.
    • Parameterizes LLMModel<TSource extends ModelSource = string> with a default of string so models.llm.* registry entries remain strictly typed as string URLs.
    • Preserves ArrayBuffer and ModelDataLoader reference identity in useModel and useResourceDownload config memoization, and skips ModelDataLoader objects during download() URL traversal.

Introduces a breaking change?

  • Yes
  • No

Type of change

  • Bug fix (change which fixes an issue)
  • New feature (change which adds functionality)
  • Documentation update (improves or adds clarity to existing documentation)
  • Other (chores, tests, code style improvements etc.)

Tested on

  • iOS
  • Android

Testing instructions

  1. Run the host C++ GoogleTest + Hermes suite:
    cd packages/react-native-executorch
    scripts/run-native-tests.sh -R ModelTest
  2. Run the TypeScript API test suites and workspace typecheck:
    yarn typecheck
    yarn workspace react-native-executorch test
  3. Build the documentation site:
    cd docs && yarn build

Related issues

Closes #1629

Checklist

  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated the documentation accordingly
  • My changes generate no new warnings

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing. Review the Copilot code review billing settings.

@Rasaboun
Rasaboun force-pushed the feat/llm-memory-loading branch from a6a894c to 27fcf37 Compare November 10, 2026 10:32
@Rasaboun
Rasaboun force-pushed the feat/llm-memory-loading branch from 27fcf37 to fc4d9b6 Compare November 10, 2026 10:37
@barhanc
barhanc self-requested a review November 10, 2026 11:03
@barhanc barhanc added core feature PRs that implement a new feature labels Nov 10, 2026

@barhanc barhanc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, I went over the proposed changes, and it would be a pretty big change for the codebase. I need to discuss how we want to approach this with @msluszniak, who unfortunately won’t be available until next Monday. In the meantime, I’ve put together a modified version of your implementation to make it simpler for us to adopt, especially the model.ts contract and native core changes. When you have a moment, please take a look at draft #1631 and let me know what you think and whether it covers your use case.

@Rasaboun

Copy link
Copy Markdown
Contributor Author

Hi, I went over the proposed changes, and it would be a pretty big change for the codebase. I need to discuss how we want to approach this with @msluszniak, who unfortunately won’t be available until next Monday. In the meantime, I’ve put together a modified version of your implementation to make it simpler for us to adopt, especially the model.ts contract and native core changes. When you have a moment, please take a look at draft #1631 and let me know what you think and whether it covers your use case.

Thanks for putting #1631 together, it covers our use case. A few things I noticed:

useModel keys the config with JSON.stringify, which drops functions. So { programLoader: a } and { programLoader: b } get the same key and the hook keeps the first model (the new test in useModel.test.ts asserts this). If an app lets users switch between encrypted models, they would silently keep the old one. In #1630 I gave each loader an id through a WeakMap in serializeHookConfig. I can port that if you want, or the docs could say to remount.

You can't mix a path and a loader, for example a plain .pte on disk with an encrypted .ptd. Wrapping the .pte in a loader works, so it doesn't block us.

The modelPath → modelSource rename will break custom configs.

I can close #1630 in favour of #1631 and test it on iOS once you've decided.

@barhanc

barhanc commented Nov 12, 2026

Copy link
Copy Markdown
Member

Thanks for pointing these out, I will resolve those problems. And yes, if you don't mind, let's proceed with #1631.

@Rasaboun

Rasaboun commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks for pointing these out, I will resolve those problems. And yes, if you don't mind, let's proceed with #1631.

Sure, I'll close #1630. Would you mind adding me as co-author on the merge commit ? Adding this line at the end of the #1631 description should carry it into the squash commit. Thanks !

Co-authored-by: Rayane 40967731+Rasaboun@users.noreply.github.com

@Rasaboun Rasaboun closed this Nov 12, 2026
@barhanc

barhanc commented Nov 12, 2026

Copy link
Copy Markdown
Member

Sure, no problem! Added you as co-author in the PR description.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core feature PRs that implement a new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(core,llm): support ArrayBuffer, custom ModelDataLoader, and external .ptd weights in loadModel

3 participants