Repository navigation
Labels
Description
Problem description
loadModel(modelPath, options) and the LLM task APIs (createLLMRunner, createLLMChatSession, createLLMTranslator, useLLMChatSession) currently require a plain local .pte file path on disk and do not expose ExecuTorch's data_map_loader for separated .ptd weight archives.
This blocks three common production use cases:
- In-memory models (
ArrayBuffer): loading small.pteprograms directly from memory without writing a temporary file to disk. - Separated
.ptdexternal weight archives: ExecuTorch supports exporting a small.pteprogram alongside an external.ptdnamed-data archive (Module(data_loader, ..., data_map_loader)), where backend delegates such as XNNPACK unpack named weight segments during method initialization and immediately free the source buffers. - Chunked / on-demand loading (
DataLoader): apps that store models in custom containers or encrypted chunks cannot materialize a 200 MB–2 GB model into a temporary plaintext file or a single full-fileArrayBufferwithout doubling peak/steady-state resident memory. They need a random-access loader (size()+load(offset, size)) so ExecuTorch only requests the byte ranges it needs and frees eachFreeableBuffersegment as soon as the backend unpacks it.
Proposed solution
Extend the core loadModel primitive and thread it through the LLM runner/task layer while staying 100% backward-compatible with existing path-based callers:
- Add
ModelDataLoaderandModelSourceinsrc/core/model.ts:export type ModelDataLoader = { size(): number; load(offset: number, size: number): ArrayBuffer; }; export type ModelSource = string | ArrayBuffer | ModelDataLoader;
- Extend
LoadModelOptionsandloadModel:export type LoadModelOptions = { readonly eagerLoadMethods?: boolean; readonly dataMapPath?: ModelSource; }; export function loadModel(modelSource: ModelSource, options?: LoadModelOptions): Model;
stringcontinues to useMmapDataLoader(NoMlock).ArrayBufferuses a 64-byte-aligned owningBufferDataLoader(model.path === '<buffer>').ModelDataLoaderuses a JSI-backedDataLoader(model.path === '<loader>') that returns 64-byte-alignedFreeableBuffers with a deallocation callback so ExecuTorch frees each segment once unpacked, and detaches its JS function/runtime handles beforeloadModel()returns so nojsi::Runtime*or JS closure outlives the load call.
- Thread
ModelSourceanddataMapPaththrough LLM APIs & hooks:createLLMRunner(modelPath: ModelSource, tokenizerPath: string, modalities?, runtime?, options?: LoadModelOptions)LLMModel<TSource extends ModelSource = string>(defaulting tostringsomodels.llm.*registry entries remain typed asstringURLs, whilecreateLLMChatSession,createLLMTranslator, anduseLLMChatSessionacceptModelSourceanddataMapPath).- Preserve
ArrayBufferandModelDataLoaderreference identity inuseModelanduseResourceDownloadconfig memoization, and skipModelDataLoaderobjects indownload()traversal.
Alternative solutions
- Writing temporary
.pte/.ptdfiles to disk before callingloadModel(path): undesirable for encrypted/protected model assets (writes plaintext weights to flash storage) and adds unnecessary disk I/O and storage pressure. - Passing only whole
ArrayBuffers: works for small.pteprograms, but for large models it keeps a full copy of the raw weights in memory alongside the backend's packed weights (~170+ MB extra RSS on a 180 MB model). Supporting bothArrayBufferandModelDataLoaderplus.ptdexternal weights lets XNNPACK free each weight segment right after packing.
Benefits to React Native ExecuTorch
- Unlocks separated
.ptdweight archives (data_map_loader) acrossloadModeland LLM tasks. - Enables zero-temp-file in-memory loading (
ArrayBuffer) and low-memory streaming/chunked loading (ModelDataLoader) for custom asset containers or encrypted models. - Lives cleanly in
core/modelwith no breaking changes to existing APIs ormodelsregistry types.
Additional context
I have opened PR #1630 implementing this with host C++ GoogleTest (ModelTest), Jest API tests, and documentation updates.
Activity
barhanc commentedon Nov 10, 2026
Thanks for raising this issue and opening a PR. I will look at it shortly.