Skip to content

QVAC-23327 fix: use the mmap weights path for CPU SmolVLA loads - #3905

Open
DmitryMalishev wants to merge 4 commits into
mainfrom
fix-ios-vla-model-load
Open

QVAC-23327 fix: use the mmap weights path for CPU SmolVLA loads#3905
DmitryMalishev wants to merge 4 commits into
mainfrom
fix-ios-vla-model-load

Conversation

@DmitryMalishev

Copy link
Copy Markdown
Contributor

🎯 What problem does this PR solve?

  • Four VLA tests (vla-hparams-shape, vla-run-synthetic-shape, vla-run-stats, vla-invalid-img-size) fail intermittently on iOS during a full SDK e2e run with Failed to load SmolVLA model. Reproduced on a local iPhone 16e and on iPhone 16 Pro in CI.
  • smolvlaLoadModel prefers mapping the GGUF and wrapping it with ggml_backend_dev_buffer_from_host_ptr, falling back to one large buffer plus a copy. The choice reads capabilities off the device that owns the default buffer type — but ggml declares the CPU buffer type with /* .device = */ NULL (see the FIXME in ggml_backend_cpu_buffer_type), so ggml_backend_buft_get_device returned nullptr, both gates stayed false, and the mmap path was unreachable on every CPU load. qvac-fabric's own loader resolves the CPU device by type for exactly this reason (create_backend_buffers, src/llama-model.cpp); this port dropped that fallback.
  • Consequence on mobile: the e2e loads the 1.9 GB smolvla-libero-vision-q8 GGUF with backend: 'cpu', so every load committed the whole weight set to anonymous memory. Device samples from a passing iPhone 16 Pro run show footprint and resident rising together by 2028 MB and peaking at 2693 MB footprint / 3010 MB resident — within ~10% of the per-process jetsam ceiling. (A file-backed mapping would instead widen the resident-minus-footprint gap by ~1.9 GB; that gap stayed flat at ~320 MB.) Once earlier tests leave the process near the ceiling the allocation is refused outright: footprint never moves and the load fails ~25 ms after the backend comes up. Hence "intermittent", and hence more frequent in a full-suite run than in a filtered one.
  • Nothing said so. A missing device produced no log line at all, so the fallback looked like the intended route, and the thrown error named only the GGUF path.

📝 How does it solve it?

  • smolvlaResolveDevice() falls back to ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU) when a buffer type carries no device, matching qvac-fabric's loader. CPU loads now map the file: weights become clean, evictable, file-backed pages, so only the compute buffers (~577 MB in the run above) count against the jetsam limit. Projected peak footprint drops from ~2.7 GB to ~1.2 GB.
  • Log why the fast path was skipped, with the device name and both gate values, so this can never regress silently again.
  • Reject a mapping whose tensor data is not buffer-type aligned. The CPU backend asserts on an unaligned host pointer ("buffer pointer must be aligned"), and an assert aborts the process instead of returning NULL; that code only becomes live for CPU devices with this change, so it needs a guard rather than a crash.
  • Carry the failing step in model.load_error, so the JS error reads failed to load SmolVLA model from: <path>: out of memory: 1889 MB weights buffer instead of naming only the path.

Verified that no weight tensor is written during inference (every ggml_backend_tensor_set target is a graph input), which is what makes the read-only mapping safe. Behaviour on Metal/Vulkan/CUDA is unchanged — those buffer types already carry a device. Windows is unaffected (the mmap path is #ifndef _WIN32).

No version bump or changelog here, per the repo's split between change PRs and chore: bump vla-ggml release PRs.

🧪 How was it tested?

  • New test/unit/test_smolvla_weights_device.cpp pins both fast-path gates for a CPU load, so swapping the helper back for a plain ggml_backend_buft_get_device fails the suite.
  • cpp-tests-vla dispatched on this branch ahead of the PR: linux-x64, darwin-arm64, darwin-x64, windows-x64.
  • Labelled for run-cpp-addon-tests, run-desktop-addon-tests and run-mobile-addon-tests: the desktop integration tests load the real SmolVLA GGUF on both the preferred backend and forced CPU, which is the path this PR makes live, and the mobile leg exercises it on iOS/Android hardware.

🔌 API Changes

No API surface change. One behaviour change worth noting for consumers: a failed SmolVLA load now appends the reason to the existing error message.

smolvlaLoadModel prefers mapping the GGUF and wrapping it with
ggml_backend_dev_buffer_from_host_ptr, and falls back to allocating one
buffer and copying the weights into it. The choice reads capabilities off
the device that owns the default buffer type, but ggml declares the CPU
buffer type with `/* .device = */ NULL`, so on a CPU load the lookup
returned nullptr, both gates stayed false, and the fast path was
unreachable. qvac-fabric's own loader resolves the CPU device by type for
exactly this reason (create_backend_buffers in src/llama-model.cpp); this
adds the same fallback.

The fallback path is what made SmolVLA loads fail intermittently on iOS.
The mobile e2e suite loads the 1.9 GB smolvla-libero-vision-q8 GGUF with
backend 'cpu', so every load committed the whole weight set to anonymous
memory: device samples from a passing iPhone 16 Pro run show footprint and
resident rising together by 2028 MB and peaking at 2693 MB footprint /
3010 MB resident, within ~10% of the per-process jetsam ceiling. Once
earlier tests leave the process near that ceiling the allocation is
refused outright — footprint stays flat and the load fails ~25 ms after
the backend comes up. Mapping the file keeps the weights in clean,
evictable, file-backed pages instead, so only the compute buffers count
against the limit.

Also:

- Log why the fast path was skipped. A missing device produced no
  diagnostic at all, so the fallback looked like the intended route.
- Reject a mapping whose tensor data is not buffer-type aligned. The CPU
  backend asserts on an unaligned host pointer, and an assert aborts the
  process rather than returning NULL; this path only now becomes live for
  CPU devices.
- Carry the failing step in model.load_error so the JS error reads
  "... : out of memory: 1889 MB weights buffer" instead of naming only the
  GGUF path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@DmitryMalishev
DmitryMalishev requested review from a team as code owners August 17, 2026 15:43
@DmitryMalishev DmitryMalishev added run-cpp-addon-tests CI: run C++ addon tests (requires verified) run-desktop-addon-tests CI: run desktop integration tests (requires verified) run-mobile-addon-tests CI: run mobile integration tests (requires verified) labels Aug 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

  • ./.github/actions/release-merge-guard
  • ./docs/website
  • ./packages/ggml-coload-smoke
  • ./packages/fabric/test/integration
  • ./packages/inference-addon-cpp/mobile
  • ./packages/sdk/e2e
  • ./packages/llm-llamacpp/benchmarks/performance
  • ./packages/llm-llamacpp/benchmarks/server
  • ./packages/vla-ggml/sim/server
  • ./packages/embed-llamacpp/benchmarks/performance
  • ./packages/embed-llamacpp/benchmarks/server
  • ./packages/asr-ggml/benchmarks/server

The end-to-end integration test proved a load succeeded but not how the
weights were placed, so the alloc+copy fallback that pushed iOS into a
jetsam-refused 1.9 GB anonymous allocation looked identical to a healthy
run. Capture the native load log and require the mmap marker on every CPU
load, except on Windows where that path is compiled out.
@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Mobile integration tests — @qvac/vla-ggml (Android)

Result: passed

metric value
Devices passed 9
Devices failed 0
Test cases total 27
Test cases passed 27
Test cases failed 0
Test cases skipped 0

View workflow run

@github-actions

github-actions Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Mobile integration tests — @qvac/vla-ggml (iOS)

Result: passed

metric value
Devices passed 6
Devices failed 0
Test cases total 18
Test cases passed 18
Test cases failed 0
Test cases skipped 0

View workflow run

The addon queues native log lines and drains them onto the JS thread through
a uv_async handle, so they can arrive after the load promise settles: the
weights-path assertion read an empty log and failed on Linux and darwin-arm64
while winning the race on darwin-x64. Wait for the load's terminal line first,
which is safe because the queue is FIFO, and print the load lines when the
assertion fails so a fallback names itself instead of looking like a flake.
@github-actions

Copy link
Copy Markdown
Contributor

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-cpp-addon-tests CI: run C++ addon tests (requires verified) run-desktop-addon-tests CI: run desktop integration tests (requires verified) run-mobile-addon-tests CI: run mobile integration tests (requires verified)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant