QVAC-23327 fix: use the mmap weights path for CPU SmolVLA loads - #3905
Open
DmitryMalishev wants to merge 4 commits into
Open
QVAC-23327 fix: use the mmap weights path for CPU SmolVLA loads#3905DmitryMalishev wants to merge 4 commits into
DmitryMalishev wants to merge 4 commits into
Conversation
smolvlaLoadModel prefers mapping the GGUF and wrapping it with ggml_backend_dev_buffer_from_host_ptr, and falls back to allocating one buffer and copying the weights into it. The choice reads capabilities off the device that owns the default buffer type, but ggml declares the CPU buffer type with `/* .device = */ NULL`, so on a CPU load the lookup returned nullptr, both gates stayed false, and the fast path was unreachable. qvac-fabric's own loader resolves the CPU device by type for exactly this reason (create_backend_buffers in src/llama-model.cpp); this adds the same fallback. The fallback path is what made SmolVLA loads fail intermittently on iOS. The mobile e2e suite loads the 1.9 GB smolvla-libero-vision-q8 GGUF with backend 'cpu', so every load committed the whole weight set to anonymous memory: device samples from a passing iPhone 16 Pro run show footprint and resident rising together by 2028 MB and peaking at 2693 MB footprint / 3010 MB resident, within ~10% of the per-process jetsam ceiling. Once earlier tests leave the process near that ceiling the allocation is refused outright — footprint stays flat and the load fails ~25 ms after the backend comes up. Mapping the file keeps the weights in clean, evictable, file-backed pages instead, so only the compute buffers count against the limit. Also: - Log why the fast path was skipped. A missing device produced no diagnostic at all, so the fallback looked like the intended route. - Reject a mapping whose tensor data is not buffer-type aligned. The CPU backend asserts on an unaligned host pointer, and an assert aborts the process rather than returning NULL; this path only now becomes live for CPU devices. - Carry the failing step in model.load_error so the JS error reads "... : out of memory: 1889 MB weights buffer" instead of naming only the GGUF path. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
License compliance — cleanNo new dependency license findings in this PR. Warn-only (shadow) mode — this check does not block merges yet. Updated automatically by the canonical license compliance workflow. NOTICE presence (advisory)Missing NOTICE (advisory, does not block):
|
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:47 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:48 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:48 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:48 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:48 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:48 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:48 — with
GitHub Actions
Failure
The end-to-end integration test proved a load succeeded but not how the weights were placed, so the alloc+copy fallback that pushed iOS into a jetsam-refused 1.9 GB anonymous allocation looked identical to a healthy run. Capture the native load log and require the mmap marker on every CPU load, except on Windows where that path is compiled out.
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:53 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:56 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:56 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:56 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 15:56 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 18:33 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 18:33 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 18:33 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 18:33 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 18:33 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 18:33 — with
GitHub Actions
Failure
DmitryMalishev
had a problem deploying
to
release
August 17, 2026 18:33 — with
GitHub Actions
Failure
Contributor
Mobile integration tests — @qvac/vla-ggml (Android)Result: passed
|
Contributor
Mobile integration tests — @qvac/vla-ggml (iOS)Result: passed
|
The addon queues native log lines and drains them onto the JS thread through a uv_async handle, so they can arrive after the load promise settles: the weights-path assertion read an empty log and failed on Linux and darwin-arm64 while winning the race on darwin-x64. Wait for the load's terminal line first, which is safe because the queue is FIFO, and print the load lines when the assertion fails so a fallback names itself instead of looking like a flake.
Contributor
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 What problem does this PR solve?
vla-hparams-shape,vla-run-synthetic-shape,vla-run-stats,vla-invalid-img-size) fail intermittently on iOS during a full SDK e2e run withFailed to load SmolVLA model. Reproduced on a local iPhone 16e and on iPhone 16 Pro in CI.smolvlaLoadModelprefers mapping the GGUF and wrapping it withggml_backend_dev_buffer_from_host_ptr, falling back to one large buffer plus a copy. The choice reads capabilities off the device that owns the default buffer type — but ggml declares the CPU buffer type with/* .device = */ NULL(see theFIXMEinggml_backend_cpu_buffer_type), soggml_backend_buft_get_devicereturnednullptr, both gates stayed false, and the mmap path was unreachable on every CPU load. qvac-fabric's own loader resolves the CPU device by type for exactly this reason (create_backend_buffers,src/llama-model.cpp); this port dropped that fallback.smolvla-libero-vision-q8GGUF withbackend: 'cpu', so every load committed the whole weight set to anonymous memory. Device samples from a passing iPhone 16 Pro run show footprint and resident rising together by 2028 MB and peaking at 2693 MB footprint / 3010 MB resident — within ~10% of the per-process jetsam ceiling. (A file-backed mapping would instead widen the resident-minus-footprint gap by ~1.9 GB; that gap stayed flat at ~320 MB.) Once earlier tests leave the process near the ceiling the allocation is refused outright: footprint never moves and the load fails ~25 ms after the backend comes up. Hence "intermittent", and hence more frequent in a full-suite run than in a filtered one.📝 How does it solve it?
smolvlaResolveDevice()falls back toggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU)when a buffer type carries no device, matching qvac-fabric's loader. CPU loads now map the file: weights become clean, evictable, file-backed pages, so only the compute buffers (~577 MB in the run above) count against the jetsam limit. Projected peak footprint drops from ~2.7 GB to ~1.2 GB."buffer pointer must be aligned"), and an assert aborts the process instead of returningNULL; that code only becomes live for CPU devices with this change, so it needs a guard rather than a crash.model.load_error, so the JS error readsfailed to load SmolVLA model from: <path>: out of memory: 1889 MB weights bufferinstead of naming only the path.Verified that no weight tensor is written during inference (every
ggml_backend_tensor_settarget is a graph input), which is what makes the read-only mapping safe. Behaviour on Metal/Vulkan/CUDA is unchanged — those buffer types already carry a device. Windows is unaffected (the mmap path is#ifndef _WIN32).No version bump or changelog here, per the repo's split between change PRs and
chore: bump vla-ggmlrelease PRs.🧪 How was it tested?
test/unit/test_smolvla_weights_device.cpppins both fast-path gates for a CPU load, so swapping the helper back for a plainggml_backend_buft_get_devicefails the suite.cpp-tests-vladispatched on this branch ahead of the PR: linux-x64, darwin-arm64, darwin-x64, windows-x64.run-cpp-addon-tests,run-desktop-addon-testsandrun-mobile-addon-tests: the desktop integration tests load the real SmolVLA GGUF on both the preferred backend and forced CPU, which is the path this PR makes live, and the mobile leg exercises it on iOS/Android hardware.🔌 API Changes
No API surface change. One behaviour change worth noting for consumers: a failed SmolVLA load now appends the reason to the existing error message.