Skip to content

docs(adrs): close cross-process VRAM sharing with ComfyUI (ADR-0011) - #182

Open
laurigates wants to merge 2 commits into
mainfrom
claude/comfyui-loractl-shared-vram-eq0oid
Open

docs(adrs): close cross-process VRAM sharing with ComfyUI (ADR-0011)#182
laurigates wants to merge 2 commits into
mainfrom
claude/comfyui-loractl-shared-vram-eq0oid

Conversation

@laurigates

@laurigates laurigates commented Jul 31, 2026

Copy link
Copy Markdown
Owner

What

Adds docs/adrs/0011-comfyui-cross-process-vram-sharing.md and a pointer
paragraph in docs/roadmap.md. Docs only — no Rust touched.

ADR-0011 records a spike into a proposed architecture: run
shared-tensor inside ComfyUI
as a "VRAM model server", have loractl import the exported cudaIpcMemHandle_t
with cudarc, and wrap the device pointer in a burn/cubecl buffer so training
reads ComfyUI's already-resident Krea 2 weights instead of allocating its own.

The spike was deliberately ordered to test the most-likely-to-fail rung first —
foreign-pointer adoption, which needs no ComfyUI, no socket and no
shared-tensor — and it stopped there.

Why

The load-bearing step was "wrap the pointer for CubeCL/Burn". ADR-0008 and
ADR-0010 both already asserted the negative ("cubecl exposes no managed-memory
allocation path (verified)"; "burn 0.21 exposes no pinned-allocation API to
call"), but neither quoted source, so nothing in the repo could settle it. This
upgrades both from asserted to demonstrated, at pinned versions.

Decision 1 — cubecl 0.10 cannot hold a pointer it did not allocate.
ComputeStorage (cubecl-runtime-0.10.0/src/storage/base.rs:74-95) has five
methods and admits memory only through fn alloc(&mut self, size: u64) — a
size, not a pointer. In cubecl-cuda-0.10.0/src/compute/storage/gpu.rs,
GpuStorage's pointer map is private (:19), its only public constructor is
new(mem_alignment, stream) (:50), and the sole insertion sits inside alloc
(:169) after an unconditional malloc_async/malloc_sync. AllocationKind
(:9-12) is private with exactly two variants, both meaning "cubecl owns this".
The back door is shut too: a hand-built StorageHandle panics in get
(:148-152). A grep for register|adopt|from_ptr|from_raw|from_device_ptr| external|import|wrap|attach across all of cubecl-runtime and cubecl-cuda
returns zero hits.

Decision 2 — forcing one in would corrupt ComfyUI.
perform_deallocations (gpu.rs:63-80) frees every pointer in the map
unconditionally, by AllocationKind. There is no ownership flag to consult
because there is no variant meaning "not mine". This answered the spike's
second rung without hardware.

Decision 3 — the reverse direction is separately unavailable. cubecl's
alloc prefers malloc_async; NVIDIA documents sharing stream-ordered
allocations through a different API family than cudaIpcGetMemHandle, requiring
IPC capability at pool creation, which GpuStorage::new never requests. Costs
nothing — the reverse channel already ships over disk
(diffusion_trainer.rs:1650-1663, keys pinned by tests/krea2_lora_keys.rs).

Decision 4 — the budget fails independently of the mechanism. Measured from
the safetensors header of the checkpoint actually in use (Comfy-Org/Krea-2 @
7b75ff3, diffusion_models/krea2_turbo_int8_convrot.safetensors):
13,492,686,496 bytes, 878 tensors — 224 I8 weights (12.16 GB) each with an
F32 weight_scale and a U8 comfy_quant marker, 76 BF16 left unquantized
(1.33 GB). So the only shareable copy is ~13.5 GB, larger than the ~10.1 GB
int4 base loractl already uses; even a perfect zero-copy import is ~23 GB before
ComfyUI's own encoder, VAE and generation activations. loractl trains int4 Q4S
(packed u32, block 32) quantized on-device — aliasing either ComfyUI scheme as
that is not a cast.

Decision 5 — the int8 repack is not a loadable loractl input today, over any
transport.
The only actionable item here, and unrelated to IPC. The file
carries no F8_E4M3, so is_fp8_checkpoint is false and it routes to the plain
load_module path, where nothing pairs a weight with its scale. Three layers
decline to catch it: burn-store maps Dtype::I8 => DType::I8 without error
(store.rs:1055); CastFloatsAdapter tests is_float and returns non-float
snapshots unchanged; and the applier validates shape but not dtype
(applier.rs:228-240), so the same-shaped I8 weight raises no ShapeMismatch.
The 224 weight_scale keys land as "Unused Tensors". Supported denoiser inputs
remain bf16/f32 or a scaled-fp8 repack.

Net: the architecture is closed, co-tenancy stays a time-division problem, and
reading ComfyUI's model directory was never the blocked part
(config/examples/krea2-comfyui.yaml). No probe example, no cudarc
dependency, and no GPU hours were spent.

Verification

  • cargo fmt --all --check passes
  • just lint / just testgreen in CI on feb7b36 (fmt + clippy + test, clippy (opt-in feature paths), cargo-deny, surf lint + check
    all passed). They could not be run locally (just is not installed in
    this environment) and the diff is markdown-only, so CI is the gate of
    record; re-running on f0c7602.
  • Not a feature-gated path — no cfg code touched
  • No dependency changes — just audit / just deny not applicable
  • No new ML code — every claim is cited to vendored upstream source at
    pinned versions (cubecl 0.10.0, the burn 0.21 fork) or measured
    directly from a safetensors header, with byte counts given

Scope of evidence, stated plainly: all findings come from reading the load
path and upstream source at pinned versions, plus HTTP Range reads of
safetensors headers. Nothing was executed — this host has no GPU and no CUDA
toolkit. Decision 5's final materialization behavior (Tensor::from_data with a
dtype-mismatched snapshot) is therefore called out in the ADR as
backend-dependent and untested; what is certain is that the scale is never
applied.

The ADR is explicitly version-scoped, not a permanent law: it asks for a
re-check on the burn 0.22 migration (#79), which moves cubecl.

Linked issue

None. This ADR closes a question rather than opening work. Decision 5 is worth
an issue if anyone is pointing model.denoiser at an int8 ComfyUI checkpoint.

A proposal to have loractl train against Krea 2 weights already resident in
ComfyUI's VRAM — exported by `shared-tensor` over a UDS and imported with
`cudaIpcOpenMemHandle` — was spiked at its most-likely-to-fail rung first: can
burn/cubecl adopt a device pointer it did not allocate?

It cannot, and this records the mechanism at pinned versions rather than
asserting it. cubecl 0.10's `ComputeStorage` (storage/base.rs:74-95) admits
memory only through `alloc(size: u64)`; `GpuStorage`'s pointer map is private
(gpu.rs:19) and its sole insertion sits inside `alloc` after an unconditional
`malloc_async`/`malloc_sync`; `AllocationKind` has no non-owning variant; and a
hand-built `StorageHandle` panics in `get` (gpu.rs:148-152). A grep for every
adoption-shaped name across cubecl-runtime and cubecl-cuda returns zero hits.
Had a pointer been forced in, `perform_deallocations` (gpu.rs:63-80) would free
it out from under ComfyUI — silent corruption in another process.

This upgrades ADR-0008's and ADR-0010's asserted "verified" to demonstrated,
with quoted source. It is version-scoped: re-check on the burn 0.22 migration
(#79).

The budget fails independently of the mechanism. The only shareable copy is
ComfyUI's ~13.1 GB scaled-fp8 base — larger than the ~10.1 GB int4 base loractl
already uses — so even a perfect zero-copy import lands at ~26 GB against a
24 GB card. Co-tenancy stays a time-division problem. Reading ComfyUI's model
*directory* was never blocked and remains shipped, as does the disk-based
reverse channel for in-progress adapters.

No probe example, cudarc dependency, or GPU hours were spent: the spike stopped
at its first rung by design.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QNa5Xn2QW7fimm5XHokguK
@laurigates
laurigates marked this pull request as ready for review August 1, 2026 10:58
…n use

Decision 4 cited the scaled-fp8 repack (~13.1 GB). The checkpoint in use is
`Comfy-Org/Krea-2` @ 7b75ff3 `diffusion_models/krea2_turbo_int8_convrot.safetensors`.
Measured from its safetensors header: 13,492,686,496 bytes, 878 tensors — 224
`I8` weights (12.16 GB) each with an F32 `weight_scale` and a U8 `comfy_quant`
marker, 76 `BF16` left unquantized (1.33 GB), 354 F32 norm scales. The budget
conclusion is unchanged and now rests on a measured number rather than an
assumed one: ~13.5 GB shared base + loractl's ~9.3 GB non-base is already ~23 GB
before ComfyUI's own encoder, VAE and generation activations.

Adds Decision 5, found while measuring the above and the only actionable item in
the ADR — unrelated to IPC. The int8 repack carries no `F8_E4M3`, so
`is_fp8_checkpoint` is false and it routes to the plain `load_module` path, where
nothing pairs a weight with its scale. Three layers decline to catch it:
burn-store maps `Dtype::I8 => DType::I8` without error (store.rs:1055);
`CastFloatsAdapter` tests `is_float` and returns non-float snapshots unchanged;
and the applier validates shape but not dtype (applier.rs:228-240), so the
same-shaped `I8` weight raises no ShapeMismatch. The 224 `weight_scale` keys land
as "Unused Tensors". Supported denoiser inputs remain bf16/f32 or a scaled-fp8
repack.

Decisions 1-3 are untouched — the cubecl storage blocker is dtype-agnostic.

Also records that a same-named community mirror
(AX1Y2JP/Krea-2-Turbo-INT8-ConvRot) is a materially different artifact — 16.17 GB,
816 tensors, 192 quantized sites, 4.16 GB left in BF16 — so sizes quoted for this
scheme need their source pinned.

Read from the load path at pinned versions; not executed (no GPU on this host).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QNa5Xn2QW7fimm5XHokguK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants