Activity

1. novkien commented on Sep 30, 2026

   

   novkien

   on Sep 30, 2026 – with ChatGPT Codex Connector

   OwnerAuthor

   More actions

   Implementation plan: selective CUDA peer access for the two NVLink pairs

   Status: proposed implementation plan for owner review, not an implemented or validated feature. This comment is the requested deliverable. It does not authorize or report a source patch, installation, production restart, route mutation, or BIOS/IOMMU/ACS change.

   1. Recommended decision

   Implement an opt-in pair allowlist, enforced consistently at CUDA peer initialization, VMM access assignment, and both CUDA tensor-copy callbacks. Preserve the existing four-GPU layer-split route. For a pair outside the allowlist, use an explicit host-memory copy path rather than trusting cudaMemcpyPeerAsync to choose a safe transport implicitly.

   Proposed configuration, not available in the inspected backend yet:

   GGML_CUDA_P2P_PAIRS=0-1,2-3
   

   The intended policy for the effective CUDA order recorded in this issue is:

   CUDA0 <==== NVLink P2P ====> CUDA1
     |                           |
     +----- host staging --------+----- between groups, in either direction
     |                           |
   CUDA2 <==== NVLink P2P ====> CUDA3
   

   More precisely, the policy matrix is:

   Source / destination	CUDA0	CUDA1	CUDA2	CUDA3
   CUDA0	Local D2D	Peer allowed	Host fallback	Host fallback
   CUDA1	Peer allowed	Local D2D	Host fallback	Host fallback
   CUDA2	Host fallback	Host fallback	Local D2D	Peer allowed
   CUDA3	Host fallback	Host fallback	Peer allowed	Local D2D

   There are four permitted ordered peer edges and eight forbidden ordered cross-group edges. Do not restrict the fix to CUDA1-CUDA2: draft/target and other graph transfers can have nonadjacent endpoints.

   The first implementation should support the required NVIDIA CUDA, four-GPU, non-collective layer-split workload. Keep NCCL compiled in if the build already has it, but reject CUDA collective initialization while this restrictive policy is active until that path has its own validated policy integration. This avoids turning a focused transport change into a tensor-parallel redesign.

   2. Evidence and source baseline

   Repository evidence

   The accessible fork master was rechecked at 526c43b8f7dfea9032e9f35e7a1be9183ca7cc20. The relevant CUDA source blob is b50d05d5f76fb7010d921b8e71bd4ff83af5b23c. The issue's 2138591d... source reference did not resolve in the earlier GitHub read; do not use its line links as the implementation baseline without resolving that revision first. The host revision recorded in the issue is a separate deployment fact, not proof that the accessible fork commit is installed.

   Read these exact source files before implementing against a newer head:

   • CUDA backend: initialization, allocation, copies, communicator dispatch.
   • Generic backend copies and scheduler.
   • CUDA context and shared declarations.
   • Existing test registration.

   The implementation must follow the candidate checkout's instructions and preserve concurrent changes. The pointers above identify the reviewed design baseline, not permission to overwrite newer code.

   Recorded hardware evidence

   The bounded copy test in llama-proxy #470 tested CUDA0->CUDA1, CUDA1->CUDA2, and CUDA2->CUDA3. It enabled peer access symmetrically, but the reported data-transfer measurements are for those three directions; reverse-direction transfer correctness still needs testing.

   The two NVLink directions measured 131.30 and 132.82 GB/s with no mismatching bytes. The middle PCIe peer direction measured 0.191 GB/s with 16,711,680 mismatches out of 16,777,216 bytes; its peer-disabled condition was correct. The live rollback recovered approximately 45 tokens/s and the earlier draft acceptance. These are recorded experiments, not tests performed for this plan.

   The latest recorded effective UUID-prefix order is 5ca3,b2e,e149,d4d. Older comments describe other host-index orderings. Resolve full UUIDs and actual process enumeration again for any later execution; do not reconstruct full UUID strings from prefixes or copy an older host-index sequence.

   External research and its implementation implications

   Source-backed finding	Consequence for this design
   CUDA peer access is directional; capability and enabled access are different concepts.	Expand each configured pair into two validated directions. A successful capability query is not a transport correctness test.
   The Volta tuning guide says transfers between NVLink-connected endpoints are routed transparently through NVLink.	Select known-good endpoint pairs; do not invent a CUDA API flag that pins a particular physical cable or bus.
   VMM permissions are controlled by mapping/access APIs.	Runtime peer enablement and VMM access descriptors must agree; neither replaces the other.
   The CUDA multi-GPU guide permits cross-device stream waits on events and warns about bare-metal Linux IOMMU PCIe P2P.	Retain event ordering and avoid the known-bad PCIe peer path. Do not claim the host's exact IOMMU/ACS/firmware subcause has been isolated.
   NCCL environment controls govern NCCL transport selection, not all CUDA operations in an application.	NCCL_P2P_LEVEL=NVL alone is not a fix for llama.cpp's own peer-enable, VMM, or copy paths.
   NCCL's P2P implementation has direct peer-enable, VMM import/map/access, and legacy IPC paths.	A backend allowlist cannot assume a later communicator will leave CUDA access state unchanged. Guard actual communicator initialization.
   CUDA API synchronization rules do not make pageable-host copies reliably asynchronous just because the API name ends in Async.	Treat the initial generic host fallback as a correctness path; profile it before claiming good overlap or a speedup.
   NVML NVLink APIs expose link state and remote PCI identities.	They can support topology validation, but CUDA and NVML device indices must not be equated. No per-copy shell command or NVML poll is needed.

   Relevant upstream history also informs the scope. Merged PR #21891 moved NCCL communicators into externally managed contexts; do not restore the older assumption that compiling NCCL necessarily initializes a communicator over every GPU at backend startup. Discussion #26156 requests a list of P2P-enabled GPUs, but does not supply this pair-policy implementation. A list containing all four GPUs would still permit the wrong cross-group edges if interpreted as a clique.

   Issue #21883 is a separate report about peer-mapping exhaustion on a larger system. It supports checking resource/error handling; its proposed blanket error clearing is not the policy adopted here. Do not use unrelated platform reports as proof of the exact V100 fault.

   3. Configuration contract

   Implement the following as explicit, documented semantics rather than leaving them to incidental parser behavior.

   3.1 Endpoint identity

   Endpoints are CUDA runtime device ordinals in the current process after CUDA_VISIBLE_DEVICES, in the range [0, physical_device_count). They are not:

   • host nvidia-smi indices;
   • positions in a shortened --device argument;
   • ggml virtual-device ordinals created through GGML_CUDA_DEVICES;
   • RPC-device identifiers.

   The official CUDA environment-variable documentation describes visibility and enumeration ordering. In the fork, each logical/virtual CUDA backend already maps to a backing physical CUDA ordinal. Apply that existing mapping before looking up a copy policy.

   Example: with two physical devices and four virtual ggml devices, virtual devices 0 and 2 share physical device 0. A copy between them is local D2D, not a peer edge. Configuration 0-1 describes the two physical runtime devices. Reject endpoint 2 in this example instead of silently interpreting it as a virtual device.

   3.2 Grammar and validation

   Proposed grammar: a nonempty comma-separated list of unsigned_decimal-unsigned_decimal pairs. Permit ASCII whitespace around tokens/separators, but require complete parsing of every token. Leading zeros may be accepted and normalized as decimal. Reject signs, integer overflow, trailing garbage, missing endpoints, self-pairs, empty entries, and out-of-range endpoints. Deduplicate repeated or reversed pairs.

   Examples:

   0-1,2-3        valid, two independent undirected pairs
   1-0,0-1       valid, normalized to one pair
   0-1,1-2       valid general allowlist; does NOT imply 0-2
   0-0           invalid self-pair
   0-1,          invalid empty entry
   0-1,2         invalid incomplete pair
   1             invalid; never interpreted as enable-all
   <empty>       invalid; absence of the variable is the legacy mode
   

   Validate the entire configuration before the first peer-enable operation. Check both directions of every requested pair with cudaDeviceCanAccessPeer. An explicitly requested direction that cannot be enabled must produce a clear startup error; do not quietly broaden the policy or claim it was applied successfully.

   The allowlist is general endpoint selection, not automatic topology discovery. Its NVLink-only property on this deployment comes from choosing the two verified NVLink pairs. A user can still explicitly configure an unsafe PCIe pair. Documentation and startup diagnostics must not call an unverified pair NVLink merely because it was allowlisted.

   3.3 Interaction with existing settings

   Configuration	Proposed behavior
   New variable absent	Preserve existing behavior, including existing NCCL/VMM handling.
   GGML_CUDA_P2P_PAIRS present, old P2P variable absent	Activate the restricted policy.
   Both GGML_CUDA_P2P_PAIRS and GGML_CUDA_P2P present	Reject as ambiguous, even if the old value is 0 or empty.
   Restricted policy with build-time GGML_CUDA_NO_PEER_COPY=ON	Reject the unsupported combination explicitly. Do not advertise accelerated pairs when peer copies are compiled out.
   Restricted policy with GGML_CUDA_ENABLE_UNIFIED_MEMORY present	Reject in v1; managed-memory placement/migration is not covered by this transport contract.
   GGML_CUDA_NO_PINNED present	Preserve its meaning. The correctness fallback remains available; do not allocate an application pinned-host pool behind the user's back.
   HIP/MUSA backend	Do not silently claim support for the new NVIDIA-specific policy; keep legacy behavior unchanged when it is absent and diagnose an attempted unsupported opt-in.
   RPC devices/connections	Leave their transport and remote GPU enumeration unchanged. This policy applies only to local CUDA endpoints.

   The existing backend checks the presence of GGML_CUDA_P2P, so GGML_CUDA_P2P=0 does not disable it. Do not change this legacy behavior incidentally in the feature patch; document that rollback removes the key.

   Policy is immutable for the lifetime of the CUDA backend/process. Parse it once, before model buffers or scratch pools are allocated. Changing a route requires a newly launched child process; mutating getenv state during inference is not a supported reconfiguration mechanism.

   4. Minimal code change map

   Target	Required change
   ggml/src/ggml-cuda/ggml-cuda.cu: ggml_cuda_init()	Parse/validate policy; enable only permitted ordered physical pairs; publish initialized policy state.
   Same file: ggml_cuda_pool_vmm::alloc()	For restricted mode, assign VMM access to the owner and permitted readers only, regardless of NCCL compile flag.
   Same file: ggml_backend_cuda_cpy_tensor_async()	Keep same-physical and allowed-peer paths; decline excluded cross-device copies before enqueueing any work.
   Same file: ggml_backend_cuda_buffer_cpy_tensor()	Apply the same exclusion so the synchronous fallback cannot re-enter peer copy.
   Same file: ggml_backend_cuda_comm_init() and its dispatch boundary	Reject unsupported collective use of restricted mode before NCCL/internal/meta fallback setup.
   ggml/src/ggml-cuda/common.cuh, or one small CUDA-private policy header if needed	Only the internal state/declarations needed to share and test the policy. No general routing framework or public llama API is required.
   Existing backend tests and tests/CMakeLists.txt only where necessary	Reuse the backend test infrastructure for policy/copy integration coverage and isolated CUDA fixtures.
   docs/build.md CUDA section	Document syntax, ordinal mapping, supported scope, conflict/error behavior, verification, and rollback.

   The existing generic copy and scheduler in ggml/src/ggml-backend.cpp should not need algorithm changes for the first implementation. They are essential review and test targets, not automatic refactor targets.

   5. Policy state and initialization

   Use one authoritative internal representation. For example, maintain a restricted-mode flag and fixed-size physical-device matrices for requested directions and successfully established directions. Keep capability, policy permission, and successful initialization conceptually separate. Diagonal local access is not peer enablement.

   A single classifier should yield the following for CUDA-owned copies:

   same physical source and destination -> LOCAL
   restricted mode and selected/initialized pair -> PEER
   restricted mode and any other distinct pair -> HOST
   legacy mode -> existing decision path
   

   Implementation sequence:

   1. Obtain the actual runtime physical count and build the existing logical-to-physical mapping.
   2. Parse the complete allowlist into an initially empty matrix. Check conflicts and platform/build restrictions first.
   3. Validate endpoint ranges and both capabilities for requested pairs. Do not probe forbidden pairs by enabling them.
   4. Enable each selected ordered edge once, on the correct current CUDA device.
   5. Handle cudaErrorPeerAccessAlreadyEnabled as the documented idempotent case, with deliberate expected-error handling. Any other error terminates this opt-in initialization with the exact pair, UUID/PCI identity where available, API name, and CUDA error.
   6. Publish the immutable policy only after all selected directions have succeeded. Restore the caller's current CUDA device where the new initialization code changes it.
   7. Log the normalized allowlist, runtime ordinal-to-UUID/PCI mapping, and that all other distinct local CUDA pairs use host fallback.

   Initialization trap: ggml_cuda_info() is a function-local static initialized by ggml_cuda_init(), and the existing logical-to-physical helper calls ggml_cuda_info(). Do not call that helper recursively while constructing the same static. Build initialization policy from the local info object and raw runtime ordinals; use normal helpers only after publication.

   Do not repeatedly enable/disable access per request, batch, or graph. Do not call cudaDeviceReset, disable arbitrary peer mappings belonging to another library, or swallow every CUDA error to make startup appear successful.

   This is an application transport policy, not a security boundary against arbitrary CUDA code in the same process. The supported child process must initialize and retain CUDA state through the audited paths. Previously initialized external communicators or other libraries that independently add mappings are outside v1's guarantee. The implementation must prevent its own collective path from creating that situation rather than trying to undo live third-party mappings afterward.

   6. VMM must use the same directional policy

   The current VMM branch grants peer access when the old P2P environment variable exists or the build defines GGML_USE_NCCL. Filtering only the initialization loop leaves this broad allocation policy intact.

   For every newly mapped VMM span owned by physical device owner, restricted mode must construct exactly:

   access(owner) = READWRITE
   access(reader) = READWRITE only when policy permits reader -> owner
   all other physical devices = no access on this new mapping
   

   Use backing physical devices and deduplicate them. Precompute the per-owner reader list after initialization if that simplifies the allocator. No getenv, topology shell execution, or repeated all-pairs capability scan belongs in the allocation hot path.

   Keep the current allocate/map/release-handle lifecycle. Apply access before returning the new span for use. Maintain the local-owner descriptor even when it has no selected peer. The restricted branch must take precedence over the existing NCCL compile-time widening; the legacy branch remains unchanged when the new variable is absent.

   Do not confuse CU_MEM_ALLOCATION_TYPE_PINNED with pinned host staging: this allocator specifies CU_MEM_LOCATION_TYPE_DEVICE, so its backing allocation is on the GPU. The NVIDIA VMM explanation and driver API describe this separate allocation/access model.

   Also, omitting a device from a later cuMemSetAccess call is not an acceptable substitute for proving that an already accessible region was revoked. V1 avoids runtime permission changes by fixing policy before allocation. Test newly allocated spans, pool growth, and any unmap/remap lifecycle; use cuMemGetAccess to verify owner/allowed/denied access flags explicitly.

   Ordinary CUDA backend buffers in this snapshot use ggml_cuda_device_malloc; VMM is used by temporary pools. Tests must cover both families. Do not assume that exercising a cudaMalloc microbenchmark or an ordinary tensor copy automatically exercises VMM-backed scratch memory.

   If future code introduces cudaMallocAsync pools, imported allocations, IPC, or managed allocations into the supported path, their access rules must be reviewed before extending this guarantee. V1 does not claim that the existing runtime peer-enable loop controls every allocation API CUDA offers.

   7. Copy dispatch: close both fallback escape routes

   7.1 Why one callback change is insufficient

   The generic API has this path:

   ggml_backend_tensor_copy_async
     -> CUDA cpy_tensor_async
          -> if declined: synchronize source/destination
             -> ggml_backend_tensor_copy
                -> CUDA buffer cpy_tensor
                     -> if declined: malloc host storage
                                     source D2H
                                     destination H2D
                                     free host storage
   

   The scheduler also has a direct path in ggml_backend_sched_compute_splits:

   wait for destination copy slot's prior use
     -> call split_backend->iface.cpy_tensor_async directly
          -> if declined: synchronize producer
                          synchronize destination reuse event/backend
                          ggml_backend_tensor_copy
   

   It does not always go through the generic async wrapper. This is why policy enforcement belongs in both CUDA callbacks and why both scheduler and public-copy entry points need coverage.

   7.2 Required callback behavior

   Preserve all existing backend/buffer type checks and logical-device ownership checks. Resolve actual backing physical IDs, including tensor views through their actual backing buffers.

   For an excluded cross-device pair, return false before calling a CUDA copy API, recording a copy-completion event, or modifying destination data. In the buffer callback, return false for the same pair so the generic host path is actually selected.

   For an allowed pair, preserve cudaMemcpyPeerAsync and the existing source-stream completion event plus destination-stream wait. For same-physical copies, preserve local D2D, even when source and destination are different virtual backends. Do not convert same-device copies into host transfers.

   The forbidden-pair path must not substitute cudaMemcpyDefault, a cross-device cudaMemcpyDeviceToDevice, a different peer API, or an implicit remote-pointer kernel access. It is deliberately DeviceToHost followed by HostToDevice.

   Do not switch on global GGML_CUDA_NO_PEER_COPY to implement selection. In this source it affects both peer-copy availability and event capabilities, and would remove the desired fast path for every pair.

   7.3 Correctness argument for the minimal version

   Under the supported process/allocation scope:

   • Peer-enable operations are limited to selected directions.
   • VMM permissions are limited to the same physical owner/reader relationships.
   • Excluded cross-device copies cannot reach either CUDA peer-copy callback path.
   • Existing fallback synchronization completes the producer and protects destination reuse before the blocking host transfer.
   • Allowed transfers retain the backend's event ordering.

   These properties give a source-level reason to expect correctness. They do not replace hardware verification, and they do not establish an inference speedup.

   8. NCCL: preserve the build, constrain actual use

   The current CUDA communicator dispatcher is separate from layer copies. In addition, upstream PR #21891 explicitly removed communicator creation from generic GPU initialization. This is the reason a build with NCCL support need not be discarded for this layer-split feature.

   However, the inspected NCCL P2P source calls cudaDeviceEnablePeerAccess for same-process ranks and contains separate VMM/IPC import paths. This is evidence that another transport owner can modify relevant access state; it is not a claim about the exact NCCL library currently installed on the host.

   For v1, place a clear restricted-mode guard at the CUDA collective initialization boundary, before calling ncclCommInitAll, setting up an internal reduction, or silently choosing the generic meta reduction. A null return that merely activates an unreviewed fallback is not a rejection. Use the backend's established error mechanism and name the unsupported combination.

   The guard must be based on actual collective use, not merely on GGML_USE_NCCL being defined. It must cover direct library callers as well as the server CLI. Keep legacy tensor/NCCL behavior unchanged without the new opt-in.

   Do not force GGML_CUDA_ALLREDUCE=internal as a four-GPU solution: the inspected internal path is a two-GPU implementation. Similarly, GGML_CUDA_ALLREDUCE=none means fallback behavior, not proof that every tensor-parallel path has disappeared.

   NCCL_P2P_LEVEL=NVL is relevant to a future NCCL-specific implementation, but it is a topology cutoff rather than this arbitrary endpoint matrix. NCCL_P2P_DIRECT_DISABLE=1 restricts direct access to user buffers; it does not disable every NCCL P2P transport. Neither variable repairs the two llama.cpp copy callbacks.

   Extending the feature to tensor/collective inference requires separate evidence for communicator transport choices, imported/mapped buffers, allocation access, reductions, and teardown. It is not a hidden prerequisite for shipping the requested layer-split support.

   9. Host fallback performance: minimum first, bounded optimization if needed

   9.1 Required initial path

   Use the existing generic host fallback after both CUDA callbacks decline. It is deliberately conservative and adds no new asynchronous resource lifecycle. Its per-copy malloc/free and synchronization can be expensive, so performance against the peer-disabled baseline is an open measurement, not an assumed win.

   The recorded route environment includes GGML_CUDA_NO_PINNED. Keep that setting unchanged in the first controlled comparison. Do not make a pinned-memory optimization look like a P2P-only improvement by silently changing another variable.

   9.2 Optional second step, only if profiling justifies it

   If host staging dominates and the first candidate does not meet the measured performance target, add a bounded reusable staging implementation behind the same classifier. This is an optimization of excluded transfers, not a relaxation of the exclusion.

   Design requirements:

   • Respect GGML_CUDA_NO_PINNED: when present, retain the non-pinned correctness path. Testing a pinned pool requires an explicitly distinguished configuration experiment.
   • When pinning is allowed, use a small reusable, portable pinned-host pool rather than pin/unpin per tensor. Allocate lazily and cap total staging memory across active transfers; chunk large copies.
   • Keep state owned by CUDA backend/context resources, not a server-global service. No new background thread, host-callback CUDA calls, or global device synchronization is required by the design.
   • Pool exhaustion causes bounded local backpressure or the established blocking host path, never peer-copy fallback on a forbidden pair.

   One conservative ordering for each reusable chunk is:

   source stream: producer -> D2H(chunk) -> record E_d2h
                                               |
   destination stream: prior destination use -> wait E_d2h
                       -> H2D(chunk) -> record E_h2d
                                               |
   source stream:                      wait E_h2d before later reuse
   

   The destination stream naturally orders its consumer after H2D. Retaining a source-side wait until H2D completion is a conservative way to preserve the existing copy-completion/lifetime contract; relaxing it requires a separate proof that every source-buffer reuse and synchronization observer is still correct.

   The host slot is reusable only after H2D completion, not after D2H completion or successful enqueue. Event records must be issued before dependent waits, and event/slot generations must not be overwritten while an earlier transfer still observes them. Avoid cyclic waits across simultaneous opposite-direction transfers. Use fresh per-slot generations and exercise wraparound.

   Cancellation does not cancel already queued CUDA work. Drain or retain the relevant transfers before releasing staging memory, source/destination buffers, streams, or events. A failed partially enqueued transfer must not return false and cause the generic path to execute a second copy as though nothing happened.

   Keep transfers byte-preserving; do not introduce FP16/BF16 conversion as a transport optimization. Check graph-capture compatibility before choosing a path: do not attempt a blocking fallback after capture has already become invalid, and do not globally disable CUDA graphs to conceal a staging defect. Scheduler transfer boundaries and the backend's kernel-capture boundaries must be tested separately.

   This optional optimization must beat or justify its complexity against the conservative implementation. Do not add it merely to complete an architectural wish list.

   10. Topology verification and diagnostics

   At startup, log enough to audit what was selected:

   restricted CUDA peer policy: 0-1,2-3
   runtime CUDA ordinal -> full GPU UUID -> PCI bus ID
   selected directional peer edges: 0->1,1->0,2->3,3->2
   all other distinct local CUDA pairs: explicit host fallback
   VMM access policy: owner + selected readers
   collective mode with restricted policy: unsupported in v1
   host staging: existing non-pinned fallback / validated pinned path
   

   These are proposed log contents, not existing output. Do not print credentials or unrelated environment values.

   For the target machine, confirm that the two selected UUID pairs still have active direct NVLink connectivity before the isolated run. An external preflight can use NVML link state and remote PCI identity, matched to CUDA UUID/PCI identities. Do not use NVML enumeration indices as CUDA indices. Do not infer NVLink from a performance-rank number or from CanAccessPeer=1.

   Automatic NVLink discovery is not needed for the minimal manual allowlist. A future strict topology-validated mode would need its own missing-library, unsupported-query, link-down, and mapping-change semantics. V1 must not promise that an endpoint allowlist alone pins physical routing or detects every link failure after launch.

   For test/profiling builds, expose per-direction peer/host copy counts and bytes and record actual peer-enable/VMM-access operations. Keep this bounded and outside production per-token logging. Current NVML documentation deprecates older utilization-counter APIs; use counters supported by the installed driver or a compatible profiler, and label unavailable evidence rather than reporting fabricated zero traffic.

   11. Proxy/bridge integration dependency

   Backend implementation alone will not make a new route environment key usable end to end.

   Fresh reads of the current proxy launcher and bridge validator show GGML_CUDA_P2P in ALLOWED_ENV_KEYS, but not the proposed GGML_CUDA_P2P_PAIRS. PR #472 provides the relevant existing two-boundary integration pattern.

   A narrowly scoped companion change in llama-proxy is therefore required before managed-route acceptance:

   1. Add the new key to both allowlists and validate its bounded textual grammar consistently.
   2. Reject simultaneous old and new P2P keys at both boundaries, including any conflicting inherited effective child environment. Do not alter unrelated route environments.
   3. Let the CUDA child remain authoritative for actual device count, capability, and mapped identities; the proxy cannot infer these from its own machine's GPU enumeration.
   4. Preserve the existing full-UUID CUDA_VISIBLE_DEVICES validation, device arguments, tensor split, MTP placement, context, parallelism, RPC configuration, and GGML_CUDA_NO_PINNED setting.
   5. Include the new value in the existing rendered launch envelope/hash and verify that the launched child's effective environment and startup diagnostics agree with route readback.
   6. Extend tests/test_launcher_spec.py and tests/test_bridge_ports.py for acceptance, invalid syntax, conflicts, pass-through, and launch-hash changes. Do not confuse those unit tests with GPU acceptance.
   7. Update the route/bridge documentation at the existing location rather than create a second configuration rulebook.

   This comment does not request or perform that companion repository write. It records an actual deployment dependency so the later implementation is not declared complete after a direct CLI-only test. Do not enable the new key against an old binary that silently ignores it and then claim selective mode was activated.

   12. Verification plan

   12.1 Parser and decision tests

   Exercise valid pairs, reversed duplicates, whitespace, overflow, negative signs, trailing junk, empty entries, self-pairs, missing endpoints, endpoint count changes, and old/new variable conflicts. Test complete validation before peer-enable side effects.

   Verify that 0-1,1-2 never permits 0-2; that all eight cross-group directions in the target matrix remain forbidden; and that logical aliases of the same physical device select local D2D.

   Test failure of either direction of a selected pair, expected already-enabled handling, and unexpected CUDA errors. A configuration error must not leave a model serving with a partially applied policy.

   Prefer a small CUDA-private pure policy helper that existing test infrastructure can exercise, rather than duplicating the production parser in a test script. GPU-specific allocation fixtures may need a small test support translation unit; do not add a large standalone framework or a public runtime API solely to facilitate tests.

   12.2 Four-GPU transfer correctness

   In an isolated candidate process, keep all four physical GPUs visible in the verified UUID order. Exercise all 12 ordered distinct-device transfers, plus same-physical behavior. For forbidden directions, test the host fallback without ever enabling that direction.

   Cover at least zero-length handling, tiny/odd sizes, alignment boundaries, 4 KiB, 64 KiB, 1 MiB, the recorded 16 MiB case, and chunk-boundary sizes if chunking is introduced. Bound peak GPU and host allocation sizes.

   Use data patterns that encode source, destination, iteration, and offset. Reset destinations to a different sentinel for every iteration and validate every byte plus guards around copied ranges. A stale successful previous copy or an all-zero tensor must not pass the test accidentally.

   Allocation/entry-point coverage must include:

   • ordinary cudaMalloc allocations;
   • actual ggml CUDA tensor buffers and valid views;
   • actual VMM-backed temporary allocations, growth/remap, and cuMemGetAccess checks;
   • public async-copy wrapper, synchronous buffer copy, and direct scheduler callback/fallback path;
   • a small producer/copy/consumer graph that checks results, not only a raw runtime memcpy loop.

   For every forbidden direction, test instrumentation must show zero forbidden peer-enable calls, zero forbidden VMM grants, and zero forbidden peer-copy dispatches. Review alternate D2D/default copies and imported-memory paths to ensure they cannot bypass the policy in the supported workload.

   Do not run an unmodified all-pairs P2P benchmark on this machine. Some generic diagnostic samples enable every capable pair; that would deliberately recreate the known-bad condition rather than test the exclusion.

   12.3 Ordering, reuse, and failure cases

   Queue producer work before a copy and a destination checksum/consumer afterward. Test forward/reverse traffic, repeated copies between the same endpoints, different backend instances, multiple scheduler copy slots, and source/destination buffer reuse.

   Exercise request cancellation and teardown while work is queued, repeated model load/unload in the supported process model, and graph capture/replay where enabled. For a staging pool, additionally test capacity exhaustion, slot wraparound, allocation failure before enqueue, error after partial enqueue, and destruction with pending H2D work.

   Preserve valid destination-reuse events from the scheduler. Do not treat a successful CUDA API return or lack of a kernel Xid as proof that bytes or ordering are correct.

   12.4 Build and scope matrix

   Required candidate coverage includes NVIDIA CUDA with VMM enabled and disabled, NCCL compiled in and out, restricted mode absent and present, and explicit conflict handling for NO_PEER_COPY and managed-memory opt-in. Verify that a collective attempt is rejected before it can enable or map an excluded pair.

   Legacy single-GPU and multi-GPU behavior without the new variable must remain unchanged. Check relevant HIP/MUSA compile guards so the NVIDIA change does not break their builds; do not report runtime support on hardware that was not tested.

   Use the verified host toolchain that can actually target the V100, and record compiler/runtime/driver/NCCL versions. Reading newer documentation is not an instruction to upgrade the host toolkit or change its architecture flags during this experiment.

   13. End-to-end Qwen acceptance

   13.1 Control the experiment

   There are two separate comparisons:

   1. Same candidate binary, restricted mode absent vs present. This isolates the new transport policy.
   2. Delivered candidate vs the recorded/remeasured production baseline. This checks whether source-revision or build differences introduce additional regressions.

   Do not compare a new fork binary against an old installed binary and attribute every difference to P2P. Record candidate commit, local diff state, binary identity, build flags, actual loaded CUDA libraries, model identity, and full launch configuration.

   Retain the exact four-GPU route recorded in this issue:

   --device CUDA0,CUDA1,CUDA2,CUDA3
   -sm layer
   -ts 28,28,29,15
   --spec-type draft-mtp
   --spec-draft-device CUDA3
   --spec-draft-n-max 3
   --spec-draft-n-min 1
   -c 400000
   --parallel 4
   

   Keep the existing model, UUID order, RPC configuration, host settings, and other environment values. A two-GPU substitute, a smaller model, disabling MTP, reducing context, or changing the split is not final acceptance. Target-only and smaller synthetic tests are diagnostics, not replacements for this workload.

   13.2 Run sequence and measurements

   After isolated correctness passes and runtime execution is separately authorized, use fresh children for baseline -> restricted -> baseline rollback. Do not insert a known-bad enable-all condition. Start with the same bounded deterministic 256-token request and then exercise sustained/multiple-slot operation at the configured parallelism.

   Use several repeated requests and report individual results plus a summary, not one favorable run. Match prompt-cache state and warmup policy across conditions. Record GPU clocks, temperature, competing load, and throttling sufficiently to identify an invalid comparison.

   Capture:

   • output/completion correctness and token IDs where available;
   • MTP draft generated/accepted counts and target verification work;
   • native decode tokens/s, prompt-processing tokens/s, TTFT, and client elapsed time;
   • per-pair copy bytes/counts and time spent in host fallback/synchronization;
   • CPU/pinned-host memory use if the optional staging path is tested;
   • CUDA errors, timeout/cancellation behavior, and cleanup results.

   The recorded reference is roughly 44-45 decode tokens/s with 159/285 draft acceptance for that request, not the broken 9.5 tokens/s condition. These exact counts are not universal constants for every prompt or build.

   For genuinely deterministic same-build cases, investigate any output or draft-acceptance divergence. If baseline runs themselves vary, establish that variation first and compare against it; do not dismiss a large divergence as ordinary nondeterminism. Byte mismatches in isolated copies are always a failure.

   Time GPU events only in valid same-device configurations. For a staged transfer crossing two devices, use an end-to-end completion boundary and appropriate host timing rather than subtracting timestamps from unrelated GPU clocks. Report whether bandwidth is one-directional or aggregate and what byte count was divided by elapsed time.

   13.3 What the performance result can mean

   A 131-133 GB/s copy measurement does not imply an equivalent inference speedup. Let f be the fraction of baseline time spent in transfers actually accelerated by the two NVLink links and s their measured speedup. Even an idealized model gives 1 / ((1-f) + f/s) before adding any change in cross-group fallback cost. The real workload also includes computation, synchronization, MTP, and other copies.

   Report correctness, transport selection, and end-to-end performance separately. Correct selective transport with no demonstrated workload gain is useful implementation evidence, but it is not proof that this production route should be promoted as faster. A correctness failure or forbidden-path use is FAILED; missing hardware evidence is PARTIAL, not PASS.

   14. Implementation sequence and completion checklist

   The steps below are coding/verification dependencies, not extra owner approval stages. The present authorization stops at this plan comment.

   Step A: policy and source-path coverage

   • Recheck candidate head/instructions and inventory every relevant peer-enable, VMM grant, peer/default cross-device copy, and communicator entry point.
   • Implement the strict, immutable physical-pair policy and startup diagnostics.
   • Cover parsing, normalization, identity mapping, conflicts, and error behavior with focused tests.

   Step B: minimal complete backend implementation

   • Gate peer initialization, VMM grants, async copy, and buffer copy with the same policy.
   • Preserve local D2D, virtual aliases, allowed-pair events, and existing scheduler fallback ordering.
   • Reject unsupported collective initialization before any external transport setup.
   • Keep old behavior when the new option is absent; document the new scope and rollback.

   Step C: isolated correctness and profiling

   • Test all four GPUs, all 12 ordered cross-device directions, and both ordinary/VMM allocation families.
   • Show zero forbidden enable/grant/peer-copy operations and zero byte mismatches.
   • Exercise actual ggml callbacks, scheduler copies, views, reuse, cancellation, and teardown.
   • Measure NVLink acceleration and the real host-fallback cost. Add the bounded optimized fallback only if needed and validate it independently, respecting GGML_CUDA_NO_PINNED.

   Step D: managed-route integration and live acceptance

   • Deliver the narrowly scoped proxy/bridge environment validation dependency and its tests when that repository work is authorized.
   • Verify route readback, launch envelope/hash, actual child environment, selected binary, UUID mapping, and effective policy logs.
   • Complete fresh-process baseline/restricted/rollback tests on the unchanged four-GPU Qwen/MTP route.
   • Exercise the configured four-slot operation and provide raw correctness, draft acceptance, decode/prefill, latency, copy-path, and cleanup evidence.

   Required evidence before closing this issue as implemented and accepted

   • All required implementation and correctness items above have evidence tied to the delivered candidate.
   • The two NVLink pairs are accelerated while every cross-group direction uses a correct explicit host path.
   • NCCL build support does not widen the policy or initialize an unreviewed communicator in the accepted layer route.
   • No model/split/context/MTP/RPC downgrade was used to make acceptance pass.
   • End-to-end results are compared with a fresh healthy baseline, and any regression or inconclusive gain is disclosed rather than converted to PASS.
   • Documentation, launcher dependency, and rollback procedure match the delivered implementation.
   • Any optional step not implemented is explicitly marked not shipped, with a reason and no claim that its tests passed.
   • The issue is not auto-closed merely because a PR merged or a model loaded successfully.

   15. Remaining uncertainties and final recommendation

   No isolated test has yet established reverse-direction NVLink correctness, VMM behavior under this exact restrictive matrix, or a four-GPU Qwen/MTP performance gain for a patched backend. No source-only argument can establish those runtime results.

   The generic fallback may be slower than CUDA's implicit peer-disabled copy on this particular driver. The endpoint allowlist does not repair the PCIe platform fault and does not itself provide continuous NVLink health enforcement. Third-party CUDA state and future allocator/collective paths require explicit scope control.

   Nevertheless, the requested feature has a concrete, bounded implementation path: one physical-pair policy, matching VMM permissions, both copy callbacks gated, existing scheduler synchronization retained, and an explicit host fallback for excluded pairs. This directly addresses the demonstrated unsafe path without sacrificing the VRAM of the four-GPU route or relying on undocumented transport selection. Keep production P2P disabled until the candidate demonstrates correctness and the measured performance outcome is reviewed.
2. novkien commented on Sep 30, 2026

   

   novkien

   on Sep 30, 2026 – with ChatGPT Codex Connector

   OwnerAuthor

   More actions

   Pre-implementation experiment and documentation draft

   Prepared draft PR #4: docs: prepare selective CUDA P2P implementation and preflight.

   It contains only seven Markdown files at documentation commit 990a75740c7b4b4b2d708cc479cf53d213050771, based on 526c43b8f7dfea9032e9f35e7a1be9183ca7cc20. No feature source, executable repository test, build/workflow, launcher, or production configuration was changed. The proposed environment option is still unimplemented.

   Coder entry point | Implementation map | Pre-code experiment specification | Actual results and raw transcript.

   1. Question tested

   Before implementing a backend feature, can the existing copy/scheduler branches preserve the permitted peer path and reach the generic host fallback for the excluded pair in a four-stage handoff?

   Result: PASS for the executed source-branch scope; PARTIAL for native four-GPU feasibility. A meaningful local branch experiment was possible, but this environment could not run the requested CUDA/NVLink hardware confirmation.

   The container has no NVIDIA device nodes, nvidia-smi, nvcc, or loadable CUDA driver library. Its actual driver probe returned:

   find_library(cuda)= None
   CUDA_LOAD_ERROR: libcuda.so.1: cannot open shared object file: No such file or directory
   

   No remote GPU host was connected or operated. Do not interpret the following as a real CUDA execution or a confirmed production fix.

   2. What actually ran

   A disposable host-only C++ probe compiled these five copied function bodies from the pinned fork source:

   • ggml_backend_buffer_copy_tensor
   • ggml_backend_tensor_copy
   • ggml_backend_tensor_copy_async
   • ggml_backend_cuda_buffer_cpy_tensor
   • ggml_backend_cuda_cpy_tensor_async

   It also compiled the exact ordinary split-input fallback block and immediate user-input copy block from the scheduler inside fixture wrappers. The full scheduler, graph allocator, model placement, VMM allocator, peer initialization, and NCCL were not executed.

   CUDA APIs, memory accessors, streams/events, physical mapping, and external policy decorators were software test doubles. The decorators supplied the proposed matrix; they were not a new production parser or backend patch. Peer API doubles perform CPU copies and record endpoints.

   Two final builds/runs completed successfully, using -Wall -Wextra -Werror with UBSan and with ASan+UBSan. Both executions returned 0, produced the same transcript, and emitted no sanitizer diagnostics for this harness. The initial scaffold-only compile warnings and their correction are disclosed in the results document.

   Executed scope	Observed result
   12 directed pairs x 6 entry paths x 8 sizes x 3 patterns	1,728 combinations; zero mismatching bytes and zero forbidden peer dispatches with both guards.
   Same-backend copies	24 combinations passed with local D2D selection.
   Physical aliases across different backend instances	48 combinations passed with local D2D selection.
   Valid views with populated backing buffers	72 combinations passed.
   Synthetic forward and reverse handoff chains	Both selected PEER, HOST, PEER and matched an independently maintained CPU result.

   Total: 1,874 positive scenario combinations, not 1,874 independent GPU regression tests.

   3. Actual traces and negative controls

   For the excluded CUDA1->CUDA2 direction, the copied scheduler branch and both guarded callbacks produced:

   async_decline
   -> backend_sync:1
   -> event_sync
   -> buffer_decline
   -> D2H:1
   -> H2D:2
   

   For allowed CUDA0->CUDA1:

   peer_API:0->1
   -> record_copy_event
   -> wait_copy_event
   

   The synthetic chains produced:

   0 -> 1 -> 2 -> 3 : PEER, HOST, PEER
   3 -> 2 -> 1 -> 0 : PEER, HOST, PEER
   

   Deliberately incomplete guards were also tested. The detector checks requested peer-call endpoints independently of whether CPU bytes happened to match:

   Negative control	Forbidden peer dispatches detected
   Guard only the async callback	48/48 denied-edge entry combinations
   Guard only the buffer callback	24/48 denied-edge entry combinations
   Guard neither callback	48/48 denied-edge entry combinations

   The async-only trace exposes the escape route:

   async_decline
   -> backend_sync:1
   -> event_sync
   -> peer_API:1->2
   -> stream_sync
   

   Thus, a change that merely rejects the async copy is insufficient: the original buffer-copy callback can still invoke the peer API. Conversely, a buffer-only guard misses async-success paths. The scheduler's immediate user-input path also needs the buffer guard because it calls synchronous tensor copy directly.

   These negative controls do not reproduce PCIe corruption. Their software memory copies still match; they establish that byte correctness alone cannot prove the required transport exclusion.

   4. What this confirms, and what it does not

   This supports a bounded implementation strategy: keep layer placement and the existing scheduler algorithm, but gate both CUDA copy callbacks using one physical-pair policy. The generic host fallback is reachable for an excluded pair while the permitted callback path is retained.

   It does not establish actual four-GPU graph placement, native event timing, physical NVLink routing, VMM permissions, NCCL isolation, or Qwen/MTP performance. It also does not confirm the proposed parser because the local policy was explicitly supplied by the test harness.

   "Normal fallback" needs a precise interpretation. With the old P2P key absent, current CUDA callbacks can still call cudaMemcpyPeerAsync and rely on runtime transport selection. The proposal instead uses the existing explicit host get/set fallback for excluded pairs. This is a correctness-oriented path, not a promise of identical legacy latency or buffering.

   5. Native confirmation before committing to the feature

   The PREFLIGHT specification defines two remaining experiments without requiring a production feature patch:

   B. Disposable four-GPU native transport/VMM probe. Resolve full UUID/PCI identities and verify the intended NVLink pairs. Explicitly enable only 0->1, 1->0, 2->3, 3->2. Test all 12 directed transfers, ordinary allocations and separately created VMM mappings, with owner/selected-reader access queried by cuMemGetAccess. Use real peer copy only for allowed edges and explicit D2H/H2D for excluded edges. Validate every byte/guard and both forward/reverse chains. Never enable the known-bad PCIe pair as a negative control.

   C. Whole ggml scheduler with actual CUDA backends. Construct a small four-stage graph, explicitly assign stages to four real CUDA backends with the current scheduler API, then verify actual graph splits and node placement. Run independent producer/consumer checks and trace each activation handoff. Expected adjacent route is PEER/HOST/PEER; add reverse and nonadjacent cases, input-copy paths, views, and scheduler copy-slot reuse.

   Before feature implementation, C can use clearly identified test-only callback decorators in a disposable native harness, preserving original callbacks for allowed/local edges. This requires a version-specific adapter; it is not already implemented here. A callback-only test does not validate the current VMM allocator's NCCL-driven access widening: actual-pool coverage needs a separately identified test seam or remains PARTIAL. Do not disable VMM/NCCL and claim production-equivalent confirmation.

   After implementation, rerun C without test policy decorators against the exact delivered candidate, then run unchanged four-GPU Qwen/MTP baseline -> selective -> rollback plus four-slot operation. The TESTING checklist specifies the remaining acceptance evidence. No two-GPU/model/context/MTP downgrade substitutes for that test.

   6. Reproducibility and disposition

   The result document includes the complete transcript, environment, compiler commands, and SHA256 hashes of source specimens, scaffold, and logs. A separate conversation artifact contains those disposable files and the docs snapshot; no executable binaries or feature code were added to the draft PR.

   Disposition: documentation preparation complete; source-branch preflight passed within its stated scope; native feasibility remains PARTIAL. Keep this issue open and PR #4 in draft. No merge, deployment, service restart, route edit, or host-setting change was performed.
3. novkien

   added a commit that references this issue on Sep 30, 2026

   cuda: implement selective CUDA P2P transport

   ...

   ed1d2b9
4. novkien commented on Sep 30, 2026

   

   novkien

   on Sep 30, 2026

   Last edited by novkien

   OwnerAuthor

   More actions

   PR #4 (cuda: implement selective CUDA P2P transport) is the reviewed delivery head ed1d2b9. It preserves the PR #2 cache tree and adds the selective physical CUDA peer transport plus its evidence pack. Independent review found no source defect. Merge is followed by clean native/RPC builds and live acceptance; this report does not close issue #3.
5. novkien commented on Sep 30, 2026

   

   novkien

   on Sep 30, 2026

   OwnerAuthor

   More actions

   Work report - backend delivered; live acceptance in progress

   Owner and scope: Codex root task owns fork issues #1/#3. The owner explicitly requested build/deployment, Qwen-27B backend change and tests/closure in this task on 2026-09-30. Later steering explicitly authorized proxy hard-restart recovery, required CUDA order starting with the Gen3 x8 GPU, and requested reducing context/rebalancing the split so pipeline parallelism remains enabled. The resulting configuration is intended for their continued use.

   Done and evidence: PR #2 merged at 7865d7f6b3761cd21b4ab7dbfc91c3d78b83164a; its combined native tests passed 33 checks and 14 focused HTTP cases. PR #4 merged at 95e6565970621aa508e80fd805dcf73af413093c after main review and independent review found no actionable defect in the exact CUDA source. Native and RPC guest checkouts advanced cleanly, both builds passed and all installed executable/library hashes were verified. Both manifests identify 95e6565970621aa508e80fd805dcf73af413093c:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855. Native uses CUDA 12.9/sm70/NCCL/RPC; guest uses CUDA 13.3/sm120/RPC. Guest retained NCCL=ON configuration but the package was unavailable and NCCL was not linked there. The supported RPC helper started and verified the fork daemon on port 50052.

   Launcher validation merged in novkien/llama-proxy#484. The managed activation stalled on retirement of four long-lived Godot observer streams. Under the owner's subsequent explicit hard-restart authorization, only the proxy service underwent cold recovery; the job was not called safely completed. Those streams reconnected. The reviewed bridge code is active and its authenticated child readback matches the fork executable hash cd5d9185c1e781e9716cbd323fb6295ff6b89c14aabc8010ac75f424bfb0ac78. The old global P2P key is absent.

   Configuration steering: actual hardware is four V100-SXM2 16 GB, not 32 GB. This corrects the capacity typo in NATIVE-RESULTS; the UUIDs, native test bytes and transport results are unchanged. A documentation correction accompanies the final acceptance report. Current runtime order is PCI 08:00 (Gen3 x8) -> NV6 -> 0c:00 -> PIX -> 0d:00 -> NV6 -> 09:00, matching UUID order 5ca3,b2e,e149,d4d. Startup and effective child environment both confirm that order.

   The original -c 400000 -ts 28,28,29,15 DFlash route loaded but its main scheduler retried without pipeline parallelism after compute-buffer OOM. The owner's new request supersedes that context/split constraint. The current trial uses -c 200000 -ts 29,29,27,15, preserves four GPUs/four slots and the current DFlash2 draft, and keeps GGML_CUDA_NO_PINNED=1. Native allocates an aligned 200192-token unified context; startup reports pipeline parallelism enabled, approximately 4431 MiB main compute buffers per GPU and no fallback. Small four-slot requests left approximately 1901-2873 MiB free per GPU. Different context/split profiles are not used as a P2P-only performance comparison.

   Remaining and next: finish fresh same-profile selector-off/on/rollback requests and historical MTP-method comparison under the new owner-selected memory profile; run meaningful long-context Qwen/Nex RAM retention, borrowed snapshot, cache opt-out and media checks; report actual outputs/latency/memory and retain the verified usable Qwen profile. Optional SSD spill is not implemented. Both fork issues stay open until their applicable live checks pass.
6. novkien commented on Sep 30, 2026

   

   novkien

   on Sep 30, 2026

   OwnerAuthor

   More actions

   Deployed acceptance complete for selective four-GPU transport

   PR #4 merged at 95e6565970621aa508e80fd805dcf73af413093c; native and RPC guest built and installed this same clean source, with matching source IDs and verified libraries/service. This includes PR #2. The launcher/bridge dependency, llama-proxy PR ggml-org#484, is deployed. Each live phase checked the actual child's executable, source manifest, arguments and CUDA environment.

   The owner later explicitly approved the persistent Qwen context/split adjustment to -c 200000 -ts 29,29,27,15 to preserve pipeline parallelism. Matched trials used that profile, four slots, all four V100-SXM2 16 GB GPUs, full UUID order 5ca3,b2e,e149,d4d, q8_0 KV and unchanged per-method speculation. This supersedes the original 400000-context/split equality constraint; no two-GPU substitute was used. Pipeline parallelism remained enabled through the long cache prefills.

   Native coverage from PR #4 passed byte checks on all four permitted and eight excluded directions, actual malloc/VMM pool permissions, both callbacks, physical aliases and whole-scheduler graphs. Selected 16 MiB copies measured 126-132 GB/s, versus 5.5-7.6 GB/s with explicit peer enable absent. Excluded pairs used the explicit host fallback correctly. No global/unsafe PCIe peer test was repeated; BIOS, driver, IOMMU and ACS were unchanged.

   Each fresh live phase ran three sequential deterministic historical requests and four simultaneous requests. All 42 completed:

   Method / phase	Median sequential tokens/s	Draft accepted/generated	Four-request max wall time, seconds
   MTP off	44.770	159/285	12.048
   MTP selected pairs	46.992	159/285	11.708
   MTP rollback off	44.408	159/285	12.099
   DFlash2 off	36.799	173/563	13.213
   DFlash2 selected pairs	36.736	173/563	13.720
   DFlash2 rollback off	40.720	173/563	14.018

   All nine sequential outputs within each method matched exactly across phases. MTP measured about 5% sequential improvement. DFlash2 has no demonstrated general speed gain; rollback variation and the simultaneous-request slowdown versus the first baseline remain disclosed. Parallel outputs/draft counts varied across fresh phases, so those requests certify four-slot serving, not deterministic equivalence. Copy throughput is not an inference-speed promise.

   Qwen real long-context RAM/hybrid/DFlash2 and focused media append/restore checks passed. Nex RAM-output correctness remains unresolved in #1, and its independent cold-media draft failure is #6; neither is hidden by closure of this transport issue. Optional SSD is not implemented. Native limitations remain: no claim for complete scheduler copy-slot rotation, cancellation/race coverage, uncompiled platform variants or selective collectives (which intentionally reject startup).

   The final persistent Qwen route was revision-checked and read back: fork backend, DFlash2, context 200000, split 29,29,27,15, four slots and GGML_CUDA_P2P_PAIRS=0-1,2-3, with the correct full UUID pinning and global P2P absent. Nex was restored to its original backend/specification. After the owner requested idle release, Qwen was not loaded again; all agent-owned idle holds were released, renewal stopped, block=false and idle_shutdown_active=true was verified. Installed binaries remain at the tested source above; the follow-up documentation does not require a GPU rebuild.

   Raw Qwen phase records remain under /tmp/issue3-live-evidence on the GPU host. The native harness/evidence archive is retained in the owner's acceptance workspace. The docs-only follow-up corrects the erroneous 32 GB label and records deployed results. Issue #3 can close for this approved transport scope; #1 and #6 remain open.
7. novkien

   closed this as completedon Sep 30, 2026