[tang-vu]

Summary

Add install.ps1 -WindowsLlmBackend cuda to run x64 NVIDIA inference in the pinned Windows llama.cpp runtime. Portal services still run in Ubuntu/WSL and Docker Desktop. The default NVIDIA container route, AMD Vulkan route and Arm64 CUDA helpers remain intact.

Refs #7968. This supplies a shared Windows inference building block; it does not implement the full Docker-free desktop profile.

Why this matters

The existing native CUDA pin supports Arm64 only. An x64 NVIDIA owner cannot select Windows inference through the public installer even though the upstream x64 release works on the hardware. The new option completes driver/compute checks, catalog selection, verified two-archive acquisition, exact adapter qualification, authenticated native binding and managed lifecycle.

The choice persists across omitted-option reruns. Missing metadata, incompatible profiles, unsupported GPUs and failed qualification stop visibly. Failed cutover restores the previous task definition and files. Desktop CUDA model control does not apply Spark's unified-memory sizing. Verified old x64 installations can be retired while unknown directories remain untouched.

AI Assistance

tang-vu leads this contribution and its scope and quality requirements, collaborating with Codex on implementation and validation. Codex assisted with analysis, code, tests and independent static review. Recorded command-based checks below were agent-executed.

Release Lane

• Mainline change targeting main

Changed Surface

• Installer / bootstrap / lifecycle
• Model routing / capabilities
• Network exposure / auth / proxy
• Dependencies / runtime wiring

Risk And Validation

Risk: High, opt-in operational change. Candidate: 91acdc78d12280d068ecadbc86c79a8c4901140d; base: 6513ad2e0e205525f148b68fffd095cb1b347e2c.

• GitHub checks on the final head: 74/74 successful (checks).

• PowerShell 5.1 and 7: native runtime 148, CUDA runtime 146, new public CUDA route 48, Portal setup 257, model controller 223, recovery 45, and existing AMD/restart/migration/legacy suites pass. Regressions first demonstrated failures for route selection, SKU matching, task-definition rollback and architecture-specific memory handling. The new suite executes in Windows CI under both interpreters.

• PSScriptAnalyzer: zero errors/warnings on changed production scripts. Parser and test-workflow inventory pass. llama.cpp compatibility contract passes using an explicitly selected Git Bash interpreter. Commit-identity, secret scan and git diff --check pass.

• Real Windows 10 19045 / RTX 3060 12 GiB / driver 616.92: both official CUDA 12.4 b9014 archives downloaded and verified against release SHA-256 and sizes. The actual catalog recommendation, Qwen3.5-9B Q4_K_M, was downloaded and SHA-256 verified. Actual launcher proves 65,536 allocated context and 33/33 layers offloaded. Missing/wrong inference keys return 401; authenticated chat and terminal SSE completion pass. The final 86-token response measured 46.57 tokens/s; this is one-host evidence, not a fleet benchmark. Initial Vulkan comparison on the same host motivated the CUDA choice.

• Real private scheduled task: shared preparation/cutover, private-file validation, controller status, Stop, Start, Activate and post-activation model generation pass. Saved omitted-option routing also passes. QA used an isolated task/root and synthetic WSL binding metadata; no WSL stack was installed or exercised. The task was removed and its port freed afterward. Both the task lifecycle and final direct launcher/auth/SSE checks used this candidate's production source.

• Independent static review found and verified fixes for task-definition rollback, exact GPU matching, Tesla names and x64 runtime cleanup.

The Windows driver floor is conservatively 551.78 for CUDA 12.4 Update 1 (NVIDIA release notes). Only explicitly compiled compute targets are accepted; unlisted targets, including Blackwell/RTX 50, refuse this opt-in rather than guessing compatibility.

Operational Change Check

• This is an operational change with the scoped native-hardware validation above.
• Full non-elevated Windows 10/11 Portal installation, real WSL/Docker handoff, reboot/sign-in and wider GPU fleet release validation remain deferred. Maintainer review and release acceptance remain required.

Explicit -WindowsLlmBackend auto does not migrate an installed native route back to containers; it refuses before prerequisite changes. The documented supported rerun preserves the native choice. Specialized catalog runtime profiles are also refused until their native launch settings are qualified.

Overlap check

Searched open/closed Windows, NVIDIA and Vulkan work and inspected changed-file inventories. #7789 covers RTX Spark/Arm64; this keeps its pin and memory policy separate while sharing lifecycle helpers. #7970 and the #7968 desktop series concern the native Dashboard profile. #7953 concerns supervisor experimentation. #7938 is the separate desktop preview. None supplies this public x64 NVIDIA CUDA route; shared-file coordination with #7789 is expected.

[tang-vu]

@Lightheartdevs could you review the opt-in x64 NVIDIA CUDA route here, especially the shared cutover/rollback path and its interaction with the RTX Spark/Arm64 work in #7789?

Default routing is preserved, and x64 CUDA is kept separate from Spark's unified-memory policy. Validation includes the catalog 9B model at 64K context on a real RTX 3060 (~46.6 tokens/s), actual Windows task Stop/Start/Activate, and 74 passing checks. Portal services still use WSL/Docker; the remaining full-install and fleet release gates are documented in the PR.

[Lightheartdevs]

Thanks for carrying the x64 CUDA option through model selection, authenticated launch, and rollback while keeping the existing NVIDIA default.

Advisory status: needs human decision

The remaining decision is whether to accept this as an opt-in pre-release path with the documented validation limits. install.ps1 and ods/installers/windows/lib/wsl-portal-setup.ps1 expose the route through the public installer, but the reported RTX 3060/task validation used synthetic WSL binding metadata. A complete non-elevated install with real Ubuntu/Docker handoff, followed by reboot/sign-in and a Portal chat, is still deferred. That is an integration acceptance gap, rather than a reproduced code defect in this review.

If that evidence is required before merge, the smallest next validation is one real Windows x64/NVIDIA install using -WindowsLlmBackend cuda, then an omitted-option rerun and reboot/sign-in. Verify Portal chat, authenticated model access, and Dashboard Stop/Start/Activate, recording the exact commit and host. Keep the separate Arm64 work in #7789 coordinated because the runtime/control helpers are shared.

What I checked:

• Exact-head diff and surrounding route, runtime, rollback, and model-control code against current main's installer/runtime boundaries; all 74 reported checks passed.
• Independently ran the new x64 CUDA route fixture under Windows PowerShell, plus the CUDA runtime (146), recovery (45), and model-control (223) contract checks: all passed.
• Read the contributor's hardware evidence and explicit limitations; I did not repeat the hardware/full-install acceptance run.

The explicit refusal paths and preserved default make the remaining acceptance decision much clearer. Thanks for documenting the limits.

[tang-vu]

Agreed: the real Ubuntu/Docker handoff, omitted-option rerun, reboot/sign-in and Portal/model-control checks remain an open integration acceptance gate for this CLI route. The 91acdc78d RTX 3060/task evidence used the documented synthetic WSL binding metadata; its passing fixtures do not close that gate. The PR retains the opt-in scope and separate maintainer release decision, including coordination with #7789.

The desktop integration in #7938 now carries its own CPU/CUDA Portal and document evidence without Docker or WSL. Its native application receipts remain separate from this CLI route's full-install acceptance. Thank you for making that distinction explicit in the advisory review.