fix: gate kv_inplace_write GPU copy on a GPU backend build

TECHNICAL_REPORTS/2108-cpu-only-link-kv-inplace-write-20261005.en.md

@@ -0,0 +1,74 @@
+# Technical Report: Issue #2108 - CPU-only Linux build fails to link kv_inplace_write
+
+**Date**: 2026-10-05
+
+**Status**: Implemented and validated on GB10 (CPU-only and CUDA builds); based on origin/main `33c45053`, pending merge.
+
+**Languages**: C++ (bridge preprocessor guard), Rust (`build.rs`, build-policy predicate and unit tests), Markdown (installation guide)
+
+**Risk Level**: Low (the GPU translation unit is unchanged apart from one predefined macro; the CPU-only build gains a throwing `eval_gpu` that no code path can reach)
+
+## Executive Summary
+
+`mlxcel-core/build.rs` compiles `src/lib/mlx-cpp/turbo/kv_inplace_write.cpp` (#1959) for every backend. Its `InplaceSliceWrite::eval_gpu` calls `mlx::core::copy_gpu_inplace`, which MLX defines only in `mlx/backend/gpu/copy.cpp`, a file built only with a GPU backend. A Linux build with no `cuda`, `metal` or `rocm` feature therefore failed at link time. The fix adds `MLXCEL_BRIDGE_GPU_BACKEND`, defined by `build.rs` exactly when MLX builds Metal, CUDA or ROCm, and puts the `copy.h` include and the `eval_gpu` body behind it. Without it `eval_gpu` throws, as `eval_cpu` already does.
+
+## 1. Problem Statement
+
+The bug was found while validating #2093, which needed `--unresolved-symbols=ignore-in-object-files` to link a CPU-only test binary. No CI job could catch it: the `ubuntu-latest` jobs only lint, every GB10 job passes `--features cuda`, and the remaining `cargo check` step does not link.
+
+Reproduced on this branch by restoring main's `kv_inplace_write.cpp` and running `CARGO_TARGET_DIR=target/cpu cargo test -p mlxcel-core --profile test-fast --lib --no-run`: the link fails with two `undefined reference to mlx::core::copy_gpu_inplace(...)` errors from `InplaceSliceWrite::eval_gpu`, and no other unresolved symbol. `copy_gpu_inplace` is the only GPU-only MLX symbol the bridge references.
+
+## 2. Change Summary
+
+| Area | Change |
+|---|---|
+| `build_support/metal_backend.rs` | `gpu_backend_enabled(metal_backend, cuda, rocm)`, a pure predicate next to `resolve_metal_backend` |
+| `build.rs` | Defines `MLXCEL_BRIDGE_GPU_BACKEND` from that predicate, using the resolved `metal_backend` value (not the `metal` Cargo feature) and the `cuda` and `rocm` features, which `build_mlx` maps one-to-one onto `MLX_BUILD_CUDA` and `MLX_BUILD_ROCM` |
+| `kv_inplace_write.cpp` | `#include <mlx/backend/gpu/copy.h>` and the `eval_gpu` body under `#ifdef MLXCEL_BRIDGE_GPU_BACKEND`; otherwise `eval_gpu` throws `std::runtime_error` |
+| `build_support/metal_backend_tests.rs` | Three tests: CPU-only is false, each backend alone is true, macOS with `MLXCEL_BUILD_METAL=OFF` and no other backend is false |
+| `docs/installation.md` | "CPU-only link check" section with the local command and its measured cost |
+
+## 3. Design Decisions
+
+### 3.1 Why a new define instead of reusing the Metal and ROCm ones
+
+`MLXCEL_BRIDGE_METAL_BACKEND` and `MLXCEL_BRIDGE_ROCM_BACKEND` gate backend-specific headers. `copy_gpu_inplace` is a backend-neutral GPU helper present in all three GPU builds, so its condition is the union, and CUDA had no bridge define at all. The predicate keys Metal on the resolved value because macOS enables Metal without the Cargo feature and `MLXCEL_BUILD_METAL=OFF` disables it with the feature (#1988).
+
+### 3.2 Why keep compiling the file
+
+The issue rejected dropping the `.file()` line: `cpp/mlx_cxx_ext.cpp` includes the header and the cxx FFI symbol `inplace_slice_write` must still link. At runtime every Rust caller (`cache.rs` decode write, rotating warmup write, and `cache/paged.rs` slab write) already requires `ffi::default_device_is_gpu()`, and `inplace_slice_write` itself refuses a non-GPU default device, so the throwing `eval_gpu` is unreachable in a CPU-only build.
+
+### 3.3 GPU builds are byte-for-byte the same source
+
+The include stays at its original position, wrapped in the guard, and the original body is inside the `#else` branch. With the define set, the preprocessed translation unit equals main's, which is the argument for Metal and ROCm, neither of which can be built on the development host.
+
+## 4. Validation
+
+| Check | Result |
+|---|---|
+| CPU-only `cargo test -p mlxcel-core --profile test-fast --lib --no-run` (separate `target/cpu`) | Links. Cold 2m24s including the CPU-only MLX tree; relink after touching the bridge file 6.0s |
+| Same command with main's `kv_inplace_write.cpp` | Fails: `undefined reference to mlx::core::copy_gpu_inplace` (2 sites) |
+| CPU-only `cargo build --release` | Links, 9m48s cold |
+| CPU-only `metal_backend_build_policy` tests | 7 passed |
+| CPU-only `inplace` tests | 4 passed (the CPU path keeps `slice_update`) |
+| CPU-only `cargo clippy -p mlxcel-core --release --lib --tests -- -D warnings` | Clean |
+| CUDA validation | See section 5 |
+
+## 5. CUDA Regression
+
+| Check | Result |
+|---|---|
+| `cargo build --release --features cuda --bin mlxcel-server` | Builds and links (17m41s cold) |
+| CUDA `kv_inplace_write.o` | Still references `mlx::core::copy_gpu_inplace` and lacks the CPU-only error string, so the GPU body is compiled; the CPU-only object is the reverse |
+| `cargo test --release -p mlxcel-core --features cuda --lib`, filter `inplace`, under `gpu-lock`, `--test-threads=1` | 4 passed |
+| Same binary, filter `cache::` | 541 passed |
+| Same binary, filter `metal_backend_build_policy` | 7 passed |
+| `cargo clippy --release --features cuda --lib --tests -- -D warnings` | Clean |
+
+## 6. Not Validated
+
+Metal and ROCm could not be built on the GB10 host. Metal: `metal_backend` is true on a default macOS build, so the define is set and the source is main's. ROCm: `CARGO_FEATURE_ROCM` sets the define and `build_mlx` builds MLX with `MLX_BUILD_ROCM=ON`, whose GPU backend provides `copy_gpu_inplace`, so the source is again main's.
+
+## 7. CI
+
+No CI step was added here; `.github/workflows/ci.yml` is edited by #2111 in the same run. The proposed step is a GB10 job with its own persistent target dir (the pattern at `ci.yml:657-661`, for example `$HOME/.cargo-target/mlxcel-cpu-link-ci`) running `cargo test -p mlxcel-core --profile test-fast --lib --no-run`. Measured locally, a warm run costs 6.0s after a bridge C++ change and 6.6s after touching `mlxcel-core/src/lib.rs`, plus checkout; the first (cold) run costs about 2.5 minutes. That is cheap enough that the step is recommended for #2111. Until it lands, the command is documented in `docs/installation.md`.

TECHNICAL_REPORTS/2108-cpu-only-link-kv-inplace-write-20261005.ko.md

@@ -0,0 +1,74 @@
+# 기술 보고서: Issue #2108 - CPU 전용 Linux build에서 kv_inplace_write link 실패
+
+**날짜**: 2026-10-05
+
+**상태**: GB10에서 구현 및 검증 완료 (CPU 전용 build와 CUDA build). origin/main `33c45053` 기준, 머지 대기 중.
+
+**언어**: C++ (bridge preprocessor guard), Rust (`build.rs`, build-policy predicate와 unit test), Markdown (설치 문서)
+
+**위험도**: 낮음 (GPU translation unit은 미리 정의되는 macro 하나를 제외하면 바뀌지 않습니다. CPU 전용 build에는 어떤 코드 경로로도 도달할 수 없는, 예외를 던지는 `eval_gpu`가 생깁니다)
+
+## 요약
+
+`mlxcel-core/build.rs`는 모든 backend에서 `src/lib/mlx-cpp/turbo/kv_inplace_write.cpp` (#1959)를 compile합니다. 이 파일의 `InplaceSliceWrite::eval_gpu`는 `mlx::core::copy_gpu_inplace`를 호출하는데, MLX는 이 함수를 GPU backend가 있을 때만 build되는 `mlx/backend/gpu/copy.cpp`에서만 정의합니다. 그래서 `cuda`, `metal`, `rocm` feature가 모두 없는 Linux build는 link 단계에서 실패했습니다. 이번 수정은 MLX가 Metal, CUDA, ROCm 중 하나를 build할 때만 `build.rs`가 정의하는 `MLXCEL_BRIDGE_GPU_BACKEND`를 추가하고, `copy.h` include와 `eval_gpu` 본문을 그 뒤에 둡니다. 정의되지 않으면 `eval_gpu`는 `eval_cpu`와 마찬가지로 예외를 던집니다.
+
+## 1. 문제
+
+이 버그는 #2093을 검증하던 중 발견되었습니다. 당시 CPU 전용 test binary를 link하려면 `--unresolved-symbols=ignore-in-object-files`가 필요했습니다. CI는 이를 잡을 수 없었습니다. `ubuntu-latest` job은 lint만 하고, 모든 GB10 job은 `--features cuda`를 넘기며, 남은 `cargo check` 단계는 link하지 않습니다.
+
+이 branch에서 main의 `kv_inplace_write.cpp`를 되돌려 놓고 `CARGO_TARGET_DIR=target/cpu cargo test -p mlxcel-core --profile test-fast --lib --no-run`을 실행해 재현했습니다. `InplaceSliceWrite::eval_gpu`에서 나온 `undefined reference to mlx::core::copy_gpu_inplace(...)` 오류 두 개로 link가 실패했고, 다른 미해결 symbol은 없었습니다. bridge가 참조하는 GPU 전용 MLX symbol은 `copy_gpu_inplace` 하나뿐입니다.
+
+## 2. 변경 요약
+
+| 영역 | 변경 |
+|---|---|
+| `build_support/metal_backend.rs` | `resolve_metal_backend` 옆에 순수 predicate `gpu_backend_enabled(metal_backend, cuda, rocm)` 추가 |
+| `build.rs` | 그 predicate로 `MLXCEL_BRIDGE_GPU_BACKEND`를 정의. Metal은 Cargo feature가 아닌 resolve된 `metal_backend` 값을, CUDA와 ROCm은 `build_mlx`가 `MLX_BUILD_CUDA`, `MLX_BUILD_ROCM`에 그대로 대응시키는 feature를 사용 |
+| `kv_inplace_write.cpp` | `#include <mlx/backend/gpu/copy.h>`와 `eval_gpu` 본문을 `#ifdef MLXCEL_BRIDGE_GPU_BACKEND` 아래에 둠. 아니면 `eval_gpu`가 `std::runtime_error`를 던짐 |
+| `build_support/metal_backend_tests.rs` | test 세 개: CPU 전용은 false, backend 하나만 켜도 true, `MLXCEL_BUILD_METAL=OFF`이고 다른 backend가 없는 macOS는 false |
+| `docs/installation.md` | 로컬 명령과 측정 비용을 적은 "CPU-only link check" 절 추가 |
+
+## 3. 설계 결정
+
+### 3.1 Metal, ROCm define을 재사용하지 않고 새 define을 둔 이유
+
+`MLXCEL_BRIDGE_METAL_BACKEND`와 `MLXCEL_BRIDGE_ROCM_BACKEND`는 backend 전용 header를 막습니다. `copy_gpu_inplace`는 세 GPU build 모두에 있는 backend 중립 GPU helper이므로 조건은 합집합이어야 하고, CUDA에는 bridge define 자체가 없었습니다. macOS는 Cargo feature 없이도 Metal을 켜고 `MLXCEL_BUILD_METAL=OFF`는 feature가 있어도 끄므로 (#1988), predicate는 Metal을 resolve된 값으로 판단합니다.
+
+### 3.2 파일을 계속 compile하는 이유
+
+issue는 `.file()` 줄을 빼는 방법을 기각했습니다. `cpp/mlx_cxx_ext.cpp`가 header를 include하고 cxx FFI symbol `inplace_slice_write`는 계속 link되어야 하기 때문입니다. runtime에는 모든 Rust 호출 지점 (`cache.rs`의 decode write와 rotating warmup write, `cache/paged.rs`의 slab write)이 이미 `ffi::default_device_is_gpu()`를 요구하고, `inplace_slice_write` 자체도 GPU가 아닌 기본 device를 거부합니다. 따라서 CPU 전용 build에서 예외를 던지는 `eval_gpu`에는 도달할 수 없습니다.
+
+### 3.3 GPU build의 source는 그대로
+
+include는 원래 위치에 guard로 감싼 채 남았고, 원래 본문은 `#else` 쪽에 그대로 있습니다. define이 설정되면 preprocess된 translation unit은 main과 같습니다. 개발 호스트에서 build할 수 없는 Metal과 ROCm에 대한 근거가 이것입니다.
+
+## 4. 검증
+
+| 검사 | 결과 |
+|---|---|
+| CPU 전용 `cargo test -p mlxcel-core --profile test-fast --lib --no-run` (별도 `target/cpu`) | link 성공. CPU 전용 MLX tree 포함 cold 2분 24초, bridge 파일을 고친 뒤 다시 link하는 데 6.0초 |
+| main의 `kv_inplace_write.cpp`로 같은 명령 | 실패: `undefined reference to mlx::core::copy_gpu_inplace` (2곳) |
+| CPU 전용 `cargo build --release` | link 성공, cold 9분 48초 |
+| CPU 전용 `metal_backend_build_policy` test | 7개 통과 |
+| CPU 전용 `inplace` test | 4개 통과 (CPU 경로는 `slice_update`를 유지) |
+| CPU 전용 `cargo clippy -p mlxcel-core --release --lib --tests -- -D warnings` | 경고 없음 |
+| CUDA 검증 | 5절 참조 |
+
+## 5. CUDA 회귀 검사
+
+| 검사 | 결과 |
+|---|---|
+| `cargo build --release --features cuda --bin mlxcel-server` | build 및 link 성공 (cold 17분 41초) |
+| CUDA `kv_inplace_write.o` | 여전히 `mlx::core::copy_gpu_inplace`를 참조하고 CPU 전용 오류 문자열이 없으므로 GPU 본문이 compile됨. CPU 전용 object는 그 반대 |
+| `cargo test --release -p mlxcel-core --features cuda --lib`, filter `inplace`, `gpu-lock` 아래 `--test-threads=1` | 4개 통과 |
+| 같은 binary, filter `cache::` | 541개 통과 |
+| 같은 binary, filter `metal_backend_build_policy` | 7개 통과 |
+| `cargo clippy --release --features cuda --lib --tests -- -D warnings` | 경고 없음 |
+
+## 6. 검증하지 못한 부분
+
+GB10 호스트에서는 Metal과 ROCm을 build할 수 없었습니다. Metal: 기본 macOS build에서는 `metal_backend`가 true이므로 define이 설정되고 source는 main과 같습니다. ROCm: `CARGO_FEATURE_ROCM`이 define을 설정하고 `build_mlx`가 `MLX_BUILD_ROCM=ON`으로 MLX를 build하며, 그 GPU backend가 `copy_gpu_inplace`를 제공하므로 이 경우도 source는 main과 같습니다.
+
+## 7. CI
+
+이 PR은 CI 단계를 추가하지 않았습니다. 같은 실행에서 #2111이 `.github/workflows/ci.yml`을 수정합니다. 제안하는 단계는 자체 persistent target dir (`ci.yml:657-661`의 패턴, 예: `$HOME/.cargo-target/mlxcel-cpu-link-ci`)를 쓰는 GB10 job에서 `cargo test -p mlxcel-core --profile test-fast --lib --no-run`을 실행하는 것입니다. 로컬에서 측정한 warm 실행 비용은 bridge C++ 변경 뒤 6.0초, `mlxcel-core/src/lib.rs` 변경 뒤 6.6초에 checkout 시간을 더한 정도이고, 처음 (cold) 실행은 약 2.5분입니다. 비용이 충분히 작으므로 #2111에서 이 단계를 추가할 것을 권장합니다. 그 단계가 들어가기 전까지는 `docs/installation.md`에 명령을 문서화해 두었습니다.

docs/installation.md

@@ -743,6 +743,16 @@ covered is a defect that reproduces only under release LTO or `codegen-units = 1
 Reach for `cargo test --release --features metal,accelerate` by hand when you
 are chasing one of those.
 
+### CPU-only link check
+
+A Linux build with no GPU feature compiles MLX without any GPU backend, so MLX's GPU-side helpers (for example `copy_gpu_inplace`) do not exist in it. Bridge C++ under `src/lib/mlx-cpp/turbo/` and `src/lib/mlxcel-core/cpp/` that calls one must sit behind `MLXCEL_BRIDGE_GPU_BACKEND`, which `mlxcel-core/build.rs` defines exactly when MLX builds Metal, CUDA or ROCm. A missed guard is a link error that only this configuration shows: `cargo check` does not link, and every GB10 CI job passes `--features cuda`. Check it locally with a separate target directory so it does not evict a CUDA cache:
+
+```bash
+CARGO_TARGET_DIR=target/cpu cargo test -p mlxcel-core --profile test-fast --lib --no-run
+```
+
+Measured on GB10 (2026-10, lablup/mlxcel#2108): about 2.5 minutes cold, including the CPU-only MLX tree, and about 6 seconds to rebuild and relink after a one-file change in the bridge or in `mlxcel-core`. CI does not run this step yet.
+
 ## Why the gate says `--workspace`
 
 `make verify-clippy` and `make verify-test` pass `--workspace`, and dropping it

src/lib/mlx-cpp/turbo/kv_inplace_write.cpp

@@ -15,7 +15,11 @@
 #include "kv_inplace_write.h"
 
 #include <mlx/backend/common/slicing.h>
+// `copy_gpu_inplace` exists only in an MLX build with a GPU backend; build.rs
+// defines MLXCEL_BRIDGE_GPU_BACKEND exactly then (#2108).
+#ifdef MLXCEL_BRIDGE_GPU_BACKEND
 #include <mlx/backend/gpu/copy.h>
+#endif
 #include <mlx/primitives.h>
 #include <mlx/utils.h>
 
@@ -41,6 +45,12 @@ class InplaceSliceWrite : public mx::UnaryPrimitive {
   }
 
   void eval_gpu(const std::vector<mx::array>& inputs, mx::array& out) override {
+#ifndef MLXCEL_BRIDGE_GPU_BACKEND
+    // CPU-only build: there is no GPU stream to reach this, and
+    // `inplace_slice_write` already refuses a non-GPU default device.
+    throw std::runtime_error(
+        "[inplace_slice_write] this build has no GPU backend");
+#else
     const auto& dst = inputs[0];
     const auto& rows = inputs[1];
     // Adopt dst's buffer unconditionally: this is the whole point, and the
@@ -62,6 +72,7 @@ class InplaceSliceWrite : public mx::UnaryPrimitive {
         /* o_offset = */ data_offset,
         mx::CopyType::GeneralGeneral,
         stream());
+#endif
   }
 
   const char* name() const override {

src/lib/mlxcel-core/build.rs

@@ -181,6 +181,21 @@ fn main() {
         bridge.define("MLXCEL_BRIDGE_ROCM_BACKEND", None);
     }
 
+    // Any GPU backend (#2108). MLX defines its GPU-side helpers, such as
+    // `copy_gpu_inplace` in `mlx/backend/gpu/copy.cpp`, only when it builds a
+    // GPU backend, so a bridge file that calls them must not reference them in
+    // a CPU-only Linux build or the final link fails. Keyed on the resolved
+    // `metal_backend` (not the Cargo feature, see above) and on the CUDA and
+    // ROCm features, which are exactly the conditions under which `build_mlx`
+    // turns a GPU backend on.
+    if metal_backend::gpu_backend_enabled(
+        metal_backend,
+        env::var_os("CARGO_FEATURE_CUDA").is_some(),
+        env::var_os("CARGO_FEATURE_ROCM").is_some(),
+    ) {
+        bridge.define("MLXCEL_BRIDGE_GPU_BACKEND", None);
+    }
+
     // Add optimization flags for release builds
     #[cfg(not(debug_assertions))]
     {

src/lib/mlxcel-core/build_support/metal_backend.rs

@@ -32,6 +32,16 @@ pub fn resolve_metal_backend(is_macos: bool, override_value: Option<&str>) -> Re
     }
 }
 
+/// Used by: mlxcel-core's C++ bridge (`MLXCEL_BRIDGE_GPU_BACKEND`, #2108).
+/// Whether MLX builds any GPU backend, and with it the GPU-side helpers such as
+/// `copy_gpu_inplace` that a bridge file may only reference in that case.
+/// `metal_backend` is the resolved value above, not the Cargo feature; `cuda`
+/// and `rocm` are the Cargo features, which `build_mlx` maps one-to-one onto
+/// `MLX_BUILD_CUDA` and `MLX_BUILD_ROCM`.
+pub fn gpu_backend_enabled(metal_backend: bool, cuda: bool, rocm: bool) -> bool {
+    metal_backend || cuda || rocm
+}
+
 #[cfg(test)]
 #[path = "metal_backend_tests.rs"]
 mod tests;

src/lib/mlxcel-core/build_support/metal_backend_tests.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use super::resolve_metal_backend;
+use super::{gpu_backend_enabled, resolve_metal_backend};
 
 #[test]
 fn macos_default_enables_metal_without_a_cargo_feature() {
@@ -42,3 +42,23 @@ fn invalid_macos_override_retains_the_named_build_error() {
     assert!(error.contains(r#"Invalid MLXCEL_BUILD_METAL value "maybe""#));
     assert!(error.contains("1/0, on/off, true/false, yes/no"));
 }
+
+#[test]
+fn cpu_only_build_has_no_gpu_backend() {
+    // A Linux build with no GPU feature: the bridge must not reference
+    // `copy_gpu_inplace` (#2108).
+    assert!(!gpu_backend_enabled(false, false, false));
+}
+
+#[test]
+fn any_single_gpu_backend_enables_the_gpu_bridge_paths() {
+    assert!(gpu_backend_enabled(true, false, false));
+    assert!(gpu_backend_enabled(false, true, false));
+    assert!(gpu_backend_enabled(false, false, true));
+}
+
+#[test]
+fn macos_with_metal_disabled_and_no_other_backend_is_cpu_only() {
+    let metal = resolve_metal_backend(true, Some("OFF")).unwrap();
+    assert!(!gpu_backend_enabled(metal, false, false));
+}