[bong-water-water-bong]

1bit serve --device hybrid: the prompt's KV cache is computed by an external prefill provider (the NPU engine), llama-server restores it into its one slot and decodes on HRX (the CPU in a build without HRX, so the route runs anywhere). Public plumbing only: the provider is a program with a stated contract, the way -DONEBIT_NPU_PRIVATE keeps the NPU kernels out of this tree. Stacked on #529 (no ONNX left).

Per request (app/serve_hybrid.h, docs/hybrid.md): template + tokenize on the backend (/apply-template, /tokenize, so the provider sees the backend's own ids) -> provider writes a llama.cpp sequence-state file into the backend's --slot-save-path -> POST /slots/0?action=restore, checked to hold all the prompt's tokens -> the request forwarded with id_slot 0 and cache_prompt true; llama-server re-evaluates the last token and generates. One line per request on stderr: prompt tokens, prefill ms, restore ms, first byte ms. One slot (-np 1); handoffs serialized until the backend has started on the slot. Any failure (no provider, provider exit/timeout/no file, restore short) is said with its reason and the request is forwarded as it came: the device is never unusable because the provider is. Refused with hybrid: --mtp, --dflash, --mmproj, --moe-slots, --parallel > 1, roles, --pm; the GGUF's own MTP head is off. --print-backend exits 5 (needs serve).

Provider contract: <provider> --tokens <ids file, one decimal id per line> --out <state file, bare name in the slot dir> --model <path> [--elfs DIR]; stdout one JSON line {"n_tokens": N, "argmax": id, "prefill_ms": x}; exit 0. Named by --prefill-provider or ONEBIT_HYBRID_PREFILL; ONEBIT_HYBRID_ELFS adds --elfs; the provider also gets ONEBIT_HYBRID_BACKEND_URL (the decoding server's address). State file = what llama_state_seq_save_file writes (ids, then seq 0's cells: per layer K type/row size/rows, then V; v_trans 0 with flash attention; the backend's cache type, f16 by default), matching the pinned llama.cpp's version.

Tests

• tests/hybrid_route.sh (CTest hybrid_route, no model): fake backend + fake provider (FAKE_PREFILL_COPY=1); checks id_slot 0 / cache_prompt reach the backend, the stderr lines, the slot directory lifecycle, the no-provider and failing-provider fallbacks, the refused flags, --print-backend exit 5. 21 checks pass.
• tests/hybrid_e2e.sh (CTest hybrid_e2e with ONEBIT_SERVE_TEST_GGUF, skips without a model or llama-server): Qwen3-0.6B-Q4_K_M on the CPU llama-server built from the pinned fork, on ryzen. The fake provider makes a real state file by having the decoding server prefill the ids into slot 0, save, and erase the slot, so the file is the only path for the KV. Result: the hybrid greedy reply equals the plain --device cpu reply ("Three colours of the rainbow are red, orange, and yellow."), the backend's prompt_n is 1 (the re-evaluated last token) against 22 for the plain run, no fallback line, streaming works, a second request answers the same.

Not in this PR: the NPU prefill provider itself (private; the A1 proof on strixhalo writes the state file from the NPU engine's KV).

🤖 Generated with Claude Code

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 882a77b)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 529 - Partially compliant Compliant requirements: Remove --device onnx in serve Remove the flag that fails with a removed-device message that names Lemonade's backends Remove the onnx/ directory of the Linux package Remove ONEBIT_ONNX, the onnx_npu_smoke test, scripts/build-onnx.sh, the SIGTERM shim, the third_party/ryzenai-server submodule Remove the ryzenai-server pin in the weekly changes, and the ryzenai-server / ONNX Runtime entries in NOTICE Remove docs/onnx.md Checked on ryzen: configure + build of onebit, smoke_serve, watchdog_serve and metrics_serve pass 1bit serve --device onnx prints the removed-device message --help mentions no ONNX Non-compliant requirements: Untouched: Lemonade's own onnxruntime:cpu backend in the 1bit OS image, and the ONNX-export vocabulary of the model census Requires further human verification: None

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Race Condition in Slot Management The hybrid device implementation uses a mutex (slot_mu_) to serialize handoffs, but there's a potential race condition in how the lock is managed. Specifically, in app/serve_hybrid.h, the prepare method creates a std::unique_lock on the mutex, but the lock is only released in the lambda passed to forward in app/serve.cpp. If the lambda is not executed (e.g., due to an early return or exception), the lock may not be released, leading to a deadlock. This could cause the hybrid device to become unresponsive under certain failure conditions. for (int32_t id : ids) f << id << '\n'; if (!f) { o.why = "cannot write " + tokens.string(); return o; } } Incorrect Error Handling in Hybrid Prefill In app/serve.cpp, the hybrid prefill logic (lines 2018-2052) does not properly handle cases where the prefill provider fails to produce a valid state file. The code checks for the existence of the state file, but does not validate its contents or structure. If the provider writes a malformed or incomplete state file, the restore operation could fail silently or cause undefined behavior. This could lead to incorrect responses or crashes. if (backends[pick].hybrid && hybrid_prefill) { // the handoff (app/serve_hybrid.h): template + tokenize on the backend, the provider's prefill, // restore into slot 0, then the request pinned to that slot. One slot: one request at a time, // held through its decode so the next handoff cannot replace a KV still in use. json body; try { body = json::parse(q.body); } catch (const std::exception&) { r.status = 400; r.set_content(R"({"error":{"message":"request body is not JSON"}})", "application/json"); return; } // The lock is held until the backend has started on the slot (its first byte): from then // on llama-server itself defers a restore into a busy slot. httplib runs the handler and // the streamed reply on one thread, so the unlock is on the locking thread. auto slot = std::make_shared<std::unique_lock<std::mutex>>(hybrid_prefill->slot_mutex()); const auto t0 = std::chrono::steady_clock::now(); const hybrid::Outcome h = hybrid_prefill->prepare(backends[pick].port, q.path, body); if (!h.handed_over) { std::fprintf(stderr, "1bit serve: hybrid: %s; the backend prefills this request itself\n", h.why.c_str()); slot->unlock(); forward(backends[pick].port, held, -1, id, drop_model, set_model, o.precise, q, r, &watch); return; } const size_t n = h.n_tokens; const double prefill_ms = h.prefill_ms, restore_ms = h.restore_ms; forward(backends[pick].port, held, -1, id, drop_model, set_model, o.precise, q, r, &watch, &body, [t0, n, prefill_ms, restore_ms, slot] { const double ttft = std::chrono::duration<double, std::milli>(std::chrono::steady_clock::now() - t0).count(); std::fprintf(stderr, "1bit serve: hybrid: %zu prompt tokens: prefill %.0f ms, restore %.0f ms, first byte %.0f ms\n", n, prefill_ms, restore_ms, ttft); if (slot->owns_lock()) slot->unlock(); }); return; } Potential Memory Leak in Slot Directory Cleanup In app/serve.cpp, the SlotDirGuard class (lines 1764-1767) is used to clean up the slot directory when the hybrid device is used. However, if an exception is thrown during the initialization of the hybrid prefill or during the setup of the slot directory, the destructor of SlotDirGuard may not be called, leading to a potential memory leak or leftover temporary directories. This could cause issues on long-running systems or under high load. struct SlotDirGuard { std::string dir; ~SlotDirGuard() { if (!dir.empty()) { std::error_code ec; fs::remove_all(dir, ec); } } } slot_dir_guard{backends[0].hybrid ? backends[0].slot_dir : ""};

[github-actions]

Persistent review updated to latest commit fe391df

[github-actions]

Persistent review updated to latest commit fe391df

[github-actions]

Persistent review updated to latest commit 999d885

[github-actions]

Persistent review updated to latest commit 102cf06

[github-actions]

Persistent review updated to latest commit 882a77b