diff --git a/README.md b/README.md index 8211434..4067d72 100644 --- a/README.md +++ b/README.md @@ -66,6 +66,33 @@ An [independent full-vocabulary measurement](https://github.com/malaiwah/quant-f The measurement used 25 windows, the full 154,880-token vocabulary, teacher forcing, FP64 accumulation, and two cold runs with identical results. The checkpoint was quantized from the official FP8 release; the measurement reference is BF16. Machine-readable summary: [`benchmarks/quality/quantization-analysis.json`](benchmarks/quality/quantization-analysis.json). +## Single-Spark memory budget, prefix-cache fix, and runtime knobs (2026-09-06) + +Findings from running this recipe as a daily driver on one DGX Spark (details, numbers and repro scripts: +[issue #3](https://github.com/gitcommit90/glm-5.3-one-spark/issues/3), upstream bug +[vllm-project/vllm#55600](https://github.com/vllm-project/vllm/issues/55600)): + +- **`--gpu-memory-utilization 0.90` leaves the host ~6 GB.** On GB10 the GPU pool is the OS RAM; the first + real request allocates ~4–7 GB outside the profiler budget (JIT, allocator growth) and the NVIDIA driver + starts failing page-table allocations (`NV_ERR_NO_MEMORY` → `Xid 31` or a hung engine with a thrashing host). + `--memory` on the container does not help (GPU allocations are not charged to the cgroup). +- **Where the KV goes at 262k:** context is 37 blocks (1.9 GiB); the DFlash2 drafter's padded 64-token blocks + reserve 257 blocks (13 GiB) with async scheduling. `ONE_SPARK_ASYNC=0` + `ONE_SPARK_DRAFT_BLOCK=1024` + + `GLM53_INDEXER_WORKSPACE=rightsize` bring the single-request need from 16.3 GiB to 3.9 GiB, so + `ONE_SPARK_UTIL=0.80` serves 262144 (2.6×) or 524288 (1.7×) with ~20 GB host headroom, decode/prefill unchanged. +- **Prefix-cache hits of ≥ 8 mamba blocks crashed the engine** (upstream vLLM seed bug, see above); + `ONE_SPARK_MAMBA_SEED_FIX=1` (default) patches it at container start. With it, warm requests at 100k–450k + return the same answers as cold, TTFT 8.7 s vs 139 s at 100k. + +Example of the measured-safe single-Spark configuration: + +```bash +ONE_SPARK_UTIL=0.80 ONE_SPARK_CTX=524288 ONE_SPARK_SEQS=1 ONE_SPARK_ASYNC=0 ONE_SPARK_DRAFT_BLOCK=1024 \ +GLM53_INDEXER_WORKSPACE=rightsize ./start.sh +``` + +All knobs default to the shipped behaviour except the seed fix, which is on by default. + ## What this project contributes The two-Spark work by [Mia's AI Lab](https://github.com/MiaAI-Lab/GLM-5.3-Flash-EXL3-2x-DGX-Sparks) established the vLLM/EXL3/DFlash foundation. This project adapts and extends that foundation for a very different target: diff --git a/scripts/serve-one-spark.sh b/scripts/serve-one-spark.sh index 46323e6..1ab2f59 100755 --- a/scripts/serve-one-spark.sh +++ b/scripts/serve-one-spark.sh @@ -5,17 +5,52 @@ set -euo pipefail python3 /opt/glm53/patch_glm_video_placeholders.py K="${ONE_SPARK_K:-5}" # DFlash2 draft depth; 5 = best prose/code, 8 = best structured (see README K sweep) SPEC='{"method":"dflash","model":"/draft","num_speculative_tokens":'"$K"',"kv_cache_dtype":"auto","draft_sample_method":"probabilistic","rejection_sample_method":"standard","draft_tensor_parallel_size":1}' +# ---- Runtime knobs (all default to the shipped recipe behaviour) ---- +# ONE_SPARK_CTX / ONE_SPARK_UTIL / ONE_SPARK_SEQS / ONE_SPARK_MNBT: context, gpu-memory-utilization, +# max-num-seqs, max-num-batched-tokens. +# ONE_SPARK_ASYNC=0|1: force --no-async-scheduling / --async-scheduling (unset = vLLM default, async for DFlash). +# With async on, the DFlash2 drafter reserves a 2047+2*MNBT token in-flight window; each 64-token drafter +# block is padded to a full 7168-token MLA page, so that window costs 257 blocks (13 GiB) for 262k context. +# ONE_SPARK_ASYNC=0 halves it; bs=1 decode is unchanged. +# ONE_SPARK_APC=0|1: --no-enable-prefix-caching / --enable-prefix-caching (default 1). +# ONE_SPARK_DRAFT_BLOCK=N (e.g. 1024): raise the drafter's compact block in the padded slot-share path +# (kv_cache_utils.py) so the drafter reserves ~10 blocks instead of 145/257. FlashAttention reports +# MultipleOf(16) and select_common_block_size returns the manager block, so kernel block == manager block. +# Measured lossless (acceptance 3.3-3.6, decode unchanged). Unset = shipped 64. +# ONE_SPARK_MAMBA_SEED_FIX=1 (default): fix for vllm-project/vllm#55600 — add_request seeds the mamba state +# index with cache_config.block_size, which EngineCore lowers to the drafter's block (64/1024) while mamba +# state lives in 7168-token blocks; prefix hits of >= 8 blocks then read past the block-table row (Xid 31) +# and shorter hits silently restore the wrong KDA state. Fail-closed: if the anchor is missing the container +# refuses to start (set ONE_SPARK_MAMBA_SEED_FIX=0 to start anyway; then also set ONE_SPARK_APC=0). +case "${ONE_SPARK_ASYNC:-}" in 0) ASYNC_FLAG=--no-async-scheduling ;; 1) ASYNC_FLAG=--async-scheduling ;; *) ASYNC_FLAG= ;; esac +case "${ONE_SPARK_APC:-1}" in 0) APC_FLAG=--no-enable-prefix-caching ;; *) APC_FLAG=--enable-prefix-caching ;; esac +if [ -n "${ONE_SPARK_DRAFT_BLOCK:-}" ]; then + KVU=/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_utils.py + sed -i "s/compact_block = 64$/compact_block = ${ONE_SPARK_DRAFT_BLOCK}/; s/s.block_size != 64 or s.page_size_padded != mla_page/s.block_size != ${ONE_SPARK_DRAFT_BLOCK} or s.page_size_padded != mla_page/" "$KVU" + echo "[one-spark] drafter block patch: $(grep -c "compact_block = ${ONE_SPARK_DRAFT_BLOCK}" "$KVU") + $(grep -c "block_size != ${ONE_SPARK_DRAFT_BLOCK}" "$KVU") sites" +fi +if [ "${ONE_SPARK_MAMBA_SEED_FIX:-1}" = "1" ]; then + MH=/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu/model_states/mamba_hybrid.py + sed -i "s|(new_req_data.num_computed_tokens - 1) // self.cache_config.block_size|(new_req_data.num_computed_tokens - 1) // self.cache_config.mamba_block_size|" "$MH" + N=$(grep -c "num_computed_tokens - 1) // self.cache_config.mamba_block_size" "$MH") + echo "[one-spark] mamba seed fix (vllm#55600): $N site" + if [ "$N" != "1" ]; then + echo "[one-spark] FATAL: mamba seed fix not applied (expected 1 match, got $N) - did the image change? Without it a prefix-cache hit of >= 8 blocks faults (Xid 31). Set ONE_SPARK_MAMBA_SEED_FIX=0 ONE_SPARK_APC=0 to start deliberately." >&2 + exit 97 + fi +fi exec vllm serve /model \ --served-model-name GLM-5.3-Flash-EXL3-2.05 \ --host "${ONE_SPARK_HOST:-127.0.0.1}" --port "${ONE_SPARK_PORT:-18080}" \ --tensor-parallel-size 1 \ --tool-call-parser glm47 --enable-auto-tool-choice \ --reasoning-parser glm45 \ - --enable-prefix-caching --no-enable-flashinfer-autotune \ + $APC_FLAG --no-enable-flashinfer-autotune \ --quantization exl3 \ - --max-model-len 262144 \ - --gpu-memory-utilization 0.90 \ - --max-num-seqs 4 --max-num-batched-tokens 7168 \ + --max-model-len "${ONE_SPARK_CTX:-262144}" \ + --gpu-memory-utilization "${ONE_SPARK_UTIL:-0.90}" \ + --max-num-seqs "${ONE_SPARK_SEQS:-4}" --max-num-batched-tokens "${ONE_SPARK_MNBT:-7168}" \ + $ASYNC_FLAG \ --kv-cache-dtype fp8 \ --speculative-config "$SPEC" \ --chat-template /opt/glm53/chat_template.jinja \ diff --git a/start.sh b/start.sh index 18ef744..852f66a 100755 --- a/start.sh +++ b/start.sh @@ -41,10 +41,13 @@ docker run -d --name "$CONTAINER" --gpus all --network host --ipc=host \ -e EXL3_FAT_KERNEL=1 \ -e GLM53_SUPPRESS_STOPS_IN_REASONING=1 \ -e GLM53_MIXED_PREFILL_CHUNK=skip \ - -e GLM53_INDEXER_WORKSPACE=stock \ + -e GLM53_INDEXER_WORKSPACE="${GLM53_INDEXER_WORKSPACE:-stock}" \ -e GLM53_SPINWAIT_MS=stock \ -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \ -e ONE_SPARK_HOST="$HOST" -e ONE_SPARK_PORT="$PORT" -e ONE_SPARK_K="${ONE_SPARK_K:-5}" \ + -e ONE_SPARK_CTX="${ONE_SPARK_CTX:-262144}" -e ONE_SPARK_UTIL="${ONE_SPARK_UTIL:-0.90}" -e ONE_SPARK_SEQS="${ONE_SPARK_SEQS:-4}" \ + -e ONE_SPARK_MNBT="${ONE_SPARK_MNBT:-7168}" -e ONE_SPARK_ASYNC="${ONE_SPARK_ASYNC:-}" -e ONE_SPARK_APC="${ONE_SPARK_APC:-1}" \ + -e ONE_SPARK_DRAFT_BLOCK="${ONE_SPARK_DRAFT_BLOCK:-}" -e ONE_SPARK_MAMBA_SEED_FIX="${ONE_SPARK_MAMBA_SEED_FIX:-1}" \ -v "$MODEL_DIR:/model:ro" -v "$DFLASH_DIR:/draft:ro" \ -v "$ROOT/scripts/serve-one-spark.sh:/start.sh:ro" \ -v "${CACHE_ROOT:-$HOME/.cache/glm53-one-spark}/vllm:/root/.cache/vllm" \