feat(ds4): add default-off HIP GPU phase profiler

server/CMakeLists.txt

@@ -254,6 +254,7 @@ add_library(dflash_common STATIC
     # DeepSeek V4 Flash target arch
     src/deepseek4/deepseek4_loader.cpp
     src/deepseek4/deepseek4_graph.cpp
+    src/deepseek4/deepseek4_gpu_profiler.cpp
     src/deepseek4/deepseek4_backend.cpp
     src/deepseek4/deepseek4_daemon.cpp
     src/deepseek4/deepseek4_layer_split_adapter.cpp
@@ -388,7 +389,7 @@ endif()
 if(DFLASH27B_GPU_BACKEND STREQUAL "cuda")
     target_link_libraries(dflash_common PUBLIC CUDA::cudart)
 elseif(DFLASH27B_GPU_BACKEND STREQUAL "hip")
-    target_link_libraries(dflash_common PUBLIC hip::host)
+    target_link_libraries(dflash_common PUBLIC hip::host ${CMAKE_DL_LIBS})
 endif()
 
 # FlashPrefill custom kernels.

server/docs/DS4.md

@@ -129,13 +129,27 @@ The runtime logs the chosen split with a `[deepseek4-split] auto-split:` banner.
 | Variable | Purpose |
 |----------|---------|
 | `DFLASH_DS4_CUDA_LAYERS` | Override the auto-split heuristic and pin the first `N` DeepSeek4 layers to CUDA. The remaining `43 - N` layers run on the Halo shard. |
+| `DFLASH_DS4_GPU_PROFILE` | Enable HIP-event phase records and ROCTX ranges for DS4 forward and verification work. Default off; `0` is disabled. |
 | `DFLASH_DS4_TIMING` | Enable DS4 timing logs for the layer-split parent and target-shard daemon. Useful for profiling prefill/decode breakdowns; leave unset for normal runs. |
 
 `DFLASH_DS4_TIMING` enables the existing timing banners:
 
 - parent / local shard: `[deepseek4-split-timing]`
 - remote Halo shard: `[deepseek4-target-timing]`
 
+`DFLASH_DS4_GPU_PROFILE=1` emits stable records with the prefix
+`[ds4-gpu-profile]`, `clock=hip_event`, a scope and mode, a phase name, token
+width and position, GPU elapsed milliseconds, and the number of timed calls.
+Records from `deepseek4_step_layer_range` also carry the `layers=<begin>-<end>`
+range so shards are distinguishable in a layer-split run.
+Core exact-verification phases are `hc_pre`, `attention`, `moe_ffn`, `hc_post`,
+and `output_projection`; phases with no HIP work report zero calls. Exact
+verification also emits `verification_step`. Fused decode and the explicit
+approximate fused-verification research path emit whole-graph records because
+their phase boundaries are inside one captured graph. When `libroctx64` is
+available, the same phase names are pushed as ROCTX ranges; builds and runs do
+not require the library.
+
 DeepSeek4 no longer uses the old expert-split environment variables or expert-worker tuning knobs. Those retired knobs were removed from the codebase rather than left behind as unsupported debug switches.
 
 ## DSpark Speculative Decode
@@ -183,13 +197,13 @@ export DFLASH_DS4_SPEC_Q=4
   --ds4-expert-top-k 4
 ```
 
-`DFLASH_DS4_FUSED_VERIFY=1` is the opt-in throughput profile. Its persistent
-whole-model GPU graph uses stable padded reduction shapes, so near-tied greedy
-logits can select a different token than the normal causal verifier even at
-temperature 0. Leave it unset when comparing against the normal verifier, or
-set `DFLASH_DS4_SEQ_VERIFY=1` for the slower token-at-a-time verification
-diagnostic. Neither fused verification nor the separate
-`--ds4-expert-top-k 4` approximation should be presented as byte-identical AR.
+`DFLASH_DS4_FUSED_VERIFY=1` requests the fused throughput profile, but falls
+back to the normal verifier because the persistent whole-model GPU graph can
+select a different greedy token at near-tied logits. The approximate graph is
+available only for explicit research runs by also setting
+`DFLASH_DS4_ALLOW_APPROX_FUSED_VERIFY=1`; it must not be presented as
+byte-identical. `DFLASH_DS4_SEQ_VERIFY=1` remains the slower token-at-a-time
+diagnostic. The separate `--ds4-expert-top-k 4` policy is also approximate.
 
 DSpark currently requires monolithic target placement. On HIP,
 `--ds4-fused-decode` selects that placement; if the target falls back to hybrid
@@ -216,6 +230,8 @@ confidence of the proposed prefix. It adds the projection to the same fused
 Markov graph and reads its scores in the existing token-id synchronization; no
 additional host round trip is introduced. Artifacts without a compatible
 confidence head transparently retain the existing acceptance-EWMA policy.
+Set `DFLASH_DS4_ADAPTIVE_WIDTH=0` to hold the verify width fixed for parity
+tests; the environment value takes precedence over the legacy `/tmp` control.
 
 On the gfx1151 validation host, confidence-adaptive width retained 10/10
 GSM+Math accuracy and measured 31.94 tok/s weighted, within 0.6% of fixed q=4

server/docs/ENVIRONMENT.md

@@ -25,11 +25,15 @@ consolidation of this list into CLI flags is tracked as follow-up work.
 | `DFLASH_ADAPTIVE_K_DENSE` | per-model default | CSV of MoE layers kept dense under adaptive-K (DFlash capture layers). Warned-inert on families that do not thread layer indices yet. |
 | `DFLASH_MMID_GROUPED` | unset | Grouped MUL_MAT_ID kernel for small verify batches; candidate for CLI promotion. |
 | `DFLASH_MMID_TELEMETRY` | unset | DEBUG: report MUL_MAT_ID dispatch, MMVQ variant, and per-node graph compatibility. |
+| `DFLASH_DS4_GPU_PROFILE` | unset | DEBUG: HIP-event DS4 phase timings and optional ROCTX ranges. `0` is disabled. |
+| `DFLASH_DS4_PARITY_TRACE` | unset | DEBUG: generated IDs plus per-row top-two logits for verified tokens. Scans the full vocabulary per row and prints one stderr line per row, so keep it to short debug runs. |
 | `DFLASH_KVFLASH` | unset | Prefer the CLI: `--kvflash` (token count or `auto`). |
 
 ## Full inventory (generated)
 
 `grep -rE 'getenv\("[A-Z0-9_]+"\)' server/src` - regenerate when adding or removing variables.
+Variables read through wrapper helpers (for example `ds4_env_flag`, `spec_env_flag`,
+`env_flag_enabled`) do not match that grep; their inventory entries are curated manually.
 
 - `DFLASH27B_CHUNKED` - qwen35_target_graph.cpp
 - `DFLASH27B_DRAFT_FP16` - draft_safetensors_loader.cpp
@@ -57,6 +61,8 @@ consolidation of this list into CLI flags is tracked as follow-up work.
 - `DFLASH_DRAFT_KV` - laguna_backend.cpp, qwen35_backend.cpp
 - `DFLASH_DRAFT_PERSIST` - laguna_backend.cpp
 - `DFLASH_DROP_COLD` - qwen35moe_backend.cpp, qwen35moe_pipelined_decode.cpp
+- `DFLASH_DS4_GPU_PROFILE` - deepseek4_graph.cpp, deepseek4_dspark_spec.cpp
+- `DFLASH_DS4_PARITY_TRACE` - deepseek4_backend.cpp, deepseek4_dspark_spec.cpp
 - `DFLASH_DS4_TIMING` - deepseek4_target_shard_ipc_daemon.cpp
 - `DFLASH_EXPERT_BUDGET_MB` - deepseek4_backend.cpp, laguna_backend.cpp, qwen35moe_backend.cpp
 - `DFLASH_EXPERT_BUDGET_PCT` - laguna_backend.cpp

server/scripts/test_ds4_fused_verify_parity.py

@@ -0,0 +1,137 @@
+#!/usr/bin/env python3
+"""Model-backed token parity regression for the public DS4 fused-verify flag."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import re
+import subprocess
+import tempfile
+import time
+import urllib.request
+from pathlib import Path
+
+
+TOKEN_RE = re.compile(r"\[ds4-parity-tokens\] n=\d+ ids=\[([0-9 ]*)\]")
+
+
+def wait_ready(port: int, proc: subprocess.Popen[bytes], timeout: float) -> None:
+    deadline = time.monotonic() + timeout
+    url = f"http://127.0.0.1:{port}/health"
+    while time.monotonic() < deadline:
+        if proc.poll() is not None:
+            raise RuntimeError(f"server exited before readiness: {proc.returncode}")
+        try:
+            with urllib.request.urlopen(url, timeout=2) as response:
+                if response.status == 200:
+                    return
+        except OSError:
+            time.sleep(1)
+    raise TimeoutError(f"server did not become ready within {timeout:.0f}s")
+
+
+def stop_server(proc: subprocess.Popen[bytes]) -> None:
+    if proc.poll() is not None:
+        return
+    proc.terminate()
+    try:
+        proc.wait(timeout=30)
+    except subprocess.TimeoutExpired:
+        proc.kill()
+        proc.wait(timeout=10)
+
+
+def run_case(args: argparse.Namespace, fused_requested: bool, log_path: Path) -> list[int]:
+    env = os.environ.copy()
+    for name in (
+        "DFLASH_DS4_FUSED_VERIFY",
+        "DFLASH_DS4_ALLOW_APPROX_FUSED_VERIFY",
+        "DFLASH_DS4_SEQ_VERIFY",
+        "DFLASH_DS4_DSPARK_DEBUG",
+    ):
+        env.pop(name, None)
+    env.update({
+        "DFLASH_DS4_SPEC": "1",
+        "DFLASH_DS4_DRAFT": str(args.draft),
+        "DFLASH_DS4_SPEC_Q": "4",
+        "DFLASH_DS4_ADAPTIVE_WIDTH": "0",
+        "DFLASH_DS4_PARITY_TRACE": "1",
+        "LUCE_MMVQ_MAX_NCOLS": "4",
+    })
+    if fused_requested:
+        env["DFLASH_DS4_FUSED_VERIFY"] = "1"
+
+    cmd = [
+        str(args.server_bin), str(args.target),
+        "--host", "127.0.0.1", "--port", str(args.port),
+        "--max-ctx", str(args.max_ctx), "--chunk", "512",
+        "--target-device", "hip:0", "--ds4-fused-decode",
+    ]
+    with log_path.open("wb") as log:
+        proc = subprocess.Popen(cmd, env=env, stdout=log, stderr=subprocess.STDOUT)
+        try:
+            wait_ready(args.port, proc, args.startup_timeout)
+            body = json.dumps({
+                "model": "dflash",
+                "messages": [{"role": "user", "content": args.prompt}],
+                "temperature": 0,
+                "seed": 1234,
+                "max_tokens": args.max_tokens,
+                "stream": False,
+            }).encode()
+            request = urllib.request.Request(
+                f"http://127.0.0.1:{args.port}/v1/chat/completions",
+                data=body,
+                headers={"Content-Type": "application/json"},
+            )
+            with urllib.request.urlopen(request, timeout=args.request_timeout) as response:
+                if response.status != 200:
+                    raise RuntimeError(f"generation returned HTTP {response.status}")
+                json.load(response)
+        finally:
+            stop_server(proc)
+
+    matches = TOKEN_RE.findall(log_path.read_text(errors="replace"))
+    if not matches:
+        raise RuntimeError(f"token trace missing from {log_path}")
+    return [int(token) for token in matches[-1].split()]
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--server-bin", required=True, type=Path)
+    parser.add_argument("--target", required=True, type=Path)
+    parser.add_argument("--draft", required=True, type=Path)
+    parser.add_argument("--port", type=int, default=18084)
+    parser.add_argument("--max-ctx", type=int, default=4096)
+    parser.add_argument("--max-tokens", type=int, default=32)
+    parser.add_argument("--startup-timeout", type=float, default=180)
+    parser.add_argument("--request-timeout", type=float, default=600)
+    parser.add_argument(
+        "--prompt",
+        default="Explain why a bicycle stays upright while moving.",
+    )
+    args = parser.parse_args()
+    for path in (args.server_bin, args.target, args.draft):
+        if not path.is_file():
+            parser.error(f"file not found: {path}")
+
+    with tempfile.TemporaryDirectory(prefix="ds4-fused-parity-") as tmp:
+        tmp_path = Path(tmp)
+        normal = run_case(args, False, tmp_path / "normal.log")
+        fused_flag = run_case(args, True, tmp_path / "fused-flag.log")
+
+    if normal != fused_flag:
+        limit = min(len(normal), len(fused_flag))
+        first = next((i for i in range(limit) if normal[i] != fused_flag[i]), limit)
+        print(f"FAIL: first token mismatch at {first}: normal={normal[first:first+4]} "
+              f"fused_flag={fused_flag[first:first+4]}")
+        return 1
+    print(f"PASS: {len(normal)} generated token IDs are identical")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

server/src/deepseek4/deepseek4_backend.cpp

@@ -1040,6 +1040,13 @@ GenerateResult DeepSeek4Backend::generate_impl(const GenerateRequest & req,
         }
         result.succeed();
         result.tokens = std::move(gen);
+        if (env_flag_enabled("DFLASH_DS4_PARITY_TRACE")) {
+            std::fprintf(stderr, "[ds4-parity-tokens] n=%zu ids=[", result.tokens.size());
+            for (size_t i = 0; i < result.tokens.size(); ++i) {
+                std::fprintf(stderr, "%s%d", i == 0 ? "" : " ", result.tokens[i]);
+            }
+            std::fprintf(stderr, "]\n");
+        }
         result.decode_s = elapsed_s(t1);
         result.accept_rate = accept_rate;
         result.spec_decode_ran = spec_ran;

server/src/deepseek4/deepseek4_dspark_spec.cpp

@@ -22,6 +22,7 @@
 // DFLASH_DS4_FULL_SNAP=1 for A/B validation.
 
 #include "deepseek4_dspark.h"
+#include "deepseek4_gpu_profiler.h"
 #include "deepseek4_internal.h"
 #include "internal.h"
 #include "common/dspark_head.h"
@@ -39,6 +40,10 @@
 
 namespace dflash::common {
 
+namespace {
+bool spec_env_flag(const char * name);
+}
+
 // ── DFlashTarget adapter over the DS4 target ────────────────────────────
 class DeepSeek4DFlashTarget : public DFlashTarget {
 public:
@@ -71,7 +76,20 @@ class DeepSeek4DFlashTarget : public DFlashTarget {
             const char * v = std::getenv("DFLASH_DS4_SEQ_VERIFY");
             return v && *v && *v != '0';
         }();
+        const bool approximate_fused =
+            spec_env_flag("DFLASH_DS4_FUSED_VERIFY") &&
+            spec_env_flag("DFLASH_DS4_ALLOW_APPROX_FUSED_VERIFY");
+        const char * profile_mode = seq_verify ? "exact_sequential" :
+            (approximate_fused ? "approx_fused" : "exact");
+        Ds4GpuProfileOptions profile_opts;
+        profile_opts.scope = "verification";
+        profile_opts.mode = profile_mode;
+        profile_opts.n_tokens = n;
+        profile_opts.kv_start = base_pos;
+        Ds4GpuProfiler verify_profiler(
+            spec_env_flag("DFLASH_DS4_GPU_PROFILE"), profile_opts);
         if (seq_verify) {
+            verify_profiler.begin(Ds4GpuPhase::VerificationStep);
             std::vector<int32_t> am_all;
             std::vector<float> feat_all;
             std::vector<float> logits_all;
@@ -98,22 +116,27 @@ class DeepSeek4DFlashTarget : public DFlashTarget {
             last_tok = am_all.back();
             verify_n_ = n;
             if (all_argmax) *all_argmax = std::move(am_all);
+            verify_profiler.end(Ds4GpuPhase::VerificationStep);
+            verify_profiler.emit();
             return true;
         }
         std::vector<int32_t> am;
         // n==1 must take the dynamic (non-reuse) path: the reused decode graph
         // skips the capture/all-logits hooks (backend HC), which this needs.
-        if (!deepseek4_dspark_verify_forward(backend_, device_, w_, cache_, capture_ids_,
-                                             embed_buf_.data(), tokens.data(), n, base_pos, am,
-                                             keep_logits_ ? &verify_logits_ : nullptr,
-                                             verify_features_, telemetry_,
-                                             /*allow_graph_reuse=*/n > 1)) {
+        verify_profiler.begin(Ds4GpuPhase::VerificationStep);
+        const bool verify_ok = deepseek4_dspark_verify_forward(
+            backend_, device_, w_, cache_, capture_ids_, embed_buf_.data(), tokens.data(),
+            n, base_pos, am, keep_logits_ ? &verify_logits_ : nullptr,
+            verify_features_, telemetry_, /*allow_graph_reuse=*/n > 1);
+        verify_profiler.end(Ds4GpuPhase::VerificationStep);
+        if (!verify_ok) {
             return false;
         }
         if (am.empty()) return false;
         last_tok = am.back();
         verify_n_ = n;
         if (all_argmax) *all_argmax = std::move(am);
+        verify_profiler.emit();
         return true;
     }
 
@@ -426,13 +449,17 @@ bool run_deepseek4_dspark_spec_decode(
 
     const bool debug = spec_env_flag("DFLASH_DS4_DSPARK_DEBUG");
     const bool timing = spec_env_flag("DFLASH_DS4_TIMING");
+    const bool parity_trace = spec_env_flag("DFLASH_DS4_PARITY_TRACE");
     const bool full_snap = spec_env_flag("DFLASH_DS4_FULL_SNAP");
     const bool seq_verify_mode = spec_env_flag("DFLASH_DS4_SEQ_VERIFY");
     // Laguna-style adaptive verify width: EWMA of accepted candidates, width =
     // ewma + 2 (avg_commit << block means the wide tail is usually wasted).
     // /tmp/ds4_awidth: 1 = on, 0 = off (default on).
     bool adaptive_width = true;
-    if (std::FILE * f = std::fopen("/tmp/ds4_awidth", "r")) {
+    const char * adaptive_env = std::getenv("DFLASH_DS4_ADAPTIVE_WIDTH");
+    if (adaptive_env && *adaptive_env) {
+        adaptive_width = *adaptive_env != '0';
+    } else if (std::FILE * f = std::fopen("/tmp/ds4_awidth", "r")) {
         int v = 1;
         if (std::fscanf(f, "%d", &v) == 1) adaptive_width = (v != 0);
         std::fclose(f);
@@ -480,6 +507,7 @@ bool run_deepseek4_dspark_spec_decode(
 
     DeepSeek4DFlashTarget target(target_w, target_cache, backend, device, snap_backend,
                                  drafter.capture_layer_ids, drafter.mask_token_id);
+    target.set_keep_logits(parity_trace);
     DraftWeights dw = make_dspark_shim(drafter);
     DeepSeek4SpecRollback rollback;
     DeepSeek4StepTelemetry tel{};
@@ -680,6 +708,31 @@ bool run_deepseek4_dspark_spec_decode(
         }
         tm_verify += spec_ms_since(t0);
 
+        if (parity_trace) {
+            std::vector<float> trace_logits;
+            if (target.read_verify_logits(q, trace_logits)) {
+                for (int row = 0; row < q; ++row) {
+                    const float * logits = trace_logits.data() + (size_t) row * target_w.n_vocab;
+                    int best = 0;
+                    int second = -1;
+                    for (int tok = 1; tok < target_w.n_vocab; ++tok) {
+                        if (logits[tok] > logits[best]) {
+                            second = best;
+                            best = tok;
+                        } else if (second < 0 || logits[tok] > logits[second]) {
+                            second = tok;
+                        }
+                    }
+                    const float second_logit = second >= 0 ? logits[second] : logits[best];
+                    std::fprintf(stderr,
+                        "[ds4-parity-logits] step=%ld pos=%d row=%d best=%d val=%.9g "
+                        "second=%d val2=%.9g margin=%.9g\n",
+                        steps, pos, row, best, logits[best], second, second_logit,
+                        logits[best] - second_logit);
+                }
+            }
+        }
+
         // Accept the longest matching prefix. accept counts the seed (slot 0)
         // plus each candidate the target agrees with.
         int accept = 1;