From d38018053b5034109ed4335986b7eef3db60b3c7 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Sun, 4 Oct 2026 18:51:30 +0900 Subject: [PATCH 1/8] update(rocm): port the fused MoE decode kernels to HIP The fused two-kernel decode MoE path (gate-up with SwiGLU or GeGLU, then down with f32 partials) had Metal and CUDA ports only, and the Rust gates in front of it read the backend-wide custom_kernels_available(), so a HIP port would have stayed unreachable. Add MOE_GATEUP_HIP_SOURCE and MOE_DOWN_HIP_SOURCE, ported from the CUDA sources with the same inputs, grid and template args, the wave32 guard, and explicit-width shuffles; the down kernel keeps its 6-bit branch and the single final rounding (#886). Fill both tables' .rocm slots. New predicates fused_moe_kernels_available() and moe_down_kernel_available() read the tables and gate SwitchGLU (switch_layers.rs) and Nemotron-H (nemotron_h.rs). The MLXCEL_FUSED_MOE_RELU2 branch now also requires the fc1_relu2 and down ports, so it declines to gather_qmm on CUDA and ROCm instead of refusing. The parity tests gate on the predicate, fail rather than skip on a GPU backend without it, and gain a SwiGLU case at 4/4, 8/8 and 4/6 bits. Validated on gfx1151: fused_moe_parity_tests (5 passed; a broken shuffle ladder makes both reference tests fail), qwen3_moe and qwen3_vl_moe positive controls, clippy, and the dtype-key and port-dispatch gates. Closes #2065 --- README.md | 2 +- docs/environment-variables.md | 6 +- docs/installation.md | 2 +- src/lib/mlxcel-core/cpp/mlx_cxx_bridge.h | 8 + src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp | 229 +++++++++++++- .../mlxcel-core/src/fused_moe_parity_tests.rs | 299 ++++++++++++------ src/lib/mlxcel-core/src/lib.rs | 14 + src/models/nemotron_h.rs | 23 +- src/models/qwen3_moe_tests.rs | 4 +- src/models/qwen3_vl_moe_tests.rs | 4 +- src/models/switch_layers.rs | 21 +- 11 files changed, 490 insertions(+), 122 deletions(-) diff --git a/README.md b/README.md index fe05e9684..e2494f535 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ Apple Silicon is the primary target. Linux/CUDA is supported as a secondary targ The current `main` branch is v0.7.0 plus unreleased work. Install from source to use features that have not reached the latest tagged release; see the [changelog](CHANGELOG.md) for the release-by-release record. -- **Experimental AMD GPU (ROCm) support.** `--features rocm` builds mlxcel on Linux against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (`gfx1151`); fused kernels fall back to MLX graphs and affine MoE models need `MLXCEL_FUSED_MOE=0` for now. See [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental) for the build and the open gaps. +- **Experimental AMD GPU (ROCm) support.** `--features rocm` builds mlxcel on Linux against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (`gfx1151`); the fused kernels with a HIP port (BitLinear, the decode-MoE pair) run natively and the rest fall back to MLX graphs. See [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental) for the build and the open gaps. - **Verified `llama-server` compatibility.** A frozen b10621 manifest classifies all 376 pinned options, routes, and native request fields, with no deferred entries. Native completion, embedding, tokenization, template, infill, props, slots, metrics, resumable-stream, router, and LoRA surfaces are implemented or explicitly classified instead of being silently ignored. - **OpenAI, Anthropic, and Vertex-compatible serving.** Chat Completions, Completions, Responses, Embeddings, Reranking, Audio, and Anthropic Messages are available alongside the native `llama-server` routes. Optional Vertex AI custom-container routing is supported through the standard `AIP_*` variables. - **Multi-model routing and live adapters.** Router mode discovers checkpoints from the model store, a model directory, or INI presets; loads them on demand; and bounds the resident set with LRU eviction. Multiple LoRA adapters can remain unfused for per-request or live scale changes, or be fused for zero decode overhead. diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 368039ac9..dc6e2ab8f 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -535,10 +535,10 @@ recommended as normal deployment settings. | `MLXCEL_SDPA_FALLBACK_MAX_QUERIES` | non-negative integer | `32` | **CUDA only.** A masked SDPA call with 2 to N query rows over a longer key sequence (the speculative verify shape: a block appended to a KV cache) bypasses cuDNN and takes MLX's own ops fallback (issue #1799). cuDNN caches its execution plan by the exact shapes, and a verify round's key length changes every round, so with cuDNN every such layer class rebuilt a plan on the host each round: about 22 ms per build on GB10, 67 to 76 ms per round on the Laguna DFlash pairing, the whole fixed floor of that round, and the plan cache's lifetime miss counter then aborted the process. The one-row decode step takes the vector kernel and is unaffected; prefill keeps cuDNN (its key length equals its query length, or it has more rows than the bound), except the trailing short chunk of a chunked prefill and a short incremental prefill over a reused prefix-cache prefix, which take the fallback. The fallback's cost grows with the key length (a `[B, heads, q_len, k_len]` score matrix, about 10 ms more per round than cuDNN at block 2 with 350 keys) while the plan build it replaces is constant, so on very long contexts (tens of thousands of keys at block 16, unmeasured) cuDNN could be the cheaper side again. Issue #1820 left this gate's shipped behaviour unchanged: bucketing is off by default, so this gate still claims every array-masked verify block as it did before. It is narrowed only when `MLXCEL_SDPA_PLAN_BUCKET_MAX_QUERIES` is set non-zero, in which case it stops claiming calls whose plan-cache key can be bucketed and covers only a causal-mode block with no array mask, where there is no mask to widen. `0` restores upstream dispatch without a rebuild, which is the kill switch used for the A/B in `docs/benchmark_results/laguna-dflash-verify-cost-gb10-2026-09-11.md`. | | `MLXCEL_PIPELINE_GRANULARITY` | `off`, `layer`, `block:N` | `off` | Inserts layer-boundary async-eval hints for pipeline experiments. | | `MLXCEL_FUSED_ADD_NORM` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fuses a parallel-residual block's `(attn + mlp) + x` with the LayerNorm that consumes it (the next block's input norm, or the final norm) into one Metal kernel, saving one dispatch and one barrier level per layer during decode. The kernel copies MLX's single-row `layer_norm` reduction, so its outputs are byte-identical to the unfused add and norm (pinned by `residual_add3_layer_norm_matches_the_unfused_pair`); it falls back to the unfused pair off Metal, above a 6656-wide row, or on mixed dtypes. Used by Cohere2: command-r7b 4-bit decode on M1 Ultra +1.1% (eight ABBA pairs, 112.45 to 113.72 tok/s). | -| `MLXCEL_FUSED_MOE` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fused single-token decode-MoE kernel (#268), on by default since #282 (Metal) and #319 (CUDA, via `mx.fast.cuda_kernel`); validated on M1 Ultra, M5, and GB10. Set to `0` to force the proven `gather_qmm`/`SwitchGLU` path. Active for afmoe, bailing_moe, cohere2_moe, dbrx, dots.llm1, gemma4, klear, laguna, lfm2, mellum, minimax, mixtral, olmoe, phimoe, qwen2_moe, qwen3_moe, qwen3_next (and Qwen3.5), qwen3_vl_moe, and the qwen3_omni_moe thinker and talker decode. Byte-identical greedy output is checkpoint- and prompt-dependent and was never a general property (#1045): it held on `qwen3-30b-a3b` for the #1045 prompt but not for every prompt (on GB10 one diverges at generated token 39, #1884), and not on Klear. This is not a defect, since the kernel measures roughly 6x closer to an all-f32 ground truth than `gather_qmm` on both, but `gather_qmm` is what mlx-lm mirrors, so set this to `0` when reference-diffing a new MoE port. On the experimental ROCm build the fused kernel has no ROCm port and aborts on affine MoE models, so set this to `0` there until lablup/mlxcel#1803. | -| `MLXCEL_FUSED_MOE_SGY` | `1`-`32` | `8` | Simdgroups (Metal) / warps-per-block (CUDA) per threadgroup for the fused decode-MoE kernel; tune per hardware. | +| `MLXCEL_FUSED_MOE` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fused single-token decode-MoE kernel (#268), on by default since #282 (Metal) and #319 (CUDA, via `mx.fast.cuda_kernel`); validated on M1 Ultra, M5, and GB10. Set to `0` to force the proven `gather_qmm`/`SwitchGLU` path. Active for afmoe, bailing_moe, cohere2_moe, dbrx, dots.llm1, gemma4, klear, laguna, lfm2, mellum, minimax, mixtral, olmoe, phimoe, qwen2_moe, qwen3_moe, qwen3_next (and Qwen3.5), qwen3_vl_moe, and the qwen3_omni_moe thinker and talker decode. Byte-identical greedy output is checkpoint- and prompt-dependent and was never a general property (#1045): it held on `qwen3-30b-a3b` for the #1045 prompt but not for every prompt (on GB10 one diverges at generated token 39, #1884), and not on Klear. This is not a defect, since the kernel measures roughly 6x closer to an all-f32 ground truth than `gather_qmm` on both, but `gather_qmm` is what mlx-lm mirrors, so set this to `0` when reference-diffing a new MoE port. On the experimental ROCm build the kernel runs through `mx.fast.hip_kernel` since lablup/mlxcel#2065 on wave32 targets such as gfx1151 (a wave64 target fails to compile it rather than computing a wrong result), with the CUDA default for `MLXCEL_FUSED_MOE_MAX_DFF`. | +| `MLXCEL_FUSED_MOE_SGY` | `1`-`32` | `8` | Simdgroups (Metal) / warps (CUDA) / wavefronts (ROCm) per threadgroup for the fused decode-MoE kernel; tune per hardware. | | `MLXCEL_FUSED_MOE_MAX_DFF` | positive int | `4096` (Metal) / `8192` (CUDA) | Expert-intermediate (Dff) upper bound for the fused path; above it the caller falls back to `gather_qmm`. The fused path wins only while `gather_qmm` underutilizes the GPU (small experts), so the break-even is backend-dependent and the default is chosen from the live backend: `4096` on Metal (M1 Ultra tuning) and `8192` on CUDA (GB10 re-measured under MLX pin e9463bb, #626; fused wins through Dff 6400 and is break-even at 8192). An explicit value overrides the default on both backends: lower it to force `gather_qmm` sooner, raise it (e.g. `20000`) to force the fused kernel on larger experts such as mixtral (Dff 14336, where it is a slight net loss). Read by every family on the shared `SwitchGLU` fused path, qwen3_moe and qwen3_vl_moe (and the qwen3_omni_moe thinker through it) included since #1884; qwen3_next (and Qwen3.5 and the qwen3_omni_moe talker through it) and gemma4 dispatch their own kernel and do not read it yet. | -| `MLXCEL_FUSED_MOE_RELU2` | presence enables | off | Enables the squared-ReLU fused routed-expert path for nemotron-class experts. It is performance-neutral on nemotron-h even though MoE is its largest block because it replaces only the already-efficient routed fc1/fc2 GEMVs; the router, always-on shared expert, and combine remain outside this kernel. Kept for a future squared-ReLU model whose routed-expert slice dominates. | +| `MLXCEL_FUSED_MOE_RELU2` | presence enables | off | Enables the squared-ReLU fused routed-expert path for nemotron-class experts. It is performance-neutral on nemotron-h even though MoE is its largest block because it replaces only the already-efficient routed fc1/fc2 GEMVs; the router, always-on shared expert, and combine remain outside this kernel. Kept for a future squared-ReLU model whose routed-expert slice dominates. The fc1 kernel has a Metal port only, so on CUDA and ROCm the flag leaves the default `gather_qmm` routed path in place (lablup/mlxcel#2065; ROCm port tracked in lablup/mlxcel#2069). | | `MLXCEL_FUSED_MOE_PARITY_CHECK` | unset or `0` disables; nonzero integer enables | off | In-situ per-call parity/determinism probe for the fused decode-MoE kernel, added for the #886 corruption triage. Re-runs the fused kernel pair on identical inputs (bitwise-determinism check), computes the `gather_qmm` production-fallback reference, and computes an all-f32 dequantize-and-matmul ground truth, then logs to stderr any call that is non-deterministic or whose fused-vs-reference, fused-vs-ground-truth, or reference-vs-ground-truth deviation exceeds `MLXCEL_FUSED_MOE_PARITY_THRESHOLD` (normalized RMS). Heavy: adds several eager evals per MoE call. Diagnostic only, kept for future corruption triage; never enable in production. | | `MLXCEL_FUSED_MOE_PARITY_THRESHOLD` | float | `0.05` | Normalized-RMS deviation threshold for the `MLXCEL_FUSED_MOE_PARITY_CHECK` probe. | | `MLXCEL_GATHER_QMM_GROUPED` | `0` disables; any other value or unset enables | on | **CUDA only.** Gates the sorted MoE prefill fast path in `GatherQMM::eval_gpu` (issue #629). When the sorted-indices `M == 1` prefill contract holds (right-sorted, transpose, one activation row pre-gathered per (token, expert) pair, non-nvfp4, float activation dtype, `E <= 1024`) and the batch is large enough to amortize (see `MLXCEL_GATHER_QMM_GROUPED_MIN_ROWS`), the expert weight stack is dequantized once and routed through `cutlass_grouped_gemm_unaligned` instead of one 1-row `qmm_sm80` GEMM per (token, expert) pair. Fixes a 5-10x CUDA MoE prefill collapse relative to Metal M1 Ultra; measured 3.6-40x prefill speedup across mixtral-8x7b, phi-3.5-moe, llama-4-scout, minimax-m2, gpt-oss-20b, and solar-open-100b on GB10, with no decode-path regression. Set to `0` to force the legacy per-row dispatch (A/B, rollback). See `docs/benchmark_results/moe-prefill-grouped-gemm-gb10-2026-07-10.md`. | diff --git a/docs/installation.md b/docs/installation.md index a6dac83c8..928e272f6 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -551,7 +551,7 @@ so `mlxcel inspect` and `--estimate-memory` give the same answers as before. | Affine 4-bit / 8-bit checkpoints | Run natively. | | mxfp8 and mxfp4 checkpoints | Run natively, including MoE experts through `gather_qmm` (for example gpt-oss-20b-MXFP4-Q4). The load log says so once per mode (`Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion ...`). mxfp4 `quantize`, `quantized_matmul` and `gather_qmm` are checked against CPU references by `tests/rocm_mxfp4_quant.rs` (lablup/mlxcel#1808). Vendor FP8 block checkpoints (`quant_method: fp8`, 128x128 blocks) are requantized to mxfp8 at load and run natively; verified end to end on a dense 0.8B Qwen3.5 FP8 checkpoint, with mxfp8 `gather_qmm` and `quantized_matmul` checked by `models::switch_layers::mxfp_tests`. The official Qwen3.5 FP8 MoE releases have not been run on ROCm (lablup/mlxcel#1807). | | NVFP4 checkpoints | No native kernel. ModelOpt NVFP4 checkpoints (for example the Gemma 4 NVFP4 exports) are converted to affine 4-bit at load with no environment variables, and the load log names the route and the reason. A layer that cannot be converted fails the load with its name and the reason. MLX-native NVFP4 exports (`"mode": "nvfp4"` in `config.json`, such as `mlx-community/*-nvfp4`) have no load-time conversion and are refused at load; use an affine export instead (lablup/mlxcel#1806). | -| Affine MoE models (for example Qwen3-30B-A3B) | Run natively. The fused MoE path used to abort on ROCm; lablup/mlxcel#1803 routes it to the MLX graph fallback, so `MLXCEL_FUSED_MOE=0` is no longer needed. | +| Affine MoE models (for example Qwen3-30B-A3B) | Run natively. Single-token decode takes the fused two-kernel MoE path (gate-up, then down) through its HIP port (lablup/mlxcel#2065), as on Metal and CUDA; `MLXCEL_FUSED_MOE=0` selects the `gather_qmm` graph path instead. Experts wider than `MLXCEL_FUSED_MOE_MAX_DFF` (8192 by default, as on CUDA; Mixtral-8x7B's 14336) stay on `gather_qmm`. | | mlxcel's fused kernels (sampling, fused norm, RoPE + KV append, paged attention) | Run as MLX graph fallbacks on the paths that have one (lablup/mlxcel#1803); ROCm ports are lablup/mlxcel#1814. | | Hybrid SSM decode (granite-4.0-h, falcon-h1, plamo-2, Nemotron-H) | The single-token Mamba2 SSM update runs the fused HIP kernel instead of the ~55-op SSD graph (lablup/mlxcel#2067): decode on `gfx1151` went from 60.4 to 88.4 tok/s on granite-4.0-h-tiny-4bit and from 51.4 to 74.4 tok/s on Nemotron-3-Nano-30B-A3B-4bit, with model logits matching the graph and Metal's kernel at every decided position ([results](benchmark_results/rocm-ssm-update-kernel-gfx1151-2026-10-04.md)). Measured on granite and Nemotron-H; falcon-h1 and plamo-2 take the same gate but have not been run on ROCm. `MLXCEL_SSM_KERNEL=0` forces the graph. | | GPU faults | Reported as errors (lablup/mlxcel#1804). A launch HIP rejects (an oversized block, no code object for the device) fails the evaluation that issued it and the device stays usable. An asynchronous fault (an out-of-bounds access) fails the evaluation waiting on it within about a second instead of hanging or returning NaN; the server fails that request and answers later ones with the same error, because after a queue fault the HIP runtime rejects every call for the rest of the process, so the process has to restart, and its shutdown may need a SIGKILL (HIP's teardown waits on callbacks the faulted queue never runs). A kernel that never finishes is not detected; `MLX_ROCM_GPU_WATCHDOG_SECS` (default off) fails any single host wait that outlives it. | diff --git a/src/lib/mlxcel-core/cpp/mlx_cxx_bridge.h b/src/lib/mlxcel-core/cpp/mlx_cxx_bridge.h index 90a3ced83..e68c271d6 100644 --- a/src/lib/mlxcel-core/cpp/mlx_cxx_bridge.h +++ b/src/lib/mlxcel-core/cpp/mlx_cxx_bridge.h @@ -1420,6 +1420,14 @@ std::unique_ptr rocm_fault_probe_array(int32_t kind); // (issues #1803, #1862). bool bitlinear_kernel_available(); +// Fused decode-MoE port predicates (lablup/mlxcel#2065), read from the +// kernels' own `KernelPorts` tables through `has_kernel_port`: the gate-up and +// down pair that `fused_moe_expert_kernel` and `fused_moe_geglu_kernel` +// launch, and the down kernel alone, which Nemotron-H's `fused_moe_forward` +// reuses. Metal, CUDA and ROCm. +bool fused_moe_kernels_available(); +bool moe_down_kernel_available(); + // Whether `quantized_matmul(x, weight, scales, biases)` with a transposed // affine weight runs the same dense GEMM that `dequantize` + `matmul` runs, so // the two return the same bytes (lablup/mlxcel#2081). On ROCm this reads the diff --git a/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp b/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp index 506d9feb5..8a272dfbb 100644 --- a/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp +++ b/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp @@ -1446,6 +1446,8 @@ namespace { // holders are defined further down, after this function). mlx::core::fast::CustomKernelFunction& moe_fc1_relu2_kernel_fn(); mlx::core::fast::CustomKernelFunction& moe_down_kernel_fn(); +const mlxcel::KernelPorts& moe_fc1_relu2_ports(); +const mlxcel::KernelPorts& moe_down_ports(); } // namespace std::unique_ptr fused_moe_forward( @@ -1531,8 +1533,15 @@ std::unique_ptr fused_moe_forward( // wired behind the dedicated flag so it stays referenceable for a model // where that narrower routed-expert slice dominates; the default path below // stays on gather_qmm. + // + // The opt-in needs both kernels it launches. On a backend missing either + // port (ROCm has the down port since #2065 but no fc1_relu2 port until + // #2069) it declines to the gather_qmm branch below rather than reaching + // `select_kernel_port` and refusing the whole forward. bool fused_relu2 = x_shape[0] == 1 && (bits == 4 || bits == 8) && - std::getenv("MLXCEL_FUSED_MOE_RELU2"); + std::getenv("MLXCEL_FUSED_MOE_RELU2") && + mlxcel::has_kernel_port(moe_fc1_relu2_ports()) && + mlxcel::has_kernel_port(moe_down_ports()); array result = x.inner; // placeholder; overwritten in both branches if (fused_relu2) { int din = (int)x_shape[1]; @@ -2464,6 +2473,105 @@ namespace { return holder; } + // ---- ROCm ports of the two fused decode-MoE kernels (#2065). ---- + // The CUDA bodies above with the warp shuffle changed, following the + // bitlinear HIP port (#1862): HIP's `__shfl_down_sync` is a compatibility + // shim that ignores its mask, so the native `__shfl_down(var, delta, width)` + // is used with the width stated. The launch geometry is the CUDA one (one + // 32-lane wavefront per output row on threadIdx.x, `sgy` rows per block on + // threadIdx.y, the expert slot on grid.z), and the template args are the + // same, so the hipRTC cache key carries `T` exactly as the CUDA key does. + // Precise `expf`/`tanhf` as in CUDA; hipRTC compiles at -O3 without + // fast-math, so they stay the precise library calls. + static const char* MOE_GATEUP_HIP_SOURCE = R"( + // The fold below starts at 16, which is correct only for a 32-lane + // wavefront; on a wave64 target (CDNA: gfx90a, gfx942) it would drop + // half the lanes and still return a finite, plausible, wrong result. + // Preprocessor checks, not `static_assert(warpSize == 32)`, which does + // not compile in HIP (see the bitlinear HIP source). + #if defined(__AMDGCN_WAVEFRONT_SIZE__) && __AMDGCN_WAVEFRONT_SIZE__ != 32 + #error "moe_gateup_kernel_hip assumes a 32-lane wavefront" + #endif + #if defined(__AMDGCN_WAVEFRONT_SIZE) && __AMDGCN_WAVEFRONT_SIZE != 32 + #error "moe_gateup_kernel_hip assumes a 32-lane wavefront" + #endif + uint32_t lane = threadIdx.x; // 0..31 (one wavefront) + uint32_t f = blockIdx.y * blockDim.y + threadIdx.y; // output row 0..Dff-1 + uint32_t eslot = blockIdx.z; // 0..K-1 + if (f >= (uint32_t)Dff) return; // wavefront-uniform + uint32_t e = indices[eslot]; + + constexpr uint32_t vpw = 32u / bits; + constexpr uint32_t wmask = (1u << bits) - 1u; + constexpr uint32_t Din_p = Din / vpw; + constexpr uint32_t G = Din / group_size; + + uint32_t row = e * Dff + f; + const uint32_t* gwr = gate_w + row * Din_p; + const T* gsr = gate_s + row * G; + const T* gbr = gate_b + row * G; + const uint32_t* uwr = up_w + row * Din_p; + const T* usr = up_s + row * G; + const T* ubr = up_b + row * G; + float g = 0.0f, u = 0.0f; + for (uint32_t p = lane; p < Din_p; p += 32u) { + uint32_t base = p * vpw; + uint32_t grp = base / group_size; + float gs = (float)gsr[grp], gb = (float)gbr[grp]; + float us = (float)usr[grp], ub = (float)ubr[grp]; + uint32_t gpk = gwr[p]; + uint32_t upk = uwr[p]; + for (uint32_t j = 0; j < vpw; ++j) { + float xv = (float)x[base + j]; + g += xv * ((float)((gpk >> (j * bits)) & wmask) * gs + gb); + u += xv * ((float)((upk >> (j * bits)) & wmask) * us + ub); + } + } + #pragma unroll + for (int o = 16; o > 0; o >>= 1) { + g += __shfl_down(g, o, 32); + u += __shfl_down(u, o, 32); + } + if (lane == 0u) { + if (act == 1) { + // GeGLU (gelu tanh approx) * up (gemma4 experts). + float g3 = g * g * g; + float inner = 0.7978845608028654f * (g + 0.044715f * g3); + float gelu = 0.5f * g * (1.0f + tanhf(inner)); + act_g[eslot * Dff + f] = gelu * u; + } else { + // SwiGLU (silu) * up, precise expf as in the CUDA port. + act_g[eslot * Dff + f] = (g / (1.0f + expf(-g))) * u; + } + } + )"; + + struct MoeGateUpKernelHolderHip { + std::optional kernel; + bool initialized = false; + mlx::core::fast::CustomKernelFunction& get() { + if (!initialized) { +#ifdef MLXCEL_BRIDGE_ROCM_BACKEND + kernel = mlx::core::fast::hip_kernel( + "moe_gateup_kernel_hip", + {"x", "indices", "gate_w", "gate_s", "gate_b", + "up_w", "up_s", "up_b"}, + {"act_g"}, + MOE_GATEUP_HIP_SOURCE); +#else + throw std::runtime_error( + "[fused_moe] this build has no ROCm backend"); +#endif + initialized = true; + } + return *kernel; + } + }; + static MoeGateUpKernelHolderHip& get_moe_gateup_kernel_hip() { + static MoeGateUpKernelHolderHip holder; + return holder; + } + // This kernel's ports, in one place. `has_kernel_port` and `select_kernel_port` // both read it, so a support predicate and the dispatch cannot answer // differently (#1801). @@ -2475,8 +2583,9 @@ const mlxcel::KernelPorts& moe_gateup_ports() { .cuda = +[]() -> mlx::core::fast::CustomKernelFunction& { return get_moe_gateup_kernel_cuda().get(); }, - // No HIP port yet (#1814). - .rocm = nullptr, + .rocm = +[]() -> mlx::core::fast::CustomKernelFunction& { + return get_moe_gateup_kernel_hip().get(); + }, }; return ports; } @@ -2502,6 +2611,103 @@ const mlxcel::KernelPorts& moe_gateup_ports() { return holder; } + // ROCm port of the down kernel (#2065): the CUDA body with the shuffle + // changed as in MOE_GATEUP_HIP_SOURCE. Serves all three callers of + // `moe_down_ports()`: SwitchGLU and GeGLU through + // `run_fused_moe_two_kernel`, and Nemotron-H's opt-in squared-ReLU path + // through `moe_down_kernel_fn()`. + static const char* MOE_DOWN_HIP_SOURCE = R"( + // 32-lane wavefront only; see MOE_GATEUP_HIP_SOURCE. + #if defined(__AMDGCN_WAVEFRONT_SIZE__) && __AMDGCN_WAVEFRONT_SIZE__ != 32 + #error "moe_down_kernel_hip assumes a 32-lane wavefront" + #endif + #if defined(__AMDGCN_WAVEFRONT_SIZE) && __AMDGCN_WAVEFRONT_SIZE != 32 + #error "moe_down_kernel_hip assumes a 32-lane wavefront" + #endif + uint32_t lane = threadIdx.x; // 0..31 (one wavefront) + uint32_t h = blockIdx.y * blockDim.y + threadIdx.y; // output row 0..Din-1 + uint32_t eslot = blockIdx.z; // 0..K-1 + if (h >= (uint32_t)Din) return; // wavefront-uniform + uint32_t e = indices[eslot]; + + constexpr uint32_t Gd = Dff / group_size; + uint32_t row = e * Din + h; + const T* dsr = down_s + row * Gd; + const T* dbr = down_b + row * Gd; + const float* a = act_g + eslot * Dff; + float d = 0.0f; + + if (bits == 6) { + // 6-bit: MLX packs 4 weights into 3 bytes; read the row as bytes. + // Layout matches quantized.h qdot: v0=b0&0x3f, + // v1=(b0>>6)|((b1&0x0f)<<2), v2=(b1>>4)|((b2&0x03)<<4), v3=b2>>2. + constexpr uint32_t packs = Dff / 4u; + constexpr uint32_t row_u32 = Dff * 3u / 16u; + const uint8_t* wb = reinterpret_cast(down_w + row * row_u32); + for (uint32_t p = lane; p < packs; p += 32u) { + uint32_t base = p * 4u; + uint32_t grp = base / group_size; + float ds = (float)dsr[grp], db = (float)dbr[grp]; + uint32_t b0 = wb[p * 3u], b1 = wb[p * 3u + 1u], b2 = wb[p * 3u + 2u]; + uint32_t v0 = b0 & 0x3fu; + uint32_t v1 = (b0 >> 6) | ((b1 & 0x0fu) << 2); + uint32_t v2 = (b1 >> 4) | ((b2 & 0x03u) << 4); + uint32_t v3 = (b2 >> 2); + d += a[base + 0u] * ((float)v0 * ds + db); + d += a[base + 1u] * ((float)v1 * ds + db); + d += a[base + 2u] * ((float)v2 * ds + db); + d += a[base + 3u] * ((float)v3 * ds + db); + } + } else { + // Power-of-2 bits (4/8): vpw weights per 32-bit pack. + constexpr uint32_t vpw = 32u / bits; + constexpr uint32_t wmask = (1u << bits) - 1u; + constexpr uint32_t Dff_p = Dff / vpw; + const uint32_t* dwr = down_w + row * Dff_p; + for (uint32_t p = lane; p < Dff_p; p += 32u) { + uint32_t base = p * vpw; + uint32_t grp = base / group_size; + float ds = (float)dsr[grp], db = (float)dbr[grp]; + uint32_t dpk = dwr[p]; + for (uint32_t j = 0; j < vpw; ++j) { + d += a[base + j] * ((float)((dpk >> (j * bits)) & wmask) * ds + db); + } + } + } + #pragma unroll + for (int o = 16; o > 0; o >>= 1) d += __shfl_down(d, o, 32); + if (lane == 0u) { + // f32 partial (score folded in f32), rounded to the activation + // dtype once by the host after the K-sum (#886). + out[eslot * Din + h] = (float)scores[eslot] * d; + } + )"; + + struct MoeDownKernelHolderHip { + std::optional kernel; + bool initialized = false; + mlx::core::fast::CustomKernelFunction& get() { + if (!initialized) { +#ifdef MLXCEL_BRIDGE_ROCM_BACKEND + kernel = mlx::core::fast::hip_kernel( + "moe_down_kernel_hip", + {"indices", "down_w", "down_s", "down_b", "act_g", "scores"}, + {"out"}, + MOE_DOWN_HIP_SOURCE); +#else + throw std::runtime_error( + "[fused_moe] this build has no ROCm backend"); +#endif + initialized = true; + } + return *kernel; + } + }; + static MoeDownKernelHolderHip& get_moe_down_kernel_hip() { + static MoeDownKernelHolderHip holder; + return holder; + } + // This kernel's ports, in one place. `has_kernel_port` and `select_kernel_port` // both read it, so a support predicate and the dispatch cannot answer // differently (#1801). @@ -2513,8 +2719,9 @@ const mlxcel::KernelPorts& moe_down_ports() { .cuda = +[]() -> mlx::core::fast::CustomKernelFunction& { return get_moe_down_kernel_cuda().get(); }, - // No HIP port yet (#1814). - .rocm = nullptr, + .rocm = +[]() -> mlx::core::fast::CustomKernelFunction& { + return get_moe_down_kernel_hip().get(); + }, }; return ports; } @@ -2606,6 +2813,18 @@ const mlxcel::KernelPorts& moe_fc1_relu2_ports() { } } +// Support predicates for the fused decode-MoE kernels (#2065), read from the +// kernels' own tables so a gate and the dispatch cannot disagree. Metal, CUDA +// and ROCm today. +bool fused_moe_kernels_available() { + return mlxcel::has_kernel_port(moe_gateup_ports()) && + mlxcel::has_kernel_port(moe_down_ports()); +} + +bool moe_down_kernel_available() { + return mlxcel::has_kernel_port(moe_down_ports()); +} + namespace { // Shared body for the two fused decode-MoE FFIs. `act` selects the gate/up // activation: 0 = SwiGLU (silu), 1 = GeGLU (gelu tanh approx, gemma4). gate/up diff --git a/src/lib/mlxcel-core/src/fused_moe_parity_tests.rs b/src/lib/mlxcel-core/src/fused_moe_parity_tests.rs index 9b5439292..3b58bc138 100644 --- a/src/lib/mlxcel-core/src/fused_moe_parity_tests.rs +++ b/src/lib/mlxcel-core/src/fused_moe_parity_tests.rs @@ -42,14 +42,18 @@ //! change per-row addressing (`row = e * Dff + f`), only the range of `e`. //! //! GPU-only: the kernels JIT through `mx.fast.metal_kernel` / -//! `mx.fast.cuda_kernel`, so the tests skip (return early) on CPU-only builds, -//! matching the convention of the fused paged-decode tests in `ffi_tests.rs`. +//! `mx.fast.cuda_kernel` / `mx.fast.hip_kernel`, so the tests skip (return +//! early) wherever `fused_moe_kernels_available()` is false, matching the +//! convention of the fused paged-decode tests in `ffi_tests.rs`. //! //! Run on CUDA (GB10 etc.): //! MLX_CUDA_ARCHITECTURES=121 cargo test -p mlxcel-core --release \ //! --features cuda fused_moe_geglu //! Run on Apple Silicon: //! cargo test -p mlxcel-core --release fused_moe_geglu +//! Run on ROCm (gfx1151 etc.): +//! cargo test -p mlxcel-core --release --features rocm --lib \ +//! fused_moe_parity_tests -- --test-threads=1 use super::*; @@ -86,14 +90,25 @@ fn random_quantized_expert_stack( (w, s, b) } -/// One randomized GeGLU MoE decode case: bf16 activations/scales/biases, -/// 4-bit affine weights, K distinct experts, positive normalized scores. +/// Gate/up activation the fused gate-up kernel applies (its `act` template +/// arg): SwiGLU for the SwitchGLU families, tanh-approx GeGLU for Gemma4. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum MoeAct { + SwiGlu, + GeGlu, +} + +/// One randomized MoE decode case: bf16 activations/scales/biases, affine +/// weights (gate/up at `gu_bits`, down at `d_bits`), K distinct experts, +/// positive normalized scores. struct GegluMoeCase { din: i32, dff: i32, k: i32, group_size: i32, - bits: i32, + gu_bits: i32, + d_bits: i32, + act: MoeAct, x: UniquePtr, // [din] bf16 indices: UniquePtr, // [k] u32, distinct scores: UniquePtr, // [k] bf16 (pre-rounded so both paths see identical values) @@ -109,8 +124,21 @@ struct GegluMoeCase { } fn build_geglu_case(seed: u64, din: i32, dff: i32, num_experts: i32, k: i32) -> GegluMoeCase { + build_moe_case(seed, din, dff, num_experts, k, MoeAct::GeGlu, 4, 4) +} + +#[allow(clippy::too_many_arguments)] +fn build_moe_case( + seed: u64, + din: i32, + dff: i32, + num_experts: i32, + k: i32, + act: MoeAct, + gu_bits: i32, + d_bits: i32, +) -> GegluMoeCase { let group_size = 64; - let bits = 4; random_seed(seed); let x_f32 = unsafe { random_normal(&[din], dtype::FLOAT32, std::ptr::null()) }; @@ -144,17 +172,20 @@ fn build_geglu_case(seed: u64, din: i32, dff: i32, num_experts: i32, k: i32) -> eval(&scores); let (gate_w, gate_s, gate_b) = - random_quantized_expert_stack(num_experts, dff, din, group_size, bits); - let (up_w, up_s, up_b) = random_quantized_expert_stack(num_experts, dff, din, group_size, bits); + random_quantized_expert_stack(num_experts, dff, din, group_size, gu_bits); + let (up_w, up_s, up_b) = + random_quantized_expert_stack(num_experts, dff, din, group_size, gu_bits); let (down_w, down_s, down_b) = - random_quantized_expert_stack(num_experts, din, dff, group_size, bits); + random_quantized_expert_stack(num_experts, din, dff, group_size, d_bits); GegluMoeCase { din, dff, k, group_size, - bits, + gu_bits, + d_bits, + act, x, indices, scores, @@ -170,10 +201,15 @@ fn build_geglu_case(seed: u64, din: i32, dff: i32, num_experts: i32, k: i32) -> } } -/// Invoke the fused two-dispatch GeGLU kernel exactly as -/// `SwitchGeGLU::forward_fused_kernel` does and force evaluation. +/// Invoke the fused two-dispatch kernel exactly as +/// `SwitchGeGLU::forward_fused_kernel` (GeGLU) or +/// `SwitchGLU::forward_fused_kernel` (SwiGLU) does and force evaluation. fn run_fused(case: &GegluMoeCase) -> UniquePtr { - let out = fused_moe_geglu_kernel( + let launch = match case.act { + MoeAct::GeGlu => fused_moe_geglu_kernel, + MoeAct::SwiGlu => fused_moe_expert_kernel, + }; + let out = launch( &case.x, &case.indices, &case.gate_w, @@ -189,8 +225,8 @@ fn run_fused(case: &GegluMoeCase) -> UniquePtr { case.din, case.dff, case.k, - case.bits, - case.bits, + case.gu_bits, + case.d_bits, case.group_size, ) .expect("gpu_backend_or_skip() reported a port, so the launcher must not refuse"); @@ -209,7 +245,7 @@ fn reference_dense_f32(case: &GegluMoeCase) -> UniquePtr { // bf16-dequantized reference would round every weight to bf16 and carry // MORE error than either production path (bf16 -> f32 is exact, so the // upcast does not change the stored values). - let dq = |w: &MlxArray, s: &MlxArray, b: &MlxArray| -> UniquePtr { + let dq = |w: &MlxArray, s: &MlxArray, b: &MlxArray, bits: i32| -> UniquePtr { let s32 = astype(s, dtype::FLOAT32); let b32 = astype(b, dtype::FLOAT32); unsafe { @@ -218,14 +254,14 @@ fn reference_dense_f32(case: &GegluMoeCase) -> UniquePtr { &s32, &b32 as &MlxArray as *const MlxArray, case.group_size, - case.bits, + bits, "affine", ) } }; - let gate_dq = dq(&case.gate_w, &case.gate_s, &case.gate_b); // [E, dff, din] - let up_dq = dq(&case.up_w, &case.up_s, &case.up_b); // [E, dff, din] - let down_dq = dq(&case.down_w, &case.down_s, &case.down_b); // [E, din, dff] + let gate_dq = dq(&case.gate_w, &case.gate_s, &case.gate_b, case.gu_bits); // [E, dff, din] + let up_dq = dq(&case.up_w, &case.up_s, &case.up_b, case.gu_bits); // [E, dff, din] + let down_dq = dq(&case.down_w, &case.down_s, &case.down_b, case.d_bits); // [E, din, dff] let gate_sel = take(&gate_dq, &case.indices, 0); // [k, dff, din] let up_sel = take(&up_dq, &case.indices, 0); @@ -234,7 +270,7 @@ fn reference_dense_f32(case: &GegluMoeCase) -> UniquePtr { let x_col = reshape(&astype(&case.x, dtype::FLOAT32), &[case.din, 1]); let g = squeeze_axis(&matmul(&gate_sel, &x_col), -1); // [k, dff] let u = squeeze_axis(&matmul(&up_sel, &x_col), -1); // [k, dff] - let act = compiled_geglu_approx_activation(&g, &u); // [k, dff] f32 + let act = activation(case.act, &g, &u); // [k, dff] f32 let act_col = reshape(&act, &[case.k, case.dff, 1]); let per_expert = squeeze_axis(&matmul(&down_sel, &act_col), -1); // [k, din] @@ -252,7 +288,12 @@ fn reference_dense_f32(case: &GegluMoeCase) -> UniquePtr { fn reference_gather_qmm(case: &GegluMoeCase) -> UniquePtr { let x4 = reshape(&case.x, &[1, 1, 1, case.din]); let idx2 = reshape(&case.indices, &[1, case.k]); - let gq = |w: &MlxArray, s: &MlxArray, b: &MlxArray, x: &MlxArray| -> UniquePtr { + let gq = |w: &MlxArray, + s: &MlxArray, + b: &MlxArray, + x: &MlxArray, + bits: i32| + -> UniquePtr { unsafe { gather_qmm( x, @@ -263,16 +304,16 @@ fn reference_gather_qmm(case: &GegluMoeCase) -> UniquePtr { &idx2 as &MlxArray as *const MlxArray, true, case.group_size, - case.bits, + bits, false, "affine", ) } }; - let gate = gq(&case.gate_w, &case.gate_s, &case.gate_b, &x4); // [1, 1, k, dff] - let up = gq(&case.up_w, &case.up_s, &case.up_b, &x4); - let act = compiled_geglu_approx_activation(&gate, &up); - let down = gq(&case.down_w, &case.down_s, &case.down_b, &act); // [1, 1, k, din] + let gate = gq(&case.gate_w, &case.gate_s, &case.gate_b, &x4, case.gu_bits); // [1, 1, k, dff] + let up = gq(&case.up_w, &case.up_s, &case.up_b, &x4, case.gu_bits); + let act = activation(case.act, &gate, &up); + let down = gq(&case.down_w, &case.down_s, &case.down_b, &act, case.d_bits); // [1, 1, k, din] let per_expert = reshape(&down, &[case.k, case.din]); let w_col = reshape(&astype(&case.scores, dtype::FLOAT32), &[case.k, 1]); @@ -282,6 +323,14 @@ fn reference_gather_qmm(case: &GegluMoeCase) -> UniquePtr { out } +/// The gate/up activation both references apply, matching the kernel's `act`. +fn activation(act: MoeAct, gate: &MlxArray, up: &MlxArray) -> UniquePtr { + match act { + MoeAct::GeGlu => compiled_geglu_approx_activation(gate, up), + MoeAct::SwiGlu => compiled_swiglu_activation(gate, up), + } +} + fn flatten_f32(arr: &MlxArray) -> Vec { let a = astype(arr, dtype::FLOAT32); eval(&a); @@ -316,16 +365,29 @@ fn normalized_deviation(a: &[f32], b: &[f32]) -> (f64, f64) { ) } +/// The GPU backend's name, or `None` (skip) on a build without one. Every GPU +/// backend has both fused MoE kernel ports since issue #2065 (Metal, CUDA, +/// ROCm), so on a GPU backend `fused_moe_kernels_available()`, which reads the +/// same tables the launcher dispatches through, must answer true: a `false` +/// there is a defect the test reports instead of skipping past. fn gpu_backend_or_skip() -> Option<&'static str> { - if crate::metal_is_available() { - Some("metal") - } else if crate::cuda_is_available() { - Some("cuda") - } else { - // The fused kernels JIT through mx.fast.metal_kernel / - // mx.fast.cuda_kernel; a CPU-only build cannot launch either body. - None - } + use crate::hardware::{GpuBackendKind, gpu_backend_kind}; + let name = match gpu_backend_kind() { + GpuBackendKind::Metal => "metal", + GpuBackendKind::Cuda => "cuda", + GpuBackendKind::Rocm => "rocm", + GpuBackendKind::None => { + // The kernels JIT through mx.fast.{metal,cuda,hip}_kernel; a + // CPU-only build cannot launch either body. + eprintln!("skipping: no GPU backend, so no fused MoE kernel port"); + return None; + } + }; + assert!( + crate::fused_moe_kernels_available(), + "{name} has both fused MoE kernel ports, so fused_moe_kernels_available() must be true" + ); + Some(name) } /// Absolute ceiling on the `gather_qmm` fallback's own deviation from the @@ -372,54 +434,81 @@ fn fused_moe_geglu_kernel_matches_references_gemma4_shape() { // Din/Dff/K/bits/group_size exactly as gemma-4-26b-a4b; E cut to 16. for seed in [886u64, 887, 888] { let case = build_geglu_case(seed, 2816, 704, 16, 8); - let fused = flatten_f32(&run_fused(&case)); - let dense_raw = reference_dense_f32(&case); - let dense = flatten_f32(&dense_raw); - // The fused output has been rounded to bf16 once; apply the identical - // rounding to the dense reference before comparing so the assertion - // measures kernel math, not the unavoidable output-dtype quantum. - let dense_bf16 = flatten_f32(&astype(&dense_raw, dtype::BFLOAT16)); - let gather = flatten_f32(&reference_gather_qmm(&case)); - - let (nrms_dense, nmax_dense) = normalized_deviation(&fused, &dense_bf16); - let (nrms_gather, nmax_gather) = normalized_deviation(&fused, &gather); - let (nrms_ref_jitter, nmax_ref_jitter) = normalized_deviation(&gather, &dense); - println!( - "fused_moe_geglu parity [{backend}, seed {seed}]: vs dense f32 (bf16-rounded) \ - nrms={nrms_dense:e} nmax={nmax_dense:e}; vs gather_qmm nrms={nrms_gather:e} \ - nmax={nmax_gather:e}; gather-vs-dense baseline nrms={nrms_ref_jitter:e} \ - nmax={nmax_ref_jitter:e}" - ); - assert!( - nrms_dense < 5e-4 && nmax_dense < 2e-2, - "fused vs bf16-rounded dense f32 reference deviates (seed {seed}): \ - nrms={nrms_dense:e} nmax={nmax_dense:e}" - ); - // The production fallback held to the same ground truth as the kernel. - assert!( - nrms_ref_jitter < GATHER_JITTER_NRMS_CEILING - && nmax_ref_jitter < GATHER_JITTER_NMAX_CEILING, - "gather_qmm fallback deviates from the dense f32 reference (seed {seed}): \ - nrms={nrms_ref_jitter:e} nmax={nmax_ref_jitter:e}" - ); - // Cross-path agreement. The fused kernel matches the dense reference - // to `nrms_dense` above, which pins its distance from `gather_qmm` to - // `gather_qmm`'s own bf16 jitter; a fixed constant here measures that - // jitter rather than either path. #964: the previous 5e-3 nrms bound - // sat below the ~1.16e-2 baseline the test already computed and was - // unsatisfiable at every seed and both shapes. Scale with the jitter - // this run measured, never tighter than the project's 5e-3 / 5e-2 - // fp16-parity allowance (which binds on a backend whose fallback is - // closer to exact than Metal's). - let nrms_gather_limit = (GATHER_JITTER_NRMS_MARGIN * nrms_ref_jitter).max(5e-3); - let nmax_gather_limit = (GATHER_JITTER_NMAX_MARGIN * nmax_ref_jitter).max(5e-2); - assert!( - nrms_gather < nrms_gather_limit && nmax_gather < nmax_gather_limit, - "fused and gather_qmm disagree beyond the fallback's own jitter (seed {seed}): \ - nrms={nrms_gather:e} (limit {nrms_gather_limit:e}) \ - nmax={nmax_gather:e} (limit {nmax_gather_limit:e}); \ - baseline nrms={nrms_ref_jitter:e} nmax={nmax_ref_jitter:e}" - ); + assert_fused_matches_references(&case, backend, "geglu 4/4-bit", seed); + } +} + +/// The parity assertions shared by every activation and bit-width case: the +/// fused kernel against the bf16-rounded all-f32 dense reference (the sharp +/// gate), the `gather_qmm` fallback against the same truth, and the two +/// production paths against each other within the fallback's own jitter. +fn assert_fused_matches_references(case: &GegluMoeCase, backend: &str, label: &str, seed: u64) { + let fused = flatten_f32(&run_fused(case)); + let dense_raw = reference_dense_f32(case); + let dense = flatten_f32(&dense_raw); + // The fused output has been rounded to bf16 once; apply the identical + // rounding to the dense reference before comparing so the assertion + // measures kernel math, not the unavoidable output-dtype quantum. + let dense_bf16 = flatten_f32(&astype(&dense_raw, dtype::BFLOAT16)); + let gather = flatten_f32(&reference_gather_qmm(case)); + + let (nrms_dense, nmax_dense) = normalized_deviation(&fused, &dense_bf16); + let (nrms_gather, nmax_gather) = normalized_deviation(&fused, &gather); + let (nrms_ref_jitter, nmax_ref_jitter) = normalized_deviation(&gather, &dense); + println!( + "fused_moe parity [{backend}, {label}, seed {seed}]: vs dense f32 (bf16-rounded) \ + nrms={nrms_dense:e} nmax={nmax_dense:e}; vs gather_qmm nrms={nrms_gather:e} \ + nmax={nmax_gather:e}; gather-vs-dense baseline nrms={nrms_ref_jitter:e} \ + nmax={nmax_ref_jitter:e}" + ); + assert!( + nrms_dense < 5e-4 && nmax_dense < 2e-2, + "fused vs bf16-rounded dense f32 reference deviates ({label}, seed {seed}): \ + nrms={nrms_dense:e} nmax={nmax_dense:e}" + ); + // The production fallback held to the same ground truth as the kernel. + assert!( + nrms_ref_jitter < GATHER_JITTER_NRMS_CEILING + && nmax_ref_jitter < GATHER_JITTER_NMAX_CEILING, + "gather_qmm fallback deviates from the dense f32 reference ({label}, seed {seed}): \ + nrms={nrms_ref_jitter:e} nmax={nmax_ref_jitter:e}" + ); + // Cross-path agreement. The fused kernel matches the dense reference + // to `nrms_dense` above, which pins its distance from `gather_qmm` to + // `gather_qmm`'s own bf16 jitter; a fixed constant here measures that + // jitter rather than either path. #964: the previous 5e-3 nrms bound + // sat below the ~1.16e-2 baseline the test already computed and was + // unsatisfiable at every seed and both shapes. Scale with the jitter + // this run measured, never tighter than the project's 5e-3 / 5e-2 + // fp16-parity allowance (which binds on a backend whose fallback is + // closer to exact than Metal's). + let nrms_gather_limit = (GATHER_JITTER_NRMS_MARGIN * nrms_ref_jitter).max(5e-3); + let nmax_gather_limit = (GATHER_JITTER_NMAX_MARGIN * nmax_ref_jitter).max(5e-2); + assert!( + nrms_gather < nrms_gather_limit && nmax_gather < nmax_gather_limit, + "fused and gather_qmm disagree beyond the fallback's own jitter ({label}, seed {seed}): \ + nrms={nrms_gather:e} (limit {nrms_gather_limit:e}) \ + nmax={nmax_gather:e} (limit {nmax_gather_limit:e}); \ + baseline nrms={nrms_ref_jitter:e} nmax={nmax_ref_jitter:e}" + ); +} + +/// SwitchGLU decode-shape parity for the SwiGLU arm of the gate-up kernel and +/// every down-kernel bit width, which the Gemma4 GeGLU case above does not +/// reach: 4-bit throughout, 8-bit throughout, and 4-bit gate/up with a 6-bit +/// down (dots.llm1's mix, the down kernel's byte-packed branch). Din, Dff and +/// K are Qwen3-30B-A3B's (2048, 768, 8); E is cut to 16 as above. +#[test] +fn fused_moe_swiglu_kernel_matches_references_every_down_width() { + let Some(backend) = gpu_backend_or_skip() else { + return; + }; + for (gu_bits, d_bits) in [(4, 4), (8, 8), (4, 6)] { + let label = format!("swiglu {gu_bits}/{d_bits}-bit"); + for seed in [30u64, 31] { + let case = build_moe_case(seed, 2048, 768, 16, 8, MoeAct::SwiGlu, gu_bits, d_bits); + assert_fused_matches_references(&case, backend, &label, seed); + } } } @@ -457,14 +546,48 @@ fn fused_moe_kernel_source_structure() { src.contains("for (int o = 16; o > 0; o >>= 1)"), "warp reduction ladder must cover offsets 16..1" ); - // The down kernels (CUDA and Metal) must write f32 partials, not round - // per-expert outputs to the activation dtype before the host K-sum. + // The down kernels (Metal, CUDA and HIP) must write f32 partials, not + // round per-expert outputs to the activation dtype before the host K-sum. assert_eq!( src.matches("out[eslot * Din + h] = (float)scores[eslot] * d;") .count(), - 2, - "both MOE_DOWN kernel twins must emit f32 partials (score in f32)" + 3, + "all three MOE_DOWN kernel ports must emit f32 partials (score in f32)" ); + // Each fused-MoE HIP source refuses to compile for a wave64 target, under + // both spellings of the wavefront macro, and every shuffle in it states a + // width of 32: the native `__shfl_down` defaults to the device wavefront, + // so an implied width would fold 64 lanes on a wave64 part while the + // ladder above covers only 32 (#2065). + for name in ["MOE_GATEUP_HIP_SOURCE", "MOE_DOWN_HIP_SOURCE"] { + let start = src + .find(&format!("{name} = R\"(")) + .unwrap_or_else(|| panic!("{name} must exist")); + let body = &src[start..start + src[start..].find(")\";").expect("raw string end")]; + for guard in [ + "#if defined(__AMDGCN_WAVEFRONT_SIZE__) && __AMDGCN_WAVEFRONT_SIZE__ != 32", + "#if defined(__AMDGCN_WAVEFRONT_SIZE) && __AMDGCN_WAVEFRONT_SIZE != 32", + ] { + assert!( + body.contains(guard), + "{name} must carry the wave32 guard `{guard}`" + ); + } + assert!( + !body.contains("__shfl_down_sync("), + "{name} must use the native __shfl_down, not the mask-ignoring shim" + ); + let mut shuffles = 0usize; + for (off, _) in body.match_indices("__shfl_down(") { + shuffles += 1; + let call = &body[off..off + body[off..].find(')').expect("unterminated call")]; + assert!( + call.ends_with(", 32"), + "{name}: __shfl_down without an explicit width of 32: {call})" + ); + } + assert!(shuffles >= 1, "{name} must reduce with __shfl_down"); + } assert!( !src.contains("out[eslot * Din + h] = (T)("), "MOE_DOWN must not round per-expert partials to the activation dtype" diff --git a/src/lib/mlxcel-core/src/lib.rs b/src/lib/mlxcel-core/src/lib.rs index edd3a8b1f..06f3d39e1 100644 --- a/src/lib/mlxcel-core/src/lib.rs +++ b/src/lib/mlxcel-core/src/lib.rs @@ -2348,6 +2348,20 @@ mod ffi { /// its first port. fn bitlinear_kernel_available() -> bool; + /// True when this backend has both fused decode-MoE kernel ports + /// (gate-up and down), that is Metal, CUDA or ROCm (issues #1803, + /// #2065). Read from the kernels' own port tables, so the + /// `MLXCEL_FUSED_MOE` gate and the launchers behind it + /// ([`fused_moe_expert_kernel`], [`fused_moe_geglu_kernel`]) cannot + /// disagree. + fn fused_moe_kernels_available() -> bool; + + /// True when this backend has the fused decode-MoE down kernel port + /// alone, which Nemotron-H's [`fused_moe_forward`] reuses for its fc2 + /// (issue #2065). Its opt-in `MLXCEL_FUSED_MOE_RELU2` branch also needs + /// the fc1 squared-ReLU port and declines to `gather_qmm` without it. + fn moe_down_kernel_available() -> bool; + /// Whether `quantized_matmul` for this transposed affine projection /// runs the same dense GEMM as `dequantize` + `matmul`, so the two /// return the same bytes (lablup/mlxcel#2081). Reads the diff --git a/src/models/nemotron_h.rs b/src/models/nemotron_h.rs index bfe472fdb..e05b341ee 100644 --- a/src/models/nemotron_h.rs +++ b/src/models/nemotron_h.rs @@ -1346,17 +1346,18 @@ impl NemotronHMoE { mlxcel_core::copy(x) }; - // `custom_kernels_available()` because the fused MoE kernels are Metal - // and CUDA only, and nothing else on this path checks for a port: - // `fused_moe_forward` has no `metal::is_available()` fallback of its own, - // unlike `fused_xielu`. So on a backend without a port the launcher - // refuses and this selects `forward_nonfused` instead, which is what the - // non-quantized arm below already does (#1801). - // - // Added defensively rather than in response to an observed abort: no - // Nemotron-H checkpoint was available to run on the ROCm host, so the - // path was traced rather than executed. - let use_fused = !self.has_latent_proj() && mlxcel_core::custom_kernels_available(); + // `moe_down_kernel_available()` reads the down kernel's own port table, + // the one fused MoE kernel `fused_moe_forward` can reach (its opt-in + // `MLXCEL_FUSED_MOE_RELU2` branch reuses it for fc2). Metal, CUDA and, + // since issue #2065, ROCm. The opt-in branch also needs the fc1 + // squared-ReLU port and checks for both itself, declining to its + // `gather_qmm` branch without them; the default branch launches no + // custom kernel today. The term stays rather than being dropped for + // that reason, so that a kernel added to the default branch later + // cannot silently widen this gate to a backend without its port; on + // such a backend this selects `forward_nonfused`, as the non-quantized + // arm below already does (#1801). + let use_fused = !self.has_latent_proj() && mlxcel_core::moe_down_kernel_available(); // Try fused MoE forward (quantized path only, no latent projection) let result = if use_fused { diff --git a/src/models/qwen3_moe_tests.rs b/src/models/qwen3_moe_tests.rs index e9124f735..049125325 100644 --- a/src/models/qwen3_moe_tests.rs +++ b/src/models/qwen3_moe_tests.rs @@ -351,7 +351,7 @@ fn fused_geometry_block(dff: i32) -> SparseMoeBlock { /// guard passes, so the decline above is the bound and nothing else. It /// launches the kernel, which on a backend without a port throws across an /// infallible bridge and ends the process (issue #1803), so it runs only when -/// `custom_kernels_available()` says a port exists. +/// `fused_moe_kernels_available()` says both ports exist. #[test] fn qwen3_moe_fused_kernel_declines_experts_wider_than_the_dff_bound() { if std::env::var_os("MLXCEL_FUSED_MOE_MAX_DFF").is_some() { @@ -388,7 +388,7 @@ fn qwen3_moe_fused_kernel_declines_experts_wider_than_the_dff_bound() { "a single token over too-wide experts must dispatch to gather_qmm" ); - if !mlxcel_core::custom_kernels_available() { + if !mlxcel_core::fused_moe_kernels_available() { eprintln!( "skipping the positive control: this backend has no fused MoE kernel port (#1803)" ); diff --git a/src/models/qwen3_vl_moe_tests.rs b/src/models/qwen3_vl_moe_tests.rs index c1b24a1df..502712205 100644 --- a/src/models/qwen3_vl_moe_tests.rs +++ b/src/models/qwen3_vl_moe_tests.rs @@ -147,7 +147,7 @@ fn qwen3_vl_moe_switch_linear_rejects_quantization_params_that_would_abort_gathe /// must now decline the fused kernel, while Dff 64 on the same geometry must /// still dispatch it. The positive half launches the kernel, which aborts on a /// backend without a port (issue #1803), so it is gated on -/// `custom_kernels_available()`. +/// `fused_moe_kernels_available()`. #[test] fn qwen3_vl_moe_fused_kernel_declines_experts_wider_than_the_dff_bound() { if std::env::var_os("MLXCEL_FUSED_MOE_MAX_DFF").is_some() { @@ -189,7 +189,7 @@ fn qwen3_vl_moe_fused_kernel_declines_experts_wider_than_the_dff_bound() { "Dff 8256 is above both default bounds, so the fused kernel must decline" ); - if !mlxcel_core::custom_kernels_available() { + if !mlxcel_core::fused_moe_kernels_available() { eprintln!( "skipping the positive control: this backend has no fused MoE kernel port (#1803)" ); diff --git a/src/models/switch_layers.rs b/src/models/switch_layers.rs index 130ec86b1..12ca48fcc 100644 --- a/src/models/switch_layers.rs +++ b/src/models/switch_layers.rs @@ -91,16 +91,19 @@ pub fn fused_moe_enabled() -> bool { static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); *ENABLED .get_or_init(|| fused_moe_enabled_from(std::env::var("MLXCEL_FUSED_MOE").ok().as_deref())) - // The fused MoE launcher has Metal and CUDA ports only. This term was - // load-bearing for safety under issue #1803, when the bridge function - // did not return `Result` and the throw on a portless backend ended the - // process; issue #1885 made it `Result`, and `forward_fused_kernel` now - // turns that refusal into the `None` it already returns for any - // unsupported config. The term stays because deciding here is cheaper - // and clearer than building the flattened arguments only to have the - // launcher refuse, and it selects the same SwitchGLU path + // Reads the fused MoE kernels' own port tables (gate-up and down), so + // this gate opens exactly where the launcher has both ports: Metal, + // CUDA, and ROCm since issue #2065. It was a backend-wide + // `custom_kernels_available()` before, which kept a ROCm port + // unreachable. Load-bearing for safety under issue #1803, when the + // bridge function did not return `Result`; issue #1885 made it + // `Result`, and `forward_fused_kernel` turns a refusal into the `None` + // it already returns for any unsupported config. The term stays + // because deciding here is cheaper and clearer than building the + // flattened arguments only to have the launcher refuse, and on a + // backend without the ports it selects the same SwitchGLU path // `MLXCEL_FUSED_MOE=0` selects. - && mlxcel_core::custom_kernels_available() + && mlxcel_core::fused_moe_kernels_available() } /// Pure decision behind [`fused_moe_enabled`], split out so it can be unit-tested From 721ff14a8aaffa9c9dccafb3c95d0bf47d46dcb0 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Sun, 4 Oct 2026 19:12:54 +0900 Subject: [PATCH 2/8] docs(rocm): state what the fused MoE wave32 guard actually covers AMD clang 23 (HIP 7.15) defines neither __AMDGCN_WAVEFRONT_SIZE spelling for gfx942 or gfx1151 (checked with hipcc -E -dM), so the #1814 preprocessor guard never fires on this toolchain. The explicit shuffle width of 32 is what keeps each row's fold inside its own 32 lanes on a wave64 part. Correct the kernel and test comments and the MLXCEL_FUSED_MOE doc, which claimed a wave64 target fails to compile, and say wave64 is untested. Also list ROCm under the MLXCEL_FUSED_MOE_MAX_DFF default and refresh the backend-wide "no ROCm ports yet" comments now that per-kernel predicates open ported kernels on ROCm. Refs #2065 --- docs/environment-variables.md | 4 ++-- src/lib/mlx-cpp/turbo/gpu_backend.h | 7 ++++--- src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp | 14 +++++++++----- src/lib/mlxcel-core/src/fused_moe_parity_tests.rs | 11 ++++++----- src/lib/mlxcel-core/src/lib.rs | 8 +++++--- 5 files changed, 26 insertions(+), 18 deletions(-) diff --git a/docs/environment-variables.md b/docs/environment-variables.md index dc6e2ab8f..919489552 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -535,9 +535,9 @@ recommended as normal deployment settings. | `MLXCEL_SDPA_FALLBACK_MAX_QUERIES` | non-negative integer | `32` | **CUDA only.** A masked SDPA call with 2 to N query rows over a longer key sequence (the speculative verify shape: a block appended to a KV cache) bypasses cuDNN and takes MLX's own ops fallback (issue #1799). cuDNN caches its execution plan by the exact shapes, and a verify round's key length changes every round, so with cuDNN every such layer class rebuilt a plan on the host each round: about 22 ms per build on GB10, 67 to 76 ms per round on the Laguna DFlash pairing, the whole fixed floor of that round, and the plan cache's lifetime miss counter then aborted the process. The one-row decode step takes the vector kernel and is unaffected; prefill keeps cuDNN (its key length equals its query length, or it has more rows than the bound), except the trailing short chunk of a chunked prefill and a short incremental prefill over a reused prefix-cache prefix, which take the fallback. The fallback's cost grows with the key length (a `[B, heads, q_len, k_len]` score matrix, about 10 ms more per round than cuDNN at block 2 with 350 keys) while the plan build it replaces is constant, so on very long contexts (tens of thousands of keys at block 16, unmeasured) cuDNN could be the cheaper side again. Issue #1820 left this gate's shipped behaviour unchanged: bucketing is off by default, so this gate still claims every array-masked verify block as it did before. It is narrowed only when `MLXCEL_SDPA_PLAN_BUCKET_MAX_QUERIES` is set non-zero, in which case it stops claiming calls whose plan-cache key can be bucketed and covers only a causal-mode block with no array mask, where there is no mask to widen. `0` restores upstream dispatch without a rebuild, which is the kill switch used for the A/B in `docs/benchmark_results/laguna-dflash-verify-cost-gb10-2026-09-11.md`. | | `MLXCEL_PIPELINE_GRANULARITY` | `off`, `layer`, `block:N` | `off` | Inserts layer-boundary async-eval hints for pipeline experiments. | | `MLXCEL_FUSED_ADD_NORM` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fuses a parallel-residual block's `(attn + mlp) + x` with the LayerNorm that consumes it (the next block's input norm, or the final norm) into one Metal kernel, saving one dispatch and one barrier level per layer during decode. The kernel copies MLX's single-row `layer_norm` reduction, so its outputs are byte-identical to the unfused add and norm (pinned by `residual_add3_layer_norm_matches_the_unfused_pair`); it falls back to the unfused pair off Metal, above a 6656-wide row, or on mixed dtypes. Used by Cohere2: command-r7b 4-bit decode on M1 Ultra +1.1% (eight ABBA pairs, 112.45 to 113.72 tok/s). | -| `MLXCEL_FUSED_MOE` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fused single-token decode-MoE kernel (#268), on by default since #282 (Metal) and #319 (CUDA, via `mx.fast.cuda_kernel`); validated on M1 Ultra, M5, and GB10. Set to `0` to force the proven `gather_qmm`/`SwitchGLU` path. Active for afmoe, bailing_moe, cohere2_moe, dbrx, dots.llm1, gemma4, klear, laguna, lfm2, mellum, minimax, mixtral, olmoe, phimoe, qwen2_moe, qwen3_moe, qwen3_next (and Qwen3.5), qwen3_vl_moe, and the qwen3_omni_moe thinker and talker decode. Byte-identical greedy output is checkpoint- and prompt-dependent and was never a general property (#1045): it held on `qwen3-30b-a3b` for the #1045 prompt but not for every prompt (on GB10 one diverges at generated token 39, #1884), and not on Klear. This is not a defect, since the kernel measures roughly 6x closer to an all-f32 ground truth than `gather_qmm` on both, but `gather_qmm` is what mlx-lm mirrors, so set this to `0` when reference-diffing a new MoE port. On the experimental ROCm build the kernel runs through `mx.fast.hip_kernel` since lablup/mlxcel#2065 on wave32 targets such as gfx1151 (a wave64 target fails to compile it rather than computing a wrong result), with the CUDA default for `MLXCEL_FUSED_MOE_MAX_DFF`. | +| `MLXCEL_FUSED_MOE` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fused single-token decode-MoE kernel (#268), on by default since #282 (Metal) and #319 (CUDA, via `mx.fast.cuda_kernel`); validated on M1 Ultra, M5, and GB10. Set to `0` to force the proven `gather_qmm`/`SwitchGLU` path. Active for afmoe, bailing_moe, cohere2_moe, dbrx, dots.llm1, gemma4, klear, laguna, lfm2, mellum, minimax, mixtral, olmoe, phimoe, qwen2_moe, qwen3_moe, qwen3_next (and Qwen3.5), qwen3_vl_moe, and the qwen3_omni_moe thinker and talker decode. Byte-identical greedy output is checkpoint- and prompt-dependent and was never a general property (#1045): it held on `qwen3-30b-a3b` for the #1045 prompt but not for every prompt (on GB10 one diverges at generated token 39, #1884), and not on Klear. This is not a defect, since the kernel measures roughly 6x closer to an all-f32 ground truth than `gather_qmm` on both, but `gather_qmm` is what mlx-lm mirrors, so set this to `0` when reference-diffing a new MoE port. On the experimental ROCm build the kernel runs through `mx.fast.hip_kernel` since lablup/mlxcel#2065, with the CUDA default for `MLXCEL_FUSED_MOE_MAX_DFF`. It is verified on gfx1151 (wave32) only; on a wave64 part (CDNA) it is untested, and `0` here keeps such a host on `gather_qmm`. | | `MLXCEL_FUSED_MOE_SGY` | `1`-`32` | `8` | Simdgroups (Metal) / warps (CUDA) / wavefronts (ROCm) per threadgroup for the fused decode-MoE kernel; tune per hardware. | -| `MLXCEL_FUSED_MOE_MAX_DFF` | positive int | `4096` (Metal) / `8192` (CUDA) | Expert-intermediate (Dff) upper bound for the fused path; above it the caller falls back to `gather_qmm`. The fused path wins only while `gather_qmm` underutilizes the GPU (small experts), so the break-even is backend-dependent and the default is chosen from the live backend: `4096` on Metal (M1 Ultra tuning) and `8192` on CUDA (GB10 re-measured under MLX pin e9463bb, #626; fused wins through Dff 6400 and is break-even at 8192). An explicit value overrides the default on both backends: lower it to force `gather_qmm` sooner, raise it (e.g. `20000`) to force the fused kernel on larger experts such as mixtral (Dff 14336, where it is a slight net loss). Read by every family on the shared `SwitchGLU` fused path, qwen3_moe and qwen3_vl_moe (and the qwen3_omni_moe thinker through it) included since #1884; qwen3_next (and Qwen3.5 and the qwen3_omni_moe talker through it) and gemma4 dispatch their own kernel and do not read it yet. | +| `MLXCEL_FUSED_MOE_MAX_DFF` | positive int | `4096` (Metal) / `8192` (CUDA, ROCm) | Expert-intermediate (Dff) upper bound for the fused path; above it the caller falls back to `gather_qmm`. The fused path wins only while `gather_qmm` underutilizes the GPU (small experts), so the break-even is backend-dependent and the default is chosen from the live backend: `4096` on Metal (M1 Ultra tuning) and `8192` on CUDA (GB10 re-measured under MLX pin e9463bb, #626; fused wins through Dff 6400 and is break-even at 8192). An explicit value overrides the default on every backend: lower it to force `gather_qmm` sooner, raise it (e.g. `20000`) to force the fused kernel on larger experts such as mixtral (Dff 14336, where it is a slight net loss). Read by every family on the shared `SwitchGLU` fused path, qwen3_moe and qwen3_vl_moe (and the qwen3_omni_moe thinker through it) included since #1884; qwen3_next (and Qwen3.5 and the qwen3_omni_moe talker through it) and gemma4 dispatch their own kernel and do not read it yet. | | `MLXCEL_FUSED_MOE_RELU2` | presence enables | off | Enables the squared-ReLU fused routed-expert path for nemotron-class experts. It is performance-neutral on nemotron-h even though MoE is its largest block because it replaces only the already-efficient routed fc1/fc2 GEMVs; the router, always-on shared expert, and combine remain outside this kernel. Kept for a future squared-ReLU model whose routed-expert slice dominates. The fc1 kernel has a Metal port only, so on CUDA and ROCm the flag leaves the default `gather_qmm` routed path in place (lablup/mlxcel#2065; ROCm port tracked in lablup/mlxcel#2069). | | `MLXCEL_FUSED_MOE_PARITY_CHECK` | unset or `0` disables; nonzero integer enables | off | In-situ per-call parity/determinism probe for the fused decode-MoE kernel, added for the #886 corruption triage. Re-runs the fused kernel pair on identical inputs (bitwise-determinism check), computes the `gather_qmm` production-fallback reference, and computes an all-f32 dequantize-and-matmul ground truth, then logs to stderr any call that is non-deterministic or whose fused-vs-reference, fused-vs-ground-truth, or reference-vs-ground-truth deviation exceeds `MLXCEL_FUSED_MOE_PARITY_THRESHOLD` (normalized RMS). Heavy: adds several eager evals per MoE call. Diagnostic only, kept for future corruption triage; never enable in production. | | `MLXCEL_FUSED_MOE_PARITY_THRESHOLD` | float | `0.05` | Normalized-RMS deviation threshold for the `MLXCEL_FUSED_MOE_PARITY_CHECK` probe. | diff --git a/src/lib/mlx-cpp/turbo/gpu_backend.h b/src/lib/mlx-cpp/turbo/gpu_backend.h index 7e401c8d5..5995aa322 100644 --- a/src/lib/mlx-cpp/turbo/gpu_backend.h +++ b/src/lib/mlx-cpp/turbo/gpu_backend.h @@ -38,9 +38,10 @@ enum class GpuKernelBackend { None = 0, Metal = 1, Cuda = 2, - // ROCm is a real GPU backend with no `fast::hip_kernel` ports yet (issue - // #1814). Callers treat it like `None` and take the graph fallback; they must - // not call `fast::cuda_kernel`. + // ROCm is a real GPU backend with `fast::hip_kernel` ports for some kernels + // only (issue #1814). Backend-wide checks treat it like `None`; a ported + // kernel's own `has_kernel_port` predicate opens it there. Callers must not + // call `fast::cuda_kernel`. Rocm = 3, }; diff --git a/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp b/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp index 8a272dfbb..9188d1693 100644 --- a/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp +++ b/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp @@ -2484,11 +2484,15 @@ namespace { // Precise `expf`/`tanhf` as in CUDA; hipRTC compiles at -O3 without // fast-math, so they stay the precise library calls. static const char* MOE_GATEUP_HIP_SOURCE = R"( - // The fold below starts at 16, which is correct only for a 32-lane - // wavefront; on a wave64 target (CDNA: gfx90a, gfx942) it would drop - // half the lanes and still return a finite, plausible, wrong result. - // Preprocessor checks, not `static_assert(warpSize == 32)`, which does - // not compile in HIP (see the bitlinear HIP source). + // The fold below starts at 16 and covers one 32-lane row. Every + // shuffle states width 32, so on a wave64 target (CDNA: gfx90a, + // gfx942) it stays inside its own 32-lane half instead of folding + // half the row. The preprocessor guard is the #1814 port rule, not + // `static_assert(warpSize == 32)`, which does not compile in HIP. It + // is inert on AMD clang 23 (HIP 7.15), which defines neither macro + // for gfx942 or gfx1151 (checked with `hipcc -E -dM`, #2065), so the + // explicit width is what keeps the fold correct there; wave64 is not + // tested. #if defined(__AMDGCN_WAVEFRONT_SIZE__) && __AMDGCN_WAVEFRONT_SIZE__ != 32 #error "moe_gateup_kernel_hip assumes a 32-lane wavefront" #endif diff --git a/src/lib/mlxcel-core/src/fused_moe_parity_tests.rs b/src/lib/mlxcel-core/src/fused_moe_parity_tests.rs index 3b58bc138..0452530c7 100644 --- a/src/lib/mlxcel-core/src/fused_moe_parity_tests.rs +++ b/src/lib/mlxcel-core/src/fused_moe_parity_tests.rs @@ -554,11 +554,12 @@ fn fused_moe_kernel_source_structure() { 3, "all three MOE_DOWN kernel ports must emit f32 partials (score in f32)" ); - // Each fused-MoE HIP source refuses to compile for a wave64 target, under - // both spellings of the wavefront macro, and every shuffle in it states a - // width of 32: the native `__shfl_down` defaults to the device wavefront, - // so an implied width would fold 64 lanes on a wave64 part while the - // ladder above covers only 32 (#2065). + // Each fused-MoE HIP source carries the #1814 wave32 guard under both + // spellings of the wavefront macro, and every shuffle in it states a width + // of 32: the native `__shfl_down` defaults to the device wavefront, so an + // implied width would fold 64 lanes on a wave64 part while the ladder + // above covers only 32. The width is the load-bearing half: AMD clang 23 + // defines neither macro, so the guard does not fire there (#2065). for name in ["MOE_GATEUP_HIP_SOURCE", "MOE_DOWN_HIP_SOURCE"] { let start = src .find(&format!("{name} = R\"(")) diff --git a/src/lib/mlxcel-core/src/lib.rs b/src/lib/mlxcel-core/src/lib.rs index 06f3d39e1..bcc7b13c5 100644 --- a/src/lib/mlxcel-core/src/lib.rs +++ b/src/lib/mlxcel-core/src/lib.rs @@ -2316,9 +2316,11 @@ mod ffi { fn gpu_backend_available() -> bool; /// True when the resolved GPU backend has mlxcel's fused kernel ports, - /// that is Metal or CUDA (issue #1803). ROCm has a GPU but no ports - /// yet, so it answers `false` and callers take the MLX graph fallback - /// their family already has. This is deliberately narrower than + /// that is Metal or CUDA (issue #1803). ROCm has a GPU and HIP ports + /// for some kernels only (#1814), so it answers `false`; a ported + /// kernel's own predicate (for example `bitlinear_kernel_available`, + /// `fused_moe_kernels_available`) reads its port table and is what + /// gates it there. This is deliberately narrower than /// `gpu_backend_available`, which only says a GPU exists: conflating /// the two is what sent ROCm into `fast::cuda_kernel` and aborted the /// process. From 4ab91f82497477fad40443fe63080df491347dda Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Sun, 4 Oct 2026 19:47:46 +0900 Subject: [PATCH 3/8] fix(bench): stop rocm_gpu_guard rejecting the command's own exited children The kernel removes a process's /sys/class/kfd/kfd/proc entry from a deferred work item, so the entry outlives the reaped process for a moment. foreign_holders() then could not read the pid's parentage and counted it as a foreign GPU tenant, so the sample taken just after bench_decode.sh's rocminfo probe or the bench binary exited rejected the attempt. On gfx1151 most attempts of a decode run were rejected this way, in this unit's runs and in the parallel #2067 unit's. A holder with no /proc entry has exited and is not using the GPU, so it is skipped. A foreign process that ran during the attempt is still caught by the samples taken while it was alive. New test test_a_kfd_entry_left_by_an_exited_process_is_not_contention fails on the old guard (exit 75, CONTENDED) and passes now; the other guard tests pass. Refs #2065 --- scripts/rocm_gpu_guard.sh | 9 +++++++++ tests/test_rocm_decode_profile.py | 11 +++++++++++ 2 files changed, 20 insertions(+) diff --git a/scripts/rocm_gpu_guard.sh b/scripts/rocm_gpu_guard.sh index 03b2a196c..e3b2d4a18 100755 --- a/scripts/rocm_gpu_guard.sh +++ b/scripts/rocm_gpu_guard.sh @@ -123,11 +123,20 @@ descends_from() { } # Holders in $2 (a kfd_holders reading) that are not $1 or its descendants. +# +# A holder whose /proc entry is gone has already exited and been reaped: the +# kernel tears a process's /sys/class/kfd/kfd/proc entry down from a deferred +# work item, so it outlives the process by a moment. Its parentage can no +# longer be read, and it is not using the GPU, so it is not contention. Without +# this, the command's own children (bench_decode.sh's rocminfo probe, the bench +# binary itself) read as foreign in the sample taken just after they exit, and +# most attempts were rejected (#2065). foreign_holders() { local root="$1" entry pid out=() entries IFS=, read -r -a entries <<<"$2" for entry in ${entries[@]+"${entries[@]}"}; do pid="${entry%%:*}" + [[ -e "/proc/$pid" ]] || continue descends_from "$pid" "$root" || out+=("$entry") done local IFS=, diff --git a/tests/test_rocm_decode_profile.py b/tests/test_rocm_decode_profile.py index 00adba794..11aae44a4 100644 --- a/tests/test_rocm_decode_profile.py +++ b/tests/test_rocm_decode_profile.py @@ -79,6 +79,17 @@ def test_the_commands_own_gpu_process_is_not_contention(self): self.assertEqual(r.returncode, 0, r.stderr) self.assertNotIn("CONTENDED", r.stderr) + def test_a_kfd_entry_left_by_an_exited_process_is_not_contention(self): + with tempfile.TemporaryDirectory() as kfd: + # KFD removes a process's proc entry after the process is reaped, so + # for a moment the entry names a pid with no /proc directory. That + # is the command's own child finishing, not another GPU tenant. + cmd = f"true & p=$!; wait $p; mkdir -p {kfd}/$p; sleep 2.5; rmdir {kfd}/$p" + r = run_guard(kfd, "--idle-secs", "1", "--max-attempts", "1", "--", + "bash", "-c", cmd) + self.assertEqual(r.returncode, 0, r.stderr) + self.assertNotIn("CONTENDED", r.stderr) + def test_a_busy_gpu_before_the_run_times_out_without_running(self): with tempfile.TemporaryDirectory() as kfd, tempfile.TemporaryDirectory() as out: os.mkdir(pathlib.Path(kfd) / "1") From 359f67ed9dff1750a1c71a045c303719182a83a4 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Sun, 4 Oct 2026 20:52:02 +0900 Subject: [PATCH 4/8] update(rocm): default the fused MoE SGY to 2 on ROCm With the Metal-tuned default of 8 rows per threadgroup, the HIP port made Qwen3-30B-A3B-4bit decode on gfx1151 slower than the gather_qmm path it replaces (about 59.9 tok/s against about 61). A guarded sweep of MLXCEL_FUSED_MOE_SGY over 1, 2, 4, 8, 16 and 32 put 2 on top at about 62.3 tok/s, with 1 at about 61.2 and the rest at or below gather_qmm. SGY only shapes the threadgroup, so the output is unchanged (fused_moe_geglu_kernel_bitwise_invariant_across_sgy). Metal and CUDA keep 8; the environment variable still overrides on every backend. Refs #2065 --- docs/environment-variables.md | 2 +- src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/environment-variables.md b/docs/environment-variables.md index 919489552..0bbf87c1b 100644 --- a/docs/environment-variables.md +++ b/docs/environment-variables.md @@ -536,7 +536,7 @@ recommended as normal deployment settings. | `MLXCEL_PIPELINE_GRANULARITY` | `off`, `layer`, `block:N` | `off` | Inserts layer-boundary async-eval hints for pipeline experiments. | | `MLXCEL_FUSED_ADD_NORM` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fuses a parallel-residual block's `(attn + mlp) + x` with the LayerNorm that consumes it (the next block's input norm, or the final norm) into one Metal kernel, saving one dispatch and one barrier level per layer during decode. The kernel copies MLX's single-row `layer_norm` reduction, so its outputs are byte-identical to the unfused add and norm (pinned by `residual_add3_layer_norm_matches_the_unfused_pair`); it falls back to the unfused pair off Metal, above a 6656-wide row, or on mixed dtypes. Used by Cohere2: command-r7b 4-bit decode on M1 Ultra +1.1% (eight ABBA pairs, 112.45 to 113.72 tok/s). | | `MLXCEL_FUSED_MOE` | `0`/`false`/`off`/`no` disable; any other value or unset enables | on | Fused single-token decode-MoE kernel (#268), on by default since #282 (Metal) and #319 (CUDA, via `mx.fast.cuda_kernel`); validated on M1 Ultra, M5, and GB10. Set to `0` to force the proven `gather_qmm`/`SwitchGLU` path. Active for afmoe, bailing_moe, cohere2_moe, dbrx, dots.llm1, gemma4, klear, laguna, lfm2, mellum, minimax, mixtral, olmoe, phimoe, qwen2_moe, qwen3_moe, qwen3_next (and Qwen3.5), qwen3_vl_moe, and the qwen3_omni_moe thinker and talker decode. Byte-identical greedy output is checkpoint- and prompt-dependent and was never a general property (#1045): it held on `qwen3-30b-a3b` for the #1045 prompt but not for every prompt (on GB10 one diverges at generated token 39, #1884), and not on Klear. This is not a defect, since the kernel measures roughly 6x closer to an all-f32 ground truth than `gather_qmm` on both, but `gather_qmm` is what mlx-lm mirrors, so set this to `0` when reference-diffing a new MoE port. On the experimental ROCm build the kernel runs through `mx.fast.hip_kernel` since lablup/mlxcel#2065, with the CUDA default for `MLXCEL_FUSED_MOE_MAX_DFF`. It is verified on gfx1151 (wave32) only; on a wave64 part (CDNA) it is untested, and `0` here keeps such a host on `gather_qmm`. | -| `MLXCEL_FUSED_MOE_SGY` | `1`-`32` | `8` | Simdgroups (Metal) / warps (CUDA) / wavefronts (ROCm) per threadgroup for the fused decode-MoE kernel; tune per hardware. | +| `MLXCEL_FUSED_MOE_SGY` | `1`-`32` | `8` (Metal, CUDA) / `2` (ROCm) | Simdgroups (Metal) / warps (CUDA) / wavefronts (ROCm) per threadgroup for the fused decode-MoE kernel; tune per hardware. The output does not depend on it. On gfx1151 `8` made Qwen3-30B-A3B decode slower than `gather_qmm` and `2` faster (lablup/mlxcel#2065), hence the ROCm default. | | `MLXCEL_FUSED_MOE_MAX_DFF` | positive int | `4096` (Metal) / `8192` (CUDA, ROCm) | Expert-intermediate (Dff) upper bound for the fused path; above it the caller falls back to `gather_qmm`. The fused path wins only while `gather_qmm` underutilizes the GPU (small experts), so the break-even is backend-dependent and the default is chosen from the live backend: `4096` on Metal (M1 Ultra tuning) and `8192` on CUDA (GB10 re-measured under MLX pin e9463bb, #626; fused wins through Dff 6400 and is break-even at 8192). An explicit value overrides the default on every backend: lower it to force `gather_qmm` sooner, raise it (e.g. `20000`) to force the fused kernel on larger experts such as mixtral (Dff 14336, where it is a slight net loss). Read by every family on the shared `SwitchGLU` fused path, qwen3_moe and qwen3_vl_moe (and the qwen3_omni_moe thinker through it) included since #1884; qwen3_next (and Qwen3.5 and the qwen3_omni_moe talker through it) and gemma4 dispatch their own kernel and do not read it yet. | | `MLXCEL_FUSED_MOE_RELU2` | presence enables | off | Enables the squared-ReLU fused routed-expert path for nemotron-class experts. It is performance-neutral on nemotron-h even though MoE is its largest block because it replaces only the already-efficient routed fc1/fc2 GEMVs; the router, always-on shared expert, and combine remain outside this kernel. Kept for a future squared-ReLU model whose routed-expert slice dominates. The fc1 kernel has a Metal port only, so on CUDA and ROCm the flag leaves the default `gather_qmm` routed path in place (lablup/mlxcel#2065; ROCm port tracked in lablup/mlxcel#2069). | | `MLXCEL_FUSED_MOE_PARITY_CHECK` | unset or `0` disables; nonzero integer enables | off | In-situ per-call parity/determinism probe for the fused decode-MoE kernel, added for the #886 corruption triage. Re-runs the fused kernel pair on identical inputs (bitwise-determinism check), computes the `gather_qmm` production-fallback reference, and computes an all-f32 dequantize-and-matmul ground truth, then logs to stderr any call that is non-deterministic or whose fused-vs-reference, fused-vs-ground-truth, or reference-vs-ground-truth deviation exceeds `MLXCEL_FUSED_MOE_PARITY_THRESHOLD` (normalized RMS). Heavy: adds several eager evals per MoE call. Diagnostic only, kept for future corruption triage; never enable in production. | diff --git a/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp b/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp index 9188d1693..5f32f6814 100644 --- a/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp +++ b/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp @@ -2848,8 +2848,12 @@ std::unique_ptr run_fused_moe_two_kernel( // SGY = simdgroups per threadgroup (one per output row). 8 is the measured // sweet spot on qwen3-30b-a3b (M1 Ultra); env-overridable for other - // hardware. It feeds an MLX template arg, which JIT-specializes at runtime. - int sgy = 8; + // hardware. It sets the threadgroup shape, so it never changes the result + // (`fused_moe_geglu_kernel_bitwise_invariant_across_sgy`). On ROCm the + // default is 2: on gfx1151 qwen3-30b-a3b decode measured about 62.3 tok/s + // at 2 against about 59.9 at 8 and about 61 on gather_qmm (#2065), so 8 + // would make the port slower than the fallback it replaces. + int sgy = mlxcel::gpu_kernel_backend() == mlxcel::GpuKernelBackend::Rocm ? 2 : 8; if (const char* s = std::getenv("MLXCEL_FUSED_MOE_SGY")) { int v = std::atoi(s); if (v >= 1 && v <= 32) sgy = v; From 826dfa8ceaa4052899b0ed69e4de12f9b1c390c2 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Sun, 4 Oct 2026 22:24:21 +0900 Subject: [PATCH 5/8] fix(rocm): pick the fused MoE SGY default from the build flag make verify-kernel-port-dispatch rejects comparing gpu_kernel_backend() in a launcher file, because backend choice belongs to select_kernel_port. A ROCm build has no Metal or CUDA backend, so MLXCEL_BRIDGE_ROCM_BACKEND selects the ROCm default of 2 at compile time with the same effect. Refs #2065 --- src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp b/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp index 5f32f6814..ca844c8cf 100644 --- a/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp +++ b/src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp @@ -2852,8 +2852,15 @@ std::unique_ptr run_fused_moe_two_kernel( // (`fused_moe_geglu_kernel_bitwise_invariant_across_sgy`). On ROCm the // default is 2: on gfx1151 qwen3-30b-a3b decode measured about 62.3 tok/s // at 2 against about 59.9 at 8 and about 61 on gather_qmm (#2065), so 8 - // would make the port slower than the fallback it replaces. - int sgy = mlxcel::gpu_kernel_backend() == mlxcel::GpuKernelBackend::Rocm ? 2 : 8; + // would make the port slower than the fallback it replaces. A ROCm build + // has no Metal or CUDA backend, so the build flag is the backend here and + // no runtime backend comparison is needed (that is reserved for + // `select_kernel_port`). +#ifdef MLXCEL_BRIDGE_ROCM_BACKEND + int sgy = 2; +#else + int sgy = 8; +#endif if (const char* s = std::getenv("MLXCEL_FUSED_MOE_SGY")) { int v = std::atoi(s); if (v >= 1 && v <= 32) sgy = v; From 77afbb1d8320ec40bf5d97365df02c0fa2feced0 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Sun, 4 Oct 2026 23:02:38 +0900 Subject: [PATCH 6/8] docs(rocm): list the SSM update step among the HIP-ported kernels Rebased onto #2099, which ported ssm_update_kernel to HIP; the README sentence this branch rewrote named only BitLinear and the decode-MoE pair. Refs #2065 --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index e2494f535..e4c58aece 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ Apple Silicon is the primary target. Linux/CUDA is supported as a secondary targ The current `main` branch is v0.7.0 plus unreleased work. Install from source to use features that have not reached the latest tagged release; see the [changelog](CHANGELOG.md) for the release-by-release record. -- **Experimental AMD GPU (ROCm) support.** `--features rocm` builds mlxcel on Linux against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (`gfx1151`); the fused kernels with a HIP port (BitLinear, the decode-MoE pair) run natively and the rest fall back to MLX graphs. See [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental) for the build and the open gaps. +- **Experimental AMD GPU (ROCm) support.** `--features rocm` builds mlxcel on Linux against an MLX ROCm backend vendored into the source tree as the ROCm part of mlxcelverse, applied on the same pinned MLX commit as the Metal and CUDA builds. Affine, mxfp8, and mxfp4 checkpoints, including gpt-oss MoE experts, run on RDNA 3.5 (`gfx1151`); the fused kernels with a HIP port (BitLinear, the SSM update step, the decode-MoE pair) run natively and the rest fall back to MLX graphs. See [Linux with AMD ROCm](docs/installation.md#linux-with-amd-rocm-experimental) for the build and the open gaps. - **Verified `llama-server` compatibility.** A frozen b10621 manifest classifies all 376 pinned options, routes, and native request fields, with no deferred entries. Native completion, embedding, tokenization, template, infill, props, slots, metrics, resumable-stream, router, and LoRA surfaces are implemented or explicitly classified instead of being silently ignored. - **OpenAI, Anthropic, and Vertex-compatible serving.** Chat Completions, Completions, Responses, Embeddings, Reranking, Audio, and Anthropic Messages are available alongside the native `llama-server` routes. Optional Vertex AI custom-container routing is supported through the standard `AIP_*` variables. - **Multi-model routing and live adapters.** Router mode discovers checkpoints from the model store, a model directory, or INI presets; loads them on demand; and bounds the resident set with LRU eviction. Multiple LoRA adapters can remain unfused for per-request or live scale changes, or be fused for zero decode overhead. From eade24a1ea29d8af5f278e9f1c5b20b5ecde30cb Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 5 Oct 2026 02:26:49 +0900 Subject: [PATCH 7/8] docs(rocm): publish the fused MoE decode and logit results on gfx1151 Add docs/benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md with the guarded, alternated before/after decode runs (Qwen3-30B-A3B-4bit 61.29 to 62.51 tok/s, Nemotron-3-Nano 74.37 to 75.09, Mixtral unchanged path within noise), the raw rows under benchmarks/, and the logit traces under benchmarks/logit_traces/rocm_gfx1151_77afbb1d/ (zero decided-position mismatches against Metal and ROCm main, gather_qmm traces byte-identical). Link it from installation.md and the 2026-09-30 correctness page, and drop a pre-rebase commit hash from the #2067 page. Refs #2065 --- .../rocm_gfx1151_77afbb1d/METADATA.txt | 17 + .../rocm_gfx1151_77afbb1d/README.md | 20 + .../rocm_gfx1151_77afbb1d/RUNS.txt | 8 + .../rocm_gfx1151_77afbb1d/SHA256SUMS | 8 + ...fbb1d_mixtral-8x7b-instruct_default_w1.tsv | 135 ++++ ...b1d_nemotron-3-nano-30b-a3b_default_w1.tsv | 135 ++++ ...motron-3-nano-30b-a3b_default_w1ctx512.tsv | 135 ++++ ...b1d_nemotron-3-nano-30b-a3b_default_w8.tsv | 647 ++++++++++++++++++ ...1151_77afbb1d_qwen3-30b-a3b_default_w1.tsv | 135 ++++ ...1151_77afbb1d_qwen3-30b-a3b_default_w8.tsv | 647 ++++++++++++++++++ ...x1151_77afbb1d_qwen3-30b-a3b_fused0_w1.tsv | 135 ++++ ...x1151_77afbb1d_qwen3-30b-a3b_fused0_w8.tsv | 647 ++++++++++++++++++ ...alo-gfx1151_2026-10-05_fused-moe-after.csv | 9 + ...lo-gfx1151_2026-10-05_fused-moe-before.csv | 10 + .../rocm-correctness-gfx1151-2026-09-30.md | 1 + .../rocm-fused-moe-gfx1151-2026-10-05.md | 59 ++ ...cm-ssm-update-kernel-gfx1151-2026-10-04.md | 2 +- docs/installation.md | 2 +- 18 files changed, 2750 insertions(+), 2 deletions(-) create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/METADATA.txt create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/README.md create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/RUNS.txt create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/SHA256SUMS create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_mixtral-8x7b-instruct_default_w1.tsv create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1.tsv create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1ctx512.tsv create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w8.tsv create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w1.tsv create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w8.tsv create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w1.tsv create mode 100644 benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w8.tsv create mode 100644 benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-after.csv create mode 100644 benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-before.csv create mode 100644 docs/benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/METADATA.txt b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/METADATA.txt new file mode 100644 index 000000000..84819ed16 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/METADATA.txt @@ -0,0 +1,17 @@ +host: AMD Ryzen AI MAX+ 395 with Radeon 8060S (gfx1151, RDNA 3.5), 96 GiB VRAM carve-out, 31 GiB host RAM +os: Debian GNU/Linux 13 (trixie), kernel 6.18.12+deb13-amd64 +rocm: 10.0.0 packages (HIP 7.15.26333), ROCM_PATH=/opt/rocm +mlxcel_commit: 77afbb1d (branch update/issue-2065-rocm-fused-moe-ports, lablup/mlxcel#2098, on origin/main 57d8ed29). The HIP ports of moe_gateup and moe_down, the table predicates, and the ROCm SGY default of 2. +mlx_pin: 81ba1c6a0e50a9268b931579c2d4f1158b9aab5a +rocm_overlay_fork: NripeshN/mlx rocm-support @ 75915908dfe5028335d318b10340313744fd3a8d (src/lib/mlx-cpp/patches-rocm/UPSTREAM), plus the local fixes in LOCAL_FIXES.md +logit_trace_sha256: ba41ee7a57f1c4e0999112d071062239241a571eca59aa8aa20751e480fd8a14 +baseline_logit_trace_sha256: 31bd9eb0ab06c232fc0e374a4675d6c92f380f145ffc1c495dce043f05b240f8 (origin/main 57d8ed29, used for the main-vs-branch rows in the results page; those traces are not committed here) +build: cargo build --release --features rocm --example logit_trace +corpus: tests/fixtures/wikitext2_excerpt.txt sha256 16721fc1057d69efc8493718fcad562de9e4554ce05369876cd06dcba431394e +checkpoints: the same local directories as ../rocm_gfx1151_bec64748/ (Qwen3-30B-A3B-4bit@d388dead, Mixtral-8x7B-Instruct-v0.1-4bit@4e8f7045) and ../rocm_gfx1151_96cbce84/ (NVIDIA-Nemotron-3-Nano-30B-A3B-4bit@832f602eba5d22436c258c1462bdedc5afddb42b) +args: w1='1 128 8 0'; w8='8 80 8 512'; w1ctx512='1 128 8 512' with MLXCEL_TRACE_START_TOKEN=512 (CHUNK_TOKENS MAX_CHUNKS TOPK PREFILL) +tags: 'default' = no override. 'fused0' = MLXCEL_FUSED_MOE=0 (gather_qmm). The fused MoE kernels run only on a single-token forward, so on Qwen3-30B-A3B only the w1 'default' trace takes the HIP kernels; its w8 'default' and 'fused0' traces are identical. Mixtral-8x7B's experts (Dff 14336) are above the 8192 MLXCEL_FUSED_MOE_MAX_DFF default, so its 'default' runs gather_qmm. Nemotron-H's 'default' runs fused_moe_forward (gather_qmm routed experts) where main ran forward_nonfused. +filtering: the Nemotron-H files are stored with the five '[NemotronH] ' loader lines removed (grep -v '^\['), so they compare directly; the older Nemotron-H sets keep those lines. +env: LD_LIBRARY_PATH unset; MLXCEL_KEEP_BF16 unset; MLXCEL_FUSED_MOE unset except on 'fused0'; MLXCEL_FUSED_MOE_SGY unset (ROCm default 2) +gpu: correctness traces, not timings; another development unit may have used the GPU during a run +references: ../metal_m1u_bec64748/ (Qwen3, Mixtral), ../rocm_gfx1151_bec64748/ (ROCm gather_qmm 'fused0'), ../rocm_gfx1151_c5fe9a16/ (Nemotron-H w1, w8 on forward_nonfused), ../rocm_gfx1151_96cbce84/ (Nemotron-H w1ctx512 on main), ../metal_m5_d1128266/ (Nemotron-H Metal) diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/README.md b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/README.md new file mode 100644 index 000000000..284cfecc5 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/README.md @@ -0,0 +1,20 @@ +# ROCm logit traces with the HIP fused MoE decode kernels (lablup/mlxcel#2065) + +Teacher-forced traces from `examples/logit_trace` on the Radeon 8060S (`gfx1151`) after `moe_gateup` and `moe_down` gained HIP ports. The fused MoE path runs only on a single-token forward, so the `w1` rows are the ones that reach the kernels on Qwen3-30B-A3B; the `w8` rows check that the multi-token path is unchanged. `METADATA.txt` records the host, versions, commit and binary hash; `RUNS.txt` the exit status and row count of every run; `SHA256SUMS` covers every trace. The comparisons are in `docs/benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md`. + +```bash +cargo build --release --features rocm --example logit_trace +T=./target/release/examples/logit_trace +C=tests/fixtures/wikitext2_excerpt.txt +P=rocm_gfx1151_77afbb1d +for w in "w1:1 128 8 0" "w8:8 80 8 512"; do + tag=${w%%:*}; args=${w#*:} + $T models/mlx/Qwen3-30B-A3B-4bit $C $args > ${P}_qwen3-30b-a3b_default_$tag.tsv + MLXCEL_FUSED_MOE=0 $T models/mlx/Qwen3-30B-A3B-4bit $C $args > ${P}_qwen3-30b-a3b_fused0_$tag.tsv + $T models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit $C $args | grep -v '^\[' > ${P}_nemotron-3-nano-30b-a3b_default_$tag.tsv +done +MLXCEL_TRACE_START_TOKEN=512 $T models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit $C 1 128 8 512 | grep -v '^\[' > ${P}_nemotron-3-nano-30b-a3b_default_w1ctx512.tsv +$T models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit $C 1 128 8 0 > ${P}_mixtral-8x7b-instruct_default_w1.tsv +``` + +Unlike the older Nemotron-H sets, the files here are stored without the five `[NemotronH]` loader lines. Filter the older references with `grep -v '^\[NemotronH\] '` before `scripts/compare_logit_traces.py`. diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/RUNS.txt b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/RUNS.txt new file mode 100644 index 000000000..75d90bf91 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/RUNS.txt @@ -0,0 +1,8 @@ +rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w8 exit=0 rows=647 elapsed=92s +rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w8 exit=0 rows=647 elapsed=89s +rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w1 exit=0 rows=135 elapsed=13s +rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w1 exit=0 rows=135 elapsed=7s +rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w8 exit=0 rows=647 elapsed=100s (loader lines filtered) +rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1 exit=0 rows=135 elapsed=13s (loader lines filtered) +rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1ctx512 exit=0 rows=135 elapsed=not timed (loader lines filtered) +rocm_gfx1151_77afbb1d_mixtral-8x7b-instruct_default_w1 exit=0 rows=135 elapsed=35s diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/SHA256SUMS b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/SHA256SUMS new file mode 100644 index 000000000..4ad82bc7c --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/SHA256SUMS @@ -0,0 +1,8 @@ +48e6537dc7be46fcef35f86fe24ece6778c63d2affcae2d95f098cc47edfccad rocm_gfx1151_77afbb1d_mixtral-8x7b-instruct_default_w1.tsv +d70dff2d7a2ef4b67310c7a029282b415ba4465e2260907bfa307f792252ffb8 rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1ctx512.tsv +281d6189a3d4ca3f49ee17ac8a51928a0b942edecfcd831d5f331f94d2aa268f rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1.tsv +c7373682196b8425837cb0edb9b2a86c39464bd073bda5fb5c38fd2acc97fb15 rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w8.tsv +84f585edd756753803a1e266d7bad8ac4d4573febbc39a9671b2788f89cbd1f4 rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w1.tsv +0c159e49996cc60fa9caf7aa38ff50020bb464b96596569a8926ba336996e7e2 rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w8.tsv +cc0114a1af1b71ca47c7583699b77d645b64de8d6e4c9f0ffe27e4045ed53a44 rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w1.tsv +0c159e49996cc60fa9caf7aa38ff50020bb464b96596569a8926ba336996e7e2 rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w8.tsv diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_mixtral-8x7b-instruct_default_w1.tsv b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_mixtral-8x7b-instruct_default_w1.tsv new file mode 100644 index 000000000..54bd1b406 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_mixtral-8x7b-instruct_default_w1.tsv @@ -0,0 +1,135 @@ +# model models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit +# device GPU +# corpus tests/fixtures/wikitext2_excerpt.txt tokens 326253 +# chunks 128 chunk_tokens 1 topk 8 prefill 0 +# columns chunk pos target nll top_ids top_logits +# start_token 0 +# contiguous false +0 0 12217 10.522325 1247,422,1186,3660,22478,726,7299,589 12.023438,10.265625,9.773438,9.507812,9.257812,8.929688,8.726562,8.414062 +1 0 570 7.930460 21861,546,278,28707,515,309,313,593 18.343750,16.843750,16.468750,15.421875,14.968750,14.890625,14.562500,14.500000 +2 0 510 4.347628 338,13,28723,28717,28729,28744,28730,327 13.914062,13.195312,12.570312,12.468750,12.437500,12.062500,11.445312,11.437500 +3 0 28733 5.776094 6730,327,28707,311,13,28709,28764,282 12.078125,11.210938,10.585938,10.523438,10.500000,10.023438,9.867188,9.789062 +4 0 28750 6.222353 260,5629,4258,6227,6434,315,415,422 17.484375,17.421875,15.890625,15.812500,15.812500,15.539062,15.429688,15.343750 +5 0 3735 12.651031 28734,28782,28781,28783,28770,28750,28784,28740 19.890625,18.171875,18.140625,18.046875,18.015625,17.937500,17.843750,17.687500 +6 0 3198 11.690868 288,321,8731,564,28723,574,286,28733 16.750000,16.000000,15.484375,14.960938,14.835938,14.203125,14.085938,13.617188 +7 0 20272 4.323570 19445,266,282,12992,27092,267,3748,3123 16.718750,16.671875,15.437500,14.906250,14.890625,14.804688,14.789062,14.703125 +8 0 327 5.915341 28730,28723,13,28733,28725,28736,28739,325 13.054688,12.609375,12.351562,12.179688,12.023438,11.703125,11.593750,11.585938 +9 0 13 5.950287 1357,3047,1647,22568,965,1811,16339,13 16.937500,16.859375,15.039062,14.859375,14.554688,14.421875,13.210938,11.890625 +10 0 13 3.375843 28771,1064,1014,13,348,28737,657,28767 13.820312,12.914062,12.781250,12.585938,12.296875,12.179688,12.078125,12.046875 +11 0 4220 9.575062 28771,1064,1014,13,348,28737,657,28767 13.820312,12.914062,12.781250,12.585938,12.296875,12.179688,12.078125,12.046875 +12 0 260 11.140512 28747,714,13,764,302,9066,7596,17297 20.562500,16.593750,16.421875,15.093750,15.046875,14.843750,14.828125,14.656250 +13 0 714 8.886410 415,28740,28750,560,315,13,851,12 15.843750,15.031250,14.562500,14.390625,13.968750,13.898438,13.773438,13.765625 +14 0 4449 10.422524 28733,1398,13,7591,28707,15845,6051,28705 15.117188,13.281250,13.015625,12.953125,12.890625,12.140625,11.656250,11.281250 +15 0 1508 0.001547 1508,28747,421,589,714,28723,13,327 21.140625,13.187500,12.343750,11.945312,11.367188,11.359375,11.320312,10.859375 +16 0 28716 7.391541 2849,421,3016,13,28748,1409,16239,9301 12.289062,11.648438,10.796875,10.515625,10.437500,10.171875,10.109375,9.976562 +17 0 786 5.500553 28725,327,28740,4362,28746,2020,1067,9813 12.539062,11.421875,10.906250,9.921875,9.890625,9.875000,9.687500,9.585938 +18 0 3080 7.114605 469,324,28705,346,28709,1097,13,5904 13.187500,13.085938,12.164062,11.640625,11.500000,11.164062,10.921875,10.687500 +19 0 1797 11.975202 263,449,858,404,298,9358,3773,13 15.617188,11.757812,11.554688,11.242188,11.210938,11.046875,11.015625,10.687500 +20 0 28723 3.068218 13,28723,28748,28725,325,28705,28733,28747 16.203125,14.171875,13.414062,13.343750,13.085938,13.062500,12.882812,12.789062 +21 0 1115 9.876192 13,1056,5515,3549,842,1631,11698,7227 15.554688,15.257812,14.640625,14.296875,13.312500,12.921875,12.804688,12.593750 +22 0 28748 6.912628 496,327,282,4318,1126,28733,28716,3015 12.460938,12.156250,11.695312,11.664062,11.320312,11.242188,11.218750,11.203125 +23 0 4201 13.130695 1393,13,565,812,8193,9329,259,732 17.281250,13.578125,13.250000,12.289062,12.093750,11.812500,11.632812,11.609375 +24 0 20363 3.797662 269,293,375,316,314,323,28723,377 11.742188,11.593750,11.421875,11.023438,10.890625,10.765625,10.406250,10.390625 +25 0 28748 4.447999 327,28723,13,28725,28730,1410,28746,354 14.250000,13.640625,12.312500,11.414062,11.171875,11.054688,11.054688,10.945312 +26 0 28735 10.513508 1393,13,565,812,8193,9329,259,732 17.281250,13.578125,13.250000,12.289062,12.093750,11.812500,11.632812,11.609375 +27 0 2230 4.639923 1251,8367,2729,269,915,1086,833,3389 15.117188,14.867188,13.976562,13.906250,13.460938,13.460938,13.335938,13.328125 +28 0 5712 7.200614 28716,28818,13,28710,262,360,753,28740 14.531250,14.125000,13.039062,12.351562,12.078125,11.648438,11.546875,11.335938 +29 0 28748 5.515120 13,327,28723,28730,28733,325,28725,349 15.859375,13.507812,13.484375,13.125000,12.882812,12.781250,12.750000,12.671875 +30 0 18509 16.071125 1393,13,565,812,8193,9329,259,732 17.281250,13.578125,13.250000,12.289062,12.093750,11.812500,11.632812,11.609375 +31 0 570 4.629711 21861,515,593,313,278,449,508,1588 14.335938,13.781250,12.992188,12.765625,12.734375,12.507812,11.937500,11.757812 +32 0 510 4.347628 338,13,28723,28717,28729,28744,28730,327 13.914062,13.195312,12.570312,12.468750,12.437500,12.062500,11.445312,11.437500 +33 0 13 3.377657 6730,327,28707,311,13,28709,28764,282 12.078125,11.210938,10.585938,10.523438,10.500000,10.023438,9.867188,9.789062 +34 0 3540 7.953968 28771,1064,1014,13,348,28737,657,28767 13.820312,12.914062,12.781250,12.585938,12.296875,12.179688,12.078125,12.046875 +35 0 673 4.739233 10286,303,3901,28714,11666,6306,28719,24344 14.476562,12.921875,12.859375,12.539062,12.382812,12.039062,12.031250,11.937500 +36 0 260 9.687000 387,28732,714,14484,28730,4831,325,648 16.765625,15.601562,15.039062,13.765625,13.578125,13.250000,12.687500,12.226562 +37 0 714 8.886410 415,28740,28750,560,315,13,851,12 15.843750,15.031250,14.562500,14.390625,13.968750,13.898438,13.773438,13.765625 +38 0 275 11.461587 28733,1398,13,7591,28707,15845,6051,28705 15.117188,13.281250,13.015625,12.953125,12.890625,12.140625,11.656250,11.281250 +39 0 849 3.654298 527,327,28727,28725,28713,849,10273,336 12.773438,11.617188,10.898438,10.710938,10.531250,10.328125,10.007812,9.648438 +40 0 570 9.199478 3306,283,266,941,280,405,6463,1900 12.835938,11.593750,11.554688,11.453125,11.312500,11.218750,11.109375,10.960938 +41 0 510 4.347628 338,13,28723,28717,28729,28744,28730,327 13.914062,13.195312,12.570312,12.468750,12.437500,12.062500,11.445312,11.437500 +42 0 28733 5.776094 6730,327,28707,311,13,28709,28764,282 12.078125,11.210938,10.585938,10.523438,10.500000,10.023438,9.867188,9.789062 +43 0 28750 6.222353 260,5629,4258,6227,6434,315,415,422 17.484375,17.421875,15.890625,15.812500,15.812500,15.539062,15.429688,15.343750 +44 0 28733 5.479156 28734,28782,28781,28783,28770,28750,28784,28740 19.890625,18.171875,18.140625,18.046875,18.015625,17.937500,17.843750,17.687500 +45 0 1507 15.622744 260,5629,4258,6227,6434,315,415,422 17.484375,17.421875,15.890625,15.812500,15.812500,15.539062,15.429688,15.343750 +46 0 28733 5.260419 28712,13,22078,282,288,28713,28714,28730 13.265625,12.664062,11.398438,10.765625,10.546875,10.476562,10.296875,10.054688 +47 0 28728 12.437197 260,5629,4258,6227,6434,315,415,422 17.484375,17.421875,15.890625,15.812500,15.812500,15.539062,15.429688,15.343750 +48 0 28740 3.077340 321,327,4743,28740,28734,28723,357,28746 13.257812,11.500000,11.234375,11.156250,10.742188,10.445312,10.367188,10.273438 +49 0 13 2.832172 28723,28734,28750,13,28774,28783,28782,28784 19.484375,18.234375,18.000000,17.906250,17.812500,17.562500,17.546875,17.531250 +50 0 16990 11.887562 28771,1064,1014,13,348,28737,657,28767 13.820312,12.914062,12.781250,12.585938,12.296875,12.179688,12.078125,12.046875 +51 0 355 9.164753 13,360,470,28733,3578,272,349,264 15.070312,14.992188,14.890625,13.945312,13.937500,13.757812,13.742188,13.554688 +52 0 714 6.629436 415,315,560,13,422,842,28740,963 15.710938,14.156250,13.898438,13.750000,13.687500,13.539062,13.531250,13.429688 +53 0 1369 11.828774 28733,1398,13,7591,28707,15845,6051,28705 15.117188,13.281250,13.015625,12.953125,12.890625,12.140625,11.656250,11.281250 +54 0 325 6.905531 28730,327,618,857,387,2210,13,28723 16.140625,15.335938,15.156250,14.539062,14.257812,13.945312,13.289062,13.171875 +55 0 28781 6.373415 2628,3260,28743,908,28740,28705,470,4176 14.312500,13.046875,12.468750,12.242188,11.750000,11.695312,11.679688,11.609375 +56 0 28770 3.262199 13,28723,28734,28782,28783,28750,28770,28747 17.359375,17.312500,16.562500,16.546875,16.421875,16.343750,16.218750,16.187500 +57 0 28782 3.141729 13,28723,28734,28740,28783,28782,28750,28770 16.750000,16.578125,16.437500,16.296875,15.929688,15.914062,15.820312,15.781250 +58 0 28783 3.324972 28734,13,28723,28782,28747,28750,28783,28770 17.437500,17.390625,17.328125,16.859375,16.718750,16.625000,16.625000,16.578125 +59 0 11926 9.722061 13,28783,28747,362,28723,28734,28782,28748 17.796875,17.718750,17.703125,17.562500,17.531250,17.453125,17.343750,17.328125 +60 0 28731 9.132744 327,28725,28730,28746,6210,13,302,12303 17.437500,16.062500,14.007812,13.781250,13.742188,12.843750,12.695312,12.210938 +61 0 13 1.607132 13,28705,415,304,315,371,349,560 16.656250,15.054688,14.921875,14.570312,14.250000,14.210938,14.125000,14.117188 +62 0 9428 12.452015 28771,1064,1014,13,348,28737,657,28767 13.820312,12.914062,12.781250,12.585938,12.296875,12.179688,12.078125,12.046875 +63 0 2287 7.136992 28747,13,564,28732,351,1641,28706,327 16.843750,15.984375,15.117188,14.164062,13.929688,13.648438,13.437500,13.414062 +64 0 714 5.823467 798,908,2463,422,963,6081,801,443 15.117188,14.648438,14.414062,14.390625,14.382812,14.195312,14.164062,14.148438 +65 0 16900 16.202065 28733,1398,13,7591,28707,15845,6051,28705 15.117188,13.281250,13.015625,12.953125,12.890625,12.140625,11.656250,11.281250 +66 0 11732 1.639950 11732,11527,28734,28733,9833,13,28777,3469 15.343750,14.835938,14.460938,13.843750,13.367188,12.976562,12.937500,12.703125 +67 0 28733 5.726730 475,351,28747,399,393,365,524,318 17.515625,17.484375,17.125000,17.000000,16.703125,16.687500,16.687500,16.625000 +68 0 5530 15.388369 260,5629,4258,6227,6434,315,415,422 17.484375,17.421875,15.890625,15.812500,15.812500,15.539062,15.429688,15.343750 +69 0 28705 6.778433 28762,2945,3728,28765,28790,5790,28779,4874 14.976562,14.710938,14.632812,13.945312,13.898438,13.843750,13.625000,13.156250 +70 0 28770 3.414432 28750,28740,28734,1177,28770,13,28784,28783 10.710938,10.546875,9.421875,8.960938,8.859375,8.679688,8.578125,8.492188 +71 0 28723 2.477667 13,28723,28734,28740,28783,28782,28750,28770 16.750000,16.578125,16.437500,16.296875,15.929688,15.914062,15.820312,15.781250 +72 0 28734 6.208223 13,1056,5515,3549,842,1631,11698,7227 15.554688,15.257812,14.640625,14.296875,13.312500,12.921875,12.804688,12.593750 +73 0 325 6.858860 13,28783,28782,28770,28750,28774,28784,28740 20.015625,18.796875,18.718750,18.656250,18.640625,18.625000,18.609375,18.500000 +74 0 3887 9.205446 2628,3260,28743,908,28740,28705,470,4176 14.312500,13.046875,12.468750,12.242188,11.750000,11.695312,11.679688,11.609375 +75 0 1508 0.005081 1508,28723,28747,13,327,421,714,28713 19.656250,12.484375,12.351562,11.656250,11.265625,11.195312,10.867188,10.640625 +76 0 16346 9.520447 2849,421,3016,13,28748,1409,16239,9301 12.289062,11.648438,10.796875,10.515625,10.437500,10.171875,10.109375,9.976562 +77 0 495 0.622168 495,271,1771,288,274,1238,482,734 17.843750,16.718750,15.343750,15.226562,15.179688,14.984375,14.929688,14.812500 +78 0 26527 13.917536 750,13,528,582,297,378,264,272 14.437500,14.062500,13.851562,13.820312,13.757812,13.617188,13.585938,13.539062 +79 0 28723 0.773268 28723,1058,13,28747,3907,28733,596,28748 15.812500,14.804688,14.296875,13.281250,12.820312,12.726562,12.359375,12.312500 +80 0 1909 6.575411 13,1056,5515,3549,842,1631,11698,7227 15.554688,15.257812,14.640625,14.296875,13.312500,12.921875,12.804688,12.593750 +81 0 28748 4.155936 28723,13,327,28730,28747,28748,12,28733 12.031250,10.187500,10.101562,8.875000,8.757812,8.632812,8.460938,8.234375 +82 0 9146 17.397892 1393,13,565,812,8193,9329,259,732 17.281250,13.578125,13.250000,12.289062,12.093750,11.812500,11.632812,11.609375 +83 0 28748 4.685043 13,327,28723,28730,28746,28747,28733,304 15.875000,15.140625,14.304688,13.703125,13.320312,13.070312,12.859375,12.585938 +84 0 1403 8.970539 1393,13,565,812,8193,9329,259,732 17.281250,13.578125,13.250000,12.289062,12.093750,11.812500,11.632812,11.609375 +85 0 28733 7.947021 28747,475,524,393,399,3304,5459,2215 17.296875,16.703125,16.640625,16.562500,16.515625,16.515625,16.515625,16.484375 +86 0 3820 13.464540 260,5629,4258,6227,6434,315,415,422 17.484375,17.421875,15.890625,15.812500,15.812500,15.539062,15.429688,15.343750 +87 0 28748 6.827408 13,28733,11091,293,554,375,28730,508 13.953125,13.562500,13.242188,13.007812,12.945312,12.898438,12.804688,12.750000 +88 0 28770 10.771320 1393,13,565,812,8193,9329,259,732 17.281250,13.578125,13.250000,12.289062,12.093750,11.812500,11.632812,11.609375 +89 0 28723 2.477667 13,28723,28734,28740,28783,28782,28750,28770 16.750000,16.578125,16.437500,16.296875,15.929688,15.914062,15.820312,15.781250 +90 0 28734 6.208223 13,1056,5515,3549,842,1631,11698,7227 15.554688,15.257812,14.640625,14.296875,13.312500,12.921875,12.804688,12.593750 +91 0 19840 15.183079 13,28783,28782,28770,28750,28774,28784,28740 20.015625,18.796875,18.718750,18.656250,18.640625,18.625000,18.609375,18.500000 +92 0 13 1.354422 13,28804,28832,371,1365,28713,325,387 14.148438,12.101562,11.703125,11.640625,11.382812,11.359375,11.226562,11.171875 +93 0 2517 8.250843 28771,1064,1014,13,348,28737,657,28767 13.820312,12.914062,12.781250,12.585938,12.296875,12.179688,12.078125,12.046875 +94 0 1030 2.780453 2148,468,595,1030,3310,2694,15292,10899 15.406250,14.804688,14.679688,14.093750,14.054688,13.945312,13.898438,13.320312 +95 0 296 0.917910 296,594,286,734,271,288,522,28709 17.906250,17.187500,16.968750,16.484375,15.851562,15.445312,14.015625,13.570312 +96 0 28747 5.440016 294,13,28733,28730,753,28723,515,470 13.867188,12.781250,11.625000,11.578125,11.335938,11.046875,10.960938,10.734375 +97 0 21740 12.173933 13,28707,15845,28727,7591,28705,376,3901 13.609375,12.781250,12.671875,12.500000,12.195312,12.101562,12.078125,12.023438 +98 0 26091 7.044914 349,18791,659,13,2204,28747,2627,28725 19.265625,19.156250,18.984375,18.875000,18.640625,18.625000,18.250000,18.187500 +99 0 28725 5.522213 13,28747,327,298,28723,325,28730,28705 19.734375,17.000000,15.718750,15.625000,15.023438,14.890625,14.796875,14.625000 +100 0 16377 11.016953 13,325,272,732,28705,264,1939,304 17.796875,16.078125,15.976562,15.968750,15.960938,15.726562,15.648438,15.429688 +101 0 5712 0.914383 5712,302,28747,304,21721,460,354,8234 20.703125,18.953125,18.171875,17.953125,17.734375,17.640625,17.406250,17.375000 +102 0 16107 12.483870 13,327,28723,28730,28733,325,28725,349 15.859375,13.507812,13.484375,13.125000,12.882812,12.781250,12.750000,12.671875 +103 0 7982 6.997429 28733,349,304,3477,297,659,13,354 17.187500,16.843750,16.546875,16.359375,16.156250,16.156250,15.882812,15.687500 +104 0 13 9.871391 404,659,297,356,4370,486,12308,304 22.390625,19.671875,19.453125,19.031250,18.906250,18.890625,18.843750,18.796875 +105 0 558 8.953968 28771,1064,1014,13,348,28737,657,28767 13.820312,12.914062,12.781250,12.585938,12.296875,12.179688,12.078125,12.046875 +106 0 325 4.494858 13,415,28740,842,583,315,523,28750 15.351562,13.734375,13.171875,13.164062,12.742188,12.664062,12.484375,12.406250 +107 0 28735 5.912477 2628,3260,28743,908,28740,28705,470,4176 14.312500,13.046875,12.468750,12.242188,11.750000,11.695312,11.679688,11.609375 +108 0 28723 6.194611 1251,8367,2729,269,915,1086,833,3389 15.117188,14.867188,13.976562,13.906250,13.460938,13.460938,13.335938,13.328125 +109 0 5625 11.930880 13,1056,5515,3549,842,1631,11698,7227 15.554688,15.257812,14.640625,14.296875,13.312500,12.921875,12.804688,12.593750 +110 0 472 10.606995 643,338,2742,490,606,291,705,2294 16.656250,16.468750,15.648438,15.460938,15.148438,15.109375,15.031250,14.976562 +111 0 28725 3.040771 302,28723,28725,304,297,349,28733,298 15.664062,15.320312,14.960938,14.453125,14.351562,14.210938,13.710938,13.601562 +112 0 418 5.759140 13,325,272,732,28705,264,1939,304 17.796875,16.078125,15.976562,15.968750,15.960938,15.726562,15.648438,15.429688 +113 0 28723 5.432296 327,28725,473,2934,3000,374,12322,3972 13.851562,12.984375,12.750000,11.976562,11.968750,11.960938,11.664062,11.500000 +114 0 25959 13.329317 13,1056,5515,3549,842,1631,11698,7227 15.554688,15.257812,14.640625,14.296875,13.312500,12.921875,12.804688,12.593750 +115 0 789 4.600626 2047,941,262,300,912,789,276,12553 19.843750,17.937500,17.828125,16.437500,16.031250,15.703125,15.695312,15.687500 +116 0 524 9.350296 13,276,28723,6234,28727,28712,849,28729 14.304688,13.789062,12.546875,12.507812,12.250000,12.062500,12.039062,12.000000 +117 0 274 6.289841 3676,1251,14821,1087,2192,2951,14468,3665 13.671875,13.195312,12.820312,12.601562,12.601562,12.546875,12.539062,12.335938 +118 0 13600 9.754205 28784,13,1391,517,381,822,4083,2644 11.765625,11.679688,11.398438,11.062500,11.000000,10.851562,10.773438,10.765625 +119 0 28725 4.796036 13,705,380,293,375,276,357,321 13.875000,13.507812,12.156250,12.148438,11.929688,11.906250,11.578125,11.343750 +120 0 399 6.384140 13,325,272,732,28705,264,1939,304 17.796875,16.078125,15.976562,15.968750,15.960938,15.726562,15.648438,15.429688 +121 0 28723 4.172563 2014,4947,20336,11672,28723,1906,1604,1383 12.617188,12.187500,11.539062,11.343750,11.070312,10.960938,10.953125,10.937500 +122 0 1537 10.384005 13,1056,5515,3549,842,1631,11698,7227 15.554688,15.257812,14.640625,14.296875,13.312500,12.921875,12.804688,12.593750 +123 0 6113 11.573893 315,28725,368,1236,456,272,767,378 20.296875,20.062500,19.921875,18.671875,18.500000,18.484375,18.343750,17.859375 +124 0 1040 8.899893 631,6113,13,25062,338,17825,11746,28723 15.468750,15.046875,13.929688,13.046875,12.796875,11.515625,11.382812,11.335938 +125 0 345 7.764133 13,27920,28821,28705,486,12789,2463,415 18.187500,17.421875,16.375000,16.312500,16.015625,15.609375,15.546875,15.484375 +126 0 8638 13.664902 1730,28737,1014,1976,3260,28741,13,8543 18.468750,14.312500,13.843750,12.820312,12.820312,12.820312,12.812500,12.789062 +127 0 318 8.678164 13,327,28723,1551,28747,564,349,298 15.507812,14.367188,13.781250,13.593750,12.500000,12.187500,12.046875,11.906250 diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1.tsv b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1.tsv new file mode 100644 index 000000000..662d69769 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1.tsv @@ -0,0 +1,135 @@ +# model models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit +# device GPU +# corpus tests/fixtures/wikitext2_excerpt.txt tokens 301749 +# chunks 128 chunk_tokens 1 topk 8 prefill 0 +# columns chunk pos target nll top_ids top_logits +# start_token 0 +# contiguous false +0 0 20211 14.614285 59199,1256,1032,2251,1603,1050,1319,1317 12.687500,12.562500,12.187500,11.937500,11.875000,11.750000,11.750000,11.437500 +1 0 1752 12.305206 109431,1046,1044,1034,2251,59199,1319,1338 14.312500,13.312500,13.187500,12.937500,12.687500,12.437500,12.125000,12.062500 +2 0 1637 13.316227 1044,109431,1046,1034,59199,1338,1048,1041 13.000000,12.812500,12.687500,12.375000,12.375000,11.812500,11.750000,11.687500 +3 0 1045 4.683241 109431,1044,59199,1046,1034,1048,1041,1319 13.875000,13.250000,13.125000,12.687500,12.625000,12.500000,11.937500,11.937500 +4 0 1050 3.996655 1256,59199,1032,1603,1050,11,2251,1319 13.000000,12.937500,12.375000,12.250000,12.062500,11.937500,11.812500,11.750000 +5 0 7637 10.189596 1034,1044,1438,1317,109431,1046,1338,2251 12.687500,12.250000,12.125000,12.000000,12.000000,11.875000,11.875000,11.687500 +6 0 103920 11.596970 109431,1034,1317,1044,1046,59199,1338,2251 13.125000,12.750000,12.562500,12.500000,12.375000,11.875000,11.687500,11.562500 +7 0 1376 8.037281 1044,1034,109431,1046,59199,1438,2251,1319 13.750000,13.562500,13.437500,13.125000,13.000000,12.687500,12.562500,12.500000 +8 0 1267 7.323183 1256,59199,1032,1603,2251,1319,1364,1317 12.687500,12.437500,12.187500,12.062500,11.687500,11.562500,11.562500,11.187500 +9 0 8263 13.938139 59199,1034,1256,1438,11,1032,2251,109431 12.437500,12.375000,12.312500,12.312500,11.687500,11.687500,11.625000,11.562500 +10 0 1260 10.368208 109431,1034,1044,59199,1046,1438,1319,1041 13.875000,13.812500,13.562500,13.187500,13.000000,12.875000,12.437500,12.312500 +11 0 1737 8.968542 1034,1256,1438,11,59199,1032,1044,1338 13.062500,12.812500,12.812500,12.562500,12.375000,12.187500,12.187500,12.000000 +12 0 7155 10.904697 1256,59199,1032,1603,11,1364,1429,2129 14.000000,13.437500,13.312500,13.000000,12.937500,12.750000,12.375000,12.375000 +13 0 2345 12.545601 1044,1046,1034,1338,1317,109431,2251,1438 13.312500,13.250000,12.937500,12.625000,12.562500,12.437500,12.375000,11.937500 +14 0 1104 9.288986 1034,1046,109431,1044,59199,2251,1438,1338 13.375000,13.000000,12.875000,12.625000,12.625000,12.500000,12.312500,12.062500 +15 0 1936 11.063481 1034,1044,109431,1438,1046,1317,59199,1338 13.250000,12.562500,12.312500,12.250000,12.000000,11.937500,11.812500,11.687500 +16 0 7366 12.798203 109431,1044,1046,59199,1034,1048,1319,1338 13.750000,13.375000,13.000000,12.875000,12.812500,12.000000,12.000000,12.000000 +17 0 3136 15.766094 1034,1044,1046,109431,1278,1338,1317,1438 12.875000,12.750000,12.500000,12.500000,12.437500,12.062500,12.000000,12.000000 +18 0 20524 12.196472 1034,109431,1044,1046,1042,1438,59199,1041 13.625000,13.625000,13.312500,12.750000,12.437500,12.437500,12.062500,11.937500 +19 0 6406 9.263884 1034,109431,1044,2251,1046,1438,59199,1414 13.937500,13.750000,13.375000,12.812500,12.750000,12.750000,12.625000,12.250000 +20 0 71129 16.407085 1034,109431,1046,1044,1317,1338,1438,59199 13.000000,13.000000,12.750000,12.437500,12.187500,12.000000,12.000000,11.750000 +21 0 14327 10.770994 1034,1044,1438,1046,2251,1321,1338,1319 13.625000,13.625000,13.187500,12.687500,12.687500,12.125000,12.125000,12.062500 +22 0 3289 13.670559 109431,1034,1046,1044,59199,1438,1338,1319 13.562500,13.125000,12.937500,12.687500,12.687500,12.187500,12.000000,11.812500 +23 0 17271 16.537022 1044,1046,109431,59199,1034,1048,1319,1338 13.062500,12.625000,12.562500,12.375000,12.125000,11.937500,11.687500,11.687500 +24 0 13001 13.339697 109431,1044,1034,1046,1438,59199,1041,1338 13.812500,13.187500,13.000000,13.000000,12.687500,12.250000,11.937500,11.875000 +25 0 1552 13.081118 109431,1034,1046,1044,1438,1338,59199,1317 13.812500,13.562500,13.125000,13.062500,12.437500,12.312500,12.312500,12.187500 +26 0 1752 13.266277 109431,1044,1046,59199,1034,1319,1048,1338 13.437500,13.250000,12.875000,12.687500,12.500000,12.000000,11.937500,11.937500 +27 0 1637 13.316227 1044,109431,1046,1034,59199,1338,1048,1041 13.000000,12.812500,12.687500,12.375000,12.375000,11.812500,11.750000,11.687500 +28 0 1010 5.620741 109431,1044,59199,1046,1034,1048,1041,1319 13.875000,13.250000,13.125000,12.687500,12.625000,12.500000,11.937500,11.937500 +29 0 7324 13.304087 1256,59199,11,1438,1032,1034,1338,109431 12.875000,12.375000,12.187500,12.187500,11.937500,11.937500,11.687500,11.625000 +30 0 2683 11.822817 109431,1034,59199,1046,1044,1317,1438,1319 14.937500,13.562500,13.187500,13.000000,12.875000,12.687500,12.625000,12.250000 +31 0 1260 10.170143 1034,1044,109431,1317,1046,1438,1307,1338 12.937500,12.687500,12.687500,12.250000,12.187500,12.125000,11.937500,11.812500 +32 0 1737 8.968542 1034,1256,1438,11,59199,1032,1044,1338 13.062500,12.812500,12.812500,12.562500,12.375000,12.187500,12.187500,12.000000 +33 0 1285 10.373447 1256,59199,1032,1603,11,1364,1429,2129 14.000000,13.437500,13.312500,13.000000,12.937500,12.750000,12.375000,12.375000 +34 0 1552 12.441151 1034,109431,1317,1044,1046,1338,1408,1438 12.875000,12.812500,12.625000,12.437500,12.125000,12.062500,11.937500,11.937500 +35 0 1752 13.266277 109431,1044,1046,59199,1034,1319,1048,1338 13.437500,13.250000,12.875000,12.687500,12.500000,12.000000,11.937500,11.937500 +36 0 1637 13.316227 1044,109431,1046,1034,59199,1338,1048,1041 13.000000,12.812500,12.687500,12.375000,12.375000,11.812500,11.750000,11.687500 +37 0 1045 4.683241 109431,1044,59199,1046,1034,1048,1041,1319 13.875000,13.250000,13.125000,12.687500,12.625000,12.500000,11.937500,11.937500 +38 0 1050 3.996655 1256,59199,1032,1603,1050,11,2251,1319 13.000000,12.937500,12.375000,12.250000,12.062500,11.937500,11.812500,11.750000 +39 0 1045 6.127096 1034,1044,1438,1317,109431,1046,1338,2251 12.687500,12.250000,12.125000,12.000000,12.000000,11.875000,11.875000,11.687500 +40 0 3416 13.762280 1256,59199,1032,1603,1050,11,2251,1319 13.000000,12.937500,12.375000,12.250000,12.062500,11.937500,11.812500,11.750000 +41 0 9332 11.023411 109431,1034,1044,1046,59199,1438,1041,1278 14.000000,13.250000,13.000000,12.750000,12.375000,12.125000,12.062500,12.062500 +42 0 1049 5.338951 109431,1034,1044,1046,59199,1438,1338,1041 13.562500,13.125000,12.750000,12.687500,12.125000,11.937500,11.875000,11.812500 +43 0 1010 5.317123 1034,1438,1044,59199,109431,1046,1338,2251 12.625000,12.125000,11.875000,11.875000,11.562500,11.500000,11.500000,11.312500 +44 0 44149 14.225962 1256,59199,11,1438,1032,1034,1338,109431 12.875000,12.375000,12.187500,12.187500,11.937500,11.937500,11.687500,11.625000 +45 0 1652 9.689829 109431,1034,1044,59199,1046,1317,1438,1414 14.562500,13.750000,13.125000,13.062500,12.812500,12.750000,12.750000,12.500000 +46 0 1737 9.173991 1034,59199,1438,1256,2251,1032,11,1317 12.750000,12.500000,12.375000,12.187500,12.000000,11.937500,11.875000,11.812500 +47 0 2688 9.998447 1256,59199,1032,1603,11,1364,1429,2129 14.000000,13.437500,13.312500,13.000000,12.937500,12.750000,12.375000,12.375000 +48 0 1319 4.268900 109431,1044,1046,1034,1338,1041,1438,59199 13.187500,13.062500,12.937500,12.250000,12.250000,11.562500,11.562500,11.562500 +49 0 1052 6.134815 1256,59199,1032,2251,1603,1317,1438,1364 12.625000,12.250000,11.687500,11.687500,11.500000,11.437500,11.437500,11.375000 +50 0 1051 5.817419 1034,1046,1044,1438,109431,59199,1338,1042 12.875000,12.312500,12.250000,12.250000,12.187500,12.000000,11.875000,11.687500 +51 0 1053 4.616913 1034,59199,1438,109431,1044,1046,1338,1256 12.937500,12.562500,12.500000,12.375000,12.187500,12.187500,12.000000,11.812500 +52 0 1056 6.391600 1034,1044,1046,109431,1438,59199,1338,1319 12.687500,12.312500,12.250000,12.250000,12.000000,11.937500,11.750000,11.437500 +53 0 12964 8.355261 1034,1044,109431,1046,59199,1438,1338,1319 12.875000,12.437500,12.375000,12.312500,12.312500,12.187500,11.875000,11.687500 +54 0 1456 6.685975 1044,1046,1034,1338,109431,1041,1319,1438 13.437500,12.875000,12.375000,12.375000,12.375000,11.937500,11.875000,11.812500 +55 0 93552 13.652958 1256,11,59199,1438,1032,1034,1050,2251 13.750000,12.937500,12.875000,12.750000,12.625000,12.312500,12.000000,12.000000 +56 0 1293 10.214773 1034,109431,1044,1046,59199,1438,1414,2251 14.125000,13.750000,13.562500,13.500000,12.937500,12.875000,12.500000,12.312500 +57 0 1737 8.728725 1256,59199,11,1032,2251,1317,1438,1034 13.250000,12.625000,12.500000,12.437500,12.250000,12.187500,12.125000,11.812500 +58 0 25079 11.342197 1256,59199,1032,1603,11,1364,1429,2129 14.000000,13.437500,13.312500,13.000000,12.937500,12.750000,12.375000,12.375000 +59 0 20992 13.370882 109431,1044,1034,59199,1046,1438,2251,1317 13.375000,13.125000,13.000000,12.875000,12.750000,12.687500,12.625000,12.250000 +60 0 5624 9.612956 1034,1044,1046,109431,1438,1338,1317,1278 12.937500,12.625000,12.562500,12.500000,12.312500,12.187500,12.062500,11.750000 +61 0 1065 7.425694 109431,1034,1438,59199,1044,1046,1050,1338 13.250000,12.875000,12.562500,12.562500,12.437500,12.437500,11.812500,11.750000 +62 0 1032 4.857929 1034,1044,59199,1438,109431,1046,1041,1319 12.750000,12.312500,12.125000,12.062500,12.062500,11.937500,11.812500,11.625000 +63 0 1051 4.993608 1256,59199,1032,2251,1050,1603,1317,1319 13.312500,12.500000,12.312500,12.250000,12.000000,12.000000,11.937500,11.937500 +64 0 1046 3.679413 1034,59199,1438,109431,1044,1046,1338,1256 12.937500,12.562500,12.500000,12.375000,12.187500,12.187500,12.000000,11.812500 +65 0 1048 5.436569 1256,59199,1034,1438,1032,2251,1338,1603 12.250000,12.125000,12.000000,11.875000,11.687500,11.625000,11.375000,11.312500 +66 0 1319 4.121596 1034,1438,1044,1338,1317,1046,59199,109431 12.687500,12.375000,12.187500,11.937500,11.875000,11.812500,11.437500,11.437500 +67 0 3299 13.728565 1256,59199,1032,2251,1603,1317,1438,1364 12.625000,12.250000,11.687500,11.687500,11.500000,11.437500,11.437500,11.375000 +68 0 2345 11.916069 1034,1044,1046,109431,1438,59199,1042,1041 13.687500,13.312500,13.312500,13.312500,13.125000,12.750000,12.562500,12.500000 +69 0 1860 15.433517 1034,1046,109431,1044,59199,2251,1438,1338 13.375000,13.000000,12.875000,12.625000,12.625000,12.500000,12.312500,12.062500 +70 0 2277 12.343655 109431,1034,1044,1046,59199,1438,1319,1050 13.937500,13.562500,13.062500,13.000000,12.750000,12.437500,12.125000,12.062500 +71 0 1730 12.353148 1044,1034,1046,109431,1338,1438,1319,1042 12.875000,12.562500,12.437500,12.187500,11.812500,11.625000,11.375000,11.250000 +72 0 15807 14.968241 109431,1034,1044,1046,1438,59199,1338,1317 14.062500,13.437500,12.875000,12.875000,12.562500,12.312500,12.125000,12.062500 +73 0 6766 11.121965 1044,1034,1046,109431,1438,1338,2251,1042 13.375000,13.062500,13.000000,12.500000,12.437500,12.187500,12.062500,12.000000 +74 0 1047 4.584736 1044,1046,109431,59199,1034,1048,1041,1338 14.000000,13.625000,13.250000,13.187500,13.062500,12.687500,12.625000,12.500000 +75 0 1676 11.672932 59199,1034,1319,1438,2251,1317,109431,1050 12.250000,11.812500,11.750000,11.750000,11.562500,11.500000,11.500000,11.375000 +76 0 12204 15.654541 1044,109431,1046,59199,1034,1048,1053,1319 13.375000,13.125000,12.937500,12.625000,12.437500,12.375000,12.312500,12.187500 +77 0 1047 6.514873 1034,1044,1046,1438,1042,1278,1317,1338 12.375000,12.312500,11.937500,11.937500,11.625000,11.500000,11.500000,11.500000 +78 0 3326 11.422932 59199,1034,1319,1438,2251,1317,109431,1050 12.250000,11.812500,11.750000,11.750000,11.562500,11.500000,11.500000,11.375000 +79 0 2553 7.708835 1034,109431,1044,1046,1278,1338,1438,59199 12.750000,12.750000,12.437500,12.437500,11.750000,11.750000,11.750000,11.687500 +80 0 1097 9.623739 1034,109431,1046,1044,1438,59199,1338,1042 13.062500,12.937500,12.250000,12.187500,12.125000,12.062500,11.750000,11.687500 +81 0 1047 6.252901 1034,1044,1046,1438,1338,109431,1319,59199 12.312500,12.125000,11.687500,11.687500,11.500000,11.500000,11.375000,11.375000 +82 0 1051 5.516682 59199,1034,1319,1438,2251,1317,109431,1050 12.250000,11.812500,11.750000,11.750000,11.562500,11.500000,11.500000,11.375000 +83 0 1046 3.679413 1034,59199,1438,109431,1044,1046,1338,1256 12.937500,12.562500,12.500000,12.375000,12.187500,12.187500,12.000000,11.812500 +84 0 1048 5.436569 1256,59199,1034,1438,1032,2251,1338,1603 12.250000,12.125000,12.000000,11.875000,11.687500,11.625000,11.375000,11.312500 +85 0 85501 14.098159 1034,1438,1044,1338,1317,1046,59199,109431 12.687500,12.375000,12.187500,11.937500,11.875000,11.812500,11.437500,11.437500 +86 0 5780 13.317652 1034,1438,1042,1046,1044,59199,2251,109431 13.437500,12.937500,12.437500,12.375000,12.250000,12.250000,12.187500,12.062500 +87 0 5604 15.242256 1034,109431,1317,59199,1438,1044,1408,1414 13.937500,13.750000,13.687500,13.437500,13.375000,12.937500,12.750000,12.625000 +88 0 1058 4.260566 1044,1034,1046,1438,109431,1338,1041,1319 12.875000,12.750000,12.437500,12.375000,12.250000,11.875000,11.625000,11.562500 +89 0 40377 12.071075 1256,59199,1603,1032,1364,2251,1429,1317 13.062500,12.625000,12.312500,12.250000,12.000000,12.000000,11.750000,11.625000 +90 0 79858 14.427614 1044,1046,1034,1338,2251,109431,1438,1317 13.500000,13.375000,12.937500,12.500000,12.500000,12.375000,11.937500,11.875000 +91 0 1044 2.354645 1044,1046,1034,1338,2251,1317,1438,1041 13.437500,13.125000,12.625000,12.562500,12.062500,12.000000,11.750000,11.687500 +92 0 37910 9.368361 1256,59199,1032,1603,2251,1438,1034,1364 12.187500,11.875000,11.562500,11.437500,11.312500,11.250000,11.187500,11.187500 +93 0 17271 14.133179 109431,1044,1034,1046,1438,2251,59199,1317 13.562500,13.500000,13.312500,12.937500,12.875000,12.875000,12.562500,12.375000 +94 0 26554 8.808447 109431,1044,1034,1046,1438,59199,1041,1338 13.812500,13.187500,13.000000,13.000000,12.687500,12.250000,11.937500,11.875000 +95 0 11071 9.528511 1044,1034,1046,1317,109431,1438,2251,1338 13.125000,12.750000,12.625000,12.250000,12.187500,12.125000,12.000000,11.937500 +96 0 1010 5.965549 1044,1034,1046,109431,1438,59199,2251,1338 13.250000,13.062500,12.875000,12.875000,12.625000,12.562500,12.500000,12.000000 +97 0 6359 9.382212 1256,59199,11,1438,1032,1034,1338,109431 12.875000,12.375000,12.187500,12.187500,11.937500,11.937500,11.687500,11.625000 +98 0 1319 4.326357 1034,1438,59199,1046,1044,109431,1042,2251 13.187500,12.437500,12.437500,12.375000,12.312500,12.312500,12.000000,11.937500 +99 0 1083 10.134815 1256,59199,1032,2251,1603,1317,1438,1364 12.625000,12.250000,11.687500,11.687500,11.500000,11.437500,11.437500,11.375000 +100 0 1046 3.880977 1034,1044,1438,109431,59199,1046,2251,1317 13.062500,12.375000,12.312500,12.250000,12.187500,11.812500,11.812500,11.750000 +101 0 10007 9.811569 1256,59199,1034,1438,1032,2251,1338,1603 12.250000,12.125000,12.000000,11.875000,11.687500,11.625000,11.375000,11.312500 +102 0 1532 12.315712 109431,1034,1046,1044,11,59199,1317,1338 13.812500,13.250000,13.250000,13.000000,12.687500,12.500000,12.312500,12.312500 +103 0 1044 2.715154 1044,109431,1046,1034,59199,1338,1041,1048 12.750000,12.687500,12.500000,12.062500,12.062500,11.750000,11.625000,11.625000 +104 0 1464 6.712111 1256,59199,1032,1603,2251,1438,1034,1364 12.187500,11.875000,11.562500,11.437500,11.312500,11.250000,11.187500,11.187500 +105 0 1046 3.373620 109431,1034,1044,1046,59199,1317,1338,1319 12.937500,12.687500,12.625000,12.437500,12.187500,12.125000,12.125000,11.937500 +106 0 60018 16.303757 1256,59199,1034,1438,1032,2251,1338,1603 12.250000,12.125000,12.000000,11.875000,11.687500,11.625000,11.375000,11.312500 +107 0 2140 9.907478 1034,109431,1044,1046,59199,1050,1319,2251 13.750000,13.625000,13.187500,13.187500,13.000000,12.312500,12.062500,12.062500 +108 0 1550 7.651908 1044,109431,1046,1034,59199,1048,1041,1338 13.312500,13.125000,12.937500,12.625000,12.312500,12.062500,11.937500,11.937500 +109 0 12336 13.331959 109431,1044,1034,59199,1046,1317,1338,1319 12.750000,12.562500,12.437500,12.250000,12.187500,11.937500,11.937500,11.750000 +110 0 1277 10.842323 1044,109431,1034,1046,1438,2251,59199,1338 13.687500,13.687500,13.312500,13.187500,12.812500,12.812500,12.500000,12.250000 +111 0 1044 3.228549 1034,1044,109431,1046,1278,1438,1261,1338 12.937500,12.500000,12.312500,12.250000,11.812500,11.812500,11.687500,11.687500 +112 0 1434 7.774611 1256,59199,1032,1603,2251,1438,1034,1364 12.187500,11.875000,11.562500,11.437500,11.312500,11.250000,11.187500,11.187500 +113 0 1046 3.708103 1034,109431,1438,59199,1044,1046,1338,2251 13.000000,12.625000,12.562500,12.562500,12.437500,12.062500,11.937500,11.812500 +114 0 4493 6.686569 1256,59199,1034,1438,1032,2251,1338,1603 12.250000,12.125000,12.000000,11.875000,11.687500,11.625000,11.375000,11.312500 +115 0 5757 16.356056 1034,1046,1044,1317,1438,59199,1338,2251 12.562500,12.250000,12.125000,12.062500,12.000000,12.000000,11.937500,11.812500 +116 0 2251 4.494226 1034,1044,1046,109431,1338,59199,1261,1438 13.312500,13.187500,12.937500,12.875000,12.187500,12.125000,12.062500,11.937500 +117 0 1429 4.291838 1256,1364,59199,1603,1032,1278,2251,2129 12.562500,12.250000,12.125000,12.062500,12.000000,12.000000,11.937500,11.875000 +118 0 28218 15.201368 1256,59199,2251,1032,1603,1319,1438,1364 12.687500,12.437500,12.187500,12.000000,11.812500,11.750000,11.750000,11.687500 +119 0 1335 8.560122 1034,109431,1044,59199,1046,1438,1041,1414 13.937500,13.937500,13.687500,13.250000,13.187500,13.125000,12.375000,12.312500 +120 0 102817 17.327961 1034,1044,109431,59199,1317,1046,1438,1338 12.500000,12.375000,12.250000,12.187500,12.125000,12.000000,11.937500,11.875000 +121 0 13064 12.507549 1034,1044,1438,109431,1046,59199,2251,1319 13.687500,13.687500,13.125000,13.125000,12.625000,12.562500,12.250000,12.187500 +122 0 10773 17.097748 109431,1046,1034,1044,59199,1338,1319,2251 14.062500,13.312500,13.062500,13.000000,12.500000,12.125000,12.062500,12.062500 +123 0 52928 12.979118 1044,1034,1046,1307,1438,1338,109431,2251 12.812500,12.750000,12.437500,12.375000,12.062500,11.812500,11.812500,11.500000 +124 0 1897 5.900103 1044,1034,2251,1046,1438,1317,1338,1319 13.437500,12.875000,12.750000,12.687500,12.500000,12.375000,12.187500,12.000000 +125 0 1032 4.111757 1034,59199,1438,2251,1032,1256,1319,1050 12.875000,12.750000,12.187500,11.687500,11.500000,11.500000,11.437500,11.250000 +126 0 1050 3.931108 1256,59199,1032,2251,1050,1603,1317,1319 13.312500,12.500000,12.312500,12.250000,12.000000,12.000000,11.937500,11.937500 +127 0 1048 6.064596 1034,1044,1438,1317,109431,1046,1338,2251 12.687500,12.250000,12.125000,12.000000,12.000000,11.875000,11.875000,11.687500 diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1ctx512.tsv b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1ctx512.tsv new file mode 100644 index 000000000..80b873749 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w1ctx512.tsv @@ -0,0 +1,135 @@ +# model models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit +# device GPU +# corpus tests/fixtures/wikitext2_excerpt.txt tokens 301749 +# chunks 128 chunk_tokens 1 topk 8 prefill 512 +# columns chunk pos target nll top_ids top_logits +# start_token 512 +# contiguous false +0 0 1048 0.049204 1048,24926,1052,1805,1054,1462,1053,1056 24.375000,20.125000,19.500000,19.375000,18.875000,18.375000,17.875000,17.625000 +1 0 1048 0.259848 1048,1049,1050,1462,1411,2545,40438,15011 25.500000,24.250000,20.500000,18.125000,17.750000,17.375000,17.125000,16.500000 +2 0 1054 1.249781 1055,1054,1056,1053,1057,1052,1051,1050 24.125000,23.875000,23.250000,22.750000,22.375000,20.750000,20.375000,20.250000 +3 0 15011 0.478422 15011,19895,41500,22876,14696,8281,17058,5395 20.250000,18.750000,17.750000,17.625000,17.000000,16.500000,16.375000,15.937500 +4 0 1307 0.006128 1307,2126,1319,29818,1408,4418,28140,1576 26.125000,19.250000,19.000000,18.750000,18.625000,18.125000,18.125000,17.875000 +5 0 1278 0.338804 1278,1531,51288,19895,22876,109492,2813,1429 20.000000,17.000000,16.500000,16.125000,16.125000,15.937500,15.812500,15.750000 +6 0 14696 0.278820 14696,19895,22876,7520,8281,41500,81873,9371 20.875000,18.750000,18.250000,17.375000,16.750000,16.125000,16.125000,15.625000 +7 0 5395 0.128090 5395,25865,18823,41500,2629,3467,14593,18944 22.625000,19.750000,19.250000,18.000000,17.875000,17.875000,16.875000,16.750000 +8 0 3897 6.084591 51288,1531,2813,43193,15087,109492,1488,1373 17.500000,16.875000,16.500000,16.250000,15.750000,15.562500,15.375000,15.000000 +9 0 43193 4.245365 2813,1531,51288,1799,3140,109492,19148,43193 15.750000,15.437500,14.750000,14.562500,14.562500,14.000000,13.812500,13.812500 +10 0 1839 0.000273 1839,1606,1270,1428,6094,69555,2398,82089 27.500000,18.875000,16.375000,15.937500,15.750000,15.750000,15.687500,15.187500 +11 0 3897 1.171562 2247,3897,1435,18988,1294,4295,11159,12440 22.500000,22.250000,20.625000,20.500000,19.750000,19.625000,19.625000,19.375000 +12 0 7125 9.735878 12440,1435,1294,22594,4418,28140,2478,1321 23.000000,21.250000,20.250000,19.500000,19.250000,19.125000,19.000000,18.625000 +13 0 1536 0.029520 1536,1294,1454,29598,2015,1278,1261,1408 25.875000,21.375000,20.125000,20.125000,20.000000,19.750000,18.750000,18.625000 +14 0 1261 0.454359 1261,1420,1278,3866,24836,1794,15720,12440 21.625000,19.750000,19.000000,19.000000,19.000000,18.750000,18.000000,17.375000 +15 0 4934 1.459671 4934,24836,17317,47715,96432,1032,68393,3709 19.375000,19.375000,18.375000,18.250000,18.125000,17.500000,17.500000,17.375000 +16 0 1294 0.271597 1294,1435,1408,22594,12440,1307,3897,15019 24.875000,23.375000,21.250000,21.125000,20.750000,19.625000,19.000000,18.750000 +17 0 1278 0.089676 1278,1032,1261,22876,1531,1420,19895,3866 21.125000,16.750000,16.750000,16.500000,16.125000,15.875000,15.312500,15.000000 +18 0 1032 1.837973 3354,1032,4076,14696,19895,4958,22876,37848 19.625000,19.000000,18.625000,18.625000,18.500000,17.250000,17.000000,16.500000 +19 0 1050 0.003214 1050,1049,1055,1051,1010,1053,1052,1054 27.500000,20.750000,19.625000,19.375000,19.250000,19.125000,19.000000,18.625000 +20 0 1048 0.001022 1048,1054,1051,1055,1052,1053,1056,1298 26.375000,17.250000,17.125000,17.125000,17.000000,17.000000,16.875000,16.875000 +21 0 1048 0.018784 1048,1049,1050,1411,1055,1462,1051,2247 27.500000,23.500000,19.250000,19.125000,16.625000,16.375000,15.687500,15.062500 +22 0 1055 0.310822 1055,1056,1054,1057,1053,1052,1051,1050 25.750000,24.125000,23.625000,22.500000,20.875000,18.625000,18.000000,17.375000 +23 0 37848 3.002918 4076,14696,19895,3354,37848,22876,4958,15011 19.250000,18.500000,17.875000,17.750000,17.250000,16.625000,16.375000,16.375000 +24 0 5466 0.065973 5466,53458,3354,63239,6267,25349,2629,13079 25.000000,21.750000,21.000000,18.625000,18.250000,18.125000,17.625000,17.625000 +25 0 1307 0.050234 1307,3897,1531,1576,1294,2247,29818,4418 23.250000,18.875000,17.750000,17.625000,16.750000,16.500000,16.500000,16.125000 +26 0 3075 8.192232 1278,1531,1429,15723,22511,1349,6959,19420 19.000000,17.500000,15.312500,15.187500,15.187500,14.937500,14.437500,14.187500 +27 0 1317 2.794657 1278,1362,1531,1317,3870,9595,2247,4159 18.125000,17.000000,16.750000,16.000000,14.750000,14.750000,14.500000,13.812500 +28 0 18275 1.499737 40659,18275,3094,64561,93616,19008,13516,107178 17.875000,17.500000,17.125000,15.937500,15.687500,15.437500,15.187500,14.937500 +29 0 1415 0.115271 1415,3108,1419,1938,39664,86674,2392,3741 20.500000,17.875000,16.875000,15.500000,15.250000,14.750000,14.500000,12.750000 +30 0 15179 15.787373 1294,1278,1454,1261,2247,1513,1321,1420 21.750000,19.375000,19.250000,19.125000,19.000000,18.500000,17.875000,17.750000 +31 0 1536 0.010662 1536,2790,8360,1321,6421,1513,2126,5785 25.750000,19.875000,19.375000,19.125000,18.500000,18.375000,18.375000,17.375000 +32 0 8233 4.690252 15723,26004,26009,16566,6031,3986,7456,16136 16.750000,16.625000,15.937500,15.562500,15.375000,15.312500,15.187500,15.125000 +33 0 1386 0.000467 1386,3390,16926,97044,2755,33151,8101,1704 23.375000,13.937500,13.437500,13.312500,13.125000,13.125000,12.812500,12.500000 +34 0 1434 0.232876 1434,10307,1757,1535,1550,7456,13225,16006 18.250000,15.250000,15.125000,14.437500,14.062500,13.875000,13.187500,13.187500 +35 0 1479 1.257416 10917,1479,1539,2765,25763,15392,47111,4940 15.125000,14.875000,13.750000,12.437500,11.937500,11.250000,11.250000,11.187500 +36 0 1965 0.000132 1965,1107,2247,1121,1985,12623,1046,1837 25.125000,15.375000,13.937500,13.562500,13.062500,12.375000,12.250000,12.125000 +37 0 2247 0.463770 2247,1513,1321,3897,1294,1454,18988,47715 24.375000,22.875000,22.500000,21.500000,21.375000,20.500000,20.500000,19.625000 +38 0 3075 7.250097 2182,1032,1656,1531,2409,8052,89656,6304 20.250000,19.625000,19.500000,18.750000,18.375000,17.875000,17.250000,17.000000 +39 0 1317 0.044668 1317,3870,1278,9327,6793,4058,6421,1576 22.375000,18.750000,16.125000,15.500000,15.437500,15.187500,15.187500,15.000000 +40 0 18275 0.010262 18275,121786,1359,56669,83955,6421,44477,43060 26.125000,20.250000,19.875000,18.750000,18.625000,18.500000,18.000000,17.500000 +41 0 1415 0.000036 1415,1314,1672,2247,1778,32571,3108,1419 27.000000,15.500000,14.500000,13.875000,13.812500,13.562500,13.500000,13.312500 +42 0 1486 0.789555 1486,1395,20209,56671,18166,3897,57062,8308 21.000000,20.250000,18.125000,18.000000,17.875000,17.625000,17.625000,17.500000 +43 0 5537 0.325165 5537,8360,2158,1261,1420,2095,4057,8460 22.125000,20.375000,18.750000,18.250000,18.000000,17.625000,17.625000,17.500000 +44 0 1513 0.181145 1513,1294,1435,22594,1394,1317,1536,1562 25.250000,23.000000,21.500000,20.750000,20.625000,20.000000,19.750000,19.750000 +45 0 27640 9.979752 1278,2960,22410,1531,1335,1261,34614,7893 23.125000,18.875000,18.250000,17.625000,17.375000,16.875000,16.375000,15.687500 +46 0 22410 0.008114 22410,10786,27677,2247,19404,35229,1109,28869 23.375000,17.125000,17.125000,15.937500,15.750000,15.625000,15.500000,15.500000 +47 0 1294 1.246471 1321,1294,2247,3897,1454,4295,2478,1435 22.750000,22.625000,22.500000,20.500000,20.000000,19.250000,19.125000,19.000000 +48 0 1278 10.314337 7893,1032,36541,11250,2247,1278,75728,5663 25.750000,19.000000,16.625000,15.812500,15.500000,15.437500,15.375000,15.187500 +49 0 7893 2.064966 5046,7893,3354,1032,14719,2158,10485,9959 17.500000,16.500000,16.375000,15.812500,14.875000,14.562500,14.562500,14.312500 +50 0 81947 0.193039 81947,13375,2247,70574,3714,37848,5046,4457 20.625000,17.750000,17.625000,17.375000,16.125000,16.125000,15.312500,15.000000 +51 0 1307 0.004328 1307,1281,9214,2247,6421,18988,11159,35549 26.125000,19.500000,19.000000,18.875000,17.625000,17.625000,17.125000,16.500000 +52 0 38199 2.967102 108492,53875,38199,3316,16210,45261,7893,35549 21.500000,21.000000,19.125000,18.000000,17.500000,16.875000,16.750000,16.375000 +53 0 1428 0.000006 1428,1920,52636,8315,49887,1966,2247,58401 30.750000,18.125000,17.250000,16.125000,15.250000,14.625000,14.500000,14.500000 +54 0 52829 0.000177 52829,23825,10786,6989,1425,11564,1109,40158 25.875000,16.000000,15.250000,14.812500,14.437500,14.062500,13.687500,13.562500 +55 0 1321 0.020554 1321,2247,1748,18988,11159,4295,3897,6421 25.750000,21.375000,19.625000,19.500000,18.750000,18.125000,18.000000,17.375000 +56 0 118799 0.012533 118799,38199,75728,44841,10330,21374,2247,39185 22.000000,16.250000,16.000000,15.687500,14.500000,14.250000,14.187500,14.187500 +57 0 2247 0.335772 2247,18988,1321,11159,1294,3897,1454,4295 23.750000,21.750000,21.500000,20.625000,20.125000,20.125000,19.250000,19.000000 +58 0 89656 4.957855 1032,1656,2182,1531,8052,2409,2157,6304 20.000000,19.625000,19.500000,18.500000,17.625000,17.125000,16.875000,16.750000 +59 0 1397 0.001352 1397,5032,1400,3701,1868,2683,1781,1923 24.125000,15.875000,15.812500,15.812500,14.750000,14.187500,13.687500,13.437500 +60 0 56671 3.455008 8308,10354,1576,1880,2095,1486,56671,5537 20.500000,19.000000,18.750000,18.750000,18.500000,18.250000,18.000000,17.750000 +61 0 1294 2.072529 22594,1294,1435,1454,15019,1408,2790,1513 25.250000,23.375000,22.125000,20.125000,19.625000,19.000000,18.875000,18.625000 +62 0 2295 5.073614 1278,1261,1420,1531,1032,1593,2295,22876 20.250000,18.250000,16.375000,15.875000,15.500000,15.437500,15.437500,15.250000 +63 0 13068 4.059656 23981,4958,1032,11733,14696,4514,13068,19895 20.125000,18.000000,17.250000,17.250000,17.000000,16.500000,16.500000,16.250000 +64 0 1294 1.010454 1294,7358,15179,3897,8360,8077,1455,12800 20.625000,19.750000,19.375000,18.875000,18.625000,18.250000,18.125000,17.375000 +65 0 1032 0.022177 1032,1278,3897,5124,7276,6032,1766,7586 25.500000,20.875000,18.625000,18.625000,18.250000,18.125000,18.000000,17.750000 +66 0 1050 0.000196 1050,1049,1052,1048,1032,1051,1056,1053 26.625000,16.625000,16.125000,15.875000,15.250000,15.250000,15.125000,15.062500 +67 0 1048 0.000151 1048,1032,1050,1049,3897,1051,1010,1052 26.250000,16.250000,15.562500,14.750000,14.187500,13.812500,13.500000,13.500000 +68 0 1048 0.113371 1048,1049,1050,1737,2247,1056,1032,1051 27.000000,24.875000,19.500000,14.750000,14.750000,14.500000,14.187500,14.062500 +69 0 1056 0.791615 1056,1055,1057,1054,1053,1052,1051,1050 23.750000,23.375000,22.750000,21.625000,20.000000,18.250000,17.375000,17.125000 +70 0 3897 1.306955 2247,3897,1737,1775,4295,1319,4254,15179 22.000000,21.625000,20.125000,20.125000,20.125000,19.000000,18.750000,18.750000 +71 0 10298 8.818853 15179,2960,1278,1531,3907,28140,1799,1925 15.812500,15.687500,15.500000,15.437500,15.312500,15.312500,14.937500,14.375000 +72 0 7421 1.163446 1307,7421,5661,2247,1032,15247,32022,53728 18.500000,18.125000,16.750000,16.500000,15.312500,14.937500,14.687500,14.687500 +73 0 9225 1.973724 9225,1576,92791,1321,7114,1681,1766,1353 16.125000,15.937500,15.875000,15.812500,14.437500,14.375000,14.375000,14.187500 +74 0 67465 0.000202 67465,14397,1575,6575,2097,2249,1616,3862 33.000000,24.375000,21.500000,21.500000,18.125000,16.125000,15.437500,15.375000 +75 0 1536 6.314869 1321,15179,22594,1319,3897,8360,47715,2247 23.875000,21.625000,21.250000,20.875000,20.750000,20.250000,19.250000,19.000000 +76 0 130537 5.049042 41520,16566,3986,7456,6031,12245,15723,36356 17.250000,16.875000,16.500000,16.500000,16.375000,16.375000,16.375000,16.250000 +77 0 6993 11.021158 6031,7456,3986,15723,16566,1349,8720,46212 17.250000,16.875000,16.750000,16.375000,16.125000,15.937500,15.812500,15.687500 +78 0 2301 6.511566 2163,10307,15450,13586,5303,2301,2247,39316 19.250000,17.000000,14.500000,13.625000,13.187500,12.875000,12.250000,11.875000 +79 0 125371 2.120897 125371,11461,48227,7976,1398,73607,51039,1760 13.250000,13.125000,12.625000,12.312500,11.812500,11.625000,11.125000,11.000000 +80 0 3897 1.869087 4728,1321,3897,75094,1305,4295,3614,2247 20.000000,19.875000,19.000000,16.625000,16.375000,15.812500,15.562500,15.187500 +81 0 1321 0.440508 1321,22594,1799,7125,47715,1294,12440,2478 21.875000,20.000000,19.500000,19.375000,19.125000,18.000000,17.625000,17.500000 +82 0 6797 7.101797 1531,1278,1294,2095,1656,29251,1349,1434 17.250000,15.812500,15.062500,14.500000,14.312500,14.187500,14.000000,14.000000 +83 0 3892 1.300848 3892,1576,5303,14258,60878,20040,2405,6174 17.000000,16.500000,16.000000,14.937500,14.750000,14.437500,14.375000,14.375000 +84 0 1390 7.023306 2247,1536,3897,2662,2314,10689,4295,1319 15.875000,14.437500,14.125000,13.812500,13.250000,12.625000,12.500000,12.375000 +85 0 8919 0.000053 8919,61245,1906,1518,6465,3257,1783,12200 26.375000,15.312500,14.125000,13.937500,13.937500,13.562500,13.437500,13.000000 +86 0 15179 0.969007 15179,2247,1536,47715,3897,1319,8360,4295 21.250000,21.125000,19.375000,19.125000,19.000000,18.625000,18.500000,18.250000 +87 0 1536 0.004633 1536,1321,3897,22594,2790,1294,2126,2247 25.625000,18.750000,17.625000,17.500000,17.375000,17.250000,17.250000,17.250000 +88 0 8595 1.739104 8595,26305,43252,36356,33407,40006,25656,7114 13.562500,13.250000,11.750000,11.625000,11.312500,11.250000,11.187500,10.875000 +89 0 1405 0.082834 1405,1269,5625,2302,6291,11254,19019,3230 20.875000,17.625000,17.375000,15.000000,14.687500,14.437500,14.250000,13.687500 +90 0 88542 0.827280 88542,10307,28958,26515,2247,78734,66533,84292 16.000000,14.625000,14.625000,13.250000,13.125000,12.750000,12.625000,12.562500 +91 0 2247 0.125179 2247,18988,11159,4295,3897,1319,1321,1294 25.125000,22.500000,21.125000,21.000000,20.875000,18.875000,18.875000,18.625000 +92 0 1656 3.092863 2182,1032,12953,6797,1531,1656,10298,6304 21.000000,20.125000,19.875000,19.125000,19.000000,18.875000,18.750000,17.750000 +93 0 4454 6.100456 1032,1278,2960,6346,6866,6633,4454,5821 24.625000,21.625000,19.875000,19.250000,19.000000,18.750000,18.625000,18.500000 +94 0 1032 0.081915 1032,1307,3897,1321,1462,1278,1514,6633 23.750000,20.500000,19.625000,19.125000,17.250000,16.875000,16.875000,16.750000 +95 0 1050 0.000898 1050,1048,1049,1051,1056,1053,1054,1052 27.250000,19.000000,18.750000,18.250000,17.750000,17.500000,17.250000,17.125000 +96 0 1048 0.000654 1048,1032,1050,1055,1053,1051,1057,1056 26.000000,16.625000,16.500000,16.375000,16.250000,16.000000,16.000000,15.937500 +97 0 1048 0.584106 1048,1049,1050,1057,3897,1051,1056,2247 25.000000,24.750000,20.750000,15.687500,14.437500,14.312500,13.687500,13.687500 +98 0 1056 0.931129 1057,1056,1055,1054,1053,1052,1050,1051 24.500000,24.125000,21.125000,20.000000,19.500000,18.125000,17.375000,17.250000 +99 0 3897 0.677832 3897,1514,89656,1278,1794,43378,1321,2247 19.875000,19.125000,18.625000,17.000000,15.562500,14.812500,14.562500,14.312500 +100 0 89656 1.101381 1514,89656,1278,1531,10298,1494,1794,2182 17.125000,17.125000,15.687500,15.000000,14.187500,13.687500,13.250000,13.000000 +101 0 1397 0.009214 1397,5032,1400,3701,21985,2306,31814,1781 21.125000,15.125000,14.812500,14.625000,13.750000,12.875000,12.437500,12.125000 +102 0 3214 4.642870 8308,10354,5537,1486,1880,52158,37101,56671 19.000000,18.500000,18.000000,17.875000,17.625000,17.625000,17.375000,17.250000 +103 0 1261 0.624857 1261,1794,1420,24836,96432,1278,3866,29857 23.750000,23.250000,22.250000,18.625000,18.375000,18.250000,17.625000,17.625000 +104 0 24836 0.729671 24836,96432,4958,13426,6061,1850,8602,4076 20.500000,20.125000,18.375000,17.250000,17.000000,16.500000,16.000000,15.875000 +105 0 13988 0.020309 13988,47715,96432,14696,8281,2126,9655,29857 23.250000,17.500000,17.125000,17.000000,16.375000,16.250000,16.125000,15.812500 +106 0 1408 0.479612 1408,1294,1435,22594,12440,1454,1513,1321 25.250000,24.500000,22.750000,21.875000,20.500000,19.125000,19.125000,18.875000 +107 0 1261 5.023712 1278,22876,1531,19895,14279,1420,1261,14696 20.250000,17.875000,17.625000,17.625000,16.250000,15.875000,15.562500,15.375000 +108 0 2295 6.726089 19895,22876,6061,78172,14696,1875,12321,41500 16.500000,16.500000,16.375000,15.875000,15.750000,15.500000,15.500000,15.312500 +109 0 2126 6.839066 24926,1462,1805,109431,1045,50824,2126,15011 23.375000,20.125000,19.750000,18.625000,17.750000,17.750000,16.625000,16.625000 +110 0 1045 0.066589 1045,1462,93023,1032,1123,18242,1925,1064 21.125000,17.875000,15.062500,15.000000,14.687500,14.375000,13.750000,13.687500 +111 0 1064 0.001776 1064,93023,115587,2126,73856,59962,3332,1050 27.250000,20.375000,18.875000,18.000000,18.000000,17.125000,16.625000,16.125000 +112 0 1805 0.081581 1805,12271,15011,1635,31194,14400,24926,106817 23.625000,20.250000,20.125000,18.500000,18.125000,16.500000,16.500000,16.375000 +113 0 15011 0.108194 15011,8303,41500,6061,19895,5395,13268,2247 20.875000,16.375000,16.375000,16.125000,15.937500,15.625000,15.437500,15.125000 +114 0 19976 13.704632 1307,29818,1408,4418,1294,1319,28140,3897 25.875000,20.625000,20.375000,19.125000,19.000000,18.750000,18.750000,18.250000 +115 0 1307 0.368771 1307,1408,1294,29818,28140,1435,1394,4418 22.750000,21.625000,19.625000,18.500000,18.375000,18.125000,17.125000,16.750000 +116 0 1278 0.110964 1278,1531,15087,2813,3797,22876,43193,14696 20.625000,16.250000,15.625000,15.250000,15.187500,15.187500,15.187500,15.062500 +117 0 14696 0.483349 14696,19895,7520,22876,9371,81873,5150,8281 20.125000,18.250000,18.000000,18.000000,17.125000,15.812500,15.750000,15.562500 +118 0 5395 0.103451 5395,25865,18823,2629,14593,50584,3467,119435 22.500000,19.500000,18.875000,17.875000,17.375000,16.625000,16.000000,15.812500 +119 0 1488 3.685127 3897,15087,51288,1531,2813,43193,1488,10495 17.500000,15.937500,15.500000,15.375000,15.375000,15.250000,14.937500,14.375000 +120 0 6812 0.323984 6812,89220,24591,6970,10178,2939,5635,25632 19.250000,18.125000,14.625000,14.250000,13.437500,13.375000,13.187500,13.187500 +121 0 1278 0.025702 1278,1531,8250,29251,3897,2247,2015,1032 21.500000,17.125000,15.687500,15.375000,14.437500,14.250000,14.125000,13.625000 +122 0 29251 0.030287 29251,19213,1407,8776,39447,49944,3897,49377 21.875000,16.000000,15.687500,15.687500,15.375000,15.312500,15.125000,14.937500 +123 0 3897 3.342592 2247,18988,22594,1435,3897,11159,29818,1454 22.625000,20.875000,19.875000,19.750000,19.750000,19.750000,19.250000,19.000000 +124 0 7125 3.889521 12440,28140,1435,1294,29818,7125,22594,8308 20.625000,17.750000,17.500000,17.375000,17.375000,17.125000,17.125000,16.875000 +125 0 1536 0.068686 1536,1294,29598,2015,4025,1261,1278,17558 25.625000,22.250000,21.000000,20.250000,20.000000,19.625000,19.375000,19.375000 +126 0 1420 2.181309 1261,1420,3866,24836,1794,1278,15720,29857 22.250000,20.500000,19.875000,19.625000,19.250000,19.000000,18.250000,17.375000 +127 0 13988 0.388712 13988,15011,1457,7481,47353,87170,107254,1408 21.250000,19.750000,19.000000,16.875000,16.625000,16.000000,16.000000,15.875000 diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w8.tsv b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w8.tsv new file mode 100644 index 000000000..7652a2c4d --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_nemotron-3-nano-30b-a3b_default_w8.tsv @@ -0,0 +1,647 @@ +# model models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit +# device GPU +# corpus tests/fixtures/wikitext2_excerpt.txt tokens 301749 +# chunks 80 chunk_tokens 8 topk 8 prefill 512 +# columns chunk pos target nll top_ids top_logits +# start_token 0 +# contiguous false +0 0 20211 14.958650 1256,59199,1032,1050,1603,1049,2251,1319 13.000000,12.937500,12.562500,12.312500,12.250000,11.875000,11.875000,11.812500 +0 1 1752 5.124104 1327,3522,1275,1586,1612,1631,1485,1329 15.750000,14.875000,14.750000,14.312500,13.812500,13.625000,13.375000,12.625000 +0 2 1637 0.026117 1637,1331,1120,1325,55906,3002,3173,11310 21.500000,17.125000,15.875000,15.875000,14.937500,14.687500,14.562500,13.937500 +0 3 1045 2.398348 109431,1044,1045,1395,1319,1040,1050,1256 16.125000,15.187500,15.000000,13.687500,13.562500,13.500000,13.187500,13.125000 +0 4 1050 0.534906 1050,1049,1054,1051,1055,1053,1052,1032 23.750000,23.375000,18.375000,17.875000,17.250000,17.125000,16.750000,16.250000 +0 5 7637 8.982275 1044,1319,16910,1010,1395,1454,1321,10835 15.187500,14.375000,13.937500,13.375000,13.187500,12.937500,12.812500,12.812500 +0 6 103920 6.722225 7916,2016,2683,44376,109431,85980,6672,114040 19.250000,16.625000,15.687500,15.562500,15.437500,15.187500,15.062500,14.875000 +0 7 1376 8.853880 1319,1044,1438,3871,1010,1034,1256,1042 16.125000,15.875000,15.750000,15.437500,15.187500,15.187500,14.875000,14.812500 +1 0 1267 4.451839 1032,1256,1349,1319,1429,1766,1267,1424 14.750000,13.187500,13.125000,12.875000,12.562500,12.562500,12.500000,12.500000 +1 1 8263 9.639622 1784,1042,1124,1045,1065,4380,1087,1049 20.250000,18.625000,18.375000,17.750000,17.750000,17.750000,17.500000,17.375000 +1 2 1260 7.501720 1058,1737,1307,1256,1877,1319,1010,3403 18.875000,16.500000,15.562500,15.250000,14.875000,14.750000,14.562500,14.250000 +1 3 1737 2.567188 1312,1737,30864,36609,1376,23762,17952,27675 19.750000,17.625000,15.687500,15.687500,15.625000,15.562500,15.250000,15.250000 +1 4 7155 0.658005 7155,1534,1032,1766,1256,16871,15783,40377 18.750000,17.000000,16.875000,15.812500,15.437500,15.250000,14.875000,14.875000 +1 5 2345 0.000458 2345,1058,1555,1737,73686,25771,2164,1895 25.875000,17.500000,15.000000,14.937500,14.937500,14.875000,14.000000,13.687500 +1 6 1104 0.681343 1104,8027,1112,6132,1119,92390,1277,13843 21.000000,19.750000,19.375000,19.250000,18.125000,17.625000,17.500000,16.750000 +1 7 1936 0.000051 1936,1102,1103,1099,13444,2762,6754,8300 28.875000,18.750000,17.125000,15.375000,14.375000,13.500000,13.437500,13.312500 +2 0 7366 0.000175 7366,10553,2891,6683,14397,38102,15611,3333 26.375000,17.250000,15.312500,15.250000,14.500000,14.312500,13.812500,13.187500 +2 1 3136 0.000069 3136,17598,4788,33995,11303,57557,96166,2651 30.750000,20.875000,18.250000,18.250000,18.125000,17.750000,16.875000,16.250000 +2 2 20524 0.000540 20524,3256,3602,10081,84202,2354,3628,26577 26.625000,18.375000,17.125000,16.625000,16.250000,16.125000,15.062500,14.687500 +2 3 6406 0.122326 6406,30045,1047,27165,13126,23857,3766,33437 25.375000,23.000000,20.750000,20.750000,20.500000,18.875000,18.500000,18.375000 +2 4 71129 0.000017 71129,1269,18492,4339,24941,1115,31380,10191 30.625000,19.000000,17.375000,17.250000,17.125000,16.875000,16.500000,15.937500 +2 5 14327 7.947184 13001,10001,10383,3766,3826,27432,36362,59548 24.250000,19.750000,19.750000,19.125000,19.000000,18.875000,18.750000,18.500000 +2 6 3289 0.057474 3289,1299,4421,1107,1075,6174,3673,4910 20.375000,15.750000,15.187500,15.000000,14.375000,14.062500,13.937500,13.937500 +2 7 17271 0.027159 17271,41577,2559,1045,12082,7697,46142,2975 24.250000,20.250000,18.375000,18.125000,17.125000,17.000000,16.250000,16.000000 +3 0 13001 0.002577 13001,1047,76888,27385,56803,27432,16244,23409 26.500000,18.375000,18.375000,18.125000,18.000000,17.875000,17.750000,17.375000 +3 1 1552 0.000027 1552,36078,1116,6578,75934,9772,1889,1276 31.250000,20.375000,18.625000,18.625000,17.000000,15.312500,15.187500,14.375000 +3 2 1752 0.014561 1752,1290,3293,23953,1612,1386,2149,1318 25.500000,20.375000,20.000000,19.625000,17.625000,17.250000,16.875000,16.750000 +3 3 1637 0.000008 1637,10773,1120,1331,63931,29059,1325,1099 33.000000,20.000000,19.625000,19.625000,18.250000,18.125000,15.562500,15.437500 +3 4 1010 1.455435 1256,1010,1267,1045,1050,1293,11753,1260 22.000000,21.375000,21.250000,20.125000,19.250000,17.750000,17.375000,17.000000 +3 5 7324 6.536573 14209,5348,93552,10547,44149,35333,35273,56179 20.875000,20.250000,20.125000,19.875000,19.875000,19.750000,19.250000,19.125000 +3 6 2683 0.103832 2683,57744,24941,4994,48511,20150,61686,109431 23.125000,19.750000,18.750000,18.125000,18.125000,18.000000,17.875000,17.375000 +3 7 1260 1.575556 3539,1260,1652,2434,3848,43168,2564,1369 22.125000,21.625000,21.500000,21.375000,19.750000,19.625000,18.375000,17.125000 +4 0 1737 0.000212 1737,1376,123993,10869,37305,7295,4295,55462 31.375000,22.500000,20.250000,20.125000,19.875000,19.750000,18.875000,18.750000 +4 1 1285 2.834385 2688,1429,2000,1285,2129,1576,88564,7637 22.125000,21.000000,20.625000,20.125000,20.125000,19.875000,19.250000,19.125000 +4 2 1552 0.000946 1552,75934,36078,71625,11073,7857,55468,1049 27.750000,20.625000,17.750000,17.750000,16.875000,16.250000,15.687500,14.562500 +4 3 1752 0.001331 1752,1318,1290,23953,1612,3293,1631,1521 25.750000,18.250000,18.000000,15.812500,15.750000,15.562500,15.375000,15.250000 +4 4 1637 0.000023 1637,1120,10773,1331,63931,29059,56566,2510 29.750000,18.625000,17.750000,15.562500,15.500000,14.812500,14.750000,13.875000 +4 5 1045 0.397186 1045,109431,1050,1287,1044,1319,13683,1256 22.500000,21.625000,19.375000,17.375000,16.750000,16.125000,16.125000,15.562500 +4 6 1050 0.001482 1050,1049,1052,3416,1053,1057,1051,1054 29.750000,23.125000,19.625000,19.500000,18.875000,18.375000,18.250000,18.250000 +4 7 1045 2.008131 1010,1319,1045,109431,1256,1044,1032,1287 19.500000,19.500000,18.750000,18.500000,18.000000,17.500000,16.875000,16.250000 +5 0 3416 0.002117 3416,1049,5983,2132,1666,1050,33263,15828 24.500000,16.625000,16.500000,16.375000,15.562500,15.250000,15.250000,15.000000 +5 1 9332 0.204727 9332,11221,91785,1010,1045,1319,109431,36362 21.625000,18.875000,18.500000,18.250000,18.250000,17.125000,16.625000,15.187500 +5 2 1049 0.001196 1049,1045,1050,1453,22823,1299,2335,1095 24.375000,16.125000,15.750000,15.750000,14.750000,14.687500,14.687500,13.250000 +5 3 1010 0.554089 1010,1319,1256,1267,1044,1032,1046,1293 20.625000,19.625000,18.625000,17.875000,17.375000,17.000000,16.750000,16.250000 +5 4 44149 1.053345 44149,14209,10547,56179,73554,5348,10080,93552 21.875000,20.125000,20.000000,20.000000,19.875000,19.750000,19.500000,19.375000 +5 5 1652 0.012457 1652,3848,2564,2434,9928,1260,3539,9878 25.000000,19.250000,18.625000,18.500000,18.250000,17.875000,17.625000,16.875000 +5 6 1737 0.000309 1737,1376,10869,7295,55462,1312,7485,37305 31.250000,23.000000,19.875000,19.375000,18.875000,18.750000,18.625000,18.500000 +5 7 2688 0.058634 2688,1576,1364,18028,1429,2129,2000,1916 24.750000,20.875000,20.125000,19.625000,19.375000,19.250000,19.000000,18.625000 +6 0 1319 4.206076 1010,1256,1319,1267,1032,2250,1293,62717 22.375000,18.750000,18.250000,18.125000,17.375000,16.375000,15.562500,15.312500 +6 1 1052 5.316050 34094,33951,1049,3265,1050,7942,1270,7347 19.875000,19.750000,19.375000,19.000000,18.875000,18.375000,18.250000,18.250000 +6 2 1051 4.568428 1044,1048,1049,1056,59199,1057,1054,1055 20.750000,19.125000,17.500000,17.500000,17.500000,17.375000,17.125000,17.125000 +6 3 1053 3.202375 59199,1048,1051,1044,1056,1050,1055,1054 21.500000,19.750000,19.625000,19.500000,19.500000,19.375000,19.375000,19.125000 +6 4 1056 2.939169 59199,1050,1053,1055,1054,1056,34421,1057 19.750000,19.000000,18.250000,18.125000,18.000000,18.000000,17.750000,17.500000 +6 5 12964 8.159634 16647,35699,18028,14170,7280,11277,34421,6896 18.500000,18.250000,18.125000,17.750000,17.125000,17.125000,17.000000,16.875000 +6 6 1456 0.766512 1456,1044,1801,1041,10039,1307,8464,1775 24.000000,23.250000,22.875000,21.875000,21.375000,21.000000,20.500000,19.500000 +6 7 93552 5.416395 56179,33586,3210,29130,10547,5348,15063,7705 22.375000,22.250000,20.750000,20.750000,20.000000,19.875000,19.750000,19.625000 +7 0 1293 0.267027 1293,1032,1256,1260,1737,1652,1058,3539 25.750000,24.500000,21.500000,19.000000,18.750000,18.625000,18.250000,17.125000 +7 1 1737 0.000191 1737,1376,4295,10869,55462,7295,37305,1312 32.000000,23.000000,21.250000,20.375000,20.250000,20.125000,19.500000,19.000000 +7 2 25079 1.309153 60518,25079,44295,52558,9453,51641,1353,2000 21.750000,21.500000,21.250000,19.625000,19.125000,19.125000,18.875000,18.000000 +7 3 20992 0.948793 5630,20992,109431,1048,1045,9964,59199,1050 23.750000,23.375000,20.500000,19.625000,19.000000,19.000000,18.125000,17.000000 +7 4 5624 3.823448 9964,109431,5624,1032,1045,1010,21128,1882 27.375000,24.000000,23.625000,23.000000,22.125000,18.125000,18.125000,17.875000 +7 5 1065 0.000608 1065,1349,1080,66118,5909,25698,23303,3590 24.875000,15.625000,15.187500,15.000000,14.750000,14.500000,14.437500,14.312500 +7 6 1032 0.172388 1032,1045,1010,109431,1319,1330,1267,59199 24.125000,21.875000,21.000000,20.125000,19.375000,18.125000,18.000000,17.250000 +7 7 1051 1.706396 1052,1051,1050,1049,1056,1032,1055,1054 25.750000,24.250000,20.500000,17.625000,17.125000,16.875000,16.375000,15.625000 +8 0 1046 0.000950 1046,1047,1010,1048,1319,1043,109431,1882 25.000000,15.687500,15.562500,15.250000,15.187500,15.062500,14.812500,14.500000 +8 1 1048 0.000059 1048,1049,1032,1050,1053,1319,1051,1256 28.125000,17.250000,16.375000,15.562500,15.125000,14.937500,14.750000,14.125000 +8 2 1319 2.240608 1010,1319,1267,1256,1032,1987,1044,1775 23.375000,21.375000,20.250000,20.125000,18.875000,18.000000,17.875000,17.625000 +8 3 3299 3.119263 1569,57223,3299,10272,6270,7439,37434,1730 19.125000,18.500000,18.250000,18.250000,18.125000,18.125000,18.125000,18.000000 +8 4 2345 0.000059 2345,73686,25771,1058,1092,1456,1555,40048 26.875000,16.000000,15.125000,14.812500,13.875000,13.437500,13.312500,13.250000 +8 5 1860 0.028553 1860,99899,1558,8027,67594,1104,2308,6132 25.250000,20.500000,19.750000,19.250000,19.000000,18.250000,18.250000,18.250000 +8 6 2277 0.000036 2277,7588,3156,4032,1097,23728,36083,6094 31.000000,20.000000,19.500000,18.750000,17.625000,16.750000,16.750000,16.375000 +8 7 1730 0.000008 1730,26277,1860,12396,2009,35344,2320,22952 34.250000,21.125000,20.625000,20.375000,19.625000,19.625000,18.375000,18.375000 +9 0 15807 -0.000000 15807,2232,6650,2943,65195,12759,8383,49790 35.750000,19.000000,18.125000,18.000000,17.875000,16.750000,16.500000,16.500000 +9 1 6766 0.000011 6766,68217,2354,1010,1046,3256,4229,2247 27.000000,14.187500,13.500000,12.937500,12.812500,12.625000,12.437500,12.375000 +9 2 1047 0.002779 1047,61048,67566,104553,10361,20578,3826,10383 31.625000,25.500000,22.500000,21.625000,21.125000,21.125000,21.000000,21.000000 +9 3 1676 0.000366 1676,67594,99571,57387,26184,16387,10764,91700 27.875000,18.375000,17.250000,16.875000,16.625000,16.500000,16.375000,16.250000 +9 4 12204 0.000278 12204,3224,35316,1262,29520,108827,1047,2383 27.875000,19.625000,16.375000,14.250000,14.000000,13.937500,13.250000,12.375000 +9 5 1047 0.000710 1047,3826,15836,23015,26630,85501,108866,3326 32.000000,24.000000,23.375000,21.125000,21.125000,21.000000,21.000000,20.875000 +9 6 3326 0.012747 3326,1666,1676,47616,11807,8283,5136,4600 28.875000,24.250000,22.125000,21.875000,19.750000,19.625000,19.500000,19.250000 +9 7 2553 0.001539 2553,1047,6603,12641,109431,3134,67566,61470 29.875000,23.250000,21.000000,20.000000,18.125000,16.500000,16.375000,16.250000 +10 0 1097 0.000811 1097,1065,1509,7839,1413,3088,7983,3490 26.125000,18.625000,16.375000,16.000000,15.812500,14.875000,14.562500,14.437500 +10 1 1047 0.000856 1047,11801,76228,1456,10361,85501,6766,19602 28.500000,21.250000,18.250000,17.375000,17.250000,17.125000,16.875000,16.750000 +10 2 1051 0.000095 1051,1052,1049,1050,2189,1048,7240,1456 29.875000,20.250000,18.250000,18.000000,16.625000,15.812500,15.812500,15.687500 +10 3 1046 0.000196 1046,1456,1047,85501,1048,1626,1801,104042 28.000000,18.000000,17.625000,17.125000,16.750000,16.750000,16.375000,16.250000 +10 4 1048 0.000010 1048,1049,1051,1050,1032,85501,110737,1052 28.625000,15.437500,14.750000,14.375000,14.312500,14.000000,13.875000,13.187500 +10 5 85501 0.158094 85501,104042,44280,1047,1456,10361,36958,126501 28.000000,25.875000,24.500000,24.000000,21.500000,20.000000,19.875000,19.875000 +10 6 5780 11.929655 5348,56179,14209,2731,10547,35333,29130,33586 22.875000,22.875000,21.250000,21.000000,21.000000,20.500000,19.875000,19.375000 +10 7 5604 0.004782 5604,4242,82671,62574,8997,5565,3963,1444 24.750000,18.500000,18.250000,17.750000,16.625000,15.000000,13.562500,13.500000 +11 0 1058 0.145283 1058,62638,1737,5074,32505,1061,1124,1032 23.125000,20.250000,20.000000,18.375000,17.750000,17.375000,17.375000,17.250000 +11 1 40377 5.165777 13980,2129,37910,1429,7155,71996,85980,40659 20.000000,19.500000,19.250000,18.875000,18.875000,18.125000,17.500000,17.125000 +11 2 79858 2.773752 15837,1319,1044,1010,9369,79858,3403,1748 19.000000,18.625000,18.500000,17.875000,17.875000,17.875000,17.375000,17.125000 +11 3 1044 2.063042 1319,1010,1044,1267,1059,1775,1046,1626 23.000000,22.375000,21.875000,21.375000,21.125000,20.375000,19.500000,19.375000 +11 4 37910 1.765017 37910,1032,2425,25079,52558,2129,53379,7155 17.500000,17.250000,17.250000,17.000000,16.250000,16.000000,16.000000,15.812500 +11 5 17271 0.002020 17271,41577,2975,107205,6613,2559,109431,1906 25.500000,18.750000,18.000000,16.250000,15.375000,14.875000,14.875000,14.500000 +11 6 26554 0.951014 11071,26554,1010,1267,1319,2354,1044,4909 22.500000,22.500000,21.500000,19.875000,19.375000,18.875000,18.625000,18.375000 +11 7 11071 0.034491 11071,15812,1010,1267,4909,1626,1044,4995 24.000000,20.125000,18.625000,17.875000,17.875000,17.125000,16.625000,16.375000 +12 0 1010 0.264002 1010,1267,1044,1256,1319,1626,1338,1520 25.500000,23.750000,22.375000,21.750000,21.750000,20.875000,20.500000,19.875000 +12 1 6359 8.461706 2731,35333,93552,14209,58643,31990,5348,10547 21.125000,20.750000,20.375000,20.250000,19.875000,19.750000,19.625000,19.125000 +12 2 1319 0.935408 1319,7155,10078,26593,25079,13980,1462,1534 21.250000,21.250000,19.375000,17.625000,17.500000,17.375000,17.000000,17.000000 +12 3 1083 9.187790 3299,6270,19399,7619,1569,1288,20358,68782 22.500000,21.625000,20.000000,19.750000,19.125000,18.625000,18.625000,18.625000 +12 4 1046 4.976086 1497,1479,52512,7421,2551,93400,7730,63785 15.937500,15.187500,15.000000,14.875000,14.625000,13.812500,13.187500,13.062500 +12 5 10007 2.807364 1488,1335,59199,10007,17188,1507,1373,26913 16.625000,15.312500,15.125000,14.937500,14.937500,14.875000,14.687500,14.000000 +12 6 1532 0.023706 1532,19092,3173,4792,1331,1525,13221,2448 19.875000,14.750000,14.187500,13.125000,13.062500,12.562500,12.125000,12.062500 +12 7 1044 0.149557 1044,1568,1748,1321,1534,59199,1456,1801 25.500000,23.500000,21.375000,19.375000,18.750000,18.750000,18.375000,18.250000 +13 0 1464 10.330406 1359,1353,1568,1335,1373,1257,1507,1407 22.000000,20.500000,18.750000,17.875000,16.625000,16.500000,16.500000,15.750000 +13 1 1046 0.003748 1046,3051,13028,3109,5226,40933,1791,6565 22.125000,16.125000,14.187500,12.750000,12.500000,11.937500,11.875000,11.625000 +13 2 60018 7.776047 1335,1550,2272,1945,3264,21472,3664,20226 17.500000,14.250000,13.250000,13.062500,12.937500,12.875000,12.437500,12.437500 +13 3 2140 0.191027 2140,4172,4263,26672,1258,92469,2712,2208 17.750000,14.437500,14.375000,14.125000,13.812500,13.125000,12.750000,12.625000 +13 4 1550 2.013887 1044,1550,97857,65248,57640,111358,111352,7491 18.875000,17.125000,15.812500,14.437500,14.000000,13.875000,13.375000,12.937500 +13 5 12336 0.195637 12336,1046,18294,1044,2958,3109,15324,67524 17.250000,15.250000,13.125000,12.687500,12.062500,11.812500,11.375000,11.187500 +13 6 1277 0.000025 1277,1269,2912,1325,1859,1552,1270,2575 28.625000,16.375000,16.125000,15.500000,15.437500,15.125000,14.625000,14.625000 +13 7 1044 0.241314 1044,1748,1321,1456,1568,1520,1801,1041 24.875000,23.500000,19.625000,19.000000,19.000000,18.875000,18.375000,17.750000 +14 0 1434 3.294538 1359,1321,1748,1434,1568,1032,1945,2628 22.000000,19.750000,19.625000,19.000000,18.625000,17.875000,17.500000,17.375000 +14 1 1046 0.003431 1046,24326,3051,2472,1982,4700,20231,1626 21.375000,14.562500,13.875000,13.187500,12.625000,12.562500,12.000000,11.937500 +14 2 4493 0.007135 4493,14550,1335,1878,27402,18055,1464,8498 22.125000,15.937500,15.750000,14.187500,13.937500,13.375000,13.187500,13.000000 +14 3 5757 0.002869 5757,1331,1044,1111,9272,1667,103796,24748 21.750000,14.750000,13.812500,12.812500,12.625000,12.562500,12.187500,12.062500 +14 4 2251 3.180590 1044,1456,1775,1520,2251,1059,1801,1032 25.250000,24.000000,23.250000,22.750000,22.625000,22.250000,21.875000,21.250000 +14 5 1429 3.342297 1032,2129,1364,1429,7155,1531,1686,32830 22.875000,22.875000,21.000000,20.375000,18.875000,17.875000,17.625000,17.500000 +14 6 28218 8.596323 80498,1065,25609,36253,4051,1087,11993,50566 21.125000,20.875000,17.625000,17.125000,17.000000,16.875000,16.875000,16.375000 +14 7 1335 2.548616 109431,7491,5624,1335,1010,1083,15611,70979 19.875000,19.250000,18.750000,18.250000,17.500000,17.375000,16.250000,15.875000 +15 0 102817 0.094498 102817,9245,23119,5464,15765,109166,25009,7052 21.375000,18.875000,16.500000,16.125000,13.187500,13.187500,13.125000,13.062500 +15 1 13064 6.876423 2241,2580,11512,15699,4410,1034,14564,3768 18.750000,18.125000,16.125000,15.875000,15.687500,15.562500,15.375000,15.250000 +15 2 10773 0.004681 10773,1637,29059,3074,8148,1032,45623,1034 24.500000,18.375000,18.375000,14.875000,14.375000,14.125000,13.687500,13.500000 +15 3 52928 0.106052 52928,11512,13144,1307,6056,1010,129491,19344 22.000000,19.250000,18.375000,16.750000,16.625000,16.000000,15.812500,14.875000 +15 4 1897 1.318594 3768,1897,14032,2580,1034,4428,39960,110964 23.750000,23.375000,23.125000,21.625000,20.500000,20.250000,20.250000,20.000000 +15 5 1032 0.018110 1032,1686,32830,73469,4671,7155,1364,5167 25.750000,20.625000,20.125000,19.250000,19.125000,18.750000,18.625000,17.250000 +15 6 1050 0.001957 1050,1010,1049,1052,1051,1032,1053,1048 28.875000,22.500000,19.500000,19.375000,18.000000,17.750000,17.375000,17.250000 +15 7 1048 0.000050 1048,1049,1050,1051,1298,1052,59199,1057 28.375000,16.250000,15.937500,15.750000,15.500000,15.062500,15.062500,15.000000 +16 0 1049 0.004702 1049,1050,1048,1052,1051,1054,1053,1055 30.500000,25.125000,20.625000,18.625000,18.500000,17.750000,17.500000,17.500000 +16 1 1054 2.103777 1055,1054,1056,1057,1053,1052,1049,1051 24.750000,22.875000,22.125000,20.875000,19.875000,18.625000,16.125000,15.875000 +16 2 1456 0.463516 1456,1801,1044,1041,1059,109431,1882,1520 25.250000,24.625000,20.500000,20.375000,19.750000,19.625000,19.500000,19.375000 +16 3 6359 2.420685 2731,6359,35333,14209,58643,93552,35417,5348 20.750000,20.125000,19.875000,19.250000,19.250000,19.125000,19.000000,18.750000 +16 4 85980 5.649828 1319,7155,10078,8464,7085,1766,2251,1364 20.625000,20.000000,19.125000,19.000000,18.500000,18.250000,18.125000,18.000000 +16 5 6309 6.543623 13236,79207,4622,22618,16541,4524,13233,2434 17.375000,17.250000,16.875000,16.625000,16.375000,15.937500,15.875000,15.812500 +16 6 1058 0.113049 1058,1319,1536,4622,8464,26555,1408,6360 23.500000,19.250000,18.875000,18.875000,18.500000,18.375000,18.250000,18.000000 +16 7 7155 0.013597 7155,1766,1032,119685,1534,37910,2000,71996 25.000000,18.625000,18.500000,18.250000,18.125000,17.625000,17.500000,17.250000 +17 0 2345 0.000015 2345,25771,1058,2880,40048,13833,9283,1267 27.750000,14.375000,13.937500,13.875000,13.750000,13.250000,13.125000,13.000000 +17 1 1104 0.034918 1104,8027,74754,11897,1277,1114,1112,16698 26.375000,21.875000,21.125000,20.750000,20.000000,19.250000,19.125000,18.750000 +17 2 1936 0.000395 1936,1102,1099,62746,1103,8300,19988,43371 26.750000,18.625000,15.125000,15.125000,14.812500,14.750000,14.750000,14.625000 +17 3 7366 0.000074 7366,10553,38102,14397,6683,3136,8865,2793 27.875000,17.875000,16.375000,15.625000,14.562500,14.250000,13.812500,13.687500 +17 4 3136 0.000042 3136,17598,57557,4788,115119,11303,33995,1102 32.000000,21.250000,20.625000,19.250000,18.750000,18.625000,18.500000,17.375000 +17 5 20524 0.000015 20524,3136,2354,6406,3256,99359,35177,84202 30.750000,17.875000,17.375000,17.125000,16.875000,16.875000,16.750000,16.625000 +17 6 6406 0.226492 6406,30045,14327,99359,5938,34221,18392,35685 29.250000,27.875000,21.000000,21.000000,20.250000,20.250000,20.000000,19.500000 +17 7 71129 0.000145 71129,18492,1115,24941,10191,2476,31380,3083 32.000000,23.000000,20.625000,19.875000,18.875000,18.250000,18.125000,17.375000 +18 0 14327 0.000998 14327,13001,56803,1047,3826,1063,14787,35840 29.500000,21.500000,21.125000,20.250000,19.875000,19.250000,18.875000,18.500000 +18 1 3289 0.000002 3289,2013,4717,82166,1279,2068,58936,2675 31.750000,18.375000,15.687500,15.500000,15.375000,15.250000,15.000000,14.687500 +18 2 17271 0.000574 17271,2975,107205,41577,2559,7697,2751,6613 29.125000,21.375000,19.375000,18.875000,18.375000,17.000000,16.375000,16.125000 +18 3 13001 0.000294 13001,56803,1047,31991,23409,3826,6406,36362 30.875000,22.250000,20.000000,19.625000,19.500000,19.000000,18.750000,18.625000 +18 4 1552 0.000103 1552,36078,1290,1327,75934,7857,78231,129210 25.625000,14.875000,14.375000,13.875000,13.812500,13.312500,12.750000,12.375000 +18 5 1752 0.000004 1752,1631,1290,1612,3293,1318,1592,9059 32.250000,19.250000,17.125000,17.125000,17.125000,16.875000,16.125000,15.937500 +18 6 1637 0.000015 1637,1120,10773,1331,2510,29059,56566,32225 35.500000,24.500000,20.000000,19.625000,17.875000,17.125000,16.625000,16.500000 +18 7 1267 1.341103 1010,1267,1035,1256,104553,1032,1319,1260 25.500000,24.500000,21.250000,20.625000,19.750000,19.625000,18.750000,18.250000 +19 0 41838 12.908844 14209,56179,93552,31990,1784,3210,10547,43888 21.125000,19.750000,18.875000,18.250000,18.125000,18.125000,18.125000,17.875000 +19 1 1652 0.300331 1652,2564,3539,1307,9878,1260,1293,40803 20.375000,16.875000,16.625000,16.375000,16.250000,16.125000,16.125000,16.000000 +19 2 1737 0.001383 1737,1376,10869,7295,123993,1312,55462,7485 28.375000,21.000000,20.375000,19.000000,19.000000,18.500000,18.375000,17.875000 +19 3 5275 4.555410 2688,1032,4566,10018,1531,18028,3797,8030 18.500000,17.375000,17.125000,17.000000,16.875000,16.750000,16.375000,16.000000 +19 4 2688 2.216223 1319,16910,2688,1010,13683,15503,13156,56359 19.375000,19.000000,18.500000,18.375000,18.125000,17.625000,17.375000,16.500000 +19 5 13156 0.304585 13156,2016,48801,18003,109431,2181,21065,53597 23.625000,22.500000,18.250000,17.750000,17.250000,17.125000,17.000000,16.875000 +19 6 1319 0.289509 1319,1010,2251,8464,1044,27175,1256,1267 24.500000,22.875000,21.625000,20.500000,19.750000,19.500000,19.250000,19.000000 +19 7 1599 3.194242 2649,1052,1599,8957,1049,7942,1948,34094 23.625000,21.000000,20.875000,20.375000,20.250000,19.625000,19.500000,19.375000 +20 0 1032 0.542315 1032,34421,11277,7280,16647,35699,12964,9399 22.500000,20.625000,20.250000,20.125000,20.000000,19.750000,19.375000,18.875000 +20 1 1052 0.014595 1052,1050,1049,1053,1051,1056,1057,1055 27.375000,22.500000,21.500000,20.875000,20.750000,20.375000,18.875000,18.625000 +20 2 1051 0.857155 1044,1051,59199,109431,1046,51076,1049,35699 24.000000,23.750000,20.875000,17.625000,17.375000,16.750000,16.500000,16.375000 +20 3 1053 0.000443 1053,1054,1056,1055,1052,1051,1050,1044 27.125000,18.125000,17.875000,17.500000,16.750000,16.625000,16.125000,16.000000 +20 4 1056 0.000778 1056,1055,1054,1050,1048,1053,1057,1052 26.000000,17.625000,17.250000,16.375000,15.937500,15.937500,15.875000,15.750000 +20 5 12964 3.645088 16647,11277,14170,7280,18028,35699,12964,2688 21.875000,20.875000,20.500000,20.250000,20.125000,19.625000,19.125000,19.000000 +20 6 56421 9.887840 1456,1584,1801,1041,2251,2434,1044,1307 23.125000,21.000000,21.000000,20.875000,20.750000,20.750000,20.625000,19.875000 +20 7 1621 0.001097 1621,1419,1370,2283,2130,49257,43220,35061 26.000000,19.125000,14.812500,14.625000,14.250000,13.500000,13.125000,12.562500 +21 0 1456 1.376930 2203,1456,1435,8464,1801,1059,1044,1394 24.250000,23.500000,22.375000,21.375000,20.875000,20.500000,20.375000,20.250000 +21 1 1071 9.420601 2731,56179,5348,3210,33586,35333,15063,44149 20.125000,19.875000,19.250000,18.750000,18.750000,18.750000,18.625000,18.625000 +21 2 11130 3.962610 1090,12256,14020,19038,83950,13754,38083,11130 17.750000,17.375000,17.000000,16.625000,16.000000,15.437500,15.250000,15.062500 +21 3 1045 1.590174 109431,1045,1050,12114,9964,29738,32545,10835 21.375000,20.250000,18.500000,18.000000,17.500000,17.250000,17.250000,16.250000 +21 4 1050 0.043743 1050,1051,8875,1052,1049,5323,17116,43888 25.750000,22.375000,20.500000,19.125000,17.750000,17.125000,17.125000,16.750000 +21 5 1398 8.296207 11512,3729,3244,29738,30926,10835,7513,1652 17.250000,16.500000,16.375000,16.375000,15.750000,15.687500,15.625000,15.375000 +21 6 7134 0.174244 7134,6964,11130,13752,3561,1446,6918,12381 20.500000,18.000000,17.250000,16.375000,15.937500,15.812500,15.250000,15.125000 +21 7 34421 5.210947 1032,128405,29738,10835,1256,93353,1058,34421 22.625000,20.875000,20.125000,18.875000,18.500000,18.375000,18.125000,17.750000 +22 0 1319 1.975178 1058,1319,1737,1651,2434,1376,1032,89738 19.500000,19.000000,19.000000,18.375000,18.125000,18.000000,17.625000,17.625000 +22 1 120102 7.746767 1049,5011,19266,34094,1053,65888,1050,21610 19.125000,18.750000,18.625000,18.625000,18.500000,18.500000,18.375000,18.375000 +22 2 15017 0.007193 15017,11754,65877,10835,13843,3978,1638,44784 21.625000,15.187500,14.687500,14.312500,14.250000,12.750000,12.625000,12.625000 +22 3 4244 2.075047 17731,1041,4244,1044,4469,1047,3244,2000 18.125000,17.750000,17.500000,16.750000,16.500000,16.250000,16.125000,15.750000 +22 4 1032 0.910093 1032,7849,109863,89738,1256,1603,1766,93353 21.625000,21.500000,19.875000,19.750000,18.500000,17.750000,17.625000,17.000000 +22 5 1050 1.880491 1049,1050,1052,1053,1051,1054,1055,1032 21.000000,20.625000,20.500000,20.500000,20.000000,19.875000,19.625000,19.125000 +22 6 1056 2.716267 1044,1053,1055,1050,1049,1054,1056,1057 22.250000,22.250000,22.125000,22.000000,21.875000,21.750000,21.500000,21.500000 +22 7 1051 3.633308 1044,59199,1055,1053,1054,1056,1049,1050 22.625000,22.500000,22.250000,22.000000,22.000000,21.500000,21.125000,21.125000 +23 0 1050 3.177383 59199,1044,1055,1053,1050,1057,1049,1056 21.750000,21.000000,20.500000,20.125000,19.625000,19.625000,19.500000,19.375000 +23 1 1056 2.422956 1055,1053,1050,1056,1054,1049,1052,1051 21.375000,20.500000,20.375000,20.375000,20.125000,20.000000,20.000000,19.875000 +23 2 1055 4.502510 34421,1010,1032,8464,8877,1267,1319,3788 21.500000,20.375000,18.625000,18.625000,18.500000,18.375000,18.250000,18.125000 +23 3 1010 2.285713 34421,1010,1053,1055,1054,1050,1056,1052 21.875000,20.375000,19.750000,19.625000,19.500000,19.375000,19.125000,19.000000 +23 4 5629 9.900187 2731,1071,57816,8693,58643,13458,65788,1066 19.625000,18.625000,18.250000,18.125000,18.125000,18.000000,17.875000,17.750000 +23 5 21830 11.578989 10662,4403,1278,1032,2493,6325,1260,3539 17.125000,15.562500,14.062500,13.937500,13.875000,13.875000,13.062500,13.062500 +23 6 1076 0.006144 1076,111268,45623,1077,72458,1376,18369,1061 23.125000,16.875000,16.500000,15.312500,15.312500,15.000000,14.750000,14.625000 +23 7 31200 15.745977 1061,1376,1058,1032,13472,4473,1256,3539 17.625000,16.875000,15.312500,15.062500,15.000000,14.750000,14.500000,14.312500 +24 0 8929 0.982176 1075,8929,45165,1079,15348,38866,1058,110718 18.000000,18.000000,17.250000,15.125000,14.250000,14.187500,13.375000,13.062500 +24 1 1075 0.048981 1075,110718,17131,1107,49744,128745,1376,1071 25.000000,22.000000,16.375000,14.937500,14.250000,13.812500,13.625000,13.500000 +24 2 65456 3.932190 21214,1061,1376,1058,65456,1032,53342,1319 18.000000,16.500000,15.875000,14.750000,14.500000,14.062500,13.875000,13.187500 +24 3 1061 1.037979 1376,1061,34421,1319,1058,2351,5966,1032 20.000000,19.625000,17.125000,16.250000,15.937500,15.750000,15.625000,15.375000 +24 4 1052 4.125631 1049,1050,1053,1051,1056,1052,1054,1048 23.000000,22.750000,22.500000,20.875000,20.125000,19.875000,19.875000,18.875000 +24 5 1048 0.915262 1048,1053,1057,34421,1056,1055,1044,1050 20.875000,19.500000,19.375000,19.125000,19.000000,18.500000,18.375000,18.375000 +24 6 1057 1.103163 1057,1048,34421,8464,1044,1319,1058,1877 20.250000,19.500000,19.125000,18.875000,18.375000,18.000000,17.750000,17.750000 +24 7 1054 0.035387 1054,1055,1053,1056,1050,1048,1052,1057 26.875000,22.875000,22.500000,20.875000,19.625000,19.125000,19.000000,18.625000 +25 0 1593 5.209959 8464,34421,1044,1319,1877,1058,10835,1278 19.250000,19.125000,18.875000,18.375000,17.875000,17.375000,17.250000,16.750000 +25 1 28248 0.403774 28248,1395,10177,33197,11006,23368,78804,23843 22.375000,19.750000,19.750000,19.375000,19.250000,19.000000,18.875000,18.125000 +25 2 1032 1.065147 1032,7849,1877,89738,1058,109863,1464,10693 22.625000,21.875000,21.500000,21.000000,20.875000,20.875000,20.000000,19.625000 +25 3 1054 0.453955 1054,1050,1055,1052,1049,1051,1053,1032 24.250000,22.875000,22.375000,21.750000,21.000000,19.500000,19.500000,19.250000 +25 4 1057 0.033936 1057,1056,1054,1053,1055,1050,1051,1048 24.250000,20.500000,18.375000,18.000000,18.000000,17.250000,17.125000,17.000000 +25 5 2767 4.425056 63308,34421,10835,5275,51076,7716,1046,7662 19.250000,17.500000,17.125000,17.000000,16.875000,16.500000,16.125000,15.875000 +25 6 41869 1.817556 109431,41869,92999,1045,3245,4693,12641,43426 24.625000,23.000000,19.375000,18.000000,18.000000,17.250000,17.125000,17.125000 +25 7 1108 0.000471 1108,7641,59172,114978,1279,7132,109431,8084 27.375000,18.875000,18.875000,15.250000,15.000000,14.375000,14.250000,14.125000 +26 0 7641 0.000097 7641,13194,2195,13633,4677,15643,43869,35077 31.625000,22.375000,16.000000,15.812500,15.312500,15.312500,14.187500,13.062500 +26 1 63308 0.091410 63308,51076,21830,34421,16144,2688,13664,1032 22.875000,19.000000,18.500000,18.250000,17.875000,17.500000,17.250000,17.125000 +26 2 1520 4.159533 1010,1626,1267,1319,1338,1877,1651,1307 24.125000,23.000000,22.000000,22.000000,21.750000,21.250000,21.125000,20.875000 +26 3 32189 10.089449 1778,28366,6510,3265,11018,5133,22634,2209 21.250000,21.000000,19.750000,19.500000,19.500000,19.250000,19.125000,19.000000 +26 4 4118 0.005241 4118,1638,1302,15006,6171,27673,4219,1405 23.875000,18.500000,15.562500,15.000000,13.750000,13.625000,12.937500,12.437500 +26 5 63118 4.232075 34336,3006,3746,5956,4455,1534,63118,30652 22.125000,19.625000,19.375000,19.000000,18.625000,18.375000,18.250000,18.125000 +26 6 1278 0.473148 1278,2143,1032,2948,1261,2258,15421,21830 22.375000,20.750000,20.375000,19.750000,18.750000,18.500000,18.375000,17.750000 +26 7 1032 1.023853 1032,11835,5074,24130,4566,15421,7849,30652 20.625000,19.375000,19.250000,19.000000,18.750000,18.750000,18.000000,17.375000 +27 0 1050 1.871981 1053,1049,1050,1054,1051,1052,1056,1055 25.500000,25.125000,24.875000,24.875000,24.500000,24.000000,22.750000,22.500000 +27 1 1048 0.951342 1048,1053,109431,1056,1054,1055,1049,1052 20.875000,20.500000,19.000000,18.625000,18.375000,18.375000,18.250000,18.250000 +27 2 4693 2.112816 109431,1048,4693,1052,5074,63308,90938,6618 21.125000,19.625000,19.625000,18.500000,18.125000,17.875000,17.625000,17.000000 +27 3 5148 0.000349 5148,34382,51615,1384,13305,5241,2395,2699 24.750000,16.375000,14.750000,13.562500,12.937500,12.375000,12.125000,12.062500 +27 4 21609 1.849707 11835,21609,13472,3579,4953,11237,18204,6139 23.125000,21.750000,21.375000,20.500000,19.375000,19.125000,18.375000,18.125000 +27 5 1307 5.675303 1626,1338,1394,1046,3016,1319,1321,1636 25.000000,24.125000,20.875000,20.625000,20.500000,20.250000,20.000000,19.750000 +27 6 1278 1.119772 1278,1032,2143,21830,2000,2948,7849,1010 18.000000,17.125000,16.250000,16.250000,16.000000,15.500000,15.500000,15.437500 +27 7 1398 8.061003 8466,1010,13664,21830,4618,54093,9519,2688 18.000000,17.875000,17.375000,16.750000,16.375000,15.812500,15.750000,15.687500 +28 0 1051 10.046696 9793,11130,3561,7134,10504,6918,2794,9034 18.125000,16.000000,14.750000,14.437500,14.062500,13.437500,13.000000,12.875000 +28 1 7353 14.205479 1076,2553,30631,38180,54093,7619,1405,6255 15.562500,15.187500,14.500000,14.500000,14.437500,14.125000,13.562500,13.437500 +28 2 15085 1.862261 28966,15085,1010,54093,109431,3244,4206,10097 19.250000,18.750000,18.000000,17.875000,17.875000,17.000000,17.000000,17.000000 +28 3 1338 1.199076 1626,1338,1010,1046,1520,1365,4043,19486 24.000000,23.250000,20.625000,19.750000,19.625000,18.375000,18.375000,18.375000 +28 4 4380 4.505823 55839,1784,93552,58643,4756,29384,31990,43888 19.875000,19.000000,18.875000,18.750000,18.625000,18.500000,18.500000,18.500000 +28 5 3323 2.947390 1395,8466,48801,3323,16910,35573,3403,62195 16.750000,16.750000,16.625000,16.250000,16.250000,16.000000,15.375000,15.375000 +28 6 1395 0.756207 1395,1710,8688,2084,2188,2715,7905,8222 21.500000,20.250000,19.750000,19.625000,18.500000,18.375000,18.375000,18.250000 +28 7 2434 2.818298 13650,1278,2434,8360,70440,1261,1805,9553 18.750000,18.500000,18.000000,17.875000,17.875000,17.750000,17.500000,17.500000 +29 0 1435 1.145292 1435,1536,1394,1317,2342,7655,53021,36333 23.875000,23.250000,23.125000,23.000000,22.875000,20.875000,20.875000,20.625000 +29 1 1261 1.615841 1278,1261,4292,1420,109431,1805,2000,2688 21.875000,20.875000,20.375000,19.500000,18.125000,17.500000,17.500000,17.500000 +29 2 47478 6.772449 1364,6752,7353,54093,2688,4249,21830,86668 17.875000,17.875000,17.500000,17.125000,17.000000,16.875000,16.750000,16.625000 +29 3 1394 0.464800 1394,1294,1317,2200,1319,2342,27175,1536 22.750000,21.750000,19.750000,19.125000,18.750000,18.625000,18.375000,18.250000 +29 4 1278 1.502436 1278,28814,35822,1398,3244,10418,1877,21830 18.250000,17.250000,17.125000,16.875000,16.750000,16.625000,16.500000,16.375000 +29 5 66560 14.818583 1398,2000,2129,21830,1429,71996,1010,1364 18.250000,18.000000,17.875000,17.250000,16.875000,16.250000,16.125000,16.000000 +29 6 28858 8.332088 109431,1080,21763,1071,43698,47782,5119,48670 15.750000,14.312500,13.625000,13.562500,13.062500,12.937500,11.937500,11.937500 +29 7 98473 9.888069 1398,1461,54093,21830,3244,7353,31587,3229 15.562500,15.187500,15.187500,14.937500,14.812500,14.625000,14.437500,14.187500 +30 0 15009 0.396387 15009,52779,109431,71978,5849,17536,54093,82658 20.125000,18.250000,18.250000,17.250000,16.625000,16.375000,15.125000,14.562500 +30 1 98430 10.948738 54093,130898,1010,2688,15701,9741,58615,113791 17.000000,16.625000,16.250000,16.000000,16.000000,15.625000,15.437500,15.312500 +30 2 1532 0.000393 1532,2754,109431,1010,1286,1105,5077,15611 27.375000,18.625000,18.125000,17.500000,17.000000,15.500000,15.375000,14.875000 +30 3 15085 4.106987 1010,2688,13664,54093,4206,15085,82202,28966 21.875000,20.125000,19.750000,19.250000,18.375000,18.375000,18.375000,18.250000 +30 4 1010 1.107758 1626,1010,1520,1338,1365,2688,1877,1046 22.250000,22.000000,20.750000,20.000000,19.500000,19.375000,18.875000,17.625000 +30 5 4039 6.996796 1259,1040,1421,4417,13800,1611,25695,67617 21.875000,20.750000,20.375000,20.375000,20.250000,20.125000,20.000000,20.000000 +30 6 11680 0.584536 11680,12436,1103,15981,125167,18869,1113,1328 18.250000,16.875000,16.000000,15.312500,15.000000,14.687500,14.375000,14.375000 +30 7 10001 1.955458 1047,10001,13126,15836,8273,1294,23015,33818 19.125000,17.875000,17.375000,16.875000,16.250000,15.562500,15.312500,15.312500 +31 0 125167 3.275562 12436,125167,1113,1103,1328,5636,10648,1098 20.375000,17.250000,16.875000,16.500000,15.187500,15.187500,15.187500,15.062500 +31 1 21227 4.739552 1095,18582,21227,2480,1045,3602,3480,3557 22.125000,20.625000,17.625000,17.000000,16.375000,16.375000,15.812500,15.750000 +31 2 1118 0.002176 1118,22160,1098,2137,15009,11221,46434,42625 25.000000,17.750000,17.125000,16.625000,16.250000,15.875000,15.750000,15.500000 +31 3 13196 10.141668 46434,3480,40431,15009,19848,1095,8261,16244 23.875000,18.250000,17.625000,17.500000,17.250000,17.125000,17.125000,17.125000 +31 4 1050 1.679115 26584,43807,1050,4957,1977,5408,103561,3838 19.875000,19.500000,19.125000,18.375000,17.125000,15.875000,15.750000,15.437500 +31 5 1101 0.003145 1101,1304,13196,1048,1095,1116,1286,11999 25.875000,18.875000,18.750000,17.500000,17.000000,16.500000,16.375000,16.250000 +31 6 50648 8.765461 8261,11898,13683,1095,2480,1047,3445,16244 19.125000,17.500000,17.375000,16.750000,15.812500,15.750000,15.625000,15.625000 +31 7 4342 2.583527 1574,1041,1577,2519,4342,1044,8464,1035 20.875000,20.750000,20.500000,19.875000,19.625000,18.875000,18.750000,17.750000 +32 0 16820 8.409203 58643,2731,100312,1975,55839,27266,43035,14135 23.250000,22.625000,21.500000,21.250000,21.000000,20.125000,19.250000,19.000000 +32 1 29737 10.541842 112318,5266,23996,26840,4036,22454,11616,7147 18.625000,18.000000,17.750000,16.875000,16.750000,16.500000,15.750000,15.687500 +32 2 2210 4.981409 2196,6113,1307,31380,4016,1505,16910,1455 19.625000,18.875000,18.000000,17.125000,17.000000,16.875000,16.875000,16.750000 +32 3 4016 0.663229 4016,2715,1319,1307,1395,10867,1505,1394 22.000000,20.750000,20.250000,20.125000,19.875000,19.125000,18.125000,17.875000 +32 4 13881 0.945145 13881,60441,3945,33972,4626,5807,56819,10399 24.125000,23.750000,23.125000,22.375000,20.875000,20.875000,20.875000,20.500000 +32 5 1593 1.587641 1278,1593,113472,5275,4618,2960,1877,2246 23.500000,22.250000,20.250000,19.375000,19.125000,18.875000,17.750000,17.750000 +32 6 25079 6.684517 11957,113472,15803,48896,28332,90536,7739,82381 21.250000,20.750000,19.375000,19.250000,18.875000,18.875000,17.875000,17.625000 +32 7 20992 0.794954 20992,109431,5630,1051,26630,28332,1032,59199 23.375000,23.125000,22.375000,19.625000,19.125000,18.250000,17.750000,17.750000 +33 0 5624 0.034018 5624,109431,1045,21128,9964,1882,28332,11941 27.500000,24.125000,18.500000,17.500000,17.375000,17.250000,17.125000,16.250000 +33 1 1065 0.000046 1065,1349,1051,1067,1032,3590,6292,1076 26.750000,15.562500,14.500000,14.437500,13.750000,13.687500,13.375000,12.812500 +33 2 1032 0.059185 1032,113472,15803,1045,11957,28332,109431,48896 26.750000,23.125000,22.750000,21.375000,20.875000,20.875000,20.500000,20.000000 +33 3 1051 0.000099 1051,1052,1050,1049,1053,1048,113472,1054 31.375000,22.000000,20.125000,17.250000,15.875000,15.437500,14.812500,14.687500 +33 4 1046 0.000025 1046,109431,1048,28234,113472,7989,1051,6662 29.000000,17.000000,16.375000,15.937500,15.687500,15.312500,15.125000,14.812500 +33 5 1048 0.000008 1048,1049,1051,1050,1054,1053,1056,1052 29.875000,17.500000,16.625000,15.437500,14.125000,13.812500,13.375000,12.750000 +33 6 113472 0.742531 113472,15803,28332,11957,48896,34140,82381,90536 22.000000,21.125000,21.125000,19.875000,19.000000,18.000000,17.875000,17.750000 +33 7 1338 1.709961 1626,1321,1338,11957,15803,1046,1294,1010 23.250000,22.375000,22.375000,21.375000,20.875000,20.375000,20.000000,19.125000 +34 0 1376 9.569738 58643,100312,8129,55839,2731,25197,47964,1035 19.375000,19.250000,18.875000,18.375000,17.750000,17.625000,17.500000,17.375000 +34 1 8052 18.429634 1474,5184,1032,56484,1045,29960,1267,3518 16.250000,15.312500,15.187500,14.937500,14.812500,14.562500,13.750000,13.250000 +34 2 89656 12.953844 8343,1507,1424,44593,27768,42180,50359,22922 14.937500,14.625000,14.125000,13.750000,13.625000,13.625000,13.437500,13.375000 +34 3 1397 6.250668 82852,6500,2306,1417,1346,3554,43926,1450 16.625000,16.500000,16.125000,13.875000,12.437500,12.375000,12.312500,11.625000 +34 4 1376 5.727118 1044,1319,1534,1010,1775,2251,1032,1748 19.375000,18.625000,18.000000,16.750000,16.750000,16.625000,16.375000,16.250000 +34 5 1032 1.624329 1032,1267,4410,1376,1319,1534,1256,11 16.625000,16.625000,14.750000,14.125000,13.937500,13.937500,13.687500,13.625000 +34 6 1010 3.085823 1050,1010,1049,1048,1267,1051,1534,1055 22.500000,19.625000,19.125000,18.750000,18.750000,18.125000,17.750000,17.625000 +34 7 8052 7.769659 2731,1975,58643,100312,1376,1032,1256,67499 18.625000,18.500000,17.500000,17.250000,15.562500,15.437500,15.437500,15.250000 +35 0 89656 0.154305 89656,1395,59199,79853,1349,1398,1681,1934 19.000000,15.750000,14.187500,14.062500,14.000000,13.937500,13.437500,13.375000 +35 1 1397 0.240656 1397,2306,2186,6500,5032,1010,1575,2723 19.750000,18.250000,16.125000,14.625000,13.125000,12.625000,12.625000,12.562500 +35 2 1395 2.279289 1534,1319,1395,1010,1044,2251,1934,4648 19.125000,17.750000,17.750000,17.250000,16.875000,16.875000,16.125000,16.000000 +35 3 1420 1.952190 1261,1278,1420,15566,10910,1605,4629,40759 21.250000,20.250000,19.875000,17.750000,16.750000,16.375000,16.125000,16.125000 +35 4 7846 3.207357 5150,34943,6089,9353,26554,7846,14055,27089 19.375000,19.375000,18.500000,18.375000,18.000000,17.750000,17.250000,17.250000 +35 5 4076 7.200390 19474,23385,9314,109431,7278,53143,6089,56905 18.000000,17.375000,16.875000,16.875000,16.125000,16.125000,16.000000,16.000000 +35 6 3897 10.430376 1321,12245,13823,22122,109431,1044,17226,12383 20.250000,20.250000,18.875000,18.500000,17.875000,16.875000,16.375000,16.250000 +35 7 14696 0.381119 14696,8281,1261,7316,25865,7476,8077,16890 19.250000,16.750000,16.125000,16.125000,15.812500,15.500000,15.500000,15.250000 +36 0 1321 0.068733 1321,3897,1044,6421,12245,41500,1748,7316 25.125000,22.125000,19.500000,19.250000,19.125000,18.750000,18.625000,18.250000 +36 1 37848 1.336792 7316,37848,8602,9160,11885,7476,73593,41228 20.000000,19.375000,18.875000,17.000000,16.500000,16.375000,15.500000,15.437500 +36 2 17226 0.152132 17226,12245,36009,7316,12383,3897,4076,1321 21.500000,19.125000,17.375000,17.250000,16.125000,15.625000,15.437500,15.375000 +36 3 2247 1.144463 2247,18988,11159,1321,3897,2274,3560,43378 22.375000,22.375000,20.875000,20.750000,20.625000,20.125000,19.625000,19.375000 +36 4 2182 1.558199 1032,2182,4410,11,6304,1256,1531,2241 21.875000,21.125000,20.625000,20.250000,18.625000,18.250000,17.750000,17.625000 +36 5 1880 5.679789 1395,1934,8308,1486,1010,10354,22088,56671 23.375000,21.875000,21.875000,20.750000,19.875000,19.500000,19.250000,19.250000 +36 6 1261 0.493572 1261,1278,1420,17317,1010,15720,8924,2830 22.125000,19.875000,19.125000,19.125000,19.000000,19.000000,18.875000,18.750000 +36 7 24836 4.332037 68393,4934,17317,1010,1032,2830,3709,4616 20.250000,18.250000,18.125000,17.750000,17.125000,17.125000,17.125000,16.875000 +37 0 2126 14.742516 4934,1010,13988,47715,1032,9655,14104,3820 22.125000,20.625000,20.250000,19.875000,19.625000,19.125000,18.375000,18.000000 +37 1 1045 7.376080 1032,1010,93023,1278,14338,4934,1261,2308 15.875000,15.062500,14.625000,13.312500,13.125000,12.875000,12.750000,12.687500 +37 2 1064 4.333725 14338,3332,4934,79113,1064,2126,1035,1049 17.625000,15.562500,14.437500,14.312500,13.812500,13.562500,13.437500,13.125000 +37 3 47715 5.322048 13988,4934,1032,1294,1805,1010,2479,29857 16.250000,16.125000,14.875000,14.687500,14.375000,13.937500,13.875000,13.875000 +37 4 4934 0.065878 4934,2126,1010,15720,1294,6421,1032,1408 22.750000,19.000000,18.375000,17.125000,16.750000,16.500000,16.250000,16.250000 +37 5 1408 1.750006 1294,1408,1435,2126,1010,1307,12440,22594 23.875000,22.500000,21.625000,19.750000,19.375000,19.250000,19.000000,18.875000 +37 6 1278 0.825708 1278,1531,1429,1420,19895,1364,2126,1617 18.500000,16.750000,15.875000,15.562500,15.437500,15.187500,15.125000,14.937500 +37 7 14696 2.990259 19895,8281,14696,36230,5150,17267,81873,22876 18.500000,17.750000,16.750000,16.750000,16.625000,16.375000,16.375000,16.000000 +38 0 5395 0.591333 5395,25865,2629,18823,3467,50584,14593,18944 20.875000,19.750000,19.125000,18.625000,18.000000,17.625000,16.875000,16.500000 +38 1 1531 2.241734 1531,1576,1429,3897,1364,22377,13446,1032 16.375000,16.125000,16.000000,15.625000,15.437500,15.000000,14.687500,14.437500 +38 2 11630 1.621571 11630,14505,67971,13649,11732,68960,9629,1398 17.250000,16.500000,16.375000,15.500000,15.000000,14.812500,14.625000,14.437500 +38 3 1294 2.096800 2247,1294,1321,1435,12440,18988,11159,3897 22.625000,21.250000,21.250000,21.000000,20.625000,19.750000,19.125000,18.875000 +38 4 1032 0.127588 1032,1278,15011,1420,38544,2126,2246,2247 24.000000,21.625000,19.000000,18.750000,18.250000,18.125000,17.625000,17.500000 +38 5 1050 0.694815 1049,1050,1032,1052,1051,1057,1053,1056 23.125000,23.125000,15.125000,15.062500,14.812500,14.750000,14.687500,14.500000 +38 6 1048 0.001755 1048,1051,1056,1053,1049,1050,1055,1057 26.000000,17.625000,17.625000,17.250000,17.000000,17.000000,17.000000,17.000000 +38 7 1048 0.349873 1048,1049,2247,1050,2126,18988,1032,3308 24.250000,23.375000,16.625000,15.875000,15.375000,14.812500,14.750000,14.750000 +39 0 1048 2.975449 1049,1053,1056,1055,1052,1054,1050,1051 22.875000,21.750000,21.500000,21.250000,21.125000,21.125000,21.000000,21.000000 +39 1 2247 0.564980 2247,12440,1321,1435,3897,18988,1294,11159 22.750000,21.125000,21.000000,20.250000,20.250000,19.625000,19.375000,19.375000 +39 2 2409 5.874550 2182,6304,1656,1531,4410,8052,11,1032 20.625000,18.750000,18.625000,18.500000,18.375000,18.250000,17.875000,17.875000 +39 3 1486 2.387520 1395,1486,4076,15011,4934,9369,7125,7520 16.750000,16.250000,16.000000,16.000000,15.875000,15.875000,15.187500,15.187500 +39 4 7125 0.551123 7125,1261,1794,1925,1278,1420,2095,2453 21.000000,19.000000,18.375000,18.375000,18.250000,17.875000,17.750000,17.125000 +39 5 1536 0.113508 1536,1454,1294,2015,1278,2126,1408,3897 25.625000,22.750000,22.125000,21.000000,20.125000,20.125000,19.625000,19.500000 +39 6 1261 0.785614 1261,1420,15720,24836,1278,3866,29857,17317 20.750000,19.250000,18.750000,18.625000,18.500000,18.000000,17.750000,17.500000 +39 7 47715 4.506365 24836,68393,4934,17317,8924,2782,3709,96432 18.625000,18.250000,18.125000,18.125000,17.125000,16.625000,16.625000,16.500000 +40 0 4934 0.024948 4934,2126,8602,4076,3820,6421,1321,3897 23.375000,18.750000,17.500000,16.750000,16.375000,16.125000,16.000000,15.687500 +40 1 1294 0.316877 1294,1408,1435,22594,12440,1454,1307,2126 24.250000,22.750000,21.375000,20.875000,19.875000,19.250000,19.000000,19.000000 +40 2 1278 0.315382 1278,1531,22876,1261,1420,1032,81873,1349 19.375000,16.750000,15.687500,15.500000,15.250000,14.812500,14.437500,14.187500 +40 3 3354 4.491108 4076,14696,19895,7326,8281,1032,22876,4958 18.500000,18.500000,17.625000,16.875000,16.875000,16.625000,16.500000,16.125000 +40 4 4973 7.066765 1349,1531,1576,1429,45776,9966,3897,1373 16.625000,16.500000,16.375000,15.625000,14.812500,14.500000,14.437500,14.312500 +40 5 2232 1.992670 2232,1264,26080,11070,1464,11732,14660,1032 15.312500,15.250000,15.250000,14.687500,14.562500,14.500000,13.750000,13.562500 +40 6 8360 9.284960 2247,1294,18988,1321,1319,1513,1455,11159 16.875000,16.625000,15.687500,15.437500,15.312500,15.312500,15.187500,14.562500 +40 7 1536 0.061224 1536,1321,1394,1294,2126,13703,6421,5652 25.375000,21.875000,21.250000,19.875000,19.875000,18.625000,18.500000,17.750000 +41 0 15723 3.858459 22511,16566,3986,6031,9342,7456,14969,6959 15.875000,15.687500,15.375000,15.250000,15.250000,15.125000,15.000000,14.875000 +41 1 97698 0.110102 97698,2247,32173,41098,3395,104972,24898,84040 20.000000,16.250000,16.000000,15.187500,15.000000,14.125000,13.750000,13.750000 +41 2 3897 4.027647 2247,1321,1294,1513,18988,1394,3897,11159 23.875000,22.375000,22.250000,21.125000,21.000000,20.375000,20.375000,20.125000 +41 3 1799 1.768070 15179,1799,5537,1321,8077,1294,1261,1394 18.625000,18.375000,18.375000,17.500000,17.000000,16.500000,16.375000,16.375000 +41 4 1486 1.192461 1486,9491,77197,1514,6307,10467,1880,2095 19.750000,19.000000,18.125000,18.000000,17.750000,17.750000,17.250000,17.250000 +41 5 5537 0.713930 5537,8077,69105,15179,2158,4326,4283,46828 21.125000,19.875000,19.500000,18.875000,18.375000,17.875000,17.750000,17.375000 +41 6 1294 1.847166 1513,1294,2126,1435,1562,1408,3184,1536 24.125000,22.625000,21.375000,20.250000,20.125000,19.625000,19.625000,19.125000 +41 7 1032 2.105356 7893,22820,1032,1278,1261,37121,4454,6776 18.875000,18.500000,17.875000,17.250000,16.000000,15.687500,15.625000,15.625000 +42 0 1050 0.010138 1050,1049,1010,1051,1267,1052,1053,1054 25.500000,20.375000,18.750000,17.625000,17.625000,17.125000,17.125000,16.875000 +42 1 1048 0.003628 1048,7893,1051,23981,1047,1032,1052,1050 22.625000,13.750000,13.625000,13.562500,13.375000,13.312500,13.312500,13.187500 +42 2 1048 0.182285 1048,1049,1050,2247,1051,2126,1032,1052 26.250000,24.625000,20.250000,17.250000,15.812500,15.750000,15.500000,15.500000 +42 3 1049 2.487535 1053,1050,1051,1054,1055,1049,1052,1056 23.875000,23.625000,23.375000,23.250000,23.125000,23.000000,23.000000,22.875000 +42 4 1513 1.943197 2247,1513,18988,1321,1294,11159,1435,1454 23.750000,22.250000,21.500000,21.000000,20.750000,20.750000,18.875000,18.750000 +42 5 1278 0.065956 1278,1531,22410,1335,1261,7893,22820,126355 22.375000,18.750000,17.500000,16.875000,16.625000,16.375000,16.000000,15.562500 +42 6 11552 1.642296 27640,11552,20361,22410,14505,1335,94670,29531 18.375000,18.125000,17.250000,17.250000,17.125000,16.375000,16.125000,16.000000 +42 7 11570 1.562757 5805,39661,11570,20361,34614,22410,34984,16653 21.500000,21.250000,20.875000,18.750000,18.625000,18.500000,17.500000,17.125000 +43 0 22410 0.103413 22410,8906,2247,37848,11159,18988,1294,1321 22.125000,18.625000,18.250000,18.250000,17.125000,16.875000,16.625000,16.500000 +43 1 2247 0.513088 2247,1294,18988,11159,1321,3897,1319,7893 23.125000,21.875000,21.125000,20.375000,20.250000,19.750000,18.750000,18.625000 +43 2 2182 1.699100 2182,1656,8052,11,4973,1032,1531,4410 19.250000,18.750000,18.250000,18.125000,18.125000,18.000000,18.000000,17.875000 +43 3 1880 1.729385 2095,8308,1880,2430,1934,1395,6387,10354 21.875000,21.750000,21.500000,20.250000,20.000000,19.875000,19.500000,19.500000 +43 4 1261 0.507122 1261,3866,24836,17317,1420,1278,1794,68393 23.125000,22.125000,20.250000,19.625000,19.500000,19.375000,19.000000,18.625000 +43 5 24836 0.807985 24836,68393,17317,8924,47715,2830,4934,3820 20.125000,19.250000,18.625000,17.625000,17.625000,17.375000,17.375000,16.750000 +43 6 4934 0.911324 4934,47715,2126,13988,8924,1867,3820,9655 21.375000,21.125000,20.500000,19.125000,18.125000,17.875000,17.000000,16.875000 +43 7 1294 0.710949 1294,1408,1435,12440,22594,2126,3897,1454 24.000000,23.875000,21.250000,20.625000,19.750000,19.000000,18.000000,17.875000 +44 0 1278 0.658991 1278,1032,1420,15087,1531,1373,109492,4682 19.500000,17.500000,17.375000,17.125000,16.875000,16.375000,16.375000,16.125000 +44 1 14696 1.137373 14696,19895,22876,8281,2730,81873,4076,7520 19.125000,18.250000,18.125000,17.125000,17.000000,16.875000,16.625000,16.500000 +44 2 5395 0.257853 5395,25865,18823,50584,2629,4076,14593,3467 21.875000,20.000000,19.000000,17.625000,17.500000,17.125000,17.000000,15.875000 +44 3 42839 7.506424 15087,43193,1373,1531,7613,13446,109492,16202 17.125000,16.750000,16.375000,16.125000,16.000000,15.875000,15.437500,15.250000 +44 4 3986 0.002331 3986,81559,14979,2126,41379,7275,71990,2247 23.500000,16.125000,15.312500,14.500000,14.500000,14.125000,14.125000,13.937500 +44 5 2491 0.011656 2491,65174,2247,3761,1311,1558,3897,67180 22.125000,16.625000,15.687500,15.500000,14.875000,14.437500,13.812500,13.500000 +44 6 1286 0.002840 1286,10694,112030,7051,21541,103291,1076,14111 22.250000,15.375000,14.562500,13.500000,13.375000,12.250000,12.187500,12.125000 +44 7 1294 0.729509 1294,2247,1321,18988,1435,3897,11159,1319 22.875000,22.625000,20.625000,19.500000,19.125000,18.625000,18.500000,18.125000 +45 0 1032 0.071575 1032,1278,1420,2126,1261,6866,15011,2247 25.250000,22.125000,20.625000,19.250000,19.000000,18.625000,18.250000,18.125000 +45 1 1050 0.143312 1050,1049,1032,1052,1010,1057,2126,1051 25.250000,23.375000,16.000000,15.625000,15.437500,15.437500,15.250000,15.187500 +45 2 1048 0.000710 1048,1057,1049,1050,23981,1051,1053,1056 23.875000,14.375000,14.312500,14.062500,13.812500,13.375000,13.125000,13.062500 +45 3 1048 0.013214 1048,1049,1050,2247,1051,1052,1053,1054 27.750000,23.375000,20.250000,17.750000,17.125000,16.500000,16.125000,16.000000 +45 4 1050 1.196178 1050,1049,1048,1053,1051,1052,1054,1055 24.750000,24.500000,24.125000,23.750000,23.500000,23.250000,22.125000,21.000000 +45 5 2247 0.297291 2247,18988,1321,11159,3897,1435,12440,1294 24.250000,22.125000,21.625000,21.500000,20.750000,19.875000,19.875000,19.125000 +45 6 1656 2.559082 2182,1656,6304,8052,11,1032,2409,4410 19.875000,18.500000,18.500000,18.375000,18.250000,18.125000,18.125000,17.625000 +45 7 1032 0.195248 1032,1278,8531,4076,5821,6633,14696,6346 23.375000,20.750000,19.375000,19.000000,18.625000,18.625000,18.625000,18.500000 +46 0 1050 0.010147 1050,1049,1052,1048,1051,1057,1010,1056 27.625000,23.000000,18.000000,17.750000,17.625000,17.625000,17.250000,17.000000 +46 1 1048 0.001173 1048,1049,1057,1050,1052,1051,1056,1053 24.375000,15.687500,15.625000,15.562500,15.562500,15.312500,15.312500,14.875000 +46 2 1048 0.216986 1048,1049,1050,1051,2247,1052,1053,2545 25.500000,24.000000,21.500000,16.875000,16.500000,16.000000,16.000000,14.812500 +46 3 1052 1.755558 1053,1051,1052,1054,1055,1050,1056,1057 23.750000,23.125000,23.125000,22.750000,22.250000,21.875000,21.750000,21.125000 +46 4 89656 9.055080 1514,3897,8052,1321,1794,2247,2165,2182 22.000000,22.000000,18.875000,18.250000,17.875000,17.875000,17.250000,17.125000 +46 5 1397 0.028429 1397,2306,5032,3701,2221,6500,1400,1781 23.250000,19.500000,17.500000,16.250000,14.000000,13.937500,13.750000,13.750000 +46 6 37101 5.759346 8308,1880,10354,5537,56671,47775,1486,18166 22.125000,21.000000,20.750000,19.250000,19.125000,18.625000,18.125000,18.125000 +46 7 1261 0.430397 1261,1278,1794,1420,3866,2126,1408,15720 23.875000,22.500000,22.000000,20.875000,19.750000,19.125000,19.000000,18.750000 +47 0 4934 1.116394 4934,68393,24836,47715,17317,8924,2830,3820 19.500000,19.250000,18.375000,17.750000,17.125000,17.000000,16.875000,16.625000 +47 1 1435 2.438059 1294,1408,1435,12440,1307,1455,22594,1454 24.500000,22.500000,22.375000,21.750000,20.000000,20.000000,19.125000,18.125000 +47 2 1429 7.725399 1261,14686,1420,33550,8001,5939,9225,1278 15.625000,15.312500,14.875000,14.500000,14.312500,14.250000,14.125000,14.062500 +47 3 33190 5.234924 28372,7492,14686,6031,10133,7871,5939,24344 14.000000,13.875000,13.562500,13.500000,13.500000,13.375000,13.250000,13.125000 +47 4 1429 1.278692 1429,1390,1294,1576,76649,12083,28125,118407 17.500000,15.937500,15.312500,15.250000,15.250000,15.125000,15.062500,15.062500 +47 5 1294 0.165510 1294,1408,3897,1319,22594,1278,2126,2274 23.750000,21.625000,19.000000,18.375000,18.375000,18.000000,17.875000,17.875000 +47 6 1278 0.126772 1278,22876,1420,1261,1531,1429,1335,81873 21.375000,18.125000,17.625000,17.375000,16.125000,15.562500,15.312500,15.062500 +47 7 15011 6.080334 22876,14696,19895,4076,7520,81873,2730,8281 19.000000,18.875000,18.250000,17.500000,17.125000,16.750000,16.375000,16.375000 +48 0 1429 1.071444 1429,1576,1307,1531,2126,29818,25994,6799 18.500000,16.875000,16.500000,16.250000,16.125000,16.125000,15.937500,15.625000 +48 1 75653 8.670643 1531,1349,1032,19213,32671,1398,11788,17903 17.500000,15.875000,15.312500,15.187500,14.937500,14.875000,14.750000,14.750000 +48 2 1576 5.571468 28588,39447,1429,16790,64266,21143,54719,1390 21.625000,18.625000,18.000000,18.000000,17.875000,16.750000,16.750000,16.625000 +48 3 1115 0.016808 1115,1265,1576,1294,1429,1321,1319,3897 22.125000,17.000000,15.125000,14.562500,14.375000,13.937500,13.875000,13.750000 +48 4 22479 5.892893 16195,29136,85410,29251,14660,16544,34992,43325 15.875000,15.187500,14.812500,14.437500,14.375000,14.250000,14.062500,14.000000 +48 5 1429 0.057667 1429,1576,4515,3897,14931,30766,2247,25994 23.875000,20.000000,19.125000,18.250000,17.750000,17.375000,17.250000,17.125000 +48 6 1307 0.634089 1307,1294,1408,1562,1319,3897,2247,6421 24.250000,23.375000,22.375000,22.250000,22.000000,20.125000,19.875000,19.375000 +48 7 1278 0.190184 1278,22876,81873,1531,19895,8281,3797,74143 20.875000,18.750000,16.375000,16.125000,15.750000,15.187500,14.937500,14.750000 +49 0 14696 0.485128 14696,22876,19895,7520,5395,8281,81873,25865 20.125000,18.500000,18.000000,17.375000,16.625000,16.375000,16.250000,15.562500 +49 1 5395 0.198286 5395,25865,18823,2629,3467,50584,18944,119435 22.750000,20.750000,19.625000,18.750000,17.500000,16.625000,16.125000,16.000000 +49 2 1531 2.208658 1531,15087,43193,13446,7613,1373,57122,21273 16.500000,15.812500,15.812500,15.750000,15.625000,15.500000,15.500000,15.125000 +49 3 9595 7.286432 11630,16544,11420,16611,104680,8776,22649,64882 18.000000,15.625000,15.250000,14.875000,14.875000,14.750000,14.562500,14.437500 +49 4 77314 6.693283 1372,9595,19232,3897,72269,20681,3060,11420 18.625000,17.500000,14.187500,14.125000,13.875000,13.625000,13.437500,13.125000 +49 5 4295 6.811514 2247,18988,11159,1408,3897,1319,1321,1294 23.750000,21.375000,20.500000,19.875000,19.875000,18.750000,18.375000,17.625000 +49 6 1514 2.570671 1278,1261,1514,1593,1294,1799,1321,1494 18.875000,17.625000,17.500000,17.250000,17.000000,17.000000,16.875000,16.375000 +49 7 56671 5.654018 8308,2095,1486,1880,1395,10354,2168,2430 20.625000,20.000000,19.375000,19.125000,19.000000,18.500000,18.000000,17.750000 +50 0 22594 0.086132 22594,1294,1435,1454,15019,4837,2790,4275 25.625000,22.375000,21.500000,21.500000,20.875000,19.875000,18.250000,18.000000 +50 1 28761 3.140110 46212,6031,9640,13488,20574,28761,6959,37926 18.125000,16.625000,16.625000,16.375000,16.125000,16.125000,15.687500,15.375000 +50 2 6627 6.466169 2516,46212,3907,6959,6031,7456,26291,9640 18.750000,17.750000,17.125000,17.000000,16.875000,16.500000,15.687500,15.500000 +50 3 44258 0.913935 44258,1434,125654,9456,5666,2247,1766,1398 15.750000,14.000000,14.000000,13.937500,13.812500,13.750000,12.812500,12.750000 +50 4 1321 0.277309 1321,3897,2247,1319,1294,43378,1435,4295 24.750000,23.500000,19.875000,19.375000,19.125000,19.000000,18.625000,18.375000 +50 5 46201 7.958225 6031,37926,7871,13488,16192,22511,6959,8720 18.375000,17.500000,17.125000,16.250000,16.250000,16.250000,16.125000,16.000000 +50 6 17379 5.757019 1434,17379,50782,14846,82277,51813,13488,69460 22.875000,17.125000,16.500000,15.187500,14.562500,13.687500,13.562500,13.125000 +50 7 1105 0.001003 1105,1121,2247,1623,1911,4743,1558,3897 23.125000,14.812500,14.500000,13.937500,13.187500,13.062500,12.500000,12.250000 +51 0 2247 0.156485 2247,18988,11159,1294,3897,1321,1435,4295 25.625000,23.375000,22.250000,21.625000,19.500000,19.000000,18.750000,18.625000 +51 1 2182 1.333767 2182,1656,8052,1531,4410,6304,1032,2409 19.375000,18.875000,17.875000,17.750000,17.750000,17.750000,17.250000,17.250000 +51 2 1486 4.376007 8308,2095,1880,10354,56671,1934,1395,2430 22.250000,22.000000,21.375000,20.375000,20.000000,19.750000,19.625000,19.500000 +51 3 7958 2.886438 2095,1294,18166,7958,1261,2430,41516,17732 19.250000,17.500000,17.500000,17.250000,17.125000,17.125000,16.500000,16.375000 +51 4 1294 1.228516 1435,1294,22594,1536,2790,1394,1317,2453 24.125000,23.500000,21.875000,21.625000,20.250000,20.125000,19.750000,18.500000 +51 5 1278 0.334345 1278,1261,22876,1032,1420,1531,3866,1794 19.750000,17.500000,16.875000,16.625000,16.375000,16.000000,15.125000,15.062500 +51 6 1032 2.858027 4076,19895,4934,14696,4798,1032,3354,15011 18.375000,17.625000,17.500000,17.500000,17.375000,17.125000,17.125000,16.625000 +51 7 1050 0.016838 1050,1049,1051,1052,1054,1053,1057,1056 27.500000,23.000000,20.750000,20.750000,20.500000,20.375000,20.000000,19.875000 +52 0 1048 0.003670 1048,1298,1049,1054,1050,1051,1052,1053 26.125000,18.875000,18.500000,18.250000,17.875000,17.875000,17.750000,17.500000 +52 1 1048 0.437099 1048,1049,1050,1411,1462,2545,17008,40438 25.250000,24.625000,20.125000,20.000000,16.875000,15.812500,15.625000,15.625000 +52 2 1053 1.116598 1053,1055,1054,1056,1052,1057,1051,1050 23.875000,23.625000,23.500000,22.500000,22.000000,21.750000,20.250000,19.875000 +52 3 37848 8.346161 4076,14696,19895,7326,15011,81873,5395,8281 20.000000,18.125000,17.750000,17.250000,16.750000,16.375000,16.250000,16.250000 +52 4 53458 6.115427 5466,3354,25349,6267,4934,13079,53458,6103 23.875000,21.000000,20.250000,19.250000,18.125000,18.125000,17.875000,17.625000 +52 5 1307 0.098192 1307,1531,1576,1429,1435,1321,1349,1294 21.875000,17.750000,17.750000,16.875000,16.875000,15.375000,15.312500,15.250000 +52 6 1278 3.347721 1531,1349,1429,1278,1576,1032,9342,4123 17.750000,16.250000,16.250000,15.750000,14.937500,14.750000,14.687500,14.625000 +52 7 14881 7.201817 3354,11733,11552,12226,34614,5046,22511,63239 18.750000,18.000000,17.125000,16.500000,16.000000,15.562500,15.500000,15.000000 +53 0 65812 1.512934 69551,65812,12658,51663,6573,5142,1390,2247 16.625000,16.500000,15.562500,15.437500,14.250000,14.187500,13.812500,13.750000 +53 1 4172 0.005074 4172,1944,1282,1103,20550,3561,18985,1660 25.500000,19.875000,18.875000,15.375000,14.937500,14.125000,13.812500,13.625000 +53 2 3354 0.282866 3354,1576,11733,15179,73768,2196,25865,8466 21.750000,19.125000,18.750000,18.750000,18.375000,17.875000,17.500000,17.125000 +53 3 50349 3.911150 1531,1429,18480,2247,22747,4925,45776,1576 14.937500,14.812500,13.812500,13.500000,13.500000,13.437500,13.062500,13.000000 +53 4 52564 1.056019 52564,1321,1737,22410,2662,3897,4254,11384 16.375000,15.625000,14.812500,14.437500,14.187500,14.125000,13.125000,13.125000 +53 5 3897 3.708563 1321,1513,1319,4295,3897,22594,1294,2247 22.375000,20.625000,19.750000,19.375000,19.125000,19.000000,18.875000,18.750000 +53 6 1799 2.349787 12440,1321,8360,1799,22594,1294,15179,47715 18.750000,18.375000,18.375000,18.250000,18.250000,17.625000,17.500000,17.500000 +53 7 1486 1.160292 1486,9491,56671,10467,1395,1514,18166,6307 19.875000,19.250000,18.750000,18.250000,17.625000,17.500000,17.250000,17.000000 +54 0 5537 0.166100 5537,69105,15179,8360,8077,2158,2095,4326 23.000000,20.375000,19.000000,18.750000,18.500000,17.875000,17.625000,17.625000 +54 1 1513 0.075806 1513,1294,1394,1435,1562,2396,2126,22594 26.000000,23.125000,20.000000,20.000000,19.750000,19.625000,19.500000,19.500000 +54 2 1278 0.023397 1278,2960,1531,22410,1335,1261,14505,7893 23.875000,19.125000,18.500000,18.500000,16.750000,16.125000,15.625000,15.562500 +54 3 73435 10.898415 11552,14505,22410,94670,20361,25172,27640,1353 20.125000,19.250000,18.750000,18.375000,18.125000,17.875000,17.875000,16.750000 +54 4 22410 0.003977 22410,37848,5184,2247,27677,3897,1464,29950 24.250000,18.000000,16.375000,16.250000,16.125000,15.750000,14.937500,14.812500 +54 5 1294 0.579830 1294,3897,2247,1321,69526,62138,4295,22410 22.250000,21.000000,20.250000,20.000000,20.000000,17.875000,17.625000,17.500000 +54 6 69526 0.050829 69526,7893,39337,44099,57909,64599,1278,29531 22.000000,18.375000,17.250000,17.125000,15.750000,15.125000,14.750000,14.750000 +54 7 1321 1.140347 2247,1321,3897,4295,18988,11159,1294,1319 22.875000,22.625000,21.750000,20.875000,20.125000,19.875000,18.500000,18.250000 +55 0 1278 0.454952 1278,2430,4025,2095,1513,1294,1531,77197 20.125000,17.750000,17.375000,17.125000,16.875000,16.625000,16.625000,16.625000 +55 1 12674 7.303574 14505,22410,11552,17241,20361,27640,94670,1353 20.000000,19.125000,18.875000,18.750000,18.375000,18.250000,18.125000,17.250000 +55 2 1720 0.103712 1720,129530,1307,8297,17279,1104,2736,80449 18.875000,14.625000,14.062500,14.062500,14.062500,13.500000,13.500000,12.875000 +55 3 113959 0.022091 113959,1576,22410,43978,22649,66716,73101,1032 23.000000,18.500000,16.750000,16.750000,15.312500,15.062500,14.812500,14.437500 +55 4 42305 0.200829 42305,22410,19404,27270,46897,11384,26182,12190 20.625000,18.875000,15.375000,15.250000,15.062500,14.875000,14.875000,14.812500 +55 5 1294 0.176235 1294,2247,22410,18988,37848,19404,1408,1321 23.875000,21.000000,20.625000,20.375000,19.625000,19.000000,18.750000,18.625000 +55 6 7893 0.002054 7893,1278,1827,36541,1032,56473,6443,2247 25.000000,17.000000,16.125000,16.125000,16.000000,15.500000,15.375000,15.312500 +55 7 2247 0.197456 2247,18988,4295,3897,11159,1321,1294,1454 24.750000,22.250000,21.625000,21.250000,21.125000,20.250000,19.750000,18.500000 +56 0 2182 1.385862 2182,1656,4410,1531,8052,1032,2351,6304 19.625000,19.375000,18.375000,18.250000,18.125000,17.625000,17.500000,17.500000 +56 1 1486 3.058811 8308,2095,1880,56671,10354,5537,1486,1934 21.875000,21.125000,20.750000,20.250000,20.125000,20.000000,19.875000,18.750000 +56 2 15179 8.131947 2095,7958,2430,1294,17732,1261,18166,1805 20.500000,20.125000,18.250000,18.125000,18.125000,17.500000,17.500000,17.000000 +56 3 1536 0.214369 1536,1294,22594,1408,1317,2790,1513,2156 24.750000,23.000000,20.250000,20.000000,19.750000,19.500000,19.375000,19.250000 +56 4 3986 2.958284 26009,15723,6031,3986,26004,1278,14881,9342 17.250000,16.125000,16.000000,15.437500,14.812500,14.750000,14.750000,14.500000 +56 5 94131 2.990242 68099,5511,96789,1434,94131,16286,2301,1407 16.875000,16.000000,15.500000,14.875000,14.812500,14.687500,14.375000,13.187500 +56 6 1321 2.282652 2247,1294,1321,18988,1394,3897,1513,11159 23.250000,23.000000,21.875000,21.250000,20.250000,20.250000,20.000000,20.000000 +56 7 56671 4.104652 2160,5537,9378,1880,1278,1794,22594,8308 17.500000,16.500000,16.500000,16.375000,16.125000,15.937500,15.687500,15.125000 +57 0 22594 0.015835 22594,1454,15019,4837,1294,1435,1408,1513 26.750000,21.875000,21.125000,20.125000,19.875000,19.625000,18.125000,17.875000 +57 1 7492 5.578518 28761,17226,36009,6031,20574,1420,26973,7456 17.875000,16.375000,15.812500,15.375000,15.375000,14.875000,14.812500,14.562500 +57 2 2550 1.312340 2550,112713,20745,116329,47081,10330,68662,73381 16.375000,15.687500,14.375000,14.375000,14.312500,14.125000,14.000000,14.000000 +57 3 2140 0.015478 2140,1275,17939,2383,21631,2783,5818,2476 22.625000,17.875000,16.000000,15.312500,15.312500,15.250000,15.250000,14.687500 +57 4 3026 0.000088 3026,1392,1119,12407,13759,60600,10232,4761 27.125000,16.375000,15.312500,15.187500,14.937500,14.812500,14.750000,14.625000 +57 5 3897 4.085096 2247,1321,18988,1294,11159,3897,1319,4295 24.250000,24.000000,21.875000,21.125000,21.125000,20.875000,19.625000,19.250000 +57 6 51503 9.598694 1454,2274,6031,2430,1321,7456,27555,22765 16.500000,16.375000,15.625000,15.125000,14.875000,14.687500,14.687500,14.500000 +57 7 2163 0.370506 2163,10436,1535,1507,8720,2452,16229,35312 16.625000,14.812500,14.687500,12.562500,11.750000,11.250000,11.250000,10.937500 +58 0 11155 0.497524 11155,1816,1509,2146,16034,2443,11236,11548 16.625000,13.125000,12.937500,12.875000,12.812500,12.750000,12.750000,12.750000 +58 1 3897 2.246742 1321,3897,43378,2247,4295,1101,6421,1097 24.125000,22.000000,17.375000,16.625000,16.625000,16.500000,16.250000,16.125000 +58 2 17690 6.558224 1321,6031,7456,9342,1335,2048,27555,1349 18.500000,16.625000,16.375000,16.000000,15.687500,15.312500,15.250000,15.125000 +58 3 21386 12.102793 2182,1424,1757,27193,36437,44094,12121,1550 14.937500,14.750000,14.187500,14.125000,13.812500,13.500000,13.375000,13.187500 +58 4 3897 4.529589 1321,3897,11564,2140,18988,1263,1353,1319 21.000000,16.500000,15.312500,15.250000,14.125000,14.062500,13.812500,13.000000 +58 5 60952 7.853539 1321,7456,6031,9342,1335,21697,15723,7492 17.625000,16.000000,15.750000,15.250000,14.812500,14.812500,14.687500,14.625000 +58 6 22433 4.806189 12944,1535,5670,5303,17099,11455,1550,7643 15.187500,15.125000,14.500000,14.312500,14.250000,14.062500,14.000000,13.312500 +58 7 1770 1.173275 2306,1770,1273,1428,2432,2454,3641,11423 21.500000,20.750000,17.875000,17.875000,12.125000,12.125000,11.750000,11.500000 +59 0 3897 1.516459 1321,3897,43378,2247,4295,1748,18988,6421 24.625000,23.375000,18.625000,18.375000,17.500000,17.375000,17.375000,17.125000 +59 1 36274 5.732525 1321,7456,6031,9342,7492,7871,16192,26004 16.250000,16.000000,15.812500,15.500000,15.187500,14.875000,14.812500,14.687500 +59 2 101944 3.194407 7456,18112,28297,5303,101944,8972,33554,25832 16.500000,15.312500,15.187500,14.625000,14.500000,14.312500,14.312500,14.000000 +59 3 1321 0.542398 1321,3897,2247,43378,18988,4295,11159,1748 24.125000,23.750000,19.125000,19.125000,18.125000,17.750000,17.625000,17.500000 +59 4 69810 8.384796 6031,7456,1335,9342,6337,16192,6627,26004 15.937500,15.312500,15.187500,15.000000,14.812500,14.687500,14.562500,14.562500 +59 5 9798 7.861500 27374,95197,5046,113148,1704,19533,1359,3582 16.500000,15.687500,15.625000,15.062500,14.875000,14.750000,14.687500,14.625000 +59 6 2247 0.183504 2247,51002,18988,1105,1580,60513,7268,10288 20.000000,16.875000,16.000000,15.375000,15.250000,15.000000,14.937500,14.937500 +59 7 1032 3.368322 1656,2182,1531,1032,8052,4410,6304,89656 19.500000,19.000000,17.625000,17.250000,17.125000,16.875000,16.875000,16.750000 +60 0 1010 0.621542 1010,1267,1050,1049,11,1051,1052,4688 23.250000,22.750000,21.500000,20.000000,17.625000,17.625000,17.625000,17.625000 +60 1 1656 4.436117 1376,2731,34625,1785,1975,3452,1032,8052 16.625000,16.375000,16.375000,15.750000,15.562500,15.312500,15.187500,14.750000 +60 2 1032 0.224707 1032,1278,6866,6633,7199,6346,1794,4454 22.625000,19.750000,18.625000,18.500000,18.500000,18.375000,18.250000,18.125000 +60 3 1050 0.003889 1050,1049,1048,1010,1051,1052,1056,1267 27.375000,21.750000,17.750000,17.000000,17.000000,16.875000,16.500000,16.500000 +60 4 1048 0.000910 1048,1056,1049,1051,1050,1055,1052,1057 23.125000,14.437500,14.062500,13.812500,13.500000,13.500000,13.250000,13.187500 +60 5 1048 0.184443 1048,1049,1050,1054,1053,1051,1055,2247 26.500000,24.875000,21.250000,17.500000,16.375000,16.125000,16.125000,15.750000 +60 6 1054 0.779648 1054,1055,1056,1053,1057,1052,1051,1050 24.000000,23.500000,22.500000,22.375000,21.750000,20.000000,19.500000,19.375000 +60 7 3897 2.213247 1514,89656,3897,8052,1321,1278,1045,1794 19.625000,19.000000,18.125000,16.250000,16.000000,15.812500,15.062500,14.937500 +61 0 89656 1.680552 1514,89656,8052,1278,1294,22594,1794,1454 18.625000,17.625000,16.875000,15.812500,15.375000,14.812500,14.750000,14.437500 +61 1 1397 0.002281 1397,5032,3701,2306,1400,1868,2221,21985 23.625000,16.125000,16.000000,15.937500,15.062500,14.062500,13.750000,13.125000 +61 2 56671 3.877934 8308,10354,1880,37101,5537,56671,24836,1486 21.625000,20.500000,20.250000,19.375000,19.250000,18.500000,18.375000,18.250000 +61 3 22594 0.846823 1294,22594,1435,1454,15019,1408,2790,1513 25.500000,25.500000,24.250000,21.375000,21.125000,20.000000,19.000000,18.875000 +61 4 2550 12.544928 6031,7456,6959,50551,28761,3986,9640,8246 16.625000,16.625000,15.750000,15.750000,15.250000,14.937500,14.937500,14.812500 +61 5 2140 0.565186 2140,3636,1416,21631,1586,1279,1275,88980 17.500000,15.875000,15.812500,14.500000,14.375000,13.625000,13.562500,13.500000 +61 6 3026 0.004623 3026,1392,1325,3326,1786,1417,1816,4308 22.125000,15.562500,14.625000,14.500000,13.937500,13.437500,13.125000,12.875000 +61 7 1294 1.443808 1321,1294,2790,3897,1435,5785,1319,1394 24.125000,23.500000,23.250000,22.625000,20.500000,20.250000,19.625000,18.375000 +62 0 1278 0.122866 1278,1261,1531,1420,1429,3866,1794,38544 20.000000,16.375000,16.250000,15.812500,14.750000,14.500000,14.125000,14.062500 +62 1 3354 2.326672 19895,4076,14696,3354,22876,4958,8281,7316 18.750000,18.625000,18.375000,18.000000,18.000000,17.000000,16.125000,16.000000 +62 2 86315 13.495268 1531,1349,1429,1576,1032,5646,6304,3897 17.625000,16.125000,15.187500,14.937500,14.437500,14.375000,14.250000,14.125000 +62 3 3218 0.301661 3218,3897,1513,1294,1799,2247,1307,1394 19.000000,16.625000,16.000000,15.437500,15.375000,15.250000,15.000000,14.750000 +62 4 8360 2.244915 1513,8360,3897,1294,2247,5537,1435,15179 20.250000,18.750000,18.625000,18.250000,18.125000,17.000000,16.875000,16.625000 +62 5 1536 0.059433 1536,1321,2126,2790,1294,1394,6421,1435 26.500000,23.500000,20.875000,20.250000,19.625000,19.625000,19.125000,18.875000 +62 6 6627 5.552267 14969,15723,1550,20226,6959,8015,24305,16566 16.500000,15.875000,15.375000,15.375000,14.750000,14.750000,14.687500,14.437500 +62 7 49698 2.229761 115520,34852,49698,8631,11737,87236,83480,1535 15.250000,14.437500,14.250000,13.687500,13.625000,12.937500,12.562500,12.437500 +63 0 63649 0.000071 63649,1104,60600,52509,123364,88458,96335,13441 26.875000,16.750000,14.937500,14.437500,14.125000,14.062500,13.562500,13.375000 +63 1 2247 0.672176 2247,1321,3897,1513,4295,1294,1394,1435 24.125000,23.000000,22.625000,21.875000,21.750000,21.375000,20.875000,20.250000 +63 2 2182 2.661970 1531,1032,2157,2182,86315,1656,2409,1534 22.375000,21.375000,20.875000,20.500000,20.000000,19.625000,19.625000,18.125000 +63 3 10354 3.619726 8308,2095,1486,1880,5537,10354,56671,2430 21.250000,20.375000,19.875000,19.125000,19.125000,18.500000,18.250000,17.875000 +63 4 1408 2.558693 1294,22594,1408,1435,1513,1454,2790,27457 23.875000,23.375000,22.000000,21.250000,20.750000,20.375000,19.875000,18.875000 +63 5 1261 4.193077 1278,14696,22876,7316,19895,1420,7026,50462 20.125000,20.000000,18.875000,18.750000,18.625000,17.875000,17.875000,17.750000 +63 6 1032 4.485210 19895,22876,14696,41500,6061,8281,1032,11440 19.625000,17.875000,16.125000,16.125000,15.875000,15.812500,15.625000,15.187500 +63 7 1050 0.051201 1050,1049,1051,1052,1053,1054,1055,1057 26.000000,21.625000,21.500000,21.375000,21.000000,20.625000,20.000000,19.750000 +64 0 1048 0.054482 1048,24926,1052,1805,1054,1462,1053,1056 24.250000,20.125000,19.625000,19.250000,18.750000,18.500000,17.750000,17.500000 +64 1 1048 0.443832 1048,1049,1050,1462,2545,1411,40438,1047 25.125000,24.500000,21.000000,18.875000,17.750000,17.625000,17.375000,17.250000 +64 2 1054 1.351202 1055,1054,1056,1057,1053,1052,1051,1050 23.625000,23.250000,22.875000,22.125000,21.875000,20.250000,19.875000,19.750000 +64 3 15011 0.499495 15011,19895,41500,22876,14696,8281,17058,19055 20.250000,18.750000,17.875000,17.625000,17.250000,16.750000,16.375000,16.125000 +64 4 1307 0.006052 1307,2126,1408,1319,29818,1429,1576,28140 25.875000,18.875000,18.625000,18.500000,18.250000,17.875000,17.875000,17.750000 +64 5 1278 0.328135 1278,1531,51288,19895,109492,2813,22876,1429 20.000000,17.125000,16.250000,16.000000,16.000000,15.750000,15.750000,15.562500 +64 6 14696 0.393539 14696,19895,22876,7520,8281,41500,81873,9371 20.375000,18.500000,18.000000,17.125000,17.000000,16.750000,15.875000,15.812500 +64 7 5395 0.138199 5395,18823,25865,41500,2629,3467,14593,18944 23.000000,20.000000,20.000000,18.750000,18.500000,18.125000,16.875000,16.875000 +65 0 3897 6.085804 51288,1531,2813,43193,15087,109492,1488,1373 17.500000,16.750000,16.500000,16.000000,15.625000,15.437500,15.250000,15.000000 +65 1 43193 4.252312 2813,1531,51288,3140,1799,109492,19148,43193 15.875000,15.312500,14.875000,14.562500,14.437500,13.875000,13.750000,13.750000 +65 2 1839 0.000170 1839,1606,1270,1428,2398,3267,6094,69555 27.500000,17.875000,16.375000,16.250000,15.875000,15.812500,15.687500,15.687500 +65 3 3897 1.029720 2247,3897,1435,18988,1294,11159,4418,12440 22.500000,22.500000,20.875000,20.375000,20.000000,19.750000,19.500000,19.500000 +65 4 7125 9.536839 12440,1435,1294,4418,28140,22594,29818,2478 22.750000,21.250000,20.375000,20.125000,20.125000,19.875000,19.375000,19.250000 +65 5 1536 0.029514 1536,1294,2015,29598,1454,1278,1261,1408 25.875000,21.375000,20.250000,20.125000,19.875000,19.625000,18.875000,18.750000 +65 6 1261 0.495865 1261,1420,1278,24836,1794,3866,15720,12440 21.250000,19.250000,19.125000,18.750000,18.500000,18.375000,17.750000,17.375000 +65 7 4934 1.374912 4934,24836,17317,47715,96432,1032,68393,3709 19.500000,19.500000,18.375000,18.375000,17.875000,17.500000,17.500000,17.375000 +66 0 1294 0.300020 1294,1435,1408,22594,12440,1307,3897,15019 24.750000,23.375000,21.250000,21.000000,20.750000,19.500000,19.000000,18.625000 +66 1 1278 0.089294 1278,1032,1261,22876,1531,1420,19895,3866 21.125000,16.875000,16.625000,16.375000,16.250000,15.937500,15.187500,14.875000 +66 2 1032 1.829241 3354,1032,4076,14696,19895,4958,22876,37848 19.500000,19.000000,18.750000,18.625000,18.500000,17.250000,17.125000,16.500000 +66 3 1050 0.003069 1050,1049,1055,1051,1010,1053,1052,1054 27.625000,21.000000,19.750000,19.250000,19.125000,19.000000,18.875000,18.625000 +66 4 1048 0.000690 1048,1049,1051,1053,1050,1055,1052,1054 26.250000,17.375000,16.500000,16.500000,16.250000,16.250000,16.125000,16.000000 +66 5 1048 0.030497 1048,1049,1050,1411,1055,1462,1051,1054 27.375000,23.875000,19.500000,19.000000,16.625000,16.250000,15.687500,14.812500 +66 6 1055 0.339228 1055,1056,1054,1057,1053,1052,1051,1050 25.750000,24.125000,23.875000,22.625000,20.875000,18.750000,18.125000,17.500000 +66 7 37848 2.836218 4076,14696,19895,3354,37848,4958,22876,15011 19.375000,18.625000,18.000000,17.750000,17.500000,16.500000,16.500000,16.125000 +67 0 5466 0.058907 5466,53458,3354,63239,6267,25349,2629,13079 25.125000,21.750000,21.000000,18.625000,18.375000,18.125000,17.750000,17.750000 +67 1 1307 0.053768 1307,3897,1531,1576,1294,2247,29818,4418 23.125000,18.750000,17.750000,17.625000,16.750000,16.375000,16.250000,16.000000 +67 2 3075 8.138632 1278,1531,1429,15723,22511,1349,6959,19420 19.000000,17.500000,15.437500,15.375000,15.312500,14.875000,14.625000,14.187500 +67 3 1317 2.640059 1278,1362,1531,1317,3870,9595,2247,1265 17.875000,17.000000,16.500000,16.000000,14.750000,14.750000,14.250000,13.875000 +67 4 18275 1.501122 40659,18275,3094,64561,93616,19008,13516,107178 18.125000,17.500000,16.500000,15.812500,15.750000,15.250000,15.187500,14.937500 +67 5 1415 0.188896 1415,3108,1419,1938,39664,2392,86674,81859 19.750000,17.625000,16.625000,15.437500,15.375000,14.375000,14.312500,12.375000 +67 6 15179 16.176008 1294,1278,1261,1454,2247,1513,1321,1737 21.375000,19.750000,19.125000,19.125000,19.125000,19.000000,18.375000,18.000000 +67 7 1536 0.012289 1536,2790,8360,1321,2126,6421,1513,5785 25.625000,19.750000,19.375000,19.250000,18.625000,18.625000,18.500000,17.375000 +68 0 8233 4.726301 15723,26004,26009,16566,6031,3986,7456,16136 16.750000,16.625000,15.875000,15.500000,15.312500,15.250000,15.125000,15.125000 +68 1 1386 0.000416 1386,3390,16926,97044,33151,2755,8101,22719 23.375000,13.937500,13.312500,13.125000,13.062500,12.750000,12.562500,12.375000 +68 2 1434 0.389526 1434,10307,1757,1535,1550,7456,14857,30377 17.875000,15.625000,14.812500,14.562500,14.500000,14.500000,13.562500,13.375000 +68 3 1479 2.510841 10917,1539,1479,2765,25763,3164,4940,15392 15.000000,14.500000,13.500000,12.750000,12.000000,11.500000,11.500000,11.500000 +68 4 1965 0.000135 1965,1107,2247,1985,1121,2454,12623,1837 25.250000,15.750000,13.750000,13.375000,12.875000,12.687500,12.375000,11.750000 +68 5 2247 0.450438 2247,1513,1321,1294,3897,18988,1454,47715 24.375000,22.750000,22.500000,21.375000,21.250000,20.875000,20.375000,20.000000 +68 6 3075 6.792654 2182,1032,1656,1531,2409,8052,4973,3264 20.250000,19.625000,19.375000,18.875000,18.250000,17.875000,17.250000,17.125000 +68 7 1317 0.040453 1317,3870,1278,9327,6793,6421,2247,4058 22.250000,18.500000,15.750000,15.437500,15.062500,15.000000,14.937500,14.875000 +69 0 18275 0.009983 18275,121786,1359,56669,83955,6421,44477,43060 26.125000,20.250000,19.750000,18.750000,18.625000,18.375000,18.000000,17.500000 +69 1 1415 0.000019 1415,1314,1672,2247,32571,1778,1419,3108 27.500000,15.062500,14.625000,13.687500,13.625000,13.312500,13.125000,12.937500 +69 2 1486 0.537025 1486,1395,56671,3897,51677,18166,57062,9491 21.750000,20.375000,18.625000,18.375000,18.125000,18.000000,17.875000,17.625000 +69 3 5537 0.517582 5537,8360,2158,1261,1420,4057,8460,46828 22.125000,21.250000,19.000000,18.625000,18.375000,17.750000,17.750000,17.750000 +69 4 1513 0.178122 1513,1294,1435,1394,22594,1317,1408,1536 25.250000,23.000000,21.625000,20.500000,20.500000,20.000000,19.750000,19.750000 +69 5 27640 9.973724 1278,2960,22410,1531,1335,1261,34614,7893 23.250000,18.625000,18.250000,17.875000,17.750000,16.625000,16.500000,15.750000 +69 6 22410 0.008812 22410,10786,27677,2247,1109,19404,28869,35229 23.250000,17.250000,16.875000,16.000000,15.750000,15.625000,15.437500,15.437500 +69 7 1294 1.200583 1294,2247,1321,3897,1454,18988,1435,1394 22.625000,22.625000,22.500000,20.375000,20.000000,19.250000,19.125000,18.875000 +70 0 1278 10.189384 7893,1032,36541,11250,1278,2247,75728,5663 25.625000,18.875000,16.500000,15.875000,15.437500,15.375000,15.312500,15.125000 +70 1 7893 2.370991 5046,7893,1032,3354,14719,2158,6509,10159 17.000000,15.812500,15.750000,15.687500,14.750000,14.625000,14.312500,14.125000 +70 2 81947 0.126022 81947,13375,2247,70574,3714,37848,4457,5046 20.875000,17.500000,17.375000,17.125000,16.125000,15.875000,14.937500,14.750000 +70 3 1307 0.004030 1307,9214,2247,1281,6421,18988,11159,35549 25.875000,19.000000,18.875000,18.375000,17.250000,17.125000,16.875000,16.500000 +70 4 38199 4.465088 108492,53875,3316,38199,16210,35549,7893,45261 22.125000,21.000000,18.250000,18.000000,17.500000,16.875000,16.750000,16.625000 +70 5 1428 0.000004 1428,1920,52636,8315,49887,95522,1325,58401 31.750000,18.625000,18.250000,16.125000,15.812500,15.250000,14.812500,14.687500 +70 6 52829 0.000206 52829,23825,10786,6989,1425,11564,40158,2247 25.875000,16.375000,15.437500,15.000000,14.500000,14.250000,13.750000,13.562500 +70 7 1321 0.019262 1321,2247,1748,18988,11159,3897,4295,6421 25.500000,21.125000,19.250000,18.875000,18.375000,17.625000,17.625000,16.750000 +71 0 118799 0.011860 118799,38199,75728,44841,10330,39185,21374,8394 22.125000,16.250000,16.000000,15.937500,14.625000,14.437500,14.375000,14.187500 +71 1 2247 0.304941 2247,1321,18988,3897,1294,11159,1454,4295 23.750000,21.625000,21.375000,20.250000,20.000000,20.000000,19.625000,19.125000 +71 2 89656 5.044851 1032,1656,2182,1531,8052,2409,2157,6304 20.250000,19.875000,19.875000,18.500000,17.750000,17.250000,17.125000,16.875000 +71 3 1397 0.001402 1397,5032,3701,1400,1868,2683,1781,2221 24.250000,16.125000,16.000000,15.937500,15.062500,14.250000,13.812500,13.625000 +71 4 56671 3.457333 8308,10354,1576,1880,2095,1486,56671,5537 20.500000,18.875000,18.750000,18.750000,18.625000,18.375000,18.000000,17.875000 +71 5 1294 2.296755 22594,1294,1435,1454,15019,1408,1513,2790 25.500000,23.375000,22.500000,20.125000,20.000000,18.875000,18.750000,18.750000 +71 6 2295 4.886812 1278,1261,1420,1593,1531,2295,1032,22876 20.250000,18.125000,16.375000,16.125000,15.687500,15.625000,15.562500,15.500000 +71 7 13068 4.027369 23981,4958,11733,1032,4514,14696,13068,19895 20.250000,18.000000,17.625000,17.000000,16.750000,16.750000,16.625000,16.125000 +72 0 1294 1.097185 1294,7358,15179,3897,8360,8077,1455,1307 20.625000,20.000000,19.500000,18.875000,18.750000,18.375000,18.000000,17.750000 +72 1 1032 0.020216 1032,1278,3897,5124,7276,1766,6032,2960 25.500000,20.750000,18.625000,18.500000,18.125000,18.000000,18.000000,17.875000 +72 2 1050 0.000174 1050,1049,1052,1048,1051,1056,1053,1032 27.000000,16.750000,16.625000,16.125000,15.500000,15.437500,15.375000,15.312500 +72 3 1048 0.000187 1048,1032,1050,1049,3897,1051,1010,1052 26.000000,16.125000,15.687500,14.625000,14.187500,13.937500,13.687500,13.500000 +72 4 1048 0.113440 1048,1049,1050,1737,2247,1056,1032,1051 27.125000,25.000000,19.750000,14.875000,14.687500,14.500000,14.375000,14.187500 +72 5 1056 0.745590 1056,1055,1057,1054,1053,1052,1051,1050 24.000000,23.500000,23.000000,21.750000,20.125000,18.250000,17.500000,17.125000 +72 6 3897 1.408869 2247,3897,1737,4295,1775,1319,15179,2251 22.125000,21.625000,20.500000,20.250000,20.125000,19.125000,19.125000,19.000000 +72 7 10298 9.025219 15179,2960,1278,3907,1531,1925,28140,12440 15.750000,15.562500,15.437500,15.437500,15.250000,14.875000,14.812500,14.750000 +73 0 7421 1.164299 1307,7421,5661,2247,1032,15247,32022,53728 18.500000,18.125000,16.750000,16.500000,15.375000,14.937500,14.750000,14.500000 +73 1 9225 1.779373 9225,1576,1321,92791,7114,1032,1353,1681 16.625000,16.500000,16.125000,16.000000,14.937500,14.625000,14.562500,14.562500 +73 2 67465 0.000309 67465,14397,1575,6575,2097,3862,2249,1616 33.500000,25.375000,21.625000,21.250000,18.000000,16.250000,16.000000,15.562500 +73 3 1536 6.210867 1321,15179,22594,3897,1319,8360,2247,47715 23.750000,21.750000,21.375000,20.500000,20.375000,20.125000,19.000000,19.000000 +73 4 130537 4.747353 41520,16566,3986,6031,7456,12245,15723,36356 17.125000,16.875000,16.375000,16.375000,16.375000,16.375000,16.375000,16.125000 +73 5 6993 10.911230 6031,7456,3986,15723,16566,8720,1349,8343 17.250000,16.875000,16.750000,16.375000,16.250000,16.000000,15.937500,15.625000 +73 6 2301 6.295448 2163,10307,15450,2301,13586,5303,2247,49305 19.500000,17.000000,14.250000,13.312500,13.312500,13.187500,12.625000,12.312500 +73 7 125371 1.967705 125371,11461,48227,7976,1398,73607,1760,51039 13.312500,12.812500,12.437500,12.187500,11.437500,11.312500,11.187500,10.937500 +74 0 3897 1.867010 4728,1321,3897,75094,1305,4295,3614,2247 20.000000,19.875000,19.000000,16.625000,16.375000,15.750000,15.625000,15.000000 +74 1 1321 0.342402 1321,22594,1799,7125,47715,1294,12440,2478 22.000000,19.750000,19.375000,19.000000,18.750000,18.000000,18.000000,17.500000 +74 2 6797 7.231401 1531,1278,1294,2095,1656,1349,1434,29251 17.500000,15.937500,15.312500,14.750000,14.625000,14.062500,14.062500,14.000000 +74 3 3892 1.486950 1576,3892,5303,14258,60878,1257,2405,1681 17.000000,16.750000,15.625000,14.562500,14.562500,14.500000,14.437500,14.375000 +74 4 1390 8.094606 2247,3897,2662,1536,2314,4295,10689,11159 15.875000,14.312500,13.562500,13.250000,13.125000,13.062500,12.812500,12.250000 +74 5 8919 0.000078 8919,61245,3257,1518,6465,1906,12200,1783 25.875000,14.687500,14.375000,14.250000,13.937500,13.875000,13.312500,13.187500 +74 6 15179 1.081272 2247,15179,1536,3897,47715,1319,8360,4295 21.250000,21.125000,19.750000,18.750000,18.750000,18.375000,18.250000,18.125000 +74 7 1536 0.004560 1536,1321,2790,3897,22594,1294,6421,2247 25.375000,18.375000,17.375000,17.375000,17.375000,17.125000,17.000000,16.750000 +75 0 8595 1.751838 8595,26305,43252,36356,33407,25656,40006,7114 13.562500,13.187500,11.812500,11.625000,11.500000,11.250000,11.250000,10.875000 +75 1 1405 0.084127 1405,5625,1269,2302,19019,11254,6291,23666 20.875000,17.750000,17.250000,14.937500,14.500000,14.437500,14.375000,13.312500 +75 2 88542 1.026257 88542,10307,28958,2247,26515,66533,84292,78734 15.625000,14.562500,14.437500,13.375000,13.250000,12.750000,12.625000,12.500000 +75 3 2247 0.109041 2247,18988,11159,4295,3897,1319,1321,1294 25.375000,22.625000,21.250000,21.125000,20.875000,18.875000,18.875000,18.750000 +75 4 1656 3.184423 2182,1032,12953,1531,1656,6797,10298,6304 20.750000,20.250000,19.750000,18.875000,18.625000,18.500000,18.125000,17.625000 +75 5 4454 5.970627 1032,1278,2960,6346,6866,4454,6633,5821 24.625000,21.500000,19.500000,19.375000,19.125000,18.750000,18.750000,18.625000 +75 6 1032 0.081230 1032,1307,3897,1321,1462,1278,1050,1576 23.875000,20.625000,19.750000,19.375000,17.375000,17.000000,16.875000,16.875000 +75 7 1050 0.001472 1050,1049,1048,1056,1053,1051,1052,1055 26.500000,18.625000,18.125000,17.875000,17.750000,17.625000,17.375000,17.000000 +76 0 1048 0.000719 1048,1032,1050,1055,1053,1051,1057,1056 25.875000,16.625000,16.500000,16.250000,16.125000,16.000000,16.000000,15.937500 +76 1 1048 0.530743 1048,1049,1050,1057,3897,1051,2247,1032 25.125000,24.750000,20.750000,15.625000,15.000000,14.187500,14.187500,13.812500 +76 2 1056 0.884495 1057,1056,1055,1054,1053,1052,1050,1051 24.375000,24.125000,21.625000,20.625000,20.000000,18.625000,17.625000,17.625000 +76 3 3897 0.501106 3897,1514,89656,1278,1794,43378,1321,2165 20.125000,18.875000,18.625000,16.750000,15.250000,14.812500,14.500000,14.437500 +76 4 89656 0.990620 89656,1514,1278,1531,1494,1794,8052,10298 17.250000,17.125000,15.500000,14.812500,13.812500,13.062500,13.000000,12.937500 +76 5 1397 0.015303 1397,5032,1400,21985,3701,31814,3504,1781 20.875000,15.437500,15.125000,14.687500,14.562500,12.812500,12.750000,12.437500 +76 6 3214 4.576775 8308,10354,1486,1880,5537,37101,52158,56671 19.125000,18.625000,18.000000,18.000000,18.000000,18.000000,17.500000,17.375000 +76 7 1261 0.571478 1261,1794,1420,24836,1278,3866,96432,29857 23.875000,23.250000,22.250000,19.000000,18.375000,18.125000,18.000000,17.750000 +77 0 24836 0.721910 24836,96432,4958,13426,6061,1850,8602,4076 20.500000,20.125000,18.250000,17.250000,17.000000,16.500000,15.937500,15.812500 +77 1 13988 0.027514 13988,47715,96432,14696,8281,2126,4934,8602 23.125000,18.000000,17.375000,17.125000,16.500000,16.250000,16.125000,16.125000 +77 2 1408 0.446144 1408,1294,1435,22594,12440,1513,1454,1321 25.250000,24.375000,22.750000,21.875000,20.750000,19.375000,19.125000,18.750000 +77 3 1261 4.890537 1278,22876,19895,1531,1420,14696,1261,14279 20.375000,18.125000,17.625000,17.375000,16.250000,16.250000,15.812500,15.812500 +77 4 2295 6.582502 6061,19895,22876,14696,1875,15011,78172,41500 16.875000,16.750000,16.125000,16.000000,15.687500,15.437500,15.437500,15.375000 +77 5 2126 5.770285 24926,1462,1805,109431,1045,50824,2126,15011 22.875000,20.250000,19.625000,18.375000,17.875000,17.500000,17.250000,16.875000 +77 6 1045 0.033033 1045,1462,1123,93023,1032,39450,1035,18242 21.625000,17.625000,15.375000,15.187500,15.062500,14.000000,13.312500,13.125000 +77 7 1064 0.001209 1064,2126,73856,93023,115587,59962,3332,1035 26.625000,18.500000,18.500000,18.000000,17.375000,16.375000,15.812500,15.500000 +78 0 1805 0.084648 1805,15011,12271,1635,31194,14400,24926,106817 23.750000,20.500000,20.250000,18.500000,18.125000,16.750000,16.500000,16.500000 +78 1 15011 0.106480 15011,41500,8303,19895,6061,5395,13268,25865 20.750000,16.250000,15.937500,15.937500,15.812500,15.062500,14.875000,14.875000 +78 2 19976 13.645441 1307,29818,1408,4418,1294,28140,1319,2126 25.625000,20.625000,20.250000,19.000000,18.750000,18.750000,18.625000,18.125000 +78 3 1307 0.307863 1307,1408,1294,29818,28140,1435,1394,3897 22.750000,21.375000,19.500000,18.375000,18.125000,18.000000,17.000000,16.750000 +78 4 1278 0.142071 1278,15087,1531,81873,19895,43193,2813,22876 20.375000,16.125000,16.000000,15.437500,15.312500,15.000000,14.937500,14.875000 +78 5 14696 0.585827 14696,19895,22876,7520,9371,5150,81873,8281 19.750000,18.250000,17.625000,17.500000,17.125000,16.125000,15.875000,15.750000 +78 6 5395 0.103643 5395,25865,18823,2629,14593,50584,3467,119435 22.500000,19.500000,18.875000,17.875000,17.500000,16.625000,16.000000,15.750000 +78 7 1488 4.048735 3897,15087,1531,2813,8238,51288,18736,43193 17.375000,16.000000,15.500000,14.937500,14.750000,14.750000,14.562500,14.500000 +79 0 6812 0.349485 6812,89220,24591,6970,2939,5635,1493,25632 19.375000,18.375000,14.562500,14.062500,13.500000,13.375000,13.312500,13.312500 +79 1 1278 0.023243 1278,1531,8250,29251,3897,2015,2247,1032 21.500000,17.000000,15.875000,14.750000,14.312500,14.250000,14.062500,13.375000 +79 2 29251 0.039413 29251,19213,1407,8776,49944,3897,39447,34627 21.750000,16.250000,16.000000,16.000000,15.812500,15.250000,15.250000,15.125000 +79 3 3897 3.202885 2247,18988,29818,3897,1435,22594,11159,1454 22.750000,20.625000,20.125000,20.000000,19.750000,19.750000,19.625000,19.125000 +79 4 7125 4.195463 12440,28140,29818,1294,1435,8308,7125,22594 20.625000,18.250000,18.000000,17.625000,17.625000,17.000000,16.875000,16.750000 +79 5 1536 0.064726 1536,1294,29598,2015,4025,1454,1261,1278 25.875000,22.500000,21.000000,20.500000,20.250000,19.875000,19.750000,19.625000 +79 6 1420 2.290291 1261,1420,3866,24836,1278,1794,15720,29857 22.375000,20.500000,20.000000,19.750000,19.250000,19.250000,18.500000,17.625000 +79 7 13988 0.377918 13988,15011,1457,7481,47353,87170,107254,19976 21.375000,19.750000,19.375000,16.750000,16.625000,16.125000,15.937500,15.812500 diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w1.tsv b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w1.tsv new file mode 100644 index 000000000..35c3f2a8c --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w1.tsv @@ -0,0 +1,135 @@ +# model models/mlx/Qwen3-30B-A3B-4bit +# device GPU +# corpus tests/fixtures/wikitext2_excerpt.txt tokens 299190 +# chunks 128 chunk_tokens 1 topk 8 prefill 0 +# columns chunk pos target nll top_ids top_logits +# start_token 0 +# contiguous false +0 0 42105 16.055737 1124,370,273,2533,37018,709,1318,654 21.125000,18.750000,16.500000,15.687500,15.687500,15.375000,15.312500,15.250000 +1 0 275 12.277327 1124,11520,2533,1318,921,273,9640,277 21.750000,17.250000,16.125000,15.000000,14.875000,14.687500,14.562500,14.187500 +2 0 427 14.390408 1124,370,11520,654,37018,2533,273,408 22.875000,20.000000,19.500000,19.125000,17.250000,17.125000,16.750000,16.625000 +3 0 12 6.239361 273,1124,370,8,11,37018,1318,354 14.687500,14.625000,14.375000,14.062500,13.937500,13.937500,13.750000,13.062500 +4 0 17 8.380493 1124,2533,921,1318,11520,1125,370,408 19.375000,15.000000,14.750000,14.562500,14.500000,14.375000,14.250000,14.062500 +5 0 3393 14.111338 1124,2533,11520,8,921,701,60,568 17.375000,15.812500,15.812500,15.625000,15.562500,14.937500,14.000000,14.000000 +6 0 75734 16.384632 1124,370,273,67,220,11,408,8 15.812500,15.187500,12.812500,12.687500,12.562500,12.375000,12.375000,12.250000 +7 0 79483 16.187334 1124,370,3808,654,220,15,516,668 20.625000,18.625000,16.875000,16.750000,16.000000,15.937500,15.750000,15.750000 +8 0 3608 15.645130 1124,654,370,220,39022,516,668,3044 21.250000,18.875000,18.125000,17.125000,16.250000,16.125000,16.125000,15.625000 +9 0 257 11.067087 220,315,25,12,15,17,323,2038 15.375000,15.062500,14.937500,13.875000,13.812500,13.812500,13.437500,13.187500 +10 0 549 11.604107 1124,353,1125,2533,921,3044,15,60 19.750000,14.750000,14.562500,14.187500,13.125000,13.125000,13.062500,12.875000 +11 0 3703 12.460676 1124,353,1125,2533,1318,921,701,9533 16.500000,13.062500,12.875000,12.812500,12.500000,11.937500,11.625000,11.437500 +12 0 1110 9.560026 1124,59,220,25,11,13,16,5305 16.125000,12.187500,12.062500,11.937500,11.750000,11.687500,11.625000,11.625000 +13 0 71 5.047022 1124,370,59,220,1699,85,15,12 17.625000,14.687500,14.500000,14.437500,14.187500,14.000000,13.812500,13.625000 +14 0 35268 15.583735 1124,11520,2533,8,921,37018,408,701 23.375000,19.250000,15.750000,15.375000,15.375000,15.000000,14.937500,14.750000 +15 0 1564 8.877424 1124,220,71,709,304,370,3422,65 16.250000,14.312500,13.812500,12.875000,12.812500,12.750000,12.687500,12.625000 +16 0 6830 13.037117 1124,370,654,668,516,59,220,284 20.625000,17.875000,16.750000,16.750000,16.625000,16.250000,16.250000,16.000000 +17 0 3446 5.859240 1124,59,346,60,273,370,9533,17 18.875000,17.125000,15.937500,15.750000,15.750000,15.687500,15.625000,15.500000 +18 0 76649 14.339861 8,1124,921,1318,408,60,701,11 15.375000,15.375000,14.625000,14.562500,14.500000,14.250000,14.187500,14.125000 +19 0 11374 7.959442 1124,370,13,516,11,7370,25,59 15.437500,14.687500,14.062500,14.062500,14.000000,13.937500,13.875000,13.875000 +20 0 3831 13.737196 1124,8,11,701,273,60,25,2533 17.375000,15.125000,14.875000,14.687500,14.312500,14.062500,13.875000,13.875000 +21 0 8833 15.266094 1124,3226,516,323,315,11,220,4260 14.562500,14.250000,13.125000,13.000000,12.812500,12.687500,12.312500,12.312500 +22 0 6324 13.884331 654,1124,370,516,4730,273,12,85 20.250000,18.375000,17.250000,17.000000,16.375000,16.000000,15.937500,15.812500 +23 0 60403 14.132729 1124,408,273,8,2533,60,921,11520 18.000000,17.000000,15.562500,15.437500,15.125000,15.062500,15.062500,14.812500 +24 0 427 14.991070 370,12,220,1124,516,3226,15,85 14.937500,14.750000,13.875000,13.500000,13.125000,13.000000,12.812500,12.750000 +25 0 198 5.239361 273,1124,370,8,11,37018,1318,354 14.687500,14.625000,14.375000,14.062500,13.937500,13.937500,13.750000,13.062500 +26 0 70584 21.144226 1124,370,654,1699,59,3044,516,3422 24.375000,20.750000,19.750000,18.125000,17.500000,17.000000,16.625000,16.625000 +27 0 257 11.794857 1124,370,273,65,220,15,315,516 18.375000,15.375000,14.562500,14.125000,14.000000,13.687500,13.687500,13.687500 +28 0 549 11.604107 1124,353,1125,2533,921,3044,15,60 19.750000,14.750000,14.562500,14.187500,13.125000,13.125000,13.062500,12.875000 +29 0 289 9.866926 1124,353,1125,2533,1318,921,701,9533 16.500000,13.062500,12.875000,12.812500,12.500000,11.937500,11.625000,11.437500 +30 0 60403 16.127209 1124,11520,8,2533,408,921,701,11 19.000000,16.875000,16.500000,16.125000,15.875000,15.875000,15.750000,15.375000 +31 0 427 14.991070 370,12,220,1124,516,3226,15,85 14.937500,14.750000,13.875000,13.500000,13.125000,13.000000,12.812500,12.750000 +32 0 12 6.239361 273,1124,370,8,11,37018,1318,354 14.687500,14.625000,14.375000,14.062500,13.937500,13.937500,13.750000,13.062500 +33 0 17 8.380493 1124,2533,921,1318,11520,1125,370,408 19.375000,15.000000,14.750000,14.562500,14.500000,14.375000,14.250000,14.062500 +34 0 12 8.236338 1124,2533,11520,8,921,701,60,568 17.375000,15.812500,15.812500,15.625000,15.562500,14.937500,14.000000,14.000000 +35 0 1041 15.349243 1124,2533,921,1318,11520,1125,370,408 19.375000,15.000000,14.750000,14.562500,14.500000,14.375000,14.250000,14.062500 +36 0 8273 10.944777 1124,59,62,12,75738,284,79,220 19.000000,17.875000,17.875000,17.125000,16.750000,16.500000,16.000000,15.687500 +37 0 16 10.794870 1124,11520,2533,408,921,9640,1318,37018 19.125000,19.125000,18.625000,17.500000,17.375000,16.875000,16.375000,16.375000 +38 0 198 4.899763 13,16,17,15,5373,18,24,20 21.000000,20.000000,19.375000,19.250000,18.875000,18.750000,18.750000,18.250000 +39 0 20193 20.628601 1124,370,654,1699,59,3044,516,3422 24.375000,20.750000,19.750000,18.125000,17.500000,17.000000,16.625000,16.625000 +40 0 414 10.604345 1124,220,12,370,59,654,17,75 16.875000,15.437500,14.812500,14.687500,14.375000,14.062500,13.625000,13.437500 +41 0 549 13.814505 1124,2533,921,1125,353,1318,60,273 21.125000,16.125000,15.312500,15.187500,14.625000,14.312500,14.187500,13.625000 +42 0 1273 11.210676 1124,353,1125,2533,1318,921,701,9533 16.500000,13.062500,12.875000,12.812500,12.500000,11.937500,11.625000,11.437500 +43 0 320 6.181293 1124,370,67,11,273,8,3226,220 14.625000,14.375000,12.437500,12.000000,11.812500,11.562500,11.562500,11.500000 +44 0 19 12.063049 1124,37018,370,2533,11520,1318,273,701 22.875000,17.875000,17.750000,17.750000,17.750000,17.250000,17.125000,16.500000 +45 0 18 3.332907 13,15,17,9754,20,23,18,19 19.625000,18.375000,18.250000,18.250000,17.875000,17.875000,17.750000,17.750000 +46 0 20 8.196136 1124,2533,8,701,11520,921,11,273 18.250000,16.000000,15.875000,15.750000,15.250000,15.187500,14.625000,14.562500 +47 0 23 3.715046 13,20,15,9754,17,18,23,5373 20.000000,19.125000,18.875000,18.875000,18.625000,18.250000,17.875000,17.750000 +48 0 6978 9.627481 13,9754,15,23,17,18,19,20 19.125000,18.625000,18.250000,18.250000,18.000000,17.750000,17.500000,17.500000 +49 0 340 10.299179 1124,220,315,323,59,11,516,1125 18.375000,14.250000,13.187500,12.562500,12.375000,12.312500,12.250000,12.187500 +50 0 9827 17.907396 1124,654,370,3044,7370,15,220,85 21.250000,18.250000,17.625000,16.375000,16.375000,16.000000,15.937500,15.500000 +51 0 262 8.578988 25,198,315,323,12,85,271,9 13.187500,12.812500,12.625000,12.375000,12.312500,12.312500,12.250000,12.125000 +52 0 549 14.336964 1124,2533,370,921,1125,1318,273,654 21.125000,15.875000,14.875000,14.500000,14.062500,14.000000,13.937500,13.687500 +53 0 13534 14.226301 1124,353,1125,2533,1318,921,701,9533 16.500000,13.062500,12.875000,12.812500,12.500000,11.937500,11.625000,11.437500 +54 0 7710 13.541580 1124,8,11,60,701,273,15,2533 15.687500,13.812500,13.437500,12.812500,12.812500,12.625000,12.437500,12.250000 +55 0 92214 12.990833 220,12,1124,62,3044,85,716,516 17.625000,17.000000,16.250000,15.375000,15.125000,15.062500,15.000000,14.937500 +56 0 220 2.211271 220,1124,15,17,16,12,85,3226 13.750000,13.687500,12.500000,12.500000,12.312500,12.125000,12.125000,11.750000 +57 0 18 8.400600 1124,370,2533,1125,15,921,17,16 21.125000,15.687500,15.062500,14.500000,14.375000,13.937500,13.812500,13.750000 +58 0 13 7.383636 1124,2533,8,701,11520,921,11,273 18.250000,16.000000,15.875000,15.750000,15.250000,15.187500,14.625000,14.562500 +59 0 15 11.005911 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.375000,16.000000,15.812500,15.125000,15.062500,14.625000 +60 0 320 8.640917 15,13,271,16,17,11,18,198 23.625000,19.375000,18.500000,18.375000,18.250000,18.125000,17.875000,17.875000 +61 0 2428 17.906799 1124,37018,370,2533,11520,1318,273,701 22.875000,17.875000,17.750000,17.750000,17.750000,17.250000,17.125000,16.500000 +62 0 1110 0.002611 1110,4,25,14375,322,549,60896,23817 21.250000,13.750000,13.687500,12.375000,12.312500,12.000000,11.250000,11.000000 +63 0 85231 14.781397 1124,370,59,220,1699,85,15,12 17.625000,14.687500,14.500000,14.437500,14.187500,14.000000,13.812500,13.625000 +64 0 2659 1.338520 2659,12,905,9533,1125,60,17,25 16.250000,15.437500,15.000000,14.125000,13.750000,13.687500,13.625000,13.500000 +65 0 6971 9.640205 9533,14186,13,11,25,3470,220,1125 12.687500,12.125000,11.687500,11.562500,11.562500,11.562500,11.500000,11.500000 +66 0 67733 9.276325 14,25815,3470,14186,12,315,23303,2899 14.562500,13.625000,13.500000,13.500000,13.437500,13.125000,13.062500,13.000000 +67 0 1331 5.217037 12,85,1124,71,15,67,73,86 17.500000,17.375000,17.250000,17.125000,16.875000,16.875000,16.750000,16.750000 +68 0 64 10.657888 1124,2533,701,11,8,273,1125,921 18.625000,16.500000,16.125000,15.812500,15.750000,15.500000,15.312500,15.187500 +69 0 14 8.413801 370,1124,284,668,220,16,17,62 20.625000,20.375000,18.875000,18.625000,18.250000,17.875000,17.250000,17.250000 +70 0 18 5.813028 1124,370,273,3226,15,17,709,16 18.625000,15.625000,15.250000,14.500000,14.312500,14.312500,14.312500,14.187500 +71 0 13 7.383636 1124,2533,8,701,11520,921,11,273 18.250000,16.000000,15.875000,15.750000,15.250000,15.187500,14.625000,14.562500 +72 0 15 11.005911 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.375000,16.000000,15.812500,15.125000,15.062500,14.625000 +73 0 53560 17.047167 15,13,271,16,17,11,18,198 23.625000,19.375000,18.500000,18.375000,18.250000,18.125000,17.875000,17.875000 +74 0 19665 15.734356 370,12,220,1124,71,65,67,3226 16.375000,14.937500,14.562500,14.312500,13.812500,13.750000,13.750000,13.687500 +75 0 1488 9.138861 273,59,258,75,408,654,613,198 13.187500,12.687500,12.250000,11.937500,11.875000,11.687500,11.500000,11.437500 +76 0 25 5.722244 370,1124,668,220,315,516,323,11 16.625000,15.000000,13.937500,13.875000,13.500000,13.437500,13.000000,12.875000 +77 0 26587 18.819664 1124,370,273,2533,1125,15,1318,37018 23.250000,17.125000,14.875000,14.812500,14.750000,14.500000,14.500000,14.187500 +78 0 20343 10.281675 220,13,15,9533,25,12,905,516 13.187500,12.812500,12.562500,12.500000,12.312500,12.250000,12.187500,11.937500 +79 0 11 5.076550 1124,1699,497,3808,59,5305,9533,25 16.000000,12.875000,12.500000,12.500000,12.062500,12.062500,12.000000,11.875000 +80 0 80935 22.978584 1124,2533,1125,1318,701,709,37018,370 22.125000,16.875000,16.500000,16.000000,15.750000,15.375000,15.312500,15.250000 +81 0 15235 13.192146 370,1124,220,3226,17,13,16,25 17.125000,16.750000,15.125000,14.687500,14.562500,14.437500,14.250000,14.125000 +82 0 8319 13.997744 1124,13,11,59,25,85,12,62 19.625000,14.562500,14.500000,14.437500,14.125000,14.000000,13.812500,13.687500 +83 0 198 7.368225 1124,7110,9533,13,11,60,370,220 16.875000,14.062500,13.937500,13.562500,13.375000,13.187500,13.062500,12.812500 +84 0 1797 14.300476 1124,370,654,1699,59,3044,516,3422 24.375000,20.750000,19.750000,18.125000,17.500000,17.000000,16.625000,16.625000 +85 0 320 8.408138 1124,353,220,1125,3044,15,1154,8358 17.625000,16.750000,13.750000,13.375000,13.125000,12.500000,12.187500,12.125000 +86 0 50 13.313049 1124,37018,370,2533,11520,1318,273,701 22.875000,17.875000,17.750000,17.750000,17.750000,17.250000,17.125000,16.500000 +87 0 13 5.219259 1124,370,11,8,15,67,17,25 18.250000,16.125000,14.812500,14.437500,14.250000,14.125000,14.062500,14.062500 +88 0 8755 19.349661 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.375000,16.000000,15.812500,15.125000,15.062500,14.625000 +89 0 487 13.800753 1124,11520,2533,408,921,273,709,1318 19.750000,18.375000,17.375000,16.500000,16.500000,15.812500,15.687500,15.687500 +90 0 11 3.898251 315,1124,220,323,516,11,304,13 15.500000,14.625000,14.250000,14.187500,13.750000,13.500000,13.375000,13.187500 +91 0 451 13.033272 1124,2533,1125,1318,701,709,37018,370 22.125000,16.875000,16.500000,16.000000,15.750000,15.375000,15.312500,15.250000 +92 0 13 7.153843 1124,8,11,11520,921,60,701,2533 18.875000,14.312500,14.062500,13.875000,13.437500,13.375000,13.125000,12.625000 +93 0 49256 22.529348 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.375000,16.000000,15.812500,15.125000,15.062500,14.625000 +94 0 812 13.452112 1124,3226,370,11520,4260,408,273,277 23.500000,18.125000,17.500000,16.750000,15.625000,15.562500,15.375000,15.250000 +95 0 82977 18.236521 370,1124,3226,4260,67,408,273,5305 20.250000,19.500000,16.000000,14.250000,14.187500,14.187500,14.062500,13.812500 +96 0 28134 12.714342 1124,3226,370,71,59,4260,273,277 22.250000,16.500000,15.875000,15.562500,15.500000,15.437500,15.375000,14.937500 +97 0 11 7.998283 1124,370,11520,654,273,408,2533,277 21.875000,19.250000,17.500000,17.250000,16.125000,15.625000,15.437500,15.312500 +98 0 431 13.158272 1124,2533,1125,1318,701,709,37018,370 22.125000,16.875000,16.500000,16.000000,15.750000,15.375000,15.312500,15.250000 +99 0 13 8.368755 1124,370,11520,408,2533,37018,921,273 18.875000,17.000000,16.750000,16.500000,15.750000,15.500000,15.187500,15.125000 +100 0 2055 18.912161 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.375000,16.000000,15.812500,15.125000,15.062500,14.625000 +101 0 9034 20.459425 1124,370,11,11520,654,2533,67,273 22.250000,15.687500,15.312500,15.187500,14.625000,14.625000,14.562500,14.562500 +102 0 1959 15.593395 1124,3226,273,408,4260,11,71,709 20.000000,16.875000,15.687500,15.500000,15.375000,15.125000,14.937500,14.812500 +103 0 330 7.192925 220,1154,323,370,1124,11,481,701 10.937500,10.687500,9.812500,9.687500,9.562500,9.500000,9.437500,9.375000 +104 0 9084 20.640753 1124,370,1318,2533,37018,709,11520,1125 23.000000,17.750000,17.250000,16.500000,16.375000,16.000000,16.000000,15.875000 +105 0 56793 18.188000 353,9,1124,220,12,59,486,323 19.000000,16.000000,15.750000,15.625000,14.937500,14.437500,14.250000,14.187500 +106 0 386 5.937124 220,12,15,16,17,353,516,1124 14.625000,13.562500,13.500000,12.937500,12.937500,12.687500,12.687500,12.437500 +107 0 12735 20.770061 1124,8,11520,11,921,701,2533,60 19.750000,14.937500,14.875000,14.375000,14.062500,13.875000,13.687500,13.437500 +108 0 26874 9.759089 315,1124,220,17,370,323,62,7370 16.000000,15.562500,15.312500,15.062500,15.000000,14.687500,14.625000,14.625000 +109 0 497 5.607233 315,1124,220,323,13,11,304,25 14.437500,14.000000,13.875000,13.812500,13.500000,13.125000,12.937500,12.687500 +110 0 220 4.757244 1124,1125,2533,6878,220,1318,15,1467 19.375000,16.125000,15.437500,15.250000,14.875000,14.625000,14.375000,14.375000 +111 0 17 7.338100 1124,370,2533,1125,15,921,17,16 21.125000,15.687500,15.062500,14.500000,14.375000,13.937500,13.812500,13.750000 +112 0 15 5.048838 1124,2533,11520,8,921,701,60,568 17.375000,15.812500,15.812500,15.625000,15.562500,14.937500,14.000000,14.000000 +113 0 16 5.328417 15,13,271,16,17,11,18,198 23.625000,19.375000,18.500000,18.375000,18.250000,18.125000,17.875000,17.875000 +114 0 21 4.399763 13,16,17,15,5373,18,24,20 21.000000,20.000000,19.375000,19.250000,18.875000,18.750000,18.750000,18.250000 +115 0 340 6.756157 13,21,9754,15,17,18,19,23 19.875000,19.375000,19.375000,18.875000,18.875000,18.875000,18.625000,18.500000 +116 0 1797 11.563646 1124,654,370,3044,7370,15,220,85 21.250000,18.250000,17.625000,16.375000,16.375000,16.000000,15.937500,15.500000 +117 0 39183 16.861263 1124,353,220,1125,3044,15,1154,8358 17.625000,16.750000,13.750000,13.375000,13.125000,12.500000,12.187500,12.125000 +118 0 3701 10.518444 1124,220,15,17,13,315,370,323 14.875000,14.562500,13.437500,13.250000,12.937500,12.937500,12.937500,12.687500 +119 0 25 7.978191 1124,370,668,353,59,220,516,323 19.000000,17.875000,15.875000,14.562500,13.937500,13.750000,13.750000,13.250000 +120 0 3703 16.132164 1124,370,273,2533,1125,15,1318,37018 23.250000,17.125000,14.875000,14.812500,14.750000,14.500000,14.500000,14.187500 +121 0 1110 9.560026 1124,59,220,25,11,13,16,5305 16.125000,12.187500,12.062500,11.937500,11.750000,11.687500,11.625000,11.625000 +122 0 71 5.047022 1124,370,59,220,1699,85,15,12 17.625000,14.687500,14.500000,14.437500,14.187500,14.000000,13.812500,13.625000 +123 0 35268 15.583735 1124,11520,2533,8,921,37018,408,701 23.375000,19.250000,15.750000,15.375000,15.375000,15.000000,14.937500,14.750000 +124 0 1564 8.877424 1124,220,71,709,304,370,3422,65 16.250000,14.312500,13.812500,12.875000,12.812500,12.750000,12.687500,12.625000 +125 0 6830 13.037117 1124,370,654,668,516,59,220,284 20.625000,17.875000,16.750000,16.750000,16.625000,16.250000,16.250000,16.000000 +126 0 3446 5.859240 1124,59,346,60,273,370,9533,17 18.875000,17.125000,15.937500,15.750000,15.750000,15.687500,15.625000,15.500000 +127 0 76649 14.339861 8,1124,921,1318,408,60,701,11 15.375000,15.375000,14.625000,14.562500,14.500000,14.250000,14.187500,14.125000 diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w8.tsv b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w8.tsv new file mode 100644 index 000000000..b08a65860 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_default_w8.tsv @@ -0,0 +1,647 @@ +# model models/mlx/Qwen3-30B-A3B-4bit +# device GPU +# corpus tests/fixtures/wikitext2_excerpt.txt tokens 299190 +# chunks 80 chunk_tokens 8 topk 8 prefill 512 +# columns chunk pos target nll top_ids top_logits +# start_token 0 +# contiguous false +0 0 42105 15.901297 1124,370,273,37018,2533,709,1318,654 20.875000,18.750000,16.375000,15.562500,15.437500,15.187500,15.125000,15.062500 +0 1 275 5.916378 285,685,83964,27391,81080,10980,579,307 17.375000,17.375000,16.500000,15.875000,15.187500,14.812500,14.625000,14.312500 +0 2 427 0.002298 427,644,765,779,706,29551,1897,327 24.875000,17.625000,16.875000,16.125000,15.875000,15.562500,15.187500,14.937500 +0 3 12 5.890383 369,47761,25,311,320,4035,198,374 18.375000,18.375000,17.875000,17.750000,17.750000,17.750000,17.625000,17.500000 +0 4 17 1.253744 17,16,18,19,15,6570,20,5097 17.875000,17.375000,16.250000,16.000000,15.812500,15.687500,15.000000,14.562500 +0 5 3393 6.649996 320,10337,198,75734,12,39183,15,42094 20.125000,19.250000,18.875000,18.625000,18.500000,18.500000,18.375000,18.000000 +0 6 75734 2.964243 2885,220,320,2573,4035,75734,39183,25 19.625000,19.250000,19.000000,19.000000,19.000000,19.000000,18.875000,18.125000 +0 7 79483 3.072130 320,4035,198,79483,25,271,348,284 21.500000,20.375000,19.875000,19.500000,19.375000,18.750000,18.750000,18.500000 +1 0 3608 8.997362 785,1986,418,58,15505,54,2,32 23.375000,20.625000,19.625000,19.125000,19.125000,19.000000,18.750000,18.375000 +1 1 257 11.779507 25,510,1467,198,320,323,1447,3542 26.375000,22.375000,22.375000,21.625000,21.625000,21.500000,21.500000,21.250000 +1 2 549 0.197664 549,760,284,508,320,481,3703,8478 26.250000,24.375000,22.500000,20.625000,20.500000,20.250000,20.000000,19.250000 +1 3 3703 1.033129 3703,508,26587,576,1758,220,256,662 22.250000,21.250000,21.250000,20.750000,20.125000,20.000000,19.625000,19.000000 +1 4 1110 0.000206 1110,25,549,14375,322,198,510,4 25.625000,16.000000,15.125000,14.687500,14.562500,14.250000,13.812500,13.312500 +1 5 71 0.589958 71,2136,82,11659,2911,1041,5204,268 23.625000,22.875000,21.375000,20.750000,20.250000,20.000000,19.875000,19.500000 +1 6 35268 0.000048 35268,2596,13619,23518,768,12,7876,4076 30.125000,19.250000,19.000000,17.125000,16.750000,16.625000,16.625000,16.375000 +1 7 1564 0.000019 1564,3579,18596,29088,16281,30985,46031,22185 34.000000,22.750000,21.250000,19.875000,18.875000,18.250000,17.500000,17.250000 +2 0 6830 0.000004 6830,71433,11021,29425,1858,35883,18798,57409 32.000000,17.375000,16.750000,16.500000,16.250000,16.250000,15.750000,15.437500 +2 1 3446 0.000080 3446,1737,14,14953,25967,75988,29425,3183 31.375000,21.250000,20.375000,19.625000,19.500000,17.500000,17.375000,17.000000 +2 2 76649 0.000011 76649,8369,4917,2096,4886,746,82,29425 37.000000,25.625000,20.500000,19.375000,19.375000,19.250000,18.375000,18.250000 +2 3 11374 9.202957 6324,4846,14,22175,15792,25502,23354,90936 29.375000,26.000000,25.625000,24.875000,23.250000,23.000000,22.875000,22.375000 +2 4 3831 0.422424 3831,309,3288,54,1466,74,306,3466 24.375000,23.375000,20.750000,20.500000,20.375000,19.500000,19.500000,19.250000 +2 5 8833 0.004412 8833,18573,48796,82010,824,49099,2566,10458 29.625000,23.500000,22.125000,22.000000,21.000000,20.875000,20.125000,20.000000 +2 6 6324 0.051945 6324,22175,14,2899,10776,46928,2687,4846 33.000000,30.000000,27.250000,24.250000,23.500000,22.125000,21.625000,21.375000 +2 7 60403 0.011055 60403,1579,14939,417,17,18677,2963,90669 33.250000,28.750000,21.000000,18.875000,18.625000,17.875000,17.625000,17.625000 +3 0 427 0.000460 427,38649,477,327,4788,480,2252,1002 28.375000,19.875000,19.750000,17.500000,17.500000,17.125000,16.000000,15.875000 +3 1 198 0.694378 198,271,12,320,17,715,34827,11 25.750000,25.500000,23.000000,23.000000,22.250000,21.125000,21.125000,20.875000 +3 2 70584 8.182957 13806,1712,5009,6262,929,36930,33363,34613 20.000000,19.875000,19.875000,19.875000,19.250000,19.250000,19.125000,19.125000 +3 3 257 0.018347 257,549,414,256,262,25,315,981 27.750000,22.000000,21.500000,21.250000,21.250000,21.125000,21.125000,20.875000 +3 4 549 0.000038 549,6260,760,30372,937,284,14512,481 32.500000,21.625000,20.375000,20.250000,19.375000,19.250000,18.250000,17.875000 +3 5 289 0.155863 289,330,97175,364,28609,1273,42105,220 28.250000,25.625000,25.250000,24.375000,23.000000,22.875000,22.625000,22.125000 +3 6 60403 -0.000000 60403,14628,417,31580,5840,96363,2501,5566 35.250000,19.125000,17.625000,17.250000,15.875000,15.875000,15.687500,15.125000 +3 7 427 0.000015 427,2252,477,327,5722,1303,38649,1318 38.500000,27.250000,24.625000,21.625000,20.125000,19.750000,19.750000,18.375000 +4 0 12 0.000099 12,16839,481,17,198,1273,1944,50342 35.000000,25.625000,22.750000,22.375000,21.625000,21.000000,20.750000,20.750000 +4 1 17 0.000027 17,16,1041,4266,15,18,21037,19 36.500000,26.000000,21.125000,19.375000,18.875000,17.750000,17.750000,17.000000 +4 2 12 0.257523 12,198,8273,271,2852,320,715,16839 26.500000,24.875000,23.875000,21.750000,20.875000,20.625000,20.625000,20.375000 +4 3 1041 0.000296 1041,7112,21550,586,5761,39884,17,9199 30.625000,21.625000,20.625000,19.500000,19.375000,19.125000,18.750000,18.750000 +4 4 8273 0.001163 8273,198,271,98056,12,2268,13333,25502 25.750000,18.250000,16.250000,15.875000,15.750000,15.750000,15.750000,15.375000 +4 5 16 0.000008 16,17,15,18,449,1272,59854,275 29.625000,17.375000,15.812500,14.750000,14.750000,14.437500,13.375000,13.062500 +4 6 198 0.031952 198,271,320,13,715,2303,11,1406 29.375000,25.500000,24.125000,23.375000,23.125000,22.500000,20.250000,20.125000 +4 7 20193 2.131334 13806,20193,6262,5009,34613,5637,2833,59286 21.750000,20.375000,19.625000,19.375000,19.375000,19.125000,18.875000,18.125000 +5 0 414 0.020603 414,257,981,549,262,286,256,260 27.250000,22.750000,21.875000,20.625000,20.125000,19.750000,19.500000,19.375000 +5 1 549 0.000010 549,760,320,30372,284,6260,937,3504 31.625000,18.500000,18.250000,18.125000,18.000000,17.625000,17.250000,16.375000 +5 2 1273 1.922794 5426,1273,10519,330,364,4862,2697,1044 29.875000,28.125000,24.500000,23.875000,23.750000,23.750000,23.500000,22.125000 +5 3 320 5.180874 198,271,320,715,11,2303,1406,4710 29.500000,26.500000,24.375000,21.875000,21.250000,21.125000,20.750000,20.000000 +5 4 19 3.892439 16,17,18,20,19,22,23,24 24.125000,22.625000,22.250000,21.375000,21.000000,20.750000,20.750000,20.750000 +5 5 18 2.721973 15,17,20,11,16,18,19,23 27.250000,26.000000,25.750000,25.625000,25.625000,25.625000,25.625000,25.500000 +5 6 20 2.766611 11,17,15,18,13,19,23,21 26.125000,25.750000,25.500000,25.375000,25.250000,25.250000,25.250000,25.125000 +5 7 23 2.919186 11,17,15,18,21,19,16,20 25.500000,25.375000,25.250000,25.125000,25.000000,24.875000,24.625000,24.625000 +6 0 6978 7.336081 10295,15,17,18,21,19,16,20 23.875000,23.625000,23.625000,23.625000,23.625000,23.500000,23.375000,23.250000 +6 1 340 0.515198 340,692,11,8,304,701,608,2790 29.375000,28.750000,26.625000,26.250000,24.375000,23.875000,23.375000,23.375000 +6 2 9827 6.201912 13806,5009,13965,3323,6262,4061,7595,2505 20.375000,19.500000,19.250000,19.125000,19.000000,18.625000,18.500000,18.375000 +6 3 262 0.006216 262,257,25,549,256,414,220,981 28.125000,22.250000,21.125000,21.000000,20.750000,19.625000,19.375000,19.000000 +6 4 549 0.000072 549,6260,14512,937,320,30372,220,481 32.500000,22.375000,21.375000,20.875000,20.000000,19.250000,18.125000,18.125000 +6 5 13534 1.780506 14865,13534,8914,24263,576,12527,5264,21693 25.250000,24.000000,23.875000,22.500000,21.000000,20.875000,20.500000,20.250000 +6 6 7710 1.858629 7671,7710,15,12,220,58318,14319,92214 24.125000,22.500000,21.250000,18.875000,17.625000,17.250000,17.000000,15.625000 +6 7 92214 1.338308 12,92214,220,16589,20059,320,7778,95942 24.500000,23.625000,22.625000,19.875000,19.375000,17.875000,17.000000,16.125000 +7 0 220 0.077545 220,198,320,271,12,323,11,348 27.000000,23.750000,23.125000,22.750000,21.500000,19.125000,18.750000,18.375000 +7 1 18 1.146996 19,18,17,16,15,20,220,320 28.875000,28.125000,24.625000,21.500000,18.500000,17.875000,17.875000,17.125000 +7 2 13 0.000019 13,198,271,320,15,11,624,323 30.750000,18.625000,17.875000,17.750000,17.375000,17.000000,16.750000,16.375000 +7 3 15 -0.000000 15,16,17,20,18,19,21,24 35.750000,21.250000,19.625000,18.500000,18.250000,18.125000,17.375000,17.000000 +7 4 320 2.209612 198,271,320,323,476,11,715,1876 26.625000,25.000000,24.750000,22.125000,21.625000,21.500000,20.750000,20.750000 +7 5 2428 0.779636 2428,4060,300,1958,2427,4197,62946,1782 22.750000,20.500000,20.125000,19.875000,19.875000,19.750000,19.750000,19.625000 +7 6 1110 0.000168 1110,25,14375,549,340,692,322,486 25.125000,15.562500,14.500000,13.437500,13.375000,13.250000,12.875000,12.812500 +7 7 85231 0.406576 85231,268,2136,71,5204,2154,67,79 22.750000,21.250000,20.625000,20.125000,19.375000,18.375000,17.250000,17.125000 +8 0 2659 0.000004 2659,5713,13,61501,8382,11021,24282,53834 31.875000,18.375000,17.625000,16.375000,15.187500,15.125000,15.125000,15.062500 +8 1 6971 0.000553 6971,43986,14,10845,53560,6580,33914,340 27.125000,18.375000,17.500000,17.500000,17.250000,17.000000,16.625000,16.500000 +8 2 67733 0.000034 67733,14,87310,553,340,692,14319,2687 35.250000,24.375000,23.250000,22.375000,21.625000,21.500000,21.500000,21.125000 +8 3 1331 0.000420 1331,14,12,2687,92214,5894,1112,2268 32.500000,24.625000,21.875000,21.000000,20.250000,17.125000,16.250000,15.812500 +8 4 64 0.000004 64,32,14,309,14186,264,1478,1112 36.250000,23.000000,21.375000,18.375000,17.625000,17.500000,17.500000,17.375000 +8 5 14 0.000021 14,53560,50807,5894,33149,340,8680,692 29.625000,18.000000,17.125000,16.875000,16.500000,15.375000,14.937500,14.375000 +8 6 18 0.003857 18,17,19,16,1112,31218,61827,692 34.000000,27.750000,27.750000,20.375000,18.250000,16.500000,16.500000,16.250000 +8 7 13 0.000004 13,14,1993,94329,22445,15,4338,1725 36.000000,22.500000,21.500000,20.000000,18.625000,18.500000,18.000000,17.500000 +9 0 15 -0.000000 15,16,20,24,18,17,21,19 32.000000,17.250000,15.500000,15.500000,14.125000,13.250000,12.375000,12.187500 +9 1 53560 0.700508 53560,14,50807,340,47036,692,22549,34545 24.750000,24.500000,23.000000,21.125000,20.250000,19.875000,19.500000,19.125000 +9 2 19665 8.808187 5009,13806,8856,1043,6262,5637,1695,11377 20.625000,19.750000,18.375000,18.250000,18.250000,17.375000,17.125000,17.125000 +9 3 1488 0.002705 1488,3977,2774,1070,262,22705,13,332 25.875000,18.625000,18.250000,18.125000,17.250000,17.250000,17.000000,17.000000 +9 4 25 0.001860 25,549,510,1447,91,5293,1995,3546 28.500000,21.250000,19.875000,19.875000,18.875000,18.875000,18.125000,18.125000 +9 5 26587 4.281845 576,1096,3703,330,5209,467,80935,89840 21.000000,20.750000,20.250000,19.750000,19.375000,19.000000,18.875000,18.875000 +9 6 20343 4.534290 320,323,374,11,198,10337,271,594 22.625000,22.625000,22.375000,21.875000,21.750000,21.500000,21.375000,21.125000 +9 7 11 2.217674 271,13,198,11,320,382,26,624 27.625000,27.375000,27.375000,26.625000,26.625000,24.375000,23.125000,23.000000 +10 0 80935 9.615000 330,220,1036,364,323,508,714,89840 24.500000,22.250000,21.625000,20.625000,20.375000,20.375000,20.375000,20.375000 +10 1 15235 3.177839 198,271,11,8319,4848,15235,320,382 22.875000,22.875000,22.375000,22.375000,21.625000,21.125000,20.875000,20.625000 +10 2 8319 0.035326 8319,271,3412,320,11,198,1800,2083 27.000000,22.000000,22.000000,21.875000,21.750000,21.625000,20.500000,20.375000 +10 3 198 1.709133 271,198,320,11,382,13,1406,2083 27.000000,25.750000,24.875000,24.625000,23.500000,22.750000,22.625000,22.125000 +10 4 1797 7.400085 5009,13806,198,8856,6262,1043,1695,2833 20.500000,18.875000,18.625000,18.625000,18.250000,18.000000,17.625000,17.625000 +10 5 320 0.259911 320,3703,323,508,330,198,481,366 24.000000,22.250000,20.125000,19.875000,19.500000,18.875000,18.500000,18.375000 +10 6 50 8.674345 2428,4060,9830,785,1254,17,3703,1958 25.625000,20.875000,19.875000,19.375000,19.375000,19.250000,19.250000,19.000000 +10 7 13 0.231306 13,1466,36544,524,6128,10080,3779,6383 20.750000,17.625000,17.125000,16.750000,16.375000,16.250000,15.937500,15.875000 +11 0 8755 7.757391 431,425,73580,444,479,36020,468,87900 21.875000,21.000000,18.625000,18.375000,18.250000,18.250000,17.750000,17.750000 +11 1 487 0.003477 487,75,2352,385,31245,291,12212,28600 26.125000,18.875000,18.375000,17.750000,17.250000,17.125000,17.000000,16.875000 +11 2 11 0.038940 11,1842,323,13,26,609,701,320 30.000000,26.750000,22.375000,20.375000,20.125000,20.000000,19.875000,19.750000 +11 3 451 0.747910 451,1842,362,356,431,328,422,468 27.750000,27.000000,26.125000,25.875000,25.500000,25.250000,24.250000,23.875000 +11 4 13 0.000051 13,1785,2572,875,808,10003,11,624 31.375000,21.000000,19.375000,18.875000,18.750000,18.375000,17.375000,16.750000 +11 5 49256 11.108501 425,14695,328,90019,1863,38979,11840,96160 18.625000,16.125000,15.562500,15.250000,15.125000,15.125000,14.812500,14.625000 +11 6 812 0.662867 812,5559,585,276,12466,19042,6190,88 22.500000,22.375000,18.125000,17.375000,17.250000,17.125000,17.000000,16.875000 +11 7 82977 3.781088 11,82977,1842,323,38767,13,328,2932 22.125000,18.375000,15.187500,14.687500,14.687500,14.437500,14.375000,14.375000 +12 0 28134 0.001785 28134,404,838,5840,8698,86,261,5059 27.000000,19.375000,19.375000,19.250000,18.000000,16.500000,16.500000,16.375000 +12 1 11 0.002996 11,323,1842,609,13,320,340,345 30.375000,24.125000,23.250000,20.500000,20.375000,19.750000,19.750000,19.375000 +12 2 431 0.626019 431,323,1842,444,451,425,479,386 27.125000,26.750000,23.750000,23.500000,23.500000,23.375000,23.375000,22.750000 +12 3 13 0.000015 13,3503,836,5142,808,875,11,11352 34.000000,21.625000,20.875000,20.750000,20.375000,20.375000,20.125000,19.500000 +12 4 2055 2.167864 1863,2055,619,431,328,74787,25867,730 17.125000,16.750000,16.375000,15.812500,15.687500,15.687500,15.625000,15.562500 +12 5 9034 0.000803 9034,11437,331,5403,71,28064,9917,20658 27.750000,20.125000,19.250000,17.000000,16.250000,16.125000,15.500000,15.437500 +12 6 1959 9.414848 11,692,13,340,320,8,26,701 28.000000,28.000000,26.875000,26.375000,26.000000,25.500000,24.000000,23.875000 +12 7 330 2.531332 220,3703,330,1490,369,1036,279,5813 21.375000,21.000000,20.125000,19.875000,19.125000,18.875000,18.125000,18.000000 +13 0 9084 8.570395 46358,2082,8813,32,12092,49,785,33876 23.625000,21.500000,21.375000,21.250000,20.750000,20.375000,20.375000,20.250000 +13 1 56793 1.776150 38710,56793,14155,80778,12,8141,23361,198 22.875000,21.375000,19.750000,19.125000,17.375000,17.375000,17.000000,16.500000 +13 2 386 1.104485 19219,386,27343,198,85471,18977,7487,34098 21.875000,21.750000,20.125000,19.750000,19.125000,18.625000,18.500000,18.500000 +13 3 12735 0.047470 12735,51525,95296,2375,17020,13470,10298,31102 24.000000,20.625000,19.500000,17.250000,15.875000,15.562500,14.937500,14.937500 +13 4 26874 0.000389 26874,4903,198,1,220,38710,715,4119 31.875000,23.375000,22.250000,21.000000,21.000000,20.875000,20.750000,20.625000 +13 5 497 1.328856 1,497,5130,1138,899,698,1335,3263 27.875000,27.125000,25.750000,25.625000,23.875000,23.375000,22.875000,22.875000 +13 6 220 0.160736 220,796,3703,358,1644,715,330,19288 24.625000,22.125000,21.000000,20.625000,19.625000,19.125000,18.500000,18.500000 +13 7 17 0.000729 17,16,18,220,3703,19,796,20 30.875000,22.875000,21.500000,21.250000,20.375000,20.125000,19.875000,19.750000 +14 0 15 0.000013 15,303,16,18,17,23,19,24 30.000000,17.750000,17.000000,16.875000,16.500000,15.812500,15.500000,15.187500 +14 1 16 0.018217 16,17,15,4146,18,19,339,6148 29.250000,25.250000,19.625000,14.000000,13.875000,13.812500,13.812500,13.812500 +14 2 21 0.828245 21,23,22,24,20,15,19,18 25.750000,25.500000,24.875000,23.375000,18.250000,15.187500,14.687500,14.312500 +14 3 340 2.651638 692,340,11,9272,8,1959,13,3593 28.500000,26.000000,24.500000,24.375000,23.875000,23.625000,23.500000,23.125000 +14 4 1797 1.068933 1797,5009,715,3374,6262,13806,36930,1060 21.375000,20.625000,18.750000,18.750000,18.750000,18.375000,18.250000,18.125000 +14 5 39183 8.501850 320,3703,323,508,330,1959,3496,576 24.750000,22.750000,21.625000,20.375000,20.125000,19.625000,19.250000,19.000000 +14 6 3701 9.062365 25,4008,5548,2319,374,3465,3565,7662 21.875000,20.750000,20.750000,20.500000,20.250000,20.250000,19.625000,19.625000 +14 7 25 0.418365 25,320,323,518,369,549,271,198 28.000000,26.500000,25.125000,25.000000,24.500000,24.500000,24.375000,24.250000 +15 0 3703 0.018906 3703,320,330,508,220,305,366,715 27.500000,22.125000,21.500000,21.375000,21.125000,21.125000,20.250000,20.125000 +15 1 1110 0.000086 1110,25,14375,60896,322,549,486,14111 26.875000,16.375000,16.125000,14.875000,14.812500,14.750000,14.000000,14.000000 +15 2 71 0.001699 71,44754,5204,4951,2136,65546,4489,47293 30.000000,23.125000,22.250000,20.625000,20.500000,18.625000,17.875000,17.750000 +15 3 35268 0.000023 35268,2596,768,45124,13299,20146,723,17 36.000000,25.250000,20.000000,18.875000,18.375000,17.000000,16.250000,16.125000 +15 4 1564 -0.000000 1564,22185,7605,46031,3579,16281,29088,18596 38.750000,20.875000,20.750000,20.125000,20.000000,19.375000,18.250000,17.750000 +15 5 6830 -0.000000 6830,11021,8639,71433,15593,22057,4245,52114 33.000000,18.875000,17.250000,17.250000,16.750000,16.625000,16.500000,16.125000 +15 6 3446 0.000027 3446,25967,14,37462,11374,1737,2687,20268 34.750000,23.875000,23.000000,19.375000,19.250000,18.250000,18.250000,17.750000 +15 7 76649 0.000004 76649,8369,4917,2096,4886,746,19346,2032 36.000000,23.375000,21.750000,16.750000,16.750000,16.625000,15.687500,15.125000 +16 0 11374 0.001936 11374,6324,2687,14,22175,3446,3183,25502 31.375000,24.375000,24.125000,22.875000,21.125000,20.375000,19.375000,19.375000 +16 1 3831 -0.000000 3831,68130,1574,15829,35418,13,29041,95929 42.500000,23.375000,22.625000,21.125000,19.750000,19.375000,17.000000,16.750000 +16 2 8833 0.000011 8833,824,1324,1958,627,18573,1192,48796 33.500000,21.500000,20.500000,19.250000,18.500000,18.375000,18.250000,18.125000 +16 3 6324 -0.000000 6324,14,22175,25502,198,271,5894,15889 37.250000,22.750000,21.875000,20.875000,20.750000,20.125000,20.125000,19.250000 +16 4 60403 0.000004 60403,31580,1579,97401,96363,275,7570,8226 37.000000,24.500000,23.875000,19.125000,18.250000,18.125000,18.125000,17.875000 +16 5 427 -0.000000 427,327,37220,2252,1303,38649,5722,477 40.500000,24.500000,24.000000,23.875000,23.375000,22.375000,21.375000,20.750000 +16 6 271 0.190680 271,198,4710,1406,2,715,382,26809 27.750000,26.125000,22.125000,21.875000,21.750000,20.375000,20.250000,20.250000 +16 7 33236 13.025576 785,418,28,1986,5009,565,2,9112 21.750000,20.125000,19.625000,19.625000,18.500000,18.250000,18.125000,18.000000 +17 0 414 1.366865 414,25,315,549,981,320,257,369 22.375000,21.250000,20.750000,20.750000,20.750000,20.375000,20.125000,20.125000 +17 1 549 0.000242 549,6260,760,320,284,30372,14512,481 30.000000,21.000000,19.375000,19.125000,18.500000,18.375000,18.125000,17.625000 +17 2 2480 5.025223 220,330,1273,576,320,508,289,1156 21.750000,21.500000,21.500000,21.250000,20.750000,20.500000,20.375000,20.250000 +17 3 1273 5.296442 320,198,1467,9345,4326,22870,271,1318 21.375000,21.000000,20.750000,20.125000,19.500000,19.375000,18.750000,17.750000 +17 4 6718 5.017664 738,320,198,10337,271,1467,6718,42094 26.750000,24.750000,24.250000,22.750000,22.375000,22.375000,22.000000,21.875000 +17 5 320 1.050947 198,320,271,11,315,448,369,438 29.000000,28.750000,27.750000,26.000000,25.500000,25.500000,24.125000,24.125000 +17 6 541 2.651321 19,2152,541,300,1921,16,28996,1811 24.000000,23.750000,22.500000,21.625000,21.500000,21.375000,21.375000,20.875000 +17 7 220 2.286644 6978,220,10295,821,10469,1467,22870,11211 24.000000,22.875000,22.875000,22.750000,22.375000,22.000000,22.000000,21.875000 +18 0 19 0.014612 19,16,18,17,20,23,21,22 29.500000,24.000000,23.875000,23.375000,23.250000,23.000000,22.125000,21.500000 +18 1 18 0.007820 18,11,74,17,15,13,16,19 29.750000,24.250000,23.375000,22.250000,22.125000,21.125000,21.125000,21.125000 +18 2 20 0.000673 20,23,19,21,17,18,22,15 30.625000,22.000000,21.625000,21.625000,21.125000,21.000000,20.500000,20.125000 +18 3 23 0.000231 23,22,24,21,17,18,20,19 30.500000,21.375000,21.000000,19.250000,18.625000,18.625000,18.625000,18.375000 +18 4 6978 1.028637 6978,10295,10469,22870,5128,10695,13121,9293 24.500000,24.000000,23.750000,23.000000,22.125000,21.750000,21.375000,21.250000 +18 5 97534 14.308701 340,692,8,315,701,11,304,568 28.750000,28.750000,27.000000,25.750000,25.750000,25.375000,25.250000,24.125000 +18 6 340 1.634146 340,448,1119,692,438,304,11,323 27.375000,27.250000,27.250000,27.000000,26.500000,26.250000,25.875000,25.625000 +18 7 38 8.317982 1972,4373,13806,54615,3323,4061,929,5009 19.125000,19.125000,19.125000,19.000000,18.750000,18.500000,18.375000,18.375000 +19 0 2828 0.662407 2828,9379,417,380,1684,2537,5372,44433 21.125000,20.750000,18.375000,17.625000,17.375000,17.250000,16.250000,16.125000 +19 1 12 0.115736 12,9934,59501,17,18,19,12009,12681 24.375000,20.375000,20.125000,20.000000,19.625000,19.625000,19.500000,18.500000 +19 2 17 1.392046 18,19,17,16,97857,4416,20,1552 24.500000,24.000000,23.875000,19.000000,18.500000,16.750000,16.500000,16.500000 +19 3 425 7.542877 414,9934,59501,320,1273,1614,2331,5603 20.750000,20.750000,20.000000,19.375000,18.125000,18.125000,18.000000,17.875000 +19 4 1740 0.005785 1740,3126,375,2828,3399,18454,2264,19701 25.750000,19.375000,19.250000,18.750000,18.375000,17.500000,16.250000,16.250000 +19 5 11211 4.009090 220,3950,256,45958,320,9660,262,549 22.125000,22.125000,21.875000,21.250000,21.125000,21.125000,20.375000,20.375000 +19 6 320 0.932634 320,25,549,448,198,11,510,817 26.750000,26.375000,26.125000,24.125000,24.000000,23.250000,22.625000,22.625000 +19 7 83 8.619921 20,18,19,1499,4197,2428,86847,2528 22.750000,21.500000,21.500000,21.250000,21.125000,20.875000,20.750000,20.625000 +20 0 1579 0.596680 1579,20,28,724,38007,2850,48883,12 22.625000,22.375000,18.250000,17.625000,16.750000,16.250000,16.125000,16.000000 +20 1 5839 0.043171 5839,89,30566,85593,57,11687,12738,34841 31.000000,27.875000,21.375000,21.375000,19.375000,19.125000,18.875000,18.750000 +20 2 1648 1.284651 8,1648,11170,65118,340,11,982,701 22.875000,22.375000,21.625000,20.250000,19.875000,19.500000,19.250000,18.375000 +20 3 220 0.288483 220,3968,3703,508,9848,320,715,937 26.250000,24.500000,22.875000,22.625000,22.000000,21.125000,20.875000,20.625000 +20 4 17 1.501976 16,17,18,19,20,21,22,23 29.375000,29.250000,29.000000,28.375000,28.125000,27.750000,27.625000,27.625000 +20 5 23 2.560568 17,18,15,16,19,20,23,24 26.875000,26.750000,26.625000,26.500000,26.500000,26.250000,26.125000,26.000000 +20 6 18 2.018089 18,11,15,16,17,23,20,24 26.500000,26.375000,26.250000,26.250000,26.250000,26.125000,26.000000,26.000000 +20 7 17 2.433577 11,18,17,15,16,19,20,23 28.000000,26.750000,26.625000,26.375000,26.375000,26.250000,26.250000,26.250000 +21 0 23 2.392651 18,15,17,16,19,23,24,20 26.500000,26.125000,26.125000,26.000000,25.875000,25.875000,25.875000,25.750000 +21 1 22 3.406303 11211,198,271,18,15,17,20,320 25.875000,25.750000,25.375000,24.875000,24.625000,24.500000,24.500000,24.500000 +21 2 198 1.234829 11211,198,271,320,2790,4244,3950,18 25.625000,25.375000,24.750000,24.625000,22.000000,20.750000,20.625000,20.250000 +21 3 1655 9.947481 38,29300,7595,23489,84564,10879,32516,33 20.125000,19.625000,19.000000,19.000000,19.000000,18.875000,18.750000,18.625000 +21 4 393 11.142710 1517,2780,1429,220,8028,1273,1944,74936 21.750000,19.750000,18.875000,18.375000,18.250000,18.125000,18.125000,17.875000 +21 5 2916 0.949825 2916,28,457,2045,284,12,16,17 22.375000,21.750000,21.625000,21.250000,19.250000,18.750000,18.625000,18.500000 +21 6 70508 17.162922 25,315,16460,21937,320,11,34447,5603 22.625000,22.625000,21.875000,21.875000,21.750000,21.500000,21.500000,21.375000 +21 7 15536 4.765327 4098,284,25,50,15023,28,15536,1379 22.500000,19.000000,18.250000,18.250000,18.125000,18.000000,17.875000,17.625000 +22 0 28 1.503681 284,28,320,11,25,315,3968,7354 27.625000,26.625000,25.375000,24.750000,24.500000,24.250000,23.000000,22.250000 +22 1 19 2.380747 16,17,20,19,18,23,21,220 25.625000,25.500000,24.750000,24.250000,23.375000,22.375000,21.875000,20.625000 +22 2 15 0.004538 15,20,17,23,19,16,18,11 29.750000,22.750000,22.625000,22.625000,22.000000,21.875000,21.875000,21.750000 +22 3 24 0.082619 24,15,11,23,17,21,18,19 29.250000,26.750000,22.875000,22.125000,21.625000,20.375000,20.125000,20.125000 +22 4 21 0.000669 21,20,17,22,23,15,19,18 30.250000,21.750000,21.500000,20.875000,20.875000,20.500000,20.375000,18.750000 +22 5 419 7.875584 11,25,320,11211,198,510,1447,323 28.250000,25.125000,25.000000,23.750000,23.500000,23.125000,22.750000,22.625000 +22 6 35408 2.545738 1035,374,3059,6696,35408,686,9044,18644 24.000000,23.875000,23.750000,23.750000,23.250000,23.125000,23.125000,22.375000 +22 7 220 0.099796 220,25,320,264,3968,510,911,393 28.500000,24.750000,24.750000,23.875000,23.750000,23.500000,23.375000,23.000000 +23 0 21 1.641081 22,21,17,16,18,19,23,20 28.625000,27.375000,26.625000,24.750000,24.000000,24.000000,23.500000,23.000000 +23 1 24 0.027018 24,23,22,21,15,18,20,19 28.125000,23.750000,22.625000,21.875000,21.500000,21.500000,21.375000,21.125000 +23 2 2477 9.244780 26757,13,11879,2480,320,11211,3950,40467 23.625000,20.875000,19.625000,19.250000,18.875000,18.875000,18.500000,18.500000 +23 3 28252 0.020832 28252,12,1975,39433,23642,34072,14859,49817 27.000000,22.000000,21.250000,21.250000,20.250000,20.125000,19.875000,19.750000 +23 4 90619 0.000837 90619,75,15958,278,3629,4260,654,89722 29.750000,22.375000,21.250000,15.750000,15.750000,15.750000,15.437500,15.437500 +23 5 26757 0.084089 26757,220,320,44792,11879,1467,11211,2266 25.625000,22.125000,20.375000,20.250000,20.125000,19.875000,19.875000,19.750000 +23 6 345 5.634905 320,198,271,382,11,315,13,369 28.500000,27.875000,27.625000,27.625000,27.375000,27.000000,26.875000,26.500000 +23 7 19411 15.054314 9547,4197,8206,437,8088,7138,1958,704 22.625000,22.125000,21.125000,20.500000,20.250000,19.750000,19.625000,19.500000 +24 0 2845 0.062716 2845,480,287,11307,369,448,304,82 27.250000,24.500000,19.500000,17.750000,17.125000,17.000000,16.250000,16.000000 +24 1 47905 7.074774 2878,1212,26345,3685,2686,4946,366,18304 23.500000,23.375000,22.875000,22.375000,21.125000,20.500000,20.125000,19.750000 +24 2 279 0.101637 279,220,479,8028,1039,2567,393,894 26.375000,22.750000,22.000000,21.250000,21.125000,21.125000,21.000000,21.000000 +24 3 220 1.064806 220,8028,7192,2567,1932,17377,1372,3930 23.000000,22.000000,21.625000,21.375000,20.875000,20.750000,20.375000,20.250000 +24 4 17 2.159409 21,16,18,17,20,19,23,22 27.500000,26.500000,26.500000,26.250000,26.125000,25.750000,23.625000,22.250000 +24 5 15 0.230696 15,20,61,19,17,23,21,16 27.875000,26.125000,23.875000,23.625000,23.250000,23.250000,23.000000,22.750000 +24 6 11582 3.488510 26757,2567,11879,15,11582,12,4362,8028 22.750000,22.500000,20.500000,20.250000,20.250000,19.875000,19.625000,19.375000 +24 7 3122 0.006176 3122,15296,573,359,1942,15832,3077,31883 31.750000,26.625000,23.250000,20.750000,20.250000,19.750000,18.625000,18.625000 +25 0 16287 3.072573 8028,3930,12171,16287,4722,1308,2567,44279 23.125000,22.125000,21.000000,20.625000,19.250000,18.875000,18.750000,18.750000 +25 1 315 1.927896 369,382,315,320,624,271,13,311 28.375000,28.125000,27.375000,26.000000,25.625000,25.500000,25.250000,25.125000 +25 2 279 0.352917 279,419,479,1565,1273,264,330,342 25.000000,23.500000,21.125000,21.000000,20.375000,20.125000,20.000000,20.000000 +25 3 425 5.274380 1273,16460,4024,28431,479,9934,5567,5603 21.250000,20.625000,19.875000,19.875000,19.375000,19.250000,19.125000,19.000000 +25 4 18 8.606319 3399,1740,18454,1593,96190,2916,4872,12457 19.375000,19.250000,18.875000,18.125000,17.250000,16.875000,16.875000,16.250000 +25 5 4271 11.648369 1273,16460,12,18266,10519,25869,323,5456 20.875000,20.250000,19.000000,19.000000,18.250000,18.250000,17.875000,17.750000 +25 6 18126 5.132542 18266,1273,1779,25869,12890,28431,35957,198 23.500000,22.500000,22.000000,20.750000,20.625000,20.500000,20.500000,20.375000 +25 7 382 0.170805 382,624,320,271,13,198,369,345 28.625000,25.750000,25.500000,24.875000,24.625000,24.500000,24.250000,22.625000 +26 0 1986 3.217823 785,418,28,1986,9112,2461,44364,4416 21.250000,19.750000,19.250000,19.125000,18.625000,18.375000,17.875000,17.750000 +26 1 1034 4.209267 374,1273,10337,1034,821,15983,2150,5316 24.625000,24.500000,22.375000,21.375000,21.125000,21.125000,20.875000,20.875000 +26 2 374 0.407141 374,5610,572,320,5646,646,594,5707 28.000000,26.625000,25.000000,24.625000,24.625000,23.750000,23.625000,23.625000 +26 3 1483 2.112885 949,369,1483,264,7907,10602,279,8791 23.375000,23.000000,23.000000,22.875000,22.750000,22.375000,22.250000,21.125000 +26 4 438 2.532818 311,369,304,438,553,448,2337,1172 27.625000,27.375000,27.000000,26.125000,26.125000,23.750000,22.750000,21.875000 +26 5 264 1.100914 264,279,1946,1273,949,458,5785,4862 26.625000,26.500000,25.750000,25.375000,24.750000,24.000000,23.875000,22.125000 +26 6 12507 10.571642 1273,5785,5297,25869,28431,425,2530,4185 23.375000,22.750000,21.375000,20.250000,20.250000,19.875000,19.875000,19.625000 +26 7 369 0.929350 304,369,553,311,320,2337,979,448 28.125000,27.875000,25.500000,25.250000,24.375000,24.125000,22.500000,22.000000 +27 0 279 1.060913 7497,279,425,37563,7032,1273,4303,17590 24.000000,23.875000,21.750000,21.250000,20.625000,20.250000,20.250000,20.250000 +27 1 47175 15.302465 425,1273,1565,330,479,364,7032,4271 23.250000,21.625000,21.125000,20.625000,20.625000,20.375000,20.000000,19.750000 +27 2 44220 8.506355 33,61437,2271,34,12,8963,9285,9967 18.375000,18.250000,18.000000,17.875000,17.625000,17.250000,17.125000,17.125000 +27 3 84648 12.967186 1273,372,425,6733,1614,3135,7032,2022 22.875000,20.625000,20.500000,20.375000,20.125000,20.125000,20.125000,20.000000 +27 4 6500 0.708683 6500,19479,8233,1273,36680,47430,13850,4938 23.375000,22.750000,21.000000,20.500000,19.875000,19.875000,19.375000,19.000000 +27 5 73152 11.031029 1273,4271,28431,5068,7497,7032,10272,62019 21.875000,21.750000,21.625000,20.625000,20.250000,20.000000,20.000000,19.875000 +27 6 487 0.000019 487,34494,1361,16460,3147,367,275,22814 29.250000,17.000000,16.625000,16.000000,15.250000,15.062500,14.937500,14.750000 +27 7 18126 6.213497 1273,28431,198,7032,16460,7497,62019,4271 25.500000,23.500000,22.625000,22.625000,22.125000,21.375000,21.375000,21.250000 +28 0 198 2.704977 1273,382,7032,624,198,320,13,345 25.625000,25.125000,24.500000,24.250000,24.125000,24.125000,23.375000,23.375000 +28 1 8623 5.908091 1944,258,437,7,23841,2428,300,5349 21.250000,20.875000,20.875000,20.750000,20.375000,20.250000,20.125000,20.000000 +28 2 82 0.671923 82,287,291,620,880,25,5523,53346 23.375000,22.250000,22.000000,21.000000,20.000000,19.500000,18.875000,18.750000 +28 3 5523 1.545790 12697,5523,14,4846,43985,78311,3470,4322 20.125000,19.875000,19.375000,18.875000,18.125000,18.125000,18.000000,17.375000 +28 4 324 0.014227 324,9048,80,65,85964,269,9713,1677 31.625000,27.375000,21.000000,19.875000,19.250000,19.125000,19.000000,18.875000 +28 5 749 0.000004 749,6331,17184,5970,2011,126394,15439,21028 40.250000,27.625000,24.750000,24.250000,23.250000,22.750000,22.625000,21.750000 +28 6 97066 14.513279 30448,53346,80,1751,8976,14,4452,44220 28.125000,23.750000,19.500000,18.500000,18.250000,18.125000,17.750000,17.750000 +28 7 2204 8.749159 11529,620,5678,9360,53346,76776,14,43985 22.750000,21.875000,20.875000,19.625000,18.750000,18.250000,17.875000,17.500000 +29 0 17 1.546122 19642,17,1409,20615,3334,27480,23819,12536 24.000000,22.875000,21.375000,20.500000,19.875000,19.750000,19.000000,18.875000 +29 1 68 0.000195 68,57824,291,295,288,5120,16,823 32.250000,23.000000,21.125000,21.000000,20.375000,20.375000,20.000000,19.875000 +29 2 25638 6.651157 7197,4452,12697,14,880,43985,4846,54798 21.125000,20.500000,19.625000,18.625000,18.625000,18.375000,18.250000,17.875000 +29 3 4292 3.957254 8,568,701,3593,692,11,323,4292 26.125000,26.000000,25.750000,25.750000,25.375000,24.125000,23.750000,23.625000 +29 4 8610 7.209177 785,2132,2461,418,1654,2679,4416,1986 22.375000,20.875000,19.750000,19.250000,19.000000,19.000000,19.000000,18.750000 +29 5 31805 11.787052 4344,2297,50844,8837,28782,17030,2647,1273 25.250000,23.250000,23.125000,22.375000,22.375000,21.750000,20.750000,20.375000 +29 6 990 4.941956 4278,975,315,476,3542,2038,1969,990 24.375000,24.000000,23.750000,22.750000,21.500000,21.375000,21.000000,20.500000 +29 7 1969 0.614321 1969,315,320,7460,1265,374,476,1588 27.750000,27.250000,25.000000,24.625000,24.500000,24.375000,24.250000,22.250000 +30 0 14060 4.542385 25017,48453,2924,387,5978,1795,21129,25879 24.000000,22.250000,22.000000,21.750000,21.250000,21.000000,21.000000,21.000000 +30 1 419 1.572054 279,419,62024,1493,5723,678,4024,6920 26.250000,25.125000,24.250000,22.250000,22.250000,22.000000,21.500000,21.000000 +30 2 13534 7.214588 62024,5723,5293,6920,3980,4247,5185,44377 23.125000,22.125000,21.625000,20.750000,20.250000,20.250000,19.750000,18.875000 +30 3 7710 1.106220 7671,7710,15,5723,62024,12,14319,92214 24.375000,23.875000,21.875000,21.750000,20.875000,19.750000,19.500000,19.250000 +30 4 92214 0.000761 92214,16589,12,81219,220,6222,62024,5723 27.500000,19.375000,18.625000,18.000000,17.750000,17.625000,17.375000,17.250000 +30 5 220 0.383530 220,5723,62024,5293,12,348,323,6920 25.250000,24.125000,23.000000,20.875000,19.500000,19.125000,18.875000,18.250000 +30 6 18 0.000099 18,19,17,16,15,220,20,21 33.000000,23.375000,22.625000,18.250000,17.875000,16.875000,16.375000,15.000000 +30 7 13 -0.000000 13,12,624,36479,382,10845,15,1876 35.750000,17.250000,17.250000,17.000000,16.500000,16.125000,15.625000,15.437500 +31 0 15 -0.000000 15,18,24,20,16,21,17,99822 35.250000,19.500000,17.625000,15.937500,15.625000,15.312500,14.750000,14.625000 +31 1 62024 1.373686 5723,62024,5293,323,6668,6920,1876,27406 25.250000,24.250000,22.250000,19.500000,19.250000,19.125000,18.875000,18.875000 +31 2 382 0.774311 382,323,624,5293,13,198,11,345 26.750000,26.375000,24.625000,24.125000,23.750000,23.750000,23.500000,23.000000 +31 3 284 7.251692 28,418,785,44364,313,2461,4421,8707 20.500000,20.500000,20.375000,18.250000,18.125000,17.500000,17.375000,17.375000 +31 4 8397 12.745161 3972,608,11176,284,835,1406,220,4710 22.250000,19.250000,19.250000,19.125000,18.875000,18.875000,18.750000,18.000000 +31 5 425 4.568361 64,356,451,468,386,328,422,594 20.375000,17.875000,17.625000,17.500000,17.375000,17.250000,17.250000,16.875000 +31 6 10965 13.632807 13,22908,366,398,535,673,356,1826 23.625000,16.250000,16.125000,15.937500,15.937500,15.875000,15.687500,15.687500 +31 7 465 2.387480 4003,777,465,1664,1280,665,466,484 21.250000,19.375000,19.125000,16.500000,15.812500,15.312500,14.937500,14.687500 +32 0 284 2.882164 594,320,79483,11,284,271,25,481 23.250000,21.625000,21.250000,21.000000,21.000000,20.625000,20.375000,19.000000 +32 1 715 1.984297 4710,1406,715,320,2303,220,18611,374 22.625000,21.875000,21.375000,20.125000,19.750000,18.625000,18.625000,18.375000 +32 2 8397 4.085903 785,34532,40,32313,284,8397,1986,58 18.375000,18.250000,17.000000,17.000000,16.625000,16.250000,16.125000,15.937500 +32 3 425 0.021715 425,7801,3757,8397,12375,468,220,21891 22.500000,15.812500,15.187500,15.187500,15.187500,14.875000,14.750000,14.625000 +32 4 10965 0.000204 10965,13,616,2950,965,30158,220,23937 25.625000,16.375000,15.312500,14.500000,13.562500,13.500000,13.187500,12.875000 +32 5 465 0.000019 465,5045,777,665,376,1982,4160,4991 30.000000,18.125000,17.125000,16.625000,16.500000,16.125000,15.750000,15.500000 +32 6 374 1.746269 320,374,572,11,220,594,198,702 24.625000,23.125000,21.000000,19.625000,19.125000,17.875000,17.750000,17.500000 +32 7 458 1.567019 264,458,279,825,5023,3881,1850,1431 26.500000,25.250000,23.500000,22.250000,21.125000,20.625000,19.750000,19.375000 +33 0 6364 0.634359 6364,3693,13369,9489,12089,3150,17847,21946 21.625000,21.000000,19.500000,17.250000,17.000000,16.750000,16.750000,16.500000 +33 1 4531 4.786921 6584,12089,4741,8964,1515,6916,39160,3150 18.375000,18.125000,18.000000,17.125000,16.375000,16.250000,16.250000,15.625000 +33 2 1154 9.175163 7538,323,16851,6440,11,12089,4171,39450 22.750000,22.500000,21.375000,20.500000,18.750000,18.125000,18.000000,17.750000 +33 3 12425 0.069733 12425,6430,5883,2766,33496,11086,323,26705 26.750000,23.875000,21.500000,20.750000,19.625000,19.500000,19.375000,19.125000 +33 4 323 0.039146 323,1154,11,16851,220,609,7538,320 29.750000,26.500000,21.375000,21.375000,20.250000,20.000000,20.000000,19.625000 +33 5 33496 2.872169 6430,33496,7743,8887,8353,26705,2766,65233 25.125000,22.375000,21.375000,20.625000,20.125000,20.000000,19.625000,19.125000 +33 6 12089 0.958448 7538,12089,16851,23654,220,4171,6916,48154 24.375000,24.125000,22.625000,19.750000,19.250000,19.125000,19.000000,18.875000 +33 7 659 1.380764 659,879,323,1850,3881,1154,13,448 27.875000,27.875000,27.500000,27.125000,27.000000,25.875000,25.250000,24.625000 +34 0 1260 0.180466 1260,425,5301,4636,37816,758,715,362 25.500000,23.625000,20.750000,20.625000,20.625000,19.375000,19.000000,18.625000 +34 1 1030 6.402983 572,374,702,9723,6342,19476,16176,18178 25.000000,24.375000,24.375000,22.500000,21.625000,21.625000,21.500000,21.375000 +34 2 264 0.562696 264,806,12783,8597,458,279,1012,8922 25.250000,23.750000,22.750000,22.125000,21.875000,21.750000,21.750000,21.750000 +34 3 8640 5.316959 3476,2613,8922,12613,45250,3598,6388,9814 22.875000,22.250000,22.250000,21.500000,21.500000,21.000000,21.000000,20.750000 +34 4 569 14.850103 3476,11094,481,39400,7702,949,6774,26255 24.750000,23.250000,21.000000,20.875000,20.500000,20.375000,20.375000,18.625000 +34 5 12 0.444805 12,481,220,304,198,3476,1376,19191 20.000000,19.125000,15.937500,15.375000,15.187500,14.562500,14.437500,14.437500 +34 6 31 0.010168 31,16,5778,62799,17,3476,569,3157 25.625000,18.875000,18.625000,18.375000,17.875000,17.875000,17.500000,17.125000 +34 7 39400 2.449594 3476,39400,11094,6774,220,7702,949,2484 26.375000,24.125000,23.375000,22.375000,21.250000,20.875000,20.500000,20.375000 +35 0 3476 0.018923 3476,304,11094,949,12783,2309,6502,389 28.125000,23.625000,21.875000,21.500000,21.125000,20.125000,20.125000,20.000000 +35 1 389 4.579731 304,438,389,315,448,518,5619,323 29.250000,26.500000,24.750000,22.625000,22.375000,22.375000,22.125000,21.500000 +35 2 279 0.143845 279,576,18096,264,330,18635,6326,364 24.000000,20.500000,19.750000,19.375000,19.375000,18.875000,18.750000,18.625000 +35 3 12425 2.040039 18096,12425,7855,220,1156,2086,4013,13434 22.125000,21.250000,20.625000,20.375000,20.250000,19.750000,19.750000,19.625000 +35 4 4013 0.062426 4013,1473,2025,18136,19584,22358,80366,6146 26.250000,22.750000,21.875000,21.250000,20.375000,20.125000,20.125000,19.750000 +35 5 576 1.945026 330,576,3355,1154,364,18635,4678,1591 19.375000,19.125000,18.500000,18.375000,17.875000,17.375000,16.875000,16.750000 +35 6 8596 3.844187 328,8246,19378,4312,1599,1532,8596,220 16.875000,16.875000,16.625000,15.625000,15.187500,15.062500,14.875000,14.625000 +35 7 304 1.562088 323,304,1154,320,659,438,504,220 26.500000,25.750000,25.000000,24.750000,24.750000,24.125000,23.125000,21.750000 +36 0 220 0.312962 220,279,264,892,458,1181,806,5813 26.625000,25.500000,22.250000,22.250000,20.250000,20.000000,19.875000,19.875000 +36 1 17 0.523573 17,16,18,15,1154,220,19,22 22.500000,22.125000,13.250000,13.000000,12.625000,12.187500,11.875000,11.812500 +36 2 15 0.000011 15,16,220,17,18,569,23,1154 28.500000,15.562500,15.187500,14.437500,13.937500,13.875000,13.750000,13.750000 +36 3 15 0.101780 15,16,17,220,1154,569,18,549 25.250000,23.000000,18.875000,12.500000,12.312500,12.250000,11.750000,11.312500 +36 4 15 1.831144 15,16,17,18,19,20,22,21 21.250000,21.125000,21.125000,21.125000,20.750000,20.750000,20.625000,20.375000 +36 5 659 0.855446 323,659,1154,438,13,320,714,2587 27.625000,27.625000,26.125000,24.500000,23.750000,23.125000,22.625000,22.375000 +36 6 1096 7.665333 1260,425,758,5301,1913,5542,24885,8704 25.875000,24.125000,23.500000,22.000000,20.375000,20.375000,20.375000,20.000000 +36 7 572 0.333273 572,374,3476,4013,4549,6069,9234,3668 24.750000,23.000000,22.125000,20.500000,20.125000,19.625000,19.625000,19.250000 +37 0 8110 1.436741 806,8110,264,279,304,438,825,458 24.875000,24.500000,24.000000,23.500000,22.875000,21.750000,21.500000,21.375000 +37 1 553 0.029675 553,304,448,705,264,19620,2937,1378 29.375000,25.500000,23.250000,23.125000,22.375000,22.375000,22.000000,21.875000 +37 2 264 0.393456 264,806,26255,279,458,12783,2441,8640 25.250000,23.750000,22.250000,21.750000,21.750000,21.625000,21.000000,20.375000 +37 3 39400 5.183321 3476,8640,949,45250,12613,4013,2613,8922 23.000000,21.000000,20.625000,20.625000,20.375000,20.250000,20.125000,20.000000 +37 4 3476 0.003990 3476,304,949,569,2309,323,5068,6502 27.750000,20.625000,20.375000,20.125000,19.250000,19.125000,19.125000,19.125000 +37 5 304 0.129242 304,438,389,448,315,518,323,5619 29.375000,27.000000,25.875000,24.125000,23.000000,22.875000,22.500000,22.250000 +37 6 279 0.118267 279,264,576,220,458,18096,364,806 25.125000,22.125000,20.750000,19.875000,19.875000,19.000000,18.625000,18.625000 +37 7 1486 3.718361 220,12425,4531,18096,2805,1486,7855,4565 21.750000,21.250000,20.750000,20.125000,19.750000,19.250000,19.125000,18.875000 +38 0 6252 7.011986 576,364,1154,330,3355,362,518,659 20.000000,19.125000,18.625000,18.375000,17.750000,17.250000,16.875000,16.625000 +38 1 2382 1.732311 2382,263,585,288,5298,1154,18209,66244 17.625000,17.000000,16.750000,16.375000,16.250000,16.000000,16.000000,15.687500 +38 2 5326 7.861170 304,1154,518,323,553,389,320,659 22.125000,21.500000,21.000000,20.875000,20.625000,20.500000,20.250000,20.250000 +38 3 553 0.008677 553,323,304,369,448,518,569,3216 29.750000,24.750000,22.500000,22.000000,20.500000,20.500000,20.500000,19.875000 +38 4 21315 3.764839 6798,3757,11867,4389,14991,21315,33121,18095 18.250000,17.625000,17.500000,17.375000,17.375000,17.375000,17.375000,17.250000 +38 5 75923 2.961792 23366,1644,12131,8362,75923,39103,386,472 18.000000,17.500000,17.500000,16.625000,16.625000,16.375000,15.937500,15.687500 +38 6 1154 2.658560 659,323,304,1154,518,892,369,320 27.625000,27.500000,27.250000,26.125000,25.875000,24.750000,24.625000,24.000000 +38 7 892 0.375389 892,323,304,1156,10660,15540,518,8947 25.875000,24.125000,22.500000,22.500000,22.500000,22.375000,22.250000,22.000000 +39 0 572 0.705832 572,8930,10613,84070,566,90613,6342,22409 25.375000,24.125000,23.875000,23.375000,23.125000,22.750000,22.000000,21.875000 +39 1 10660 0.812071 10660,8947,50057,1156,10449,518,6839,15540 23.625000,22.500000,22.250000,21.625000,21.250000,21.000000,20.875000,20.750000 +39 2 304 2.262070 518,304,553,389,1156,369,504,569 29.875000,27.750000,25.750000,24.250000,23.750000,22.875000,22.875000,22.500000 +39 3 220 0.727556 220,279,7148,264,2176,19361,5257,576 23.375000,23.000000,21.375000,20.875000,19.625000,19.125000,18.625000,18.250000 +39 4 17 0.003731 17,16,18,19,15,20,22,21 26.500000,20.875000,16.125000,15.375000,15.125000,14.937500,14.625000,14.125000 +39 5 15 0.000004 15,16,17,18,23,24,19,1154 30.250000,16.875000,15.375000,14.437500,14.437500,14.250000,14.187500,14.062500 +39 6 15 0.008640 15,16,17,220,18,1154,19,549 28.625000,23.875000,17.875000,13.812500,13.750000,13.500000,13.312500,13.312500 +39 7 16 1.003468 16,17,18,15,19,20,21,22 22.250000,21.750000,21.500000,20.625000,20.500000,20.125000,19.500000,19.375000 +40 0 518 0.405586 518,659,323,304,553,389,1154,13 28.375000,26.750000,26.375000,26.250000,24.500000,22.875000,22.750000,22.500000 +40 1 279 0.022123 279,576,264,7148,2176,26221,19361,24688 25.625000,21.375000,19.000000,19.000000,18.000000,17.625000,17.125000,16.750000 +40 2 16194 0.918459 16194,15953,5055,1163,4312,14079,19361,26221 20.625000,19.875000,19.250000,18.000000,17.625000,17.625000,17.375000,17.375000 +40 3 7154 0.049406 7154,5055,18705,41382,323,15815,15953,65463 24.125000,20.375000,20.000000,18.250000,17.750000,17.000000,17.000000,16.375000 +40 4 26221 0.062290 26221,304,33496,315,569,659,1154,323 25.750000,21.625000,21.125000,21.000000,20.750000,20.750000,20.375000,20.125000 +40 5 659 1.773592 304,659,323,1154,3138,320,16448,481 28.000000,26.500000,24.750000,24.625000,22.375000,22.250000,21.625000,21.000000 +40 6 1260 0.875631 1260,758,425,5301,24885,576,4636,5542 24.625000,24.125000,23.875000,22.375000,21.375000,21.000000,20.750000,20.500000 +40 7 1030 3.054010 1083,1221,572,702,9723,6342,1030,2937 23.500000,23.125000,22.875000,22.875000,22.125000,22.000000,21.875000,21.875000 +41 0 264 0.078417 264,2441,806,458,1083,279,12783,4623 27.250000,23.375000,22.875000,22.750000,21.875000,21.750000,21.625000,21.125000 +41 1 8640 2.266478 3476,12613,8640,8922,2613,45250,6388,9814 21.875000,21.875000,21.125000,21.000000,20.750000,20.625000,20.375000,19.500000 +41 2 3476 0.528576 3476,569,39400,11094,7702,6774,949,25377 23.500000,22.125000,22.125000,21.500000,19.000000,18.875000,17.625000,17.375000 +41 3 304 0.152626 304,389,438,1549,448,518,5619,315 30.375000,28.375000,26.375000,24.375000,24.125000,23.875000,23.875000,23.250000 +41 4 279 0.069275 279,576,264,220,2441,458,18096,7506 25.500000,21.125000,21.000000,20.500000,20.500000,19.875000,19.500000,18.750000 +41 5 12425 0.714283 12425,18096,220,4013,7855,4531,5883,2086 23.000000,21.625000,21.500000,20.875000,19.875000,19.750000,19.625000,19.500000 +41 6 4013 0.015282 4013,19584,1473,2025,18136,6146,4531,22358 27.375000,22.125000,20.875000,20.750000,20.750000,20.625000,20.375000,20.375000 +41 7 20207 5.308310 576,7506,15696,18635,6326,4678,9414,48822 17.875000,16.250000,16.250000,16.125000,16.000000,15.812500,15.812500,15.687500 +42 0 3757 0.000729 3757,1154,659,422,54904,54058,81204,481 28.375000,20.375000,19.000000,18.750000,18.250000,18.000000,17.875000,17.625000 +42 1 1581 0.015110 1581,54904,1154,31302,82884,55156,5227,444 22.875000,17.750000,16.375000,15.937500,15.875000,15.812500,15.625000,15.500000 +42 2 291 0.001705 291,2636,2145,585,19289,1090,68,546 27.000000,20.125000,17.750000,17.500000,17.500000,17.375000,17.000000,17.000000 +42 3 304 0.307930 304,320,323,1154,659,504,892,438 28.750000,26.875000,26.375000,25.875000,24.750000,24.000000,23.875000,23.625000 +42 4 220 0.010113 220,279,264,892,5813,3217,5470,6058 28.875000,24.000000,21.000000,20.875000,20.125000,19.875000,19.750000,19.750000 +42 5 17 0.000265 17,16,18,15,20,19,22,220 28.875000,20.625000,15.812500,14.375000,13.500000,13.437500,12.687500,12.687500 +42 6 15 0.000002 15,16,220,17,23,18,1154,659 30.000000,15.687500,14.500000,14.312500,13.625000,13.437500,13.125000,13.000000 +42 7 15 0.003189 15,16,17,1154,220,549,659,481 28.375000,22.625000,16.500000,14.687500,14.000000,13.125000,13.125000,12.562500 +43 0 17 1.085657 17,18,19,16,20,21,22,23 22.500000,22.500000,21.500000,21.125000,20.750000,20.000000,19.500000,18.375000 +43 1 659 1.062925 323,659,1154,438,320,13,304,2587 29.500000,29.000000,27.375000,24.250000,23.875000,23.750000,23.625000,23.375000 +43 2 758 1.084049 1260,758,425,5301,24885,5542,5005,27133 25.250000,25.000000,24.125000,22.500000,21.500000,21.000000,20.750000,20.375000 +43 3 220 0.036356 220,279,5256,5813,3217,5470,6702,7400 28.625000,24.875000,23.000000,21.625000,21.500000,21.375000,21.375000,21.375000 +43 4 17 0.000437 17,16,15,1154,18,220,549,19 26.750000,19.000000,13.437500,13.000000,12.562500,11.875000,10.562500,10.375000 +43 5 15 -0.000000 15,16,24,17,23,18,19,220 31.625000,15.437500,14.187500,13.937500,13.625000,13.187500,12.375000,12.312500 +43 6 15 0.011084 15,16,17,1154,220,18,549,19 27.875000,23.375000,17.500000,13.875000,13.375000,13.187500,12.187500,12.000000 +43 7 19 1.610905 18,19,20,21,17,22,23,24 24.750000,23.625000,22.875000,21.875000,21.375000,21.250000,20.375000,19.750000 +44 0 425 3.314785 566,1154,425,323,806,1260,569,8397 29.625000,29.250000,26.875000,23.500000,23.250000,22.000000,21.750000,21.500000 +44 1 10965 0.000156 10965,965,581,616,6344,1080,220,425 25.625000,15.687500,14.312500,14.187500,13.687500,13.312500,13.062500,13.000000 +44 2 465 0.000019 465,5045,376,665,59385,423,4160,1982 31.125000,19.875000,17.750000,17.250000,16.500000,16.375000,16.375000,16.250000 +44 3 26120 7.430857 6342,9723,572,10859,1030,3867,1865,58235 24.500000,24.375000,23.375000,22.375000,22.250000,22.000000,21.875000,21.375000 +44 4 264 0.267038 264,279,806,458,304,2441,825,12783 28.000000,26.500000,25.375000,22.625000,21.500000,20.750000,20.000000,19.750000 +44 5 3476 0.970726 3476,6388,12613,45250,5792,949,3598,39400 22.875000,21.750000,21.500000,21.375000,21.250000,21.125000,20.625000,20.000000 +44 6 438 1.823097 304,438,389,448,5619,315,518,569 29.500000,27.875000,24.250000,24.250000,24.125000,23.500000,23.125000,21.750000 +44 7 330 8.402699 264,279,458,4389,6798,825,8364,11867 21.500000,19.625000,19.000000,17.750000,17.625000,17.500000,17.375000,17.375000 +45 0 28417 7.448458 4392,14991,21315,6798,3757,8364,32819,4389 16.250000,16.000000,15.937500,15.875000,15.812500,15.812500,15.750000,15.687500 +45 1 330 0.136829 330,3670,422,425,386,472,7401,1 21.625000,16.750000,16.000000,16.000000,15.875000,15.812500,15.812500,15.687500 +45 2 304 0.020948 304,389,1154,320,323,279,264,448 29.000000,24.000000,24.000000,22.625000,22.375000,22.000000,21.875000,21.125000 +45 3 279 0.028978 279,264,576,458,13434,18096,330,4013 26.875000,22.250000,21.625000,20.500000,20.250000,20.125000,19.375000,19.250000 +45 4 9234 8.027498 12425,4531,18096,4013,7855,13434,22358,4565 22.125000,21.125000,20.875000,20.250000,20.250000,20.125000,19.750000,19.125000 +45 5 330 0.773592 330,315,24849,1591,3355,364,19867,1154 24.375000,24.125000,22.125000,21.500000,21.250000,21.000000,20.750000,19.875000 +45 6 70066 10.947123 576,362,425,358,350,356,220,422 19.500000,16.875000,16.750000,16.250000,16.125000,16.125000,16.000000,16.000000 +45 7 364 4.697786 330,23985,364,323,16180,35968,304,8171 23.500000,19.625000,18.875000,18.375000,17.750000,17.000000,16.750000,16.750000 +46 0 82 0.002056 82,274,364,569,50,1154,67,2587 27.000000,20.625000,16.875000,16.250000,16.125000,16.000000,15.875000,15.625000 +46 1 15106 4.608822 5004,6841,8599,6059,6164,1532,7506,7996 15.562500,15.500000,15.437500,15.375000,15.062500,14.937500,14.812500,14.812500 +46 2 330 0.003899 330,320,364,3670,1,549,1154,5933 28.250000,20.375000,20.250000,20.125000,20.000000,19.625000,19.375000,19.250000 +46 3 315 0.098465 315,304,389,504,320,1154,323,892 30.125000,27.125000,26.500000,25.250000,25.000000,24.750000,24.250000,23.750000 +46 4 279 0.024517 279,576,18096,13434,4013,264,18635,7506 26.000000,21.625000,19.500000,19.125000,18.625000,18.250000,18.125000,18.000000 +46 5 12425 0.748310 12425,4013,18096,7855,13434,22358,5883,19584 23.000000,22.125000,21.750000,20.000000,19.875000,19.625000,19.375000,19.250000 +46 6 4013 0.007528 4013,19584,2025,1473,6146,22358,18136,80366 27.750000,21.375000,21.125000,21.000000,20.625000,20.500000,20.375000,19.125000 +46 7 576 1.754431 576,1154,18635,48822,15696,52325,6326,14671 18.250000,16.875000,16.625000,16.250000,16.125000,16.125000,16.000000,16.000000 +47 0 5724 8.366972 328,8246,19378,9494,1532,220,6686,8596 17.500000,16.625000,16.500000,15.875000,15.750000,15.625000,15.437500,15.375000 +47 1 37625 3.607450 477,9536,4050,7684,18971,2738,12149,37625 19.625000,18.625000,18.500000,18.125000,18.000000,17.500000,17.250000,17.125000 +47 2 2587 5.449572 659,1154,320,323,389,892,304,2587 27.375000,25.625000,25.000000,24.125000,23.375000,23.000000,22.750000,22.250000 +47 3 566 1.169445 566,419,279,323,304,4764,425,806 24.250000,24.125000,23.750000,22.500000,22.250000,21.375000,21.250000,21.250000 +47 4 58235 6.761471 572,1083,6342,5927,9723,2937,1030,1221 23.875000,23.625000,23.250000,22.875000,22.125000,21.875000,21.500000,21.000000 +47 5 16263 4.623627 304,438,1549,448,16263,1083,369,518 27.500000,26.000000,23.500000,23.125000,23.125000,22.000000,21.000000,21.000000 +47 6 19571 4.119192 6798,279,3757,4389,11867,8397,18939,24986 18.000000,17.875000,17.875000,17.875000,17.875000,17.750000,17.500000,17.500000 +47 7 4389 4.299381 1741,1075,2670,11867,18939,6798,4389,3757 22.750000,21.250000,20.500000,19.500000,19.500000,19.375000,19.250000,19.125000 +48 0 26099 4.778086 431,13612,2691,20607,350,72113,1260,26099 19.375000,17.000000,16.375000,15.562500,15.187500,14.937500,14.875000,14.875000 +48 1 323 0.065849 323,1154,320,659,304,438,2587,609 30.875000,28.125000,25.000000,23.125000,22.625000,21.625000,21.250000,21.000000 +48 2 42323 6.572186 6798,3757,11867,4389,18939,7801,11565,18095 18.250000,17.750000,17.625000,17.500000,17.500000,17.375000,17.375000,17.375000 +48 3 15035 3.184395 25164,15035,63540,431,434,98971,4264,51656 22.750000,19.625000,17.500000,17.125000,15.562500,15.125000,15.062500,15.062500 +48 4 18337 0.003052 18337,404,72,23300,18122,46291,354,1119 23.750000,17.750000,14.750000,14.437500,13.687500,13.375000,13.000000,12.812500 +48 5 659 0.089527 659,304,323,1154,16448,6762,320,2587 30.250000,27.250000,26.250000,25.125000,25.000000,24.500000,24.375000,24.000000 +48 6 1260 1.723835 758,1260,425,5301,24885,576,2938,5542 25.250000,24.000000,23.625000,21.375000,21.375000,21.250000,20.250000,20.125000 +48 7 572 1.676542 1083,572,6342,1030,702,9723,1221,2937 23.000000,22.875000,22.375000,22.250000,22.000000,22.000000,21.750000,21.250000 +49 0 6311 2.209560 1083,264,6311,304,1221,2937,14766,27420 23.750000,22.375000,22.375000,21.750000,21.375000,21.375000,20.750000,20.375000 +49 1 304 0.659798 304,438,369,311,1549,553,389,448 26.375000,26.250000,22.875000,21.625000,20.250000,19.625000,19.250000,19.125000 +49 2 279 0.159676 279,264,220,2441,458,576,1378,806 26.750000,24.625000,22.625000,22.125000,21.500000,21.250000,20.375000,19.625000 +49 3 220 1.306345 220,3476,12425,4531,1486,4013,18096,2805 22.625000,22.500000,22.375000,21.250000,20.500000,19.500000,19.500000,19.125000 +49 4 17 0.003527 17,16,18,19,20,22,21,23 28.125000,22.125000,19.875000,19.750000,19.500000,19.000000,18.875000,18.125000 +49 5 15 0.000044 15,16,17,18,19,23,20,24 28.375000,17.500000,16.000000,15.500000,15.375000,15.375000,15.125000,14.687500 +49 6 15 0.020632 15,16,17,19,339,20,18,4146 27.750000,23.875000,18.250000,14.750000,14.250000,14.187500,14.000000,13.312500 +49 7 20 0.839939 20,21,22,19,23,24,18,17 23.375000,22.875000,22.125000,22.000000,21.000000,20.250000,19.375000,18.375000 +50 0 33496 8.196558 4531,12425,4013,7855,4565,18096,5700,22358 21.375000,21.250000,19.500000,18.625000,18.500000,18.500000,18.250000,18.125000 +50 1 53148 6.371386 5670,1486,5068,33085,56005,481,1473,6573 25.625000,24.125000,21.250000,20.625000,20.625000,19.750000,19.625000,19.625000 +50 2 315 0.076632 315,576,518,1154,330,549,304,323 25.000000,20.375000,20.250000,20.000000,19.125000,19.000000,18.875000,18.875000 +50 3 279 1.840660 576,279,330,7401,362,364,422,9043 20.750000,20.125000,18.250000,17.625000,17.375000,17.125000,16.875000,16.875000 +50 4 25179 10.188784 1486,11088,22358,17795,6430,825,1852,19584 23.250000,21.625000,19.125000,19.000000,18.875000,18.500000,18.250000,17.750000 +50 5 92301 3.626781 31828,730,20734,444,92301,30563,20113,6210 19.500000,17.500000,17.000000,16.750000,16.375000,16.125000,16.000000,15.875000 +50 6 1486 0.320869 1486,364,11088,481,19584,1154,323,7054 24.250000,22.750000,21.500000,20.125000,19.875000,19.750000,19.250000,18.875000 +50 7 43562 8.485886 576,330,364,1154,25044,425,659,362 21.625000,20.125000,18.750000,18.750000,17.750000,17.000000,17.000000,16.625000 +51 0 49840 0.469931 49840,323,1154,518,304,2587,553,320 24.375000,23.125000,22.750000,20.875000,20.375000,20.000000,19.625000,19.250000 +51 1 1154 1.814772 323,1154,320,518,659,304,2587,892 27.375000,26.125000,25.375000,25.125000,25.125000,24.125000,23.500000,22.750000 +51 2 892 1.963104 323,892,15540,10660,518,5619,304,448 25.125000,23.875000,23.750000,22.875000,22.250000,22.250000,22.125000,21.625000 +51 3 572 0.242048 572,10613,90613,3867,1033,566,8930,1030 25.500000,22.875000,22.125000,21.875000,21.750000,21.625000,21.375000,21.250000 +51 4 10660 0.442640 10660,15540,50057,8947,5326,10449,264,518 24.000000,22.125000,21.500000,21.125000,20.625000,20.500000,20.125000,20.000000 +51 5 518 0.271942 518,304,553,389,438,1212,448,1549 29.625000,28.375000,25.500000,23.250000,23.000000,23.000000,22.625000,22.125000 +51 6 279 0.020451 279,576,7148,264,2055,2176,26221,5257 25.000000,20.250000,19.250000,18.125000,18.000000,18.000000,17.875000,17.375000 +51 7 45406 10.174389 16194,5055,2055,1674,15953,1163,14079,13263 20.500000,19.875000,19.500000,19.125000,19.000000,18.250000,18.125000,17.500000 +52 0 26221 0.020060 26221,33496,304,323,1154,22780,37471,320 26.125000,21.625000,20.500000,20.000000,18.875000,18.000000,18.000000,17.875000 +52 1 304 0.210781 304,1154,323,659,320,2587,315,448 28.625000,26.250000,26.000000,25.750000,23.500000,22.125000,21.375000,21.250000 +52 2 70998 6.430344 7148,220,52935,6326,279,86570,356,90157 20.250000,19.000000,19.000000,18.250000,17.750000,17.625000,17.500000,16.500000 +52 3 323 0.361603 323,659,1154,2587,304,1573,320,448 29.250000,27.875000,27.250000,25.250000,24.375000,24.000000,23.750000,23.000000 +52 4 279 0.944805 279,518,2937,1221,304,1083,572,27420 24.000000,23.375000,23.250000,22.875000,21.000000,21.000000,20.000000,19.875000 +52 5 11012 7.956448 16194,15953,1674,1163,5055,4312,1532,13263 19.125000,18.750000,18.250000,18.000000,17.750000,17.125000,17.000000,17.000000 +52 6 1268 0.002666 1268,594,1409,518,304,4813,424,18849 26.000000,19.000000,18.500000,17.875000,17.250000,16.625000,15.937500,15.937500 +52 7 38420 0.000097 38420,19295,14997,92964,17931,659,15619,1154 31.750000,22.250000,20.375000,18.625000,18.375000,18.125000,17.750000,17.625000 +53 0 16937 0.001793 16937,8633,19295,37729,1154,21618,26524,99675 28.875000,22.250000,20.750000,18.375000,18.000000,17.875000,17.750000,17.125000 +53 1 304 0.037256 304,659,1154,323,2587,320,16448,6762 30.500000,27.000000,24.625000,24.250000,23.750000,23.000000,21.875000,21.250000 +53 2 7148 0.000561 7148,4882,220,279,31302,107074,9806,659 29.625000,21.125000,21.000000,19.500000,19.500000,19.125000,18.500000,18.375000 +53 3 659 0.117085 659,2587,1154,304,323,13,16448,6762 29.875000,26.625000,26.375000,25.750000,25.750000,24.875000,24.500000,24.000000 +53 4 1260 1.222822 758,1260,425,5301,576,24885,1096,2938 24.250000,24.000000,23.750000,21.375000,21.125000,20.875000,20.000000,19.500000 +53 5 572 1.588333 572,6342,1030,1083,9723,702,1221,2937 22.750000,22.625000,22.500000,22.375000,21.875000,21.125000,21.000000,20.500000 +53 6 15540 8.320093 1083,6311,264,304,14766,1221,2937,949 23.625000,23.125000,22.250000,21.875000,21.125000,20.875000,20.625000,20.250000 +53 7 553 0.017456 553,304,369,323,1052,518,311,389 29.375000,25.000000,22.500000,22.375000,22.375000,22.000000,21.500000,20.750000 +54 0 3757 3.711679 279,18939,6798,11867,21315,4389,8397,23269 17.625000,17.625000,17.500000,17.125000,17.125000,17.000000,16.875000,16.875000 +54 1 67473 0.115610 67473,350,4509,431,26741,4264,17884,328 20.875000,16.375000,16.250000,16.000000,15.687500,15.250000,15.250000,15.187500 +54 2 323 0.386368 323,659,1154,304,2587,320,369,879 29.250000,28.000000,26.500000,26.125000,25.500000,25.000000,25.000000,24.375000 +54 3 58235 3.604206 572,1030,6342,10660,279,1062,9723,58235 21.000000,20.000000,20.000000,20.000000,19.500000,19.500000,19.500000,19.250000 +54 4 16263 1.010624 448,16263,304,438,14002,518,3156,279 25.750000,25.500000,24.500000,22.250000,21.750000,19.750000,19.625000,19.500000 +54 5 7355 6.781145 19571,12089,4389,11867,23654,20445,18939,24986 20.500000,18.125000,18.000000,18.000000,18.000000,17.875000,17.625000,17.625000 +54 6 1232 0.090221 1232,35203,26649,49535,356,38174,43854,60074 22.375000,18.750000,17.750000,17.750000,17.625000,17.000000,17.000000,16.375000 +54 7 812 0.000257 812,1442,1203,285,14810,2652,48211,139591 29.625000,20.375000,19.625000,19.500000,19.000000,18.000000,17.500000,17.500000 +55 0 672 -0.000000 672,19915,1360,14077,309,40878,87448,363 37.500000,21.625000,21.500000,21.375000,21.125000,20.125000,20.125000,20.000000 +55 1 1154 3.030670 323,659,1154,304,320,2587,16448,6762 29.375000,28.375000,26.750000,25.875000,24.750000,24.500000,23.500000,23.250000 +55 2 49975 10.025879 879,323,6693,448,4389,264,6798,7801 20.875000,20.500000,19.000000,18.375000,18.125000,18.000000,18.000000,17.625000 +55 3 1863 0.692371 1863,98974,506,13646,96882,14259,20696,431 19.750000,19.250000,16.375000,16.250000,16.250000,16.000000,16.000000,15.687500 +55 4 12707 0.951918 67592,12707,437,292,517,3923,16030,278 21.375000,21.250000,19.375000,18.625000,18.250000,17.000000,17.000000,16.750000 +55 5 1154 2.696886 323,1154,320,659,609,304,481,2587 29.250000,26.625000,21.625000,20.875000,20.125000,19.625000,19.625000,19.625000 +55 6 13928 7.981901 323,4389,11867,18939,6798,7801,3757,18095 23.125000,17.875000,17.875000,17.875000,17.750000,17.750000,17.500000,17.375000 +55 7 17734 13.498407 472,1260,45196,4483,350,444,2932,356 19.500000,19.375000,19.375000,19.250000,18.375000,17.750000,17.500000,17.125000 +56 0 1154 2.466133 323,1154,263,659,320,609,2587,1842 29.000000,26.625000,21.625000,21.250000,20.375000,20.000000,19.750000,19.250000 +56 1 55130 12.311621 323,11867,7801,18939,4389,6798,3757,21315 22.750000,18.000000,17.875000,17.875000,17.750000,17.750000,17.500000,17.500000 +56 2 23905 0.652473 23905,7801,362,730,444,4598,31071,17426 20.500000,19.125000,17.625000,17.500000,17.250000,17.250000,16.875000,16.750000 +56 3 416 0.131050 416,265,2718,388,2892,5963,5482,74 24.875000,22.000000,21.875000,20.625000,19.875000,19.125000,19.000000,18.125000 +56 4 1154 1.911043 323,1154,659,320,609,2587,304,16448 30.875000,29.125000,23.125000,21.625000,21.375000,21.375000,20.125000,19.625000 +56 5 59310 6.985514 323,11867,4389,7801,18939,3757,6798,21315 22.250000,18.250000,18.000000,18.000000,18.000000,17.750000,17.750000,17.625000 +56 6 80664 7.387667 25164,10978,4483,33031,36130,467,431,425 20.625000,19.750000,19.000000,19.000000,18.500000,18.375000,18.250000,17.750000 +56 7 323 0.161821 323,1154,659,320,609,2587,304,1842 30.625000,28.875000,23.375000,22.625000,21.875000,21.500000,20.500000,20.500000 +57 0 67034 6.638327 3800,11867,18939,20445,4389,6798,7801,8364 18.750000,18.125000,17.750000,17.750000,17.625000,17.500000,17.375000,17.375000 +57 1 10926 10.155075 10801,4312,386,3128,4483,26234,7401,472 22.250000,22.000000,21.875000,20.875000,18.125000,18.000000,17.875000,17.625000 +57 2 659 0.104498 659,304,16448,1154,2587,6762,323,72 26.125000,22.625000,21.625000,21.375000,21.375000,21.125000,20.875000,20.750000 +57 3 715 5.769854 758,1260,425,576,5301,24885,4710,43562 24.250000,23.500000,23.375000,21.625000,21.000000,20.750000,19.625000,19.625000 +57 4 758 3.454901 284,425,1260,8397,785,33,641,758 20.125000,19.500000,18.750000,18.500000,18.250000,18.125000,18.125000,18.000000 +57 5 220 0.095837 220,279,5256,3217,5813,6702,7400,5470 27.250000,24.500000,22.250000,21.250000,21.250000,21.125000,21.000000,20.875000 +57 6 17 0.000748 17,16,1154,15,18,220,279,20 27.250000,20.000000,14.562500,14.500000,14.437500,14.125000,14.062500,13.000000 +57 7 15 0.000002 15,16,220,24,2624,18,23,17 28.875000,13.750000,13.000000,12.437500,12.375000,12.250000,12.187500,11.937500 +58 0 15 0.014217 15,16,17,220,1154,20,21,549 26.875000,22.625000,16.875000,13.187500,12.687500,11.937500,11.625000,11.625000 +58 1 21 0.409723 21,22,20,23,24,19,18,17 24.000000,22.750000,21.875000,21.250000,20.250000,18.750000,18.500000,17.750000 +58 2 1154 2.889454 425,566,1154,569,806,323,8397,1260 28.875000,26.375000,26.125000,22.250000,21.250000,20.500000,20.500000,19.875000 +58 3 425 0.090849 425,566,8397,806,1260,279,1283,264 28.625000,26.250000,21.375000,20.375000,20.250000,19.625000,18.875000,18.750000 +58 4 10965 0.000008 10965,965,616,581,22604,2950,220,39337 29.750000,16.750000,16.125000,15.562500,15.062500,14.562500,14.437500,14.312500 +58 5 465 -0.000000 465,5045,56960,696,376,4160,1982,4198 43.000000,26.125000,25.250000,23.875000,23.750000,22.500000,22.250000,21.750000 +58 6 58235 4.152901 6342,572,9723,10859,3867,1030,58235,44777 24.750000,24.375000,23.875000,22.750000,22.625000,22.375000,21.625000,21.500000 +58 7 16263 4.863850 304,438,16263,448,518,389,569,14002 27.375000,26.500000,22.875000,22.125000,20.875000,20.375000,20.000000,19.625000 +59 0 1232 10.255280 4389,19571,11867,6798,8397,3757,18939,24986 18.000000,18.000000,17.750000,17.625000,17.375000,17.250000,17.250000,17.250000 +59 1 812 0.003386 812,380,3187,5853,72,28300,1442,483 23.875000,17.000000,16.000000,15.812500,14.500000,14.500000,14.437500,14.375000 +59 2 672 0.000004 672,19915,365,363,8635,6379,10303,23397 33.500000,20.500000,18.875000,17.500000,17.500000,17.250000,17.000000,16.500000 +59 3 304 0.767990 304,323,1549,1154,438,518,320,389 29.000000,28.750000,27.625000,26.625000,24.625000,24.000000,23.625000,22.250000 +59 4 279 0.085732 279,264,576,2441,458,806,13293,330 25.750000,22.625000,21.125000,20.875000,20.125000,19.125000,19.000000,18.875000 +59 5 1486 0.616413 1486,12425,4531,6430,18096,2805,4312,22358 22.750000,21.250000,21.000000,19.875000,19.375000,19.250000,19.250000,19.250000 +59 6 93736 12.688566 576,1154,425,434,362,330,422,358 19.625000,17.125000,16.750000,16.625000,16.500000,16.375000,16.375000,16.250000 +59 7 5326 1.882917 1154,518,5326,553,304,323,659,315 25.750000,25.125000,24.875000,24.500000,23.375000,23.375000,23.125000,22.250000 +60 0 553 0.001316 553,323,369,304,518,448,438,1154 29.875000,22.875000,21.250000,20.125000,20.000000,18.500000,18.375000,18.250000 +60 1 4389 3.760677 6798,21315,29405,14991,4389,11867,18095,33121 18.500000,18.500000,17.750000,17.500000,17.375000,17.375000,17.375000,17.125000 +60 2 45107 1.309662 506,45107,1967,431,472,425,25369,25164 21.875000,21.000000,17.875000,17.500000,17.500000,17.000000,16.875000,16.625000 +60 3 27507 0.000805 27507,659,1154,481,323,42241,8108,71 29.875000,22.375000,21.000000,20.250000,19.125000,18.625000,16.625000,16.125000 +60 4 659 1.622293 323,518,659,1154,2587,304,369,892 29.250000,29.000000,28.875000,28.875000,27.750000,25.750000,25.625000,25.125000 +60 5 1260 1.187468 1260,1096,576,758,425,1084,5301,2938 23.625000,23.500000,23.000000,22.750000,21.875000,21.000000,20.125000,19.750000 +60 6 9723 2.923809 572,1083,6342,1030,9723,10660,702,1221 23.875000,23.000000,22.750000,22.375000,21.875000,21.125000,21.000000,20.875000 +60 7 389 4.755703 304,438,389,518,448,16263,1549,26753 28.500000,26.250000,23.875000,23.750000,23.375000,22.625000,22.500000,21.375000 +61 0 264 4.173363 279,12425,6430,264,4171,18096,576,3807 24.250000,22.875000,22.250000,20.500000,20.125000,20.125000,19.500000,19.000000 +61 1 220 3.273695 12425,1372,4013,220,6430,18096,2805,2086 21.625000,21.125000,20.375000,19.750000,19.625000,19.500000,19.250000,19.000000 +61 2 17 0.015329 17,16,18,19,22,20,21,24 28.750000,24.125000,22.750000,21.500000,21.500000,21.375000,21.125000,20.625000 +61 3 15 0.000267 15,16,17,18,19,20,24,22 27.500000,17.750000,17.125000,16.875000,16.750000,16.500000,16.500000,16.250000 +61 4 15 0.023333 15,16,17,5534,339,5768,21,5813 27.750000,24.000000,17.750000,15.250000,14.687500,14.687500,14.625000,14.500000 +61 5 21 1.399513 22,21,23,24,20,19,18,17 23.250000,22.375000,21.625000,20.250000,18.875000,18.125000,17.750000,16.875000 +61 6 9234 1.468105 12425,9234,4013,5670,18096,481,3200,6430 20.500000,20.375000,18.750000,18.500000,18.250000,18.000000,18.000000,17.625000 +61 7 315 0.001373 315,24849,304,389,1154,19867,320,369 30.375000,22.625000,21.500000,21.125000,21.000000,20.750000,20.375000,20.125000 +62 0 279 0.133627 279,576,7506,18096,48822,18635,15696,6326 23.000000,20.250000,17.875000,17.750000,17.625000,17.375000,17.125000,16.750000 +62 1 12425 0.665522 12425,18096,22358,4013,7855,13434,80366,19584 22.625000,21.750000,20.375000,20.250000,20.000000,18.875000,18.875000,18.375000 +62 2 4013 0.027206 4013,1473,2025,18136,19584,22358,6146,80366 27.125000,22.250000,22.125000,22.000000,21.250000,20.750000,19.375000,19.375000 +62 3 1154 3.944342 576,9414,7506,18635,48822,85450,15696,3089 17.625000,16.000000,15.937500,15.875000,15.875000,15.875000,15.750000,15.500000 +62 4 53843 6.701771 576,323,892,279,220,448,48822,2670 19.000000,18.375000,17.750000,17.125000,17.000000,16.750000,16.500000,16.375000 +62 5 1154 0.638203 1154,659,320,2587,323,549,304,438 26.500000,26.000000,24.125000,23.875000,23.000000,22.875000,22.500000,22.000000 +62 6 8110 8.281076 323,438,5619,304,892,1380,448,714 25.000000,24.500000,24.375000,24.250000,23.500000,22.250000,22.125000,21.375000 +62 7 553 0.010138 553,304,2937,19620,264,448,279,705 29.250000,24.250000,22.375000,21.750000,21.500000,21.125000,20.500000,20.500000 +63 0 264 0.165865 264,806,2441,458,279,26255,12783,5619 26.500000,23.250000,23.250000,22.875000,22.625000,22.125000,21.875000,20.750000 +63 1 3476 0.553259 3476,8640,220,12613,39400,45250,6388,949 23.750000,22.250000,21.875000,20.875000,20.750000,20.750000,20.125000,19.625000 +63 2 304 0.084801 304,438,389,315,5619,448,518,369 29.375000,26.750000,24.750000,23.125000,22.500000,22.375000,22.000000,21.125000 +63 3 279 0.073317 279,264,220,576,458,2441,18096,1154 26.125000,22.750000,21.375000,21.125000,20.750000,20.750000,18.750000,18.500000 +63 4 220 0.634710 220,12425,1486,4531,18096,4013,22358,4565 23.250000,22.250000,21.750000,20.625000,20.000000,19.250000,19.000000,18.250000 +63 5 17 0.000715 17,16,18,19,22,20,21,23 28.250000,20.625000,18.750000,17.875000,17.875000,17.625000,17.375000,16.875000 +63 6 15 0.000004 15,16,17,1154,549,18,23,1635 30.000000,16.875000,14.625000,14.312500,13.437500,13.375000,13.312500,13.062500 +63 7 15 0.002481 15,16,17,21,22,481,220,1154 28.000000,22.000000,14.312500,13.250000,12.625000,12.500000,12.437500,12.312500 +64 0 22 0.341499 22,21,23,24,20,19,18,17 24.500000,23.375000,21.875000,19.750000,17.375000,16.000000,15.562500,14.812500 +64 1 33496 6.025534 12425,4013,4531,1486,9234,18096,22358,5670 22.000000,20.875000,20.625000,20.125000,19.250000,18.750000,18.625000,18.000000 +64 2 5670 0.207182 5670,1486,53148,5068,1473,33085,56005,23675 25.000000,22.875000,22.250000,20.375000,19.375000,18.875000,18.875000,18.750000 +64 3 315 0.013435 315,1154,576,659,518,8755,323,43562 27.125000,20.875000,20.500000,20.250000,20.125000,19.375000,19.125000,18.875000 +64 4 2585 7.463917 279,576,264,6798,362,1154,3757,11867 22.250000,20.500000,17.500000,17.250000,16.875000,16.875000,16.875000,16.875000 +64 5 311 1.712418 358,311,1205,279,1260,2806,23309,3155 22.625000,21.500000,20.875000,20.250000,19.125000,18.625000,18.500000,18.375000 +64 6 70627 11.506456 12190,4093,2126,2823,13655,387,11158,22749 21.250000,20.375000,19.875000,18.750000,18.750000,18.375000,18.375000,18.250000 +64 7 15540 6.837259 304,264,659,1154,518,5326,549,323 23.125000,22.375000,22.000000,22.000000,21.875000,21.625000,21.500000,21.375000 +65 0 553 0.000324 553,323,518,304,1154,448,569,3216 29.875000,21.250000,19.875000,18.625000,18.500000,18.375000,17.875000,17.500000 +65 1 27878 8.399975 4389,21315,18939,3757,11867,7801,20445,6798 18.625000,18.125000,17.750000,17.375000,17.125000,17.000000,17.000000,16.875000 +65 2 645 0.000069 645,25503,19310,472,72,6535,220,6023 29.625000,18.625000,17.875000,17.500000,17.375000,17.250000,17.125000,17.125000 +65 3 431 0.114969 431,659,451,29938,425,328,5807,444 19.875000,16.375000,16.000000,15.062500,15.000000,14.750000,14.687500,14.375000 +65 4 84434 0.139278 84434,13,12523,2667,4076,26378,9605,1021 21.875000,19.250000,18.375000,17.250000,15.812500,15.625000,15.500000,15.437500 +65 5 659 0.678722 659,323,518,1154,304,2587,369,320 28.375000,27.875000,26.750000,25.750000,25.000000,24.375000,24.000000,23.375000 +65 6 2585 10.259563 1260,758,425,576,5301,715,1096,220 24.500000,24.375000,22.625000,21.375000,21.375000,21.125000,21.000000,19.875000 +65 7 311 0.004797 311,2014,279,1521,566,653,374,569 25.375000,19.125000,17.875000,17.125000,16.750000,16.750000,16.125000,16.125000 +66 0 70627 0.000408 70627,40000,69719,45975,12886,96894,53677,42705 27.000000,18.000000,18.000000,16.750000,15.937500,15.500000,15.000000,14.625000 +66 1 572 0.144394 572,374,10613,1154,84070,8930,364,1030 27.000000,23.500000,23.125000,22.875000,22.750000,22.500000,22.000000,22.000000 +66 2 10660 0.203054 10660,8947,264,1083,50057,10449,6342,2937 25.000000,21.875000,21.625000,21.375000,21.375000,21.125000,20.125000,19.875000 +66 3 518 0.138224 518,304,553,389,438,448,504,1156 30.000000,28.000000,25.000000,23.750000,23.000000,23.000000,21.500000,21.500000 +66 4 14079 10.139832 279,576,7148,2176,264,2055,26221,350 25.250000,20.000000,19.000000,18.500000,18.250000,18.125000,17.625000,17.125000 +66 5 26221 0.005322 26221,33496,37471,576,46113,323,4678,1154 26.125000,20.625000,18.000000,17.625000,17.500000,17.125000,16.750000,16.375000 +66 6 304 0.459089 304,323,659,1154,320,2587,364,448 27.875000,26.875000,25.750000,25.250000,22.875000,21.750000,21.500000,21.250000 +66 7 279 6.399178 7148,220,279,79593,58546,107074,2932,4312 25.375000,21.500000,19.000000,17.125000,16.500000,16.000000,15.875000,15.750000 +67 0 7148 1.348850 6424,7148,3639,4312,220,1852,4882,9710 23.000000,22.375000,22.125000,19.500000,18.250000,18.125000,17.875000,17.750000 +67 1 84203 0.019958 84203,65741,659,323,1154,3082,220,4312 29.000000,24.875000,22.875000,21.250000,21.250000,21.000000,20.125000,19.250000 +67 2 315 0.000298 315,4940,82,659,1154,220,320,2587 25.000000,16.000000,14.875000,13.875000,13.875000,13.437500,13.437500,12.875000 +67 3 472 4.328083 95733,82468,34982,46943,68929,46331,31302,472 22.250000,22.125000,20.125000,20.000000,20.000000,19.875000,19.375000,18.875000 +67 4 42057 0.987804 1624,42057,3249,8985,546,529,4681,994 26.750000,26.250000,22.875000,19.000000,18.375000,18.125000,17.875000,17.250000 +67 5 43674 -0.000000 43674,659,76,8155,1763,1306,25013,19501 35.500000,20.625000,20.250000,19.500000,19.375000,18.875000,18.375000,18.250000 +67 6 323 0.023401 323,609,659,1154,481,304,569,608 30.750000,26.250000,26.250000,23.500000,22.125000,21.500000,21.250000,20.875000 +67 7 47450 0.000929 47450,659,5599,1154,279,434,4312,14010 27.500000,19.750000,19.125000,17.750000,17.500000,16.875000,16.500000,16.375000 +68 0 5604 0.000084 5604,8187,6660,73493,13515,72116,13,71 29.750000,19.625000,19.000000,18.375000,17.875000,15.437500,14.312500,14.250000 +68 1 659 0.346903 659,323,1154,304,2587,16448,320,13 29.125000,27.875000,26.125000,25.875000,24.875000,24.125000,23.750000,23.125000 +68 2 425 2.114649 758,1260,425,715,576,4710,5301,1096 24.125000,24.125000,23.000000,22.375000,20.750000,20.625000,20.625000,19.625000 +68 3 10965 0.000006 10965,616,965,6344,581,220,13,2950 29.625000,15.750000,15.750000,15.562500,15.125000,14.500000,14.062500,14.062500 +68 4 465 -0.000000 465,1536,5045,56960,665,696,1982,3667 40.000000,23.500000,23.250000,22.375000,21.875000,21.875000,21.375000,21.375000 +68 5 58235 4.778940 572,1083,6342,9723,1221,364,702,1030 23.875000,23.250000,23.000000,22.625000,22.375000,22.125000,22.125000,21.875000 +68 6 304 0.178432 304,16263,438,448,1549,518,389,14002 26.500000,24.000000,23.875000,22.750000,21.375000,20.500000,20.250000,19.750000 +68 7 1378 5.774031 279,264,220,2441,576,458,1378,3807 25.875000,23.500000,21.750000,20.750000,20.625000,20.375000,20.250000,20.250000 +69 0 12351 3.910610 11088,220,12425,53148,803,1008,2155,12783 22.000000,21.375000,20.875000,20.625000,20.500000,20.500000,20.125000,20.000000 +69 1 304 0.842764 304,549,1154,5880,2337,2587,448,15540 26.875000,26.750000,25.125000,23.500000,23.250000,23.000000,22.875000,22.750000 +69 2 220 0.053831 220,279,806,892,419,4124,1154,3309 27.375000,24.375000,20.250000,20.250000,19.500000,19.500000,19.250000,19.000000 +69 3 17 0.000345 17,16,15,1154,220,18,279,320 26.000000,18.000000,12.375000,12.312500,11.875000,11.625000,10.875000,10.875000 +69 4 15 -0.000000 15,16,17,18,220,20,23,1112 29.375000,12.562500,12.062500,11.625000,11.187500,10.812500,10.625000,10.562500 +69 5 15 0.018167 15,16,17,220,1154,481,549,198 25.875000,21.875000,14.500000,11.812500,11.500000,10.687500,10.562500,10.000000 +69 6 23 1.134024 22,23,24,21,20,19,18,17 23.125000,22.500000,20.500000,19.875000,18.000000,17.750000,17.500000,16.750000 +69 7 1154 2.915783 549,1154,2587,659,323,481,1177,1959 28.875000,26.125000,25.875000,25.625000,24.250000,23.500000,23.250000,23.000000 +70 0 6059 9.341125 2670,576,2176,279,825,5619,323,304 21.500000,21.125000,20.875000,20.500000,20.125000,19.625000,19.375000,19.250000 +70 1 4145 0.110571 4145,315,8960,323,15623,18306,220,32981 24.125000,19.875000,19.875000,19.750000,19.625000,19.500000,19.000000,18.750000 +70 2 4892 2.707407 323,4892,48438,57679,320,549,220,304 24.750000,22.375000,22.375000,22.000000,21.750000,20.625000,20.250000,19.750000 +70 3 30204 0.000919 30204,652,1902,7132,65,7964,82,1330 34.750000,27.125000,27.000000,20.000000,18.000000,16.875000,16.250000,15.375000 +70 4 553 8.252722 323,320,1154,304,448,549,15540,5326 28.750000,26.375000,24.875000,23.375000,23.250000,23.000000,22.375000,21.750000 +70 5 56237 7.056473 11867,4389,6798,3757,21315,7538,18939,14991 17.625000,17.375000,17.375000,17.250000,17.250000,17.125000,17.125000,17.000000 +70 6 12095 9.626516 4389,11867,3757,6798,18939,21315,6898,33121 17.125000,17.125000,17.000000,17.000000,16.875000,16.750000,16.625000,16.500000 +70 7 1967 0.096632 1967,67912,66097,1863,69231,350,85656,37705 20.750000,17.125000,16.250000,15.500000,15.312500,15.250000,15.000000,14.875000 +71 0 544 4.164993 288,544,50,64,300,51,4422,73814 20.375000,16.250000,14.875000,14.687500,14.375000,13.937500,13.937500,13.562500 +71 1 72 0.007160 72,285,6591,9156,98255,4738,645,859 30.500000,25.500000,22.000000,20.875000,20.375000,20.125000,20.000000,20.000000 +71 2 1154 2.487192 323,1154,320,304,2587,659,481,549 29.000000,26.625000,25.000000,22.875000,22.500000,22.000000,21.000000,20.875000 +71 3 323 0.090385 323,304,892,264,448,1380,39400,369 26.750000,23.500000,23.125000,21.250000,21.125000,20.500000,20.500000,20.375000 +71 4 4320 6.956036 576,279,264,304,13293,362,1154,425 19.875000,19.375000,17.500000,17.000000,17.000000,16.875000,16.625000,16.500000 +71 5 792 2.388796 944,792,364,1405,10563,3406,64,10810 20.875000,18.750000,18.125000,17.000000,15.875000,15.500000,15.437500,15.437500 +71 6 53175 1.880667 1154,53175,304,323,594,481,553,32342 20.375000,19.875000,19.625000,19.125000,18.750000,18.500000,18.250000,18.000000 +71 7 15540 3.252323 553,1154,15540,659,320,5326,304,549 27.500000,25.375000,24.500000,24.125000,24.000000,22.750000,22.375000,22.125000 +72 0 553 0.000980 553,323,304,1154,369,448,659,3216 29.625000,22.500000,19.375000,19.000000,18.250000,17.750000,17.750000,17.625000 +72 1 11959 7.590174 21315,6798,7801,14991,18939,35757,3757,12066 17.125000,16.750000,16.750000,16.750000,16.625000,16.625000,16.375000,16.250000 +72 2 398 0.038385 398,28206,2577,6355,8346,6743,46716,685 24.250000,20.875000,18.250000,16.375000,15.812500,15.687500,15.687500,15.625000 +72 3 88353 2.539148 13612,88353,3984,20042,468,14595,29660,356 17.125000,15.937500,15.500000,15.312500,15.187500,15.125000,15.125000,14.937500 +72 4 659 0.338022 659,323,1154,2587,16448,448,304,320 28.375000,27.000000,25.625000,25.375000,23.625000,23.250000,23.000000,22.625000 +72 5 758 0.963053 758,1260,425,11733,715,576,5301,220 24.000000,23.875000,22.625000,21.875000,21.625000,21.375000,21.125000,20.250000 +72 6 3217 7.157238 220,279,2176,264,5256,419,806,1493 26.375000,24.125000,22.625000,21.375000,21.125000,20.625000,20.500000,20.500000 +72 7 220 0.009319 220,315,323,1154,279,481,311,198 27.875000,22.250000,21.875000,21.625000,19.250000,18.875000,18.750000,18.625000 +73 0 17 0.000109 17,16,18,220,15,1154,19,20 27.000000,17.750000,14.437500,14.125000,13.625000,12.562500,12.250000,12.187500 +73 1 15 -0.000000 15,16,2624,78,17,220,18,21 33.250000,14.750000,13.625000,13.250000,13.125000,12.687500,12.000000,12.000000 +73 2 15 0.142752 15,16,17,220,1154,11,23,198 23.625000,21.750000,14.062500,10.500000,10.062500,9.500000,8.750000,8.625000 +73 3 23 0.903038 24,23,22,21,20,15,19,18 22.875000,22.500000,17.875000,15.875000,14.312500,13.937500,13.875000,13.625000 +73 4 1154 0.842928 1154,425,566,806,264,323,279,432 28.125000,27.875000,27.500000,21.375000,20.875000,20.750000,20.375000,20.375000 +73 5 425 0.640942 425,566,806,279,8397,264,432,1260 26.625000,26.500000,20.625000,20.375000,20.125000,20.000000,19.750000,19.750000 +73 6 10965 -0.000000 10965,616,965,2950,581,1080,3431,1624 32.250000,18.375000,16.625000,16.125000,15.812500,15.500000,15.500000,15.437500 +73 7 465 -0.000000 465,5045,665,696,56960,1982,3667,59385 38.500000,23.000000,20.750000,20.500000,20.375000,19.875000,19.750000,19.250000 +74 0 1865 3.828510 572,6342,9723,3867,10859,1030,10660,3949 24.375000,23.000000,23.000000,22.250000,21.875000,21.750000,21.750000,21.625000 +74 1 264 1.145658 806,264,458,279,26255,2441,825,12425 28.625000,28.000000,26.625000,23.250000,22.625000,21.375000,21.125000,21.000000 +74 2 8640 0.529167 8640,9814,94056,12425,2805,825,3281,7743 23.750000,21.875000,21.875000,21.375000,21.000000,20.125000,19.500000,19.500000 +74 3 11094 0.027109 11094,569,3476,26255,12425,39400,5068,6774 25.750000,21.500000,19.625000,19.250000,18.875000,18.875000,18.625000,18.500000 +74 4 389 0.574217 389,304,518,438,369,448,320,315 29.250000,28.875000,26.125000,25.750000,23.875000,23.875000,22.375000,21.750000 +74 5 264 4.566364 279,576,264,6326,18096,458,48822,1154 24.625000,20.500000,20.125000,19.375000,19.000000,18.875000,18.125000,17.875000 +74 6 1378 5.607271 12425,4013,220,3175,18096,3281,3887,9234 21.125000,20.250000,20.125000,20.000000,20.000000,19.875000,19.750000,19.750000 +74 7 569 2.346378 481,28037,12,569,9234,949,29509,21231 20.625000,20.375000,19.125000,19.125000,16.750000,16.500000,16.000000,15.687500 +75 0 12 0.000105 12,569,313,36519,28,1154,11,481 30.625000,19.375000,19.250000,19.250000,19.000000,18.875000,18.375000,18.375000 +75 1 31 0.000040 31,62799,569,19191,26,19347,323,39291 31.875000,21.000000,20.875000,17.875000,16.750000,16.750000,16.500000,16.250000 +75 2 949 0.805830 9234,949,2003,6460,481,1899,2254,3668 25.250000,25.125000,21.250000,21.250000,21.125000,20.750000,20.000000,19.500000 +75 3 9234 0.084539 9234,3281,4013,12425,315,18096,18635,17770 23.875000,19.875000,19.750000,19.750000,18.375000,18.250000,18.125000,17.625000 +75 4 15580 14.443222 315,304,24849,389,320,19867,1154,369 30.000000,23.500000,23.250000,23.125000,22.125000,22.000000,21.875000,21.000000 +75 5 315 0.098663 315,389,304,369,320,1154,24849,438 29.375000,26.500000,26.125000,23.125000,22.625000,22.375000,22.125000,21.875000 +75 6 279 0.045074 279,576,6326,7506,48822,15696,5748,18635 24.625000,20.750000,19.375000,18.625000,18.125000,17.625000,17.500000,17.500000 +75 7 12425 0.436167 12425,18096,4013,1293,7855,22358,13434,5411 23.250000,21.250000,21.125000,20.250000,20.250000,19.125000,18.875000,18.750000 +76 0 4013 0.006680 4013,19584,1473,2025,18136,6146,22358,80366 28.375000,22.250000,21.750000,21.375000,21.375000,20.250000,20.125000,19.500000 +76 1 467 6.097881 1154,576,7506,6326,18635,52325,48822,15696 19.625000,18.375000,17.750000,17.625000,17.625000,17.500000,16.875000,16.750000 +76 2 1765 0.070820 1765,16,3104,6179,2317,27313,449,4872 22.625000,19.000000,18.750000,17.750000,17.000000,16.125000,15.750000,15.687500 +76 3 279 0.011339 279,576,659,1154,29092,21192,30833,12826 26.625000,22.125000,16.875000,16.500000,16.375000,15.562500,15.187500,14.937500 +76 4 15021 0.004673 15021,5593,7840,17829,13532,350,82988,37806 22.000000,14.500000,14.125000,14.062500,13.937500,13.500000,13.187500,12.875000 +76 5 1154 1.428856 659,1154,320,2587,323,304,438,448 27.125000,26.375000,24.875000,24.750000,24.375000,24.250000,23.750000,23.750000 +76 6 8110 5.414007 892,5619,304,323,1380,448,6342,16263 23.750000,23.375000,23.250000,22.750000,22.375000,21.625000,21.250000,20.750000 +76 7 553 0.007786 553,304,2937,19620,264,448,389,7069 28.250000,23.125000,20.875000,20.625000,19.250000,18.875000,18.125000,18.125000 +77 0 458 3.763317 264,806,458,2441,279,26255,12783,5619 26.125000,22.750000,22.500000,22.375000,21.750000,21.375000,21.125000,20.625000 +77 1 11094 0.089790 11094,9234,5107,15358,11577,61888,650,5813 25.000000,21.625000,20.625000,20.000000,19.375000,19.125000,18.875000,18.625000 +77 2 389 2.058479 304,389,438,518,448,315,279,369 30.250000,28.375000,26.500000,26.125000,24.000000,23.625000,23.500000,23.125000 +77 3 279 0.288771 279,264,458,576,2441,220,18096,7506 24.500000,22.875000,21.250000,20.625000,20.250000,19.250000,18.500000,18.375000 +77 4 12425 0.742733 12425,220,4013,18096,22358,1378,2086,9234 23.000000,22.250000,21.000000,20.125000,19.625000,19.500000,19.250000,19.125000 +77 5 4013 0.010630 4013,1473,18136,19584,2025,22358,6146,13420 27.250000,21.250000,21.000000,21.000000,20.875000,19.500000,18.875000,18.750000 +77 6 28778 6.603430 576,1154,9414,7506,659,48822,15696,59455 17.875000,17.750000,16.500000,16.125000,16.000000,15.812500,15.500000,15.500000 +77 7 85393 0.289145 85393,2249,533,344,2221,49675,1524,3936 30.375000,29.000000,27.875000,21.625000,21.375000,20.875000,16.750000,16.500000 +78 0 304 0.277006 304,320,659,1154,323,892,438,549 27.625000,25.625000,25.375000,23.750000,22.875000,22.750000,22.125000,22.125000 +78 1 6702 4.387726 220,279,5534,6122,5768,6156,6527,6702 25.750000,23.375000,22.250000,22.250000,21.875000,21.875000,21.625000,21.625000 +78 2 220 0.032499 220,315,659,323,279,429,1154,16448 29.125000,25.500000,23.375000,21.750000,21.625000,21.625000,21.375000,21.250000 +78 3 17 0.000008 17,16,220,15,18,323,1154,315 29.000000,16.250000,14.875000,14.500000,14.500000,13.437500,13.437500,13.187500 +78 4 15 -0.000000 15,78,2624,13,659,220,16,1154 28.250000,11.312500,11.250000,10.875000,10.500000,10.062500,10.000000,9.375000 +78 5 15 0.004080 15,16,17,1154,23,220,659,2587 28.500000,23.000000,13.875000,13.687500,12.562500,11.875000,11.875000,11.625000 +78 6 23 0.033768 23,24,22,21,19,20,18,15 25.375000,22.000000,16.125000,13.875000,13.062500,13.000000,12.625000,12.375000 +78 7 659 0.068289 659,16448,323,6762,1154,2587,320,13 29.000000,25.250000,25.125000,24.250000,24.000000,23.375000,22.625000,21.875000 +79 0 1260 1.453676 758,1260,425,715,4710,5301,220,576 25.000000,24.250000,23.500000,23.375000,21.625000,21.000000,20.375000,20.375000 +79 1 1030 2.365362 572,1083,9723,6342,1030,58235,1221,1865 23.000000,22.250000,22.000000,21.875000,21.750000,20.750000,20.375000,20.375000 +79 2 264 0.023874 264,458,2441,806,279,45250,12783,8922 28.125000,22.875000,22.875000,22.125000,21.750000,21.500000,21.250000,20.875000 +79 3 45250 2.588278 3476,8640,12613,45250,8922,2613,6388,39400 23.375000,22.125000,21.500000,21.500000,21.000000,20.625000,20.625000,19.875000 +79 4 3476 0.007931 3476,8640,12613,949,12425,323,3668,11094 26.125000,20.750000,18.375000,18.000000,18.000000,17.750000,17.625000,17.625000 +79 5 304 0.302860 304,438,389,315,504,5619,369,448 28.750000,27.250000,26.500000,24.125000,23.500000,23.250000,21.875000,21.750000 +79 6 5779 13.080235 279,220,576,264,28778,18096,458,4013 26.875000,21.500000,20.875000,20.750000,20.500000,18.750000,18.625000,18.625000 +79 7 17770 0.018698 17770,315,220,9234,23921,481,12425,5479 23.500000,18.000000,17.625000,16.875000,16.750000,16.500000,16.000000,15.875000 diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w1.tsv b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w1.tsv new file mode 100644 index 000000000..fae8de093 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w1.tsv @@ -0,0 +1,135 @@ +# model models/mlx/Qwen3-30B-A3B-4bit +# device GPU +# corpus tests/fixtures/wikitext2_excerpt.txt tokens 299190 +# chunks 128 chunk_tokens 1 topk 8 prefill 0 +# columns chunk pos target nll top_ids top_logits +# start_token 0 +# contiguous false +0 0 42105 16.162960 1124,370,273,2533,37018,709,1318,654 21.250000,18.750000,16.500000,15.750000,15.750000,15.375000,15.375000,15.312500 +1 0 275 12.278282 1124,11520,2533,1318,921,273,9640,50853 21.750000,17.250000,16.250000,15.062500,14.937500,14.687500,14.625000,14.187500 +2 0 427 13.990677 1124,370,11520,654,2533,37018,273,408 21.875000,19.500000,19.000000,18.625000,17.000000,17.000000,16.625000,16.375000 +3 0 12 6.237040 1124,273,370,8,11,37018,1318,354 15.000000,14.937500,14.750000,14.312500,14.250000,14.062500,13.875000,13.312500 +4 0 17 8.112375 1124,2533,921,1318,11520,1125,370,408 18.875000,14.875000,14.625000,14.375000,14.250000,14.125000,13.937500,13.812500 +5 0 3393 14.240599 1124,11520,2533,8,921,701,568,60 17.625000,15.875000,15.812500,15.687500,15.625000,15.000000,14.062500,14.000000 +6 0 75734 16.391666 1124,370,273,67,220,11,408,8 15.812500,15.187500,12.812500,12.687500,12.625000,12.375000,12.375000,12.250000 +7 0 79483 16.252052 1124,370,654,3808,220,15,59,516 20.750000,18.625000,16.875000,16.875000,16.000000,15.937500,15.750000,15.750000 +8 0 3608 15.981388 1124,654,370,220,39022,668,516,3044 21.875000,19.625000,18.750000,17.500000,16.875000,16.750000,16.625000,16.125000 +9 0 257 11.070562 220,315,25,12,15,17,323,16 15.375000,15.062500,14.937500,13.937500,13.875000,13.875000,13.375000,13.187500 +10 0 549 11.602852 1124,353,1125,2533,3044,921,15,60 19.750000,14.750000,14.500000,14.125000,13.125000,13.062500,13.000000,12.812500 +11 0 3703 12.532848 1124,353,1125,2533,1318,921,701,9533 16.625000,13.125000,12.937500,12.875000,12.562500,11.937500,11.687500,11.500000 +12 0 1110 9.645788 1124,59,220,25,11,13,16,17 16.625000,12.625000,12.375000,12.250000,12.000000,12.000000,12.000000,12.000000 +13 0 71 5.049170 1124,370,59,220,1699,85,15,12 17.625000,14.687500,14.500000,14.500000,14.187500,14.000000,13.812500,13.625000 +14 0 35268 15.400644 1124,11520,2533,921,8,37018,408,273 23.125000,19.125000,15.937500,15.562500,15.500000,15.187500,15.000000,14.937500 +15 0 1564 8.833986 1124,220,71,304,709,370,516,3422 16.500000,14.687500,14.125000,13.187500,13.187500,13.062500,12.937500,12.937500 +16 0 6830 13.126095 1124,370,654,668,516,59,220,284 20.750000,17.875000,16.875000,16.750000,16.625000,16.250000,16.250000,16.000000 +17 0 3446 5.952713 1124,59,346,60,273,370,9533,17 19.000000,17.250000,16.000000,15.937500,15.812500,15.750000,15.750000,15.562500 +18 0 76649 14.275181 8,1124,921,1318,408,60,701,11 15.437500,15.437500,14.687500,14.625000,14.562500,14.312500,14.250000,14.187500 +19 0 11374 7.882776 1124,370,516,13,11,12,25,7370 15.250000,14.562500,14.062500,14.000000,13.937500,13.812500,13.812500,13.812500 +20 0 3831 13.739841 1124,8,11,701,273,60,25,2533 17.125000,14.937500,14.687500,14.500000,14.125000,13.875000,13.687500,13.687500 +21 0 8833 15.255756 1124,3226,516,323,315,11,220,4260 14.562500,14.250000,13.125000,13.000000,12.812500,12.687500,12.375000,12.312500 +22 0 6324 13.953758 654,1124,370,516,4730,12,273,220 20.625000,18.625000,17.625000,17.375000,16.750000,16.375000,16.375000,16.125000 +23 0 60403 14.136164 1124,408,273,8,2533,60,921,11520 18.500000,17.375000,15.875000,15.812500,15.500000,15.375000,15.375000,15.375000 +24 0 427 14.989000 370,12,220,1124,516,3226,15,654 14.937500,14.750000,13.937500,13.625000,13.187500,13.000000,12.812500,12.750000 +25 0 198 5.174540 1124,273,370,8,11,37018,1318,354 15.000000,14.937500,14.750000,14.312500,14.250000,14.062500,13.875000,13.312500 +26 0 70584 21.212875 1124,370,654,1699,59,3044,516,3422 24.750000,21.250000,20.375000,18.500000,17.875000,17.375000,17.125000,17.000000 +27 0 257 11.766346 1124,370,273,65,220,516,15,315 18.375000,15.375000,14.562500,14.125000,14.000000,13.750000,13.687500,13.687500 +28 0 549 11.602852 1124,353,1125,2533,3044,921,15,60 19.750000,14.750000,14.500000,14.125000,13.125000,13.062500,13.000000,12.812500 +29 0 289 9.939098 1124,353,1125,2533,1318,921,701,9533 16.625000,13.125000,12.937500,12.875000,12.562500,11.937500,11.687500,11.500000 +30 0 60403 16.117622 1124,11520,8,2533,921,408,701,11 19.000000,16.750000,16.500000,16.125000,15.875000,15.812500,15.750000,15.437500 +31 0 427 14.989000 370,12,220,1124,516,3226,15,654 14.937500,14.750000,13.937500,13.625000,13.187500,13.000000,12.812500,12.750000 +32 0 12 6.237040 1124,273,370,8,11,37018,1318,354 15.000000,14.937500,14.750000,14.312500,14.250000,14.062500,13.875000,13.312500 +33 0 17 8.112375 1124,2533,921,1318,11520,1125,370,408 18.875000,14.875000,14.625000,14.375000,14.250000,14.125000,13.937500,13.812500 +34 0 12 8.365599 1124,11520,2533,8,921,701,568,60 17.625000,15.875000,15.812500,15.687500,15.625000,15.000000,14.062500,14.000000 +35 0 1041 15.049875 1124,2533,921,1318,11520,1125,370,408 18.875000,14.875000,14.625000,14.375000,14.250000,14.125000,13.937500,13.812500 +36 0 8273 11.003105 1124,59,62,12,75738,284,79,220 19.250000,18.125000,18.125000,17.500000,17.000000,16.875000,16.250000,16.000000 +37 0 16 11.679699 1124,11520,2533,921,408,9640,37018,273 20.625000,19.500000,18.875000,17.500000,17.375000,17.125000,16.875000,16.750000 +38 0 198 4.909811 13,16,17,15,24,5373,18,20 21.000000,20.000000,19.375000,19.250000,18.875000,18.875000,18.750000,18.250000 +39 0 20193 20.681625 1124,370,654,1699,59,3044,516,3422 24.750000,21.250000,20.375000,18.500000,17.875000,17.375000,17.125000,17.000000 +40 0 414 10.565886 1124,220,12,370,59,654,17,75 16.750000,15.250000,14.562500,14.437500,14.187500,13.875000,13.375000,13.250000 +41 0 549 14.063318 1124,2533,921,1125,353,1318,60,273 21.250000,16.250000,15.500000,15.312500,14.500000,14.437500,14.312500,13.750000 +42 0 1273 11.282848 1124,353,1125,2533,1318,921,701,9533 16.625000,13.125000,12.937500,12.875000,12.562500,11.937500,11.687500,11.500000 +43 0 320 6.158167 1124,370,67,11,273,8,220,3226 14.562500,14.375000,12.437500,12.062500,11.812500,11.562500,11.500000,11.500000 +44 0 19 12.187044 1124,37018,370,2533,11520,1318,273,1291 23.125000,18.125000,18.000000,18.000000,18.000000,17.500000,17.375000,16.625000 +45 0 18 3.259611 13,15,9754,17,18,19,20,23 19.625000,18.375000,18.375000,18.250000,17.875000,17.875000,17.875000,17.875000 +46 0 20 7.971718 1124,2533,8,701,921,11520,11,273 17.875000,15.812500,15.687500,15.562500,15.000000,14.875000,14.562500,14.437500 +47 0 23 3.658564 13,9754,15,20,17,18,23,5373 19.375000,18.625000,18.500000,18.500000,18.125000,17.875000,17.500000,17.375000 +48 0 6978 9.611034 13,9754,23,15,17,18,19,24 19.000000,18.500000,18.250000,18.125000,17.875000,17.625000,17.500000,17.375000 +49 0 340 10.061897 1124,220,315,323,11,59,1125,516 18.000000,13.937500,12.875000,12.250000,12.062500,12.062500,12.000000,11.937500 +50 0 9827 17.977255 1124,654,370,3044,7370,15,220,85 21.375000,18.375000,17.750000,16.500000,16.375000,16.000000,15.937500,15.562500 +51 0 262 8.579953 25,198,315,12,85,323,271,9 13.187500,12.812500,12.625000,12.312500,12.312500,12.312500,12.250000,12.125000 +52 0 549 14.738266 1124,2533,921,1125,1318,273,709,11520 21.250000,15.812500,14.812500,14.437500,14.187500,14.062500,13.625000,13.625000 +53 0 13534 14.314098 1124,353,1125,2533,1318,921,701,9533 16.625000,13.125000,12.937500,12.875000,12.562500,11.937500,11.687500,11.500000 +54 0 7710 13.520891 1124,8,11,701,60,273,15,2533 15.625000,13.812500,13.437500,12.812500,12.750000,12.625000,12.375000,12.312500 +55 0 92214 13.010126 220,12,1124,62,3044,85,716,516 17.625000,17.000000,16.375000,15.375000,15.125000,15.062500,15.000000,14.875000 +56 0 220 2.203558 220,1124,15,17,16,12,85,3226 14.062500,14.062500,12.875000,12.875000,12.750000,12.562500,12.500000,12.125000 +57 0 18 8.525661 1124,370,2533,1125,15,921,17,16 21.125000,15.687500,15.125000,14.562500,14.250000,14.000000,13.750000,13.625000 +58 0 13 7.159218 1124,2533,8,701,921,11520,11,273 17.875000,15.812500,15.687500,15.562500,15.000000,14.875000,14.562500,14.437500 +59 0 15 11.005924 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.250000,16.000000,15.812500,15.187500,15.125000,14.687500 +60 0 320 8.641680 15,13,271,16,17,11,18,198 23.500000,19.250000,18.375000,18.250000,18.125000,18.000000,17.750000,17.625000 +61 0 2428 17.999544 1124,37018,370,2533,11520,1318,273,1291 23.125000,18.125000,18.000000,18.000000,18.000000,17.500000,17.375000,16.625000 +62 0 1110 0.002899 1110,4,25,14375,322,549,60896,23817 21.125000,13.687500,13.687500,12.312500,12.250000,12.000000,11.187500,11.000000 +63 0 85231 14.752295 1124,370,59,220,1699,85,15,12 17.625000,14.687500,14.500000,14.500000,14.187500,14.000000,13.812500,13.625000 +64 0 2659 1.401993 2659,12,905,9533,1125,60,17,25 16.125000,15.375000,14.937500,14.125000,13.750000,13.625000,13.562500,13.437500 +65 0 6971 10.389917 9533,14186,13,11,25,315,905,3470 12.625000,12.312500,12.125000,11.937500,11.750000,11.562500,11.562500,11.562500 +66 0 67733 9.299282 14,25815,3470,14186,12,315,2899,23303 14.562500,13.625000,13.500000,13.500000,13.437500,13.125000,13.000000,13.000000 +67 0 1331 5.236755 12,85,1124,71,15,67,73,86 17.500000,17.375000,17.375000,17.125000,17.000000,16.875000,16.750000,16.750000 +68 0 64 10.727615 1124,2533,701,11,8,273,1125,921 18.750000,16.500000,16.125000,15.875000,15.750000,15.500000,15.312500,15.125000 +69 0 14 8.641491 370,1124,284,668,220,16,17,62 21.125000,20.875000,19.000000,18.875000,18.375000,18.125000,17.500000,17.500000 +70 0 18 5.942450 1124,370,273,3226,15,709,17,5305 19.000000,15.875000,15.437500,14.687500,14.562500,14.562500,14.500000,14.375000 +71 0 13 7.159218 1124,2533,8,701,921,11520,11,273 17.875000,15.812500,15.687500,15.562500,15.000000,14.875000,14.562500,14.437500 +72 0 15 11.005924 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.250000,16.000000,15.812500,15.187500,15.125000,14.687500 +73 0 53560 16.985430 15,13,271,16,17,11,18,198 23.500000,19.250000,18.375000,18.250000,18.125000,18.000000,17.750000,17.625000 +74 0 19665 15.719305 370,12,220,1124,71,65,67,3226 16.375000,14.875000,14.500000,14.375000,13.812500,13.750000,13.750000,13.687500 +75 0 1488 9.182374 273,59,258,75,408,654,613,198 13.125000,12.687500,12.187500,11.937500,11.875000,11.687500,11.500000,11.437500 +76 0 25 5.719521 370,1124,668,220,315,516,323,11 16.625000,15.000000,13.937500,13.875000,13.500000,13.437500,13.000000,12.875000 +77 0 26587 18.851065 1124,370,273,2533,1125,1318,15,37018 23.125000,17.000000,14.875000,14.812500,14.687500,14.500000,14.312500,14.187500 +78 0 20343 10.319180 220,13,15,9533,25,12,905,516 13.500000,13.125000,12.875000,12.812500,12.687500,12.625000,12.500000,12.187500 +79 0 11 5.082237 1124,1699,497,3808,59,5305,9533,25 15.937500,12.812500,12.437500,12.437500,12.000000,12.000000,11.937500,11.875000 +80 0 80935 23.089275 1124,2533,1125,1318,701,709,37018,370 22.250000,17.000000,16.500000,16.125000,15.750000,15.375000,15.312500,15.250000 +81 0 15235 13.203253 370,1124,220,3226,17,13,16,25 17.125000,16.750000,15.187500,14.750000,14.562500,14.500000,14.312500,14.187500 +82 0 8319 14.143055 1124,13,11,59,25,85,12,62 19.750000,14.625000,14.500000,14.500000,14.125000,14.062500,13.812500,13.687500 +83 0 198 7.454596 1124,7110,9533,13,11,60,370,220 17.000000,14.125000,13.937500,13.562500,13.437500,13.250000,13.125000,12.812500 +84 0 1797 14.494125 1124,370,654,1699,59,3044,516,3422 24.750000,21.250000,20.375000,18.500000,17.875000,17.375000,17.125000,17.000000 +85 0 320 8.430004 1124,353,220,1125,3044,15,1154,8358 17.750000,16.750000,13.812500,13.375000,13.250000,12.562500,12.187500,12.187500 +86 0 50 13.374544 1124,37018,370,2533,11520,1318,273,1291 23.125000,18.125000,18.000000,18.000000,18.000000,17.500000,17.375000,16.625000 +87 0 13 5.118158 1124,370,11,8,15,67,17,25 18.125000,16.000000,14.812500,14.375000,14.187500,14.125000,14.062500,14.062500 +88 0 8755 19.380924 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.250000,16.000000,15.812500,15.187500,15.125000,14.687500 +89 0 487 13.766680 1124,11520,2533,921,408,273,1318,709 19.625000,18.250000,17.375000,16.625000,16.500000,15.812500,15.687500,15.562500 +90 0 11 3.918764 315,1124,220,323,516,11,304,13 15.312500,14.375000,14.062500,14.000000,13.562500,13.312500,13.187500,13.000000 +91 0 451 13.155682 1124,2533,1125,1318,701,709,37018,370 22.250000,17.000000,16.500000,16.125000,15.750000,15.375000,15.312500,15.250000 +92 0 13 7.322989 1124,8,11,11520,921,60,701,59 19.250000,14.437500,14.187500,14.062500,13.562500,13.500000,13.250000,12.875000 +93 0 49256 22.560612 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.250000,16.000000,15.812500,15.187500,15.125000,14.687500 +94 0 812 13.328945 1124,3226,370,11520,4260,408,273,277 23.375000,18.125000,17.500000,16.625000,15.687500,15.562500,15.375000,15.250000 +95 0 82977 18.199749 370,1124,3226,4260,67,408,273,5305 20.250000,19.625000,16.125000,14.312500,14.187500,14.187500,14.062500,13.812500 +96 0 28134 12.839138 1124,3226,370,71,59,4260,273,277 22.250000,16.500000,15.875000,15.562500,15.500000,15.437500,15.375000,14.875000 +97 0 11 7.890968 1124,370,11520,654,273,408,2533,277 21.750000,19.250000,17.625000,17.250000,16.125000,15.687500,15.437500,15.312500 +98 0 431 13.218182 1124,2533,1125,1318,701,709,37018,370 22.250000,17.000000,16.500000,16.125000,15.750000,15.375000,15.312500,15.250000 +99 0 13 8.462234 1124,370,11520,408,2533,37018,921,273 19.000000,17.000000,16.875000,16.625000,15.812500,15.500000,15.250000,15.187500 +100 0 2055 18.912174 1124,2533,370,11520,1125,921,1318,709 23.500000,16.500000,16.250000,16.000000,15.812500,15.187500,15.125000,14.687500 +101 0 9034 20.498520 1124,370,11520,11,654,2533,273,37018 22.500000,15.937500,15.625000,15.375000,14.937500,14.937500,14.875000,14.812500 +102 0 1959 15.712936 1124,3226,273,408,4260,11,71,709 19.750000,16.625000,15.687500,15.500000,15.187500,15.125000,14.812500,14.750000 +103 0 330 7.187007 220,1154,323,370,1124,11,481,701 10.937500,10.687500,9.812500,9.687500,9.500000,9.437500,9.437500,9.375000 +104 0 9084 20.603432 1124,370,1318,2533,37018,709,1125,11520 22.500000,17.375000,17.000000,16.500000,16.125000,15.812500,15.812500,15.750000 +105 0 56793 17.905731 353,9,1124,220,12,59,323,1699 18.500000,15.500000,15.312500,15.187500,14.375000,14.062500,13.750000,13.562500 +106 0 386 5.930481 220,12,15,16,17,353,516,1124 14.625000,13.562500,13.500000,12.937500,12.937500,12.687500,12.687500,12.437500 +107 0 12735 21.650307 1124,11520,8,11,921,701,2533,60 20.875000,15.562500,14.750000,14.250000,14.000000,13.875000,13.750000,13.562500 +108 0 26874 9.698305 315,1124,220,17,370,62,323,7370 16.000000,15.562500,15.312500,15.062500,15.000000,14.687500,14.625000,14.625000 +109 0 497 5.605358 315,1124,220,323,13,11,304,25 14.437500,13.937500,13.875000,13.812500,13.500000,13.125000,12.937500,12.687500 +110 0 220 4.768656 1124,1125,2533,6878,220,716,15,1318 19.750000,16.125000,15.375000,15.375000,15.187500,14.687500,14.625000,14.625000 +111 0 17 7.400661 1124,370,2533,1125,15,921,17,16 21.125000,15.687500,15.125000,14.562500,14.250000,14.000000,13.750000,13.625000 +112 0 15 5.178099 1124,11520,2533,8,921,701,568,60 17.625000,15.875000,15.812500,15.687500,15.625000,15.000000,14.062500,14.000000 +113 0 16 5.329180 15,13,271,16,17,11,18,198 23.500000,19.250000,18.375000,18.250000,18.125000,18.000000,17.750000,17.625000 +114 0 21 4.409811 13,16,17,15,24,5373,18,20 21.000000,20.000000,19.375000,19.250000,18.875000,18.875000,18.750000,18.250000 +115 0 340 6.839550 13,21,9754,15,17,18,19,23 20.000000,19.500000,19.500000,19.000000,19.000000,18.875000,18.750000,18.625000 +116 0 1797 11.617880 1124,654,370,3044,7370,15,220,85 21.375000,18.375000,17.750000,16.500000,16.375000,16.000000,15.937500,15.562500 +117 0 39183 16.914379 1124,353,220,1125,3044,15,1154,8358 17.750000,16.750000,13.812500,13.375000,13.250000,12.562500,12.187500,12.187500 +118 0 3701 10.475697 1124,220,15,17,13,315,370,16 14.875000,14.500000,13.375000,13.187500,12.937500,12.937500,12.937500,12.687500 +119 0 25 7.891834 1124,370,668,353,59,220,516,323 18.875000,17.875000,15.812500,14.500000,13.875000,13.750000,13.750000,13.250000 +120 0 3703 16.226065 1124,370,273,2533,1125,1318,15,37018 23.125000,17.000000,14.875000,14.812500,14.687500,14.500000,14.312500,14.187500 +121 0 1110 9.645788 1124,59,220,25,11,13,16,17 16.625000,12.625000,12.375000,12.250000,12.000000,12.000000,12.000000,12.000000 +122 0 71 5.049170 1124,370,59,220,1699,85,15,12 17.625000,14.687500,14.500000,14.500000,14.187500,14.000000,13.812500,13.625000 +123 0 35268 15.400644 1124,11520,2533,921,8,37018,408,273 23.125000,19.125000,15.937500,15.562500,15.500000,15.187500,15.000000,14.937500 +124 0 1564 8.833986 1124,220,71,304,709,370,516,3422 16.500000,14.687500,14.125000,13.187500,13.187500,13.062500,12.937500,12.937500 +125 0 6830 13.126095 1124,370,654,668,516,59,220,284 20.750000,17.875000,16.875000,16.750000,16.625000,16.250000,16.250000,16.000000 +126 0 3446 5.952713 1124,59,346,60,273,370,9533,17 19.000000,17.250000,16.000000,15.937500,15.812500,15.750000,15.750000,15.562500 +127 0 76649 14.275181 8,1124,921,1318,408,60,701,11 15.437500,15.437500,14.687500,14.625000,14.562500,14.312500,14.250000,14.187500 diff --git a/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w8.tsv b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w8.tsv new file mode 100644 index 000000000..b08a65860 --- /dev/null +++ b/benchmarks/logit_traces/rocm_gfx1151_77afbb1d/rocm_gfx1151_77afbb1d_qwen3-30b-a3b_fused0_w8.tsv @@ -0,0 +1,647 @@ +# model models/mlx/Qwen3-30B-A3B-4bit +# device GPU +# corpus tests/fixtures/wikitext2_excerpt.txt tokens 299190 +# chunks 80 chunk_tokens 8 topk 8 prefill 512 +# columns chunk pos target nll top_ids top_logits +# start_token 0 +# contiguous false +0 0 42105 15.901297 1124,370,273,37018,2533,709,1318,654 20.875000,18.750000,16.375000,15.562500,15.437500,15.187500,15.125000,15.062500 +0 1 275 5.916378 285,685,83964,27391,81080,10980,579,307 17.375000,17.375000,16.500000,15.875000,15.187500,14.812500,14.625000,14.312500 +0 2 427 0.002298 427,644,765,779,706,29551,1897,327 24.875000,17.625000,16.875000,16.125000,15.875000,15.562500,15.187500,14.937500 +0 3 12 5.890383 369,47761,25,311,320,4035,198,374 18.375000,18.375000,17.875000,17.750000,17.750000,17.750000,17.625000,17.500000 +0 4 17 1.253744 17,16,18,19,15,6570,20,5097 17.875000,17.375000,16.250000,16.000000,15.812500,15.687500,15.000000,14.562500 +0 5 3393 6.649996 320,10337,198,75734,12,39183,15,42094 20.125000,19.250000,18.875000,18.625000,18.500000,18.500000,18.375000,18.000000 +0 6 75734 2.964243 2885,220,320,2573,4035,75734,39183,25 19.625000,19.250000,19.000000,19.000000,19.000000,19.000000,18.875000,18.125000 +0 7 79483 3.072130 320,4035,198,79483,25,271,348,284 21.500000,20.375000,19.875000,19.500000,19.375000,18.750000,18.750000,18.500000 +1 0 3608 8.997362 785,1986,418,58,15505,54,2,32 23.375000,20.625000,19.625000,19.125000,19.125000,19.000000,18.750000,18.375000 +1 1 257 11.779507 25,510,1467,198,320,323,1447,3542 26.375000,22.375000,22.375000,21.625000,21.625000,21.500000,21.500000,21.250000 +1 2 549 0.197664 549,760,284,508,320,481,3703,8478 26.250000,24.375000,22.500000,20.625000,20.500000,20.250000,20.000000,19.250000 +1 3 3703 1.033129 3703,508,26587,576,1758,220,256,662 22.250000,21.250000,21.250000,20.750000,20.125000,20.000000,19.625000,19.000000 +1 4 1110 0.000206 1110,25,549,14375,322,198,510,4 25.625000,16.000000,15.125000,14.687500,14.562500,14.250000,13.812500,13.312500 +1 5 71 0.589958 71,2136,82,11659,2911,1041,5204,268 23.625000,22.875000,21.375000,20.750000,20.250000,20.000000,19.875000,19.500000 +1 6 35268 0.000048 35268,2596,13619,23518,768,12,7876,4076 30.125000,19.250000,19.000000,17.125000,16.750000,16.625000,16.625000,16.375000 +1 7 1564 0.000019 1564,3579,18596,29088,16281,30985,46031,22185 34.000000,22.750000,21.250000,19.875000,18.875000,18.250000,17.500000,17.250000 +2 0 6830 0.000004 6830,71433,11021,29425,1858,35883,18798,57409 32.000000,17.375000,16.750000,16.500000,16.250000,16.250000,15.750000,15.437500 +2 1 3446 0.000080 3446,1737,14,14953,25967,75988,29425,3183 31.375000,21.250000,20.375000,19.625000,19.500000,17.500000,17.375000,17.000000 +2 2 76649 0.000011 76649,8369,4917,2096,4886,746,82,29425 37.000000,25.625000,20.500000,19.375000,19.375000,19.250000,18.375000,18.250000 +2 3 11374 9.202957 6324,4846,14,22175,15792,25502,23354,90936 29.375000,26.000000,25.625000,24.875000,23.250000,23.000000,22.875000,22.375000 +2 4 3831 0.422424 3831,309,3288,54,1466,74,306,3466 24.375000,23.375000,20.750000,20.500000,20.375000,19.500000,19.500000,19.250000 +2 5 8833 0.004412 8833,18573,48796,82010,824,49099,2566,10458 29.625000,23.500000,22.125000,22.000000,21.000000,20.875000,20.125000,20.000000 +2 6 6324 0.051945 6324,22175,14,2899,10776,46928,2687,4846 33.000000,30.000000,27.250000,24.250000,23.500000,22.125000,21.625000,21.375000 +2 7 60403 0.011055 60403,1579,14939,417,17,18677,2963,90669 33.250000,28.750000,21.000000,18.875000,18.625000,17.875000,17.625000,17.625000 +3 0 427 0.000460 427,38649,477,327,4788,480,2252,1002 28.375000,19.875000,19.750000,17.500000,17.500000,17.125000,16.000000,15.875000 +3 1 198 0.694378 198,271,12,320,17,715,34827,11 25.750000,25.500000,23.000000,23.000000,22.250000,21.125000,21.125000,20.875000 +3 2 70584 8.182957 13806,1712,5009,6262,929,36930,33363,34613 20.000000,19.875000,19.875000,19.875000,19.250000,19.250000,19.125000,19.125000 +3 3 257 0.018347 257,549,414,256,262,25,315,981 27.750000,22.000000,21.500000,21.250000,21.250000,21.125000,21.125000,20.875000 +3 4 549 0.000038 549,6260,760,30372,937,284,14512,481 32.500000,21.625000,20.375000,20.250000,19.375000,19.250000,18.250000,17.875000 +3 5 289 0.155863 289,330,97175,364,28609,1273,42105,220 28.250000,25.625000,25.250000,24.375000,23.000000,22.875000,22.625000,22.125000 +3 6 60403 -0.000000 60403,14628,417,31580,5840,96363,2501,5566 35.250000,19.125000,17.625000,17.250000,15.875000,15.875000,15.687500,15.125000 +3 7 427 0.000015 427,2252,477,327,5722,1303,38649,1318 38.500000,27.250000,24.625000,21.625000,20.125000,19.750000,19.750000,18.375000 +4 0 12 0.000099 12,16839,481,17,198,1273,1944,50342 35.000000,25.625000,22.750000,22.375000,21.625000,21.000000,20.750000,20.750000 +4 1 17 0.000027 17,16,1041,4266,15,18,21037,19 36.500000,26.000000,21.125000,19.375000,18.875000,17.750000,17.750000,17.000000 +4 2 12 0.257523 12,198,8273,271,2852,320,715,16839 26.500000,24.875000,23.875000,21.750000,20.875000,20.625000,20.625000,20.375000 +4 3 1041 0.000296 1041,7112,21550,586,5761,39884,17,9199 30.625000,21.625000,20.625000,19.500000,19.375000,19.125000,18.750000,18.750000 +4 4 8273 0.001163 8273,198,271,98056,12,2268,13333,25502 25.750000,18.250000,16.250000,15.875000,15.750000,15.750000,15.750000,15.375000 +4 5 16 0.000008 16,17,15,18,449,1272,59854,275 29.625000,17.375000,15.812500,14.750000,14.750000,14.437500,13.375000,13.062500 +4 6 198 0.031952 198,271,320,13,715,2303,11,1406 29.375000,25.500000,24.125000,23.375000,23.125000,22.500000,20.250000,20.125000 +4 7 20193 2.131334 13806,20193,6262,5009,34613,5637,2833,59286 21.750000,20.375000,19.625000,19.375000,19.375000,19.125000,18.875000,18.125000 +5 0 414 0.020603 414,257,981,549,262,286,256,260 27.250000,22.750000,21.875000,20.625000,20.125000,19.750000,19.500000,19.375000 +5 1 549 0.000010 549,760,320,30372,284,6260,937,3504 31.625000,18.500000,18.250000,18.125000,18.000000,17.625000,17.250000,16.375000 +5 2 1273 1.922794 5426,1273,10519,330,364,4862,2697,1044 29.875000,28.125000,24.500000,23.875000,23.750000,23.750000,23.500000,22.125000 +5 3 320 5.180874 198,271,320,715,11,2303,1406,4710 29.500000,26.500000,24.375000,21.875000,21.250000,21.125000,20.750000,20.000000 +5 4 19 3.892439 16,17,18,20,19,22,23,24 24.125000,22.625000,22.250000,21.375000,21.000000,20.750000,20.750000,20.750000 +5 5 18 2.721973 15,17,20,11,16,18,19,23 27.250000,26.000000,25.750000,25.625000,25.625000,25.625000,25.625000,25.500000 +5 6 20 2.766611 11,17,15,18,13,19,23,21 26.125000,25.750000,25.500000,25.375000,25.250000,25.250000,25.250000,25.125000 +5 7 23 2.919186 11,17,15,18,21,19,16,20 25.500000,25.375000,25.250000,25.125000,25.000000,24.875000,24.625000,24.625000 +6 0 6978 7.336081 10295,15,17,18,21,19,16,20 23.875000,23.625000,23.625000,23.625000,23.625000,23.500000,23.375000,23.250000 +6 1 340 0.515198 340,692,11,8,304,701,608,2790 29.375000,28.750000,26.625000,26.250000,24.375000,23.875000,23.375000,23.375000 +6 2 9827 6.201912 13806,5009,13965,3323,6262,4061,7595,2505 20.375000,19.500000,19.250000,19.125000,19.000000,18.625000,18.500000,18.375000 +6 3 262 0.006216 262,257,25,549,256,414,220,981 28.125000,22.250000,21.125000,21.000000,20.750000,19.625000,19.375000,19.000000 +6 4 549 0.000072 549,6260,14512,937,320,30372,220,481 32.500000,22.375000,21.375000,20.875000,20.000000,19.250000,18.125000,18.125000 +6 5 13534 1.780506 14865,13534,8914,24263,576,12527,5264,21693 25.250000,24.000000,23.875000,22.500000,21.000000,20.875000,20.500000,20.250000 +6 6 7710 1.858629 7671,7710,15,12,220,58318,14319,92214 24.125000,22.500000,21.250000,18.875000,17.625000,17.250000,17.000000,15.625000 +6 7 92214 1.338308 12,92214,220,16589,20059,320,7778,95942 24.500000,23.625000,22.625000,19.875000,19.375000,17.875000,17.000000,16.125000 +7 0 220 0.077545 220,198,320,271,12,323,11,348 27.000000,23.750000,23.125000,22.750000,21.500000,19.125000,18.750000,18.375000 +7 1 18 1.146996 19,18,17,16,15,20,220,320 28.875000,28.125000,24.625000,21.500000,18.500000,17.875000,17.875000,17.125000 +7 2 13 0.000019 13,198,271,320,15,11,624,323 30.750000,18.625000,17.875000,17.750000,17.375000,17.000000,16.750000,16.375000 +7 3 15 -0.000000 15,16,17,20,18,19,21,24 35.750000,21.250000,19.625000,18.500000,18.250000,18.125000,17.375000,17.000000 +7 4 320 2.209612 198,271,320,323,476,11,715,1876 26.625000,25.000000,24.750000,22.125000,21.625000,21.500000,20.750000,20.750000 +7 5 2428 0.779636 2428,4060,300,1958,2427,4197,62946,1782 22.750000,20.500000,20.125000,19.875000,19.875000,19.750000,19.750000,19.625000 +7 6 1110 0.000168 1110,25,14375,549,340,692,322,486 25.125000,15.562500,14.500000,13.437500,13.375000,13.250000,12.875000,12.812500 +7 7 85231 0.406576 85231,268,2136,71,5204,2154,67,79 22.750000,21.250000,20.625000,20.125000,19.375000,18.375000,17.250000,17.125000 +8 0 2659 0.000004 2659,5713,13,61501,8382,11021,24282,53834 31.875000,18.375000,17.625000,16.375000,15.187500,15.125000,15.125000,15.062500 +8 1 6971 0.000553 6971,43986,14,10845,53560,6580,33914,340 27.125000,18.375000,17.500000,17.500000,17.250000,17.000000,16.625000,16.500000 +8 2 67733 0.000034 67733,14,87310,553,340,692,14319,2687 35.250000,24.375000,23.250000,22.375000,21.625000,21.500000,21.500000,21.125000 +8 3 1331 0.000420 1331,14,12,2687,92214,5894,1112,2268 32.500000,24.625000,21.875000,21.000000,20.250000,17.125000,16.250000,15.812500 +8 4 64 0.000004 64,32,14,309,14186,264,1478,1112 36.250000,23.000000,21.375000,18.375000,17.625000,17.500000,17.500000,17.375000 +8 5 14 0.000021 14,53560,50807,5894,33149,340,8680,692 29.625000,18.000000,17.125000,16.875000,16.500000,15.375000,14.937500,14.375000 +8 6 18 0.003857 18,17,19,16,1112,31218,61827,692 34.000000,27.750000,27.750000,20.375000,18.250000,16.500000,16.500000,16.250000 +8 7 13 0.000004 13,14,1993,94329,22445,15,4338,1725 36.000000,22.500000,21.500000,20.000000,18.625000,18.500000,18.000000,17.500000 +9 0 15 -0.000000 15,16,20,24,18,17,21,19 32.000000,17.250000,15.500000,15.500000,14.125000,13.250000,12.375000,12.187500 +9 1 53560 0.700508 53560,14,50807,340,47036,692,22549,34545 24.750000,24.500000,23.000000,21.125000,20.250000,19.875000,19.500000,19.125000 +9 2 19665 8.808187 5009,13806,8856,1043,6262,5637,1695,11377 20.625000,19.750000,18.375000,18.250000,18.250000,17.375000,17.125000,17.125000 +9 3 1488 0.002705 1488,3977,2774,1070,262,22705,13,332 25.875000,18.625000,18.250000,18.125000,17.250000,17.250000,17.000000,17.000000 +9 4 25 0.001860 25,549,510,1447,91,5293,1995,3546 28.500000,21.250000,19.875000,19.875000,18.875000,18.875000,18.125000,18.125000 +9 5 26587 4.281845 576,1096,3703,330,5209,467,80935,89840 21.000000,20.750000,20.250000,19.750000,19.375000,19.000000,18.875000,18.875000 +9 6 20343 4.534290 320,323,374,11,198,10337,271,594 22.625000,22.625000,22.375000,21.875000,21.750000,21.500000,21.375000,21.125000 +9 7 11 2.217674 271,13,198,11,320,382,26,624 27.625000,27.375000,27.375000,26.625000,26.625000,24.375000,23.125000,23.000000 +10 0 80935 9.615000 330,220,1036,364,323,508,714,89840 24.500000,22.250000,21.625000,20.625000,20.375000,20.375000,20.375000,20.375000 +10 1 15235 3.177839 198,271,11,8319,4848,15235,320,382 22.875000,22.875000,22.375000,22.375000,21.625000,21.125000,20.875000,20.625000 +10 2 8319 0.035326 8319,271,3412,320,11,198,1800,2083 27.000000,22.000000,22.000000,21.875000,21.750000,21.625000,20.500000,20.375000 +10 3 198 1.709133 271,198,320,11,382,13,1406,2083 27.000000,25.750000,24.875000,24.625000,23.500000,22.750000,22.625000,22.125000 +10 4 1797 7.400085 5009,13806,198,8856,6262,1043,1695,2833 20.500000,18.875000,18.625000,18.625000,18.250000,18.000000,17.625000,17.625000 +10 5 320 0.259911 320,3703,323,508,330,198,481,366 24.000000,22.250000,20.125000,19.875000,19.500000,18.875000,18.500000,18.375000 +10 6 50 8.674345 2428,4060,9830,785,1254,17,3703,1958 25.625000,20.875000,19.875000,19.375000,19.375000,19.250000,19.250000,19.000000 +10 7 13 0.231306 13,1466,36544,524,6128,10080,3779,6383 20.750000,17.625000,17.125000,16.750000,16.375000,16.250000,15.937500,15.875000 +11 0 8755 7.757391 431,425,73580,444,479,36020,468,87900 21.875000,21.000000,18.625000,18.375000,18.250000,18.250000,17.750000,17.750000 +11 1 487 0.003477 487,75,2352,385,31245,291,12212,28600 26.125000,18.875000,18.375000,17.750000,17.250000,17.125000,17.000000,16.875000 +11 2 11 0.038940 11,1842,323,13,26,609,701,320 30.000000,26.750000,22.375000,20.375000,20.125000,20.000000,19.875000,19.750000 +11 3 451 0.747910 451,1842,362,356,431,328,422,468 27.750000,27.000000,26.125000,25.875000,25.500000,25.250000,24.250000,23.875000 +11 4 13 0.000051 13,1785,2572,875,808,10003,11,624 31.375000,21.000000,19.375000,18.875000,18.750000,18.375000,17.375000,16.750000 +11 5 49256 11.108501 425,14695,328,90019,1863,38979,11840,96160 18.625000,16.125000,15.562500,15.250000,15.125000,15.125000,14.812500,14.625000 +11 6 812 0.662867 812,5559,585,276,12466,19042,6190,88 22.500000,22.375000,18.125000,17.375000,17.250000,17.125000,17.000000,16.875000 +11 7 82977 3.781088 11,82977,1842,323,38767,13,328,2932 22.125000,18.375000,15.187500,14.687500,14.687500,14.437500,14.375000,14.375000 +12 0 28134 0.001785 28134,404,838,5840,8698,86,261,5059 27.000000,19.375000,19.375000,19.250000,18.000000,16.500000,16.500000,16.375000 +12 1 11 0.002996 11,323,1842,609,13,320,340,345 30.375000,24.125000,23.250000,20.500000,20.375000,19.750000,19.750000,19.375000 +12 2 431 0.626019 431,323,1842,444,451,425,479,386 27.125000,26.750000,23.750000,23.500000,23.500000,23.375000,23.375000,22.750000 +12 3 13 0.000015 13,3503,836,5142,808,875,11,11352 34.000000,21.625000,20.875000,20.750000,20.375000,20.375000,20.125000,19.500000 +12 4 2055 2.167864 1863,2055,619,431,328,74787,25867,730 17.125000,16.750000,16.375000,15.812500,15.687500,15.687500,15.625000,15.562500 +12 5 9034 0.000803 9034,11437,331,5403,71,28064,9917,20658 27.750000,20.125000,19.250000,17.000000,16.250000,16.125000,15.500000,15.437500 +12 6 1959 9.414848 11,692,13,340,320,8,26,701 28.000000,28.000000,26.875000,26.375000,26.000000,25.500000,24.000000,23.875000 +12 7 330 2.531332 220,3703,330,1490,369,1036,279,5813 21.375000,21.000000,20.125000,19.875000,19.125000,18.875000,18.125000,18.000000 +13 0 9084 8.570395 46358,2082,8813,32,12092,49,785,33876 23.625000,21.500000,21.375000,21.250000,20.750000,20.375000,20.375000,20.250000 +13 1 56793 1.776150 38710,56793,14155,80778,12,8141,23361,198 22.875000,21.375000,19.750000,19.125000,17.375000,17.375000,17.000000,16.500000 +13 2 386 1.104485 19219,386,27343,198,85471,18977,7487,34098 21.875000,21.750000,20.125000,19.750000,19.125000,18.625000,18.500000,18.500000 +13 3 12735 0.047470 12735,51525,95296,2375,17020,13470,10298,31102 24.000000,20.625000,19.500000,17.250000,15.875000,15.562500,14.937500,14.937500 +13 4 26874 0.000389 26874,4903,198,1,220,38710,715,4119 31.875000,23.375000,22.250000,21.000000,21.000000,20.875000,20.750000,20.625000 +13 5 497 1.328856 1,497,5130,1138,899,698,1335,3263 27.875000,27.125000,25.750000,25.625000,23.875000,23.375000,22.875000,22.875000 +13 6 220 0.160736 220,796,3703,358,1644,715,330,19288 24.625000,22.125000,21.000000,20.625000,19.625000,19.125000,18.500000,18.500000 +13 7 17 0.000729 17,16,18,220,3703,19,796,20 30.875000,22.875000,21.500000,21.250000,20.375000,20.125000,19.875000,19.750000 +14 0 15 0.000013 15,303,16,18,17,23,19,24 30.000000,17.750000,17.000000,16.875000,16.500000,15.812500,15.500000,15.187500 +14 1 16 0.018217 16,17,15,4146,18,19,339,6148 29.250000,25.250000,19.625000,14.000000,13.875000,13.812500,13.812500,13.812500 +14 2 21 0.828245 21,23,22,24,20,15,19,18 25.750000,25.500000,24.875000,23.375000,18.250000,15.187500,14.687500,14.312500 +14 3 340 2.651638 692,340,11,9272,8,1959,13,3593 28.500000,26.000000,24.500000,24.375000,23.875000,23.625000,23.500000,23.125000 +14 4 1797 1.068933 1797,5009,715,3374,6262,13806,36930,1060 21.375000,20.625000,18.750000,18.750000,18.750000,18.375000,18.250000,18.125000 +14 5 39183 8.501850 320,3703,323,508,330,1959,3496,576 24.750000,22.750000,21.625000,20.375000,20.125000,19.625000,19.250000,19.000000 +14 6 3701 9.062365 25,4008,5548,2319,374,3465,3565,7662 21.875000,20.750000,20.750000,20.500000,20.250000,20.250000,19.625000,19.625000 +14 7 25 0.418365 25,320,323,518,369,549,271,198 28.000000,26.500000,25.125000,25.000000,24.500000,24.500000,24.375000,24.250000 +15 0 3703 0.018906 3703,320,330,508,220,305,366,715 27.500000,22.125000,21.500000,21.375000,21.125000,21.125000,20.250000,20.125000 +15 1 1110 0.000086 1110,25,14375,60896,322,549,486,14111 26.875000,16.375000,16.125000,14.875000,14.812500,14.750000,14.000000,14.000000 +15 2 71 0.001699 71,44754,5204,4951,2136,65546,4489,47293 30.000000,23.125000,22.250000,20.625000,20.500000,18.625000,17.875000,17.750000 +15 3 35268 0.000023 35268,2596,768,45124,13299,20146,723,17 36.000000,25.250000,20.000000,18.875000,18.375000,17.000000,16.250000,16.125000 +15 4 1564 -0.000000 1564,22185,7605,46031,3579,16281,29088,18596 38.750000,20.875000,20.750000,20.125000,20.000000,19.375000,18.250000,17.750000 +15 5 6830 -0.000000 6830,11021,8639,71433,15593,22057,4245,52114 33.000000,18.875000,17.250000,17.250000,16.750000,16.625000,16.500000,16.125000 +15 6 3446 0.000027 3446,25967,14,37462,11374,1737,2687,20268 34.750000,23.875000,23.000000,19.375000,19.250000,18.250000,18.250000,17.750000 +15 7 76649 0.000004 76649,8369,4917,2096,4886,746,19346,2032 36.000000,23.375000,21.750000,16.750000,16.750000,16.625000,15.687500,15.125000 +16 0 11374 0.001936 11374,6324,2687,14,22175,3446,3183,25502 31.375000,24.375000,24.125000,22.875000,21.125000,20.375000,19.375000,19.375000 +16 1 3831 -0.000000 3831,68130,1574,15829,35418,13,29041,95929 42.500000,23.375000,22.625000,21.125000,19.750000,19.375000,17.000000,16.750000 +16 2 8833 0.000011 8833,824,1324,1958,627,18573,1192,48796 33.500000,21.500000,20.500000,19.250000,18.500000,18.375000,18.250000,18.125000 +16 3 6324 -0.000000 6324,14,22175,25502,198,271,5894,15889 37.250000,22.750000,21.875000,20.875000,20.750000,20.125000,20.125000,19.250000 +16 4 60403 0.000004 60403,31580,1579,97401,96363,275,7570,8226 37.000000,24.500000,23.875000,19.125000,18.250000,18.125000,18.125000,17.875000 +16 5 427 -0.000000 427,327,37220,2252,1303,38649,5722,477 40.500000,24.500000,24.000000,23.875000,23.375000,22.375000,21.375000,20.750000 +16 6 271 0.190680 271,198,4710,1406,2,715,382,26809 27.750000,26.125000,22.125000,21.875000,21.750000,20.375000,20.250000,20.250000 +16 7 33236 13.025576 785,418,28,1986,5009,565,2,9112 21.750000,20.125000,19.625000,19.625000,18.500000,18.250000,18.125000,18.000000 +17 0 414 1.366865 414,25,315,549,981,320,257,369 22.375000,21.250000,20.750000,20.750000,20.750000,20.375000,20.125000,20.125000 +17 1 549 0.000242 549,6260,760,320,284,30372,14512,481 30.000000,21.000000,19.375000,19.125000,18.500000,18.375000,18.125000,17.625000 +17 2 2480 5.025223 220,330,1273,576,320,508,289,1156 21.750000,21.500000,21.500000,21.250000,20.750000,20.500000,20.375000,20.250000 +17 3 1273 5.296442 320,198,1467,9345,4326,22870,271,1318 21.375000,21.000000,20.750000,20.125000,19.500000,19.375000,18.750000,17.750000 +17 4 6718 5.017664 738,320,198,10337,271,1467,6718,42094 26.750000,24.750000,24.250000,22.750000,22.375000,22.375000,22.000000,21.875000 +17 5 320 1.050947 198,320,271,11,315,448,369,438 29.000000,28.750000,27.750000,26.000000,25.500000,25.500000,24.125000,24.125000 +17 6 541 2.651321 19,2152,541,300,1921,16,28996,1811 24.000000,23.750000,22.500000,21.625000,21.500000,21.375000,21.375000,20.875000 +17 7 220 2.286644 6978,220,10295,821,10469,1467,22870,11211 24.000000,22.875000,22.875000,22.750000,22.375000,22.000000,22.000000,21.875000 +18 0 19 0.014612 19,16,18,17,20,23,21,22 29.500000,24.000000,23.875000,23.375000,23.250000,23.000000,22.125000,21.500000 +18 1 18 0.007820 18,11,74,17,15,13,16,19 29.750000,24.250000,23.375000,22.250000,22.125000,21.125000,21.125000,21.125000 +18 2 20 0.000673 20,23,19,21,17,18,22,15 30.625000,22.000000,21.625000,21.625000,21.125000,21.000000,20.500000,20.125000 +18 3 23 0.000231 23,22,24,21,17,18,20,19 30.500000,21.375000,21.000000,19.250000,18.625000,18.625000,18.625000,18.375000 +18 4 6978 1.028637 6978,10295,10469,22870,5128,10695,13121,9293 24.500000,24.000000,23.750000,23.000000,22.125000,21.750000,21.375000,21.250000 +18 5 97534 14.308701 340,692,8,315,701,11,304,568 28.750000,28.750000,27.000000,25.750000,25.750000,25.375000,25.250000,24.125000 +18 6 340 1.634146 340,448,1119,692,438,304,11,323 27.375000,27.250000,27.250000,27.000000,26.500000,26.250000,25.875000,25.625000 +18 7 38 8.317982 1972,4373,13806,54615,3323,4061,929,5009 19.125000,19.125000,19.125000,19.000000,18.750000,18.500000,18.375000,18.375000 +19 0 2828 0.662407 2828,9379,417,380,1684,2537,5372,44433 21.125000,20.750000,18.375000,17.625000,17.375000,17.250000,16.250000,16.125000 +19 1 12 0.115736 12,9934,59501,17,18,19,12009,12681 24.375000,20.375000,20.125000,20.000000,19.625000,19.625000,19.500000,18.500000 +19 2 17 1.392046 18,19,17,16,97857,4416,20,1552 24.500000,24.000000,23.875000,19.000000,18.500000,16.750000,16.500000,16.500000 +19 3 425 7.542877 414,9934,59501,320,1273,1614,2331,5603 20.750000,20.750000,20.000000,19.375000,18.125000,18.125000,18.000000,17.875000 +19 4 1740 0.005785 1740,3126,375,2828,3399,18454,2264,19701 25.750000,19.375000,19.250000,18.750000,18.375000,17.500000,16.250000,16.250000 +19 5 11211 4.009090 220,3950,256,45958,320,9660,262,549 22.125000,22.125000,21.875000,21.250000,21.125000,21.125000,20.375000,20.375000 +19 6 320 0.932634 320,25,549,448,198,11,510,817 26.750000,26.375000,26.125000,24.125000,24.000000,23.250000,22.625000,22.625000 +19 7 83 8.619921 20,18,19,1499,4197,2428,86847,2528 22.750000,21.500000,21.500000,21.250000,21.125000,20.875000,20.750000,20.625000 +20 0 1579 0.596680 1579,20,28,724,38007,2850,48883,12 22.625000,22.375000,18.250000,17.625000,16.750000,16.250000,16.125000,16.000000 +20 1 5839 0.043171 5839,89,30566,85593,57,11687,12738,34841 31.000000,27.875000,21.375000,21.375000,19.375000,19.125000,18.875000,18.750000 +20 2 1648 1.284651 8,1648,11170,65118,340,11,982,701 22.875000,22.375000,21.625000,20.250000,19.875000,19.500000,19.250000,18.375000 +20 3 220 0.288483 220,3968,3703,508,9848,320,715,937 26.250000,24.500000,22.875000,22.625000,22.000000,21.125000,20.875000,20.625000 +20 4 17 1.501976 16,17,18,19,20,21,22,23 29.375000,29.250000,29.000000,28.375000,28.125000,27.750000,27.625000,27.625000 +20 5 23 2.560568 17,18,15,16,19,20,23,24 26.875000,26.750000,26.625000,26.500000,26.500000,26.250000,26.125000,26.000000 +20 6 18 2.018089 18,11,15,16,17,23,20,24 26.500000,26.375000,26.250000,26.250000,26.250000,26.125000,26.000000,26.000000 +20 7 17 2.433577 11,18,17,15,16,19,20,23 28.000000,26.750000,26.625000,26.375000,26.375000,26.250000,26.250000,26.250000 +21 0 23 2.392651 18,15,17,16,19,23,24,20 26.500000,26.125000,26.125000,26.000000,25.875000,25.875000,25.875000,25.750000 +21 1 22 3.406303 11211,198,271,18,15,17,20,320 25.875000,25.750000,25.375000,24.875000,24.625000,24.500000,24.500000,24.500000 +21 2 198 1.234829 11211,198,271,320,2790,4244,3950,18 25.625000,25.375000,24.750000,24.625000,22.000000,20.750000,20.625000,20.250000 +21 3 1655 9.947481 38,29300,7595,23489,84564,10879,32516,33 20.125000,19.625000,19.000000,19.000000,19.000000,18.875000,18.750000,18.625000 +21 4 393 11.142710 1517,2780,1429,220,8028,1273,1944,74936 21.750000,19.750000,18.875000,18.375000,18.250000,18.125000,18.125000,17.875000 +21 5 2916 0.949825 2916,28,457,2045,284,12,16,17 22.375000,21.750000,21.625000,21.250000,19.250000,18.750000,18.625000,18.500000 +21 6 70508 17.162922 25,315,16460,21937,320,11,34447,5603 22.625000,22.625000,21.875000,21.875000,21.750000,21.500000,21.500000,21.375000 +21 7 15536 4.765327 4098,284,25,50,15023,28,15536,1379 22.500000,19.000000,18.250000,18.250000,18.125000,18.000000,17.875000,17.625000 +22 0 28 1.503681 284,28,320,11,25,315,3968,7354 27.625000,26.625000,25.375000,24.750000,24.500000,24.250000,23.000000,22.250000 +22 1 19 2.380747 16,17,20,19,18,23,21,220 25.625000,25.500000,24.750000,24.250000,23.375000,22.375000,21.875000,20.625000 +22 2 15 0.004538 15,20,17,23,19,16,18,11 29.750000,22.750000,22.625000,22.625000,22.000000,21.875000,21.875000,21.750000 +22 3 24 0.082619 24,15,11,23,17,21,18,19 29.250000,26.750000,22.875000,22.125000,21.625000,20.375000,20.125000,20.125000 +22 4 21 0.000669 21,20,17,22,23,15,19,18 30.250000,21.750000,21.500000,20.875000,20.875000,20.500000,20.375000,18.750000 +22 5 419 7.875584 11,25,320,11211,198,510,1447,323 28.250000,25.125000,25.000000,23.750000,23.500000,23.125000,22.750000,22.625000 +22 6 35408 2.545738 1035,374,3059,6696,35408,686,9044,18644 24.000000,23.875000,23.750000,23.750000,23.250000,23.125000,23.125000,22.375000 +22 7 220 0.099796 220,25,320,264,3968,510,911,393 28.500000,24.750000,24.750000,23.875000,23.750000,23.500000,23.375000,23.000000 +23 0 21 1.641081 22,21,17,16,18,19,23,20 28.625000,27.375000,26.625000,24.750000,24.000000,24.000000,23.500000,23.000000 +23 1 24 0.027018 24,23,22,21,15,18,20,19 28.125000,23.750000,22.625000,21.875000,21.500000,21.500000,21.375000,21.125000 +23 2 2477 9.244780 26757,13,11879,2480,320,11211,3950,40467 23.625000,20.875000,19.625000,19.250000,18.875000,18.875000,18.500000,18.500000 +23 3 28252 0.020832 28252,12,1975,39433,23642,34072,14859,49817 27.000000,22.000000,21.250000,21.250000,20.250000,20.125000,19.875000,19.750000 +23 4 90619 0.000837 90619,75,15958,278,3629,4260,654,89722 29.750000,22.375000,21.250000,15.750000,15.750000,15.750000,15.437500,15.437500 +23 5 26757 0.084089 26757,220,320,44792,11879,1467,11211,2266 25.625000,22.125000,20.375000,20.250000,20.125000,19.875000,19.875000,19.750000 +23 6 345 5.634905 320,198,271,382,11,315,13,369 28.500000,27.875000,27.625000,27.625000,27.375000,27.000000,26.875000,26.500000 +23 7 19411 15.054314 9547,4197,8206,437,8088,7138,1958,704 22.625000,22.125000,21.125000,20.500000,20.250000,19.750000,19.625000,19.500000 +24 0 2845 0.062716 2845,480,287,11307,369,448,304,82 27.250000,24.500000,19.500000,17.750000,17.125000,17.000000,16.250000,16.000000 +24 1 47905 7.074774 2878,1212,26345,3685,2686,4946,366,18304 23.500000,23.375000,22.875000,22.375000,21.125000,20.500000,20.125000,19.750000 +24 2 279 0.101637 279,220,479,8028,1039,2567,393,894 26.375000,22.750000,22.000000,21.250000,21.125000,21.125000,21.000000,21.000000 +24 3 220 1.064806 220,8028,7192,2567,1932,17377,1372,3930 23.000000,22.000000,21.625000,21.375000,20.875000,20.750000,20.375000,20.250000 +24 4 17 2.159409 21,16,18,17,20,19,23,22 27.500000,26.500000,26.500000,26.250000,26.125000,25.750000,23.625000,22.250000 +24 5 15 0.230696 15,20,61,19,17,23,21,16 27.875000,26.125000,23.875000,23.625000,23.250000,23.250000,23.000000,22.750000 +24 6 11582 3.488510 26757,2567,11879,15,11582,12,4362,8028 22.750000,22.500000,20.500000,20.250000,20.250000,19.875000,19.625000,19.375000 +24 7 3122 0.006176 3122,15296,573,359,1942,15832,3077,31883 31.750000,26.625000,23.250000,20.750000,20.250000,19.750000,18.625000,18.625000 +25 0 16287 3.072573 8028,3930,12171,16287,4722,1308,2567,44279 23.125000,22.125000,21.000000,20.625000,19.250000,18.875000,18.750000,18.750000 +25 1 315 1.927896 369,382,315,320,624,271,13,311 28.375000,28.125000,27.375000,26.000000,25.625000,25.500000,25.250000,25.125000 +25 2 279 0.352917 279,419,479,1565,1273,264,330,342 25.000000,23.500000,21.125000,21.000000,20.375000,20.125000,20.000000,20.000000 +25 3 425 5.274380 1273,16460,4024,28431,479,9934,5567,5603 21.250000,20.625000,19.875000,19.875000,19.375000,19.250000,19.125000,19.000000 +25 4 18 8.606319 3399,1740,18454,1593,96190,2916,4872,12457 19.375000,19.250000,18.875000,18.125000,17.250000,16.875000,16.875000,16.250000 +25 5 4271 11.648369 1273,16460,12,18266,10519,25869,323,5456 20.875000,20.250000,19.000000,19.000000,18.250000,18.250000,17.875000,17.750000 +25 6 18126 5.132542 18266,1273,1779,25869,12890,28431,35957,198 23.500000,22.500000,22.000000,20.750000,20.625000,20.500000,20.500000,20.375000 +25 7 382 0.170805 382,624,320,271,13,198,369,345 28.625000,25.750000,25.500000,24.875000,24.625000,24.500000,24.250000,22.625000 +26 0 1986 3.217823 785,418,28,1986,9112,2461,44364,4416 21.250000,19.750000,19.250000,19.125000,18.625000,18.375000,17.875000,17.750000 +26 1 1034 4.209267 374,1273,10337,1034,821,15983,2150,5316 24.625000,24.500000,22.375000,21.375000,21.125000,21.125000,20.875000,20.875000 +26 2 374 0.407141 374,5610,572,320,5646,646,594,5707 28.000000,26.625000,25.000000,24.625000,24.625000,23.750000,23.625000,23.625000 +26 3 1483 2.112885 949,369,1483,264,7907,10602,279,8791 23.375000,23.000000,23.000000,22.875000,22.750000,22.375000,22.250000,21.125000 +26 4 438 2.532818 311,369,304,438,553,448,2337,1172 27.625000,27.375000,27.000000,26.125000,26.125000,23.750000,22.750000,21.875000 +26 5 264 1.100914 264,279,1946,1273,949,458,5785,4862 26.625000,26.500000,25.750000,25.375000,24.750000,24.000000,23.875000,22.125000 +26 6 12507 10.571642 1273,5785,5297,25869,28431,425,2530,4185 23.375000,22.750000,21.375000,20.250000,20.250000,19.875000,19.875000,19.625000 +26 7 369 0.929350 304,369,553,311,320,2337,979,448 28.125000,27.875000,25.500000,25.250000,24.375000,24.125000,22.500000,22.000000 +27 0 279 1.060913 7497,279,425,37563,7032,1273,4303,17590 24.000000,23.875000,21.750000,21.250000,20.625000,20.250000,20.250000,20.250000 +27 1 47175 15.302465 425,1273,1565,330,479,364,7032,4271 23.250000,21.625000,21.125000,20.625000,20.625000,20.375000,20.000000,19.750000 +27 2 44220 8.506355 33,61437,2271,34,12,8963,9285,9967 18.375000,18.250000,18.000000,17.875000,17.625000,17.250000,17.125000,17.125000 +27 3 84648 12.967186 1273,372,425,6733,1614,3135,7032,2022 22.875000,20.625000,20.500000,20.375000,20.125000,20.125000,20.125000,20.000000 +27 4 6500 0.708683 6500,19479,8233,1273,36680,47430,13850,4938 23.375000,22.750000,21.000000,20.500000,19.875000,19.875000,19.375000,19.000000 +27 5 73152 11.031029 1273,4271,28431,5068,7497,7032,10272,62019 21.875000,21.750000,21.625000,20.625000,20.250000,20.000000,20.000000,19.875000 +27 6 487 0.000019 487,34494,1361,16460,3147,367,275,22814 29.250000,17.000000,16.625000,16.000000,15.250000,15.062500,14.937500,14.750000 +27 7 18126 6.213497 1273,28431,198,7032,16460,7497,62019,4271 25.500000,23.500000,22.625000,22.625000,22.125000,21.375000,21.375000,21.250000 +28 0 198 2.704977 1273,382,7032,624,198,320,13,345 25.625000,25.125000,24.500000,24.250000,24.125000,24.125000,23.375000,23.375000 +28 1 8623 5.908091 1944,258,437,7,23841,2428,300,5349 21.250000,20.875000,20.875000,20.750000,20.375000,20.250000,20.125000,20.000000 +28 2 82 0.671923 82,287,291,620,880,25,5523,53346 23.375000,22.250000,22.000000,21.000000,20.000000,19.500000,18.875000,18.750000 +28 3 5523 1.545790 12697,5523,14,4846,43985,78311,3470,4322 20.125000,19.875000,19.375000,18.875000,18.125000,18.125000,18.000000,17.375000 +28 4 324 0.014227 324,9048,80,65,85964,269,9713,1677 31.625000,27.375000,21.000000,19.875000,19.250000,19.125000,19.000000,18.875000 +28 5 749 0.000004 749,6331,17184,5970,2011,126394,15439,21028 40.250000,27.625000,24.750000,24.250000,23.250000,22.750000,22.625000,21.750000 +28 6 97066 14.513279 30448,53346,80,1751,8976,14,4452,44220 28.125000,23.750000,19.500000,18.500000,18.250000,18.125000,17.750000,17.750000 +28 7 2204 8.749159 11529,620,5678,9360,53346,76776,14,43985 22.750000,21.875000,20.875000,19.625000,18.750000,18.250000,17.875000,17.500000 +29 0 17 1.546122 19642,17,1409,20615,3334,27480,23819,12536 24.000000,22.875000,21.375000,20.500000,19.875000,19.750000,19.000000,18.875000 +29 1 68 0.000195 68,57824,291,295,288,5120,16,823 32.250000,23.000000,21.125000,21.000000,20.375000,20.375000,20.000000,19.875000 +29 2 25638 6.651157 7197,4452,12697,14,880,43985,4846,54798 21.125000,20.500000,19.625000,18.625000,18.625000,18.375000,18.250000,17.875000 +29 3 4292 3.957254 8,568,701,3593,692,11,323,4292 26.125000,26.000000,25.750000,25.750000,25.375000,24.125000,23.750000,23.625000 +29 4 8610 7.209177 785,2132,2461,418,1654,2679,4416,1986 22.375000,20.875000,19.750000,19.250000,19.000000,19.000000,19.000000,18.750000 +29 5 31805 11.787052 4344,2297,50844,8837,28782,17030,2647,1273 25.250000,23.250000,23.125000,22.375000,22.375000,21.750000,20.750000,20.375000 +29 6 990 4.941956 4278,975,315,476,3542,2038,1969,990 24.375000,24.000000,23.750000,22.750000,21.500000,21.375000,21.000000,20.500000 +29 7 1969 0.614321 1969,315,320,7460,1265,374,476,1588 27.750000,27.250000,25.000000,24.625000,24.500000,24.375000,24.250000,22.250000 +30 0 14060 4.542385 25017,48453,2924,387,5978,1795,21129,25879 24.000000,22.250000,22.000000,21.750000,21.250000,21.000000,21.000000,21.000000 +30 1 419 1.572054 279,419,62024,1493,5723,678,4024,6920 26.250000,25.125000,24.250000,22.250000,22.250000,22.000000,21.500000,21.000000 +30 2 13534 7.214588 62024,5723,5293,6920,3980,4247,5185,44377 23.125000,22.125000,21.625000,20.750000,20.250000,20.250000,19.750000,18.875000 +30 3 7710 1.106220 7671,7710,15,5723,62024,12,14319,92214 24.375000,23.875000,21.875000,21.750000,20.875000,19.750000,19.500000,19.250000 +30 4 92214 0.000761 92214,16589,12,81219,220,6222,62024,5723 27.500000,19.375000,18.625000,18.000000,17.750000,17.625000,17.375000,17.250000 +30 5 220 0.383530 220,5723,62024,5293,12,348,323,6920 25.250000,24.125000,23.000000,20.875000,19.500000,19.125000,18.875000,18.250000 +30 6 18 0.000099 18,19,17,16,15,220,20,21 33.000000,23.375000,22.625000,18.250000,17.875000,16.875000,16.375000,15.000000 +30 7 13 -0.000000 13,12,624,36479,382,10845,15,1876 35.750000,17.250000,17.250000,17.000000,16.500000,16.125000,15.625000,15.437500 +31 0 15 -0.000000 15,18,24,20,16,21,17,99822 35.250000,19.500000,17.625000,15.937500,15.625000,15.312500,14.750000,14.625000 +31 1 62024 1.373686 5723,62024,5293,323,6668,6920,1876,27406 25.250000,24.250000,22.250000,19.500000,19.250000,19.125000,18.875000,18.875000 +31 2 382 0.774311 382,323,624,5293,13,198,11,345 26.750000,26.375000,24.625000,24.125000,23.750000,23.750000,23.500000,23.000000 +31 3 284 7.251692 28,418,785,44364,313,2461,4421,8707 20.500000,20.500000,20.375000,18.250000,18.125000,17.500000,17.375000,17.375000 +31 4 8397 12.745161 3972,608,11176,284,835,1406,220,4710 22.250000,19.250000,19.250000,19.125000,18.875000,18.875000,18.750000,18.000000 +31 5 425 4.568361 64,356,451,468,386,328,422,594 20.375000,17.875000,17.625000,17.500000,17.375000,17.250000,17.250000,16.875000 +31 6 10965 13.632807 13,22908,366,398,535,673,356,1826 23.625000,16.250000,16.125000,15.937500,15.937500,15.875000,15.687500,15.687500 +31 7 465 2.387480 4003,777,465,1664,1280,665,466,484 21.250000,19.375000,19.125000,16.500000,15.812500,15.312500,14.937500,14.687500 +32 0 284 2.882164 594,320,79483,11,284,271,25,481 23.250000,21.625000,21.250000,21.000000,21.000000,20.625000,20.375000,19.000000 +32 1 715 1.984297 4710,1406,715,320,2303,220,18611,374 22.625000,21.875000,21.375000,20.125000,19.750000,18.625000,18.625000,18.375000 +32 2 8397 4.085903 785,34532,40,32313,284,8397,1986,58 18.375000,18.250000,17.000000,17.000000,16.625000,16.250000,16.125000,15.937500 +32 3 425 0.021715 425,7801,3757,8397,12375,468,220,21891 22.500000,15.812500,15.187500,15.187500,15.187500,14.875000,14.750000,14.625000 +32 4 10965 0.000204 10965,13,616,2950,965,30158,220,23937 25.625000,16.375000,15.312500,14.500000,13.562500,13.500000,13.187500,12.875000 +32 5 465 0.000019 465,5045,777,665,376,1982,4160,4991 30.000000,18.125000,17.125000,16.625000,16.500000,16.125000,15.750000,15.500000 +32 6 374 1.746269 320,374,572,11,220,594,198,702 24.625000,23.125000,21.000000,19.625000,19.125000,17.875000,17.750000,17.500000 +32 7 458 1.567019 264,458,279,825,5023,3881,1850,1431 26.500000,25.250000,23.500000,22.250000,21.125000,20.625000,19.750000,19.375000 +33 0 6364 0.634359 6364,3693,13369,9489,12089,3150,17847,21946 21.625000,21.000000,19.500000,17.250000,17.000000,16.750000,16.750000,16.500000 +33 1 4531 4.786921 6584,12089,4741,8964,1515,6916,39160,3150 18.375000,18.125000,18.000000,17.125000,16.375000,16.250000,16.250000,15.625000 +33 2 1154 9.175163 7538,323,16851,6440,11,12089,4171,39450 22.750000,22.500000,21.375000,20.500000,18.750000,18.125000,18.000000,17.750000 +33 3 12425 0.069733 12425,6430,5883,2766,33496,11086,323,26705 26.750000,23.875000,21.500000,20.750000,19.625000,19.500000,19.375000,19.125000 +33 4 323 0.039146 323,1154,11,16851,220,609,7538,320 29.750000,26.500000,21.375000,21.375000,20.250000,20.000000,20.000000,19.625000 +33 5 33496 2.872169 6430,33496,7743,8887,8353,26705,2766,65233 25.125000,22.375000,21.375000,20.625000,20.125000,20.000000,19.625000,19.125000 +33 6 12089 0.958448 7538,12089,16851,23654,220,4171,6916,48154 24.375000,24.125000,22.625000,19.750000,19.250000,19.125000,19.000000,18.875000 +33 7 659 1.380764 659,879,323,1850,3881,1154,13,448 27.875000,27.875000,27.500000,27.125000,27.000000,25.875000,25.250000,24.625000 +34 0 1260 0.180466 1260,425,5301,4636,37816,758,715,362 25.500000,23.625000,20.750000,20.625000,20.625000,19.375000,19.000000,18.625000 +34 1 1030 6.402983 572,374,702,9723,6342,19476,16176,18178 25.000000,24.375000,24.375000,22.500000,21.625000,21.625000,21.500000,21.375000 +34 2 264 0.562696 264,806,12783,8597,458,279,1012,8922 25.250000,23.750000,22.750000,22.125000,21.875000,21.750000,21.750000,21.750000 +34 3 8640 5.316959 3476,2613,8922,12613,45250,3598,6388,9814 22.875000,22.250000,22.250000,21.500000,21.500000,21.000000,21.000000,20.750000 +34 4 569 14.850103 3476,11094,481,39400,7702,949,6774,26255 24.750000,23.250000,21.000000,20.875000,20.500000,20.375000,20.375000,18.625000 +34 5 12 0.444805 12,481,220,304,198,3476,1376,19191 20.000000,19.125000,15.937500,15.375000,15.187500,14.562500,14.437500,14.437500 +34 6 31 0.010168 31,16,5778,62799,17,3476,569,3157 25.625000,18.875000,18.625000,18.375000,17.875000,17.875000,17.500000,17.125000 +34 7 39400 2.449594 3476,39400,11094,6774,220,7702,949,2484 26.375000,24.125000,23.375000,22.375000,21.250000,20.875000,20.500000,20.375000 +35 0 3476 0.018923 3476,304,11094,949,12783,2309,6502,389 28.125000,23.625000,21.875000,21.500000,21.125000,20.125000,20.125000,20.000000 +35 1 389 4.579731 304,438,389,315,448,518,5619,323 29.250000,26.500000,24.750000,22.625000,22.375000,22.375000,22.125000,21.500000 +35 2 279 0.143845 279,576,18096,264,330,18635,6326,364 24.000000,20.500000,19.750000,19.375000,19.375000,18.875000,18.750000,18.625000 +35 3 12425 2.040039 18096,12425,7855,220,1156,2086,4013,13434 22.125000,21.250000,20.625000,20.375000,20.250000,19.750000,19.750000,19.625000 +35 4 4013 0.062426 4013,1473,2025,18136,19584,22358,80366,6146 26.250000,22.750000,21.875000,21.250000,20.375000,20.125000,20.125000,19.750000 +35 5 576 1.945026 330,576,3355,1154,364,18635,4678,1591 19.375000,19.125000,18.500000,18.375000,17.875000,17.375000,16.875000,16.750000 +35 6 8596 3.844187 328,8246,19378,4312,1599,1532,8596,220 16.875000,16.875000,16.625000,15.625000,15.187500,15.062500,14.875000,14.625000 +35 7 304 1.562088 323,304,1154,320,659,438,504,220 26.500000,25.750000,25.000000,24.750000,24.750000,24.125000,23.125000,21.750000 +36 0 220 0.312962 220,279,264,892,458,1181,806,5813 26.625000,25.500000,22.250000,22.250000,20.250000,20.000000,19.875000,19.875000 +36 1 17 0.523573 17,16,18,15,1154,220,19,22 22.500000,22.125000,13.250000,13.000000,12.625000,12.187500,11.875000,11.812500 +36 2 15 0.000011 15,16,220,17,18,569,23,1154 28.500000,15.562500,15.187500,14.437500,13.937500,13.875000,13.750000,13.750000 +36 3 15 0.101780 15,16,17,220,1154,569,18,549 25.250000,23.000000,18.875000,12.500000,12.312500,12.250000,11.750000,11.312500 +36 4 15 1.831144 15,16,17,18,19,20,22,21 21.250000,21.125000,21.125000,21.125000,20.750000,20.750000,20.625000,20.375000 +36 5 659 0.855446 323,659,1154,438,13,320,714,2587 27.625000,27.625000,26.125000,24.500000,23.750000,23.125000,22.625000,22.375000 +36 6 1096 7.665333 1260,425,758,5301,1913,5542,24885,8704 25.875000,24.125000,23.500000,22.000000,20.375000,20.375000,20.375000,20.000000 +36 7 572 0.333273 572,374,3476,4013,4549,6069,9234,3668 24.750000,23.000000,22.125000,20.500000,20.125000,19.625000,19.625000,19.250000 +37 0 8110 1.436741 806,8110,264,279,304,438,825,458 24.875000,24.500000,24.000000,23.500000,22.875000,21.750000,21.500000,21.375000 +37 1 553 0.029675 553,304,448,705,264,19620,2937,1378 29.375000,25.500000,23.250000,23.125000,22.375000,22.375000,22.000000,21.875000 +37 2 264 0.393456 264,806,26255,279,458,12783,2441,8640 25.250000,23.750000,22.250000,21.750000,21.750000,21.625000,21.000000,20.375000 +37 3 39400 5.183321 3476,8640,949,45250,12613,4013,2613,8922 23.000000,21.000000,20.625000,20.625000,20.375000,20.250000,20.125000,20.000000 +37 4 3476 0.003990 3476,304,949,569,2309,323,5068,6502 27.750000,20.625000,20.375000,20.125000,19.250000,19.125000,19.125000,19.125000 +37 5 304 0.129242 304,438,389,448,315,518,323,5619 29.375000,27.000000,25.875000,24.125000,23.000000,22.875000,22.500000,22.250000 +37 6 279 0.118267 279,264,576,220,458,18096,364,806 25.125000,22.125000,20.750000,19.875000,19.875000,19.000000,18.625000,18.625000 +37 7 1486 3.718361 220,12425,4531,18096,2805,1486,7855,4565 21.750000,21.250000,20.750000,20.125000,19.750000,19.250000,19.125000,18.875000 +38 0 6252 7.011986 576,364,1154,330,3355,362,518,659 20.000000,19.125000,18.625000,18.375000,17.750000,17.250000,16.875000,16.625000 +38 1 2382 1.732311 2382,263,585,288,5298,1154,18209,66244 17.625000,17.000000,16.750000,16.375000,16.250000,16.000000,16.000000,15.687500 +38 2 5326 7.861170 304,1154,518,323,553,389,320,659 22.125000,21.500000,21.000000,20.875000,20.625000,20.500000,20.250000,20.250000 +38 3 553 0.008677 553,323,304,369,448,518,569,3216 29.750000,24.750000,22.500000,22.000000,20.500000,20.500000,20.500000,19.875000 +38 4 21315 3.764839 6798,3757,11867,4389,14991,21315,33121,18095 18.250000,17.625000,17.500000,17.375000,17.375000,17.375000,17.375000,17.250000 +38 5 75923 2.961792 23366,1644,12131,8362,75923,39103,386,472 18.000000,17.500000,17.500000,16.625000,16.625000,16.375000,15.937500,15.687500 +38 6 1154 2.658560 659,323,304,1154,518,892,369,320 27.625000,27.500000,27.250000,26.125000,25.875000,24.750000,24.625000,24.000000 +38 7 892 0.375389 892,323,304,1156,10660,15540,518,8947 25.875000,24.125000,22.500000,22.500000,22.500000,22.375000,22.250000,22.000000 +39 0 572 0.705832 572,8930,10613,84070,566,90613,6342,22409 25.375000,24.125000,23.875000,23.375000,23.125000,22.750000,22.000000,21.875000 +39 1 10660 0.812071 10660,8947,50057,1156,10449,518,6839,15540 23.625000,22.500000,22.250000,21.625000,21.250000,21.000000,20.875000,20.750000 +39 2 304 2.262070 518,304,553,389,1156,369,504,569 29.875000,27.750000,25.750000,24.250000,23.750000,22.875000,22.875000,22.500000 +39 3 220 0.727556 220,279,7148,264,2176,19361,5257,576 23.375000,23.000000,21.375000,20.875000,19.625000,19.125000,18.625000,18.250000 +39 4 17 0.003731 17,16,18,19,15,20,22,21 26.500000,20.875000,16.125000,15.375000,15.125000,14.937500,14.625000,14.125000 +39 5 15 0.000004 15,16,17,18,23,24,19,1154 30.250000,16.875000,15.375000,14.437500,14.437500,14.250000,14.187500,14.062500 +39 6 15 0.008640 15,16,17,220,18,1154,19,549 28.625000,23.875000,17.875000,13.812500,13.750000,13.500000,13.312500,13.312500 +39 7 16 1.003468 16,17,18,15,19,20,21,22 22.250000,21.750000,21.500000,20.625000,20.500000,20.125000,19.500000,19.375000 +40 0 518 0.405586 518,659,323,304,553,389,1154,13 28.375000,26.750000,26.375000,26.250000,24.500000,22.875000,22.750000,22.500000 +40 1 279 0.022123 279,576,264,7148,2176,26221,19361,24688 25.625000,21.375000,19.000000,19.000000,18.000000,17.625000,17.125000,16.750000 +40 2 16194 0.918459 16194,15953,5055,1163,4312,14079,19361,26221 20.625000,19.875000,19.250000,18.000000,17.625000,17.625000,17.375000,17.375000 +40 3 7154 0.049406 7154,5055,18705,41382,323,15815,15953,65463 24.125000,20.375000,20.000000,18.250000,17.750000,17.000000,17.000000,16.375000 +40 4 26221 0.062290 26221,304,33496,315,569,659,1154,323 25.750000,21.625000,21.125000,21.000000,20.750000,20.750000,20.375000,20.125000 +40 5 659 1.773592 304,659,323,1154,3138,320,16448,481 28.000000,26.500000,24.750000,24.625000,22.375000,22.250000,21.625000,21.000000 +40 6 1260 0.875631 1260,758,425,5301,24885,576,4636,5542 24.625000,24.125000,23.875000,22.375000,21.375000,21.000000,20.750000,20.500000 +40 7 1030 3.054010 1083,1221,572,702,9723,6342,1030,2937 23.500000,23.125000,22.875000,22.875000,22.125000,22.000000,21.875000,21.875000 +41 0 264 0.078417 264,2441,806,458,1083,279,12783,4623 27.250000,23.375000,22.875000,22.750000,21.875000,21.750000,21.625000,21.125000 +41 1 8640 2.266478 3476,12613,8640,8922,2613,45250,6388,9814 21.875000,21.875000,21.125000,21.000000,20.750000,20.625000,20.375000,19.500000 +41 2 3476 0.528576 3476,569,39400,11094,7702,6774,949,25377 23.500000,22.125000,22.125000,21.500000,19.000000,18.875000,17.625000,17.375000 +41 3 304 0.152626 304,389,438,1549,448,518,5619,315 30.375000,28.375000,26.375000,24.375000,24.125000,23.875000,23.875000,23.250000 +41 4 279 0.069275 279,576,264,220,2441,458,18096,7506 25.500000,21.125000,21.000000,20.500000,20.500000,19.875000,19.500000,18.750000 +41 5 12425 0.714283 12425,18096,220,4013,7855,4531,5883,2086 23.000000,21.625000,21.500000,20.875000,19.875000,19.750000,19.625000,19.500000 +41 6 4013 0.015282 4013,19584,1473,2025,18136,6146,4531,22358 27.375000,22.125000,20.875000,20.750000,20.750000,20.625000,20.375000,20.375000 +41 7 20207 5.308310 576,7506,15696,18635,6326,4678,9414,48822 17.875000,16.250000,16.250000,16.125000,16.000000,15.812500,15.812500,15.687500 +42 0 3757 0.000729 3757,1154,659,422,54904,54058,81204,481 28.375000,20.375000,19.000000,18.750000,18.250000,18.000000,17.875000,17.625000 +42 1 1581 0.015110 1581,54904,1154,31302,82884,55156,5227,444 22.875000,17.750000,16.375000,15.937500,15.875000,15.812500,15.625000,15.500000 +42 2 291 0.001705 291,2636,2145,585,19289,1090,68,546 27.000000,20.125000,17.750000,17.500000,17.500000,17.375000,17.000000,17.000000 +42 3 304 0.307930 304,320,323,1154,659,504,892,438 28.750000,26.875000,26.375000,25.875000,24.750000,24.000000,23.875000,23.625000 +42 4 220 0.010113 220,279,264,892,5813,3217,5470,6058 28.875000,24.000000,21.000000,20.875000,20.125000,19.875000,19.750000,19.750000 +42 5 17 0.000265 17,16,18,15,20,19,22,220 28.875000,20.625000,15.812500,14.375000,13.500000,13.437500,12.687500,12.687500 +42 6 15 0.000002 15,16,220,17,23,18,1154,659 30.000000,15.687500,14.500000,14.312500,13.625000,13.437500,13.125000,13.000000 +42 7 15 0.003189 15,16,17,1154,220,549,659,481 28.375000,22.625000,16.500000,14.687500,14.000000,13.125000,13.125000,12.562500 +43 0 17 1.085657 17,18,19,16,20,21,22,23 22.500000,22.500000,21.500000,21.125000,20.750000,20.000000,19.500000,18.375000 +43 1 659 1.062925 323,659,1154,438,320,13,304,2587 29.500000,29.000000,27.375000,24.250000,23.875000,23.750000,23.625000,23.375000 +43 2 758 1.084049 1260,758,425,5301,24885,5542,5005,27133 25.250000,25.000000,24.125000,22.500000,21.500000,21.000000,20.750000,20.375000 +43 3 220 0.036356 220,279,5256,5813,3217,5470,6702,7400 28.625000,24.875000,23.000000,21.625000,21.500000,21.375000,21.375000,21.375000 +43 4 17 0.000437 17,16,15,1154,18,220,549,19 26.750000,19.000000,13.437500,13.000000,12.562500,11.875000,10.562500,10.375000 +43 5 15 -0.000000 15,16,24,17,23,18,19,220 31.625000,15.437500,14.187500,13.937500,13.625000,13.187500,12.375000,12.312500 +43 6 15 0.011084 15,16,17,1154,220,18,549,19 27.875000,23.375000,17.500000,13.875000,13.375000,13.187500,12.187500,12.000000 +43 7 19 1.610905 18,19,20,21,17,22,23,24 24.750000,23.625000,22.875000,21.875000,21.375000,21.250000,20.375000,19.750000 +44 0 425 3.314785 566,1154,425,323,806,1260,569,8397 29.625000,29.250000,26.875000,23.500000,23.250000,22.000000,21.750000,21.500000 +44 1 10965 0.000156 10965,965,581,616,6344,1080,220,425 25.625000,15.687500,14.312500,14.187500,13.687500,13.312500,13.062500,13.000000 +44 2 465 0.000019 465,5045,376,665,59385,423,4160,1982 31.125000,19.875000,17.750000,17.250000,16.500000,16.375000,16.375000,16.250000 +44 3 26120 7.430857 6342,9723,572,10859,1030,3867,1865,58235 24.500000,24.375000,23.375000,22.375000,22.250000,22.000000,21.875000,21.375000 +44 4 264 0.267038 264,279,806,458,304,2441,825,12783 28.000000,26.500000,25.375000,22.625000,21.500000,20.750000,20.000000,19.750000 +44 5 3476 0.970726 3476,6388,12613,45250,5792,949,3598,39400 22.875000,21.750000,21.500000,21.375000,21.250000,21.125000,20.625000,20.000000 +44 6 438 1.823097 304,438,389,448,5619,315,518,569 29.500000,27.875000,24.250000,24.250000,24.125000,23.500000,23.125000,21.750000 +44 7 330 8.402699 264,279,458,4389,6798,825,8364,11867 21.500000,19.625000,19.000000,17.750000,17.625000,17.500000,17.375000,17.375000 +45 0 28417 7.448458 4392,14991,21315,6798,3757,8364,32819,4389 16.250000,16.000000,15.937500,15.875000,15.812500,15.812500,15.750000,15.687500 +45 1 330 0.136829 330,3670,422,425,386,472,7401,1 21.625000,16.750000,16.000000,16.000000,15.875000,15.812500,15.812500,15.687500 +45 2 304 0.020948 304,389,1154,320,323,279,264,448 29.000000,24.000000,24.000000,22.625000,22.375000,22.000000,21.875000,21.125000 +45 3 279 0.028978 279,264,576,458,13434,18096,330,4013 26.875000,22.250000,21.625000,20.500000,20.250000,20.125000,19.375000,19.250000 +45 4 9234 8.027498 12425,4531,18096,4013,7855,13434,22358,4565 22.125000,21.125000,20.875000,20.250000,20.250000,20.125000,19.750000,19.125000 +45 5 330 0.773592 330,315,24849,1591,3355,364,19867,1154 24.375000,24.125000,22.125000,21.500000,21.250000,21.000000,20.750000,19.875000 +45 6 70066 10.947123 576,362,425,358,350,356,220,422 19.500000,16.875000,16.750000,16.250000,16.125000,16.125000,16.000000,16.000000 +45 7 364 4.697786 330,23985,364,323,16180,35968,304,8171 23.500000,19.625000,18.875000,18.375000,17.750000,17.000000,16.750000,16.750000 +46 0 82 0.002056 82,274,364,569,50,1154,67,2587 27.000000,20.625000,16.875000,16.250000,16.125000,16.000000,15.875000,15.625000 +46 1 15106 4.608822 5004,6841,8599,6059,6164,1532,7506,7996 15.562500,15.500000,15.437500,15.375000,15.062500,14.937500,14.812500,14.812500 +46 2 330 0.003899 330,320,364,3670,1,549,1154,5933 28.250000,20.375000,20.250000,20.125000,20.000000,19.625000,19.375000,19.250000 +46 3 315 0.098465 315,304,389,504,320,1154,323,892 30.125000,27.125000,26.500000,25.250000,25.000000,24.750000,24.250000,23.750000 +46 4 279 0.024517 279,576,18096,13434,4013,264,18635,7506 26.000000,21.625000,19.500000,19.125000,18.625000,18.250000,18.125000,18.000000 +46 5 12425 0.748310 12425,4013,18096,7855,13434,22358,5883,19584 23.000000,22.125000,21.750000,20.000000,19.875000,19.625000,19.375000,19.250000 +46 6 4013 0.007528 4013,19584,2025,1473,6146,22358,18136,80366 27.750000,21.375000,21.125000,21.000000,20.625000,20.500000,20.375000,19.125000 +46 7 576 1.754431 576,1154,18635,48822,15696,52325,6326,14671 18.250000,16.875000,16.625000,16.250000,16.125000,16.125000,16.000000,16.000000 +47 0 5724 8.366972 328,8246,19378,9494,1532,220,6686,8596 17.500000,16.625000,16.500000,15.875000,15.750000,15.625000,15.437500,15.375000 +47 1 37625 3.607450 477,9536,4050,7684,18971,2738,12149,37625 19.625000,18.625000,18.500000,18.125000,18.000000,17.500000,17.250000,17.125000 +47 2 2587 5.449572 659,1154,320,323,389,892,304,2587 27.375000,25.625000,25.000000,24.125000,23.375000,23.000000,22.750000,22.250000 +47 3 566 1.169445 566,419,279,323,304,4764,425,806 24.250000,24.125000,23.750000,22.500000,22.250000,21.375000,21.250000,21.250000 +47 4 58235 6.761471 572,1083,6342,5927,9723,2937,1030,1221 23.875000,23.625000,23.250000,22.875000,22.125000,21.875000,21.500000,21.000000 +47 5 16263 4.623627 304,438,1549,448,16263,1083,369,518 27.500000,26.000000,23.500000,23.125000,23.125000,22.000000,21.000000,21.000000 +47 6 19571 4.119192 6798,279,3757,4389,11867,8397,18939,24986 18.000000,17.875000,17.875000,17.875000,17.875000,17.750000,17.500000,17.500000 +47 7 4389 4.299381 1741,1075,2670,11867,18939,6798,4389,3757 22.750000,21.250000,20.500000,19.500000,19.500000,19.375000,19.250000,19.125000 +48 0 26099 4.778086 431,13612,2691,20607,350,72113,1260,26099 19.375000,17.000000,16.375000,15.562500,15.187500,14.937500,14.875000,14.875000 +48 1 323 0.065849 323,1154,320,659,304,438,2587,609 30.875000,28.125000,25.000000,23.125000,22.625000,21.625000,21.250000,21.000000 +48 2 42323 6.572186 6798,3757,11867,4389,18939,7801,11565,18095 18.250000,17.750000,17.625000,17.500000,17.500000,17.375000,17.375000,17.375000 +48 3 15035 3.184395 25164,15035,63540,431,434,98971,4264,51656 22.750000,19.625000,17.500000,17.125000,15.562500,15.125000,15.062500,15.062500 +48 4 18337 0.003052 18337,404,72,23300,18122,46291,354,1119 23.750000,17.750000,14.750000,14.437500,13.687500,13.375000,13.000000,12.812500 +48 5 659 0.089527 659,304,323,1154,16448,6762,320,2587 30.250000,27.250000,26.250000,25.125000,25.000000,24.500000,24.375000,24.000000 +48 6 1260 1.723835 758,1260,425,5301,24885,576,2938,5542 25.250000,24.000000,23.625000,21.375000,21.375000,21.250000,20.250000,20.125000 +48 7 572 1.676542 1083,572,6342,1030,702,9723,1221,2937 23.000000,22.875000,22.375000,22.250000,22.000000,22.000000,21.750000,21.250000 +49 0 6311 2.209560 1083,264,6311,304,1221,2937,14766,27420 23.750000,22.375000,22.375000,21.750000,21.375000,21.375000,20.750000,20.375000 +49 1 304 0.659798 304,438,369,311,1549,553,389,448 26.375000,26.250000,22.875000,21.625000,20.250000,19.625000,19.250000,19.125000 +49 2 279 0.159676 279,264,220,2441,458,576,1378,806 26.750000,24.625000,22.625000,22.125000,21.500000,21.250000,20.375000,19.625000 +49 3 220 1.306345 220,3476,12425,4531,1486,4013,18096,2805 22.625000,22.500000,22.375000,21.250000,20.500000,19.500000,19.500000,19.125000 +49 4 17 0.003527 17,16,18,19,20,22,21,23 28.125000,22.125000,19.875000,19.750000,19.500000,19.000000,18.875000,18.125000 +49 5 15 0.000044 15,16,17,18,19,23,20,24 28.375000,17.500000,16.000000,15.500000,15.375000,15.375000,15.125000,14.687500 +49 6 15 0.020632 15,16,17,19,339,20,18,4146 27.750000,23.875000,18.250000,14.750000,14.250000,14.187500,14.000000,13.312500 +49 7 20 0.839939 20,21,22,19,23,24,18,17 23.375000,22.875000,22.125000,22.000000,21.000000,20.250000,19.375000,18.375000 +50 0 33496 8.196558 4531,12425,4013,7855,4565,18096,5700,22358 21.375000,21.250000,19.500000,18.625000,18.500000,18.500000,18.250000,18.125000 +50 1 53148 6.371386 5670,1486,5068,33085,56005,481,1473,6573 25.625000,24.125000,21.250000,20.625000,20.625000,19.750000,19.625000,19.625000 +50 2 315 0.076632 315,576,518,1154,330,549,304,323 25.000000,20.375000,20.250000,20.000000,19.125000,19.000000,18.875000,18.875000 +50 3 279 1.840660 576,279,330,7401,362,364,422,9043 20.750000,20.125000,18.250000,17.625000,17.375000,17.125000,16.875000,16.875000 +50 4 25179 10.188784 1486,11088,22358,17795,6430,825,1852,19584 23.250000,21.625000,19.125000,19.000000,18.875000,18.500000,18.250000,17.750000 +50 5 92301 3.626781 31828,730,20734,444,92301,30563,20113,6210 19.500000,17.500000,17.000000,16.750000,16.375000,16.125000,16.000000,15.875000 +50 6 1486 0.320869 1486,364,11088,481,19584,1154,323,7054 24.250000,22.750000,21.500000,20.125000,19.875000,19.750000,19.250000,18.875000 +50 7 43562 8.485886 576,330,364,1154,25044,425,659,362 21.625000,20.125000,18.750000,18.750000,17.750000,17.000000,17.000000,16.625000 +51 0 49840 0.469931 49840,323,1154,518,304,2587,553,320 24.375000,23.125000,22.750000,20.875000,20.375000,20.000000,19.625000,19.250000 +51 1 1154 1.814772 323,1154,320,518,659,304,2587,892 27.375000,26.125000,25.375000,25.125000,25.125000,24.125000,23.500000,22.750000 +51 2 892 1.963104 323,892,15540,10660,518,5619,304,448 25.125000,23.875000,23.750000,22.875000,22.250000,22.250000,22.125000,21.625000 +51 3 572 0.242048 572,10613,90613,3867,1033,566,8930,1030 25.500000,22.875000,22.125000,21.875000,21.750000,21.625000,21.375000,21.250000 +51 4 10660 0.442640 10660,15540,50057,8947,5326,10449,264,518 24.000000,22.125000,21.500000,21.125000,20.625000,20.500000,20.125000,20.000000 +51 5 518 0.271942 518,304,553,389,438,1212,448,1549 29.625000,28.375000,25.500000,23.250000,23.000000,23.000000,22.625000,22.125000 +51 6 279 0.020451 279,576,7148,264,2055,2176,26221,5257 25.000000,20.250000,19.250000,18.125000,18.000000,18.000000,17.875000,17.375000 +51 7 45406 10.174389 16194,5055,2055,1674,15953,1163,14079,13263 20.500000,19.875000,19.500000,19.125000,19.000000,18.250000,18.125000,17.500000 +52 0 26221 0.020060 26221,33496,304,323,1154,22780,37471,320 26.125000,21.625000,20.500000,20.000000,18.875000,18.000000,18.000000,17.875000 +52 1 304 0.210781 304,1154,323,659,320,2587,315,448 28.625000,26.250000,26.000000,25.750000,23.500000,22.125000,21.375000,21.250000 +52 2 70998 6.430344 7148,220,52935,6326,279,86570,356,90157 20.250000,19.000000,19.000000,18.250000,17.750000,17.625000,17.500000,16.500000 +52 3 323 0.361603 323,659,1154,2587,304,1573,320,448 29.250000,27.875000,27.250000,25.250000,24.375000,24.000000,23.750000,23.000000 +52 4 279 0.944805 279,518,2937,1221,304,1083,572,27420 24.000000,23.375000,23.250000,22.875000,21.000000,21.000000,20.000000,19.875000 +52 5 11012 7.956448 16194,15953,1674,1163,5055,4312,1532,13263 19.125000,18.750000,18.250000,18.000000,17.750000,17.125000,17.000000,17.000000 +52 6 1268 0.002666 1268,594,1409,518,304,4813,424,18849 26.000000,19.000000,18.500000,17.875000,17.250000,16.625000,15.937500,15.937500 +52 7 38420 0.000097 38420,19295,14997,92964,17931,659,15619,1154 31.750000,22.250000,20.375000,18.625000,18.375000,18.125000,17.750000,17.625000 +53 0 16937 0.001793 16937,8633,19295,37729,1154,21618,26524,99675 28.875000,22.250000,20.750000,18.375000,18.000000,17.875000,17.750000,17.125000 +53 1 304 0.037256 304,659,1154,323,2587,320,16448,6762 30.500000,27.000000,24.625000,24.250000,23.750000,23.000000,21.875000,21.250000 +53 2 7148 0.000561 7148,4882,220,279,31302,107074,9806,659 29.625000,21.125000,21.000000,19.500000,19.500000,19.125000,18.500000,18.375000 +53 3 659 0.117085 659,2587,1154,304,323,13,16448,6762 29.875000,26.625000,26.375000,25.750000,25.750000,24.875000,24.500000,24.000000 +53 4 1260 1.222822 758,1260,425,5301,576,24885,1096,2938 24.250000,24.000000,23.750000,21.375000,21.125000,20.875000,20.000000,19.500000 +53 5 572 1.588333 572,6342,1030,1083,9723,702,1221,2937 22.750000,22.625000,22.500000,22.375000,21.875000,21.125000,21.000000,20.500000 +53 6 15540 8.320093 1083,6311,264,304,14766,1221,2937,949 23.625000,23.125000,22.250000,21.875000,21.125000,20.875000,20.625000,20.250000 +53 7 553 0.017456 553,304,369,323,1052,518,311,389 29.375000,25.000000,22.500000,22.375000,22.375000,22.000000,21.500000,20.750000 +54 0 3757 3.711679 279,18939,6798,11867,21315,4389,8397,23269 17.625000,17.625000,17.500000,17.125000,17.125000,17.000000,16.875000,16.875000 +54 1 67473 0.115610 67473,350,4509,431,26741,4264,17884,328 20.875000,16.375000,16.250000,16.000000,15.687500,15.250000,15.250000,15.187500 +54 2 323 0.386368 323,659,1154,304,2587,320,369,879 29.250000,28.000000,26.500000,26.125000,25.500000,25.000000,25.000000,24.375000 +54 3 58235 3.604206 572,1030,6342,10660,279,1062,9723,58235 21.000000,20.000000,20.000000,20.000000,19.500000,19.500000,19.500000,19.250000 +54 4 16263 1.010624 448,16263,304,438,14002,518,3156,279 25.750000,25.500000,24.500000,22.250000,21.750000,19.750000,19.625000,19.500000 +54 5 7355 6.781145 19571,12089,4389,11867,23654,20445,18939,24986 20.500000,18.125000,18.000000,18.000000,18.000000,17.875000,17.625000,17.625000 +54 6 1232 0.090221 1232,35203,26649,49535,356,38174,43854,60074 22.375000,18.750000,17.750000,17.750000,17.625000,17.000000,17.000000,16.375000 +54 7 812 0.000257 812,1442,1203,285,14810,2652,48211,139591 29.625000,20.375000,19.625000,19.500000,19.000000,18.000000,17.500000,17.500000 +55 0 672 -0.000000 672,19915,1360,14077,309,40878,87448,363 37.500000,21.625000,21.500000,21.375000,21.125000,20.125000,20.125000,20.000000 +55 1 1154 3.030670 323,659,1154,304,320,2587,16448,6762 29.375000,28.375000,26.750000,25.875000,24.750000,24.500000,23.500000,23.250000 +55 2 49975 10.025879 879,323,6693,448,4389,264,6798,7801 20.875000,20.500000,19.000000,18.375000,18.125000,18.000000,18.000000,17.625000 +55 3 1863 0.692371 1863,98974,506,13646,96882,14259,20696,431 19.750000,19.250000,16.375000,16.250000,16.250000,16.000000,16.000000,15.687500 +55 4 12707 0.951918 67592,12707,437,292,517,3923,16030,278 21.375000,21.250000,19.375000,18.625000,18.250000,17.000000,17.000000,16.750000 +55 5 1154 2.696886 323,1154,320,659,609,304,481,2587 29.250000,26.625000,21.625000,20.875000,20.125000,19.625000,19.625000,19.625000 +55 6 13928 7.981901 323,4389,11867,18939,6798,7801,3757,18095 23.125000,17.875000,17.875000,17.875000,17.750000,17.750000,17.500000,17.375000 +55 7 17734 13.498407 472,1260,45196,4483,350,444,2932,356 19.500000,19.375000,19.375000,19.250000,18.375000,17.750000,17.500000,17.125000 +56 0 1154 2.466133 323,1154,263,659,320,609,2587,1842 29.000000,26.625000,21.625000,21.250000,20.375000,20.000000,19.750000,19.250000 +56 1 55130 12.311621 323,11867,7801,18939,4389,6798,3757,21315 22.750000,18.000000,17.875000,17.875000,17.750000,17.750000,17.500000,17.500000 +56 2 23905 0.652473 23905,7801,362,730,444,4598,31071,17426 20.500000,19.125000,17.625000,17.500000,17.250000,17.250000,16.875000,16.750000 +56 3 416 0.131050 416,265,2718,388,2892,5963,5482,74 24.875000,22.000000,21.875000,20.625000,19.875000,19.125000,19.000000,18.125000 +56 4 1154 1.911043 323,1154,659,320,609,2587,304,16448 30.875000,29.125000,23.125000,21.625000,21.375000,21.375000,20.125000,19.625000 +56 5 59310 6.985514 323,11867,4389,7801,18939,3757,6798,21315 22.250000,18.250000,18.000000,18.000000,18.000000,17.750000,17.750000,17.625000 +56 6 80664 7.387667 25164,10978,4483,33031,36130,467,431,425 20.625000,19.750000,19.000000,19.000000,18.500000,18.375000,18.250000,17.750000 +56 7 323 0.161821 323,1154,659,320,609,2587,304,1842 30.625000,28.875000,23.375000,22.625000,21.875000,21.500000,20.500000,20.500000 +57 0 67034 6.638327 3800,11867,18939,20445,4389,6798,7801,8364 18.750000,18.125000,17.750000,17.750000,17.625000,17.500000,17.375000,17.375000 +57 1 10926 10.155075 10801,4312,386,3128,4483,26234,7401,472 22.250000,22.000000,21.875000,20.875000,18.125000,18.000000,17.875000,17.625000 +57 2 659 0.104498 659,304,16448,1154,2587,6762,323,72 26.125000,22.625000,21.625000,21.375000,21.375000,21.125000,20.875000,20.750000 +57 3 715 5.769854 758,1260,425,576,5301,24885,4710,43562 24.250000,23.500000,23.375000,21.625000,21.000000,20.750000,19.625000,19.625000 +57 4 758 3.454901 284,425,1260,8397,785,33,641,758 20.125000,19.500000,18.750000,18.500000,18.250000,18.125000,18.125000,18.000000 +57 5 220 0.095837 220,279,5256,3217,5813,6702,7400,5470 27.250000,24.500000,22.250000,21.250000,21.250000,21.125000,21.000000,20.875000 +57 6 17 0.000748 17,16,1154,15,18,220,279,20 27.250000,20.000000,14.562500,14.500000,14.437500,14.125000,14.062500,13.000000 +57 7 15 0.000002 15,16,220,24,2624,18,23,17 28.875000,13.750000,13.000000,12.437500,12.375000,12.250000,12.187500,11.937500 +58 0 15 0.014217 15,16,17,220,1154,20,21,549 26.875000,22.625000,16.875000,13.187500,12.687500,11.937500,11.625000,11.625000 +58 1 21 0.409723 21,22,20,23,24,19,18,17 24.000000,22.750000,21.875000,21.250000,20.250000,18.750000,18.500000,17.750000 +58 2 1154 2.889454 425,566,1154,569,806,323,8397,1260 28.875000,26.375000,26.125000,22.250000,21.250000,20.500000,20.500000,19.875000 +58 3 425 0.090849 425,566,8397,806,1260,279,1283,264 28.625000,26.250000,21.375000,20.375000,20.250000,19.625000,18.875000,18.750000 +58 4 10965 0.000008 10965,965,616,581,22604,2950,220,39337 29.750000,16.750000,16.125000,15.562500,15.062500,14.562500,14.437500,14.312500 +58 5 465 -0.000000 465,5045,56960,696,376,4160,1982,4198 43.000000,26.125000,25.250000,23.875000,23.750000,22.500000,22.250000,21.750000 +58 6 58235 4.152901 6342,572,9723,10859,3867,1030,58235,44777 24.750000,24.375000,23.875000,22.750000,22.625000,22.375000,21.625000,21.500000 +58 7 16263 4.863850 304,438,16263,448,518,389,569,14002 27.375000,26.500000,22.875000,22.125000,20.875000,20.375000,20.000000,19.625000 +59 0 1232 10.255280 4389,19571,11867,6798,8397,3757,18939,24986 18.000000,18.000000,17.750000,17.625000,17.375000,17.250000,17.250000,17.250000 +59 1 812 0.003386 812,380,3187,5853,72,28300,1442,483 23.875000,17.000000,16.000000,15.812500,14.500000,14.500000,14.437500,14.375000 +59 2 672 0.000004 672,19915,365,363,8635,6379,10303,23397 33.500000,20.500000,18.875000,17.500000,17.500000,17.250000,17.000000,16.500000 +59 3 304 0.767990 304,323,1549,1154,438,518,320,389 29.000000,28.750000,27.625000,26.625000,24.625000,24.000000,23.625000,22.250000 +59 4 279 0.085732 279,264,576,2441,458,806,13293,330 25.750000,22.625000,21.125000,20.875000,20.125000,19.125000,19.000000,18.875000 +59 5 1486 0.616413 1486,12425,4531,6430,18096,2805,4312,22358 22.750000,21.250000,21.000000,19.875000,19.375000,19.250000,19.250000,19.250000 +59 6 93736 12.688566 576,1154,425,434,362,330,422,358 19.625000,17.125000,16.750000,16.625000,16.500000,16.375000,16.375000,16.250000 +59 7 5326 1.882917 1154,518,5326,553,304,323,659,315 25.750000,25.125000,24.875000,24.500000,23.375000,23.375000,23.125000,22.250000 +60 0 553 0.001316 553,323,369,304,518,448,438,1154 29.875000,22.875000,21.250000,20.125000,20.000000,18.500000,18.375000,18.250000 +60 1 4389 3.760677 6798,21315,29405,14991,4389,11867,18095,33121 18.500000,18.500000,17.750000,17.500000,17.375000,17.375000,17.375000,17.125000 +60 2 45107 1.309662 506,45107,1967,431,472,425,25369,25164 21.875000,21.000000,17.875000,17.500000,17.500000,17.000000,16.875000,16.625000 +60 3 27507 0.000805 27507,659,1154,481,323,42241,8108,71 29.875000,22.375000,21.000000,20.250000,19.125000,18.625000,16.625000,16.125000 +60 4 659 1.622293 323,518,659,1154,2587,304,369,892 29.250000,29.000000,28.875000,28.875000,27.750000,25.750000,25.625000,25.125000 +60 5 1260 1.187468 1260,1096,576,758,425,1084,5301,2938 23.625000,23.500000,23.000000,22.750000,21.875000,21.000000,20.125000,19.750000 +60 6 9723 2.923809 572,1083,6342,1030,9723,10660,702,1221 23.875000,23.000000,22.750000,22.375000,21.875000,21.125000,21.000000,20.875000 +60 7 389 4.755703 304,438,389,518,448,16263,1549,26753 28.500000,26.250000,23.875000,23.750000,23.375000,22.625000,22.500000,21.375000 +61 0 264 4.173363 279,12425,6430,264,4171,18096,576,3807 24.250000,22.875000,22.250000,20.500000,20.125000,20.125000,19.500000,19.000000 +61 1 220 3.273695 12425,1372,4013,220,6430,18096,2805,2086 21.625000,21.125000,20.375000,19.750000,19.625000,19.500000,19.250000,19.000000 +61 2 17 0.015329 17,16,18,19,22,20,21,24 28.750000,24.125000,22.750000,21.500000,21.500000,21.375000,21.125000,20.625000 +61 3 15 0.000267 15,16,17,18,19,20,24,22 27.500000,17.750000,17.125000,16.875000,16.750000,16.500000,16.500000,16.250000 +61 4 15 0.023333 15,16,17,5534,339,5768,21,5813 27.750000,24.000000,17.750000,15.250000,14.687500,14.687500,14.625000,14.500000 +61 5 21 1.399513 22,21,23,24,20,19,18,17 23.250000,22.375000,21.625000,20.250000,18.875000,18.125000,17.750000,16.875000 +61 6 9234 1.468105 12425,9234,4013,5670,18096,481,3200,6430 20.500000,20.375000,18.750000,18.500000,18.250000,18.000000,18.000000,17.625000 +61 7 315 0.001373 315,24849,304,389,1154,19867,320,369 30.375000,22.625000,21.500000,21.125000,21.000000,20.750000,20.375000,20.125000 +62 0 279 0.133627 279,576,7506,18096,48822,18635,15696,6326 23.000000,20.250000,17.875000,17.750000,17.625000,17.375000,17.125000,16.750000 +62 1 12425 0.665522 12425,18096,22358,4013,7855,13434,80366,19584 22.625000,21.750000,20.375000,20.250000,20.000000,18.875000,18.875000,18.375000 +62 2 4013 0.027206 4013,1473,2025,18136,19584,22358,6146,80366 27.125000,22.250000,22.125000,22.000000,21.250000,20.750000,19.375000,19.375000 +62 3 1154 3.944342 576,9414,7506,18635,48822,85450,15696,3089 17.625000,16.000000,15.937500,15.875000,15.875000,15.875000,15.750000,15.500000 +62 4 53843 6.701771 576,323,892,279,220,448,48822,2670 19.000000,18.375000,17.750000,17.125000,17.000000,16.750000,16.500000,16.375000 +62 5 1154 0.638203 1154,659,320,2587,323,549,304,438 26.500000,26.000000,24.125000,23.875000,23.000000,22.875000,22.500000,22.000000 +62 6 8110 8.281076 323,438,5619,304,892,1380,448,714 25.000000,24.500000,24.375000,24.250000,23.500000,22.250000,22.125000,21.375000 +62 7 553 0.010138 553,304,2937,19620,264,448,279,705 29.250000,24.250000,22.375000,21.750000,21.500000,21.125000,20.500000,20.500000 +63 0 264 0.165865 264,806,2441,458,279,26255,12783,5619 26.500000,23.250000,23.250000,22.875000,22.625000,22.125000,21.875000,20.750000 +63 1 3476 0.553259 3476,8640,220,12613,39400,45250,6388,949 23.750000,22.250000,21.875000,20.875000,20.750000,20.750000,20.125000,19.625000 +63 2 304 0.084801 304,438,389,315,5619,448,518,369 29.375000,26.750000,24.750000,23.125000,22.500000,22.375000,22.000000,21.125000 +63 3 279 0.073317 279,264,220,576,458,2441,18096,1154 26.125000,22.750000,21.375000,21.125000,20.750000,20.750000,18.750000,18.500000 +63 4 220 0.634710 220,12425,1486,4531,18096,4013,22358,4565 23.250000,22.250000,21.750000,20.625000,20.000000,19.250000,19.000000,18.250000 +63 5 17 0.000715 17,16,18,19,22,20,21,23 28.250000,20.625000,18.750000,17.875000,17.875000,17.625000,17.375000,16.875000 +63 6 15 0.000004 15,16,17,1154,549,18,23,1635 30.000000,16.875000,14.625000,14.312500,13.437500,13.375000,13.312500,13.062500 +63 7 15 0.002481 15,16,17,21,22,481,220,1154 28.000000,22.000000,14.312500,13.250000,12.625000,12.500000,12.437500,12.312500 +64 0 22 0.341499 22,21,23,24,20,19,18,17 24.500000,23.375000,21.875000,19.750000,17.375000,16.000000,15.562500,14.812500 +64 1 33496 6.025534 12425,4013,4531,1486,9234,18096,22358,5670 22.000000,20.875000,20.625000,20.125000,19.250000,18.750000,18.625000,18.000000 +64 2 5670 0.207182 5670,1486,53148,5068,1473,33085,56005,23675 25.000000,22.875000,22.250000,20.375000,19.375000,18.875000,18.875000,18.750000 +64 3 315 0.013435 315,1154,576,659,518,8755,323,43562 27.125000,20.875000,20.500000,20.250000,20.125000,19.375000,19.125000,18.875000 +64 4 2585 7.463917 279,576,264,6798,362,1154,3757,11867 22.250000,20.500000,17.500000,17.250000,16.875000,16.875000,16.875000,16.875000 +64 5 311 1.712418 358,311,1205,279,1260,2806,23309,3155 22.625000,21.500000,20.875000,20.250000,19.125000,18.625000,18.500000,18.375000 +64 6 70627 11.506456 12190,4093,2126,2823,13655,387,11158,22749 21.250000,20.375000,19.875000,18.750000,18.750000,18.375000,18.375000,18.250000 +64 7 15540 6.837259 304,264,659,1154,518,5326,549,323 23.125000,22.375000,22.000000,22.000000,21.875000,21.625000,21.500000,21.375000 +65 0 553 0.000324 553,323,518,304,1154,448,569,3216 29.875000,21.250000,19.875000,18.625000,18.500000,18.375000,17.875000,17.500000 +65 1 27878 8.399975 4389,21315,18939,3757,11867,7801,20445,6798 18.625000,18.125000,17.750000,17.375000,17.125000,17.000000,17.000000,16.875000 +65 2 645 0.000069 645,25503,19310,472,72,6535,220,6023 29.625000,18.625000,17.875000,17.500000,17.375000,17.250000,17.125000,17.125000 +65 3 431 0.114969 431,659,451,29938,425,328,5807,444 19.875000,16.375000,16.000000,15.062500,15.000000,14.750000,14.687500,14.375000 +65 4 84434 0.139278 84434,13,12523,2667,4076,26378,9605,1021 21.875000,19.250000,18.375000,17.250000,15.812500,15.625000,15.500000,15.437500 +65 5 659 0.678722 659,323,518,1154,304,2587,369,320 28.375000,27.875000,26.750000,25.750000,25.000000,24.375000,24.000000,23.375000 +65 6 2585 10.259563 1260,758,425,576,5301,715,1096,220 24.500000,24.375000,22.625000,21.375000,21.375000,21.125000,21.000000,19.875000 +65 7 311 0.004797 311,2014,279,1521,566,653,374,569 25.375000,19.125000,17.875000,17.125000,16.750000,16.750000,16.125000,16.125000 +66 0 70627 0.000408 70627,40000,69719,45975,12886,96894,53677,42705 27.000000,18.000000,18.000000,16.750000,15.937500,15.500000,15.000000,14.625000 +66 1 572 0.144394 572,374,10613,1154,84070,8930,364,1030 27.000000,23.500000,23.125000,22.875000,22.750000,22.500000,22.000000,22.000000 +66 2 10660 0.203054 10660,8947,264,1083,50057,10449,6342,2937 25.000000,21.875000,21.625000,21.375000,21.375000,21.125000,20.125000,19.875000 +66 3 518 0.138224 518,304,553,389,438,448,504,1156 30.000000,28.000000,25.000000,23.750000,23.000000,23.000000,21.500000,21.500000 +66 4 14079 10.139832 279,576,7148,2176,264,2055,26221,350 25.250000,20.000000,19.000000,18.500000,18.250000,18.125000,17.625000,17.125000 +66 5 26221 0.005322 26221,33496,37471,576,46113,323,4678,1154 26.125000,20.625000,18.000000,17.625000,17.500000,17.125000,16.750000,16.375000 +66 6 304 0.459089 304,323,659,1154,320,2587,364,448 27.875000,26.875000,25.750000,25.250000,22.875000,21.750000,21.500000,21.250000 +66 7 279 6.399178 7148,220,279,79593,58546,107074,2932,4312 25.375000,21.500000,19.000000,17.125000,16.500000,16.000000,15.875000,15.750000 +67 0 7148 1.348850 6424,7148,3639,4312,220,1852,4882,9710 23.000000,22.375000,22.125000,19.500000,18.250000,18.125000,17.875000,17.750000 +67 1 84203 0.019958 84203,65741,659,323,1154,3082,220,4312 29.000000,24.875000,22.875000,21.250000,21.250000,21.000000,20.125000,19.250000 +67 2 315 0.000298 315,4940,82,659,1154,220,320,2587 25.000000,16.000000,14.875000,13.875000,13.875000,13.437500,13.437500,12.875000 +67 3 472 4.328083 95733,82468,34982,46943,68929,46331,31302,472 22.250000,22.125000,20.125000,20.000000,20.000000,19.875000,19.375000,18.875000 +67 4 42057 0.987804 1624,42057,3249,8985,546,529,4681,994 26.750000,26.250000,22.875000,19.000000,18.375000,18.125000,17.875000,17.250000 +67 5 43674 -0.000000 43674,659,76,8155,1763,1306,25013,19501 35.500000,20.625000,20.250000,19.500000,19.375000,18.875000,18.375000,18.250000 +67 6 323 0.023401 323,609,659,1154,481,304,569,608 30.750000,26.250000,26.250000,23.500000,22.125000,21.500000,21.250000,20.875000 +67 7 47450 0.000929 47450,659,5599,1154,279,434,4312,14010 27.500000,19.750000,19.125000,17.750000,17.500000,16.875000,16.500000,16.375000 +68 0 5604 0.000084 5604,8187,6660,73493,13515,72116,13,71 29.750000,19.625000,19.000000,18.375000,17.875000,15.437500,14.312500,14.250000 +68 1 659 0.346903 659,323,1154,304,2587,16448,320,13 29.125000,27.875000,26.125000,25.875000,24.875000,24.125000,23.750000,23.125000 +68 2 425 2.114649 758,1260,425,715,576,4710,5301,1096 24.125000,24.125000,23.000000,22.375000,20.750000,20.625000,20.625000,19.625000 +68 3 10965 0.000006 10965,616,965,6344,581,220,13,2950 29.625000,15.750000,15.750000,15.562500,15.125000,14.500000,14.062500,14.062500 +68 4 465 -0.000000 465,1536,5045,56960,665,696,1982,3667 40.000000,23.500000,23.250000,22.375000,21.875000,21.875000,21.375000,21.375000 +68 5 58235 4.778940 572,1083,6342,9723,1221,364,702,1030 23.875000,23.250000,23.000000,22.625000,22.375000,22.125000,22.125000,21.875000 +68 6 304 0.178432 304,16263,438,448,1549,518,389,14002 26.500000,24.000000,23.875000,22.750000,21.375000,20.500000,20.250000,19.750000 +68 7 1378 5.774031 279,264,220,2441,576,458,1378,3807 25.875000,23.500000,21.750000,20.750000,20.625000,20.375000,20.250000,20.250000 +69 0 12351 3.910610 11088,220,12425,53148,803,1008,2155,12783 22.000000,21.375000,20.875000,20.625000,20.500000,20.500000,20.125000,20.000000 +69 1 304 0.842764 304,549,1154,5880,2337,2587,448,15540 26.875000,26.750000,25.125000,23.500000,23.250000,23.000000,22.875000,22.750000 +69 2 220 0.053831 220,279,806,892,419,4124,1154,3309 27.375000,24.375000,20.250000,20.250000,19.500000,19.500000,19.250000,19.000000 +69 3 17 0.000345 17,16,15,1154,220,18,279,320 26.000000,18.000000,12.375000,12.312500,11.875000,11.625000,10.875000,10.875000 +69 4 15 -0.000000 15,16,17,18,220,20,23,1112 29.375000,12.562500,12.062500,11.625000,11.187500,10.812500,10.625000,10.562500 +69 5 15 0.018167 15,16,17,220,1154,481,549,198 25.875000,21.875000,14.500000,11.812500,11.500000,10.687500,10.562500,10.000000 +69 6 23 1.134024 22,23,24,21,20,19,18,17 23.125000,22.500000,20.500000,19.875000,18.000000,17.750000,17.500000,16.750000 +69 7 1154 2.915783 549,1154,2587,659,323,481,1177,1959 28.875000,26.125000,25.875000,25.625000,24.250000,23.500000,23.250000,23.000000 +70 0 6059 9.341125 2670,576,2176,279,825,5619,323,304 21.500000,21.125000,20.875000,20.500000,20.125000,19.625000,19.375000,19.250000 +70 1 4145 0.110571 4145,315,8960,323,15623,18306,220,32981 24.125000,19.875000,19.875000,19.750000,19.625000,19.500000,19.000000,18.750000 +70 2 4892 2.707407 323,4892,48438,57679,320,549,220,304 24.750000,22.375000,22.375000,22.000000,21.750000,20.625000,20.250000,19.750000 +70 3 30204 0.000919 30204,652,1902,7132,65,7964,82,1330 34.750000,27.125000,27.000000,20.000000,18.000000,16.875000,16.250000,15.375000 +70 4 553 8.252722 323,320,1154,304,448,549,15540,5326 28.750000,26.375000,24.875000,23.375000,23.250000,23.000000,22.375000,21.750000 +70 5 56237 7.056473 11867,4389,6798,3757,21315,7538,18939,14991 17.625000,17.375000,17.375000,17.250000,17.250000,17.125000,17.125000,17.000000 +70 6 12095 9.626516 4389,11867,3757,6798,18939,21315,6898,33121 17.125000,17.125000,17.000000,17.000000,16.875000,16.750000,16.625000,16.500000 +70 7 1967 0.096632 1967,67912,66097,1863,69231,350,85656,37705 20.750000,17.125000,16.250000,15.500000,15.312500,15.250000,15.000000,14.875000 +71 0 544 4.164993 288,544,50,64,300,51,4422,73814 20.375000,16.250000,14.875000,14.687500,14.375000,13.937500,13.937500,13.562500 +71 1 72 0.007160 72,285,6591,9156,98255,4738,645,859 30.500000,25.500000,22.000000,20.875000,20.375000,20.125000,20.000000,20.000000 +71 2 1154 2.487192 323,1154,320,304,2587,659,481,549 29.000000,26.625000,25.000000,22.875000,22.500000,22.000000,21.000000,20.875000 +71 3 323 0.090385 323,304,892,264,448,1380,39400,369 26.750000,23.500000,23.125000,21.250000,21.125000,20.500000,20.500000,20.375000 +71 4 4320 6.956036 576,279,264,304,13293,362,1154,425 19.875000,19.375000,17.500000,17.000000,17.000000,16.875000,16.625000,16.500000 +71 5 792 2.388796 944,792,364,1405,10563,3406,64,10810 20.875000,18.750000,18.125000,17.000000,15.875000,15.500000,15.437500,15.437500 +71 6 53175 1.880667 1154,53175,304,323,594,481,553,32342 20.375000,19.875000,19.625000,19.125000,18.750000,18.500000,18.250000,18.000000 +71 7 15540 3.252323 553,1154,15540,659,320,5326,304,549 27.500000,25.375000,24.500000,24.125000,24.000000,22.750000,22.375000,22.125000 +72 0 553 0.000980 553,323,304,1154,369,448,659,3216 29.625000,22.500000,19.375000,19.000000,18.250000,17.750000,17.750000,17.625000 +72 1 11959 7.590174 21315,6798,7801,14991,18939,35757,3757,12066 17.125000,16.750000,16.750000,16.750000,16.625000,16.625000,16.375000,16.250000 +72 2 398 0.038385 398,28206,2577,6355,8346,6743,46716,685 24.250000,20.875000,18.250000,16.375000,15.812500,15.687500,15.687500,15.625000 +72 3 88353 2.539148 13612,88353,3984,20042,468,14595,29660,356 17.125000,15.937500,15.500000,15.312500,15.187500,15.125000,15.125000,14.937500 +72 4 659 0.338022 659,323,1154,2587,16448,448,304,320 28.375000,27.000000,25.625000,25.375000,23.625000,23.250000,23.000000,22.625000 +72 5 758 0.963053 758,1260,425,11733,715,576,5301,220 24.000000,23.875000,22.625000,21.875000,21.625000,21.375000,21.125000,20.250000 +72 6 3217 7.157238 220,279,2176,264,5256,419,806,1493 26.375000,24.125000,22.625000,21.375000,21.125000,20.625000,20.500000,20.500000 +72 7 220 0.009319 220,315,323,1154,279,481,311,198 27.875000,22.250000,21.875000,21.625000,19.250000,18.875000,18.750000,18.625000 +73 0 17 0.000109 17,16,18,220,15,1154,19,20 27.000000,17.750000,14.437500,14.125000,13.625000,12.562500,12.250000,12.187500 +73 1 15 -0.000000 15,16,2624,78,17,220,18,21 33.250000,14.750000,13.625000,13.250000,13.125000,12.687500,12.000000,12.000000 +73 2 15 0.142752 15,16,17,220,1154,11,23,198 23.625000,21.750000,14.062500,10.500000,10.062500,9.500000,8.750000,8.625000 +73 3 23 0.903038 24,23,22,21,20,15,19,18 22.875000,22.500000,17.875000,15.875000,14.312500,13.937500,13.875000,13.625000 +73 4 1154 0.842928 1154,425,566,806,264,323,279,432 28.125000,27.875000,27.500000,21.375000,20.875000,20.750000,20.375000,20.375000 +73 5 425 0.640942 425,566,806,279,8397,264,432,1260 26.625000,26.500000,20.625000,20.375000,20.125000,20.000000,19.750000,19.750000 +73 6 10965 -0.000000 10965,616,965,2950,581,1080,3431,1624 32.250000,18.375000,16.625000,16.125000,15.812500,15.500000,15.500000,15.437500 +73 7 465 -0.000000 465,5045,665,696,56960,1982,3667,59385 38.500000,23.000000,20.750000,20.500000,20.375000,19.875000,19.750000,19.250000 +74 0 1865 3.828510 572,6342,9723,3867,10859,1030,10660,3949 24.375000,23.000000,23.000000,22.250000,21.875000,21.750000,21.750000,21.625000 +74 1 264 1.145658 806,264,458,279,26255,2441,825,12425 28.625000,28.000000,26.625000,23.250000,22.625000,21.375000,21.125000,21.000000 +74 2 8640 0.529167 8640,9814,94056,12425,2805,825,3281,7743 23.750000,21.875000,21.875000,21.375000,21.000000,20.125000,19.500000,19.500000 +74 3 11094 0.027109 11094,569,3476,26255,12425,39400,5068,6774 25.750000,21.500000,19.625000,19.250000,18.875000,18.875000,18.625000,18.500000 +74 4 389 0.574217 389,304,518,438,369,448,320,315 29.250000,28.875000,26.125000,25.750000,23.875000,23.875000,22.375000,21.750000 +74 5 264 4.566364 279,576,264,6326,18096,458,48822,1154 24.625000,20.500000,20.125000,19.375000,19.000000,18.875000,18.125000,17.875000 +74 6 1378 5.607271 12425,4013,220,3175,18096,3281,3887,9234 21.125000,20.250000,20.125000,20.000000,20.000000,19.875000,19.750000,19.750000 +74 7 569 2.346378 481,28037,12,569,9234,949,29509,21231 20.625000,20.375000,19.125000,19.125000,16.750000,16.500000,16.000000,15.687500 +75 0 12 0.000105 12,569,313,36519,28,1154,11,481 30.625000,19.375000,19.250000,19.250000,19.000000,18.875000,18.375000,18.375000 +75 1 31 0.000040 31,62799,569,19191,26,19347,323,39291 31.875000,21.000000,20.875000,17.875000,16.750000,16.750000,16.500000,16.250000 +75 2 949 0.805830 9234,949,2003,6460,481,1899,2254,3668 25.250000,25.125000,21.250000,21.250000,21.125000,20.750000,20.000000,19.500000 +75 3 9234 0.084539 9234,3281,4013,12425,315,18096,18635,17770 23.875000,19.875000,19.750000,19.750000,18.375000,18.250000,18.125000,17.625000 +75 4 15580 14.443222 315,304,24849,389,320,19867,1154,369 30.000000,23.500000,23.250000,23.125000,22.125000,22.000000,21.875000,21.000000 +75 5 315 0.098663 315,389,304,369,320,1154,24849,438 29.375000,26.500000,26.125000,23.125000,22.625000,22.375000,22.125000,21.875000 +75 6 279 0.045074 279,576,6326,7506,48822,15696,5748,18635 24.625000,20.750000,19.375000,18.625000,18.125000,17.625000,17.500000,17.500000 +75 7 12425 0.436167 12425,18096,4013,1293,7855,22358,13434,5411 23.250000,21.250000,21.125000,20.250000,20.250000,19.125000,18.875000,18.750000 +76 0 4013 0.006680 4013,19584,1473,2025,18136,6146,22358,80366 28.375000,22.250000,21.750000,21.375000,21.375000,20.250000,20.125000,19.500000 +76 1 467 6.097881 1154,576,7506,6326,18635,52325,48822,15696 19.625000,18.375000,17.750000,17.625000,17.625000,17.500000,16.875000,16.750000 +76 2 1765 0.070820 1765,16,3104,6179,2317,27313,449,4872 22.625000,19.000000,18.750000,17.750000,17.000000,16.125000,15.750000,15.687500 +76 3 279 0.011339 279,576,659,1154,29092,21192,30833,12826 26.625000,22.125000,16.875000,16.500000,16.375000,15.562500,15.187500,14.937500 +76 4 15021 0.004673 15021,5593,7840,17829,13532,350,82988,37806 22.000000,14.500000,14.125000,14.062500,13.937500,13.500000,13.187500,12.875000 +76 5 1154 1.428856 659,1154,320,2587,323,304,438,448 27.125000,26.375000,24.875000,24.750000,24.375000,24.250000,23.750000,23.750000 +76 6 8110 5.414007 892,5619,304,323,1380,448,6342,16263 23.750000,23.375000,23.250000,22.750000,22.375000,21.625000,21.250000,20.750000 +76 7 553 0.007786 553,304,2937,19620,264,448,389,7069 28.250000,23.125000,20.875000,20.625000,19.250000,18.875000,18.125000,18.125000 +77 0 458 3.763317 264,806,458,2441,279,26255,12783,5619 26.125000,22.750000,22.500000,22.375000,21.750000,21.375000,21.125000,20.625000 +77 1 11094 0.089790 11094,9234,5107,15358,11577,61888,650,5813 25.000000,21.625000,20.625000,20.000000,19.375000,19.125000,18.875000,18.625000 +77 2 389 2.058479 304,389,438,518,448,315,279,369 30.250000,28.375000,26.500000,26.125000,24.000000,23.625000,23.500000,23.125000 +77 3 279 0.288771 279,264,458,576,2441,220,18096,7506 24.500000,22.875000,21.250000,20.625000,20.250000,19.250000,18.500000,18.375000 +77 4 12425 0.742733 12425,220,4013,18096,22358,1378,2086,9234 23.000000,22.250000,21.000000,20.125000,19.625000,19.500000,19.250000,19.125000 +77 5 4013 0.010630 4013,1473,18136,19584,2025,22358,6146,13420 27.250000,21.250000,21.000000,21.000000,20.875000,19.500000,18.875000,18.750000 +77 6 28778 6.603430 576,1154,9414,7506,659,48822,15696,59455 17.875000,17.750000,16.500000,16.125000,16.000000,15.812500,15.500000,15.500000 +77 7 85393 0.289145 85393,2249,533,344,2221,49675,1524,3936 30.375000,29.000000,27.875000,21.625000,21.375000,20.875000,16.750000,16.500000 +78 0 304 0.277006 304,320,659,1154,323,892,438,549 27.625000,25.625000,25.375000,23.750000,22.875000,22.750000,22.125000,22.125000 +78 1 6702 4.387726 220,279,5534,6122,5768,6156,6527,6702 25.750000,23.375000,22.250000,22.250000,21.875000,21.875000,21.625000,21.625000 +78 2 220 0.032499 220,315,659,323,279,429,1154,16448 29.125000,25.500000,23.375000,21.750000,21.625000,21.625000,21.375000,21.250000 +78 3 17 0.000008 17,16,220,15,18,323,1154,315 29.000000,16.250000,14.875000,14.500000,14.500000,13.437500,13.437500,13.187500 +78 4 15 -0.000000 15,78,2624,13,659,220,16,1154 28.250000,11.312500,11.250000,10.875000,10.500000,10.062500,10.000000,9.375000 +78 5 15 0.004080 15,16,17,1154,23,220,659,2587 28.500000,23.000000,13.875000,13.687500,12.562500,11.875000,11.875000,11.625000 +78 6 23 0.033768 23,24,22,21,19,20,18,15 25.375000,22.000000,16.125000,13.875000,13.062500,13.000000,12.625000,12.375000 +78 7 659 0.068289 659,16448,323,6762,1154,2587,320,13 29.000000,25.250000,25.125000,24.250000,24.000000,23.375000,22.625000,21.875000 +79 0 1260 1.453676 758,1260,425,715,4710,5301,220,576 25.000000,24.250000,23.500000,23.375000,21.625000,21.000000,20.375000,20.375000 +79 1 1030 2.365362 572,1083,9723,6342,1030,58235,1221,1865 23.000000,22.250000,22.000000,21.875000,21.750000,20.750000,20.375000,20.375000 +79 2 264 0.023874 264,458,2441,806,279,45250,12783,8922 28.125000,22.875000,22.875000,22.125000,21.750000,21.500000,21.250000,20.875000 +79 3 45250 2.588278 3476,8640,12613,45250,8922,2613,6388,39400 23.375000,22.125000,21.500000,21.500000,21.000000,20.625000,20.625000,19.875000 +79 4 3476 0.007931 3476,8640,12613,949,12425,323,3668,11094 26.125000,20.750000,18.375000,18.000000,18.000000,17.750000,17.625000,17.625000 +79 5 304 0.302860 304,438,389,315,504,5619,369,448 28.750000,27.250000,26.500000,24.125000,23.500000,23.250000,21.875000,21.750000 +79 6 5779 13.080235 279,220,576,264,28778,18096,458,4013 26.875000,21.500000,20.875000,20.750000,20.500000,18.750000,18.625000,18.625000 +79 7 17770 0.018698 17770,315,220,9234,23921,481,12425,5479 23.500000,18.000000,17.625000,16.875000,16.750000,16.500000,16.000000,15.875000 diff --git a/benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-after.csv b/benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-after.csv new file mode 100644 index 000000000..4af8aa480 --- /dev/null +++ b/benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-after.csv @@ -0,0 +1,9 @@ +model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlxcel_version,build_type,max_tokens,prompt,prompt_target_len,mlxcel_commit,mlx_commit,mlx_rocm_overlay_commit,hip_version +Qwen3-30B-A3B-4bit,models/mlx/Qwen3-30B-A3B-4bit,512,128,1594.26,321.15,2047.70,62.51,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,77afbb1d,81ba1c6a,75915908,7.15.26333 +NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,512,128,1697.73,301.58,1704.08,75.11,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,77afbb1d,81ba1c6a,75915908,7.15.26333 +Mixtral-8x7B-Instruct-v0.1-4bit,models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit,512,128,19685.12,26.01,13460.80,9.51,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,77afbb1d,81ba1c6a,75915908,7.15.26333 +Qwen3-30B-A3B-4bit,models/mlx/Qwen3-30B-A3B-4bit,512,128,1640.26,312.15,2042.01,62.68,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,77afbb1d,81ba1c6a,75915908,7.15.26333 +NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,512,128,1727.24,296.43,1704.59,75.09,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,77afbb1d,81ba1c6a,75915908,7.15.26333 +Mixtral-8x7B-Instruct-v0.1-4bit,models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit,512,128,19704.32,25.98,13373.37,9.57,2026-10-05,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,77afbb1d,81ba1c6a,75915908,7.15.26333 +Qwen3-30B-A3B-4bit,models/mlx/Qwen3-30B-A3B-4bit,512,128,1580.93,323.86,2059.29,62.16,2026-10-05,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,77afbb1d,81ba1c6a,75915908,7.15.26333 +NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,512,128,1922.90,266.26,1709.43,74.88,2026-10-05,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,77afbb1d,81ba1c6a,75915908,7.15.26333 diff --git a/benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-before.csv b/benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-before.csv new file mode 100644 index 000000000..0cf13db90 --- /dev/null +++ b/benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-before.csv @@ -0,0 +1,10 @@ +model,model_path,prompt_tokens,generated_tokens,prefill_ms,prefill_tok_s,decode_ms,decode_tok_s,date,hardware,mlxcel_version,build_type,max_tokens,prompt,prompt_target_len,mlxcel_commit,mlx_commit,mlx_rocm_overlay_commit,hip_version +Qwen3-30B-A3B-4bit,models/mlx/Qwen3-30B-A3B-4bit,512,128,1580.82,323.88,2088.43,61.29,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 +NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,512,128,1709.38,299.52,1722.86,74.30,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 +Mixtral-8x7B-Instruct-v0.1-4bit,models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit,512,128,19695.92,26.00,13581.69,9.42,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 +Qwen3-30B-A3B-4bit,models/mlx/Qwen3-30B-A3B-4bit,512,128,1554.75,329.31,2096.72,61.05,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 +NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,512,128,1723.77,297.02,1715.44,74.62,2026-10-04,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 +Mixtral-8x7B-Instruct-v0.1-4bit,models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit,512,128,19738.62,25.94,14092.28,9.08,2026-10-05,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 +Qwen3-30B-A3B-4bit,models/mlx/Qwen3-30B-A3B-4bit,512,128,1550.67,330.18,2070.36,61.83,2026-10-05,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 +NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,models/mlx/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit,512,128,1934.29,264.70,1721.16,74.37,2026-10-05,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 +Mixtral-8x7B-Instruct-v0.1-4bit,models/mlx/Mixtral-8x7B-Instruct-v0.1-4bit,512,128,19887.76,25.74,11737.33,10.91,2026-10-05,AMD_Radeon_8060S_Graphics_gfx1151_ROCm10.0.0_96GB,0.7.0,release,128,"Hello, how are you today?",512,57d8ed29,81ba1c6a,75915908,7.15.26333 diff --git a/docs/benchmark_results/rocm-correctness-gfx1151-2026-09-30.md b/docs/benchmark_results/rocm-correctness-gfx1151-2026-09-30.md index 8948637c5..4c21381c2 100644 --- a/docs/benchmark_results/rocm-correctness-gfx1151-2026-09-30.md +++ b/docs/benchmark_results/rocm-correctness-gfx1151-2026-09-30.md @@ -276,5 +276,6 @@ The Metal traces were produced with the loop in `benchmarks/logit_traces/rocm_gf - The second acceptance criterion of #1809 (decided-position mismatch reported per model and width against Metal) is now met for all nine models: fifteen of these sixteen pairs pass and `nemotron-3-nano-30b-a3b` `w1` is inconclusive, with the same checkpoint's `w8` and `w256` pairs passing. The two `w1ctx512` rows, which cover the fused SSM update kernel on Metal, also pass. - Kernel against kernel for the SSM update: the `w1ctx512` rows here compare Metal's kernel with ROCm's graph. The HIP port (#2067) added kernel-against-kernel rows, with zero decided-position mismatches for both hybrids, in [rocm-ssm-update-kernel-gfx1151-2026-10-04.md](rocm-ssm-update-kernel-gfx1151-2026-10-04.md#model-logits-kernel-against-kernel). +- Two MoE implementations in the Nemotron-H pairs: since the HIP fused MoE ports (#2065) ROCm's gate reads `moe_down_kernel_available()` and takes `fused_moe_forward`, as Metal does. New Nemotron-H traces on that path pass against these Metal traces at `w8` and `w1ctx512` with zero decided-position mismatches, in [rocm-fused-moe-gfx1151-2026-10-05.md](rocm-fused-moe-gfx1151-2026-10-05.md#model-logits). - `src/models/nemotron_h.rs` prints its loading messages to stdout, which puts five non-trace lines into every Nemotron-H trace and stops `compare_logit_traces.py`. Either the loader should log to stderr or the comparison script should skip them. - No noise floor was measured; the first run's reasoning for not needing one still applies. diff --git a/docs/benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md b/docs/benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md new file mode 100644 index 000000000..6cf169d58 --- /dev/null +++ b/docs/benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md @@ -0,0 +1,59 @@ +# HIP ports of the fused MoE decode kernels on gfx1151 (2026-10-05) + +lablup/mlxcel#2065, part of #1814. Before this change ROCm ran `gather_qmm` for every routed expert at single-token decode, because `moe_gateup_ports()` and `moe_down_ports()` had no `.rocm` entry and the SwitchGLU gate read the backend-wide `custom_kernels_available()`. The decode profile ([rocm-decode-profile-gfx1151-2026-09-30.md](rocm-decode-profile-gfx1151-2026-09-30.md)) put fused-MoE-reachable work at 46.8% of Qwen3-30B-A3B decode GPU time, but 42.1 points of that are expert GEMVs already running near the host's bandwidth, so it estimated about 5% of GPU time plus about 430 dispatches per token as the most a fused kernel could recover. + +The ports (`MOE_GATEUP_HIP_SOURCE`, `MOE_DOWN_HIP_SOURCE` in `src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp`) are the CUDA kernels with `__shfl_down(v, o, 32)` for the lane fold. `fused_moe_kernels_available()` and `moe_down_kernel_available()` read the two tables, and `switch_layers.rs` and `nemotron_h.rs` gate on them. On ROCm the rows-per-threadgroup default (`MLXCEL_FUSED_MOE_SGY`) is 2 instead of Metal's 8, for the reason in the next section. + +## Environment + +AMD Ryzen AI MAX+ 395 with Radeon 8060S (`gfx1151`, RDNA 3.5), 96 GiB VRAM carve-out, Debian 13, ROCm 10.0.0 (HIP 7.15.26333). Before: `origin/main` `57d8ed29`. After: the PR branch at `77afbb1d` on that main. MLX pin `81ba1c6a`, ROCm overlay `75915908`, `cargo build --release --features rocm`. Checkpoints: the same local `mlx-community` directories as the earlier correctness rows. + +## Decode throughput + +`scripts/bench_decode.sh` at its default pp512/tg128 shape, one model per run, before and after binaries alternated run by run (before, after, next model), three rounds. Every run went through `scripts/rocm_gpu_guard.sh` with the fix from this PR, and every attempt that overlapped another unit's GPU work or a compiler was rejected and rerun. Raw rows: `benchmarks/rocm_strixhalo-gfx1151_2026-10-05_fused-moe-before.csv` and `..._fused-moe-after.csv`. + +| Model | Path change | Before tok/s | After tok/s | Change (medians) | +|---|---|---|---|---| +| Qwen3-30B-A3B-4bit | `gather_qmm` to the fused HIP pair | 61.29 / 61.05 / 61.83, median 61.29 | 62.51 / 62.68 / 62.16, median 62.51 | +2.0% | +| NVIDIA-Nemotron-3-Nano-30B-A3B-4bit | `forward_nonfused` to `fused_moe_forward` (still `gather_qmm`) | 74.30 / 74.62 / 74.37, median 74.37 | 75.11 / 75.09 / 74.88, median 75.09 | +1.0% | +| Mixtral-8x7B-Instruct-v0.1-4bit | none (Dff 14336 is above the 8192 `MLXCEL_FUSED_MOE_MAX_DFF` default) | 9.42 / 9.08 / 10.91, median 9.42 | 9.51 / 9.57 (two runs: the third never found a clean GPU window before the host's other work resumed) | +1.3% (noise; same code) | + +Qwen3's gain is consistent run to run (every after run is above every before run) and matches the profile's ceiling: the fused pair removes the activation, weighted-sum and index-building dispatches but reads the same expert weights. Nemotron-H's 1% is near its run-to-run spread; its routed experts still run `gather_qmm` (the fused fc1 kernel is #2069), and only the surrounding combine changed path. Mixtral runs the same code before and after, so its spread (9.08 to 10.91 tok/s) is this host's noise for a 26 GB checkpoint on a 31 GiB host and says nothing about the change. `compare_bench_csv.py --allow-commit-change` on the per-model median rows reports a 1.016x median over the three pairs, none moved by more than 10%. Prefill is not on the fused path (a prompt is more than one token). + +### Why the ROCm SGY default is 2 + +With Metal's default of 8 rows per threadgroup, the port made Qwen3 decode slower than the path it replaces: one guarded pair before the default changed measured 61.64 tok/s on `gather_qmm` (`c0b71344`) against 59.42 tok/s fused. A sweep of `MLXCEL_FUSED_MOE_SGY` over 1, 2, 4, 8, 16 and 32 on one binary put 2 on top in every repetition (about 62.3 tok/s, against about 61 for `MLXCEL_FUSED_MOE=0`, about 61.2 for 1, and 59.3 to 60.4 for the rest). That sweep is indicative only: each of its nine guarded attempts saw another unit's GPU process in at least one sample, so the guard rejected all of them, and the table above, measured at the new default, is the result this page stands on. SGY shapes the threadgroup only; `fused_moe_geglu_kernel_bitwise_invariant_across_sgy` pins that the output does not depend on it. + +## Correctness + +### Kernel against references + +`fused_moe_parity_tests` runs on gfx1151 for the first time, gated on `fused_moe_kernels_available()` (a GPU backend without the ports now fails the test instead of skipping it). The fused output is within 3.0e-6 normalized RMS of an all-f32 dense reference and about 3.7e-3 from `gather_qmm`, which is itself about 3.3e-3 from that reference; bounds unchanged. A new case covers SwiGLU at 4/4, 8/8 and 4/6 bits on the Qwen3 shape, reaching the SwiGLU arm and the 6-bit down branch. Starting the down kernel's lane fold at 8 instead of 16 fails both reference tests (normalized RMS 0.63 and 0.70). + +### Model logits + +New traces: `benchmarks/logit_traces/rocm_gfx1151_77afbb1d/`. The fused kernels run only on a single-token forward, so `w1` (`1 128 8 0`) is the row that reaches them on Qwen3; at `w8` its `default` and `fused0` traces are identical. `python3 scripts/compare_logit_traces.py --decided 2.0`. + +| Model, window | Reference | Candidate | Top-1 disagreement | Decided mismatches | Largest gap | Perplexity ref / cand | +|---|---|---|---|---|---|---| +| Qwen3-30B-A3B, w1 | Metal fused (`metal_m1u_bec64748` default) | ROCm fused (HIP) | 1 / 128 | 0 / 74 | 0.688 | 75742.6 / 76499.3 | +| Qwen3-30B-A3B, w1 | Metal `gather_qmm` (fused0) | ROCm `gather_qmm` (fused0) | 5 / 128 | 0 / 70 | 0.750 | 76983.0 / 78852.2 | +| Qwen3-30B-A3B, w1 | ROCm `gather_qmm` (fused0) | ROCm fused | 3 / 128 | 0 / 74 | 0.062 | 78852.2 / 76499.3 | +| Qwen3-30B-A3B, w8 | Metal default | ROCm default | 17 / 640 | 0 / 319 | 0.500 | 10.921 / 10.907 | +| Mixtral-8x7B, w1 | Metal default | ROCm default (`gather_qmm`) | 0 / 128 | 0 / 17 | none | 1589.66 / 1588.28 | +| Nemotron-3-Nano, w1ctx512 | ROCm main (`rocm_gfx1151_96cbce84`) | this branch | 3 / 128 | 0 / 68 | 0.125 | 7.442 / 7.410 | +| Nemotron-3-Nano, w1ctx512 | Metal (`metal_m5_d1128266`) | this branch | 6 / 128 | 0 / 71 | 0.250 | 7.489 / 7.410 | +| Nemotron-3-Nano, w1ctx512 | Metal (`metal_m5_d1128266`) | ROCm main (control) | 3 / 128 | 0 / 71 | 0.250 | 7.489 / 7.442 | +| Nemotron-3-Nano, w8 | ROCm main | this branch | 17 / 640 | 0 / 283 | 0.500 | 10.983 / 10.980 | +| Nemotron-3-Nano, w8 | ROCm `rocm_gfx1151_c5fe9a16` (`forward_nonfused`) | this branch | 20 / 640 | 0 / 286 | 0.500 | 10.984 / 10.980 | +| Nemotron-3-Nano, w8 | Metal (`metal_m5_d1128266`) | this branch | 22 / 640 | 0 / 287 | 0.562 | 10.956 / 10.980 | + +Every row has zero disagreements on a decided position. With the HIP kernels Qwen3's single-token logits are closer to Metal's fused kernel (1 top-1 disagreement) than ROCm's `gather_qmm` was to Metal's (5). Nemotron-H's change of path (`fused_moe_forward` instead of `forward_nonfused`) moves a few undecided tokens on both windows, at reference gaps of 0.5 or less. The `w1` perplexities are large because those chunks have no context; the `w1ctx512` and `w8` rows are the meaningful ones for perplexity. + +The `gather_qmm` path is unchanged: the branch's Qwen3 `fused0` `w1` trace and Mixtral `w1` trace are byte-identical in every data row to `rocm_gfx1151_bec64748`'s `fused0` traces, and its Qwen3 `fused0` `w8` trace is byte-identical to one built on `main` `57d8ed29` (both differ from `bec64748` at `w8`, from changes that predate this PR). + +Greedy text: Nemotron-3-Nano generates the same 40-token greedy text with and without `MLXCEL_FUSED_MOE_RELU2=1`, which now declines to `gather_qmm` on ROCm instead of refusing (its fc1 kernel has no HIP port until #2069). + +## Not measured + +Gemma 4 (GeGLU) and Qwen3-Next reach the same kernels through the same gate but have no checkpoint on this host. Wave64 (CDNA) parts are untested: the `#error` guard the #1814 ports carry is inert with AMD clang 23, which defines neither `__AMDGCN_WAVEFRONT_SIZE` spelling, and the explicit shuffle width of 32 is what keeps each fold inside one row there. diff --git a/docs/benchmark_results/rocm-ssm-update-kernel-gfx1151-2026-10-04.md b/docs/benchmark_results/rocm-ssm-update-kernel-gfx1151-2026-10-04.md index 5e7e5b5ef..4da0e7314 100644 --- a/docs/benchmark_results/rocm-ssm-update-kernel-gfx1151-2026-10-04.md +++ b/docs/benchmark_results/rocm-ssm-update-kernel-gfx1151-2026-10-04.md @@ -10,7 +10,7 @@ AMD Ryzen AI MAX+ 395 with Radeon 8060S (`gfx1151`, RDNA 3.5), 96 GiB VRAM carve ## Decode throughput -`scripts/bench_decode.sh` at its default pp512/tg128 shape, one model per run, `MLXCEL_SSM_KERNEL=0` for the graph and unset for the kernel, on the same binary. Three runs per arm, the arm order alternated between runs. Every run went through `scripts/rocm_gpu_guard.sh` (90 s with `/sys/class/kfd/kfd/proc` empty and no compiler, then 1 Hz monitoring); a parallel development unit shared the GPU, and every attempt that overlapped its work was rejected and rerun. The first run used the guard on `main`, the rest the copy fixed in lablup/mlxcel#2098 (commit `c3eab7b2`), which stops rejecting the command's own exited children; the old guard's rejections were false, never its acceptances. Raw rows: `benchmarks/rocm_strixhalo-gfx1151_2026-10-04_ssm-kernel-off.csv` and `..._ssm-kernel-on.csv`. +`scripts/bench_decode.sh` at its default pp512/tg128 shape, one model per run, `MLXCEL_SSM_KERNEL=0` for the graph and unset for the kernel, on the same binary. Three runs per arm, the arm order alternated between runs. Every run went through `scripts/rocm_gpu_guard.sh` (90 s with `/sys/class/kfd/kfd/proc` empty and no compiler, then 1 Hz monitoring); a parallel development unit shared the GPU, and every attempt that overlapped its work was rejected and rerun. The first run used the guard on `main`, the rest the copy fixed in lablup/mlxcel#2098, which stops rejecting the command's own exited children; the old guard's rejections were false, never its acceptances. Raw rows: `benchmarks/rocm_strixhalo-gfx1151_2026-10-04_ssm-kernel-off.csv` and `..._ssm-kernel-on.csv`. | Model | Graph (`MLXCEL_SSM_KERNEL=0`) tok/s | Kernel tok/s | Speedup (medians) | |---|---|---|---| diff --git a/docs/installation.md b/docs/installation.md index 928e272f6..3b9eabc47 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -551,7 +551,7 @@ so `mlxcel inspect` and `--estimate-memory` give the same answers as before. | Affine 4-bit / 8-bit checkpoints | Run natively. | | mxfp8 and mxfp4 checkpoints | Run natively, including MoE experts through `gather_qmm` (for example gpt-oss-20b-MXFP4-Q4). The load log says so once per mode (`Quantization mode mxfp4: running on native ROCm kernels, no load-time conversion ...`). mxfp4 `quantize`, `quantized_matmul` and `gather_qmm` are checked against CPU references by `tests/rocm_mxfp4_quant.rs` (lablup/mlxcel#1808). Vendor FP8 block checkpoints (`quant_method: fp8`, 128x128 blocks) are requantized to mxfp8 at load and run natively; verified end to end on a dense 0.8B Qwen3.5 FP8 checkpoint, with mxfp8 `gather_qmm` and `quantized_matmul` checked by `models::switch_layers::mxfp_tests`. The official Qwen3.5 FP8 MoE releases have not been run on ROCm (lablup/mlxcel#1807). | | NVFP4 checkpoints | No native kernel. ModelOpt NVFP4 checkpoints (for example the Gemma 4 NVFP4 exports) are converted to affine 4-bit at load with no environment variables, and the load log names the route and the reason. A layer that cannot be converted fails the load with its name and the reason. MLX-native NVFP4 exports (`"mode": "nvfp4"` in `config.json`, such as `mlx-community/*-nvfp4`) have no load-time conversion and are refused at load; use an affine export instead (lablup/mlxcel#1806). | -| Affine MoE models (for example Qwen3-30B-A3B) | Run natively. Single-token decode takes the fused two-kernel MoE path (gate-up, then down) through its HIP port (lablup/mlxcel#2065), as on Metal and CUDA; `MLXCEL_FUSED_MOE=0` selects the `gather_qmm` graph path instead. Experts wider than `MLXCEL_FUSED_MOE_MAX_DFF` (8192 by default, as on CUDA; Mixtral-8x7B's 14336) stay on `gather_qmm`. | +| Affine MoE models (for example Qwen3-30B-A3B) | Run natively. Single-token decode takes the fused two-kernel MoE path (gate-up, then down) through its HIP port (lablup/mlxcel#2065), as on Metal and CUDA: Qwen3-30B-A3B-4bit decode on `gfx1151` went from 61.3 to 62.5 tok/s, with logits matching Metal's fused kernel at every decided position ([results](benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md)); `MLXCEL_FUSED_MOE=0` selects the `gather_qmm` graph path instead. Experts wider than `MLXCEL_FUSED_MOE_MAX_DFF` (8192 by default, as on CUDA; Mixtral-8x7B's 14336) stay on `gather_qmm`. | | mlxcel's fused kernels (sampling, fused norm, RoPE + KV append, paged attention) | Run as MLX graph fallbacks on the paths that have one (lablup/mlxcel#1803); ROCm ports are lablup/mlxcel#1814. | | Hybrid SSM decode (granite-4.0-h, falcon-h1, plamo-2, Nemotron-H) | The single-token Mamba2 SSM update runs the fused HIP kernel instead of the ~55-op SSD graph (lablup/mlxcel#2067): decode on `gfx1151` went from 60.4 to 88.4 tok/s on granite-4.0-h-tiny-4bit and from 51.4 to 74.4 tok/s on Nemotron-3-Nano-30B-A3B-4bit, with model logits matching the graph and Metal's kernel at every decided position ([results](benchmark_results/rocm-ssm-update-kernel-gfx1151-2026-10-04.md)). Measured on granite and Nemotron-H; falcon-h1 and plamo-2 take the same gate but have not been run on ROCm. `MLXCEL_SSM_KERNEL=0` forces the graph. | | GPU faults | Reported as errors (lablup/mlxcel#1804). A launch HIP rejects (an oversized block, no code object for the device) fails the evaluation that issued it and the device stays usable. An asynchronous fault (an out-of-bounds access) fails the evaluation waiting on it within about a second instead of hanging or returning NaN; the server fails that request and answers later ones with the same error, because after a queue fault the HIP runtime rejects every call for the rest of the process, so the process has to restart, and its shutdown may need a SIGKILL (HIP's teardown waits on callbacks the faulted queue never runs). A kernel that never finishes is not detected; `MLX_ROCM_GPU_WATCHDOG_SECS` (default off) fails any single host wait that outlives it. | From 3ff73f9175c0a391fedc8bb2dac397cd896e4f35 Mon Sep 17 00:00:00 2001 From: Jeongkyu Shin Date: Mon, 5 Oct 2026 02:44:36 +0900 Subject: [PATCH 8/8] docs: add technical report for PR #2098 Bilingual (en, ko) report on the HIP fused MoE decode ports, the guard fix, the ROCm SGY default and the measurements. Refs #2065 --- ...098-rocm-fused-moe-hip-port-20261005.en.md | 158 ++++++++++++++++++ ...098-rocm-fused-moe-hip-port-20261005.ko.md | 158 ++++++++++++++++++ 2 files changed, 316 insertions(+) create mode 100644 TECHNICAL_REPORTS/2098-rocm-fused-moe-hip-port-20261005.en.md create mode 100644 TECHNICAL_REPORTS/2098-rocm-fused-moe-hip-port-20261005.ko.md diff --git a/TECHNICAL_REPORTS/2098-rocm-fused-moe-hip-port-20261005.en.md b/TECHNICAL_REPORTS/2098-rocm-fused-moe-hip-port-20261005.en.md new file mode 100644 index 000000000..5f90ea9a3 --- /dev/null +++ b/TECHNICAL_REPORTS/2098-rocm-fused-moe-hip-port-20261005.en.md @@ -0,0 +1,158 @@ +# Technical Report: PR #2098 - HIP ports of the fused MoE decode kernels + +**Date**: 2026-10-05 + +**Status**: Implemented and measured on the gfx1151 host; head `eade24a1` on origin/main `57d8ed29`, PR open, pending merge. + +**Languages**: C++ (HIP kernel sources through `fast::hip_kernel`, port tables, predicates), Rust (backend gates in `switch_layers.rs` and `nemotron_h.rs`, FFI, parity tests), Bash (idle-GPU guard), Python (guard unit test), Markdown, TSV/CSV (logit traces, bench rows) + +**Risk Level**: Medium (ROCm single-token MoE decode now runs new GPU kernels by default on every SwitchGLU family, and Nemotron-H on ROCm changes path. Metal is untouched; on CUDA the only behavior change is that `MLXCEL_FUSED_MOE_RELU2` declines instead of panicking. Wave64 parts are untested) + +## Executive Summary + +Issue #2065 (item 5 of #1814, ranked second by the #2086 profile) asked for HIP ports of the two fused decode-MoE kernels. Before this PR, `moe_gateup_ports()` and `moe_down_ports()` had `.rocm = nullptr`, and both Rust gates in front of them read the backend-wide `custom_kernels_available()`, which answers false on ROCm, so ROCm decode ran `gather_qmm` for every routed expert. + +The PR adds `MOE_GATEUP_HIP_SOURCE` and `MOE_DOWN_HIP_SOURCE`, translated from the CUDA kernels with `__shfl_down(v, o, 32)` for the lane fold, fills both `.rocm` slots, and replaces the backend-wide gate with two predicates that read the port tables (`fused_moe_kernels_available()`, `moe_down_kernel_available()`). On ROCm builds the rows-per-threadgroup default `MLXCEL_FUSED_MOE_SGY` becomes 2 instead of 8, because at 8 the port was slower than `gather_qmm` on gfx1151. + +On gfx1151, Qwen3-30B-A3B-4bit decode goes from a median of 61.29 to 62.51 tok/s (+2.0%, every after run above every before run), which fits the #2086 estimate that only the activation, combine and dispatch overhead is recoverable. Nemotron-3-Nano moves +1.0% (near noise) and Mixtral, whose experts are above the Dff cap, is unchanged. Logit traces show 0 decided-position mismatches in every comparison, and Qwen3's single-token logits are now closer to Metal's fused kernel (1/128 top-1 disagreements) than ROCm `gather_qmm` was (5/128). + +Two side findings were fixed or documented along the way: `scripts/rocm_gpu_guard.sh` was rejecting most measurement attempts because it counted the command's own exited children as foreign GPU tenants, and the wave32 `#error` guard that the #1814 ports carry is inert with AMD clang 23. + +## 1. Problem Statement + +### 1.1 The fused MoE path was unreachable on ROCm + +The fused two-kernel decode MoE path (a gate-up kernel that writes the activated intermediate, then a down kernel that folds the score-weighted expert outputs) had Metal and CUDA ports only. Two independent things kept ROCm off it: + +- the port tables had no `.rocm` entry, so the launcher would refuse; +- `fused_moe_enabled()` in `switch_layers.rs` and the `use_fused` decision in `nemotron_h.rs` both read `custom_kernels_available()`, which is deliberately false on ROCm (#1803). Filling the table alone would not have opened the gate. + +The #2086 profile put fused-MoE-reachable work at 46.8% of Qwen3-30B-A3B decode GPU time, but 42.1 points of that are expert GEMVs already near the host's bandwidth (about 181 GB/s), so its estimate of what a fused kernel could recover was about 5% of GPU time plus about 430 dispatches per token. + +### 1.2 Stale documentation + +`README.md`, `docs/installation.md` and `docs/environment-variables.md` still said affine MoE models abort on ROCm and need `MLXCEL_FUSED_MOE=0`, which no longer matched the code (the backend-wide gate already kept ROCm on `gather_qmm`) and after this PR is wrong in the other direction, because the kernels run. + +## 2. Change Summary + +Seven commits: + +- **`d3801805`** `update(rocm): port the fused MoE decode kernels to HIP`: the two HIP sources and holders, both `.rocm` slots, the two predicates and their FFI, the gate changes in `switch_layers.rs` and `nemotron_h.rs`, the `MLXCEL_FUSED_MOE_RELU2` decline, the test gates and the new SwiGLU parity case. +- **`721ff14a`** `docs(rocm): state what the fused MoE wave32 guard actually covers`: comments and env doc corrected after the `hipcc -E -dM` check (section 5). +- **`4ab91f82`** `fix(bench): stop rocm_gpu_guard rejecting the command's own exited children`: the guard fix and its unit test (section 4). +- **`359f67ed`** `update(rocm): default the fused MoE SGY to 2 on ROCm`. +- **`826dfa8c`** `fix(rocm): pick the fused MoE SGY default from the build flag`: replaces a runtime backend comparison that `make verify-kernel-port-dispatch` rejected. +- **`77afbb1d`** `docs(rocm): list the SSM update step among the HIP-ported kernels`: README wording after the rebase onto #2099. +- **`eade24a1`** `docs(rocm): publish the fused MoE decode and logit results on gfx1151`: `docs/benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md`, the bench CSVs and the traces under `benchmarks/logit_traces/rocm_gfx1151_77afbb1d/`. + +### 2.1 The kernels + +Both HIP sources are the CUDA bodies with one change: the warp shuffle. HIP's `__shfl_down_sync` is a compatibility shim that ignores its mask, so, following the bitlinear port (#1862), the native `__shfl_down(var, delta, 32)` is used with the width stated. Inputs, outputs, grid (one 32-lane wavefront per output row on `threadIdx.x`, `sgy` rows per block on `threadIdx.y`, the expert slot on `grid.z`) and template args are identical, so the hipRTC cache key carries `T` the same way the CUDA key does. `expf` and `tanhf` stay the precise library calls (hipRTC compiles at -O3 without fast-math). + +The down kernel keeps the CUDA kernel's 6-bit branch (four weights per three bytes, laid out as in `quantized.h`), its f32 partials with the score folded in f32, and the single final rounding to the activation dtype after the K-sum that #886 introduced. One HIP down kernel serves all three callers of `moe_down_ports()`: SwitchGLU and GeGLU through `run_fused_moe_two_kernel`, and Nemotron-H's opt-in squared-ReLU path through `moe_down_kernel_fn()`. + +### 2.2 Gates that read the port tables + +`fused_moe_kernels_available()` is `has_kernel_port(moe_gateup_ports()) && has_kernel_port(moe_down_ports())`; `moe_down_kernel_available()` reads the down table alone. Because `select_kernel_port` reads the same tables, a gate and the dispatch cannot disagree (the #1801 rule). `switch_layers.rs` gates every SwitchGLU family on the first, which includes qwen3_moe, qwen3_vl_moe, qwen3_next and gemma4; `nemotron_h.rs` gates on the second. Neither calls `custom_kernels_available()` any more. The doc comments in `lib.rs` and `gpu_backend.h` that said ROCm has no ports now say that backend-wide checks still treat ROCm like `None` and a ported kernel's own predicate opens it. + +For Nemotron-H this changes the ROCm path from `forward_nonfused` to `fused_moe_forward`. Its default branch still runs `gather_qmm` for the routed experts; only the surrounding combine changes. The term stays in the gate even though the default branch launches no custom kernel, so a kernel added to that branch later cannot silently widen the gate to a backend without its port. + +### 2.3 `MLXCEL_FUSED_MOE_RELU2` declines instead of refusing + +The opt-in squared-ReLU branch now also requires `has_kernel_port(moe_fc1_relu2_ports())` and `has_kernel_port(moe_down_ports())`. The fc1 kernel has a Metal port only (the ROCm port is #2069), so on ROCm and CUDA the flag now leaves the default `gather_qmm` branch in place. On CUDA that replaces a panic at the `.expect` in `nemotron_h.rs`; Metal is unchanged. + +### 2.4 Test gates + +`gpu_backend_or_skip()` in `fused_moe_parity_tests.rs` now skips only when there is no GPU backend; on Metal, CUDA or ROCm it asserts `fused_moe_kernels_available()` and fails if it is false, so a missing port is reported rather than skipped past. The qwen3_moe and qwen3_vl_moe positive controls read the predicate instead of `custom_kernels_available()`. A new case, `fused_moe_swiglu_kernel_matches_references_every_down_width`, runs SwiGLU at 4/4, 8/8 and 4/6 bits on the Qwen3 shape, which reaches the SwiGLU arm and the 6-bit down branch. Tolerances are unchanged. A source check asserts the HIP kernels reduce with `__shfl_down`. + +## 3. Measured Results + +### 3.1 Method + +gfx1151 (Radeon 8060S, RDNA 3.5), ROCm 10.0.0, HIP 7.15.26333, MLX pin `81ba1c6a`, overlay `75915908`. Before: origin/main `57d8ed29`. After: the branch at `77afbb1d` on that main. `scripts/bench_decode.sh` at pp512/tg128, before and after binaries alternated run by run, three rounds, every run through the fixed guard; contended attempts were rejected and rerun. + +Main moved mid-run to `57d8ed29` (#2099, the HIP `ssm_update` port), which changes Nemotron-H decode. The branch was rebased, the baseline rebuilt at `57d8ed29`, and every decode and Nemotron logit measurement was redone against it, so before and after differ by this PR only. + +### 3.2 Decode throughput + +| Model | Path change | Before tok/s (median) | After tok/s (median) | Change | +|---|---|---|---|---| +| Qwen3-30B-A3B-4bit | `gather_qmm` to the fused HIP pair | 61.29 / 61.05 / 61.83 (61.29) | 62.51 / 62.68 / 62.16 (62.51) | +2.0% | +| Nemotron-3-Nano-30B-A3B-4bit | `forward_nonfused` to `fused_moe_forward` (still `gather_qmm`) | 74.30 / 74.62 / 74.37 (74.37) | 75.11 / 75.09 / 74.88 (75.09) | +1.0%, near noise | +| Mixtral-8x7B-Instruct-v0.1-4bit | none (Dff 14336 above the 8192 `MLXCEL_FUSED_MOE_MAX_DFF` default) | 9.42 / 9.08 / 10.91 (9.42) | 9.51 / 9.57 (two runs) | +1.3%, noise (same code) | + +The Qwen3 gain is consistent (every after run above every before run) and within the #2086 ceiling: the pair removes the activation, weighted-sum and index-building dispatches but reads the same expert weights. Mixtral's third after run never found a clean GPU window; since the code it runs is identical, its 9.08 to 10.91 spread is host noise for a 26 GB checkpoint on this host. `compare_bench_csv.py --allow-commit-change` on the per-model medians reports a 1.016x median across the three pairs, none moved by more than 10%. Prefill is not on the fused path. + +### 3.3 The SGY default + +The first implementation ported the CUDA kernels verbatim with Metal's SGY default of 8. A guarded pair measured Qwen3 at 61.64 tok/s on `gather_qmm` (`c0b71344`) against 59.42 fused: the port was slower than the fallback it replaced. A sweep of `MLXCEL_FUSED_MOE_SGY` over 1, 2, 4, 8, 16 and 32 on one binary put 2 on top in every repetition (about 62.3 tok/s, against about 61 for `gather_qmm`, about 61.2 for SGY 1 and 59.3 to 60.4 for the rest). All nine guarded attempts of that sweep were rejected as contended (another unit's GPU process appeared in at least one sample), so the sweep is indicative only; the table in 3.2, measured at the new default, is the evidence. SGY shapes the threadgroup only, and `fused_moe_geglu_kernel_bitwise_invariant_across_sgy` pins that the output does not depend on it. + +### 3.4 Logits + +`compare_logit_traces.py --decided 2.0`, new traces under `benchmarks/logit_traces/rocm_gfx1151_77afbb1d/`. The fused kernels run only on a single-token forward, so `w8` traces never reach them (Qwen3 `default` and `fused0` at `w8` are identical); `w1` is the window that exercises the kernels, and Nemotron-H `w1ctx512` covers decode with state. + +| Model, window | Reference | Candidate | Top-1 disagreement | Decided mismatches | +|---|---|---|---|---| +| Qwen3-30B-A3B, w1 | Metal fused | ROCm fused (HIP) | 1 / 128 | 0 / 74 | +| Qwen3-30B-A3B, w1 | Metal `gather_qmm` | ROCm `gather_qmm` | 5 / 128 | 0 / 70 | +| Qwen3-30B-A3B, w1 | ROCm `gather_qmm` | ROCm fused | 3 / 128 | 0 / 74 | +| Qwen3-30B-A3B, w8 | Metal default | ROCm default | 17 / 640 | 0 / 319 | +| Nemotron-3-Nano, w1ctx512 | ROCm main | this branch | 3 / 128 | 0 / 68 | +| Nemotron-3-Nano, w1ctx512 | Metal M5 | this branch | 6 / 128 | 0 / 71 | +| Nemotron-3-Nano, w8 | ROCm `c5fe9a16` (`forward_nonfused`) | this branch | 20 / 640 | 0 / 286 | +| Nemotron-3-Nano, w8 | Metal M5 | this branch | 22 / 640 | 0 / 287 | + +Every row has zero decided-position mismatches. Nemotron-H's path change moves a few undecided tokens, at reference gaps of 0.5 or less. The `gather_qmm` path is unchanged: the Qwen3 `fused0` `w1` and Mixtral `w1` traces are byte-identical in every data row to `rocm_gfx1151_bec64748`, and Qwen3 `fused0` `w8` is byte-identical to a trace built on `57d8ed29`. Nemotron-3-Nano generates the same 40-token greedy text with and without `MLXCEL_FUSED_MOE_RELU2=1`. + +## 4. The Guard Bug Found While Measuring + +`scripts/rocm_gpu_guard.sh` (added in #2086) samples `/sys/class/kfd/kfd/proc` at 1 Hz and rejects an attempt if any GPU holder is not the command or a descendant. KFD removes a process's entry from deferred work after the process is reaped, so for a moment the entry names a pid with no `/proc` directory. The next sample then saw the command's own exited child (the `rocminfo` probe in `bench_decode.sh`, or the bench binary itself), `descends_from` could not read its parentage, and the guard counted it as foreign. Most attempts were rejected this way, in this unit and in the parallel #2067 unit. + +The fix skips holders whose `/proc/` is gone: an exited, reaped process is not using the GPU. `test_a_kfd_entry_left_by_an_exited_process_is_not_contention` in `tests/test_rocm_decode_profile.py` reproduces the window with a fake KFD directory; it fails on the old guard (exit 75, `CONTENDED`) and passes on the new one. A second effect, two guards started in lockstep rejecting each other, was handled operationally (a longer idle window, waiting) and not in code. + +## 5. The Wave32 Guard Is Inert + +The #1814 HIP ports carry `#error` guards on `__AMDGCN_WAVEFRONT_SIZE` and `__AMDGCN_WAVEFRONT_SIZE__`. `hipcc -E -dM` for gfx942 and gfx1151 showed that AMD clang 23 (HIP 7.15) defines neither spelling, so the guard never fires and a wave64 target would compile. What keeps the kernels correct there is the explicit shuffle width: `__shfl_down(v, o, 32)` keeps each 32-lane fold inside one row on a 64-lane wavefront. The first draft of the env doc said a wave64 target fails to compile; that was corrected, and the docs now say wave64 is untested and `MLXCEL_FUSED_MOE=0` keeps such a host on `gather_qmm`. The same holds for the earlier bitlinear port, which this PR does not change. + +## 6. Technical Decisions + +- **Gate on the port tables, not on the backend.** A backend-wide predicate cannot express "ROCm has this kernel but not that one", which is the state #1814 produces port by port. Reading the same tables as `select_kernel_port` makes the gate and the dispatch agree by construction. +- **Translate the CUDA kernel, do not redesign it.** Same geometry, template args and numerics (6-bit branch, f32 partials, one final rounding) keeps the parity bounds and the #886 fix valid unchanged, and leaves performance work as a separate, measurable step. +- **Pick the ROCm SGY default at compile time.** The first attempt compared `gpu_kernel_backend()` at runtime in the launcher; `make verify-kernel-port-dispatch` rejects runtime backend comparisons outside `select_kernel_port`. A ROCm build has no Metal or CUDA backend, so `MLXCEL_BRIDGE_ROCM_BACKEND` is the backend and satisfies the gate. Metal and CUDA keep 8. +- **Decline rather than refuse on a partial port set.** `MLXCEL_FUSED_MOE_RELU2` falls back to `gather_qmm` when fc1 or down is missing, which also removes a latent CUDA panic. +- **Fail, do not skip, on a GPU backend without the ports.** Every GPU backend now has both ports, so a false predicate there is a defect the parity tests should report. +- **Rebuild the baseline when main moved.** #2099 changed Nemotron-H decode; comparing against the old baseline would have credited this PR with #2099's effect. + +## 7. Validation + +On gfx1151: + +- `make verify-rocm` on the final rebased head: 146 suites, 11857 passed, 0 failed, 378 ignored (head `eade24a1`). On the pre-rebase head `beb8f77c`: 146 suites, 11852 passed, 0 failed, 378 ignored. +- `fused_moe_parity_tests`: 5 passed. Fused against the all-f32 reference at normalized RMS at most 3.0e-6; fused against `gather_qmm` about 3.7e-3; `gather_qmm` against the reference about 3.3e-3. Negative control: starting the down kernel's lane fold at 8 instead of 16 fails both reference tests (normalized RMS 0.63 and 0.70). +- `qwen3_moe::tests` and `qwen3_vl_moe::tests`: 12 passed, with the positive controls running the kernel. +- `switch_layers` and `nemotron_h` lib tests: 93 passed. +- `cargo clippy -D warnings` on `mlxcel-core` and `mlxcel`: clean. The fast gates pass, and the dtype-key pin is unchanged at 9 in scope. +- Guard unit test: fails on the old guard, passes on the new one. + +A static review found no CRITICAL or HIGH issues. The MEDIUM item was the wave64 concern, answered by the finding in section 5 (no hard failure; untested). The LOW items were fixed: the `MLXCEL_FUSED_MOE_MAX_DFF` row lists ROCm, and the stale "ROCm has no ports" comments in `lib.rs` and `gpu_backend.h` were rewritten. + +## 8. Residual Risks and What Was Not Verified + +- **Metal and CUDA not run.** No hardware on this host. Their `.metal` and `.cuda` entries and sources are untouched and both predicates answer true there. The CUDA-visible change is the `MLXCEL_FUSED_MOE_RELU2` decline; the SGY default change is compiled into ROCm builds only. +- **Gemma 4 and Qwen3-Next not run.** They reach the same kernels through the same gate (GeGLU for Gemma 4), but no checkpoint was available. The GeGLU arm is covered only by the parity tests. +- **Wave64 (CDNA) untested.** The compile-time guard does not protect it (section 5); correctness there rests on the explicit shuffle width. +- **One device.** All throughput and the SGY choice come from gfx1151. The SGY sweep itself was contended in every attempt and is indicative only; another RDNA or CDNA part may want a different default. +- **Small gains near noise.** Nemotron-H's +1.0% sits within its run-to-run spread, and Mixtral has only two after runs. +- **The port is naive.** It is a direct CUDA translation without vectorized loads or `x` in shared memory. Because expert GEMVs are bandwidth-bound, the headroom is expected to be small, but it was not measured. + +Recommended follow-ups: vectorized loads or `x` in shared memory for the HIP MoE kernels; a wave-size check for all ROCm ports that does not rely on the inert macro (for example a host-side `warpSize` check in port selection); routing granite's `block_sparse_moe` through `forward_fused_kernel` (noted in #2086); the Nemotron-H loader printing to stdout. + +## 9. Learning Points + +- **A port is not reachable until its gate is.** Filling `.rocm` changed nothing while the callers read a backend-wide predicate. Per-kernel predicates over the port tables are what make partial port coverage work. +- **A verbatim port can be slower than the fallback.** The CUDA launch shape tuned on other hardware lost to `gather_qmm` on gfx1151 until SGY changed from 8 to 2. A before/after measurement belongs in the port PR, not after it. +- **Choose the trace window that reaches the code.** The fused kernels run only on a single-token forward, so `w8` traces would have compared identical paths. `w1` (and `w1ctx512` for stateful decode) is where the change is visible. +- **Check that a compile-time guard can fire.** The wave32 `#error` looked like protection; `hipcc -E -dM` showed the macro is never defined. +- **Measurement tooling has bugs too.** The guard's false rejections cost time across two units and came from a kernel teardown order, not from real contention. A regression test that reproduces the window keeps it fixed. + +Refs: #2065, #1814, #2086, #2099, #2069, #2067, #1862, #1803, #1885, #1801, #886. diff --git a/TECHNICAL_REPORTS/2098-rocm-fused-moe-hip-port-20261005.ko.md b/TECHNICAL_REPORTS/2098-rocm-fused-moe-hip-port-20261005.ko.md new file mode 100644 index 000000000..58ae97308 --- /dev/null +++ b/TECHNICAL_REPORTS/2098-rocm-fused-moe-hip-port-20261005.ko.md @@ -0,0 +1,158 @@ +# 기술 보고서: PR #2098 - fused MoE decode kernel의 HIP port + +**날짜**: 2026-10-05 + +**상태**: gfx1151 호스트에서 구현 및 측정 완료. head `eade24a1`, origin/main `57d8ed29` 위, PR open 상태로 머지 대기 중. + +**언어**: C++ (`fast::hip_kernel`을 통한 HIP kernel source, port table, predicate), Rust (`switch_layers.rs`와 `nemotron_h.rs`의 backend gate, FFI, parity test), Bash (idle-GPU guard), Python (guard unit test), Markdown, TSV/CSV (logit trace, bench 결과) + +**위험도**: 중간 (ROCm의 single-token MoE decode가 모든 SwitchGLU 계열에서 기본값으로 새 GPU kernel을 실행하고, ROCm의 Nemotron-H는 경로가 바뀝니다. Metal은 그대로이고, CUDA에서 바뀌는 동작은 `MLXCEL_FUSED_MOE_RELU2`가 panic 대신 사양하는 것 하나입니다. Wave64 장치는 시험하지 않았습니다) + +## 요약 + +Issue #2065 (#1814의 항목 5, #2086 profile에서 2순위)는 두 fused decode-MoE kernel의 HIP port를 요구했습니다. 이 PR 이전에는 `moe_gateup_ports()`와 `moe_down_ports()`의 `.rocm`이 `nullptr`였고, 그 앞의 Rust gate 두 개가 모두 backend 전체 단위의 `custom_kernels_available()`을 읽었습니다. 이 값은 ROCm에서 false이므로 ROCm decode는 routed expert마다 `gather_qmm`을 실행했습니다. + +이 PR은 CUDA kernel을 옮기면서 lane fold만 `__shfl_down(v, o, 32)`로 바꾼 `MOE_GATEUP_HIP_SOURCE`와 `MOE_DOWN_HIP_SOURCE`를 추가하고, 두 `.rocm` 칸을 채우며, backend 단위 gate를 port table을 읽는 predicate 두 개 (`fused_moe_kernels_available()`, `moe_down_kernel_available()`)로 바꿉니다. ROCm build에서는 threadgroup당 row 수 기본값 `MLXCEL_FUSED_MOE_SGY`가 8이 아니라 2입니다. gfx1151에서 8로는 port가 `gather_qmm`보다 느렸기 때문입니다. + +gfx1151에서 Qwen3-30B-A3B-4bit decode는 median 61.29에서 62.51 tok/s로 올랐습니다 (+2.0%, 모든 after run이 모든 before run보다 높음). 이는 activation, combine, dispatch overhead만 회수할 수 있다는 #2086의 추정과 맞습니다. Nemotron-3-Nano는 +1.0% (noise 수준)이고, expert가 Dff 상한을 넘는 Mixtral은 변화가 없습니다. Logit trace는 모든 비교에서 decided position 불일치가 0이며, Qwen3의 single-token logit은 이제 ROCm `gather_qmm` (5/128)보다 Metal fused kernel에 더 가깝습니다 (top-1 불일치 1/128). + +작업 중 발견한 두 가지도 고치거나 문서화했습니다. `scripts/rocm_gpu_guard.sh`가 명령 자신의 종료된 자식 프로세스를 외부 GPU 사용자로 세어 대부분의 측정 시도를 거부하고 있었고, #1814 port들이 지닌 wave32 `#error` guard는 AMD clang 23에서 작동하지 않습니다. + +## 1. 문제 정의 + +### 1.1 ROCm에서 닿을 수 없던 fused MoE 경로 + +Fused two-kernel decode MoE 경로 (활성화된 intermediate를 쓰는 gate-up kernel, 이어서 score로 가중된 expert 출력을 합치는 down kernel)는 Metal과 CUDA port만 있었습니다. ROCm을 이 경로에서 막은 것은 서로 독립적인 두 가지였습니다. + +- port table에 `.rocm` 항목이 없어서 launcher가 거부했습니다. +- `switch_layers.rs`의 `fused_moe_enabled()`와 `nemotron_h.rs`의 `use_fused` 결정이 모두 `custom_kernels_available()`을 읽었고, 이 값은 ROCm에서 의도적으로 false입니다 (#1803). Table만 채웠다면 gate는 열리지 않았을 것입니다. + +#2086 profile은 Qwen3-30B-A3B decode GPU 시간의 46.8%를 fused MoE가 닿는 작업으로 보았지만, 그중 42.1 point는 이미 호스트 대역폭 (약 181 GB/s)에 가까운 expert GEMV였습니다. 그래서 fused kernel이 회수할 수 있는 양은 GPU 시간의 약 5%와 token당 약 430개의 dispatch로 추정했습니다. + +### 1.2 낡은 문서 + +`README.md`, `docs/installation.md`, `docs/environment-variables.md`는 여전히 ROCm에서 affine MoE 모델이 abort하므로 `MLXCEL_FUSED_MOE=0`이 필요하다고 적고 있었습니다. 이는 이미 코드와 맞지 않았고 (backend 단위 gate가 ROCm을 `gather_qmm`에 두고 있었음), 이 PR 이후에는 kernel이 실제로 실행되므로 반대 방향으로 틀린 설명이 됩니다. + +## 2. 변경 요약 + +Commit 일곱 개: + +- **`d3801805`** `update(rocm): port the fused MoE decode kernels to HIP`: HIP source 두 개와 holder, 두 `.rocm` 칸, predicate 두 개와 FFI, `switch_layers.rs`와 `nemotron_h.rs`의 gate 변경, `MLXCEL_FUSED_MOE_RELU2` 사양, test gate와 새 SwiGLU parity case. +- **`721ff14a`** `docs(rocm): state what the fused MoE wave32 guard actually covers`: `hipcc -E -dM` 확인 후 주석과 env 문서 수정 (5절). +- **`4ab91f82`** `fix(bench): stop rocm_gpu_guard rejecting the command's own exited children`: guard 수정과 unit test (4절). +- **`359f67ed`** `update(rocm): default the fused MoE SGY to 2 on ROCm`. +- **`826dfa8c`** `fix(rocm): pick the fused MoE SGY default from the build flag`: `make verify-kernel-port-dispatch`가 거부한 runtime backend 비교를 대체. +- **`77afbb1d`** `docs(rocm): list the SSM update step among the HIP-ported kernels`: #2099 위로 rebase한 뒤의 README 문구. +- **`eade24a1`** `docs(rocm): publish the fused MoE decode and logit results on gfx1151`: `docs/benchmark_results/rocm-fused-moe-gfx1151-2026-10-05.md`, bench CSV, `benchmarks/logit_traces/rocm_gfx1151_77afbb1d/` 아래 trace. + +### 2.1 Kernel + +두 HIP source는 CUDA 본문에서 warp shuffle 하나만 바꾼 것입니다. HIP의 `__shfl_down_sync`는 mask를 무시하는 호환용 shim이므로, bitlinear port (#1862)를 따라 width를 명시한 native `__shfl_down(var, delta, 32)`를 씁니다. 입력, 출력, grid (`threadIdx.x`에 출력 row당 32-lane wavefront 하나, `threadIdx.y`에 block당 `sgy`개 row, `grid.z`에 expert slot)와 template arg가 같으므로 hipRTC cache key도 CUDA key와 같은 방식으로 `T`를 담습니다. `expf`와 `tanhf`는 정밀한 library 호출 그대로입니다 (hipRTC는 fast-math 없이 -O3로 compile). + +Down kernel은 CUDA kernel의 6-bit 분기 (세 byte에 weight 네 개, `quantized.h`와 같은 배치), score를 f32에서 접는 f32 partial, 그리고 #886이 도입한 K-sum 뒤 activation dtype으로의 단일 반올림을 그대로 유지합니다. HIP down kernel 하나가 `moe_down_ports()`의 호출자 셋을 모두 담당합니다. `run_fused_moe_two_kernel`을 통한 SwitchGLU와 GeGLU, 그리고 `moe_down_kernel_fn()`을 통한 Nemotron-H의 opt-in squared-ReLU 경로입니다. + +### 2.2 Port table을 읽는 gate + +`fused_moe_kernels_available()`은 `has_kernel_port(moe_gateup_ports()) && has_kernel_port(moe_down_ports())`이고, `moe_down_kernel_available()`은 down table만 읽습니다. `select_kernel_port`가 같은 table을 읽으므로 gate와 dispatch는 서로 다르게 답할 수 없습니다 (#1801 규칙). `switch_layers.rs`는 모든 SwitchGLU 계열을 첫 번째 predicate로 막으며, 여기에는 qwen3_moe, qwen3_vl_moe, qwen3_next, gemma4가 포함됩니다. `nemotron_h.rs`는 두 번째를 읽습니다. 두 곳 모두 더 이상 `custom_kernels_available()`을 호출하지 않습니다. ROCm에 port가 없다고 적었던 `lib.rs`와 `gpu_backend.h`의 doc comment는 이제 backend 단위 검사는 ROCm을 여전히 `None`처럼 다루고, port된 kernel의 자체 predicate가 그 kernel을 연다고 설명합니다. + +Nemotron-H의 ROCm 경로는 `forward_nonfused`에서 `fused_moe_forward`로 바뀝니다. 기본 분기는 여전히 routed expert에 `gather_qmm`을 쓰고, 바뀌는 것은 그 주변의 combine뿐입니다. 기본 분기가 custom kernel을 실행하지 않는데도 gate 조건을 남겨 둔 것은, 나중에 그 분기에 kernel이 추가되더라도 port가 없는 backend로 gate가 조용히 넓어지지 않게 하기 위해서입니다. + +### 2.3 `MLXCEL_FUSED_MOE_RELU2`는 거부 대신 사양 + +Opt-in squared-ReLU 분기는 이제 `has_kernel_port(moe_fc1_relu2_ports())`와 `has_kernel_port(moe_down_ports())`도 요구합니다. fc1 kernel은 Metal port만 있으므로 (ROCm port는 #2069), ROCm과 CUDA에서 이 flag는 기본 `gather_qmm` 분기를 그대로 둡니다. CUDA에서는 `nemotron_h.rs`의 `.expect`에서 나던 panic이 사라지고, Metal은 그대로입니다. + +### 2.4 Test gate + +`fused_moe_parity_tests.rs`의 `gpu_backend_or_skip()`은 이제 GPU backend가 없을 때만 건너뜁니다. Metal, CUDA, ROCm에서는 `fused_moe_kernels_available()`을 assert하고 false면 실패하므로, port 누락은 건너뛰지 않고 보고됩니다. qwen3_moe와 qwen3_vl_moe의 positive control은 `custom_kernels_available()` 대신 이 predicate를 읽습니다. 새 case `fused_moe_swiglu_kernel_matches_references_every_down_width`는 Qwen3 shape에서 SwiGLU를 4/4, 8/8, 4/6 bit로 실행해 SwiGLU 분기와 6-bit down 분기에 닿습니다. 허용 오차는 바뀌지 않았습니다. HIP kernel이 `__shfl_down`으로 reduce하는지 source를 검사하는 assert도 있습니다. + +## 3. 측정 결과 + +### 3.1 방법 + +gfx1151 (Radeon 8060S, RDNA 3.5), ROCm 10.0.0, HIP 7.15.26333, MLX pin `81ba1c6a`, overlay `75915908`. Before: origin/main `57d8ed29`. After: 그 main 위의 branch `77afbb1d`. `scripts/bench_decode.sh`를 pp512/tg128로, before와 after binary를 run마다 번갈아 세 round 실행했고, 모든 run은 수정된 guard를 거쳤으며 경합이 있던 시도는 거부 후 재실행했습니다. + +작업 도중 main이 `57d8ed29` (#2099, HIP `ssm_update` port)로 이동했고, 이는 Nemotron-H decode를 바꿉니다. Branch를 rebase하고 baseline을 `57d8ed29`에서 다시 build한 뒤 decode와 Nemotron logit 측정을 모두 다시 했으므로, before와 after의 차이는 이 PR뿐입니다. + +### 3.2 Decode 처리량 + +| 모델 | 경로 변화 | Before tok/s (median) | After tok/s (median) | 변화 | +|---|---|---|---|---| +| Qwen3-30B-A3B-4bit | `gather_qmm`에서 fused HIP 쌍으로 | 61.29 / 61.05 / 61.83 (61.29) | 62.51 / 62.68 / 62.16 (62.51) | +2.0% | +| Nemotron-3-Nano-30B-A3B-4bit | `forward_nonfused`에서 `fused_moe_forward`로 (여전히 `gather_qmm`) | 74.30 / 74.62 / 74.37 (74.37) | 75.11 / 75.09 / 74.88 (75.09) | +1.0%, noise 수준 | +| Mixtral-8x7B-Instruct-v0.1-4bit | 없음 (Dff 14336이 `MLXCEL_FUSED_MOE_MAX_DFF` 기본값 8192 초과) | 9.42 / 9.08 / 10.91 (9.42) | 9.51 / 9.57 (두 run) | +1.3%, noise (같은 코드) | + +Qwen3의 향상은 일관되며 (모든 after run이 모든 before run보다 높음) #2086의 상한 안에 있습니다. Fused 쌍은 activation, weighted sum, index 생성 dispatch를 없애지만 expert weight는 똑같이 읽습니다. Mixtral의 세 번째 after run은 깨끗한 GPU 구간을 얻지 못했습니다. 실행하는 코드가 같으므로 9.08~10.91의 폭은 이 호스트에서 26 GB checkpoint가 보이는 noise입니다. 모델별 median에 대한 `compare_bench_csv.py --allow-commit-change`는 세 쌍에서 median 1.016x를 보고하며, 10% 넘게 움직인 것은 없습니다. Prefill은 fused 경로를 지나지 않습니다. + +### 3.3 SGY 기본값 + +첫 구현은 CUDA kernel을 Metal의 SGY 기본값 8과 함께 그대로 옮겼습니다. Guard를 거친 한 쌍의 측정에서 Qwen3는 `gather_qmm` (`c0b71344`) 61.64 tok/s, fused 59.42였습니다. Port가 대체하려던 fallback보다 느렸습니다. 한 binary에서 `MLXCEL_FUSED_MOE_SGY`를 1, 2, 4, 8, 16, 32로 바꾼 sweep에서는 모든 반복에서 2가 가장 빨랐습니다 (약 62.3 tok/s, `gather_qmm` 약 61, SGY 1 약 61.2, 나머지 59.3~60.4). 이 sweep의 guard 시도 아홉 번은 모두 경합으로 거부되었으므로 (적어도 한 sample에 다른 unit의 GPU 프로세스가 보임) 참고용일 뿐이고, 근거는 새 기본값에서 측정한 3.2의 표입니다. SGY는 threadgroup 모양만 정하며, 출력이 이 값에 의존하지 않는다는 것은 `fused_moe_geglu_kernel_bitwise_invariant_across_sgy`가 고정합니다. + +### 3.4 Logit + +`compare_logit_traces.py --decided 2.0`, 새 trace는 `benchmarks/logit_traces/rocm_gfx1151_77afbb1d/`. Fused kernel은 single-token forward에서만 실행되므로 `w8` trace는 여기에 닿지 않습니다 (`w8`에서 Qwen3 `default`와 `fused0`은 동일). kernel을 실행하는 window는 `w1`이고, Nemotron-H `w1ctx512`는 state가 있는 decode를 다룹니다. + +| 모델, window | 기준 | 후보 | Top-1 불일치 | Decided 불일치 | +|---|---|---|---|---| +| Qwen3-30B-A3B, w1 | Metal fused | ROCm fused (HIP) | 1 / 128 | 0 / 74 | +| Qwen3-30B-A3B, w1 | Metal `gather_qmm` | ROCm `gather_qmm` | 5 / 128 | 0 / 70 | +| Qwen3-30B-A3B, w1 | ROCm `gather_qmm` | ROCm fused | 3 / 128 | 0 / 74 | +| Qwen3-30B-A3B, w8 | Metal default | ROCm default | 17 / 640 | 0 / 319 | +| Nemotron-3-Nano, w1ctx512 | ROCm main | 이 branch | 3 / 128 | 0 / 68 | +| Nemotron-3-Nano, w1ctx512 | Metal M5 | 이 branch | 6 / 128 | 0 / 71 | +| Nemotron-3-Nano, w8 | ROCm `c5fe9a16` (`forward_nonfused`) | 이 branch | 20 / 640 | 0 / 286 | +| Nemotron-3-Nano, w8 | Metal M5 | 이 branch | 22 / 640 | 0 / 287 | + +모든 행에서 decided position 불일치는 0입니다. Nemotron-H의 경로 변화는 undecided token 몇 개를 움직이며, 그 위치의 기준 gap은 0.5 이하입니다. `gather_qmm` 경로는 바뀌지 않았습니다. Qwen3 `fused0` `w1`과 Mixtral `w1` trace는 모든 data row에서 `rocm_gfx1151_bec64748`와 byte 단위로 같고, Qwen3 `fused0` `w8`은 `57d8ed29`에서 만든 trace와 byte 단위로 같습니다. Nemotron-3-Nano는 `MLXCEL_FUSED_MOE_RELU2=1` 유무와 관계없이 같은 40-token greedy text를 생성합니다. + +## 4. 측정 중 발견한 guard 버그 + +`scripts/rocm_gpu_guard.sh` (#2086에서 추가)는 `/sys/class/kfd/kfd/proc`을 1 Hz로 sample하고, GPU를 쥔 프로세스 중 명령 자신이나 그 자손이 아닌 것이 있으면 시도를 거부합니다. KFD는 프로세스가 reap된 뒤 deferred work에서 그 항목을 지우므로, 잠깐 동안 항목이 `/proc` 디렉터리가 없는 pid를 가리킵니다. 그러면 다음 sample이 명령 자신의 종료된 자식 (`bench_decode.sh`의 `rocminfo` probe, 또는 bench binary 자체)을 보고, `descends_from`이 부모 관계를 읽지 못해 외부 프로세스로 셌습니다. 이 unit과 병렬로 돌던 #2067 unit 모두에서 대부분의 시도가 이렇게 거부되었습니다. + +수정은 `/proc/`가 없는 holder를 건너뛰는 것입니다. 종료되고 reap된 프로세스는 GPU를 쓰고 있지 않습니다. `tests/test_rocm_decode_profile.py`의 `test_a_kfd_entry_left_by_an_exited_process_is_not_contention`은 가짜 KFD 디렉터리로 그 구간을 재현하며, 옛 guard에서는 실패하고 (exit 75, `CONTENDED`) 새 guard에서는 통과합니다. 동시에 시작한 guard 두 개가 서로를 거부하는 두 번째 현상은 코드가 아니라 운영으로 (더 긴 idle 구간, 대기) 대응했습니다. + +## 5. 작동하지 않는 wave32 guard + +#1814 HIP port들은 `__AMDGCN_WAVEFRONT_SIZE`와 `__AMDGCN_WAVEFRONT_SIZE__`에 대한 `#error` guard를 지닙니다. gfx942와 gfx1151에 대해 `hipcc -E -dM`을 실행해 보니 AMD clang 23 (HIP 7.15)은 두 철자 모두 정의하지 않으므로, guard는 발동하지 않고 wave64 target도 compile됩니다. 그런 장치에서 kernel을 올바르게 유지하는 것은 명시한 shuffle width입니다. `__shfl_down(v, o, 32)`는 64-lane wavefront에서도 각 32-lane fold를 한 row 안에 가둡니다. env 문서 초안은 wave64 target이 compile에 실패한다고 잘못 적었고, 이를 고쳐 이제 wave64는 시험하지 않았으며 `MLXCEL_FUSED_MOE=0`으로 그런 호스트를 `gather_qmm`에 둘 수 있다고 설명합니다. 이전의 bitlinear port도 마찬가지이며, 이 PR은 그것을 바꾸지 않습니다. + +## 6. 기술적 선택과 그 이유 + +- **Backend가 아니라 port table로 gate합니다.** Backend 단위 predicate로는 "ROCm에 이 kernel은 있고 저 kernel은 없다"를 표현할 수 없는데, #1814는 port를 하나씩 채우므로 바로 그 상태를 만듭니다. `select_kernel_port`와 같은 table을 읽으면 gate와 dispatch가 구조적으로 일치합니다. +- **CUDA kernel을 재설계하지 않고 옮깁니다.** Geometry, template arg, 수치 처리 (6-bit 분기, f32 partial, 단일 최종 반올림)를 같게 두면 parity 범위와 #886 수정이 그대로 유효하고, 성능 작업은 따로 측정할 수 있는 단계로 남습니다. +- **ROCm SGY 기본값은 compile 시점에 고릅니다.** 첫 시도는 launcher에서 `gpu_kernel_backend()`를 runtime에 비교했고, `make verify-kernel-port-dispatch`는 `select_kernel_port` 밖의 runtime backend 비교를 거부합니다. ROCm build에는 Metal이나 CUDA backend가 없으므로 `MLXCEL_BRIDGE_ROCM_BACKEND`가 곧 backend이고 gate도 통과합니다. Metal과 CUDA는 8을 유지합니다. +- **Port 일부만 있을 때는 거부하지 않고 사양합니다.** `MLXCEL_FUSED_MOE_RELU2`는 fc1이나 down이 없으면 `gather_qmm`으로 돌아가며, 이것으로 CUDA의 잠재적 panic도 사라집니다. +- **Port가 없는 GPU backend에서는 건너뛰지 않고 실패합니다.** 이제 모든 GPU backend가 두 port를 가지므로, 거기서 predicate가 false라면 parity test가 보고해야 할 결함입니다. +- **Main이 움직이면 baseline을 다시 만듭니다.** #2099가 Nemotron-H decode를 바꿨으므로, 옛 baseline과 비교했다면 #2099의 효과가 이 PR의 것으로 잡혔을 것입니다. + +## 7. 검증 + +gfx1151에서: + +- 최종 rebase head에서 `make verify-rocm`: 146 suite, 11857 passed, 0 failed, 378 ignored (head `eade24a1`). Rebase 전 head `beb8f77c`에서는 146 suite, 11852 passed, 0 failed, 378 ignored. +- `fused_moe_parity_tests`: 5 passed. Fused 대 all-f32 기준의 normalized RMS 최대 3.0e-6, fused 대 `gather_qmm` 약 3.7e-3, `gather_qmm` 대 기준 약 3.3e-3. Negative control: down kernel의 lane fold를 16 대신 8에서 시작하면 두 기준 test가 모두 실패합니다 (normalized RMS 0.63과 0.70). +- `qwen3_moe::tests`와 `qwen3_vl_moe::tests`: 12 passed, positive control이 kernel을 실행. +- `switch_layers`와 `nemotron_h` lib test: 93 passed. +- `mlxcel-core`와 `mlxcel`에 대한 `cargo clippy -D warnings`: clean. Fast gate 통과, dtype-key pin은 범위 안에서 9로 변함없음. +- Guard unit test: 옛 guard에서 실패, 새 guard에서 통과. + +정적 review에서 CRITICAL이나 HIGH는 없었습니다. MEDIUM은 wave64 우려였고 5절의 발견으로 답했습니다 (compile 실패는 없음, 시험하지 않음). LOW 항목은 고쳤습니다. `MLXCEL_FUSED_MOE_MAX_DFF` 행에 ROCm을 넣었고, `lib.rs`와 `gpu_backend.h`의 낡은 "ROCm has no ports" 주석을 다시 썼습니다. + +## 8. 남은 위험과 검증하지 않은 부분 + +- **Metal과 CUDA는 실행하지 않았습니다.** 이 호스트에 하드웨어가 없습니다. `.metal`과 `.cuda` 항목과 source는 손대지 않았고 두 predicate는 그곳에서 true입니다. CUDA에서 보이는 변화는 `MLXCEL_FUSED_MOE_RELU2` 사양이고, SGY 기본값 변경은 ROCm build에만 compile됩니다. +- **Gemma 4와 Qwen3-Next는 실행하지 않았습니다.** 같은 gate를 거쳐 같은 kernel에 닿지만 (Gemma 4는 GeGLU) checkpoint가 없었습니다. GeGLU 분기는 parity test로만 덮여 있습니다. +- **Wave64 (CDNA)는 시험하지 않았습니다.** Compile 시점 guard는 이를 막지 못하며 (5절), 정확성은 명시한 shuffle width에 기대고 있습니다. +- **장치 하나.** 처리량과 SGY 선택은 모두 gfx1151에서 나왔습니다. SGY sweep 자체는 모든 시도가 경합 상태여서 참고용이고, 다른 RDNA나 CDNA 장치에는 다른 기본값이 맞을 수 있습니다. +- **Noise에 가까운 작은 향상.** Nemotron-H의 +1.0%는 run 간 편차 안에 있고, Mixtral은 after run이 두 개뿐입니다. +- **단순한 port.** Vectorized load나 shared memory의 `x` 없이 CUDA를 그대로 옮긴 것입니다. Expert GEMV가 대역폭에 묶여 있으므로 여유는 작을 것으로 보이지만 측정하지는 않았습니다. + +권장 후속 작업: HIP MoE kernel의 vectorized load 또는 `x`의 shared memory 적재, 작동하지 않는 macro에 기대지 않는 모든 ROCm port용 wave size 검사 (예: port 선택 시 host 쪽 `warpSize` 검사), granite의 `block_sparse_moe`를 `forward_fused_kernel`로 연결 (#2086에서 언급), Nemotron-H loader의 stdout 출력 정리. + +## 9. 학습 포인트 + +- **Gate가 열려야 port에 닿습니다.** 호출자가 backend 단위 predicate를 읽는 동안에는 `.rocm`을 채워도 아무것도 바뀌지 않았습니다. Port table 위의 kernel별 predicate가 port가 부분적으로만 있는 상태를 다룰 수 있게 합니다. +- **그대로 옮긴 port가 fallback보다 느릴 수 있습니다.** 다른 하드웨어에서 맞춘 CUDA launch 모양은 SGY를 8에서 2로 바꾸기 전까지 gfx1151에서 `gather_qmm`에 졌습니다. Before/after 측정은 port PR 안에 있어야 합니다. +- **코드에 닿는 trace window를 고릅니다.** Fused kernel은 single-token forward에서만 실행되므로 `w8` trace는 같은 경로끼리 비교했을 것입니다. 변화가 보이는 것은 `w1` (state가 있는 decode는 `w1ctx512`)입니다. +- **Compile 시점 guard가 실제로 발동할 수 있는지 확인합니다.** Wave32 `#error`는 보호처럼 보였지만, `hipcc -E -dM`은 그 macro가 정의되지 않는다는 것을 보여 주었습니다. +- **측정 도구에도 버그가 있습니다.** Guard의 잘못된 거부는 두 unit에 걸쳐 시간을 잡아먹었고, 원인은 실제 경합이 아니라 kernel의 정리 순서였습니다. 그 구간을 재현하는 regression test가 수정이 유지되도록 합니다. + +Refs: #2065, #1814, #2086, #2099, #2069, #2067, #1862, #1803, #1885, #1801, #886.