diff --git a/metainfer/tasks/dcu_kernel_auto_opt/WORKFLOW.md b/metainfer/tasks/dcu_kernel_auto_opt/WORKFLOW.md index ea1cfe68..3a62240c 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/WORKFLOW.md +++ b/metainfer/tasks/dcu_kernel_auto_opt/WORKFLOW.md @@ -3,6 +3,11 @@ > 维护说明：本文档由人工/AI 维护，内容基于 2026-08 的实际代码与现场任务。 > 代码演进后请同步更新；涉及版本、容器名、具体数值的地方会标注“以实际为准”。 > 遇到不确定的信息，先查代码/任务现场，不要凭记忆下结论。 +> +> **报错查询**：任务失败时先看 [`error_catalog/`](error_catalog/README.md) —— 那里按类别收了 +> DKAO 真实出现过的报错（原文 / 判定 / 根因 / 处置 / 是否已修），一条命令即可重新收集： +> `python3 -m metainfer.tasks.dcu_kernel_auto_opt.error_catalog.collect_errors`。 +> 遇到新报错请补进那里（规则加在 `collect_errors.py` 的 `CLASSES`，解释写在对应 `NN-*.md`）。 ## 0. 这个功能是什么 @@ -114,11 +119,28 @@ wo_b 19882、gate_up 8790、down 5546；wq_b 与 indexer.wq_b 共用一条）。 1. 用 worker 的 accepted objects 重建 `final/source`：`git restore` → 拷贝 object/HIP → `llvm-objcopy` 符号命名（`mi__` 前缀）→ 渲染 `csrc/w8a8_dispatch.cpp` → 写 `artifact_manifest.json`。 -2. **dispatch 只按 (N,K) 路由、忽略 M**（见 w8a8_dispatch.cpp），所以回归 shape 也会走自定义 object。 -3. 验证集合 = 6 个优化 shape（默认 protocol）+ 全部 API fallback shapes（`warmups=2, samples=3`）。 - 优化 shape 还有 `final ≤ 1.05× worker best` 的性能门。 -4. 任一 shape 正确性失败 → `Final prebuilt-object W8A8 validation failed: correctness failed for : `。 -5. 全部通过 → 提交 final worktree、cherry-pick 回 seed、写 `final_report.json`，任务 finished/success。 +2. **dispatch 按整个 shape `(m,n,k)` 路由**（见 `_render_prebuilt_dispatch` 的 docstring）：族内形成递减 M 链， + 精确 M 命中自己的 kernel，没优化过的 M 退到不超过它的最大 M；**pack 路由只能按 `(N,K)` 一条** + （`launch_pack_w8a8_weight` 的 ABI 里没有 `m`，真实模型里一份权重只 pack 一次）， + 取族内最小 M 的 pack 内核。 +3. **合并前的布局门（`final_layout_gate`）**：对每个 `(N,K)` 族用**成员各自的 pack 内核**在**真实 (N,K)** 上 + 打一次包（字节计数图），取 SHA256 当布局指纹。指纹不同 = 不能证明互相读得懂 → 按指纹分组、 + 打成最少的批（`pack_batches`），每批自己的 dispatch + prebuilt，串行验收一批一批来； + **批 0 渲染进 `final/source` 并发布，其余留在 `final/layout_batches/batchN`**。 + - 探针**可运行**且所有成员指纹相同 → 单批（历史行为，零代价）。 + - 探针**测不到任何成员**（无对象 / 无 GPU / 子进程失败）→ 一致性**未被证明**， + 多成员族**按成员逐个拆批**（`unproven_groups`），并发 `final_layout_probe_failed` 事件。 + 探针失败**不废任务**，但也**不再被当成"没有冲突"**——它只多花一批的时间。 + - 探针子进程必须用 `metainfer.orchestrator.interpreter.python_executable()` 解析解释器： + WebUI 起的 orchestrator 里 `sys.executable` 是**空串**（launcher 把 `argv[0]` 写成 + `metainfer-orchestrator`），`subprocess.run([''])` 会抛 + `PermissionError: [Errno 13] Permission denied: ''`。2026-10-11 前这道门因此对 + WebUI 起的任务**全程失效**（见 `wiki-metainfer/gotchas/layout-gate-probe-blind-under-webui.md`）。 +4. 验证集合 = 6 个优化 shape（默认 protocol）+ 全部 API fallback shapes（`warmups=2, samples=3`）。 + 优化 shape 还有 `final ≤ worker best + max(5%, 2 µs)` 的性能门（5% 与 2 µs 取更宽： + 40 µs 以下由 2 µs 绝对下限兜底，40 µs 以上与旧规则完全一致）。 +5. 任一 shape 正确性失败 → `Final prebuilt-object W8A8 validation failed: correctness failed for : `。 +6. 全部通过 → 提交 final worktree、cherry-pick 回 seed、写 `final_report.json`，任务 finished/success。 ### 2.6 主 agent 与子 agent 的职责边界（严格区分，不是靠自觉） @@ -181,10 +203,11 @@ TP8 另有 wq_b/wo_b (1024,4096)、gate_up (4096,512)、down (256,4096)、indexe 默认 42 不变，串行验证 fallback 不受影响）；前端三个模型的 TP8 topology 加 `mValues`（用 `W8A8_MODEL_TP8_M_VALUES` 常量）；基线表 15 条 `(8,4096,…)` 已实测 （`tools/baseline/bench_triton_tp8_m4096.py`，int8_utils.matmul_kernel + CUDA-graph replay， - 结果存 `tools/baseline/tp8_m4096_graph.json`）。DeepSeek TP8 不加 M=4096。 + 结果存 `tools/baseline/tp8_m4096_graph.json`）。当时 DeepSeek TP8 不加 M=4096 + （2026-10-11 已放开，见下条）。 - **模型目录 TP8 短 decode 边界（2026-09-20）**：Hy3/MiniMax M3/GLM5.2 的 TP8 形状再加 M=8，即 `MODEL_TP8_EXTRA_OPTIMIZATION_M_VALUES=(8, 4096)`；前端 - `W8A8_MODEL_TP8_M_VALUES = [2, 8, 16, 3072, 4096]`（DeepSeek TP1/TP8 仍是 + `W8A8_MODEL_TP8_M_VALUES = [2, 8, 16, 3072, 4096]`（当时 DeepSeek TP1/TP8 仍是 `W8A8_M_VALUES=[2,16,3072]`，TP4 仍是 `W8A8_TP4_M_VALUES=[2,16,3072,4096]`）。 基线表 15 条 `(8,8,…)` 在 worker29/gfx928 实测三次 （`tools/baseline/bench_triton_tp8_m8.py --m 8 --mode graph`，协议同 2026-08-12 目录： @@ -192,10 +215,24 @@ TP8 另有 wq_b/wo_b (1024,4096)、gate_up (4096,512)、down (256,4096)、indexe 公布值取「每次 pass median 再取 median」；原始数据 `tools/baseline/tp8_m8_graph_run{1,2,3}.json`，汇总 `tp8_m8_graph_merged.json`。 12/15 形状三次 pass 极差 ≤1.6%（共享卡上 pass 2 的 minimax shared_down_proj 被同租户 - 干扰到 33.76 µs，取 median 后为 19.98 µs）。DeepSeek TP8 不加 M=8。 + 干扰到 33.76 µs，取 median 后为 19.98 µs）。当时 DeepSeek TP8 不加 M=8。 +- **DeepSeek V4 Flash TP8 并入 M=8 / M=4096（2026-10-11）**：DeepSeek TP8 topology 加 + `mValues: W8A8_MODEL_TP8_M_VALUES`（四个模型目录现在同一套 5 个 M 值）。 + **默认 workload 不变**：契约 `DEFAULT_OPTIMIZATION_SHAPES` 仍 42 条（M=2/16/3072， + TP4 另加 4096），前端的空字段默认列表用同一集合 —— `tests/test_shape_catalog_panel.py` + 现在会**逐一比对**前端默认列表与契约默认集合，别再让面板扩 M 顺手改掉默认集。 + 基线在**本机风冷档**实测（DeepSeek TP8 的 4 个新 (N,K) × 2 个 M；`(1024,4096)` 键本来 + 就有 = Hy3 TP8 o_proj / wq_b_or_wo_b，只复核未覆盖）： + `baselines/air-k100ai.json` 的 8 条新键，原始 pass 存 + `tools/baseline/tp8_{m8,m4096}_graph_deepseek_air_20261011_run{1,2,3}.json`。 + 实测自检（同卡、同协议）：M=16 `wqkv_a` 复现 79.084→80.064 µs； + `(1024,4096)` M=4096 复现 16403.5→16397.738 µs。 + ⚠️ **液冷档（j20r1n08 / liquid-k500sm）表里没有这 8 条** —— 在那边选这些 shape 会在 + `baseline` 阶段 `BaselineUnavailable`，需要在液冷机按同一协议补测（不要跨档抄数）。 - **面板/契约/基线三层一致性测试**：`tests/test_shape_catalog_panel.py` 用 node 真实求值 `static/dkao-shape-input.js` 的 `MODEL_WORKLOADS`，断言面板给出的每个 TP4/TP8 shape - 都能过契约校验且能查到基线（否则任务要到 baseline 阶段才报 ValueError）。 + 都能过契约校验且能查到基线（否则任务要到 baseline 阶段才报 ValueError）， + 并要求 M=8/M=4096 的基线**来自本机档位**（`profile_mismatch is False`）。 - `MIN_M=1, MAX_M=4096`；`WORKSPACE_BUDGET_BYTES=16MB`。 - **M=4096 时大部分 (N,K) 的 split-K workspace 容量为 0** → 大 M kernel 必须走 2D M-tile 路径， 不能依赖 split-K workspace。 @@ -215,10 +252,13 @@ TP8 另有 wq_b/wo_b (1024,4096)、gate_up (4096,512)、down (256,4096)、indexe ## 5. 常见坑（都是真实踩过的） 1. **布局一致性（最容易翻车）**：object 的 `pack_weight` 如果重排了 B（例：down_proj (512,4096) - 打成 N-blocked 64 列），它内部**所有按 M 分发的路径必须读同一个布局**。dispatch 按 (N,K) 忽略 M， - 回归 M（2/16/3072）会走进通用路径；通用 DUMMA 若按 row-major 读 N-blocked B 就全错。 + 打成 N-blocked 64 列），它内部**所有按 M 分发的路径必须读同一个布局**。回归 M（2/16/3072） + 会走进族内递减 M 链里不超过它的那条路由；若该 kernel 按 row-major 读 N-blocked B 就全错。 修复示例：`dumma_eligible && (n,k)==(4096,512)` 时改用 N-blocked-aware 的 64x64 tile kernel， 精确 M=4096 的 128x64 路径保持不变。 + **注意**：合并时 `(N,K)` 族只有一条 pack 路由，所以**同一权重的多个 M 必须共享一份 packed 布局** —— + 这既是 §2.5 那道布局门在查的东西，也是"一个任务里放同一算子的多个 M"最容易死在第 5 步的原因。 + **根治（未做）**：让 worker 从契约层就遵守"一份权重一份布局"，而不是靠合并期拆批兜。 2. **worker 只验证分配到的 M** → 回归 bug 拖到 serial validate 才暴露。 建议：worker 验收时对每个 (K,N) 额外跑 M=2/16/3072 正确性（M-sweep）。 3. **不要手改任务仓库的 API 快照**（digest mismatch）；改 @@ -253,6 +293,13 @@ TP8 另有 wq_b/wo_b (1024,4096)、gate_up (4096,512)、down (256,4096)、indexe 7. **角色越界会自动判失败**：`generate_role_violation` / `changed files outside its HIP ownership` / `agent changed control-plane-owned extension infrastructure` 都是控制面在 agent 返回后 用 git diff/digest 检出的。修复方式是恢复被改文件、重试该 attempt，**不要绕过检查**。 +8. **合并期的门可能是"瞎的"，而瞎和"没冲突"在页面上长得一样**：布局探针失败时 + `final_artifact_link_launch.layout_groups.probed=false`、`probe_error` 里才是原因 + （2026-10-11 前是 `PermissionError: [Errno 13] Permission denied: ''` —— WebUI 起的 + orchestrator 里 `sys.executable` 是空串，探针子进程起不来）。 + 现已在 `final_layout_gate` 修掉（统一走 `metainfer.orchestrator.interpreter.python_executable()`）， + 并且**探针测不到东西时按成员拆批**、另发 `final_layout_probe_failed` 事件。 + 看一个任务的合并行为：查这两个事件的 `batches` / `probe_error`，别只看最后那行 error。 ## 6. 新 AI 快速开始 checklist diff --git a/metainfer/tasks/dcu_kernel_auto_opt/api/int8w8a8gemm/int8_w8a8_gemm_api.py b/metainfer/tasks/dcu_kernel_auto_opt/api/int8w8a8gemm/int8_w8a8_gemm_api.py index 5f52266f..dedddae2 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/api/int8w8a8gemm/int8_w8a8_gemm_api.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/api/int8w8a8gemm/int8_w8a8_gemm_api.py @@ -37,24 +37,7 @@ from typing import Final, Mapping -try: - import torch -except ModuleNotFoundError: # Metadata validation runs in CPU-only CI. - torch = None # type: ignore[assignment] - - -def _torch_no_grad(): - """Return PyTorch's decorator, or an import-only CI fallback.""" - if torch is None: - return lambda function: function - return torch.no_grad() - - -def _require_torch() -> None: - if torch is None: - raise RuntimeError( - "the INT8 W8A8 runtime API requires PyTorch in the DCU environment" - ) +import torch # These are logical (K, N) dimensions after TP partitioning, not checkpoint @@ -193,18 +176,26 @@ def _merge_kn_tables( # the original three M values, so each TP4 operator gets exactly one M=4096 # shape while TP8 defaults are unchanged. TP4_EXTRA_OPTIMIZATION_M_VALUES: Final[tuple[int, ...]] = (4096,) -# Model-catalog TP8 extra M values. The DeepSeek TP8 default workload keeps -# the original three M values; Hy3 / MiniMax M3 / GLM5.2 TP8 operators are -# additionally optimizable at these M values via "Selected shapes only" -# (their (K,N) pairs are already in the allowed sets and their Triton -# CUDA-graph baselines are measured). This constant is not applied by -# _default_optimization_shapes(), so DEFAULT_OPTIMIZATION_SHAPES -# (DeepSeek-only) and its serial-validation fallback scope are unchanged. +# Model-catalog TP8 extra M values. The DeepSeek TP8 *default* workload keeps +# the original three M values; the Hy3 / MiniMax M3 / GLM5.2 TP8 operators (and, +# since 2026-10-11, the DeepSeek V4 Flash TP8 operators) are additionally +# optimizable at these M values via "Selected shapes only" (their (K,N) pairs +# are already in the allowed sets and their Triton CUDA-graph baselines are +# measured). This constant is not applied by _default_optimization_shapes(), so +# DEFAULT_OPTIMIZATION_SHAPES (DeepSeek-only) and its serial-validation fallback +# scope are unchanged. # M=4096: model-catalog large-prefill boundary added on 2026-08-27. # M=8: model-catalog short decode boundary added on 2026-09-20; the # Triton Graph baselines for the 15 catalog (K, N) pairs were # measured on worker29/gfx928 with tools/baseline/ # bench_triton_tp8_m8.py (raw passes in tp8_m8_graph_run{1,2,3}.json). +# DeepSeek TP8 (2026-10-11): the same two boundaries for the DeepSeek V4 Flash +# TP8 catalog -- 4 new (N,K) keys per M (wqkv_a 1536x4096, +# indexer.wq_b 8192x1024, shared_gate_up_proj 512x4096, +# shared_down_proj 4096x256; the (1024,4096) key already existed for +# wq_b/wo_b). Measured on the *air* board (kme6) with the same two +# tools; raw passes in tools/baseline/ +# tp8_{m8,m4096}_graph_deepseek_air_20261011_run{1,2,3}.json. MODEL_TP8_EXTRA_OPTIMIZATION_M_VALUES: Final[tuple[int, ...]] = (8, 4096) @@ -264,7 +255,6 @@ def _check_cuda_tensor( *, contiguous: bool = True, ) -> None: - _require_torch() if not isinstance(tensor, torch.Tensor): raise TypeError(f"{name} must be a torch.Tensor") if not tensor.is_cuda: @@ -327,7 +317,6 @@ def validate_optimization_shape(shape: Mapping[str, object]) -> None: def _optional_op(namespace: str, name: str): - _require_torch() try: return getattr(getattr(torch.ops, namespace), name) except AttributeError: @@ -344,7 +333,7 @@ def _required_op(namespace: str, name: str): return op -@_torch_no_grad() +@torch.no_grad() def prepare_weight( raw_weight: torch.Tensor, weight_scale: torch.Tensor, @@ -485,7 +474,7 @@ def validate_gemm_out_inputs( return m, n, k -@_torch_no_grad() +@torch.no_grad() def w8a8_gemm_out( x_q: torch.Tensor, packed_weight: torch.Tensor, diff --git a/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py b/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py index af7af3ab..30a77717 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py @@ -15,10 +15,7 @@ import time from pathlib import Path -try: - import torch -except ModuleNotFoundError: # Allow CPU-only CI to import pure helpers. - torch = None # type: ignore[assignment] +import torch #: Device admission rule for anything that measures: VRAM <= 90% and HCU == 0. @@ -338,10 +335,6 @@ def prepare_reference(m: int, n: int, k: int, cache_dir: Path) -> Path: def main() -> int: - if torch is None: - raise RuntimeError( - "w8a8_bench.py requires PyTorch in the DCU benchmark environment" - ) parser = argparse.ArgumentParser() parser.add_argument("--source", type=Path) parser.add_argument("--m", type=int) diff --git a/metainfer/tasks/dcu_kernel_auto_opt/baselines/air-k100ai.json b/metainfer/tasks/dcu_kernel_auto_opt/baselines/air-k100ai.json new file mode 100644 index 00000000..9a1e64b0 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/baselines/air-k100ai.json @@ -0,0 +1,981 @@ +{ + "schema_version": 1, + "hw_profile": "air-k100ai", + "machine": "kme6", + "device": "cuda:4 (K100_AI)", + "card": "K100_AI / DEV-AI (AI firmware line, no HBM address hashing)", + "measured_on": "2026-09-29 + 2026-10-11 (DeepSeek TP8 M=8/M=4096)", + "kind": "local", + "protocol": { + "passes": 3, + "warmups": 10, + "samples": 20, + "launches_per_sample": 5, + "mode": "graph", + "aggregate": "per-shape median of 3 independent passes", + "kernel": "lmslim int8_utils.matmul_kernel (the W8A8 Triton baseline)" + }, + "config": { + "BLOCK_SIZE_M": 16, + "BLOCK_SIZE_N": 32, + "BLOCK_SIZE_K": 256, + "GROUP_SIZE_M": 4, + "SPLIT_K": 1, + "num_stages": 0, + "num_warps": 4 + }, + "notes": [ + "Measured on the air-cooled board because baselines are hardware", + "dependent: the reference (liquid) table is 1.06x-1.40x faster, and the", + "gap is operator dependent (worst on small-K read-heavy shapes).", + "Coverage is intentionally partial: only the shapes measured so far are", + "listed. Shapes missing here fall back to the reference table with", + "profile_mismatch=true until they are measured on this board.", + "Raw passes: baseline/tp8_m8_air_20260928/tp8_m8_graph_run{1,2,3}.json", + "2026-10-11: added the 8 DeepSeek V4 Flash TP8 keys for M=8 and M=4096 (wqkv_a, indexer.wq_b, shared_gate_up_proj, shared_down_proj - the (K,N)=(1024,4096) key already existed as wq_b_or_wo_b and was reproduced, not overwritten) so the model panel catalog resolves on this board.", + "Protocol unchanged and cross-checked on the same card: this run reproduced the published M=16 wqkv_a value (79.084 -> 80.064 us) and the (1024,4096) M=4096 key (16403.5 -> 16397.738 us).", + "Raw passes: tools/baseline/tp8_m8_graph_deepseek_air_20261011_run{1,2,3}.json and tools/baseline/tp8_m4096_graph_deepseek_air_20261011_run{1,2,3}.json" + ], + "graph": { + "8,8,1280,4096": [ + "qkv_proj", + 80.768 + ], + "8,8,4096,1024": [ + "o_proj", + 39.936 + ], + "8,8,384,4096": [ + "shared_gate_up_proj", + 63.824 + ], + "8,8,4096,192": [ + "shared_down_proj", + 20.128 + ], + "8,8,1280,6144": [ + "qkv_proj", + 104.736 + ], + "8,8,1536,6144": [ + "qkv_proj_and_indexer_qk", + 106.032 + ], + "8,8,6144,1024": [ + "o_proj", + 39.52 + ], + "8,8,768,6144": [ + "shared_gate_up_proj", + 100.336 + ], + "8,8,6144,384": [ + "shared_down_proj", + 25.936 + ], + "8,8,2624,6144": [ + "fused_qkv_a_proj", + 94.768 + ], + "8,8,2048,2048": [ + "q_b_proj", + 43.84 + ], + "8,8,3584,512": [ + "kv_b_proj", + 21.408 + ], + "8,8,6144,2048": [ + "o_proj", + 66.96 + ], + "8,8,512,6144": [ + "shared_gate_up_proj", + 97.584 + ], + "8,8,6144,256": [ + "shared_down_proj", + 21.168 + ], + "8,8,1536,4096": [ + "wqkv_a", + 74.464 + ], + "8,8,8192,1024": [ + "indexer.wq_b", + 61.184 + ], + "8,8,512,4096": [ + "shared_gate_up_proj", + 67.008 + ], + "8,8,4096,256": [ + "shared_down_proj", + 19.552 + ], + "8,4096,1280,4096": [ + "qkv_proj", + 16439.425 + ], + "8,4096,4096,1024": [ + "o_proj", + 16403.5 + ], + "8,4096,384,4096": [ + "shared_gate_up_proj", + 5961.593 + ], + "8,4096,4096,192": [ + "shared_down_proj", + 4644.426 + ], + "8,4096,1280,6144": [ + "qkv_proj", + 22143.989 + ], + "8,4096,1536,6144": [ + "qkv_proj_and_indexer_qk", + 28560.428 + ], + "8,4096,6144,1024": [ + "o_proj", + 23849.329 + ], + "8,4096,768,6144": [ + "shared_gate_up_proj", + 12796.697 + ], + "8,4096,6144,384": [ + "shared_down_proj", + 10430.34 + ], + "8,4096,2624,6144": [ + "fused_qkv_a_proj", + 57764.743 + ], + "8,4096,2048,2048": [ + "q_b_proj", + 14312.817 + ], + "8,4096,3584,512": [ + "kv_b_proj", + 7722.216 + ], + "8,4096,6144,2048": [ + "o_proj", + 45946.132 + ], + "8,4096,512,6144": [ + "shared_gate_up_proj", + 8613.861 + ], + "8,4096,6144,256": [ + "shared_down_proj", + 7961.126 + ], + "8,4096,1536,4096": [ + "wqkv_a", + 20638.416 + ], + "8,4096,8192,1024": [ + "indexer.wq_b", + 32158.441 + ], + "8,4096,512,4096": [ + "shared_gate_up_proj", + 5949.006 + ], + "8,4096,4096,256": [ + "shared_down_proj", + 5434.158 + ], + "4,4096,1536,4096": [ + "wqkv_a", + 20697.343 + ], + "4,4096,8192,1024": [ + "indexer.wq_b", + 32139.554 + ], + "4,4096,4096,2048": [ + "wo_b", + 31753.751 + ], + "4,4096,1024,4096": [ + "shared_gate_up_proj", + 12569.8 + ], + "4,4096,4096,512": [ + "shared_down_proj", + 8988.2 + ], + "4,2,1536,4096": [ + "wqkv_a", + 81.808 + ], + "4,16,1536,4096": [ + "wqkv_a", + 79.76 + ], + "4,2,8192,1024": [ + "wq_b", + 62.592 + ], + "4,16,8192,1024": [ + "wq_b", + 64.128 + ], + "4,2,4096,2048": [ + "wo_b", + 59.584 + ], + "4,16,4096,2048": [ + "wo_b", + 63.12 + ], + "4,2,1024,4096": [ + "shared_gate_up", + 69.136 + ], + "4,16,1024,4096": [ + "shared_gate_up", + 74.88 + ], + "4,2,4096,512": [ + "shared_down", + 28.384 + ], + "4,16,4096,512": [ + "shared_down", + 29.84 + ], + "4,3072,1536,4096": [ + "wqkv_a", + 13958.833 + ], + "4,3072,8192,1024": [ + "wq_b", + 23551.318 + ], + "4,3072,4096,2048": [ + "wo_b", + 21913.068 + ], + "4,3072,1024,4096": [ + "shared_gate_up", + 9035.738 + ], + "4,3072,4096,512": [ + "shared_down", + 6526.313 + ], + "1,2,1536,4096": [ + "wqkv_a", + 79.552 + ], + "1,2,2624,6144": [ + "fused_qkv_a_proj", + 100.032 + ], + "1,2,3072,4096": [ + "shared_gate_up_proj", + 81.68 + ], + "1,2,4096,1536": [ + "shared_down_proj", + 54.08 + ], + "1,2,4096,2048": [ + "shared_down_proj", + 64.384 + ], + "1,2,4096,4096": [ + "shared_gate_up_proj", + 109.104 + ], + "1,2,4096,6144": [ + "shared_gate_up_proj", + 149.12 + ], + "1,2,4096,8192": [ + "wo_b", + 189.408 + ], + "1,2,6144,2048": [ + "shared_down_proj", + 65.52 + ], + "1,2,6144,3072": [ + "shared_down_proj", + 89.328 + ], + "1,2,6144,6144": [ + "shared_gate_up_proj", + 154.544 + ], + "1,2,6144,8192": [ + "o_proj", + 197.36 + ], + "1,2,6144,16384": [ + "o_proj", + 367.04 + ], + "1,2,8192,1024": [ + "wq_b", + 65.424 + ], + "1,2,9216,6144": [ + "qkv_proj", + 236.24 + ], + "1,2,9856,6144": [ + "qkv_proj_and_indexer_qk", + 225.136 + ], + "1,2,10240,4096": [ + "qkv_proj", + 168.8 + ], + "1,2,16384,2048": [ + "q_b_proj", + 161.584 + ], + "1,2,28672,512": [ + "kv_b_proj", + 84.32 + ], + "1,2,32768,1024": [ + "wq_b", + 162.4 + ], + "1,16,1536,4096": [ + "wqkv_a", + 84.352 + ], + "1,16,2624,6144": [ + "fused_qkv_a_proj", + 103.744 + ], + "1,16,3072,4096": [ + "shared_gate_up_proj", + 96.528 + ], + "1,16,4096,1536": [ + "shared_down_proj", + 54.688 + ], + "1,16,4096,2048": [ + "shared_down_proj", + 66.896 + ], + "1,16,4096,4096": [ + "shared_gate_up_proj", + 110.72 + ], + "1,16,4096,6144": [ + "shared_gate_up_proj", + 153.616 + ], + "1,16,4096,8192": [ + "wo_b", + 210.304 + ], + "1,16,6144,2048": [ + "shared_down_proj", + 67.968 + ], + "1,16,6144,3072": [ + "shared_down_proj", + 90.608 + ], + "1,16,6144,6144": [ + "shared_gate_up_proj", + 156.784 + ], + "1,16,6144,8192": [ + "o_proj", + 207.456 + ], + "1,16,6144,16384": [ + "o_proj", + 378.608 + ], + "1,16,8192,1024": [ + "wq_b", + 67.904 + ], + "1,16,9216,6144": [ + "qkv_proj", + 240.512 + ], + "1,16,9856,6144": [ + "qkv_proj_and_indexer_qk", + 237.088 + ], + "1,16,10240,4096": [ + "qkv_proj", + 183.12 + ], + "1,16,16384,2048": [ + "q_b_proj", + 168.064 + ], + "1,16,28672,512": [ + "kv_b_proj", + 86.656 + ], + "1,16,32768,1024": [ + "wq_b", + 167.76 + ], + "1,3072,1536,4096": [ + "wqkv_a", + 13959.315 + ], + "1,3072,2624,6144": [ + "fused_qkv_a_proj", + 41782.234 + ], + "1,3072,3072,4096": [ + "shared_gate_up_proj", + 31746.498 + ], + "1,3072,4096,1536": [ + "shared_down_proj", + 16318.993 + ], + "1,3072,4096,2048": [ + "shared_down_proj", + 21915.211 + ], + "1,3072,4096,4096": [ + "shared_gate_up_proj", + 43841.151 + ], + "1,3072,4096,6144": [ + "shared_gate_up_proj", + 62162.958 + ], + "1,3072,4096,8192": [ + "wo_b", + 83061.392 + ], + "1,3072,6144,2048": [ + "shared_down_proj", + 35111.54 + ], + "1,3072,6144,3072": [ + "shared_down_proj", + 50185.201 + ], + "1,3072,6144,6144": [ + "shared_gate_up_proj", + 99491.711 + ], + "1,3072,6144,8192": [ + "o_proj", + 132177.411 + ], + "1,3072,6144,16384": [ + "o_proj", + 255563.354 + ], + "1,3072,8192,1024": [ + "wq_b", + 23547.578 + ], + "1,3072,9216,6144": [ + "qkv_proj", + 145413.721 + ], + "1,3072,9856,6144": [ + "qkv_proj_and_indexer_qk", + 161061.969 + ], + "1,3072,10240,4096": [ + "qkv_proj", + 115026.978 + ], + "1,3072,16384,2048": [ + "q_b_proj", + 90817.047 + ], + "1,3072,28672,512": [ + "kv_b_proj", + 46670.041 + ], + "1,3072,32768,1024": [ + "wq_b", + 93940.244 + ], + "4,2,768,4096": [ + "shared_gate_up_proj", + 72.864 + ], + "4,2,1024,6144": [ + "shared_gate_up_proj", + 112.496 + ], + "4,2,1536,6144": [ + "shared_gate_up_proj", + 113.248 + ], + "4,2,2304,6144": [ + "qkv_proj", + 114.336 + ], + "4,2,2560,4096": [ + "qkv_proj", + 82.496 + ], + "4,2,2560,6144": [ + "qkv_proj_and_indexer_qk", + 118.448 + ], + "4,2,2624,6144": [ + "fused_qkv_a_proj", + 100.816 + ], + "4,2,4096,384": [ + "shared_down_proj", + 29.024 + ], + "4,2,6144,512": [ + "shared_down_proj", + 29.44 + ], + "4,2,6144,768": [ + "shared_down_proj", + 36.928 + ], + "4,2,6144,2048": [ + "o_proj", + 67.216 + ], + "4,2,6144,4096": [ + "o_proj", + 112.336 + ], + "4,2,7168,512": [ + "kv_b_proj", + 29.216 + ], + "4,16,768,4096": [ + "shared_gate_up_proj", + 79.328 + ], + "4,16,1024,6144": [ + "shared_gate_up_proj", + 112.528 + ], + "4,16,1536,6144": [ + "shared_gate_up_proj", + 115.248 + ], + "4,16,2304,6144": [ + "qkv_proj", + 117.648 + ], + "4,16,2560,4096": [ + "qkv_proj", + 94.352 + ], + "4,16,2560,6144": [ + "qkv_proj_and_indexer_qk", + 117.328 + ], + "4,16,2624,6144": [ + "fused_qkv_a_proj", + 103.792 + ], + "4,16,4096,384": [ + "shared_down_proj", + 29.152 + ], + "4,16,6144,512": [ + "shared_down_proj", + 30.208 + ], + "4,16,6144,768": [ + "shared_down_proj", + 38.592 + ], + "4,16,6144,2048": [ + "o_proj", + 69.936 + ], + "4,16,6144,4096": [ + "o_proj", + 115.472 + ], + "4,16,7168,512": [ + "kv_b_proj", + 30.64 + ], + "4,3072,768,4096": [ + "shared_gate_up_proj", + 6639.05 + ], + "4,3072,1024,6144": [ + "shared_gate_up_proj", + 12757.482 + ], + "4,3072,1536,6144": [ + "shared_gate_up_proj", + 19376.759 + ], + "4,3072,2304,6144": [ + "qkv_proj", + 32814.705 + ], + "4,3072,2560,4096": [ + "qkv_proj", + 26482.025 + ], + "4,3072,2560,6144": [ + "qkv_proj_and_indexer_qk", + 37226.559 + ], + "4,3072,2624,6144": [ + "fused_qkv_a_proj", + 41989.261 + ], + "4,3072,4096,384": [ + "shared_down_proj", + 5296.845 + ], + "4,3072,6144,512": [ + "shared_down_proj", + 10050.745 + ], + "4,3072,6144,768": [ + "shared_down_proj", + 13554.101 + ], + "4,3072,6144,2048": [ + "o_proj", + 35130.563 + ], + "4,3072,6144,4096": [ + "o_proj", + 70870.343 + ], + "4,3072,7168,512": [ + "kv_b_proj", + 11325.255 + ], + "4,4096,768,4096": [ + "shared_gate_up_proj", + 9129.209 + ], + "4,4096,1024,6144": [ + "shared_gate_up_proj", + 17584.737 + ], + "4,4096,1536,6144": [ + "shared_gate_up_proj", + 28511.801 + ], + "4,4096,2304,6144": [ + "qkv_proj", + 46073.257 + ], + "4,4096,2560,4096": [ + "qkv_proj", + 35980.676 + ], + "4,4096,2560,6144": [ + "qkv_proj_and_indexer_qk", + 51360.352 + ], + "4,4096,2624,6144": [ + "fused_qkv_a_proj", + 57740.244 + ], + "4,4096,4096,384": [ + "shared_down_proj", + 7125.867 + ], + "4,4096,6144,512": [ + "shared_down_proj", + 13143.06 + ], + "4,4096,6144,768": [ + "shared_down_proj", + 17832.851 + ], + "4,4096,6144,2048": [ + "o_proj", + 45972.914 + ], + "4,4096,6144,4096": [ + "o_proj", + 92068.274 + ], + "4,4096,7168,512": [ + "kv_b_proj", + 15406.099 + ], + "8,2,384,4096": [ + "shared_gate_up_proj", + 69.904 + ], + "8,2,512,4096": [ + "shared_gate_up_proj", + 73.536 + ], + "8,2,512,6144": [ + "shared_gate_up_proj", + 100.32 + ], + "8,2,768,6144": [ + "shared_gate_up_proj", + 105.792 + ], + "8,2,1280,4096": [ + "qkv_proj", + 77.648 + ], + "8,2,1280,6144": [ + "qkv_proj", + 114.64 + ], + "8,2,1536,4096": [ + "wqkv_a", + 80.304 + ], + "8,2,1536,6144": [ + "qkv_proj_and_indexer_qk", + 113.632 + ], + "8,2,2048,2048": [ + "q_b_proj", + 46.544 + ], + "8,2,2624,6144": [ + "fused_qkv_a_proj", + 101.616 + ], + "8,2,3584,512": [ + "kv_b_proj", + 23.872 + ], + "8,2,4096,192": [ + "shared_down_proj", + 23.232 + ], + "8,2,4096,256": [ + "shared_down_proj", + 24.256 + ], + "8,2,4096,1024": [ + "wq_b_or_wo_b", + 44.768 + ], + "8,2,6144,256": [ + "shared_down_proj", + 23.824 + ], + "8,2,6144,384": [ + "shared_down_proj", + 29.248 + ], + "8,2,6144,1024": [ + "o_proj", + 43.296 + ], + "8,2,6144,2048": [ + "o_proj", + 67.408 + ], + "8,2,8192,1024": [ + "wq_b", + 66.928 + ], + "8,16,384,4096": [ + "shared_gate_up_proj", + 66.552 + ], + "8,16,512,6144": [ + "shared_gate_up_proj", + 94.112 + ], + "8,16,768,6144": [ + "shared_gate_up_proj", + 99.984 + ], + "8,16,1280,4096": [ + "qkv_proj", + 77.368 + ], + "8,16,1280,6144": [ + "qkv_proj", + 106.732 + ], + "8,16,1536,6144": [ + "qkv_proj_and_indexer_qk", + 107.56 + ], + "8,16,2048,2048": [ + "q_b_proj", + 44.604 + ], + "8,16,2624,6144": [ + "fused_qkv_a_proj", + 96.848 + ], + "8,16,3584,512": [ + "kv_b_proj", + 20.964 + ], + "8,16,4096,192": [ + "shared_down_proj", + 20.02 + ], + "8,16,6144,256": [ + "shared_down_proj", + 21.804 + ], + "8,16,6144,384": [ + "shared_down_proj", + 25.804 + ], + "8,16,6144,1024": [ + "o_proj", + 39.528 + ], + "8,16,6144,2048": [ + "o_proj", + 63.532 + ], + "8,3072,384,4096": [ + "shared_gate_up_proj", + 4411.613 + ], + "8,3072,512,6144": [ + "shared_gate_up_proj", + 6326.731 + ], + "8,3072,768,6144": [ + "shared_gate_up_proj", + 9610.361 + ], + "8,3072,1280,4096": [ + "qkv_proj", + 11585.751 + ], + "8,3072,1280,6144": [ + "qkv_proj", + 16478.803 + ], + "8,3072,1536,6144": [ + "qkv_proj_and_indexer_qk", + 19368.032 + ], + "8,3072,2048,2048": [ + "q_b_proj", + 10070.072 + ], + "8,3072,2624,6144": [ + "fused_qkv_a_proj", + 41974.527 + ], + "8,3072,3584,512": [ + "kv_b_proj", + 5548.284 + ], + "8,3072,4096,192": [ + "shared_down_proj", + 3478.653 + ], + "8,3072,6144,256": [ + "shared_down_proj", + 6042.747 + ], + "8,3072,6144,384": [ + "shared_down_proj", + 7942.937 + ], + "8,3072,6144,1024": [ + "o_proj", + 18312.34 + ], + "8,3072,6144,2048": [ + "o_proj", + 35178.548 + ] + }, + "tp8": { + "8,16,1536,4096": [ + "wqkv_a", + 79.048, + 81.432, + 79.084, + 79.936 + ], + "8,16,4096,1024": [ + "wq_b_or_wo_b", + 61.176, + 61.896, + 41.012, + 41.696 + ], + "8,16,8192,1024": [ + "indexer.wq_b", + 66.06, + 72.592, + 61.708, + 62.448 + ], + "8,16,512,4096": [ + "shared_gate_up", + 70.116, + 71.312, + 69.92, + 70.576 + ], + "8,16,4096,256": [ + "shared_down", + 60.176, + 60.712, + 21.056, + 22.736 + ], + "8,3072,1536,4096": [ + "wqkv_a", + 13938.953, + 14074.615, + 13980.82, + 14037.749 + ], + "8,3072,4096,1024": [ + "wq_b_or_wo_b", + 11445.258, + 11483.352, + 11440.072, + 11474.102 + ], + "8,3072,8192,1024": [ + "indexer.wq_b", + 23571.869, + 23637.262, + 23560.895, + 23584.621 + ], + "8,3072,512,4096": [ + "shared_gate_up", + 4433.549, + 4442.973, + 4422.589, + 4432.989 + ], + "8,3072,4096,256": [ + "shared_down", + 4104.973, + 4134.878, + 4100.317, + 4130.301 + ] + } +} diff --git a/metainfer/tasks/dcu_kernel_auto_opt/baselines/liquid-k500sm.json b/metainfer/tasks/dcu_kernel_auto_opt/baselines/liquid-k500sm.json new file mode 100644 index 00000000..9505bcc5 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/baselines/liquid-k500sm.json @@ -0,0 +1,939 @@ +{ + "schema_version": 1, + "hw_profile": "liquid-k500sm", + "machine": "j20r1n08 (worker29 node id)", + "card": "K500SM_AI / DEV-HPC (hpc firmware line)", + "measured_on": "2026-08-12 .. 2026-09-20", + "kind": "reference", + "protocol": { + "kernel": "lmslim int8_utils.matmul_kernel (the W8A8 Triton baseline)", + "timing": "GPU events, CUDA Graph replay, hot cache", + "warmups": 10, + "samples": 20, + "launches_per_sample": 5, + "config_by_m": { + "M<=32": "BM16/BN32/BK256 GROUP4 SPLIT_K1 stages0 warps4", + "M<=64": "BM32/BN32/BK256", + "M<=128": "BM64/BN64/BK256", + "M<=1024": "BM128/BN128/BK256", + "M>1024": "BM256/BN256/BK64" + }, + "aggregate": "per-shape median of three independent passes" + }, + "notes": [ + "Moved verbatim out of w8a8_baselines.py on 2026-09-29; values unchanged.", + "This is the profile every pre-2026-09-25 artifact in this workspace was", + "produced on, and it is the fallback reference for shapes the air profile", + "has not measured yet (records then carry profile_mismatch=true)." + ], + "graph": { + "4,2,1536,4096": [ + "wqkv_a", + 66.924 + ], + "4,16,1536,4096": [ + "wqkv_a", + 66.597 + ], + "4,2,8192,1024": [ + "wq_b", + 47.181 + ], + "4,16,8192,1024": [ + "wq_b", + 48.049 + ], + "4,2,4096,2048": [ + "wo_b", + 54.453 + ], + "4,16,4096,2048": [ + "wo_b", + 54.617 + ], + "4,2,1024,4096": [ + "shared_gate_up", + 60.389 + ], + "4,16,1024,4096": [ + "shared_gate_up", + 60.965 + ], + "4,2,4096,512": [ + "shared_down", + 18.975 + ], + "4,16,4096,512": [ + "shared_down", + 21.372 + ], + "4,3072,1536,4096": [ + "wqkv_a", + 9488.0 + ], + "4,3072,8192,1024": [ + "wq_b", + 15437.0 + ], + "4,3072,4096,2048": [ + "wo_b", + 14680.0 + ], + "4,3072,1024,4096": [ + "shared_gate_up", + 5985.0 + ], + "4,3072,4096,512": [ + "shared_down", + 4199.0 + ], + "4,4096,1536,4096": [ + "wqkv_a", + 13247.253 + ], + "4,4096,8192,1024": [ + "wq_b", + 20590.225 + ], + "4,4096,4096,2048": [ + "wo_b", + 19881.949 + ], + "4,4096,1024,4096": [ + "shared_gate_up", + 8790.192 + ], + "4,4096,4096,512": [ + "shared_down", + 5545.821 + ], + "1,2,1536,4096": [ + "wqkv_a", + 78.046 + ], + "1,2,2624,6144": [ + "fused_qkv_a_proj", + 96.623 + ], + "1,2,3072,4096": [ + "shared_gate_up_proj", + 81.854 + ], + "1,2,4096,1536": [ + "shared_down_proj", + 47.839 + ], + "1,2,4096,2048": [ + "shared_down_proj", + 62.351 + ], + "1,2,4096,4096": [ + "shared_gate_up_proj", + 109.998 + ], + "1,2,4096,6144": [ + "shared_gate_up_proj", + 151.045 + ], + "1,2,4096,8192": [ + "wo_b", + 194.748 + ], + "1,2,6144,2048": [ + "shared_down_proj", + 66.098 + ], + "1,2,6144,3072": [ + "shared_down_proj", + 92.174 + ], + "1,2,6144,6144": [ + "shared_gate_up_proj", + 155.341 + ], + "1,2,6144,8192": [ + "o_proj", + 190.044 + ], + "1,2,6144,16384": [ + "o_proj", + 371.539 + ], + "1,2,8192,1024": [ + "wq_b", + 55.599 + ], + "1,2,9216,6144": [ + "qkv_proj", + 232.987 + ], + "1,2,9856,6144": [ + "qkv_proj_and_indexer_qk", + 216.684 + ], + "1,2,10240,4096": [ + "qkv_proj", + 175.532 + ], + "1,2,16384,2048": [ + "q_b_proj", + 145.087 + ], + "1,2,28672,512": [ + "kv_b_proj", + 82.624 + ], + "1,2,32768,1024": [ + "wq_b", + 155.197 + ], + "1,16,1536,4096": [ + "wqkv_a", + 80.574 + ], + "1,16,2624,6144": [ + "fused_qkv_a_proj", + 98.463 + ], + "1,16,3072,4096": [ + "shared_gate_up_proj", + 82.638 + ], + "1,16,4096,1536": [ + "shared_down_proj", + 49.439 + ], + "1,16,4096,2048": [ + "shared_down_proj", + 65.567 + ], + "1,16,4096,4096": [ + "shared_gate_up_proj", + 110.606 + ], + "1,16,4096,6144": [ + "shared_gate_up_proj", + 151.685 + ], + "1,16,4096,8192": [ + "wo_b", + 205.644 + ], + "1,16,6144,2048": [ + "shared_down_proj", + 67.346 + ], + "1,16,6144,3072": [ + "shared_down_proj", + 91.838 + ], + "1,16,6144,6144": [ + "shared_gate_up_proj", + 156.653 + ], + "1,16,6144,8192": [ + "o_proj", + 192.412 + ], + "1,16,6144,16384": [ + "o_proj", + 358.435 + ], + "1,16,8192,1024": [ + "wq_b", + 56.847 + ], + "1,16,9216,6144": [ + "qkv_proj", + 235.083 + ], + "1,16,9856,6144": [ + "qkv_proj_and_indexer_qk", + 219.564 + ], + "1,16,10240,4096": [ + "qkv_proj", + 171.069 + ], + "1,16,16384,2048": [ + "q_b_proj", + 146.719 + ], + "1,16,28672,512": [ + "kv_b_proj", + 83.504 + ], + "1,16,32768,1024": [ + "wq_b", + 158.989 + ], + "1,3072,1536,4096": [ + "wqkv_a", + 9795.766 + ], + "1,3072,2624,6144": [ + "fused_qkv_a_proj", + 28944.543 + ], + "1,3072,3072,4096": [ + "shared_gate_up_proj", + 21047.242 + ], + "1,3072,4096,1536": [ + "shared_down_proj", + 11363.852 + ], + "1,3072,4096,2048": [ + "shared_down_proj", + 14921.346 + ], + "1,3072,4096,4096": [ + "shared_gate_up_proj", + 29325.629 + ], + "1,3072,4096,6144": [ + "shared_gate_up_proj", + 43830.049 + ], + "1,3072,4096,8192": [ + "wo_b", + 58484.25 + ], + "1,3072,6144,2048": [ + "shared_down_proj", + 22320.748 + ], + "1,3072,6144,3072": [ + "shared_down_proj", + 33673.799 + ], + "1,3072,6144,6144": [ + "shared_gate_up_proj", + 67255.209 + ], + "1,3072,6144,8192": [ + "o_proj", + 90545.355 + ], + "1,3072,6144,16384": [ + "o_proj", + 182662.598 + ], + "1,3072,8192,1024": [ + "wq_b", + 15305.526 + ], + "1,3072,9216,6144": [ + "qkv_proj", + 103111.029 + ], + "1,3072,9856,6144": [ + "qkv_proj_and_indexer_qk", + 110813.373 + ], + "1,3072,10240,4096": [ + "qkv_proj", + 74301.431 + ], + "1,3072,16384,2048": [ + "q_b_proj", + 59796.738 + ], + "1,3072,28672,512": [ + "kv_b_proj", + 29183.929 + ], + "1,3072,32768,1024": [ + "wq_b", + 62264.438 + ], + "4,2,768,4096": [ + "shared_gate_up_proj", + 71.711 + ], + "4,2,1024,6144": [ + "shared_gate_up_proj", + 108.278 + ], + "4,2,1536,6144": [ + "shared_gate_up_proj", + 109.822 + ], + "4,2,2304,6144": [ + "qkv_proj", + 113.134 + ], + "4,2,2560,4096": [ + "qkv_proj", + 77.854 + ], + "4,2,2560,6144": [ + "qkv_proj_and_indexer_qk", + 111.79 + ], + "4,2,2624,6144": [ + "fused_qkv_a_proj", + 98.756 + ], + "4,2,4096,384": [ + "shared_down_proj", + 21.664 + ], + "4,2,6144,512": [ + "shared_down_proj", + 23.553 + ], + "4,2,6144,768": [ + "shared_down_proj", + 33.407 + ], + "4,2,6144,2048": [ + "o_proj", + 65.135 + ], + "4,2,6144,4096": [ + "o_proj", + 110.501 + ], + "4,2,7168,512": [ + "kv_b_proj", + 25.009 + ], + "4,16,768,4096": [ + "shared_gate_up_proj", + 73.262 + ], + "4,16,1024,6144": [ + "shared_gate_up_proj", + 108.422 + ], + "4,16,1536,6144": [ + "shared_gate_up_proj", + 113.006 + ], + "4,16,2304,6144": [ + "qkv_proj", + 113.422 + ], + "4,16,2560,4096": [ + "qkv_proj", + 80.19 + ], + "4,16,2560,6144": [ + "qkv_proj_and_indexer_qk", + 112.27 + ], + "4,16,2624,6144": [ + "fused_qkv_a_proj", + 98.708 + ], + "4,16,4096,384": [ + "shared_down_proj", + 23.216 + ], + "4,16,6144,512": [ + "shared_down_proj", + 24.977 + ], + "4,16,6144,768": [ + "shared_down_proj", + 34.063 + ], + "4,16,6144,2048": [ + "o_proj", + 67.535 + ], + "4,16,6144,4096": [ + "o_proj", + 112.198 + ], + "4,16,7168,512": [ + "kv_b_proj", + 27.793 + ], + "4,3072,768,4096": [ + "shared_gate_up_proj", + 4988.476 + ], + "4,3072,1024,6144": [ + "shared_gate_up_proj", + 9708.601 + ], + "4,3072,1536,6144": [ + "shared_gate_up_proj", + 14650.233 + ], + "4,3072,2304,6144": [ + "qkv_proj", + 22634.012 + ], + "4,3072,2560,4096": [ + "qkv_proj", + 16984.091 + ], + "4,3072,2560,6144": [ + "qkv_proj_and_indexer_qk", + 25396.649 + ], + "4,3072,2624,6144": [ + "fused_qkv_a_proj", + 28947.94 + ], + "4,3072,4096,384": [ + "shared_down_proj", + 3366.413 + ], + "4,3072,6144,512": [ + "shared_down_proj", + 6270.013 + ], + "4,3072,6144,768": [ + "shared_down_proj", + 8900.2 + ], + "4,3072,6144,2048": [ + "o_proj", + 22307.655 + ], + "4,3072,6144,4096": [ + "o_proj", + 44517.157 + ], + "4,3072,7168,512": [ + "kv_b_proj", + 7308.89 + ], + "4,4096,768,4096": [ + "shared_gate_up_proj", + 6678.282 + ], + "4,4096,1024,6144": [ + "shared_gate_up_proj", + 13036.336 + ], + "4,4096,1536,6144": [ + "shared_gate_up_proj", + 19776.488 + ], + "4,4096,2304,6144": [ + "qkv_proj", + 31556.192 + ], + "4,4096,2560,4096": [ + "qkv_proj", + 24045.12 + ], + "4,4096,2560,6144": [ + "qkv_proj_and_indexer_qk", + 35918.768 + ], + "4,4096,2624,6144": [ + "fused_qkv_a_proj", + 40458.051 + ], + "4,4096,4096,384": [ + "shared_down_proj", + 4381.289 + ], + "4,4096,6144,512": [ + "shared_down_proj", + 8243.047 + ], + "4,4096,6144,768": [ + "shared_down_proj", + 11759.473 + ], + "4,4096,6144,2048": [ + "o_proj", + 29702.473 + ], + "4,4096,6144,4096": [ + "o_proj", + 58712.341 + ], + "4,4096,7168,512": [ + "kv_b_proj", + 9702.386 + ], + "8,2,384,4096": [ + "shared_gate_up_proj", + 64.511 + ], + "8,2,512,4096": [ + "shared_gate_up_proj", + 72.687 + ], + "8,2,512,6144": [ + "shared_gate_up_proj", + 97.606 + ], + "8,2,768,6144": [ + "shared_gate_up_proj", + 104.351 + ], + "8,2,1280,4096": [ + "qkv_proj", + 73.535 + ], + "8,2,1280,6144": [ + "qkv_proj", + 110.638 + ], + "8,2,1536,4096": [ + "wqkv_a", + 74.767 + ], + "8,2,1536,6144": [ + "qkv_proj_and_indexer_qk", + 110.19 + ], + "8,2,2048,2048": [ + "q_b_proj", + 40.371 + ], + "8,2,2624,6144": [ + "fused_qkv_a_proj", + 95.43 + ], + "8,2,3584,512": [ + "kv_b_proj", + 17.873 + ], + "8,2,4096,192": [ + "shared_down_proj", + 16.144 + ], + "8,2,4096,256": [ + "shared_down_proj", + 19.344 + ], + "8,2,4096,1024": [ + "wq_b_or_wo_b", + 36.271 + ], + "8,2,6144,256": [ + "shared_down_proj", + 16.881 + ], + "8,2,6144,384": [ + "shared_down_proj", + 21.968 + ], + "8,2,6144,1024": [ + "o_proj", + 37.407 + ], + "8,2,6144,2048": [ + "o_proj", + 64.948 + ], + "8,2,8192,1024": [ + "wq_b", + 56.239 + ], + "8,16,384,4096": [ + "shared_gate_up_proj", + 66.527 + ], + "8,16,512,6144": [ + "shared_gate_up_proj", + 99.318 + ], + "8,16,768,6144": [ + "shared_gate_up_proj", + 104.527 + ], + "8,16,1280,4096": [ + "qkv_proj", + 74.782 + ], + "8,16,1280,6144": [ + "qkv_proj", + 110.478 + ], + "8,16,1536,6144": [ + "qkv_proj_and_indexer_qk", + 111.486 + ], + "8,16,2048,2048": [ + "q_b_proj", + 42.035 + ], + "8,16,2624,6144": [ + "fused_qkv_a_proj", + 97.99 + ], + "8,16,3584,512": [ + "kv_b_proj", + 18.833 + ], + "8,16,4096,192": [ + "shared_down_proj", + 17.312 + ], + "8,16,6144,256": [ + "shared_down_proj", + 18.177 + ], + "8,16,6144,384": [ + "shared_down_proj", + 23.152 + ], + "8,16,6144,1024": [ + "o_proj", + 38.655 + ], + "8,16,6144,2048": [ + "o_proj", + 66.98 + ], + "8,3072,384,4096": [ + "shared_gate_up_proj", + 3009.348 + ], + "8,3072,512,6144": [ + "shared_gate_up_proj", + 4355.572 + ], + "8,3072,768,6144": [ + "shared_gate_up_proj", + 7087.451 + ], + "8,3072,1280,4096": [ + "qkv_proj", + 8489.847 + ], + "8,3072,1280,6144": [ + "qkv_proj", + 12553.842 + ], + "8,3072,1536,6144": [ + "qkv_proj_and_indexer_qk", + 14614.569 + ], + "8,3072,2048,2048": [ + "q_b_proj", + 6821.235 + ], + "8,3072,2624,6144": [ + "fused_qkv_a_proj", + 29185.017 + ], + "8,3072,3584,512": [ + "kv_b_proj", + 3685.827 + ], + "8,3072,4096,192": [ + "shared_down_proj", + 2043.847 + ], + "8,3072,6144,256": [ + "shared_down_proj", + 3593.156 + ], + "8,3072,6144,384": [ + "shared_down_proj", + 4925.667 + ], + "8,3072,6144,1024": [ + "o_proj", + 11664.713 + ], + "8,3072,6144,2048": [ + "o_proj", + 22305.402 + ], + "8,4096,1280,4096": [ + "qkv_proj", + 19381.383 + ], + "8,4096,4096,1024": [ + "o_proj", + 14151.792 + ], + "8,4096,384,4096": [ + "shared_gate_up_proj", + 16712.231 + ], + "8,4096,4096,192": [ + "shared_down_proj", + 10762.616 + ], + "8,4096,1280,6144": [ + "qkv_proj", + 41000.618 + ], + "8,4096,1536,6144": [ + "qkv_proj_and_indexer_qk", + 35604.698 + ], + "8,4096,6144,1024": [ + "o_proj", + 19897.517 + ], + "8,4096,768,6144": [ + "shared_gate_up_proj", + 31882.257 + ], + "8,4096,6144,384": [ + "shared_down_proj", + 22389.162 + ], + "8,4096,2624,6144": [ + "fused_qkv_a_proj", + 68661.417 + ], + "8,4096,2048,2048": [ + "q_b_proj", + 14385.571 + ], + "8,4096,3584,512": [ + "kv_b_proj", + 6939.939 + ], + "8,4096,6144,2048": [ + "o_proj", + 54775.797 + ], + "8,4096,512,6144": [ + "shared_gate_up_proj", + 21254.589 + ], + "8,4096,6144,256": [ + "shared_down_proj", + 17700.262 + ], + "8,8,1280,4096": [ + "qkv_proj", + 66.432 + ], + "8,8,4096,1024": [ + "o_proj", + 31.472 + ], + "8,8,384,4096": [ + "shared_gate_up_proj", + 57.008 + ], + "8,8,4096,192": [ + "shared_down_proj", + 14.592 + ], + "8,8,1280,6144": [ + "qkv_proj", + 98.865 + ], + "8,8,1536,6144": [ + "qkv_proj_and_indexer_qk", + 99.473 + ], + "8,8,6144,1024": [ + "o_proj", + 32.96 + ], + "8,8,768,6144": [ + "shared_gate_up_proj", + 92.928 + ], + "8,8,6144,384": [ + "shared_down_proj", + 19.984 + ], + "8,8,2624,6144": [ + "fused_qkv_a_proj", + 87.2 + ], + "8,8,2048,2048": [ + "q_b_proj", + 36.704 + ], + "8,8,3584,512": [ + "kv_b_proj", + 15.808 + ], + "8,8,6144,2048": [ + "o_proj", + 58.864 + ], + "8,8,512,6144": [ + "shared_gate_up_proj", + 88.448 + ], + "8,8,6144,256": [ + "shared_down_proj", + 15.136 + ] + }, + "tp8": { + "8,16,1536,4096": [ + "wqkv_a", + 97.66, + 98.404, + 66.591, + 67.347 + ], + "8,16,4096,1024": [ + "wq_b_or_wo_b", + 95.284, + 96.092, + 32.137, + 35.969 + ], + "8,16,8192,1024": [ + "indexer.wq_b", + 96.204, + 96.796, + 48.274, + 50.794 + ], + "8,16,512,4096": [ + "shared_gate_up", + 96.056, + 97.788, + 59.21, + 59.962 + ], + "8,16,4096,256": [ + "shared_down", + 94.54, + 95.74, + 14.849, + 18.001 + ], + "8,3072,1536,4096": [ + "wqkv_a", + 9423.243, + 9446.877, + 9422.539, + 9486.942 + ], + "8,3072,4096,1024": [ + "wq_b_or_wo_b", + 7637.209, + 8033.128, + 7617.88, + 7686.395 + ], + "8,3072,8192,1024": [ + "indexer.wq_b", + 15461.0, + 15528.571, + 15466.648, + 15958.569 + ], + "8,3072,512,4096": [ + "shared_gate_up", + 2775.526, + 2780.134, + 2740.757, + 2751.685 + ], + "8,3072,4096,256": [ + "shared_down", + 2414.377, + 2424.569, + 2370.583, + 2376.343 + ] + } +} diff --git a/metainfer/tasks/dcu_kernel_auto_opt/bridge/dsh/tests/test_dsh_agent.py b/metainfer/tasks/dcu_kernel_auto_opt/bridge/dsh/tests/test_dsh_agent.py index bd2d9b53..a4007444 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/bridge/dsh/tests/test_dsh_agent.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/bridge/dsh/tests/test_dsh_agent.py @@ -98,16 +98,6 @@ def __init__(self, text): def run_wrapper(module, argv, prompt, backend="sdk"): - """Run the wrapper with a scripted prompt/stdin, plus a fake API key. - - ``dsh_agent.main()`` refuses to start without credentials (TENCENT_API_KEY - / DEEPSEEK_API_KEY env or ~/.dsh/.credentials.yaml). CI runners have none - of those, so inject a dummy key for the duration of the call to keep the - tests hermetic and environment-independent. - - ``backend`` selects the DSH agent backend through ``DSH_AGENT_BACKEND`` and - is restored afterwards. - """ out, err = io.StringIO(), io.StringIO() old_stdin = sys.stdin old_backend = os.environ.get("DSH_AGENT_BACKEND") @@ -115,18 +105,12 @@ def run_wrapper(module, argv, prompt, backend="sdk"): os.environ["DSH_AGENT_BACKEND"] = backend elif old_backend is not None: os.environ.pop("DSH_AGENT_BACKEND", None) - saved_key = os.environ.get("TENCENT_API_KEY") - os.environ["TENCENT_API_KEY"] = saved_key or "ci-test-key" sys.stdin = FakeStdin(prompt) try: with redirect_stdout(out), redirect_stderr(err): code = module.main(argv) finally: sys.stdin = old_stdin - if saved_key is None: - os.environ.pop("TENCENT_API_KEY", None) - else: - os.environ["TENCENT_API_KEY"] = saved_key if old_backend is not None: os.environ["DSH_AGENT_BACKEND"] = old_backend else: diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/01-environment.md b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/01-environment.md new file mode 100644 index 00000000..7e14a43b --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/01-environment.md @@ -0,0 +1,123 @@ +# 01 · 环境类报错 + +这一层的报错**与候选内核无关**：它们说明这台机器/这个进程树现在没法起 GPU 栈。 +判断口诀：**换一个 shell、换一张卡、换一个时间点，同一个命令会不会好**—— +会好 → 环境类（本节）；不会 → 看 `04-kernel-quality.md`。 + +--- + +## ENV-SHM-SANDBOX + +**无法写 `/dev/shm`（文件沙箱）** ｜ 状态：**已修（2026-09-22）** ｜ 真出现：快照内 5 次 + 2026-09-20~22 的 10 次（现场已被 purge，原文见 `README.md` 标本） + +### 原文长什么样 + +``` +... --probe failed (3): shm_open: Permission denied +UserWarning: Can't initialize amdsmi - Error code: 32 +``` + +同族变体（同一个根因，不同 errno/不同阶段）： + +| 变体原文 | 说明 | +|---|---| +| `shm_open: Permission denied` | 建/写 `/dev/shm/rocm_smi_card*` 被拒（最常见） | +| `shm_open: Invalid argument` | 早前"清理残留 shm"把活跃会合文件删掉后，名字对不上 | +| `fchmod: Bad file descriptor` | 同上：进程手里是已被 unlink 的死 inode | +| `No HIP GPUs are available` | 栈起不来之后的 torch 侧表现（见 `ENV-NO-DEVICE`） | +| `open hydm_... : Permission denied` | 同一沙箱对其它设备节点的拒绝 | + +### 判定（30 秒） + +```bash +python3 - <<'EOF' +import tempfile, os +try: + fd, p = tempfile.mkstemp(prefix="chk-", dir="/dev/shm"); os.close(fd); os.unlink(p) + print("unconfined: /dev/shm writable") +except OSError as e: + print("CONFINED:", e.errno, e.strerror) +EOF +``` +若这里报 `EACCES`，而 `--probe` 也随之失败 → 就是本类。另可看诊断文件 +`/gpu_probe_failure.json` 的 `dev_shm_write`（`{"ok": false, "errno": 13, "denied": true}`）。 + +### 根因 + +ROCm/amdsmi 起 GPU 栈靠 `/dev/shm` 里的会合文件（`rocm_smi_card*` 等）。 +进程的文件沙箱（Landlock/bwrap 风格：只允许写自己的 workspace 与 `/tmp`）把 `/dev/shm` 变成**只读**， +`shm_open(O_CREAT)` 与"打开已有段写模式"都会 `EACCES`，于是设备永远出不来。 +关键是**继承性**：WebUI（`serve.py`）是长命父进程，Landlock 域被子进程继承且**子进程无法解除**， +所以"启动 WebUI 的那个 shell"决定了后面每一个任务。 + +2026-09-22 那次就是这么来的：WebUI 是 09-21 15:31 在一个 `workspace-write` 会话里 +`nohup` 重启的（见 `nodes/worker29/.metainfer/tasks/` 之外的会话记录），此后每个 WebUI 任务必死。 +此前 08-20/08-21/08-23/09-09 的同族报错被误判为"容器 GPU 栈偶发抖动"—— +对照实验当时是在**无沙箱**会话里做的（那个会话 09-20 21:49 起就是 `danger-full-access`）， +所以"手动跑永远成功"。 + +### 处置 + +1. **从无文件沙箱的 shell 启动 WebUI / orchestrator**（本机：`/root/zth_agent/start_webui_8765.sh`）。 + DKAO 内部**没有**修复办法：受限进程无法自行解除沙箱。 +2. 已经在跑的服务要**重启**才会脱离沙箱（子进程继承，改不了）。 +3. 现在有**三道护栏**，坏环境不会再伪装成"容器坏了"： + - 启动脚本在沙箱里**直接拒启**（`[start_webui] FATAL: this shell cannot write /dev/shm`）； + - 服务启动时 `metainfer/server/envcheck.py` 打醒目告警，spawn 任务时把原因写进该任务 `orchestrator.log`； + - `orchestrator/gpu_probe.py` 在第一次尝试前自检，命中时**不再空重试**，报错直接说 + *"restart it outside that sandbox"*，诊断里带 `dev_shm_write`。 + +### 相关代码 / 测试 + +`orchestrator/gpu_probe.py`（`shm_write_check` / `_shm_denied` / `run_gpu_probe`）、 +`metainfer/server/envcheck.py`、`metainfer/server/app.py`、`metainfer/server/launcher.py`、 +`tests/test_gpu_probe_resilience.py`、`metainfer/server/tests/test_envcheck.py`。 + +--- + +## ENV-NO-DEVICE + +**`No HIP GPUs are available` / `visible_devices: 0`** ｜ 状态：设计行为 ｜ 真出现：1（另见上类变体） + +### 判定 / 根因（两种） + +1. **栈没起来**：真正的病因在上面几行（`shm_open` / amdsmi）→ 按 `ENV-SHM-SANDBOX` 处理。 +2. **可见设备被双重过滤**：同一个**非零** index 同时设了 `HIP_VISIBLE_DEVICES` 与 `ROCR_VISIBLE_DEVICES`， + 会被过滤两次（device 1 消失，device 0 侥幸存活）。修法是**只用 `HIP_VISIBLE_DEVICES`** + （`bind_worker_gpu`），别再加 `ROCR_*`。 + +### 处置 + +先看 `shm` 症状；没有则检查环境变量是否重复绑定，再确认卡是否真的透传进容器 +（`hy-smi` / `/dev/kfd` / `/dev/dri/renderD*`）。 + +--- + +## ENV-PROBE-WRAPPER + +**`GPU probe failed after N attempt(s) (RuntimeError): …`** ｜ 状态：已修（2026-09-22） + +### 判定 + +这是 DKAO **自己的重试包装**，不是病因。**不要据此判断环境好坏**——真正的病因在冒号后面那一行 +（`shm_open` / traceback / 模块缺失…）。修复前它还会把"权限类确定性失败"重试 3 次（白等 ~25s）； +现在 `gpu_probe.py` 只在"非沙箱类"的瞬时症状上重试，命中沙箱时直接失败并把 +`attempts_performed` 记成 1（诊断里同时保留 `attempts` 预算值，便于区分"没试"和"试满"）。 + +--- + +## ENV-NO-LEASE + +**`no GPU lease became available after N checks` / `no device passed the admission gate`** ｜ 状态：环境 + +### 判定 / 根因 + +`gpu_mode=scheduled`（或 `execution_profile=production`，它强制走 broker）时，卡来自共享 broker 队列： +只发"**过闸门且没人租**"的卡（`gpu_broker.py`，优先级 `PRIORITY_PRODUCTION` 高于 harness）。 +默认 30s 一轮、最多 48 轮（`gpu_lease_wait_seconds` / `gpu_lease_max_waits`），拿不到就报本错； +生产模式措辞会明确写"共享卡会给出过期数字，所以不跳过闸门"。 + +### 处置 + +等卡空；或改用 `gpu_mode=occupy`（钉住自己的卡，不看队列）；或让占用方让出。 +默认模式的 DKAO（`occupy`）**不会**报这个错——它直接钉卡，靠测量闸门（见 `03`）兜底。 diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/02-generate-and-agent.md b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/02-generate-and-agent.md new file mode 100644 index 00000000..7b913533 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/02-generate-and-agent.md @@ -0,0 +1,132 @@ +# 02 · 生成阶段与 agent 类报错 + +发生在 `generate_kernel_repo` / worker bootstrap：**内核还没开始优化**，报错来自 agent 进程、 +脚手架校验、或工作区归属检查。 + +--- + +## GEN-CONFLICT + +**`workspace main already points to , not requested `** ｜ 状态：**已修** ｜ 真出现 **64** 次（09-11 ~ 09-15） + +### 原文 + +``` +workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab, +not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-...-it002-candidate-hy3-tp8-shared-down-proj-m16- +``` + +### 根因 + +harness-evolve 的"环境失败重试"会给同一道题**换一个新的 kernel repo 名**，却**复用同一个子任务目录** +`children/iteration_NNN/<题号>/workspace`。DKAO 会把该工作区的 `main` 符号链接**绑死**到本次要用的 repo +（`_prepare_worktrees` 的所有权校验），于是第二次尝试一进去就失败——本来 3 秒就能自愈的环境抖动， +被放大成"必然失败 3 次"，64 条记录里绝大多数是这一条。 + +### 处置 / 修法 + +让**每个评估尝试拥有独立目录**（结构上杜绝复用）：`children/iteration_001`、环境失败后的 +`iteration_001_a2`、`_a3`、泛化重考的 `iteration_00N_generalization`； +`rounds.py` 新增 `stage_dir_name` / `attempt_dirs` / `resolve_child_dir`（最新尝试获胜）。 +任何两个 DKAO run 不再共用目录，失败那次的日志原样留在旁边当证据。 +**再看到这条**：说明有新的调用方在复用目录，按"尝试目录必须唯一"去查。 + +--- + +## GEN-SCHEMA + +**表单/契约参数非法** ｜ 状态：设计行为 + +| 原文 | 说明 | +|---|---| +| `tp_size must be 4 or 8, got 1` | 前端形状编辑器曾为每个模型提供 `tp_size: 1` 的拓扑，而算子契约只认 4/8；选到 TP1 必死（`int8_w8a8_gemm_api.py`）。前端已修 | +| `gpu_mode must be occupy or scheduled` | 表单/接口传了未知模式 | +| `execution_profile must be 'default' or 'production'` | 同上 | + +这条报错是**契约在拦人**，不是环境问题：按提示改成合法值即可。 + +--- + +## GEN-CONTRACT + +**脚手架/参考不可信** ｜ 状态：设计行为 + +| 原文 | 根因 | +|---|---| +| `staged API contract digest is not trusted` | 落地的 `int8_w8a8_gemm_api.py` 与受信摘要不一致（被改动或拷贝不完整） | +| `trusted PyTorch reference self-test failed` | `w8a8_bench.py --self-test` 没过：参考实现本身不可信，拒绝在其上做测量 | +| `Generate scaffold unexpectedly contains a HIP implementation` | "新建 repo"模式要求脚手架里没有任何现成 HIP 实现（避免继承旧代码） | + +处置：不要绕过。前者查契约文件来源/摘要；后者删掉仓库里多出来的实现文件或改用"接入已有 repo"模式。 + +--- + +## GEN-OWNERSHIP + +**`bootstrap Agent changed files outside its HIP ownership: [...]`** ｜ 状态：设计行为 ｜ 真出现 4 次（07-28） + +agent 只被允许改 `_GENERATED_KERNEL_FILE`，改动清单里出现别的路径就会被拒绝： + +``` +bootstrap Agent changed files outside its HIP ownership: +['__pycache__/w8a8_backend.cpython-310.pyc', 'csrc/bindings_hip.cpp'] +``` + +`__pycache__/*.pyc` 是**最常见的误报来源**（agent 只是 import 了一下）。处置：清理缓存产物 / +确认 `PYTHONDONTWRITEBYTECODE`，真改了别人的文件则必须回退。 + +--- + +## GEN-BINARY-MISSING + +**agent 可执行文件/模块不存在** ｜ 状态：环境 ｜ 真出现 2 次（08-03） + +``` +[Errno 2] No such file or directory: 'ccb' +→ worker_3 could not generate a valid initial HIP kernel after 3 attempts +``` + +根因：宿主没有 agent 二进制（`METAINFER_CLAUDE_BIN` / PATH 缺失），三个 bootstrap 尝试全在同一处失败。 +处置：确认 `METAINFER_CLAUDE_BIN`（本机 `/usr/local/bin/ccb`）存在且可执行； +本类报错**必须**先修环境再重跑，否则只是重复失败。 + +--- + +## GEN-AGENT-EJECT + +**`Generate agent attempt N failed: nonzero exit 1`** ｜ 状态：环境 ｜ 真出现 **543** 次（最多的一类） + +### 判定 + +agent 进程非零退出（不是 DKAO 的判断）。同一任务里往往连出 attempt 1/2/3，然后外层包装成 +`GEN-COORD-FAILED`。**先看 agent 自己的日志**：`/main/.metainfer-generate-logs/*.attemptN.log` +（或 worker 的 `workers//logs/*.log`），错误原因在那里。 + +### 常见根因 + +agent 侧超时/OOM 被杀、模型/凭据不可用、prompt 里引用的文件不存在、agent CLI 版本不匹配。 +处置：按 agent 日志修环境或输入，再重跑；不要盲目加大重试次数。 + +--- + +## GEN-NO-KERNEL + +**agent 没能产出可用内核** ｜ 状态：设计行为 ｜ 真出现 18 次 + +| 原文 | 说明 | +|---|---| +| `bootstrap Agent did not generate ` | agent 没写目标文件 | +| `agent did not write proposal.json` | 缺少提案文件（迭代轮） | +| `worker_N could not generate a valid initial HIP kernel after 3 attempts` | 3 次尝试后仍无可用内核 → 该 lane 判失败（`PROC-NO-WORKER` 的前置） | + +处置：看 agent 日志与最后一次 `bootstrap_progress.json`；这是"生成质量"问题，通常伴随 `GEN-AGENT-EJECT`。 + +--- + +## GEN-COORD-FAILED + +**`Kernel coordination failed after 3 attempts. Last failure: …`** ｜ 状态：环境 + +`generate_kernel_repo` 阶段的主 agent 重试耗尽（`_MAX_GENERATE_RETRIES`）。这是**外层包装**： +真正的理由在 `Last failure:` 后面（通常是 `GEN-AGENT-EJECT` 或契约/归属类）。 +任务会以 `stopped / infra_fail` 结束在生成阶段——**不要**据此说"GPU 坏了"。 diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/03-measurement-and-gate.md b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/03-measurement-and-gate.md new file mode 100644 index 00000000..db96e50f --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/03-measurement-and-gate.md @@ -0,0 +1,175 @@ +# 03 · 测量与闸门类 + +DKAO 的核心纪律：**卡不干净就不给数字**。所以这一层"报错"很多时候是**故意的拒绝**， +先分清"拒绝测量"（设计行为）和"工具/环境坏了"（环境类）。 + +闸门（measurement gate）判据只有一条：**VRAM ≤ 90% 且 HCU == 0**（`gpu_preflight.py`）， +检查时机：每次**计时 benchmark 之前**、每次 **PMC profile 之前**（检查自己那张卡）， +以及生成阶段的 probe（`ensure_any_measurement_gate`：自己卡优先，否则任意空闲卡）。 +每查一次往 `/measurement_gate.jsonl` 写一行 `gate_ok` / `gate_blocked` +（含 gpu / site / attempt / util% / VRAM% / reasons）。 + +--- + +## ENV-GATE-TIMEOUT + +**闸门等待 / 等超时** ｜ 状态：设计行为 ｜ 真出现 **84** 次（19 个 DKAO 子任务） + +两种原文，都是同一件事： + +``` +# ① 等待中的审计行（/measurement_gate.jsonl） +{"event": "gate_blocked", "site": "benchmark", "gpu": 0, "attempt": 4, "max_waits": 48, + "wait_seconds": 1800, "usable": false, "vram_percent": 34.0, "util_percent": 100.0, + "reasons": ["device busy: HCU 100.0% > 0.0%"]} + +# ② 等满 24h 后的最终形态 +GPU 0 did not pass the measurement gate (VRAM <=90% and HCU ==0) after 48 checks: … +``` + +**真实样例**：2026-09-17 `test-9-17-1-07bb5cd8` 的 `hy3_tp8_o_proj_m4096` 子任务，17:57 / 18:27 / +20:47 / 21:17 / 23:47 … 每 30 分钟一条 `gate_blocked`，`gpu0` 的 HCU 一直是 100%—— +**这就是"卡被占住时 DKAO 的样子"**：不测、不报错、每半小时重查一次，日志里留痕。 + +### 语义（看清这几条，基本不会误判） + +| 项 | 值 / 行为 | +|---|---| +| 重查间隔 | 30 min（`METAINFER_GATE_WAIT_SECONDS`，默认 1800s） | +| 上限 | 48 次 ≈ **24 小时/每次调用**（`METAINFER_GATE_MAX_WAITS`） | +| 等待期间 | **不测**（宁可不给数，也不给被污染的数），只写 `gate_blocked` 审计行 | +| 超时后 | 抛 `MeasurementGateBlocked` → 该轮记 `worker_iteration_failed`（"record failed round, restore best commit, and continue"） | +| lane 之间 | **不互相阻塞**：别人占的卡在等，其它卡的 lane 照常优化；但整个 phase 要等所有 lane 收尾 | +| 判定为基础设施故障吗 | **不算**。`is_infrastructure_failure()` 只认 `timeout/timed out/killed/no result/exit 143`，闸门阻塞不计入那 5 次 infra 恢复额度 | +| 预算粒度 | **每次 gate 调用独立 24h**（bootstrap 每卡最多 3 次尝试 → 卡一直没空时可能挂很久） | + +### 处置 + +1. 先确认卡是不是真的被占：`hy-smi` / `foreign_kfd_pids()` / `gpu_preflight.sample_gpu_state` + （本机 2026-09-22 13:26 的实测：4 张卡 util 100%，GPU2/3 VRAM 87%，外来 KFD 进程 10 个 → 邻居容器在跑）。 +2. 等卡空；或换卡；或把等待调小让它早点报出来（`METAINFER_GATE_WAIT_SECONDS=300 METAINFER_GATE_MAX_WAITS=6`）。 +3. 彻底关闸门：`METAINFER_GPU_PREFLIGHT=0`（**卡忙也照测，数字会被污染，别用于生产/验收**）。 + +--- + +## MEAS-CMD-FAILED + +**基准/probe 子进程非零退出** ｜ 状态：环境 ｜ 真出现 **53** 次 + +``` +python3 .../w8a8_bench.py --source ... --m 1 --n 1 --k 1 --prepare-reference failed (1): shm_open: Permission denied +Command '['python3', '.../w8a8_bench.py', ...]' returned non-zero exit status 1 +``` + +判定顺序：**先在同一行后面找真病因**（`shm_open` → `ENV-SHM-SANDBOX`；traceback → 编译/接口错误； +`--prepare-reference` → 见下条）。处置：修病因后重跑；这类失败会被记为基础设施类并自动补一轮。 + +--- + +## MEAS-REF-PREPARE + +**参考缓存准备失败** ｜ 状态：环境 + +`--prepare-reference` 用来生成 int64 精确参考（正确性判据）。失败原因通常是： +磁盘/缓存目录不可写、`/dev/shm` 受限（同 `ENV-SHM-SANDBOX`）、显存不足、或参考协议版本不一致。 +处置：确认 `/cache/references/` 可写、卡上无并发占用；参考缓存可复用（同名 shape 不必重算）。 + +--- + +## MEAS-TIMEOUT + +**基准超时** ｜ 状态：环境 + +单次 benchmark 上限 `_BENCHMARK_TIMEOUT_S = 1200`（20 分钟）。卡被外部作业拖慢、内核死循环/ +挂死（`Graph` 捕获卡住）都会命中。处置：看是不是被占（闸门通过≠全程独占），或内核本身挂死 +（后者按 `04` 处理，属于候选代码问题）。 + +--- + +## MEAS-PMC-UNAVAILABLE + +**`worker_pmc_skipped … available: false`** ｜ 状态：设计行为 ｜ 真出现 **210** 次（高频，但**不是失败**） + +hipprof/PMC 证据拿不到时（权限、驱动、卡不支持某组 counter、profiler 冲突），这一轮**跳过 PMC 采集继续跑**： +调优少了 ISA 级依据，正确性与性能结论不受影响。只有把它错当成"报错"才会误导判断—— +它**不会**导致任务停止。处置：无需处理；若长期全任务都缺 PMC，检查 `hipprof` 可用性 +（`hipprof_executable` 在 `generation_preflight.json` 里有记录）。 + +--- + +## 测量中"被抢"（污染处置） + +设计上：父侧把被抢的时间窗写进 `/contention.json`，子侧收尾时 +`discard_if_contaminated()` 会**丢掉这一轮的数、隔离到 `contaminated/`、并把源码 git 回滚到 +上一次被接受的 commit**，然后重测（timeline `worker_round_discarded_contended`）。 + +**现状（2026-09-22 核查）**：仓库里**只有读方、没有写方**——harness-evolve 明确删掉了自己的 +contention watch（`adapters/eval.py`：*"a contention window: gone entirely … DKAO's gate and +contamination handling do"*）。也就是说这条"事后作废"路径目前**休眠**，真正生效的是： +测前闸门 + `measurement_suspect` 可疑标记（功率异常/外来 KFD pid → 只报告并排在 clean 卡后面） ++ 最终串行验收重测。看到 `worker_round_discarded_contended` 说明**有人**在写这个文件，需要确认是谁。 + +### 最终串行验收**自己也**会落在污染窗口里（2026-10-11 实例） + +**现象**：第 5 步报 + +``` +performance regressed for : final X us vs worker best Y us + (limit ... = best + ...) after 3 re-measures +``` + +而且 X **非常稳**（`latency_samples_us` 紧到 ±0.1%），看起来像"结构性变慢"。 + +**实例**（`glm-tp8-m8-16-4096-2663a78f` 的恢复运行，卡 0）：17:00–17:31 这个窗口里**两个 shape 同时偏慢** + +| shape | worker best | 窗口内实测 | 窗口后同一产物重测 | +|---|---|---|---| +| `glm_tp8_fused_qkv_a_proj_m4096` | 1191.7 | 2721.7 / 2720.8 / 2720.8 / 2720.8 | **1191.8 / 1188.8**（卡 0）、1214.0（卡 5）、1193.9（卡 3） | +| `glm_tp8_shared_down_proj_m16` | 10.23 | 13.65 / 13.60 | — | + +重测命令（合并产物原样，不重编、不改 kernel）： + +```bash +cd /final/layout_batches/batch1 # 或 final/source +HIP_VISIBLE_DEVICES=0 python3 w8a8_bench.py --source . \ + --m 4096 --n 2624 --k 6144 \ + --reference-cache-dir /final/cache/references \ + --warmups 30 --samples 10 --replays-per-sample 50 +``` + +**为什么重测救不回来**：`_PERF_GATE_RETRY_INTERVAL_S = 300`（3 次重测共 ~15 分钟）， +一个 ~30 分钟的共租户窗口把**每一次**重测都罩住了。窗口越界 → 4 次测量全废 → 任务挂在第 5 步。 + +**判据（真回归 vs 污染）**：同一棵合并产物、同一张卡、稍后重测一次。回到 ≈ worker best = 污染； +仍然慢 = 真回归，再去查路由/pack/workspace。 + +### 污染是**按卡**的，而 `hy-smi` 的 0% 看不出来（2026-10-11 实测） + +同一分钟、同一棵产物、同一个 µs 级 shape（`glm_tp8_shared_down_proj_m8`，门限 11.556 µs）： + +| 卡 | median | 结论 | +|---|---|---| +| 1 | 9.507 / 9.779 | 干净 | +| 3 | 9.672（p90 9.696，极紧） | 干净 | +| 2 | 16.544 | 被污染 | +| 5 | 12.771 / 10.040 | 时好时坏 | +| 0 | 13.040（同批另一个 shape 20.416） | 被污染 | + +而这几张卡在 `hy-smi` 上都显示 **DCU% = 0**（采样瞬间看不到别人**短 kernel 突发**； +1.2 ms 的大 kernel 对同一种污染几乎不敏感 —— 同一张卡 0 测 M=4096 仍是 1188.8 µs = best）。 + +**所以"等 hy-smi 全 0% 再跑"是不够的。**正确的前置检查是**先量一个已知 shape**： + +```bash +# 用已知 worker best 的 µs 级 shape 在候选卡上打一发，比 best 高 >5% 就换卡 +HIP_VISIBLE_DEVICES= python3 /w8a8_bench.py --source \ + --m 8 --n 6144 --k 256 --reference-cache-dir /final/cache/references \ + --warmups 30 --samples 30 --replays-per-sample 50 +``` + +`recover_serial_validate` 的默认卡本来就是 **3**（`METAINFER_SERIAL_VALIDATE_GPU` 的 +`setdefault`，注释写的"default to a non-zero card"）—— 恢复时**不要**随手覆盖成 0。 + +**处置**：等 `hy-smi` 全部 0% 再跑恢复；在污染窗口内得出的性能结论一律标可疑。 +（本机是**风冷**卡、显存地址散列没开 → 共租户的"跳行读"会让带宽直接腰斩，比液冷机敏感得多， +见 `wiki-metainfer/machines.md` 与 `gotchas/air-machine-no-address-hashing.md`。） diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/04-kernel-quality.md b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/04-kernel-quality.md new file mode 100644 index 00000000..90290150 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/04-kernel-quality.md @@ -0,0 +1,323 @@ +# 04 · 内核质量类（这些是 DKAO 的日常工作，不是故障） + +看到这里的报错**不要**去查机器、也不要怪沙箱：它们说的是"agent 这一版内核不行"。 +管线本来就靠重试/回退处理它们，处理不掉才升级成 `PROC-NO-WORKER` 之类的进程级失败。 + +--- + +### KQ-PACK-LAYOUT-CONFLICT / 同一个权重的多个 M 用了**不同 packed 布局** + +> 本类由 `collect_errors.py` 独立识别（`final_layout_conflict` 事件 + `packed layout` 文案）， +> **不要**当成 `KQ-CORRECTNESS`：它不是"某个内核算错了"，而是"合并把两个 M 的 pack 混用了"。 +> 2026-10-08 之前的两次事故都混在 `KQ-CORRECTNESS` 里，所以第一次修完就没人回头看。 + +**为什么会这样**：合并产物里 `launch_pack_w8a8_weight` 每个权重 `(N,K)` 只能有一条路由 +（它的 ABI 里**没有 m**），所以族内多个 M 变体必须共用一份 packed 布局。 +**而这是冻结 API 的硬要求**：`api/int8w8a8gemm/int8_w8a8_gemm_api.py::prepare_weight(raw_weight, weight_scale)` +不接 M —— 真实模型里**一个权重只 pack 一次，然后被所有 M 复用**。 +worker 侧契约又明确允许改布局（`prompts.py:600`："may change only its HIP implementation +**and the matching GEMM interpretation** to test packed layouts"），于是同一个 worker 在 +不同 M 上换布局是可能的。 + +**指纹现象**：`mismatch_count` 接近满量程（99.9% 元素错）+ `max_abs_error` 只有几十~ +几百（不是爆量级），`median_us` 又正好是**该 shape 自己 kernel 的正常耗时**。 +→ 三条同时成立 = 布局/解释不一致，不是精度问题、不是跑错 kernel、更不是卡的问题。 + +**三次实例** + +| 时间 | 任务 | 形态 | +|---|---|---| +| 2026-10-07 | `hy3-tp8-m8-16-4096-2-9dcfce15` | o_proj 的 m8/m16 共用一种布局，m4096 换了另一种；**三个 pack 源码逐字节相同**，静态比对抓不到 | +| 2026-10-08 | `minimax-tp8-m8-16-4096-df6fb501` | qkv_proj 的 m8 pack **不带 K-octet swizzle**，m4096 带；两份 pack 源码正文不同，实测 7,343,908 字节不同（首个差异字节 offset 8） | +| 2026-10-11 | `glm-tp8-m8-16-4096-2663a78f` | **门装了但没跑起来**：探针子进程用空 `sys.executable` 起不来 → `probed=false` → 退回单批 → 6 个族里 **5 个冲突**，挂在 `fused_qkv_a_proj_m4096`（mismatch 10738189/10774404 = 99.9%，max_abs 252.5，median 1192 µs = 该 shape 正常耗时）。根因与修法见下节「2026-10-11」。 | + +df6fb501 的具体证据（本机实测，可复现）： + +```bash +W=nodes/worker29/workspaces/minimax-tp8-m8-16-4096-df6fb501 +grep -n 'launch_pack_w8a8_weight' -A 6 $W/final/cache/compile_sources/*/csrc/w8a8_dispatch.cpp +# → 每个 (n,k) 一条 pack 路由，取族内最小 M = mi_..._qkv_proj_m8_launch_pack_w8a8_weight +# 把两个 prebuilt 对象各自链成 .so，用 ctypes 直接调 pack，比 packed_weight 字节： +# pw_m8 != pw_m4096（7,343,908 字节不同）；weight_scale 相同 +# 同一个 m4096 GEMM，只换 pack 来源： +# 喂 m8 pack → mismatch_count = 5237886（复现 final 的失败） +# 喂 m4096 pack → mismatch_count = 0 +# 对照：最终产物的 m8 路径用同一份 m8 pack → mismatch_count = 0（自洽） +``` + +**处置（2026-10-08/09 落地：`final_layout_gate`）** + +合并前对每个 `(N,K)` 族跑一次**探针**（一个子进程覆盖所有族，链上每个成员的 prebuilt 对象）： +拿**每个成员自己的 pack 内核**，用一份"字节计数图"权重、**按本族真实 (N,K)** 打一次包， +对打包结果取 SHA256 作为该成员的**布局指纹**。两条关键结论： + +- **必须行为式比对**：hy3 那次三份 pack 源码**逐字节相同**却布局不同，比对源码/符号名完全抓不到。 +- **必须按真实 (N,K)**：用小形状探针时 df6fb501 的两个 pack 内核**输出完全相同**（都写了同一块 32 KB）， + 只有到真实 `(1280, 6144)` 才分叉 —— swizzle 只在完整 k32 网格上生效。 + 实测成本：7.9 MB 镜像 **0.9 ms**、235 MB 镜像 8.3 ms，且不在任何计时区间内。 + 整个任务的 5 个族一次探针 **18.6 s**（含子进程的 torch/HIP 初始化，只做一次）。 + +判读：**指纹相同 = 字节完全相同**，所以这些成员互相可换、可以共用一条 pack 路由； +指纹不同一律按**不兼容**处理（合并无法证明对方读得懂，而事故正是"只认自己那份镜像的读者"）。 +这是偏保守的方向：只有真的产出了不同字节，才会多付一批。 + +分批（`family_groups` / `pack_batches`）：按指纹分组，组内自洽；再把各族的组打进**最少**的批， +每批内每个 `(N,K)` 只有一组 → 一条 pack 路由。批 0 直接渲染进 `final/source`（发布的那棵树 +必须是被验收过的那批），其余批在 `final/layout_batches/batchN` 各自带 dispatch + prebuilt 编译。 + +探针不可用（对象缺失 / 子进程失败 / 无 GPU）→ **仍不判冲突**（`LayoutCheck.ok` 保持 true， +绝不因为探测失败废掉任务），但**不再当成"没有冲突"**：一致性只是**未被证明**， +多成员族按 `unproven_groups` **逐个成员拆批**，并发 `final_layout_probe_failed` 事件 +（2026-10-11 起；旧行为是静默退回单批，正是下面这次事故）。探针可跑且指纹全同 → 仍是单批，零额外代价。 + +**2026-10-11：门对 WebUI 起的任务全程失效（第三个实例的根因）** + +`final_layout_gate._probe_families` 用 `subprocess.run([sys.executable, "-c", ...])` 起探针； +而 `server/launcher.py` 起 orchestrator 时把 `argv[0]` 写成友好名 `metainfer-orchestrator` +（真实二进制走 `Popen(executable=...)`）。CPython 从 `argv[0]` 推 `sys.executable`，推不出来时是**空串**： + +```bash +python3 -c "import subprocess,sys; print(repr(subprocess.run( + ['metainfer-orchestrator','-c','import sys;print(repr(sys.executable))'], + executable=sys.executable,capture_output=True,text=True).stdout))" +# → "''" ← subprocess.run(['']) 抛 PermissionError: [Errno 13] Permission denied: '' +``` + +对照（`workspaces/*` + `.metainfer/tasks/*/timeline.jsonl`）：`probed=true` 只出现在 +`recovered_serial_validate=true`（从 shell 手工恢复驱动）的任务上；唯一一个 WebUI 起、又跑到门这一步 +的任务就是 `glm-…-2663a78f`，`probed=false`。**也就是说这道门 2026-10-08 上线后， +对 WebUI 起的任务一次都没生效过。** + +修法：探针子进程统一走 `metainfer.orchestrator.interpreter.python_executable()` +（`METAINFER_PYTHON` → 可用的 `sys.executable` → PATH 上的 `python3`）， +`server/launcher.py` 与 `harness_evolve/.../eval.py` 的两份同类函数也接到同一个实现； +探针失败信息里带上解释器路径；回归测试见 +`tests/test_final_layout_gate.py::test_probe_spawns_a_resolvable_interpreter_without_sys_executable` +与 `metainfer/orchestrator/tests/test_interpreter.py`。 + +**2026-10-11 第二个坑：分批以后，第 N 批其实是拿第 0 批的扩展在测** + +自动分批上线后第一次真正跑起来（同一个 `glm-…-2663a78f` 的恢复）立刻暴露： +`W8A8Runner` 在合并里**只建一次**（`runner = W8A8Runner(final, serial_gpu)` → +`self.source = final/source`），而 `_validate_layout_batch` 只把**第 N 批的树**渲染到 +`final/layout_batches/batchN`，benchmark 仍走 `runner.benchmark(...)` → +**第 N 批的 shape 是被第 0 批的 dispatch 服务的**。 + +指纹（可静态复现，不用跑 GPU）： + +```bash +# 两批各自的 build key（= sha256(bindings.cpp + dispatch + prebuilt/*.o)） +batch0 = final/source prebuilt=13 key 6fd5852f… +batch1 = final/layout_batches/batch1 prebuilt=5 key 5f8f2d65… +ls final/cache/compile_sources/ # → 只有 18cee70c…(旧) 与 6fd5852f…(batch0) + # batch1 的 key 根本不存在 = 它从未被编译/加载 +``` + +后果：batch 1 的 `fused_qkv_a_proj_m4096` 走 batch 0 的 M 链退化成 **m16 kernel** +（数值正确、慢 2.28×：2721.7 µs；同形状的真产物在卡 0/3/5 实测 1188.8 / 1193.9 / 1214.0 µs， +与 worker best 1191.7 一致）→ 性能门把"从没被测量过的产物"判成回退，恢复两次都挂在这上面。 + +修法：每批用**自己的树**建 runner —— `W8A8Runner(..., source=batch_source)` +（`worker_root` 继续持有 references/torch/logs 缓存，所以 build key 仍按批内容寻址）； +回归测试 `tests/test_gen_pipeline.py::test_runner_can_point_at_a_layout_batch_tree` +与 `tests/test_final_layout_gate.py::test_each_batch_benchmarks_its_own_tree`。 + +**教训**：手动按 M 恢复（`--m-values 8/16/4096` 三次独立调用）每次都是**单批**， +所以这条路一直没暴露这个 bug；**自动分批是第一个把 batch_count>1 真正跑起来的路径**。 + +**DF6FB501 实测（2026-10-09，卡 3）** + +``` +final_layout_conflict: + conflicts: 1280x6144 (['minimax_tp8_qkv_proj_m8']; ['minimax_tp8_qkv_proj_m4096']), + 1536x6144 (['..._qkv_proj_and_indexer_qk_m8']; ['..._qkv_proj_and_indexer_qk_m4096']), + 6144x384 (['..._shared_down_proj_m8']; ['..._shared_down_proj_m4096']) +``` + +即**三个族真的分叉**；`shared_gate_up_proj`（768x6144）与 `o_proj`（6144x1024）的 m8/m4096 +指纹**完全相同**，仍留在同一批 —— gate 不会"见到多 M 就无脑拆"。 +分派据此拆成 7 + 3 两批（批 0 含两个 M 的 o_proj / gate_up 与全部 m8）。 + +同任务手工两批恢复（`--m-values 8`，5 shape）实测：**`status=success`，5/5 过性能门** +（`final_target.all_shapes_met = True`），correctness `mismatch_count=0`，median 与 worker best 同量级： +o_proj 12.91 / qkv_and_indexer 34.03 / qkv 16.79 / shared_down 10.79 / shared_gate_up 15.44（µs）。 + +**仍然要靠流程避开的**：一个任务里同一算子放多个 M 仍有"多一份打包"的代价； +**根治（未做）**是让 worker 遵守"同一权重一份 packed 布局"（布局升级为任务级参数 + +族级契约测试）。参见 `docs/dkao_suite_protocol_v2.md` 与下一条目。 + +**旧的处置（`recover_serial_validate --m-values`，仍然可用）** + +```bash +python3 -m metainfer.tasks.dcu_kernel_auto_opt.orchestrator.recover_serial_validate \ + --state-dir --workspace-dir --m-values 8 +``` + +`--m-values` / `--shape-ids` 只把该批次的 accepted artifact 合并进去，**不重跑任何优化轮次**。 +注意：`--state-dir` 要指向**任务自己的目录**（`.../.metainfer/tasks/`，里面才有 `run.json`）， +且 `METAINFER_SERIAL_VALIDATE_GPU` 必须是该任务**已分配的 worker 卡**之一（见下）。 + +- **实测**（任务 `hy3-tp8-m8-16-4096-2-9dcfce15`，12 shape，卡 2，三批各 4 shape）： + **三次全部 `status=success`，12/12 过性能门**，合并后 median 与 worker best 一致： + + | 批次 | shape | worker best | 合并后 | + |---|---|---|---| + | M=8 | o_proj / qkv_proj / shared_down / shared_gate_up | 12.27 / 13.43 / 9.41 / 12.02 | 12.60 / 13.81 / 9.45 / 11.89 | + | M=16 | 同上 | 12.22 / 14.77 / 10.29 / 12.90 | 13.35 / 14.93 / 10.19 / 12.98 | + | M=4096 | 同上 | 397.3 / 410.7 / 208.9 / 198.2 | **397.2** / 412.0 / 207.9 / 198.4 | + + 对照：整任务一次合并时，同样的 artifact 在 `o_proj_m4096` 上要么跑错 kernel（1026 µs）、要么正确性失败（99.9% 元素错）。 +- **两条踩过的坑**： + 1. `--state-dir` 给成 `.metainfer` 根目录 → `FileNotFoundError: no run.json at .../.metainfer/run.json`； + 2. `METAINFER_SERIAL_VALIDATE_GPU` 给成任务没分配过的卡（例如任务只用了 0–3，却给 5） + → `serial validation GPU must be one of the completed worker GPUs [0, 1, 2, 3], got 5`。 + 这是**有意**的约束：串行验收必须回到该任务自己的卡上，不能借别的任务的卡。 + +--- + +### KQ-CORRECTNESS + +**内核算错（正确性/Graph 校验不过）** ｜ 状态：设计行为 ｜ 真出现 9 次 +（2026-09-22 现场 1 次） + +### 两种原文 + +``` +worker_iteration_failed: exact correctness failed: +{"device_name": "K500SM_AI", "first_mismatch": {"abs_error": 13.21875, "actual": 5.78125, "expected": -8.4375, + "flat_index": 0, "m": 0, "n": 0}, "graph_capture_passed": true, "median_us": ...} + +worker_bootstrap_attempt_failed: +child-generated kernel Graph/correctness validation failed for : {…"first_mismatch": {…}…} +``` + +### 判定 + +带 `first_mismatch`（首错位置 + 绝对误差）与 `graph_capture_passed`： +- `passed: false` + `first_mismatch` → 数值不对（归约顺序、标量边界、量化尺度/零点、累加精度…）； +- `graph_capture_passed: false` → Graph 捕获失败（动态 shape/分配、host 同步、stream 用法）。 + +### 现场样例（2026-09-22 13:18，任务 `dcu-kernel-auto-opt-dee7d379`） + +``` +worker_1 bootstrap attempt 1: passed=false, first_mismatch abs_error=10.25 @ m0/n0 +13:21:03 attempt 2: passed=true → worker_bootstrap_success +``` +即 **agent 第一版算错、第二版修好**，全程无需人干预——这是正常回路。 + +### 处置 + +不用处理，看后续尝试即可；只有当某个 shape **反复多次**同一个位置错时，才值得查 +agent 用的参考实现/接口契约（`first_mismatch` 的 `flat_index/m/n` 指出位置）。 + +--- + +## KQ-INLINE-ASM + +**`inline asm acceptance gate failed: …`** ｜ 状态：设计行为 ｜ 真出现 1 次 + +``` +worker_3-...-iter17 failed: …; inline asm acceptance gate failed: +proposal.json is missing isa_optimization; trusted candidate gfx928 ISA audit is … +``` + +内联汇编（DUMMA/MTBUF 等）默认要**先有 ISA 审计证据**才允许用在候选里 +（`evaluate_inline_asm_gate`：`isa_policy.raw_inline_asm_allowed` + 已核验的目标指令）。 +缺 `isa_optimization` 声明或审计不可信 → 该提案被拒。处置：按提示补 ISA 证据，或让 agent 走无内联汇编路径。 + +--- + +## KQ-PERF-REGRESSION + +**性能回退（最终验收不过）** ｜ 状态：设计行为 ｜ 真出现 **72** 次 + +``` +performance regressed for m2_wqkv_a: final 12.34 us vs worker best 11.60 us → final_synthesis_rejected +``` + +worker 侧接受的"最好"在**最终串行验收**（独占卡、串行重测）时没达到门限 +（默认 `worker best + max(5%, 2 µs)` / 无 P90 回退；2 µs 绝对下限只作用于 best < 40 µs 的小 kernel）。 +常见原因：worker 期间的测量受同批 lane/邻居负载影响（整机 4 lane 同时跑本身就是竞争负载）、 +卡间差异（同一 shape 跨卡可差 4%）、环境地板漂移（同一份代码 median 从 884us 漂到 3047us 的实测记录存在）。 +处置：换卡重测（`METAINFER_SERIAL_VALIDATE_GPU=`）、等机器安静后重跑； +**不要**把它当成"内核变差了"。 + +### KQ-PERF-REGRESSION / 第二种根因：最终 dispatch 忽略 M（**控制面 bug，2026-10-07 已修并验证**） + +2026-10-07 实测（任务 `hy3-tp8-m8-16-4096-2-9dcfce15`）： + +``` +performance regressed for hy3_tp8_qkv_proj_m4096: + final 1026.342 us vs worker best 410.660 us (limit 431.193 us = best + 20.533 us) after 3 re-measures +``` + +**这次不是噪声，也不是内核变差**：4 次测量方差 < 0.5%（1026.85 / 1026.34 / 1026.34 / 1026.34， +每次间隔 5 分钟），同一张卡（`serial_gpu` = 第一个完成的 worker 卡 = 卡 2， +正是测出 410.66 µs 的那张）。 + +**根因**：最终合并产物的分派器**只按 `(n, k)` 路由、完全不看 `m`**。 +生成它的 `gen_and_opt_pipeline._render_prebuilt_dispatch()` 用 `seen_pairs: set[(n, k)]` 去重： + +```python +pair = (n, k) +if pair in seen_pairs: # ← 同 (N,K) 的后续 M 变体直接被丢掉 + continue +``` + +于是**同一算子、同 (N,K)、不同 M** 的多个 artifact 里，**只有第一个拿到路由** +（artifact 按 `shape_id` 排序 → `m16` < `m4096` < `m8`，所以是 **m16**）， +其余 M 变体统统落到那个分支。最终生成的 +`final/source/csrc/w8a8_dispatch.cpp` 里就是： + +```c +if (n == 1280 && k == 4096) { // qkv_proj 的 m8/m16/m4096 全部命中 + mi_hy3_tp8_qkv_proj_m16_launch_w8a8_gemm(...); // ← M=4096 也跑 M=16 调优的 kernel +``` + +M=4096 的 prefill 跑 M=16 调优的 kernel：**数值正确（能过正确性门）、慢 2.5×**（1026 vs 410）。 + +**判定特征（怎么和噪声型回退区分）** + +| 特征 | 噪声型（本节上半） | 本 bug | +|---|---|---| +| 回退幅度 | 几个 % ~ 几十 % | **巨大且整齐**（本例 2.50×） | +| 重测是否抖动 | 抖 | **几乎不变**（< 0.5%） | +| 影响范围 | 随机某几个 shape | **同族大 M 变体**（小 M 变体反而"侥幸过关"，因为借 m16 的 kernel 不会更慢） | +| 任务形态 | 无关 | shape 里有**同算子、多个 M、相同 (N,K)** | +| 直接证据 | — | `final/source/csrc/w8a8_dispatch.cpp` 里的 `if (n == … && k == …)` **没有 m** | + +**为什么以前 72 次都没暴露**：历史任务基本是**一个算子一个 M**（例：MiniMax TP8 M8 的 5 个 shape += 5 个不同算子，`(N,K)` 天然唯一），`(n,k)` 键够用。本例是第一个"每个算子三个 M 变体"的任务。 + +**影响（比"这一次失败"更大）**：即使门侥幸过了，最终 artifact 也**不是各 worker 优化出来的那套** +——小 M 变体会静默地跑 m16 的 kernel，而报告仍写着各 shape 的 worker best。 +**最终交付与 measurement 脱节**。 + +**证据链（可复现）** + +```bash +W=nodes/worker29/workspaces/hy3-tp8-m8-16-4096-2-9dcfce15 +grep -n 'if (n ==' $W/final/source/csrc/w8a8_dispatch.cpp # 只有 n / k，没有 m +sha256sum $W/final/source/accepted_sources/hy3_tp8_qkv_proj_m4096.hip +# = ff84f677… = 该 worker iteration15 的 csrc/w8a8_gemm_hip.hip（逐字节相同） +ls $W/main/candidates/worker_2/hy3_tp8_qkv_proj_m4096/iteration15/csrc/ +# 只有 bindings.cpp + w8a8_gemm_hip.hip → worker 侧量的是自己的 kernel，没有 dispatch +``` + +**临时处置（不改代码）**：提交任务时**让每个算子只出现一个 M**（`(N,K)` 便唯一，分派不会再撞车）。 +换卡重测**无效**（本例已证明与卡无关）。 + +**状态：已修（2026-10-07）并实测验证**。修法：`_render_prebuilt_dispatch()` 改为按完整 +`(m, n, k)` 路由，族内是"降序 M 链"（精确 M 命中自己的 kernel；未优化的 M 落到不大于它的最大 M， +再兜底族内最小 M），`pack` 仍每族一条（ABI 无 m）；生成物额外记录 `dispatch_families` +（`final_artifact_link_launch` 事件）便于排查。测试：`test_gen_pipeline.py` 里 6 个新用例， +含"每个 artifact 都必须可达"与"顺序无关"。 +**实测**：同一任务的 `qkv_proj_m4096` 从 **1026 µs → 411.96 µs**（worker best 410.66），并正确通过。 +**注意**：修好之后，同族里若各 M 的 packed 布局不一致，会从"静默跑错 kernel（正确但慢）"变成 +"正确性失败" —— 那是上一条根因，按 M 分批解决。 + +--- + +## 与测量层的关系 + +`KQ-*` 报错都建立在"测量可信"之上（闸门已过）。如果同时看到 `gate_blocked` / `ENV-GATE-TIMEOUT`， +先按 `03` 解决可信度问题——在忙卡上得出的"性能回退"完全可能只是噪声。 diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/05-process-and-infra.md b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/05-process-and-infra.md new file mode 100644 index 00000000..4bff51ca --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/05-process-and-infra.md @@ -0,0 +1,74 @@ +# 05 · 进程与基础设施类 + +进程被谁杀、谁没跑起来、DKAO 自己崩没崩——都在这一层。 + +--- + +## PROC-AGENT-KILLED + +**agent 被 kill** ｜ 状态：环境 ｜ 真出现 **432** 次（历史上最多） + +``` +worker_0-m16_wqkv_a-iter8 failed: killed (see events log tail) +WorkerIterationFailed: worker_1-...-iter5 failed: killed +exit 143 / SIGTERM +``` + +### 判定 / 根因 + +agent 进程被外部终止：任务被 kill、agent 栈超时/OOM、父进程退出、容器资源限制。 +同样的文本既可能来自"用户点了 Stop"，也可能来自 OOM——所以**要结合同刻的事件看**： +- `task_killed {prior_pid, force}` → 人为停止（正常）； +- 没有任何 kill 事件、却出现 `killed (see events log tail)` → agent 侧被杀（查 agent 日志末尾、 + 系统 `dmesg` OOM 记录、`Max resident set` 限制）。 + +### 处置 + +- 人为 kill：无需处理； +- 非人为：按"基础设施失败"处理（管线本就给它补一轮，`_MAX_INFRASTRUCTURE_RECOVERY_ROUNDS = 5`）， + 同时查内存/句柄限制（`/proc//limits`，诊断文件里有）与 agent 超时配置。 + +--- + +## PROC-NO-WORKER + +**`worker bootstrap has fewer than one successful GPU worker`** ｜ 状态：设计行为 ｜ 真出现 1 次 + +所有 lane 的 bootstrap 都没产出可用内核（每卡 3 次尝试全灭）→ 没有基准、没有考卷，任务停止。 +**这是"全灭"的汇总**：逐个 lane 看 `worker_failed` / `workers//failure.json` 才知道病因 +（`GEN-BINARY-MISSING` / `GEN-NO-KERNEL` / `ENV-*` 都可能）。 + +处置：先修 lane 级病因（大多数是环境/工具），再重跑；只要有一个 lane 成功任务就会继续用幸存卡跑。 + +--- + +## PROC-BUG + +**DKAO 自身崩溃（AttributeError / TypeError / KeyError / traceback）** ｜ 状态：待办 + +这次扫描里 **0 条**属于 DKAO 目录（说明近期没有这类崩溃）。 +唯一命中过的同类错误 `champion_record_failed: AttributeError("'NoneType' object has no attribute 'get'")` +（09-11，26 次）来自 **harness-evolve** 的 `orchestrator/pipeline.py`，不是 DKAO 的代码路径—— +所以它没进本目录。想连父任务一起收：`collect_errors.py --task-type any`。 + +处置：这类必须修代码（补测试），不是环境问题；排查看 traceback 的第一帧是否落在 +`metainfer/tasks/dcu_kernel_auto_opt/orchestrator/*`。 + +--- + +## 僵尸/残留进程（不是报错，但常被当成报错） + +- **僵尸 orchestrator**：任务被 kill/purge 后，`ps` 里可能留下 + `[metainfer-dkao] `（父进程是 WebUI，它不 `wait()`）。**不占卡、不占 CPU**， + 重启 WebUI 即消失。看到它不等于"任务还在跑"。 +- **判据**：`ps -o stat= -p ` 显示 `Z`；`run.json` 的 `final_status` 已终态。 +- **残留工作区**：被 kill 的任务保留 state/workspace（这是设计：留证据）； + kernel repo 需要 `DELETE /api/sys-shell/?purge=true` 或在 UI 里删除才清掉。 + +--- + +## 证据会消失（重要） + +`purge` 会同时删 state_dir、workspace 和 kernel repo。2026-09-22 那次 +`dcu-kernel-auto-opt-5ed5c157` 的现场（含 `gpu_probe_failure.json`）就是这样没了的—— +所以：**值得留的报错原文，请贴进本目录对应 md**（README 里那节标本就是因此保留的）。 diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/README.md b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/README.md new file mode 100644 index 00000000..6943fec3 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/README.md @@ -0,0 +1,93 @@ +# DKAO 报错档案（error_catalog） + +这里专门收 DKAO（`dcu-kernel-auto-opt`）**真实出现过的报错**：原文、判定特征、根因、处置、以及是否已经修掉。 + +为什么要单独建目录：同一个报错文本会**以不同根因反复出现**。最典型的是 `shm_open: Permission denied` —— +2026-08-20 / 08-21 / 08-23 / 09-09 / 09-20~09-22 都报过，前几次被当成"容器 GPU 栈偶发抖动"， +直到 09-22 才确认是**启动方被文件沙箱关住**（`01-environment.md`）。UI 只给你那一句话， +这里给你"这句话属于哪一类、该干什么"。 + +## 怎么用 + +1. **先按报错原文查**（`samples.jsonl` 里 grep 最方便）： + ```bash + grep -i "shm_open" metainfer/tasks/dcu_kernel_auto_opt/error_catalog/samples.jsonl + ``` +2. 看 `class_id`，打开对应的 `NN-*.md` 里的那一节：判定 / 根因 / 处置 / 状态。 +3. **新报错**：在 `collect_errors.py` 的 `CLASSES` 里加一条规则（id + 正则 + 状态 + 指向的 md）， + 在执行目录里补一节解释，然后重跑收集器；测试在 `tests/test_error_catalog.py`。 + +```bash +# 重新收集（默认扫 nodes/ 下所有 dcu-kernel-auto-opt 任务，含 harness 子任务） +python3 -m metainfer.tasks.dcu_kernel_auto_opt.error_catalog.collect_errors +# 只扫某个任务 +python3 -m ...collect_errors --root nodes/worker29/.metainfer/tasks/dcu-kernel-auto-opt-7892abc2 +# 连 harness-evolve 父任务的报错一起收 +python3 -m ...collect_errors --task-type any +``` + +产物：`samples.jsonl`（每条：类别 / 任务 / 来源文件 / 事件 / 时间 / 原文，按原文去重并计数）、 +`SUMMARY.md`（各分类的次数、首末时间、示例任务）。两者都是**生成物，别手改**。 + +## 类别总览 + +| 类别 | 一句话 | 层 | 状态 | 详见 | +|---|---|---|---|---| +| `ENV-SHM-SANDBOX` | 进程树写不了 `/dev/shm`（文件沙箱），GPU 栈起不来 | 环境 | **已修** | [01](01-environment.md#env-shm-sandbox) | +| `ENV-NO-DEVICE` | `No HIP GPUs are available`：栈没起来 或 可见设备被双重过滤 | 环境 | 设计行为 | [01](01-environment.md#env-no-device) | +| `ENV-PROBE-WRAPPER` | `GPU probe failed after N attempt(s)`（外层包装，病因在下一行） | 环境 | 已修 | [01](01-environment.md#env-probe-wrapper) | +| `ENV-NO-LEASE` | scheduled/生产模式租不到卡 | 环境 | 环境 | [01](01-environment.md#env-no-lease) | +| `ENV-GATE-TIMEOUT` | 闸门等满 24h：卡长期被占，宁可不测 | 环境 | 设计行为 | [03](03-measurement-and-gate.md#env-gate-timeout) | +| `GEN-CONFLICT` | `workspace main already points to …`（子任务目录被复用） | 生成 | **已修** | [02](02-generate-and-agent.md#gen-conflict) | +| `GEN-SCHEMA` | 表单/契约参数非法（如 `tp_size must be 4 or 8`） | 生成 | 设计行为 | [02](02-generate-and-agent.md#gen-schema) | +| `GEN-CONTRACT` | API 契约摘要不可信 / 参考自检失败 | 生成 | 设计行为 | [02](02-generate-and-agent.md#gen-contract) | +| `GEN-OWNERSHIP` | agent 改了自己不该改的文件 | 生成 | 设计行为 | [02](02-generate-and-agent.md#gen-ownership) | +| `GEN-BINARY-MISSING` | agent 可执行文件/模块不存在（`'ccb'`） | 生成 | 环境 | [02](02-generate-and-agent.md#gen-binary-missing) | +| `GEN-AGENT-EJECT` | `Generate agent attempt N failed: nonzero exit` | 生成 | 环境 | [02](02-generate-and-agent.md#gen-agent-eject) | +| `GEN-NO-KERNEL` | agent 没能产出可用内核 | 生成 | 设计行为 | [02](02-generate-and-agent.md#gen-no-kernel) | +| `GEN-COORD-FAILED` | `Kernel coordination failed after 3 attempts`（外层包装） | 生成 | 环境 | [02](02-generate-and-agent.md#gen-coord-failed) | +| `MEAS-CMD-FAILED` | 基准/probe 子进程非零退出 | 测量 | 环境 | [03](03-measurement-and-gate.md#meas-cmd-failed) | +| `MEAS-REF-PREPARE` | 参考缓存准备失败 | 测量 | 环境 | [03](03-measurement-and-gate.md#meas-ref-prepare) | +| `MEAS-TIMEOUT` | 基准超时 | 测量 | 环境 | [03](03-measurement-and-gate.md#meas-timeout) | +| `MEAS-PMC-UNAVAILABLE` | PMC/hipprof 证据不可用（**跳过，不是失败**） | 测量 | 设计行为 | [03](03-measurement-and-gate.md#meas-pmc-unavailable) | +| `KQ-CORRECTNESS` | 内核算错（正确性/Graph 校验不过） | 内核质量 | 设计行为 | [04](04-kernel-quality.md#kq-correctness) | +| `KQ-INLINE-ASM` | 内联汇编验收门不通过 | 内核质量 | 设计行为 | [04](04-kernel-quality.md#kq-inline-asm) | +| `KQ-PERF-REGRESSION` | 性能回退（最终验收不过） | 内核质量 | 设计行为 | [04](04-kernel-quality.md#kq-perf-regression) | +| `PROC-AGENT-KILLED` | agent 被 kill（超时/OOM/外部） | 进程 | 环境 | [05](05-process-and-infra.md#proc-agent-killed) | +| `PROC-NO-WORKER` | 一张卡都没跑通 | 进程 | 设计行为 | [05](05-process-and-infra.md#proc-no-worker) | +| `PROC-BUG` | DKAO 自身崩溃（AttributeError 之类） | 进程 | 待办 | [05](05-process-and-infra.md#proc-bug) | + +状态含义：**已修** = 代码层面消除了成因，再出现要按新根因查；**设计行为** = 管线拒绝给出不可信的数， +本来就会反复出现；**环境** = 机器被别人占用/工具缺失时的正常表现；**待办** = 已知未处理。 + +## 标本：2026-09-22 那次（现在已经修掉） + +任务 `dcu-kernel-auto-opt-5ed5c157`（WebUI 发起，shape `tp4_wqkv_a_m2`，12:53:04 → 12:53:58 就死了）： + +``` +Task stopped in generate_kernel_repo:GPU probe failed after 3 attempt(s) (RuntimeError): +python3 /root/zth_agent/kernel-repos/dcu-kernel-auto-opt-5ed5c157/w8a8_bench.py --source ... --m 1 --n 1 --k 1 --probe +failed (3): shm_open: Permission denied +/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:817: UserWarning: Can't initialize amdsmi - Error code: 32 +... +{"ts": 1790052837.94, "attempts": 3, "visible_devices_env": "0", "uid": 0, "gid": 0, + "dev_shm": [rocm_smi_card1..4(mode 0o777), c3000_topology_info, sem.mp-...], + "admission": {"checked": true, "usable": true, "device": 0, "vram_percent": 0.01, "util_percent": 0.0, + "reason": "card is idle"}} +``` + +**判定过程（可复现）**：卡是空闲的（0.01% VRAM）；同一个 repo、同一条命令在**无沙箱** shell 里 +`rc=0`、`device_name=K500SM_AI`；在**沙箱**里逐字复现 `shm_open: Permission denied`。 +用 DSH 自带的 `landlock-run`（授权范围 = workspace-write）也能复现，且 `open("/dev/shm/x", O_CREAT)` +与 `open("/dev/shm/rocm_smi_card1", O_RDWR)` 都返回 `EACCES`，`O_RDONLY` 正常—— +即 `/dev/shm` 对该进程树是只读的。根因：WebUI（`serve.py` 8765）是 09-21 15:31 从 +**workspace-write 沙箱会话**里重启的，Landlock 域被它拉起的每个 orchestrator/probe 继承， +而子进程无法解除。详见 [`01-environment.md`](01-environment.md#env-shm-sandbox)。 + +> ⚠️ 该任务与 kernel repo 后来被清理（purge）了，所以 `samples.jsonl` 里已经没有它的记录—— +> 这正是本目录存在的理由：**值得记住的原文请贴进对应 md**，磁盘上的证据会被删，文字不会。 + +## 快照信息 + +`samples.jsonl` / `SUMMARY.md` 是收集器在某一刻对磁盘证据的抓取，任务被 purge 后对应记录就没了。 +最近一次抓取：见 `SUMMARY.md` 顶部的「生成时间」与「扫描任务数」。 diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/SUMMARY.md b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/SUMMARY.md new file mode 100644 index 00000000..8ba2a51b --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/SUMMARY.md @@ -0,0 +1,36 @@ +# DKAO 报错汇总（自动生成） + +- 扫描任务数：**165** +- 命中报错条目：**1538**（去重后） +- 生成时间：2026-10-09 00:43 + +由 `collect_errors.py` 生成，请勿手改；新增/调整判定规则改那里，人类解释写在 `NN-*.md`。 + +| 类别 | 标题 | 层 | 状态 | 次数 | 首次 | 最近 | 示例任务 | +|---|---|---|---|---:|---|---|---| +| `GEN-AGENT-EJECT` | agent 非零退出（逐次尝试） | generate | environment | 565 | 2026-07-28 09:21 | 2026-10-08 11:57 | `dcu-kernel-auto-opt-d4dd2dc5`, `glm5-2-dsh-tp8-m4096-1-e6a280a2`, `hy3-dsh-tp8-m16-2-dc53295a` … (+83) | +| `PROC-AGENT-KILLED` | agent 被 kill（含超时/OOM） | process | environment | 438 | 2026-07-27 01:42 | 2026-10-08 10:23 | `dcu-kernel-auto-opt-2e64c51b`, `dcu-kernel-auto-opt-602ee1ee`, `dcu-kernel-auto-opt-7892abc2` … (+34) | +| `MEAS-PMC-UNAVAILABLE` | PMC/hipprof 证据不可用（跳过） | measurement | by design | 226 | 2026-08-03 16:05 | 2026-10-08 00:44 | `dcu-kernel-auto-opt-7892abc2`, `dcu-kernel-auto-opt-cc86e2b2`, `dcu-kernel-auto-opt-fa8de7cc` … (+104) | +| `KQ-PERF-REGRESSION` | 性能回退（最终验收不过） | kernel-quality | by design | 78 | 2026-07-28 14:40 | 2026-10-07 23:20 | `dcu-kernel-auto-opt-7eafe2cc`, `dcu-kernel-auto-opt-8c1f2eac`, `dcu-kernel-auto-opt-c7e73766` … (+26) | +| `ENV-GATE-TIMEOUT` | 闸门等待/等超时（卡被占） | environment | by design | 69 | 2026-09-17 21:44 | 2026-09-23 19:38 | `test-9-17-1-aaea0725:iteration_001/glm_tp8_fused_qkv_a_proj_m4096`, `test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16`, `test-9-17-1-aaea0725:iteration_001/minimax_tp8_qkv_proj_m4096` … (+14) | +| `GEN-CONFLICT` | 工作区 main 已被别的 repo 占用 | generate | fixed | 64 | 2026-09-11 18:34 | 2026-09-15 16:26 | `hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_shared_down_proj_m16`, `hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16`, `hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_o_proj_m16` … (+14) | +| `MEAS-CMD-FAILED` | 基准/PROBE 子进程非零退出 | measurement | environment | 55 | 2026-07-28 11:22 | 2026-09-23 21:46 | `dcu-kernel-auto-opt-2e64c51b`, `dcu-kernel-auto-opt-602ee1ee`, `dcu-kernel-auto-opt-7892abc2` … (+23) | +| `GEN-NO-KERNEL` | agent 没能产出可用内核 | generate | by design | 18 | 2026-07-28 11:49 | 2026-09-14 17:37 | `dcu-kernel-auto-opt-7892abc2`, `dcu-kernel-auto-opt-a473ffe1`, `glm5-2-dsh-tp4-m4096-1-85cb689c` … (+5) | +| `KQ-CORRECTNESS` | 内核算错（正确性/Graph 校验不过） | kernel-quality | by design | 10 | 2026-07-29 14:30 | 2026-10-09 00:40 | `dcu-kernel-auto-opt-8b6136de`, `dcu-kernel-auto-opt-c7e73766`, `dcu-kernel-auto-opt-cc86e2b2` … (+5) | +| `ENV-SHM-SANDBOX` | 无法写 /dev/shm（文件沙箱） | environment | fixed | 5 | 2026-08-21 17:15 | 2026-09-09 22:05 | `hy3-dsh-tp8-m16-9-8-0161e718`, `hy3-tp4-m4096-dsh-test1-bfdf3002`, `minimaxm3-dsh-tp4-m4096-1-0c2f84a9` | +| `GEN-OWNERSHIP` | agent 改了自己不该改的文件 | generate | by design | 4 | 2026-07-28 11:27 | 2026-07-28 11:43 | `dcu-kernel-auto-opt-a473ffe1` | +| `GEN-BINARY-MISSING` | agent 可执行文件/模块不存在 | generate | environment | 2 | 2026-08-03 16:40 | 2026-08-03 16:40 | `dcu-kernel-auto-opt-7892abc2` | +| `ENV-NO-DEVICE` | 看不到设备（No HIP GPUs are available） | environment | by design | 1 | 2026-08-20 10:43 | 2026-08-20 10:43 | `hy3-tp4-4096-sonnet-5i-1-cbcd099f` | +| `GEN-COORD-FAILED` | 生成阶段重试耗尽（外层包装） | generate | environment | 1 | 2026-07-28 19:29 | 2026-07-28 19:29 | `dcu-kernel-auto-opt-7eafe2cc` | +| `KQ-INLINE-ASM` | 内联汇编验收门不通过 | kernel-quality | by design | 1 | 2026-08-27 04:35 | 2026-08-27 04:35 | `hy3-dsh-tp8-m4096-1-0ebb994d` | +| `PROC-NO-WORKER` | 一张卡都没跑通（bootstrap 全灭） | process | by design | 1 | 2026-08-03 16:40 | 2026-08-03 16:40 | `dcu-kernel-auto-opt-7892abc2` | +| `ENV-NO-LEASE` | 租不到卡（scheduled/生产模式） | environment | environment | 0 | — | — | — | +| `ENV-PROBE-WRAPPER` | probe 重试后仍失败（外层包装） | environment | fixed | 0 | — | — | — | +| `GEN-CONTRACT` | API 契约/参考自检不可信 | generate | by design | 0 | — | — | — | +| `GEN-SCHEMA` | 表单/契约参数非法 | generate | by design | 0 | — | — | — | +| `KQ-PACK-LAYOUT-CONFLICT` | 同一权重的多个 M 用了不同 packed 布局（合并期冲突） | kernel-quality | fixed | 0 | — | — | — | +| `MEAS-REF-PREPARE` | 参考缓存准备失败 | measurement | environment | 0 | — | — | — | +| `MEAS-TIMEOUT` | 基准超时 | measurement | environment | 0 | — | — | — | +| `PROC-BUG` | DKAO 自身的代码缺陷（崩溃） | process | open | 0 | — | — | — | + +> 未命中任何类别的条目不会出现在汇总里；如果某个新报错总被漏掉，就在 `CLASSES` 里加一条规则。 diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/__init__.py b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/__init__.py new file mode 100644 index 00000000..6d7659bb --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/__init__.py @@ -0,0 +1,35 @@ +"""A dedicated home for the errors DKAO actually produces. + +Why a folder and not a wiki page: every one of these errors has cost somebody +hours, and the same *text* keeps coming back with a different root cause. The +WebUI shows the sentence that failed; what an operator needs next to it is +"which class is this, what causes it, what do I do, and has it already been +fixed" — see ``README.md`` and the ``NN-*.md`` symptom files. + +``collect_errors.py`` is the machine half: it walks the task state dirs (and the +worker workspaces) under ``nodes/*/``, pulls out every error-ish string, tags it +with a class, and writes ``samples.jsonl`` + ``SUMMARY.md``. Re-run it after a +bad night and the new evidence lands in the repository instead of in a chat log. +""" + +from .collect_errors import ( + CLASSES, + ErrorClass, + Record, + classify, + collect, + main, + render_summary, + scan_task, +) + +__all__ = [ + "CLASSES", + "ErrorClass", + "Record", + "classify", + "collect", + "main", + "render_summary", + "scan_task", +] diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/collect_errors.py b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/collect_errors.py new file mode 100644 index 00000000..c061c5cf --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/collect_errors.py @@ -0,0 +1,585 @@ +#!/usr/bin/env python3 +"""Collect the errors DKAO actually produced into one reviewable file. + +The WebUI shows one sentence when a task dies. This walks the task state dirs +(and the worker workspaces) under ``nodes/*/``, extracts every error-ish string, +tags it with one of the classes declared below, and writes: + + samples.jsonl one row per (task, class, event, message) with a count + SUMMARY.md counts per class, first/last seen, example task ids + +Run it after a failure to put the evidence in the repository rather than in a +chat log:: + + python3 -m metainfer.tasks.dcu_kernel_auto_opt.error_catalog.collect_errors \ + --root nodes --out metainfer/tasks/dcu_kernel_auto_opt/error_catalog + +The classes are the contract: adding a symptom means adding a rule with a +matching section in one of the ``NN-*.md`` files, so "what is this" and "what do +I do" stay one click apart. +""" + +from __future__ import annotations + +import argparse +import json +import re +import sys +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, Iterator, List, Optional, Sequence, Tuple + +#: Raw evidence is trimmed to keep samples.jsonl greppable. +MAX_RAW_CHARS = 1200 + + +@dataclass(frozen=True) +class ErrorClass: + """One recurring failure signature. + + ``status`` is the operationally important field: + + * ``fixed`` — a code change removed the cause; recurrence means a new cause + wearing the same words (see ENV-SHM-SANDBOX); + * ``by design`` — the pipeline refusing to produce a number it cannot trust + (a gate, an ownership check). Not a bug, and it should recur; + * ``environment`` — correct behaviour on a machine somebody else is using; + * ``open`` — known, not yet addressed. + """ + + id: str + title: str + layer: str + status: str + patterns: Tuple[str, ...] + doc: str + note: str = "" + + def matches(self, text: str) -> bool: + return any(re.search(p, text) for p in self.patterns) + + +#: Ordered most specific first: the first match wins, so a message that contains +#: both "shm_open" and "probe failed" is filed under the sandbox, not the wrapper. +CLASSES: Tuple[ErrorClass, ...] = ( + # ---------------------------------------------------------------- environment + ErrorClass( + "ENV-SHM-SANDBOX", "无法写 /dev/shm（文件沙箱）", "environment", "fixed", + (r"shm_open: (permission denied|invalid argument)", + r"fchmod: bad file descriptor", + r"can'?t initialize amdsmi"), + "01-environment.md", + "ROCm/amdsmi 用 /dev/shm 会合文件起 GPU 栈；被文件沙箱关住就必然失败。" + "2026-09-22 已修：probe 先自检、不再空重试、报错指名启动方；" + "WebUI 启动侧也有三道护栏。", + ), + ErrorClass( + "ENV-NO-DEVICE", "看不到设备（No HIP GPUs are available）", "environment", "by design", + (r"no hip gpus are available", r"visible_devices.{0,12}(0|none)"), + "01-environment.md", + "两种成因：栈没起来（见 ENV-SHM-SANDBOX），或 HIP/ROCR 可见设备被双重过滤" + "（同一非零 index 同时设 HIP_VISIBLE_DEVICES 与 ROCR_VISIBLE_DEVICES）。", + ), + ErrorClass( + "ENV-NO-LEASE", "租不到卡（scheduled/生产模式）", "environment", "environment", + (r"no gpu lease became available", r"no device passed the admission gate"), + "01-environment.md", + ), + ErrorClass( + "ENV-GATE-TIMEOUT", "闸门等待/等超时（卡被占）", "environment", "by design", + (r"did not pass the measurement gate", r"gate probe failed", + r"no device state was read", r"gate_blocked", r"gate_blocked\b"), + "03-measurement-and-gate.md", + "`gate_blocked` 审计行代表\"卡不干净、这一轮不测\"；等满 24h 的最终形态是 " + "MeasurementGateBlocked。两者都是设计行为，不是故障。", + ), + ErrorClass( + "ENV-PROBE-WRAPPER", "probe 重试后仍失败（外层包装）", "environment", "fixed", + (r"gpu probe failed after \d+ attempt",), + "01-environment.md", + "真正的病因在下一行；本类只是 DKAO 的重试包装，不要据此判断环境好坏。", + ), + # ------------------------------------------------------------- generate/agent + ErrorClass( + "GEN-CONFLICT", "工作区 main 已被别的 repo 占用", "generate", "fixed", + (r"already points to",), + "02-generate-and-agent.md", + ), + ErrorClass( + "GEN-SCHEMA", "表单/契约参数非法", "generate", "by design", + (r"tp_size must be 4 or 8", r"gpu_mode must be", + r"execution_profile must be", r"unknown task_type"), + "02-generate-and-agent.md", + ), + ErrorClass( + "GEN-CONTRACT", "API 契约/参考自检不可信", "generate", "by design", + (r"staged api contract digest is not trusted", + r"trusted pytorch reference self-test failed", + r"generate scaffold unexpectedly contains a hip implementation"), + "02-generate-and-agent.md", + ), + ErrorClass( + "GEN-OWNERSHIP", "agent 改了自己不该改的文件", "generate", "by design", + (r"outside its hip ownership", r"changed files outside"), + "02-generate-and-agent.md", + ), + ErrorClass( + "GEN-BINARY-MISSING", "agent 可执行文件/模块不存在", "generate", "environment", + (r"no such file or directory: '(ccb|claude|dsh)'", r"no module named '"), + "02-generate-and-agent.md", + ), + ErrorClass( + "GEN-AGENT-EJECT", "agent 非零退出（逐次尝试）", "generate", "environment", + (r"generate agent attempt \d+ failed: nonzero exit", + r"failed: nonzero exit \d+"), + "02-generate-and-agent.md", + ), + ErrorClass( + "GEN-NO-KERNEL", "agent 没能产出可用内核", "generate", "by design", + (r"could not generate a valid initial hip kernel", + r"did not generate", r"did not write proposal\.json"), + "02-generate-and-agent.md", + ), + ErrorClass( + "GEN-COORD-FAILED", "生成阶段重试耗尽（外层包装）", "generate", "environment", + (r"kernel coordination failed after",), + "02-generate-and-agent.md", + ), + # ---------------------------------------------------------------- measurement + ErrorClass( + "MEAS-REF-PREPARE", "参考缓存准备失败", "measurement", "environment", + (r"--prepare-reference failed", r"reference cache"), + "03-measurement-and-gate.md", + ), + ErrorClass( + "MEAS-PMC-UNAVAILABLE", "PMC/hipprof 证据不可用（跳过）", "measurement", "by design", + (r"worker_pmc_skipped", r"pmc .*(unavailable|not available)", + r"hipprof"), + "03-measurement-and-gate.md", + "不是失败：这一轮按无 PMC 证据继续，调优少了 ISA 依据但不影响正确性。", + ), + ErrorClass( + "MEAS-CMD-FAILED", "基准/PROBE 子进程非零退出", "measurement", "environment", + (r"returned non-zero exit status", r"command '\[", + r"failed \(\d+\):", r"--probe failed"), + "03-measurement-and-gate.md", + ), + ErrorClass( + "MEAS-TIMEOUT", "基准超时", "measurement", "environment", + (r"timed out", r"timeout"), + "03-measurement-and-gate.md", + ), + # ------------------------------------------------------------- kernel quality + ErrorClass( + "KQ-PACK-LAYOUT-CONFLICT", + "同一权重的多个 M 用了不同 packed 布局（合并期冲突）", + "kernel-quality", "fixed", + (r"packed layout conflicts", r"packed layout disagrees", + r"layout_conflict", r"packed layouts? disagree", + r"members of one \(n, k\) family pack"), + "04-kernel-quality.md", + "冻结的 prepare_weight ABI 没有 M，同族必须共用一份 packed 镜像；" + "final_layout_gate 在合并前比对每个成员的 pack 指纹，冲突时按布局分批验收。", + ), + ErrorClass( + "KQ-CORRECTNESS", "内核算错（正确性/Graph 校验不过）", "kernel-quality", "by design", + (r"exact correctness failed", r"graph/correctness validation failed", + r"mismatch_count", r"paired m=2 fallback failed"), + "04-kernel-quality.md", + "这是 DKAO 的日常工作：agent 第一版内核经常不对，重试/repair 会处理。", + ), + ErrorClass( + "KQ-INLINE-ASM", "内联汇编验收门不通过", "kernel-quality", "by design", + (r"inline asm acceptance gate failed",), + "04-kernel-quality.md", + ), + ErrorClass( + "KQ-PERF-REGRESSION", "性能回退（最终验收不过）", "kernel-quality", "by design", + (r"performance regressed", r"final_synthesis_rejected", r"p90 .*regress"), + "04-kernel-quality.md", + ), + # ------------------------------------------------------------ process/infra + ErrorClass( + "PROC-AGENT-KILLED", "agent 被 kill（含超时/OOM）", "process", "environment", + (r"failed: killed", r"see events log tail", r"exit 143", r"sigkill|sigterm"), + "05-process-and-infra.md", + ), + ErrorClass( + "PROC-NO-WORKER", "一张卡都没跑通（bootstrap 全灭）", "process", "by design", + (r"fewer than one successful gpu worker",), + "05-process-and-infra.md", + ), + ErrorClass( + "PROC-BUG", "DKAO 自身的代码缺陷（崩溃）", "process", "open", + (r"attributeerror", r"typeerror", r"keyerror", r"traceback \(most recent call last\)", + r"nonetype' object has no attribute"), + "05-process-and-infra.md", + ), +) + +#: Events worth extracting even when their payload has no obvious error field. +_ERRORISH_EVENT_TOKENS = ( + "error", "failed", "blocked", "rejected", "discarded", "exhausted", + "incomplete", "killed", "skipped", "timeout", "retry", + # A packed-layout conflict is reported as its own event (the merge repairs + # it by batching, so no exception is raised) and must still be counted. + "conflict", +) + + +def classify(text: str) -> Optional[ErrorClass]: + """The class of one message, or ``None`` when nothing matches.""" + lowered = str(text or "").lower() + for cls in CLASSES: + if cls.matches(lowered): + return cls + return None + + +@dataclass +class Record: + """One extracted error occurrence.""" + + class_id: str + task_id: str + task_type: str + source: str + event: str + ts: Optional[float] + raw: str + run_id: Optional[str] = None + occurrences: int = 1 + + def to_json(self) -> Dict[str, Any]: + return { + "class_id": self.class_id, + "task_id": self.task_id, + "run_id": self.run_id, + "task_type": self.task_type, + "source": self.source, + "event": self.event, + "ts": self.ts, + "occurrences": self.occurrences, + "raw": self.raw, + } + + +def _trim(text: Any, limit: int = MAX_RAW_CHARS) -> str: + s = str(text if text is not None else "") + s = re.sub(r"\s+", " ", s).strip() + return s if len(s) <= limit else s[: limit - 1] + "…" + + +def _first_text(payload: Dict[str, Any]) -> str: + for key in ("error", "reason", "detail", "message", "stderr", "failure"): + if payload.get(key): + return _trim(payload[key]) + return _trim(json.dumps(payload, ensure_ascii=False)) + + +def _is_errorish(event_type: str, payload: Dict[str, Any]) -> bool: + lowered = str(event_type).lower() + if any(tok in lowered for tok in _ERRORISH_EVENT_TOKENS): + return True + return any(payload.get(k) for k in ("error", "failure")) + + +def _read_jsonl(path: Path) -> Iterator[Dict[str, Any]]: + try: + with path.open("r", encoding="utf-8", errors="replace") as fh: + for line in fh: + line = line.strip() + if not line: + continue + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict): + yield row + except OSError: + return + + +def _read_json(path: Path) -> Optional[Dict[str, Any]]: + try: + data = json.loads(path.read_text(encoding="utf-8", errors="replace")) + except (OSError, ValueError): + return None + return data if isinstance(data, dict) else None + + +def _task_type(state_dir: Path) -> str: + run = _read_json(state_dir / "run.json") or {} + req = _read_json(state_dir / "requirements.json") or {} + return str(run.get("task_type") or req.get("task_type") or "unknown") + + +def _task_label(state_dir: Path) -> str: + """A readable id for an embedded DKAO run. + + Children of a harness run are all called ``state``, so the directory name + alone would file every one of them under the same useless id. Their path + carries the information: ``workspaces//children//``. + """ + parts = state_dir.parts + if state_dir.name == "state" and "children" in parts: + idx = parts.index("children") + parent = parts[idx - 1] if idx >= 1 else "?" + tail = "/".join(parts[idx + 1:-1]) or state_dir.parent.name + return f"{parent}:{tail}" + return state_dir.name + + +def scan_task(state_dir: Path, workspace_dir: Optional[Path] = None, + ) -> List[Record]: + """Every classifiable error string inside one task's evidence.""" + task_id = _task_label(state_dir) + task_type = _task_type(state_dir) + run = _read_json(state_dir / "run.json") + run_id = str((run or {}).get("task_id") or "") or None + out: List[Record] = [] + + def add(source: str, event: str, ts: Optional[float], text: Any) -> None: + cls = classify(f"{event} {text}") + if cls is None: + return + out.append(Record(class_id=cls.id, task_id=task_id, task_type=task_type, + source=source, event=str(event), run_id=run_id, + ts=float(ts) if ts else None, raw=_trim(text))) + + if run: + ts = run.get("last_update") or run.get("created_at") + if run.get("notes"): + add("run.json", f"note:{run.get('final_status') or 'note'}", ts, + run["notes"][-1]) + elif str(run.get("final_status") or "") in {"stopped", "failed"}: + add("run.json", f"final_status:{run.get('final_status')}", + ts, run.get("last_outcome")) + + for row in _read_jsonl(state_dir / "timeline.jsonl"): + payload = row.get("payload") if isinstance(row.get("payload"), dict) else {} + event = str(row.get("type") or "") + if not _is_errorish(event, payload): + continue + add("timeline.jsonl", event, row.get("ts"), _first_text(payload)) + + probe = _read_json(state_dir / "gpu_probe_failure.json") + if probe: + shm = probe.get("dev_shm_write") if isinstance(probe.get("dev_shm_write"), dict) else {} + text = ( + f"gpu probe failed; dev_shm_write={json.dumps(shm, ensure_ascii=False)}; " + f"attempts={probe.get('attempts')} " + f"performed={probe.get('attempts_performed')}" + ) + if not shm.get("ok", True): + text = ("shm_open: " + str(shm.get("error")) + " " + text) + add("gpu_probe_failure.json", "probe_diagnostics", probe.get("ts"), text) + + for row in _read_jsonl(state_dir / "measurement_gate.jsonl"): + if row.get("event") == "gate_blocked": + add("measurement_gate.jsonl", "gate_blocked", row.get("ts"), + f"gpu{row.get('gpu')} site={row.get('site')} " + f"vram={row.get('vram_percent')} util={row.get('util_percent')} " + f"reasons={row.get('reasons')}") + + if workspace_dir and workspace_dir.is_dir(): + for failure in sorted(workspace_dir.glob("workers/*/failure.json")): + data = _read_json(failure) or {} + add("workers/failure.json", + f"worker:{data.get('worker_id')}:{data.get('stage')}", + data.get("timestamp"), data.get("error") or json.dumps(data)) + for status in sorted(workspace_dir.glob("workers/*/status.json")): + data = _read_json(status) or {} + if str(data.get("state") or "") in {"failed", "timed_out", "error"}: + add("workers/status.json", + f"worker:{data.get('worker_id')}:{data.get('state')}", + data.get("last_update"), json.dumps(data, ensure_ascii=False)) + return out + + +def _is_state_dir(path: Path) -> bool: + return (path / "run.json").is_file() or (path / "timeline.jsonl").is_file() + + +def _node_roots(root: Path) -> List[Path]: + """Directories that hold ``.metainfer/`` and/or ``workspaces/``. + + Accepts whatever the operator points at: the ``nodes`` directory, one node + directory, or something in between. Getting this wrong is silent and costly + — pointing the collector at a node dir used to find only the WebUI tasks and + quietly drop every harness child, which is where most environment failures + are recorded. + """ + if (root / ".metainfer").is_dir() or (root / "workspaces").is_dir(): + return [root] + if root.name == "tasks" and root.parent.name == ".metainfer": + return [root.parent.parent] + return [p for p in sorted(root.glob("*")) + if p.is_dir() and ((p / ".metainfer").is_dir() + or (p / "workspaces").is_dir())] + + +def _task_dirs(root: Path, *, task_type: Optional[str] = None) -> List[Path]: + """Every DKAO state dir under ``root``. + + Two layouts hold them and both matter: + + * ``nodes//.metainfer/tasks/`` — a task started from the WebUI; + * ``nodes//workspaces//children/**/state`` — a DKAO task + started by the harness-evolve parent, one per question per attempt. These + are where most of the *environment* failures live, so skipping them would + leave the interesting half of the history out. + """ + if _is_state_dir(root): + candidates = [root] + else: + candidates = [] + for node in _node_roots(root): + tasks = node / ".metainfer" / "tasks" + if tasks.is_dir(): + candidates.extend(sorted(p for p in tasks.iterdir() if p.is_dir())) + workspaces = node / "workspaces" + if workspaces.is_dir(): + candidates.extend(sorted(workspaces.glob("*/children/**/state"))) + found = [p for p in candidates if p.is_dir() and _is_state_dir(p)] + if task_type: + found = [p for p in found if _task_type(p) == task_type] + # A task dir may be reachable through two patterns; keep each once. + seen: Dict[str, Path] = {} + for path in found: + seen.setdefault(str(path.resolve()), path) + return sorted(seen.values()) + + +def _workspace_for(state_dir: Path) -> Optional[Path]: + """The artifact tree next to a state dir, in either layout.""" + sibling = state_dir.parent / "workspace" + if sibling.is_dir(): + return sibling + parts = list(state_dir.parts) + if ".metainfer" in parts: + idx = len(parts) - 1 - parts[::-1].index(".metainfer") + if idx > 0: + candidate = Path(*parts[: idx - 1]) / "workspaces" / state_dir.name + if candidate.is_dir(): + return candidate + return None + + +@dataclass +class Collection: + records: List[Record] = field(default_factory=list) + scanned: int = 0 + + def by_class(self) -> Dict[str, List[Record]]: + out: Dict[str, List[Record]] = {} + for rec in self.records: + out.setdefault(rec.class_id, []).append(rec) + return out + + +def collect(roots: Sequence[Path], *, + task_type: Optional[str] = "dcu-kernel-auto-opt") -> Collection: + """Scan every root; identical messages collapse into one counted record.""" + merged: Dict[Tuple[str, str, str, str], Record] = {} + collection = Collection() + for root in roots: + for state_dir in _task_dirs(Path(root), task_type=task_type): + collection.scanned += 1 + for rec in scan_task(state_dir, _workspace_for(state_dir)): + key = (rec.class_id, rec.task_id, rec.source, rec.raw) + if key in merged: + merged[key].occurrences += 1 + else: + merged[key] = rec + collection.records = sorted( + merged.values(), key=lambda r: (r.class_id, r.task_id, r.source)) + return collection + + +def render_summary(collection: Collection, *, examples: int = 3) -> str: + """A markdown table: class, count, first/last seen, example tasks.""" + import datetime + + buckets = collection.by_class() + lines = [ + "# DKAO 报错汇总（自动生成）", + "", + f"- 扫描任务数：**{collection.scanned}**", + f"- 命中报错条目：**{len(collection.records)}**（去重后）", + f"- 生成时间：{datetime.datetime.now().strftime('%Y-%m-%d %H:%M')}", + "", + "由 `collect_errors.py` 生成，请勿手改；新增/调整判定规则改那里，" + "人类解释写在 `NN-*.md`。", + "", + "| 类别 | 标题 | 层 | 状态 | 次数 | 首次 | 最近 | 示例任务 |", + "|---|---|---|---|---:|---|---|---|", + ] + ordered = sorted( + CLASSES, key=lambda c: (-len(buckets.get(c.id, [])), c.id)) + for cls in ordered: + rows = buckets.get(cls.id) or [] + stamps = sorted(r.ts for r in rows if r.ts) + first = (datetime.datetime.fromtimestamp(stamps[0]).strftime("%Y-%m-%d %H:%M") + if stamps else "—") + last = (datetime.datetime.fromtimestamp(stamps[-1]).strftime("%Y-%m-%d %H:%M") + if stamps else "—") + seen: List[str] = [] + for rec in rows: + if rec.task_id not in seen: + seen.append(rec.task_id) + shown = ", ".join(f"`{t}`" for t in seen[:examples]) + if len(seen) > examples: + shown += f" … (+{len(seen) - examples})" + lines.append( + f"| `{cls.id}` | {cls.title} | {cls.layer} | {cls.status} | " + f"{len(rows)} | {first} | {last} | {shown or '—'} |") + lines += ["", "> 未命中任何类别的条目不会出现在汇总里；" + "如果某个新报错总被漏掉，就在 `CLASSES` 里加一条规则。"] + return "\n".join(lines) + "\n" + + +def main(argv: Optional[Sequence[str]] = None) -> int: + here = Path(__file__).resolve().parent + parser = argparse.ArgumentParser( + prog="dkao-collect-errors", + description="把 DKAO 的历史报错抽出来，落到 error_catalog/ 下。") + parser.add_argument("--root", action="append", default=None, + help="nodes 目录、某个节点的 .metainfer/tasks，" + "或单个任务 state 目录（可重复）") + parser.add_argument("--task-type", default="dcu-kernel-auto-opt", + help="只收这个 task_type 的任务（默认 dcu-kernel-auto-opt；" + "传 any 收全部）") + parser.add_argument("--out", default=str(here), + help="输出目录（默认就是本目录）") + parser.add_argument("--quiet", action="store_true") + args = parser.parse_args(argv) + + roots = [Path(r) for r in (args.root or [])] + if not roots: + repo = here.parents[3] # …/MetaInfer + nodes = repo / "nodes" + roots = [nodes] if nodes.is_dir() else [repo] + wanted = None if str(args.task_type).strip().lower() == "any" else args.task_type + collection = collect(roots, task_type=wanted) + out = Path(args.out) + out.mkdir(parents=True, exist_ok=True) + with (out / "samples.jsonl").open("w", encoding="utf-8") as fh: + for rec in collection.records: + fh.write(json.dumps(rec.to_json(), ensure_ascii=False) + "\n") + (out / "SUMMARY.md").write_text(render_summary(collection), encoding="utf-8") + if not args.quiet: + buckets = collection.by_class() + print(f"scanned {collection.scanned} task dir(s) from " + f"{', '.join(str(r) for r in roots)}") + for cls in sorted(CLASSES, key=lambda c: -len(buckets.get(c.id, []))): + n = len(buckets.get(cls.id, [])) + if n: + print(f" {n:>4} {cls.id:<20} {cls.title}") + print(f"wrote {out / 'samples.jsonl'} and {out / 'SUMMARY.md'}") + return 0 + + +if __name__ == "__main__": # pragma: no cover - CLI + raise SystemExit(main()) diff --git a/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/samples.jsonl b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/samples.jsonl new file mode 100644 index 00000000..689b1441 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/error_catalog/samples.jsonl @@ -0,0 +1,1538 @@ +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-2a6130", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789658486.331536, "occurrences": 2, "raw": "gpu3 site=benchmark vram=85.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-2a6130", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789660286.5759768, "occurrences": 2, "raw": "gpu3 site=benchmark vram=94.0 util=60.0 reasons=['VRAM 94% > 90% limit', 'device busy: HCU 60.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-2a6130", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789662086.8112938, "occurrences": 2, "raw": "gpu3 site=benchmark vram=96.0 util=94.2 reasons=['VRAM 96% > 90% limit', 'device busy: HCU 94.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-2a6130", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789663887.0470133, "occurrences": 4, "raw": "gpu3 site=benchmark vram=96.0 util=100.0 reasons=['VRAM 96% > 90% limit', 'device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-2a6130", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789667487.4320915, "occurrences": 2, "raw": "gpu3 site=benchmark vram=96.0 util=99.2 reasons=['VRAM 96% > 90% limit', 'device busy: HCU 99.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-o-proj-m16-3768c0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789655185.09061, "occurrences": 2, "raw": "gpu2 site=benchmark vram=44.0 util=53.3 reasons=['device busy: HCU 53.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-o-proj-m16-3768c0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789656985.2473767, "occurrences": 2, "raw": "gpu2 site=benchmark vram=96.0 util=3.3 reasons=['VRAM 96% > 90% limit', 'device busy: HCU 3.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-o-proj-m16-3768c0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789658785.4459069, "occurrences": 2, "raw": "gpu2 site=benchmark vram=85.0 util=28.3 reasons=['device busy: HCU 28.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-o-proj-m16-3768c0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789660585.5749304, "occurrences": 2, "raw": "gpu2 site=benchmark vram=94.0 util=89.2 reasons=['VRAM 94% > 90% limit', 'device busy: HCU 89.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-o-proj-m16-3768c0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789662385.822207, "occurrences": 2, "raw": "gpu2 site=benchmark vram=95.0 util=88.3 reasons=['VRAM 95% > 90% limit', 'device busy: HCU 88.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-o-proj-m16-3768c0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789664186.0499544, "occurrences": 6, "raw": "gpu2 site=benchmark vram=95.0 util=100.0 reasons=['VRAM 95% > 90% limit', 'device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-minimax-tp8-qkv-proj-m4096-d93662", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789652652.233801, "occurrences": 2, "raw": "gpu1 site=benchmark vram=57.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-minimax-tp8-qkv-proj-m4096-d93662", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789662656.9322333, "occurrences": 2, "raw": "gpu1 site=benchmark vram=46.0 util=75.0 reasons=['device busy: HCU 75.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test-9-17-1-aaea0725:iteration_001/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-minimax-tp8-qkv-proj-m4096-d93662", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789664843.070989, "occurrences": 2, "raw": "gpu1 site=benchmark vram=49.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-m16-wqkv-a-b0cfda", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789921391.4349692, "occurrences": 2, "raw": "gpu0 site=benchmark vram=2.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-m16-wqkv-a-b0cfda", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789925452.8073273, "occurrences": 2, "raw": "gpu0 site=profile_pmc vram=3.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-m16-wqkv-a-b0cfda", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789927662.7208147, "occurrences": 6, "raw": "gpu0 site=benchmark vram=87.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-m16-wqkv-a-b0cfda", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789931263.1924827, "occurrences": 2, "raw": "gpu0 site=benchmark vram=88.0 util=75.0 reasons=['device busy: HCU 75.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-m16-wqkv-a-b0cfda", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789934863.6198232, "occurrences": 2, "raw": "gpu0 site=benchmark vram=88.0 util=96.7 reasons=['device busy: HCU 96.7% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789921811.820668, "occurrences": 2, "raw": "gpu1 site=benchmark vram=4.0 util=34.2 reasons=['device busy: HCU 34.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789923640.9922445, "occurrences": 2, "raw": "gpu1 site=profile_pmc vram=3.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789925441.2388017, "occurrences": 2, "raw": "gpu1 site=profile_pmc vram=17.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789927241.4837081, "occurrences": 2, "raw": "gpu1 site=profile_pmc vram=2.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789929041.7206576, "occurrences": 2, "raw": "gpu1 site=profile_pmc vram=19.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789930841.9179556, "occurrences": 2, "raw": "gpu1 site=profile_pmc vram=31.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789932642.128478, "occurrences": 2, "raw": "gpu1 site=profile_pmc vram=10.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789934442.3626215, "occurrences": 2, "raw": "gpu1 site=profile_pmc vram=15.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789922178.8015707, "occurrences": 2, "raw": "gpu2 site=benchmark vram=89.0 util=59.2 reasons=['device busy: HCU 59.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789923978.9712908, "occurrences": 2, "raw": "gpu2 site=benchmark vram=90.0 util=87.5 reasons=['device busy: HCU 87.5% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789925779.1271057, "occurrences": 2, "raw": "gpu2 site=benchmark vram=90.0 util=93.3 reasons=['device busy: HCU 93.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789927579.2940683, "occurrences": 8, "raw": "gpu2 site=benchmark vram=93.0 util=100.0 reasons=['VRAM 93% > 90% limit', 'device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789929379.5381875, "occurrences": 4, "raw": "gpu2 site=benchmark vram=93.0 util=3.3 reasons=['VRAM 93% > 90% limit', 'device busy: HCU 3.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789932979.930035, "occurrences": 2, "raw": "gpu2 site=benchmark vram=93.0 util=75.0 reasons=['VRAM 93% > 90% limit', 'device busy: HCU 75.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789936580.420706, "occurrences": 2, "raw": "gpu2 site=benchmark vram=93.0 util=66.7 reasons=['VRAM 93% > 90% limit', 'device busy: HCU 66.7% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-shared-down-proj-m16-65601d", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789923152.9341302, "occurrences": 2, "raw": "gpu3 site=benchmark vram=90.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-shared-down-proj-m16-65601d", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789924953.1694722, "occurrences": 2, "raw": "gpu3 site=benchmark vram=90.0 util=59.2 reasons=['device busy: HCU 59.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-shared-down-proj-m16-65601d", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789926753.3772118, "occurrences": 2, "raw": "gpu3 site=benchmark vram=90.0 util=68.3 reasons=['device busy: HCU 68.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-shared-down-proj-m16-65601d", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789928553.6067533, "occurrences": 2, "raw": "gpu3 site=benchmark vram=94.0 util=0.0 reasons=['VRAM 94% > 90% limit']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-shared-down-proj-m16-65601d", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789930353.8432574, "occurrences": 8, "raw": "gpu3 site=benchmark vram=94.0 util=100.0 reasons=['VRAM 94% > 90% limit', 'device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-shared-down-proj-m16-65601d", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1789933954.2115555, "occurrences": 4, "raw": "gpu3 site=benchmark vram=94.0 util=3.3 reasons=['VRAM 94% > 90% limit', 'device busy: HCU 3.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp4-qkv-proj-m16-292420", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790065615.7603943, "occurrences": 2, "raw": "gpu0 site=benchmark vram=91.0 util=100.0 reasons=['VRAM 91% > 90% limit', 'device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp4-qkv-proj-m16-292420", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790067415.9736354, "occurrences": 50, "raw": "gpu0 site=benchmark vram=85.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp4-qkv-proj-m16-292420", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790096131.479265, "occurrences": 6, "raw": "gpu0 site=benchmark vram=83.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp8-shared-down-proj-m4096-931a67", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790065683.1968534, "occurrences": 2, "raw": "gpu1 site=benchmark vram=41.0 util=79.2 reasons=['device busy: HCU 79.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp8-shared-down-proj-m4096-931a67", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790069158.8612468, "occurrences": 2, "raw": "gpu1 site=benchmark vram=20.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-f5d1a2", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790132964.0250077, "occurrences": 2, "raw": "gpu0 site=benchmark vram=80.0 util=61.7 reasons=['device busy: HCU 61.7% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-qkv-proj-m4096-4a2913", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790134494.3558671, "occurrences": 2, "raw": "gpu1 site=benchmark vram=2.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-qkv-proj-m4096-4a2913", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790136294.5058239, "occurrences": 6, "raw": "gpu1 site=benchmark vram=5.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-qkv-proj-m4096-4a2913", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790143079.8871918, "occurrences": 4, "raw": "gpu1 site=benchmark vram=29.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-qkv-proj-m4096-4a2913", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790146680.3626301, "occurrences": 2, "raw": "gpu1 site=benchmark vram=4.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-shared-down-proj-m16-ae1f1c", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790130818.804856, "occurrences": 2, "raw": "gpu2 site=benchmark vram=96.0 util=56.7 reasons=['VRAM 96% > 90% limit', 'device busy: HCU 56.7% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-minimax-tp4-shared-down-proj-m4096-678a97", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790130612.3645391, "occurrences": 2, "raw": "gpu3 site=benchmark vram=51.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-minimax-tp4-shared-down-proj-m4096-678a97", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790133478.236259, "occurrences": 2, "raw": "gpu3 site=benchmark vram=44.0 util=93.3 reasons=['device busy: HCU 93.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_002/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-minimax-tp4-shared-down-proj-m4096-678a97", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790137833.028779, "occurrences": 2, "raw": "gpu3 site=benchmark vram=44.0 util=92.5 reasons=['device busy: HCU 92.5% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/glm_tp8_kv_b_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-glm-tp8-kv-b-proj-m4096-8135a8", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790151539.6900477, "occurrences": 2, "raw": "gpu0 site=benchmark vram=2.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-glm-tp8-shared-gate-up-proj-m4096-4b3831", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790152973.1399217, "occurrences": 2, "raw": "gpu1 site=benchmark vram=3.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-glm-tp8-shared-gate-up-proj-m4096-4b3831", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790160028.4667842, "occurrences": 2, "raw": "gpu1 site=benchmark vram=2.0 util=14.2 reasons=['device busy: HCU 14.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-glm-tp8-shared-gate-up-proj-m4096-4b3831", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790161885.2346773, "occurrences": 2, "raw": "gpu1 site=benchmark vram=4.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-hy3-tp4-shared-down-proj-m16-e92ad1", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790150816.2549133, "occurrences": 2, "raw": "gpu2 site=benchmark vram=99.0 util=95.0 reasons=['VRAM 99% > 90% limit', 'device busy: HCU 95.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-hy3-tp4-shared-down-proj-m16-e92ad1", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790152616.430118, "occurrences": 2, "raw": "gpu2 site=benchmark vram=99.0 util=0.0 reasons=['VRAM 99% > 90% limit']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-hy3-tp4-shared-down-proj-m16-e92ad1", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790154416.606018, "occurrences": 4, "raw": "gpu2 site=benchmark vram=99.0 util=100.0 reasons=['VRAM 99% > 90% limit', 'device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-hy3-tp4-shared-down-proj-m16-e92ad1", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790156216.7675676, "occurrences": 2, "raw": "gpu2 site=benchmark vram=96.0 util=64.2 reasons=['VRAM 96% > 90% limit', 'device busy: HCU 64.2% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-hy3-tp4-shared-down-proj-m16-e92ad1", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790163365.2212508, "occurrences": 2, "raw": "gpu2 site=benchmark vram=94.0 util=100.0 reasons=['VRAM 94% > 90% limit', 'device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-e9ad3b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790151711.4448369, "occurrences": 2, "raw": "gpu3 site=benchmark vram=100.0 util=0.0 reasons=['VRAM 100% > 90% limit']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-e9ad3b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790153511.6340034, "occurrences": 2, "raw": "gpu3 site=benchmark vram=100.0 util=68.3 reasons=['VRAM 100% > 90% limit', 'device busy: HCU 68.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-e9ad3b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790155311.857932, "occurrences": 2, "raw": "gpu3 site=benchmark vram=44.0 util=100.0 reasons=['device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-e9ad3b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790157112.098464, "occurrences": 2, "raw": "gpu3 site=benchmark vram=96.0 util=88.3 reasons=['VRAM 96% > 90% limit', 'device busy: HCU 88.3% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-e9ad3b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790161736.9121654, "occurrences": 2, "raw": "gpu3 site=benchmark vram=87.0 util=70.8 reasons=['device busy: HCU 70.8% > 0.0%']"} +{"class_id": "ENV-GATE-TIMEOUT", "task_id": "test9-22-1-fd2ebf70:iteration_003/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-e9ad3b", "task_type": "dcu-kernel-auto-opt", "source": "measurement_gate.jsonl", "event": "gate_blocked", "ts": 1790163537.159349, "occurrences": 2, "raw": "gpu3 site=benchmark vram=94.0 util=100.0 reasons=['VRAM 94% > 90% limit', 'device busy: HCU 100.0% > 0.0%']"} +{"class_id": "ENV-NO-DEVICE", "task_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "run_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1787193832.8758337, "occurrences": 1, "raw": "RuntimeError('python3 /root/zth_agent/kernel-repos/hy3-tp4-4096-sonnet-5i-1/w8a8_bench.py --source /root/zth_agent/kernel-repos/hy3-tp4-4096-sonnet-5i-1 --m 1 --n 1 --k 1 --probe failed (1): open hydmilib:libhydmi.so error, reaseon:libhydmi.so: cannot open shared object file: No such file or directory\\nTraceback (most recent call last):\\n File \"/root/zth_agent/kernel-repos/hy3-tp4-4096-sonnet-5i-1/w8a8_bench.py\", line 429, in \\n raise SystemExit(main())\\n File \"/root/zth_agent/kernel-repos/hy3-tp4-4096-sonnet-5i-1/w8a8_bench.py\", line 186, in main\\n props = torch.cuda.get_device_properties(0) if count else None\\n File \"/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py\", line 614, in get_device_properties\\n _lazy_init() # will define _get_device_properties\\n File \"/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py\", line 410, in _lazy_init\\n torch._C._cuda_init()\\nRuntimeError: No HIP GPUs are available\\n')"} +{"class_id": "ENV-SHM-SANDBOX", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "final_artifact_link_rejected", "ts": 1788962716.5823526, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-9-8-0161e718/final/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-9-8-0161e718/final/source --m 16 --n 1280 --k 4096 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-9-8-0161e718/final/cache/references failed (1): shm_open: Permission denied /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:817: UserWarning: Can't initialize amdsmi - Error code: 32 warnings.warn(f\"Can't initialize amdsmi - Error code: {e.err_code}\") \u001b[92mSuccessfully preprocessed all matching files.\u001b[0m Traceback (most recent call last): File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-9-8-0161e718/final/source/w8a8_bench.py\", line 516, in raise SystemExit(main()) File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-9-8-0161e718/final/source/w8a8_bench.py\", line 325, in main a, b, a_scale, b_scale = generate_w8a8_inputs( File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-9-8-0161e718/final/source/w8a8_bench.py\", line 126, in generate_w8a8_inputs a = torch.rand…"} +{"class_id": "ENV-SHM-SANDBOX", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1787303726.4221723, "occurrences": 1, "raw": "RuntimeError('python3 /root/zth_agent/kernel-repos/hy3_TP4_M4096_dsh-test1/w8a8_bench.py --source /root/zth_agent/kernel-repos/hy3_TP4_M4096_dsh-test1 --m 1 --n 1 --k 1 --probe failed (3): shm_open: Permission denied\\n/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:817: UserWarning: Can\\'t initialize amdsmi - Error code: 32\\n warnings.warn(f\"Can\\'t initialize amdsmi - Error code: {e.err_code}\")\\n')"} +{"class_id": "ENV-SHM-SANDBOX", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787464377.093136, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/source --m 4096 --n 2304 --k 6144 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/cache/references --prepare-reference failed (1): shm_open: Permission denied /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:817: UserWarning: Can't initialize amdsmi - Error code: 32 warnings.warn(f\"Can't initialize amdsmi - Error code: {e.err_code}\") Traceback (most recent call last): File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/source/w8a8_bench.py\", line 510, in raise SystemExit(main()) File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/source/w8a8_bench.py\", line 291, in main prepare_reference( File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/source/w8a8_bench.py\", line 16…"} +{"class_id": "ENV-SHM-SANDBOX", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787464406.7415535, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/source --m 4096 --n 2560 --k 6144 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/cache/references --prepare-reference failed (1): shm_open: Permission denied /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:817: UserWarning: Can't initialize amdsmi - Error code: 32 warnings.warn(f\"Can't initialize amdsmi - Error code: {e.err_code}\") Traceback (most recent call last): File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/source/w8a8_bench.py\", line 510, in raise SystemExit(main()) File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/source/w8a8_bench.py\", line 291, in main prepare_reference( File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/source/w8a8_bench.py\", line 16…"} +{"class_id": "ENV-SHM-SANDBOX", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787464412.996237, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/source --m 4096 --n 1536 --k 6144 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/cache/references --prepare-reference failed (1): shm_open: Permission denied /usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:817: UserWarning: Can't initialize amdsmi - Error code: 32 warnings.warn(f\"Can't initialize amdsmi - Error code: {e.err_code}\") Traceback (most recent call last): File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/source/w8a8_bench.py\", line 510, in raise SystemExit(main()) File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/source/w8a8_bench.py\", line 291, in main prepare_reference( File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/source/w8a8_bench.py\", line 16…"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "dcu-kernel-auto-opt-d4dd2dc5", "run_id": "dcu-kernel-auto-opt-d4dd2dc5", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1785201695.6959016, "occurrences": 1, "raw": "worker_0-m2_wo_b-iter1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "dcu-kernel-auto-opt-d4dd2dc5", "run_id": "dcu-kernel-auto-opt-d4dd2dc5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1785201695.6955757, "occurrences": 1, "raw": "RuntimeError('worker_0-m2_wo_b-iter1 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788117544.2801845, "occurrences": 1, "raw": "worker_1-glm_tp8_q_b_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787743868.7505403, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787743880.8255646, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788891744.3584793, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788891745.8783486, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "run_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1787197253.8809052, "occurrences": 3, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "run_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1787197432.0674915, "occurrences": 3, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "run_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1787197614.2608354, "occurrences": 3, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "run_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1787197614.2612684, "occurrences": 3, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1787327069.9157083, "occurrences": 2, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1787327073.9278178, "occurrences": 2, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1787327077.941299, "occurrences": 2, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1787327077.9417307, "occurrences": 2, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434806.1459322, "occurrences": 1, "raw": "worker_1-bootstrap-attempt1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434814.1852038, "occurrences": 1, "raw": "worker_1-bootstrap-attempt2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434828.166277, "occurrences": 1, "raw": "worker_3-bootstrap-attempt1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434846.218484, "occurrences": 1, "raw": "worker_0-bootstrap-attempt1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434852.1825209, "occurrences": 1, "raw": "worker_2-bootstrap-attempt1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434874.24182, "occurrences": 1, "raw": "worker_3-bootstrap-attempt2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434880.2798738, "occurrences": 1, "raw": "worker_0-bootstrap-attempt2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434904.3059084, "occurrences": 1, "raw": "worker_3-bootstrap-attempt3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_failed", "ts": 1788434904.3081098, "occurrences": 1, "raw": "worker_3 could not generate a valid initial HIP kernel after 3 attempts: worker_3-bootstrap-attempt3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434906.2703223, "occurrences": 1, "raw": "worker_2-bootstrap-attempt2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1788434906.3096454, "occurrences": 1, "raw": "worker_1-bootstrap-attempt3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_failed", "ts": 1788434906.312063, "occurrences": 1, "raw": "worker_1 could not generate a valid initial HIP kernel after 3 attempts: worker_1-bootstrap-attempt3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788449809.9401953, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788449814.471267, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788451695.0513856, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788451875.1487317, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788453949.4962344, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788454080.5050654, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter19 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788454159.8050416, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788456994.3901374, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788456996.4773946, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788457402.0012789, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788457414.93117, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788457493.0525641, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788457497.125203, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788457498.1910336, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788457503.1431117, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788457503.23862, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788460537.5332563, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788460539.2726083, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788460827.6515124, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-shared-down-proj-m16-d7f058", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789380937.0584648, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_down_proj_m16-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-indexer-wq-b-m16-492bc9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789387314.7273476, "occurrences": 1, "raw": "worker_0-hy3_tp8_indexer_wq_b_m16-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-wqkv-a-m16-b232b1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789402170.452895, "occurrences": 1, "raw": "worker_3-hy3_tp8_wqkv_a_m16-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791285498.7363942, "occurrences": 1, "raw": "worker_7-hy3_tp8_shared_down_proj_m8-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791288945.5606012, "occurrences": 1, "raw": "worker_7-hy3_tp8_shared_down_proj_m8-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791288946.7090688, "occurrences": 1, "raw": "worker_6-hy3_tp8_shared_gate_up_proj_m8-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791295327.7018664, "occurrences": 1, "raw": "worker_7-hy3_tp8_shared_down_proj_m16-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791299080.3396025, "occurrences": 1, "raw": "worker_7-hy3_tp8_shared_down_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791335455.1001382, "occurrences": 1, "raw": "worker_2-hy3_tp8_qkv_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-1e22df66", "run_id": "minimax-tp8-m8-1-1e22df66", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790255740.1032054, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m8-iter8-repair1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-1e22df66", "run_id": "minimax-tp8-m8-1-1e22df66", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790255741.6587148, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m8-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-1e22df66", "run_id": "minimax-tp8-m8-1-1e22df66", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790257465.8647256, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m8-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-1e22df66", "run_id": "minimax-tp8-m8-1-1e22df66", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790257534.1880488, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m8-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-1e22df66", "run_id": "minimax-tp8-m8-1-1e22df66", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790259990.3056324, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m8-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790171880.9406717, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m8-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790171969.102501, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m8-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790172154.2101953, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m8-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790172163.359883, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m8-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791392857.1629906, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m8-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791399949.7133925, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m8-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791416540.3879054, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m8-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791416543.8791106, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791427001.2363272, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791430048.693298, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791431865.138301, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787476341.9867713, "occurrences": 1, "raw": "worker_0-minimax_tp4_qkv_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787476345.706231, "occurrences": 1, "raw": "worker_1-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787912100.2974255, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787912100.3131373, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787912101.8085153, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-35600c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789116124.1646638, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789126216.6031396, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789126865.3450599, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789127211.804977, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789128103.7018797, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789129006.3096688, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789129807.905704, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130022.6434426, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130413.8075314, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130654.5493166, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134256.6240704, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134828.033398, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134976.3166182, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135876.859826, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136113.1776037, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136199.4083524, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136327.6642802, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136610.0472298, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136676.254795, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136778.4970872, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136976.8221965, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137017.0035002, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-down-proj-m4096-fee009", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789126803.7368984, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-down-proj-m4096-fee009", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789128145.654152, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-down-proj-m4096-fee009", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789128370.0062568, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-down-proj-m4096-fee009", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130268.3025239, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-down-proj-m4096-fee009", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789131144.2466197, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133020.387652, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133274.7477467, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133338.95336, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133407.1589289, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133487.3719242, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133733.733479, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134189.4741201, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135212.2878356, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136126.5945456, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136200.8119767, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136355.096028, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_gate_up_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789126085.393833, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789126137.581158, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789126321.8872213, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130081.996501, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130442.347352, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130597.0493057, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m4096-de2dc0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789131405.6452925, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m4096-de2dc0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132117.8729465, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m4096-de2dc0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133119.3136759, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m4096-de2dc0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133355.681186, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m4096-de2dc0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133554.0136793, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m4096-de2dc0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133732.318288, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m16-155b0d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789126142.8260376, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m16-155b0d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789127930.085653, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m16-155b0d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789128196.4758654, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m16-155b0d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789128943.398137, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m16-155b0d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789129808.8272886, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m16-155b0d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130510.386594, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130501.717467, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789131197.7878058, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789131914.5780723, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132172.9607866, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132213.1452377, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133041.7087731, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133244.0305467, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133416.3246539, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133464.5138087, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133530.7158618, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133664.9780953, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133889.310264, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m16-3c8a59", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789128092.2705677, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter6-repair1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m16-3c8a59", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789128867.79225, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter7-repair1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m16-3c8a59", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789128996.0104554, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m16-3c8a59", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130019.592358, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m16-3c8a59", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132370.5384023, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m16-3c8a59", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133309.8417172, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132371.3019338, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133277.1947784, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133307.3635726, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133381.5640876, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133519.8419588, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134981.100103, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135875.0951717, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136477.261913, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136621.5561275, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789130413.594728, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_gate_up_proj_m16-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132235.9739215, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_gate_up_proj_m16-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133488.9889529, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_gate_up_proj_m16-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133577.215421, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_gate_up_proj_m16-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133671.437684, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_gate_up_proj_m16-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134970.35432, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_gate_up_proj_m16-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135030.563222, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_gate_up_proj_m16-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133214.6896198, "occurrences": 1, "raw": "worker_3-hy3_tp8_o_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133527.1362016, "occurrences": 1, "raw": "worker_3-hy3_tp8_o_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134217.0530946, "occurrences": 1, "raw": "worker_3-hy3_tp8_o_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134849.7479694, "occurrences": 1, "raw": "worker_3-hy3_tp8_o_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136838.3612194, "occurrences": 1, "raw": "worker_3-hy3_tp8_o_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789132112.2636585, "occurrences": 1, "raw": "worker_2-bootstrap-attempt1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133374.697694, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133544.9955797, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133813.375423, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135452.8489373, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135874.9980521, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136319.7683105, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136426.0201366, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136544.282816, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137032.593441, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137194.9068265, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132847.7399225, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133279.2283318, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133483.548589, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133609.7991989, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133724.042218, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136313.2085636, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136443.472543, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136543.7233915, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137037.054272, "occurrences": 1, "raw": "worker_1-glm_tp8_shared_down_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137036.3772666, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137082.5481951, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140199.484962, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140345.761239, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140588.1157959, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140716.3809752, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140766.5711837, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140888.8276784, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140919.0008075, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141005.2193131, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141619.4420443, "occurrences": 1, "raw": "worker_2-hy3_tp4_o_proj_m16-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136618.8005667, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136673.0029094, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136733.18728, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140312.7766352, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140761.32245, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140943.6460233, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141049.891198, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141510.436489, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141722.7658205, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141758.9366016, "occurrences": 1, "raw": "worker_0-hy3_tp4_o_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132791.436409, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133071.8211892, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133282.1624994, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133342.35939, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133502.654565, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133558.8476515, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134215.4437752, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136531.8912077, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136670.182536, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136726.3821347, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136780.5736077, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136914.8455062, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789136686.6058302, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789136841.0864582, "occurrences": 1, "raw": "worker_0-bootstrap-attempt1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140288.7647955, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140477.1036978, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140679.4168487, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140881.7210088, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141764.9246414, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141989.2748399, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142257.6696908, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142565.2381997, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142571.3965669, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142577.5561895, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142583.7048042, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142589.852976, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135508.7238533, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136198.4118218, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136296.645482, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136484.954377, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136607.2058105, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136665.4262593, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136851.727712, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137012.0152125, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789139697.672684, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140344.1562042, "occurrences": 1, "raw": "worker_1-hy3_tp4_shared_down_proj_m16-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136127.2393036, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136183.42659, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136257.628834, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136429.8974736, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136664.263072, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137224.2988293, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789139516.0164404, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789139911.5793755, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140851.5552564, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_gate_up_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140345.2325237, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140531.5754395, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140715.8963435, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140780.104948, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140854.3199902, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141004.6121027, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141160.913714, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141339.2305274, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141465.4965193, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141982.4447327, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142530.8893194, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142537.0497925, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142543.2077606, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142549.3663824, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789132862.6976836, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133310.0447555, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133492.3457296, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789133732.7078564, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134847.3833456, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789134967.6240797, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135336.2001143, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135936.610378, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136112.9226985, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136295.2394302, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136657.295955, "occurrences": 1, "raw": "worker_3-hy3_tp8_qkv_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789136860.4026022, "occurrences": 1, "raw": "worker_3-bootstrap-attempt1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789137030.6003125, "occurrences": 1, "raw": "worker_3-bootstrap-attempt2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789138989.4225218, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140402.8800306, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140465.0885868, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140645.392166, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140761.6554859, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140825.8694863, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140986.1646545, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141110.4181185, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142110.2591157, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142206.498806, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142352.7830315, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789139970.2439091, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140371.7265072, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140469.9493318, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140552.1633894, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140718.4422526, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141162.146276, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141422.5304394, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142122.2414458, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142577.9162717, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142584.0699584, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142590.2256742, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142596.3789325, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142602.5352497, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789135452.761526, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136357.9299493, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136478.1905289, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136538.3794286, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136658.6556373, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789136854.953733, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137025.2484634, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789137217.5522501, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789139873.8531282, "occurrences": 1, "raw": "worker_3-m16_wqkv_a-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140582.3924847, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140828.7614079, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141638.2275596, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142577.356057, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142583.5097616, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142589.6608875, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142595.8052287, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142601.9501479, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142608.1010811, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142614.247488, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142620.4040334, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142626.5569, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142632.7014773, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142638.8554068, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140535.4780054, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140583.6559894, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140942.3323662, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141781.4938078, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142593.8086512, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142599.9611843, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142606.1113653, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142612.255069, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142618.4018397, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142624.6031084, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142630.7565744, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142636.9085395, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142643.0579646, "occurrences": 1, "raw": "worker_3-minimax_tp4_qkv_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140536.0047255, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142266.4563317, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142523.9454186, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142532.0867765, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142538.2359169, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142544.377373, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142552.5262952, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142564.6831844, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142570.8333704, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142576.983907, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142583.132121, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142589.2978013, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142595.4494267, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142601.5975227, "occurrences": 1, "raw": "worker_2-glm_tp8_fused_qkv_a_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140138.250505, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140286.4818344, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142525.751802, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142535.936582, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142542.0840333, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142548.2355452, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142554.386644, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142564.54074, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142570.6998773, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142576.8439014, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142582.9949396, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142589.1474905, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142595.3023243, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142601.4463642, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_down_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142691.0393429, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142697.187315, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142703.3339229, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142709.4841552, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142715.6325996, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142721.7819839, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142727.9240804, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142734.0743787, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142740.234375, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142746.3880463, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142752.5378518, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142758.6825025, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142764.8030322, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142770.9568977, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142777.1089795, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142783.2593312, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m4096-df16cf", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143070.116471, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m4096-df16cf", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143058.0835204, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m4096-df16cf", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143064.0993876, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m4096-df16cf", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143070.1154327, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m4096-df16cf", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143070.1159165, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789139698.4543514, "occurrences": 1, "raw": "worker_1-bootstrap-attempt1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140535.1084836, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140591.300174, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140659.4975152, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140827.7858293, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789140944.0414166, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141066.2943733, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789141771.755534, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142160.256072, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142230.473116, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142528.8870018, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142535.0363696, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142541.1874938, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142547.3371115, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142553.4894607, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142567.657009, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142573.8097837, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m4096-8deec7", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143775.899922, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m4096-8deec7", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143763.8640494, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m4096-8deec7", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143769.881639, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m4096-8deec7", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143775.8986032, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m4096-8deec7", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143775.8991468, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-shared-down-proj-m16-de44d4", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143071.1647563, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-shared-down-proj-m16-de44d4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143059.1312623, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-shared-down-proj-m16-de44d4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143065.1477053, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-shared-down-proj-m16-de44d4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143071.1635485, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-shared-down-proj-m16-de44d4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143071.1640658, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-o-proj-m4096-7cf773", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144402.320677, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-o-proj-m4096-7cf773", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144390.283687, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-o-proj-m4096-7cf773", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144396.2978892, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-o-proj-m4096-7cf773", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144402.3197973, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-o-proj-m4096-7cf773", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144402.320239, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-down-proj-m4096-1142d7", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143776.856988, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-down-proj-m4096-1142d7", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143764.8262472, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-down-proj-m4096-1142d7", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143770.843458, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-down-proj-m4096-1142d7", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143776.8560648, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-down-proj-m4096-1142d7", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143776.8564603, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-gate-up-proj-m4096-3bbf6a", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144445.3186848, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-gate-up-proj-m4096-3bbf6a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144433.2871168, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-gate-up-proj-m4096-3bbf6a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144439.301918, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-gate-up-proj-m4096-3bbf6a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144445.3177323, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp8-shared-gate-up-proj-m4096-3bbf6a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144445.3181398, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-o-proj-m4096-dae9bc", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789142736.7086692, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-o-proj-m4096-dae9bc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789142724.6756084, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-o-proj-m4096-dae9bc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789142730.6913228, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-o-proj-m4096-dae9bc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789142736.707674, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-o-proj-m4096-dae9bc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789142736.7080557, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-down-proj-m4096-8f8ce9", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143079.2445424, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-down-proj-m4096-8f8ce9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143067.2125285, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-down-proj-m4096-8f8ce9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143073.2267075, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-down-proj-m4096-8f8ce9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143079.2435338, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-down-proj-m4096-8f8ce9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143079.2440162, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142861.8730633, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142868.0142553, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142874.1598353, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142880.3094668, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142886.459221, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter5 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142892.606234, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter6 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142898.755718, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142904.8846042, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter8 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142911.0413823, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142917.1863875, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142923.3312259, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142929.4802186, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter12 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142935.6290624, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142941.7782636, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter14 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142947.9276443, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter15 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142954.080465, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter16 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142960.230574, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter17 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789142966.3805342, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter18 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-o-proj-m4096-c3b47a", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143784.6467981, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-o-proj-m4096-c3b47a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143772.611693, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-o-proj-m4096-c3b47a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143778.6288047, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-o-proj-m4096-c3b47a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143784.6458104, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-o-proj-m4096-c3b47a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143784.6462505, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-qkv-proj-m16-051c55", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144403.3405488, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-qkv-proj-m16-051c55", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144391.311695, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-qkv-proj-m16-051c55", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144397.3257272, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-qkv-proj-m16-051c55", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144403.3393767, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-qkv-proj-m16-051c55", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144403.3398573, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-shared-down-proj-m16-d72b62", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144445.4202209, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-shared-down-proj-m16-d72b62", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144433.386996, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-shared-down-proj-m16-d72b62", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144439.4025853, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-shared-down-proj-m16-d72b62", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144445.4192379, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp8-shared-down-proj-m16-d72b62", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144445.4196532, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143138.1307452, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143144.1476822, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143150.1597013, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143150.1601548, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-5707d0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143250.220229, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-5707d0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143256.235233, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-5707d0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143262.2515326, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-5707d0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143262.2519627, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144395.2713664, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144401.284904, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144407.3019223, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144407.3023715, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m4096-106d13", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143147.164738, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m4096-106d13", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143135.131698, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m4096-106d13", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143141.1477141, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m4096-106d13", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143147.1637213, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m4096-106d13", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143147.164182, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m16-b90d13", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143262.252632, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m16-b90d13", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143250.2195036, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m16-b90d13", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143256.2366617, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m16-b90d13", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143262.2517126, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m16-b90d13", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143262.2520952, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m4096-86c8ba", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143852.2649283, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m4096-86c8ba", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143840.232154, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m4096-86c8ba", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143846.2473624, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m4096-86c8ba", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143852.264004, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-qkv-proj-m4096-86c8ba", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143852.2644033, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-shared-down-proj-m16-6313d4", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143149.799923, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-shared-down-proj-m16-6313d4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143137.7693632, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-shared-down-proj-m16-6313d4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143143.781711, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-shared-down-proj-m16-6313d4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143149.798879, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-shared-down-proj-m16-6313d4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143149.7992887, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-o-proj-m4096-2f34e5", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144475.100367, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-o-proj-m4096-2f34e5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144463.0674553, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-o-proj-m4096-2f34e5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144469.0831985, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-o-proj-m4096-2f34e5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144475.0994093, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-o-proj-m4096-2f34e5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144475.099831, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-down-proj-m4096-09a8aa", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143853.6695633, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-down-proj-m4096-09a8aa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143841.6325, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-down-proj-m4096-09a8aa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143847.649006, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-down-proj-m4096-09a8aa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143853.6685052, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-down-proj-m4096-09a8aa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143853.6689482, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-gate-up-proj-m4096-e7eb49", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144513.8213315, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-gate-up-proj-m4096-e7eb49", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144501.7874727, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-gate-up-proj-m4096-e7eb49", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144507.8033807, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-gate-up-proj-m4096-e7eb49", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144513.820399, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp8-shared-gate-up-proj-m4096-e7eb49", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144513.8208058, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-o-proj-m16-8040ad", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143205.0992317, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-o-proj-m16-8040ad", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143193.0668302, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-o-proj-m16-8040ad", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143199.0819697, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-o-proj-m16-8040ad", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143205.0982518, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-o-proj-m16-8040ad", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143205.0986745, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m16-6f97a5", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143270.0035028, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m16-6f97a5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143257.971305, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m16-6f97a5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143263.9870262, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m16-6f97a5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143270.0025964, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m16-6f97a5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143270.0029752, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-eadf77", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789143861.1183836, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-eadf77", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143849.0771568, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-eadf77", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143855.0981266, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-eadf77", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789143861.1172347, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-eadf77", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143861.1177242, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-m4096-bd3658", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144407.142309, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-m4096-bd3658", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144395.1083255, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-m4096-bd3658", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144401.1249502, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-m4096-bd3658", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144407.1412544, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-qkv-proj-m4096-bd3658", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144407.1417232, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-down-proj-m4096-4a2288", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144474.7085657, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-down-proj-m4096-4a2288", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144462.6750095, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-down-proj-m4096-4a2288", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144468.6912355, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-down-proj-m4096-4a2288", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144474.7074502, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-down-proj-m4096-4a2288", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144474.707953, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-gate-up-proj-m16-3a2110", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144513.9443564, "occurrences": 1, "raw": "Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-gate-up-proj-m16-3a2110", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144501.9111943, "occurrences": 1, "raw": "Generate agent attempt 1 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-gate-up-proj-m16-3a2110", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144507.9272459, "occurrences": 1, "raw": "Generate agent attempt 2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-gate-up-proj-m16-3a2110", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "generate_agent_failed", "ts": 1789144513.9433699, "occurrences": 1, "raw": "Generate agent attempt 3 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-11-1-b01955b3:iteration_005/minimax_tp8_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-minimax-tp8-shared-gate-up-proj-m16-3a2110", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144513.943832, "occurrences": 1, "raw": "RuntimeError('Kernel coordination failed after 3 attempts. Last failure: Generate agent attempt 3 failed: nonzero exit 1')"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-qkv-proj-m16-d5c6e1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789648769.457714, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m16-iter7 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789920795.0147812, "occurrences": 1, "raw": "worker_1-minimax_tp4_shared_down_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-20-1-3f9b399d:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it002-candidate-glm-tp8-o-proj-m4096-987747", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789955652.597964, "occurrences": 1, "raw": "worker_0-glm_tp8_o_proj_m4096-iter11 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-20-1-3f9b399d:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it002-candidate-hy3-tp4-o-proj-m16-18694f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789954943.850719, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter13 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-20-1-3f9b399d:iteration_002/minimax_tp4_o_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it002-candidate-minimax-tp4-o-proj-m4096-208840", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789954220.1637335, "occurrences": 1, "raw": "worker_2-minimax_tp4_o_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-20-1-3f9b399d:iteration_003_generalization/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it003-candidate-minimax-tp8-qkv-proj-m16-74da05", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789961575.8508637, "occurrences": 1, "raw": "worker_2-minimax_tp8_qkv_proj_m16-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp8-shared-gate-up-proj-m4096-4fd75d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790070903.4124646, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter9 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-22-1-fd2ebf70:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-f5d1a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790127118.1975648, "occurrences": 1, "raw": "worker_0-glm_tp8_shared_gate_up_proj_m4096-iter4 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-qkv-proj-m4096-4a2913", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790127121.768069, "occurrences": 1, "raw": "worker_1-hy3_tp4_qkv_proj_m4096-iter2 failed: nonzero exit 1"} +{"class_id": "GEN-AGENT-EJECT", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-shared-down-proj-m16-ae1f1c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1790129440.3465292, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_down_proj_m16-iter10 failed: nonzero exit 1"} +{"class_id": "GEN-BINARY-MISSING", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785746459.4342682, "occurrences": 3, "raw": "[Errno 2] No such file or directory: 'ccb'"} +{"class_id": "GEN-BINARY-MISSING", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_failed", "ts": 1785746459.4574304, "occurrences": 1, "raw": "worker_3 could not generate a valid initial HIP kernel after 3 attempts: [Errno 2] No such file or directory: 'ccb'"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789395111.8655083, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-074ec9"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789393291.517327, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-d8083a')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789395111.864867, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-074ec9')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789460783.2225661, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-51494f"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789455779.7075458, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-ba4353')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789455807.7579358, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-3ef513')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460753.9926379, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-06e726')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460767.953296, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-0cbc01')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460783.2219152, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-51494f')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789455793.8598313, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-f9ba46"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789455765.0118508, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-17e72e')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789455793.859137, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-f9ba46')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789460789.9003277, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-fbf49d"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460754.0102558, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-864f57')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460774.6636713, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-eb75b2')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460789.8996537, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-fbf49d')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789460789.8254557, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-01835c"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460754.0242007, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-0e07a1')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460774.571526, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-02678b')"} +{"class_id": "GEN-CONFLICT", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789460789.8248065, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-01835c')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:success", "ts": 1789131717.3669229, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-2dcad7"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789122844.474622, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-9d8811')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789125546.7245278, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-f25bfc')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789128248.7487655, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-69c264')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789130951.6339197, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-2dcad7')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789130951.5913553, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-603a05"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789122844.4092977, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-477fb8')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789125546.6804268, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-3dac29')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789128248.854355, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-170879')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789130951.5906768, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-603a05')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:success", "ts": 1789139216.2431095, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-a6cf3c"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789134554.8520384, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-ba2d22')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789135037.1967475, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-a8ce85')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789136628.7427723, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-d824c9')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789138577.634055, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-b1edb3')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789138711.2486792, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-a6cf3c')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:success", "ts": 1789137277.0482376, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-81142c"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789134554.9514802, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-81142c')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789138577.5267756, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-b0a3ce')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789138711.3849065, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-dc9c8d')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:success", "ts": 1789137472.9037235, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-e49582"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789134554.8434358, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-e49582')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144392.7220201, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-9cdfb7"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789141523.996918, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-069698')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143019.0476305, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-21ed08')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143724.946864, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-5a2fe4')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144349.8043277, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-d0e46c')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144392.721398, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-9cdfb7')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:success", "ts": 1789142731.9793756, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-d73007"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789141524.0575688, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-d73007')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144392.9119196, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-f861f2"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144349.7367105, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-46c162')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144392.9111109, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-f861f2')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144462.6508598, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-59383d"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143210.785811, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-b06b9a')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789143801.7997603, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-51a3fb')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144355.8786438, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-8b3477')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144423.2291555, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-a8f93e')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144462.6502378, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-d51c96, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-fused-qkv-a-proj-m4096-59383d')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-5707d0", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144355.75875, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-5707d0, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-2603c9"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-5707d0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144355.758103, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-5707d0, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-glm-tp8-shared-down-proj-m4096-2603c9')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789144462.6692717, "occurrences": 1, "raw": "workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-3223ef"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144423.3750432, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-a9fed2')"} +{"class_id": "GEN-CONFLICT", "task_id": "test-9-11-1-b01955b3:iteration_005/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789144462.668655, "occurrences": 1, "raw": "RuntimeError('workspace main already points to /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-06589e, not requested kernel repo /root/zth_agent/ahe-kernel-repos/ahe-test-9-11-1-b01955b3-it005-candidate-hy3-tp4-o-proj-m16-3223ef')"} +{"class_id": "GEN-COORD-FAILED", "task_id": "dcu-kernel-auto-opt-7eafe2cc", "run_id": "dcu-kernel-auto-opt-7eafe2cc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1785238191.4370708, "occurrences": 1, "raw": "RuntimeError(\"Kernel coordination failed after 3 attempts. Last failure: Invalid gpu_assignment: logical operator family 'wqkv_a' is split between worker_0 and worker_2; keep all M variants on one worker\")"} +{"class_id": "GEN-NO-KERNEL", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_failed", "ts": 1785744641.8384776, "occurrences": 1, "raw": "worker_3 could not generate a valid initial HIP kernel after 3 attempts: python3 /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/source/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/source --m 3072 --n 1024 --k 4096 --reference-cache-dir /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/cache/references failed (1): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ /opt/dtk/include/du_mma.h:211:77: error: no template named 'DUFragmentBase'; did you mean 'DUFragment'? template<> class DUFragment : public DUFragmentBase {}; ^ /opt/dtk/include/du_mma.h:184:85: note: 'DUFragment' declared here template class DUFragment; ^ /opt/dtk/include/du_mma.h:211:77: error: too few template arguments for class template 'DUFragment' template<> class DUFragment : public DUFragmentBase {}; ^ /opt/dtk/include/du_mma.h:184:85: note: template is declared here template(m) * N + n] = __float2bfloat16(result); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_0/source/csrc/w8a8_gemm_hip.hip:97:26: error: unknown type name '__hip_bfloat16'; did you mean 'hip_bfloat16'? reinterpret_cast<__hip_bfloat16*>(out), ^~~~~~~~~~~~~~ hip_bfloat16 /opt/dtk/hip/include/hip/amd_detail/amd_hip_runtime.h:249:87: note: expanded from macro 'hipLaunchKernelGGL' #define hipLaunchKernelGGL(kernelName, ...) hipLaunchKernelGGLInternal((kernelName), __VA_ARGS__) ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_runtime.h:246:78: note: expanded from macro 'hipLaunchKernelGGLInternal' kernelName<<<(numBlocks), (numThreads), (memPerBlock), (streamId)>>>(__VA_ARGS__); \\ ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bfloat16.h:61:8: note: 'hip_bfl…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-602ee1ee", "run_id": "dcu-kernel-auto-opt-602ee1ee", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785213063.3764582, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_1/source --m 16 --n 1536 --k 4096 failed (1): de -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THH -isystem /opt/dtk/include -isystem /usr/include/python3.10 -fPIC -std=c++17 -O3 -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_1/source/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_1/source/csrc/w8a8_gemm_hip.hip:25:5: error: unknown type name '__hip_bfloat16' __hip_bfloat16* __restrict__ out, ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_1/source/csrc/w8a8_gemm_hip.hip:55:14: error: use o…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-602ee1ee", "run_id": "dcu-kernel-auto-opt-602ee1ee", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785213202.527374, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_1/source --m 16 --n 1536 --k 4096 failed (1): FORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_1/source/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_1/source/csrc/w8a8_gemm_hip.hip:28:5: error: unknown type name 'hip_bfloat16'; did you mean '__hip_bfloat16'? hip_bfloat16* __restrict__ out, ^~~~~~~~~~~~ __hip_bfloat16 /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:192:36: note: '__hip_bfloat16' declared here struct __attribute__((aligned(2))) __hip_bfloat16 { ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-602ee1ee/workers/worker_1/source/csrc/w8a8_gemm_hip.hip:90:19: error: unknown type name 'hip_bfloat16'; did you mean '_…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785743946.5152814, "occurrences": 2, "raw": "python3 /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/source/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/source --m 3072 --n 1024 --k 4096 --reference-cache-dir /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/cache/references failed (1): ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ /opt/dtk/include/du_mma.h:211:77: error: no template named 'DUFragmentBase'; did you mean 'DUFragment'? template<> class DUFragment : public DUFragmentBase {}; ^ /opt/dtk/include/du_mma.h:184:85: note: 'DUFragment' declared here template class DUFragment; ^ /opt/dtk/include/du_mma.h:211:77: error: too few template arguments for class template 'DUFragment' template<> class DUFragment : public DUFragmentBase {}; ^ /opt/dtk/include/du_mma.h:184:85: note: template is declared here template class DUFragmen…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785744385.288713, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/source/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/source --m 3072 --n 1024 --k 4096 --reference-cache-dir /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/cache/references failed (1): dtk/bin/hipcc -DWITH_HIP -DTORCH_EXTENSION_NAME=metainfer_w8a8_backend_622c1119987069ae11e9ee4b -DTORCH_API_INCLUDE_EXTENSION_H -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THH -isystem /opt/dtk/include -isystem /usr/include/python3.10 -fPIC -std=c++17 -O3 -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-7892abc2/workers/worker_3/cache/compile_sources/622c1119987069ae11e9ee4…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785218590.4607534, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source --m 16 --n 1536 --k 4096 failed (1): ude/du_mma.hpp:138:29: note: 'du_load_matrix_sync' declared here __DU_MMA_DEVICE_DECL__ void du_load_matrix_sync(DUFragment &a, const float *p, unsigned ldm) ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:83:16: error: no member named 'load_matrix_sync' in namespace 'du::dumma' du::dumma::load_matrix_sync(b_frag, &Bs[0][0], MmaK); ~~~~~~~~~~~^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:83:33: error: use of undeclared identifier 'b_frag' du::dumma::load_matrix_sync(b_frag, &Bs[0][0], MmaK); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:86:16: error: no member named 'mma_sync' in namespace 'du::dumma' du::dumma::mma_sync(c_frag, a_frag, b_frag, c_frag); ~~~~~~~~~~~^ /workspace/MetaInf…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785218706.1567364, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_3/source --m 16 --n 8192 --k 1024 failed (1): /dtk/include/du_mma.h:184:85: note: 'DUFragment' declared here template class DUFragment; ^ /opt/dtk/include/du_mma.h:210:77: error: too few template arguments for class template 'DUFragment' template<> class DUFragment : public DUFragmentBase {}; ^ /opt/dtk/include/du_mma.h:184:85: note: template is declared here template class DUFragment; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ /opt/dtk/include/du_mma.h:211:77: error: no template named 'DUFragmentBase'; did you mean 'DUFragment'? template<> class DUFragment : public DUFragmentBase {}; ^ /opt/dtk/include/du_mma.h:184:85: note: 'DUFragment' declared here template c…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785218901.9418402, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source --m 16 --n 1536 --k 4096 failed (1): rag' declared here A_Frag a_frag; ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:57:20: error: use of undeclared identifier 'c_frag' du_fill_fragment(c_frag, 0); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:88:25: error: use of undeclared identifier 'b_frag' du_load_matrix_sync(b_frag, &Bs[0][0], MmaK); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:91:17: error: use of undeclared identifier 'c_frag' du_mma_sync(c_frag, a_frag, b_frag, c_frag); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:91:33: error: use of undeclared identifier 'b_frag' du_mma_sync(c_frag, a_frag, b_frag, c_frag); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785219383.6413934, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source --m 16 --n 1536 --k 4096 failed (1): e A_Frag a_frag; ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:63:20: error: use of undeclared identifier 'c_frag' du_fill_fragment(c_frag, 0); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:91:25: error: use of undeclared identifier 'b_frag' du_load_matrix_sync(b_frag, &Bs[0][0], MmaK); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:92:17: error: use of undeclared identifier 'c_frag' du_mma_sync(c_frag, a_frag, b_frag, c_frag); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:92:33: error: use of undeclared identifier 'b_frag' du_mma_sync(c_frag, a_frag, b_frag, c_frag); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785220294.5847678, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_3/source --m 16 --n 8192 --k 1024 failed (1): =1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-8c1f2eac/workers/worker_3/source/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o FAILED: w8a8_gemm_hip.cuda.o /opt/dtk/bin/hipcc -DWITH_HIP -DTORCH_EXTENSION_NAME=metainfer_w8a8_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THH -isystem /opt/dtk/include -isystem /usr/include/python3.10 -fPIC -std=c++17 -O3 -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-a473ffe1", "run_id": "dcu-kernel-auto-opt-a473ffe1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785208958.3074243, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-a473ffe1/workers/worker_0/source --m 2 --n 1536 --k 4096 failed (1): md_hip_bf16.h:192:36: note: '__hip_bfloat16' declared here struct __attribute__((aligned(2))) __hip_bfloat16 { ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-a473ffe1/workers/worker_0/source/csrc/w8a8_gemm_hip.hip:122:39: error: unknown type name 'hip_bfloat16'; did you mean '__hip_bfloat16'? reinterpret_cast(out), m, n, k); ^~~~~~~~~~~~ __hip_bfloat16 /opt/dtk/hip/include/hip/amd_detail/amd_hip_runtime.h:249:87: note: expanded from macro 'hipLaunchKernelGGL' #define hipLaunchKernelGGL(kernelName, ...) hipLaunchKernelGGLInternal((kernelName), __VA_ARGS__) ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_runtime.h:246:78: note: expanded from macro 'hipLaunchKernelGGLInternal' kernelName<<<(numBlocks), (numThreads), (memPerBlock), (streamId)>>>(__VA_ARGS__); \\ ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:192:36: note: '__hip_bfloat16' declared here struct __attribute__((aligned(2))) __hip_bfloat16 { ^ 2 errors gene…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-a473ffe1", "run_id": "dcu-kernel-auto-opt-a473ffe1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785209268.7212665, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-a473ffe1/workers/worker_3/source --m 16 --n 8192 --k 1024 failed (1): e Use, int m, int n, int k, typename T, typename Layout=void> class DUFragment; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^ /opt/dtk/include/du_mma.h:211:77: error: no template named 'DUFragmentBase'; did you mean 'DUFragment'? template<> class DUFragment : public DUFragmentBase {}; ^ /opt/dtk/include/du_mma.h:184:85: note: 'DUFragment' declared here template class DUFragment; ^ /opt/dtk/include/du_mma.h:211:77: error: too few template arguments for class template 'DUFragment' template<> class DUFragment : public DUFragmentBase {}; ^ /opt/dtk/include/du_mma.h:184:85: note: template is declared here template class DUFragment; ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-a473ffe1", "run_id": "dcu-kernel-auto-opt-a473ffe1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785209913.3181977, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-a473ffe1/workers/worker_2/source --m 16 --n 1536 --k 4096 failed (1): ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-a473ffe1/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:79:9: error: use of undeclared identifier 'dumma'; did you mean 'du::dumma'? dumma::du_mma_sync(c_frag, a_frag, b_frag, c_frag); ^~~~~ du::dumma /opt/dtk/include/du_mma.hpp:135:11: note: 'du::dumma' declared here namespace dumma ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-a473ffe1/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:87:5: error: use of undeclared identifier 'dumma'; did you mean 'du::dumma'? dumma::du_store_matrix_sync(lds_out, c_frag, TILE_N, dumma::mem_row_major); ^~~~~ du::dumma /opt/dtk/include/du_mma.hpp:135:11: note: 'du::dumma' declared here namespace dumma ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-a473ffe1/workers/worker_2/source/csrc/w8a8_gemm_hip.hip:87:58: error: use of undeclared identifier 'dumma'; did you mean 'du::dumma'? dumma::du_store_matrix_sync(lds_…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-aafa5d91", "run_id": "dcu-kernel-auto-opt-aafa5d91", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1785740576.785088, "occurrences": 1, "raw": "Command '['bash', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_2/source/profile_pmc.sh', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_2/source/w8a8_bench.py', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_2/source', '3072', '4096', '2048', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_2/profiles/tp4_wo_b_m3072/iteration1']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-aafa5d91", "run_id": "dcu-kernel-auto-opt-aafa5d91", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1785740618.0707195, "occurrences": 1, "raw": "Command '['bash', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_0/source/profile_pmc.sh', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_0/source/w8a8_bench.py', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_0/source', '3072', '1536', '4096', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_0/profiles/tp4_wqkv_a_m3072/iteration1']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-aafa5d91", "run_id": "dcu-kernel-auto-opt-aafa5d91", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1785742421.153553, "occurrences": 1, "raw": "Command '['bash', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_0/source/profile_pmc.sh', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_0/source/w8a8_bench.py', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_0/source', '3072', '1536', '4096', '/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-aafa5d91/workers/worker_0/profiles/tp4_wqkv_a_m3072/iteration2']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "final_artifact_link_rejected", "ts": 1785727334.8582094, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source --m 3072 --n 1536 --k 4096 --warmups 2 --samples 3 failed (1): \u001b[92mSuccessfully preprocessed all matching files.\u001b[0m Traceback (most recent call last): File \"/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/w8a8_bench.py\", line 343, in raise SystemExit(main()) File \"/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/w8a8_bench.py\", line 204, in main workspace = api.allocate_workspace( File \"/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/int8_w8a8_gemm_api.py\", line 201, in allocate_workspace _check_target_shape(m, n, k) File \"/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/int8_w8a8_gemm_api.py\", line 110, in _check_target_shape raise ValueError(f\"M must be in [{MIN_M}, {MAX_M}], got {m}\") ValueError: M must be in [1, 16], got 3072"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1785727334.8584578, "occurrences": 1, "raw": "RuntimeError('Final prebuilt-object W8A8 validation failed: python3 /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source --m 3072 --n 1536 --k 4096 --warmups 2 --samples 3 failed (1): \\x1b[92mSuccessfully preprocessed all matching files.\\x1b[0m\\nTraceback (most recent call last):\\n File \"/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/w8a8_bench.py\", line 343, in \\n raise SystemExit(main())\\n File \"/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/w8a8_bench.py\", line 204, in main\\n workspace = api.allocate_workspace(\\n File \"/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/int8_w8a8_gemm_api.py\", line 201, in allocate_workspace\\n _check_target_shape(m, n, k)\\n File \"/workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-c9d1c1dc/final/source/int8_w8a8_gemm_api.py\", line 110, in _check_target_shape\\n raise ValueError(f\"M must be in [{MIN_M}, {MAX_M}], got {m}\")\\nValueError: M must be in [1, 16], go…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785395185.522457, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eab76420/workers/worker_1/source/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eab76420/workers/worker_1/source --m 16 --n 8192 --k 1024 failed (1): -O3 --offload-arch=gfx928 -c /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eab76420/workers/worker_1/source/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o FAILED: w8a8_gemm_hip.cuda.o /opt/dtk/bin/hipcc -DWITH_HIP -DTORCH_EXTENSION_NAME=metainfer_w8a8_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THH -isystem /opt/dtk/include -isystem /usr/include/python3.10 -fPIC -std=c++17 -O3 -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eab76420/workers/wor…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785648705.7918997, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eba3825d/workers/worker_2/source/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eba3825d/workers/worker_2/source --m 16 --n 4096 --k 2048 failed (1): __=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eba3825d/workers/worker_2/source/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o FAILED: w8a8_gemm_hip.cuda.o /opt/dtk/bin/hipcc -DWITH_HIP -DTORCH_EXTENSION_NAME=metainfer_w8a8_backend -DTORCH_API_INCLUDE_EXTENSION_H -isystem /usr/local/lib/python3.10/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THH -isystem /opt/dtk/include -isystem /usr/include/python3.10 -fPIC -std=c++17 -O3 -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /workspace/Me…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785648766.9272888, "occurrences": 1, "raw": "python3 /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eba3825d/workers/worker_0/source/w8a8_bench.py --source /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eba3825d/workers/worker_0/source --m 16 --n 1536 --k 4096 failed (1): -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THH -isystem /opt/dtk/include -isystem /usr/include/python3.10 -fPIC -std=c++17 -O3 -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eba3825d/workers/worker_0/source/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eba3825d/workers/worker_0/source/csrc/w8a8_gemm_hip.hip:49:14: error: use of undeclared identifier '__float2bfloat16' out[tid] = __float2bfloat16(f); ^ /workspace/MetaInfer/nodes/worker29/workspaces/dcu-kernel-auto-opt-eba3825d/workers/worker_0/source/csrc/w8a8_gemm_hip.hip:116:3: warning: ignoring return value of function declared w…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787724591.5258403, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-1-adf021ab/workers/worker_2/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-1-adf021ab/workers/worker_2/source --m 16 --n 4096 --k 1024 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-1-adf021ab/workers/worker_2/cache/references failed (1): rch/include/THH -isystem /opt/dtk/include -isystem /usr/include/python3.10 -fPIC -std=c++17 -O3 -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-1-adf021ab/workers/worker_2/cache/compile_sources/acb046c8eb5b28c563b2b78a0bff423a3ab1d90d849b1b689e001f90923d0747/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-dsh-tp8-m16-1-adf021ab/workers/worker_2/cache/compile_sources/acb046c8eb5b28c563b2b78a0bff423a3ab1d90d849b1b689e001f90923d0747/csrc/w8a8_gemm_hip.hip:77:16: error: use of un…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "run_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787201239.7632082, "occurrences": 3, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-4096-sonnet-5i-1-cbcd099f/workers/worker_0/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-4096-sonnet-5i-1-cbcd099f/workers/worker_0/source', '--m', '4096', '--n', '2560', '--k', '4096', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-4096-sonnet-5i-1-cbcd099f/workers/worker_0/cache/references']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787328170.67995, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/workers/worker_2/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/workers/worker_2/source --m 4096 --n 768 --k 4096 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/workers/worker_2/cache/references failed (1): d06b20afe103bbb2d622c90ef95223b04fc5e55a23e/csrc/w8a8_gemm_hip.hip:76:26: error: use of undeclared identifier '__float2hip_bfloat16'; did you mean '__float2bfloat16'? out[row * n + col] = __float2hip_bfloat16(scaled); ^~~~~~~~~~~~~~~~~~~~ __float2bfloat16 /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:532:44: note: '__float2bfloat16' declared here __BF16_HOST_DEVICE_STATIC__ __hip_bfloat16 __float2bfloat16(float f) { ^ /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/workers/worker_2/cache/compile_sources/14841ca5f0c1148de6566d06b20afe103bbb2d622c90ef95223b04fc5e55a23e/csrc/w8a8_gemm_hip.hip:220:24: error: use of undeclared identifier '__float2hip_bfloat16'; did you mean '__float2bfloat16'? out[row * n + col] = __float2hi…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787329565.0325031, "occurrences": 2, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/workers/worker_0/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/workers/worker_0/source', '--m', '4096', '--n', '2560', '--k', '4096', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/workers/worker_0/cache/references']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "final_artifact_link_rejected", "ts": 1787413599.0702448, "occurrences": 1, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/final/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/final/source', '--m', '4096', '--n', '2560', '--k', '4096', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp4-m4096-dsh-test1-bfdf3002/final/cache/references']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-wqkv-a-m16-d9c065", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789374721.7592416, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-ahe-v2-ea5b8069/children/iteration_001/hy3_tp8_wqkv_a_m16/workspace/workers/worker_3/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-ahe-v2-ea5b8069/children/iteration_001/hy3_tp8_wqkv_a_m16/workspace/workers/worker_3/source --m 16 --n 1536 --k 4096 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-ahe-v2-ea5b8069/children/iteration_001/hy3_tp8_wqkv_a_m16/workspace/workers/worker_3/cache/references failed (1): 4: note: candidate function __BF16_HOST_DEVICE__ operator long long() const { ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:368:24: note: candidate function __BF16_HOST_DEVICE__ operator short() const { return static_cast(bfloatraw_2_float(__x)); } ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:371:24: note: candidate function __BF16_HOST_DEVICE__ operator signed char() const { ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:376:24: note: candidate function __BF16_HOST_DEVICE__ operator unsigned char() const { ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:381:24: note: candidate function __BF…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-tp8-m16-test9-16-619177f8", "run_id": "hy3-tp8-m16-test9-16-619177f8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789556085.4525898, "occurrences": 1, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_0/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_0/source', '--m', '16', '--n', '1280', '--k', '4096', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_0/cache/references']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-tp8-m16-test9-16-619177f8", "run_id": "hy3-tp8-m16-test9-16-619177f8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789556226.1878679, "occurrences": 1, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_1/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_1/source', '--m', '2', '--n', '4096', '--k', '1024', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_1/cache/references']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "hy3-tp8-m16-test9-16-619177f8", "run_id": "hy3-tp8-m16-test9-16-619177f8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789577315.825007, "occurrences": 1, "raw": "Command '['bash', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_3/source/profile_pmc.sh', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_3/source/w8a8_bench.py', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_3/source', '16', '4096', '192', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/hy3-tp8-m16-test9-16-619177f8/workers/worker_3/profiles/hy3_tp8_shared_down_proj_m16/iteration8']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1790166454.15984, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_1/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_1/source --m 8 --n 1536 --k 6144 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_1/cache/references failed (1): pt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:363:24: note: candidate function __BF16_HOST_DEVICE__ operator long long() const { ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:368:24: note: candidate function __BF16_HOST_DEVICE__ operator short() const { return static_cast(bfloatraw_2_float(__x)); } ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:371:24: note: candidate function __BF16_HOST_DEVICE__ operator signed char() const { ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:376:24: note: candidate function __BF16_HOST_DEVICE__ operator unsigned char() const { ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:381:24: note: candidate function __BF16_HOST_DEVICE__ operator unsigned int() const { ^ /opt/dtk/hip/include/hip/amd_detail/amd_hip_bf16.h:386:24:…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1790169643.2626975, "occurrences": 1, "raw": "Command '['bash', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_1/source/profile_pmc.sh', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_1/source/w8a8_bench.py', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_1/source', '8', '1536', '6144', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_1/profiles/minimax_tp8_qkv_proj_and_indexer_qk_m8/iteration5']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1790171194.736767, "occurrences": 1, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_3/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_3/source', '--m', '8', '--n', '768', '--k', '6144', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimax-tp8-m8-1-e4e50cf8/workers/worker_3/cache/references']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:success", "ts": 1787481929.658918, "occurrences": 1, "raw": "Final prebuilt-object W8A8 validation failed: python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source --m 4096 --n 6144 --k 2048 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/cache/references failed (1): \u001b[92mSuccessfully preprocessed all matching files.\u001b[0m Traceback (most recent call last): File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 510, in raise SystemExit(main()) File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 402, in main save_reference_cache(reference, reference_path) File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 155, in save_reference_cache torch.save(reference.to(\"cpu\"), temporary) File \"/usr/local/lib/python3.10/dist-packages/torch/serialization.py\", line 966, in save with _open_zipfi…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787457218.1549916, "occurrences": 3, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/source', '--m', '4096', '--n', '2560', '--k', '6144', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_1/cache/references']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787457230.1347277, "occurrences": 3, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/source', '--m', '4096', '--n', '1536', '--k', '6144', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_3/cache/references']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1787457250.1754174, "occurrences": 3, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/source', '--m', '4096', '--n', '2304', '--k', '6144', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/workers/worker_0/cache/references']' timed out after 900 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "final_artifact_link_rejected", "ts": 1787475617.877153, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source --m 4096 --n 6144 --k 2048 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/cache/references failed (1): \u001b[92mSuccessfully preprocessed all matching files.\u001b[0m Traceback (most recent call last): File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 510, in raise SystemExit(main()) File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 402, in main save_reference_cache(reference, reference_path) File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 155, in save_reference_cache torch.save(reference.to(\"cpu\"), temporary) File \"/usr/local/lib/python3.10/dist-packages/torch/serialization.py\", line 966, in save with _open_zipfile_writer(f) as opened_zipfile: File \"/usr/loc…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1787475617.8776927, "occurrences": 1, "raw": "RuntimeError('Final prebuilt-object W8A8 validation failed: python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source --m 4096 --n 6144 --k 2048 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/cache/references failed (1): \\x1b[92mSuccessfully preprocessed all matching files.\\x1b[0m\\nTraceback (most recent call last):\\n File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 510, in \\n raise SystemExit(main())\\n File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 402, in main\\n save_reference_cache(reference, reference_path)\\n File \"/root/zth_agent/MetaInfer/nodes/worker29/workspaces/minimaxm3-dsh-tp4-m4096-1-0c2f84a9/final/source/w8a8_bench.py\", line 155, in save_reference_cache\\n torch.save(reference.to(\"cpu\"), temporary)\\n File \"/usr/local/lib/python3.10/dist-packages/torch/serialization.py\",…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-11-1-b01955b3:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-35600c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789107252.5269945, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_0/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_0/source --m 4096 --n 2624 --k 6144 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_0/cache/references failed (1): 9/workspaces/test-9-11-1-b01955b3/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_0/cache/compile_sources/51c7c5d4ac8871969103ae9049fa642182bda915cfaca0efc8bceec5084d9bd6/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_0/cache/compile_sources/51c7c5d4ac8871969103ae9049fa642182bda915cfaca0efc8bceec5084d9bd6/csrc/w8a8_gemm_hip.hip:179:7: error: no matching function for call to 'pack_bf16x4' pack_bf16x4(__float2bfloat16(v0), __floa…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789122755.2695506, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_002/hy3_tp4_o_proj_m16/workspace/workers/worker_1/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_002/hy3_tp4_o_proj_m16/workspace/workers/worker_1/source --m 16 --n 4096 --k 2048 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_002/hy3_tp4_o_proj_m16/workspace/workers/worker_1/cache/references failed (1): 0/dist-packages/torch/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.10/dist-packages/torch/include/THH -isystem /opt/dtk/include -isystem /usr/include/python3.10 -fPIC -std=c++17 -O3 -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fPIC -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_002/hy3_tp4_o_proj_m16/workspace/workers/worker_1/cache/compil…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1789135815.9955294, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_003/hy3_tp4_o_proj_m4096/workspace/workers/worker_0/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_003/hy3_tp4_o_proj_m4096/workspace/workers/worker_0/source --m 4096 --n 4096 --k 2048 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_003/hy3_tp4_o_proj_m4096/workspace/workers/worker_0/cache/references failed (1): RSIONS__=1 -DHIP_ENABLE_WARP_SYNC_BUILTINS=1 --offload-arch=gfx928 -fno-gpu-rdc -O3 --offload-arch=gfx928 -c /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_003/hy3_tp4_o_proj_m4096/workspace/workers/worker_0/cache/compile_sources/570a65e4f5a3d0f706370fd3cea9f60a4a84cfd26b7f3c3f352acd549de46f87/csrc/w8a8_gemm_hip.hip -o w8a8_gemm_hip.cuda.o /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-11-1-b01955b3/children/iteration_003/hy3_tp4_o_proj_m4096/workspace/workers/worker_0/cache/compile_sources/570a65e4f5a3d0f706370fd3cea9f60a4a84cfd26b7f3c3f352acd549de46f87/csrc/w8a8_gemm_hip.hip:205…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-17-1-aaea0725:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-2a6130", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789670989.0867653, "occurrences": 1, "raw": "Command '['bash', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_3/source/profile_pmc.sh', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_3/source/w8a8_bench.py', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_3/source', '4096', '2624', '6144', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/glm_tp8_fused_qkv_a_proj_m4096/workspace/workers/worker_3/profiles/glm_tp8_fused_qkv_a_proj_m4096/iteration2']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-qkv-proj-m16-d5c6e1", "task_type": "dcu-kernel-auto-opt", "source": "run.json", "event": "note:stopped", "ts": 1789651743.234284, "occurrences": 1, "raw": "Final prebuilt-object W8A8 validation failed: Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/source', '--m', '16', '--n', '2560', '--k', '4096', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/cache/references', '--warmups', '30', '--samples', '30', '--replays-per-sample', '50']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-qkv-proj-m16-d5c6e1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "final_artifact_link_rejected", "ts": 1789651743.2331605, "occurrences": 1, "raw": "Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/source', '--m', '16', '--n', '2560', '--k', '4096', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/cache/references', '--warmups', '30', '--samples', '30', '--replays-per-sample', '50']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-qkv-proj-m16-d5c6e1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1789651743.2336323, "occurrences": 1, "raw": "RuntimeError(\"Final prebuilt-object W8A8 validation failed: Command '['python3', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/source/w8a8_bench.py', '--source', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/source', '--m', '16', '--n', '2560', '--k', '4096', '--reference-cache-dir', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/hy3_tp4_qkv_proj_m16/workspace/final/cache/references', '--warmups', '30', '--samples', '30', '--replays-per-sample', '50']' timed out after 1200 seconds\")"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-17-1-aaea0725:iteration_001/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-minimax-tp8-qkv-proj-m4096-d93662", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789659417.2202265, "occurrences": 1, "raw": "Command '['bash', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/minimax_tp8_qkv_proj_m4096/workspace/workers/worker_1/source/profile_pmc.sh', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/minimax_tp8_qkv_proj_m4096/workspace/workers/worker_1/source/w8a8_bench.py', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/minimax_tp8_qkv_proj_m4096/workspace/workers/worker_1/source', '4096', '1280', '6144', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/minimax_tp8_qkv_proj_m4096/workspace/workers/worker_1/profiles/minimax_tp8_qkv_proj_m4096/iteration7']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test-9-17-1-aaea0725:iteration_001/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-minimax-tp8-qkv-proj-m4096-d93662", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789668520.0595274, "occurrences": 1, "raw": "Command '['bash', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/minimax_tp8_qkv_proj_m4096/workspace/workers/worker_1/source/profile_pmc.sh', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/minimax_tp8_qkv_proj_m4096/workspace/workers/worker_1/source/w8a8_bench.py', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/minimax_tp8_qkv_proj_m4096/workspace/workers/worker_1/source', '4096', '1280', '6144', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test-9-17-1-aaea0725/children/iteration_001/minimax_tp8_qkv_proj_m4096/workspace/workers/worker_1/profiles/minimax_tp8_qkv_proj_m4096/iteration13']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test9-20-1-3f9b399d:iteration_001/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-m16-wqkv-a-b0cfda", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789925012.2626514, "occurrences": 1, "raw": "Command '['bash', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-20-1-3f9b399d/children/iteration_001/m16_wqkv_a/workspace/workers/worker_0/source/profile_pmc.sh', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-20-1-3f9b399d/children/iteration_001/m16_wqkv_a/workspace/workers/worker_0/source/w8a8_bench.py', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-20-1-3f9b399d/children/iteration_001/m16_wqkv_a/workspace/workers/worker_0/source', '16', '1536', '4096', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-20-1-3f9b399d/children/iteration_001/m16_wqkv_a/workspace/workers/worker_0/profiles/m16_wqkv_a/iteration9']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp8-shared-down-proj-m4096-931a67", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1790068768.249131, "occurrences": 1, "raw": "Command '['bash', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/hy3_tp8_shared_down_proj_m4096/workspace/workers/worker_1/source/profile_pmc.sh', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/hy3_tp8_shared_down_proj_m4096/workspace/workers/worker_1/source/w8a8_bench.py', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/hy3_tp8_shared_down_proj_m4096/workspace/workers/worker_1/source', '4096', '4096', '192', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/hy3_tp8_shared_down_proj_m4096/workspace/workers/worker_1/profiles/hy3_tp8_shared_down_proj_m4096/iteration1']' timed out after 1200 seconds"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test9-22-1-fd2ebf70:iteration_001/minimax_tp8_o_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-minimax-tp8-o-proj-m16-85bd16", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1790065660.2458627, "occurrences": 1, "raw": "python3 /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/minimax_tp8_o_proj_m16/workspace/workers/worker_3/source/w8a8_bench.py --source /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/minimax_tp8_o_proj_m16/workspace/workers/worker_3/source --m 16 --n 6144 --k 1024 --reference-cache-dir /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/minimax_tp8_o_proj_m16/workspace/workers/worker_3/cache/references failed (1): ip -o w8a8_gemm_hip.cuda.o /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/minimax_tp8_o_proj_m16/workspace/workers/worker_3/cache/compile_sources/c89c90af43edd19f97772ba5d57d80c6927428cb5787792f9e1c5b1c4dc837e8/csrc/w8a8_gemm_hip.hip:52:1: warning: void function is missing a return statement [-Wreturn-type] } ^ /root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_001/minimax_tp8_o_proj_m16/workspace/workers/worker_3/cache/compile_sources/c89c90af43edd19f97772ba5d57d80c6927428cb5787792f9e1c5b1c4dc837e8/csrc/w8a8_gemm_hip.hip:64:1: warning: void functio…"} +{"class_id": "MEAS-CMD-FAILED", "task_id": "test9-22-1-fd2ebf70:iteration_002/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-minimax-tp4-shared-down-proj-m4096-678a97", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1790136918.8148077, "occurrences": 1, "raw": "Command '['bash', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_002/minimax_tp4_shared_down_proj_m4096/workspace/workers/worker_3/source/profile_pmc.sh', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_002/minimax_tp4_shared_down_proj_m4096/workspace/workers/worker_3/source/w8a8_bench.py', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_002/minimax_tp4_shared_down_proj_m4096/workspace/workers/worker_3/source', '4096', '6144', '768', '/root/zth_agent/MetaInfer/nodes/worker29/workspaces/test9-22-1-fd2ebf70/children/iteration_002/minimax_tp4_shared_down_proj_m4096/workspace/workers/worker_3/profiles/minimax_tp4_shared_down_proj_m4096/iteration15']' timed out after 1200 seconds"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1785744345.6928775, "occurrences": 11, "raw": "invalid hipprof memory request counters"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1785745285.558738, "occurrences": 39, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "dcu-kernel-auto-opt-cc86e2b2", "run_id": "dcu-kernel-auto-opt-cc86e2b2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1786025653.1418889, "occurrences": 51, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "dcu-kernel-auto-opt-fa8de7cc", "run_id": "dcu-kernel-auto-opt-fa8de7cc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1785766669.0188434, "occurrences": 14, "raw": "invalid hipprof memory request counters"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "dcu-kernel-auto-opt-fa8de7cc", "run_id": "dcu-kernel-auto-opt-fa8de7cc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1785767260.0841205, "occurrences": 74, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "dcu-kernel-auto-opt-feb64434", "run_id": "dcu-kernel-auto-opt-feb64434", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1785833831.9315288, "occurrences": 2, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "dcu-kernel-auto-opt-feb64434", "run_id": "dcu-kernel-auto-opt-feb64434", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1785834449.9479992, "occurrences": 37, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787535284.1932209, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=351392 exceeds TCC_EA_WRREQ=335873 by 15519 (tolerance=34)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787536064.273755, "occurrences": 59, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787543459.952994, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=361398 exceeds TCC_EA_WRREQ=340080 by 21318 (tolerance=35)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787986930.6357985, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787988089.654866, "occurrences": 70, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1788106868.4417875, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=336960 exceeds TCC_EA_WRREQ=335917 by 1043 (tolerance=34)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1788107759.7713182, "occurrences": 54, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787568809.185895, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787569207.5088112, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=58132 exceeds TCC_EA_WRREQ=384 by 57748 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787569237.1142554, "occurrences": 40, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787677848.8103042, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787678886.0494175, "occurrences": 54, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787679043.3874052, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=249441 exceeds TCC_EA_WRREQ=3840 by 245601 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787684230.930808, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=16698 exceeds TCC_EA_WRREQ=5120 by 11578 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787724690.387039, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787725098.857836, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=322209 exceeds TCC_EA_WRREQ=862 by 321347 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787725127.9988976, "occurrences": 53, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787725193.837087, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=303980 exceeds TCC_EA_WRREQ=256 by 303724 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787725442.0879698, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=105925 exceeds TCC_EA_WRREQ=2048 by 103877 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787725615.4500225, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=279013 exceeds TCC_EA_WRREQ=769 by 278244 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787743121.965121, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787743468.0346386, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=552069 exceeds TCC_EA_WRREQ=656 by 551413 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787743822.5008721, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=217886 exceeds TCC_EA_WRREQ=2048 by 215838 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787743868.757001, "occurrences": 47, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787744002.0483694, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=215444 exceeds TCC_EA_WRREQ=640 by 214804 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1788883170.4339318, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1788884903.3124964, "occurrences": 75, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787762168.4977064, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=53403 exceeds TCC_EA_WRREQ=49802 by 3601 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787762281.0346658, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=556246 exceeds TCC_EA_WRREQ=526516 by 29730 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787762771.1642704, "occurrences": 54, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787771275.8521097, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=527519 exceeds TCC_EA_WRREQ=527321 by 198 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787771995.9015434, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=533278 exceeds TCC_EA_WRREQ=525571 by 7707 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "run_id": "hy3-tp4-4096-sonnet-5i-1-cbcd099f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787200956.4342823, "occurrences": 16, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787329852.2472794, "occurrences": 46, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1788436675.5190513, "occurrences": 106, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1788448534.428119, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=911608 exceeds TCC_EA_WRREQ=50065 by 861543 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1788459502.689345, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=63456 exceeds TCC_EA_WRREQ=49563 by 13893 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-a76091f6:iteration_001/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-a76091f6-it001-candidate-hy3-tp8-o-proj-m16-495edf", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789372791.5374653, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-a76091f6:iteration_001/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-a76091f6-it001-candidate-hy3-tp8-o-proj-m16-495edf", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789373475.5142157, "occurrences": 1, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-a76091f6:iteration_001/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-a76091f6-it001-candidate-hy3-tp8-o-proj-m16-495edf", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789374027.6689174, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=18206 exceeds TCC_EA_WRREQ=16384 by 1822 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-a76091f6:iteration_001/hy3_tp8_shared_gate_up_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-a76091f6-it001-candidate-hy3-tp8-shared-gate-up-proj-m16-f4e55e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789372814.9351497, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-indexer-wq-b-m16-b7c33c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789374717.7954, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-indexer-wq-b-m16-b7c33c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789377255.9489872, "occurrences": 6, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-o-proj-m16-3273d5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789374691.0674207, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-o-proj-m16-3273d5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789375330.6301396, "occurrences": 6, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-shared-down-proj-m16-d7f058", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789374696.3469758, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-shared-down-proj-m16-d7f058", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789375708.9988034, "occurrences": 10, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-wqkv-a-m16-d9c065", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789374940.5920947, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_001/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it001-candidate-hy3-tp8-wqkv-a-m16-d9c065", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789376842.383666, "occurrences": 5, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-indexer-wq-b-m16-492bc9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789383246.4168713, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-indexer-wq-b-m16-492bc9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789385226.3458464, "occurrences": 8, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-o-proj-m16-655f2f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789383210.9635766, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-o-proj-m16-655f2f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789384668.236184, "occurrences": 9, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789383179.4754245, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-shared-down-proj-m16-c9edab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789383974.184383, "occurrences": 10, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-wqkv-a-m16-29801d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789383230.529778, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-wqkv-a-m16-29801d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789383865.5131638, "occurrences": 8, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_002/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it002-candidate-hy3-tp8-wqkv-a-m16-29801d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789384425.0999532, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=3938665 exceeds TCC_EA_WRREQ=23040 by 3915625 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-indexer-wq-b-m16-29d691", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789395624.4982195, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-indexer-wq-b-m16-29d691", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789396347.2573614, "occurrences": 8, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-o-proj-m16-828ad3", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789395623.3815775, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-o-proj-m16-828ad3", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789397665.864862, "occurrences": 5, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-o-proj-m16-828ad3", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789400432.5254366, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=4685 exceeds TCC_EA_WRREQ=2073 by 2612 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-shared-down-proj-m16-b7c81a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789395650.4716485, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-shared-down-proj-m16-b7c81a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789396415.071741, "occurrences": 9, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-wqkv-a-m16-b232b1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789395682.8048007, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_003/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it003-candidate-hy3-tp8-wqkv-a-m16-b232b1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789397880.94003, "occurrences": 9, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789404321.652668, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789405028.3246212, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789404388.6804814, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789405031.8961382, "occurrences": 7, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789456198.1522038, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_shared_down_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-down-proj-m16-5e2c3e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789457429.374814, "occurrences": 3, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_shared_gate_up_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-shared-gate-up-proj-m16-c4f02e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789461256.9210823, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789456207.8319237, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789457518.7811453, "occurrences": 3, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-test9-16-619177f8", "run_id": "hy3-tp8-m16-test9-16-619177f8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789556292.0819244, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m16-test9-16-619177f8", "run_id": "hy3-tp8-m16-test9-16-619177f8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789557038.909402, "occurrences": 43, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1791281097.6047304, "occurrences": 140, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787819642.585007, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787820390.9720197, "occurrences": 53, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787824333.4292583, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=6392 exceeds TCC_EA_WRREQ=6144 by 248 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787826478.6994498, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=18708 exceeds TCC_EA_WRREQ=18432 by 276 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787829222.7979207, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=248840 exceeds TCC_EA_WRREQ=3073 by 245767 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787830185.7002282, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=264201 exceeds TCC_EA_WRREQ=18432 by 245769 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787830778.4661703, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=19254 exceeds TCC_EA_WRREQ=18432 by 822 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787832300.9038978, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=35153 exceeds TCC_EA_WRREQ=18432 by 16721 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787837009.138831, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=9660 exceeds TCC_EA_WRREQ=3072 by 6588 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787837445.9294252, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=5312 exceeds TCC_EA_WRREQ=3072 by 2240 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787839377.9871786, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=5789 exceeds TCC_EA_WRREQ=3072 by 2717 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787840080.9280002, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=3324 exceeds TCC_EA_WRREQ=3075 by 249 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1787842541.5710418, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=4139 exceeds TCC_EA_WRREQ=3149 by 990 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-tp8-m8-1-1e22df66", "run_id": "minimax-tp8-m8-1-1e22df66", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790249027.2613294, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-tp8-m8-1-1e22df66", "run_id": "minimax-tp8-m8-1-1e22df66", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790249666.7805412, "occurrences": 55, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790166538.120865, "occurrences": 4, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-tp8-m8-1-e4e50cf8", "run_id": "minimax-tp8-m8-1-e4e50cf8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790167593.798959, "occurrences": 44, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1791391480.3659463, "occurrences": 116, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787458058.2013068, "occurrences": 38, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1787906463.5959332, "occurrences": 64, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-35600c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789109811.5887797, "occurrences": 6, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_001/glm_tp8_kv_b_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it001-candidate-glm-tp8-kv-b-proj-m4096-9267b6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789107188.0075622, "occurrences": 9, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_001/glm_tp8_kv_b_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it001-candidate-glm-tp8-kv-b-proj-m4096-9267b6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789114680.0351462, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=704378 exceeds TCC_EA_WRREQ=459077 by 245301 (tolerance=46)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_001/glm_tp8_q_b_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it001-candidate-glm-tp8-q-b-proj-m4096-bf181d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789107316.137062, "occurrences": 10, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it001-candidate-hy3-tp4-o-proj-m16-118a34", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789106726.174411, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it001-candidate-hy3-tp4-o-proj-m16-118a34", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789107402.5800312, "occurrences": 14, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-fused-qkv-a-proj-m4096-ca9b7c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789125140.488511, "occurrences": 9, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-o-proj-m4096-6096d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789133577.9259079, "occurrences": 14, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-down-proj-m4096-fee009", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789123611.7710576, "occurrences": 13, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-2c215a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789133020.3959026, "occurrences": 14, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789122968.0481167, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m16-5fb28e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789123774.8494022, "occurrences": 11, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-o-proj-m4096-de2dc0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789130213.3128543, "occurrences": 11, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m16-155b0d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789123164.0633721, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m16-155b0d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789123987.9962451, "occurrences": 12, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-qkv-proj-m4096-a53d19", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789130501.725951, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m16-3c8a59", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789125893.3994565, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m16-3c8a59", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789126497.9737442, "occurrences": 12, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789131426.5632899, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=727505 exceeds TCC_EA_WRREQ=540057 by 187448 (tolerance=55)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-down-proj-m4096-2ca7da", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789131643.2157826, "occurrences": 17, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789128565.5102975, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp4_shared_gate_up_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp4-shared-gate-up-proj-m16-782b0c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789129656.0736148, "occurrences": 10, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789131536.4615886, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=876615 exceeds TCC_EA_WRREQ=524958 by 351657 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789131890.9320698, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=701429 exceeds TCC_EA_WRREQ=526428 by 175001 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789132173.3432562, "occurrences": 14, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789135165.7121954, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=641607 exceeds TCC_EA_WRREQ=526540 by 115067 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_002/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it002-candidate-hy3-tp8-o-proj-m4096-406336", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789135817.36961, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=714802 exceeds TCC_EA_WRREQ=524794 by 190008 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789132906.9971356, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=1439124 exceeds TCC_EA_WRREQ=335872 by 1103252 (tolerance=34)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789133330.5352206, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=669218 exceeds TCC_EA_WRREQ=335889 by 333329 (tolerance=34)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789133374.70583, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-fused-qkv-a-proj-m4096-da2846", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789134183.6985908, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=7855058 exceeds TCC_EA_WRREQ=336007 by 7519051 (tolerance=34)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-glm-tp8-shared-down-proj-m4096-a90d60", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789132847.7491198, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789136994.2390335, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789137036.3814914, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789137425.6687033, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=506598 exceeds TCC_EA_WRREQ=2048 by 504550 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m16-9b025d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789138513.8527946, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=5274465 exceeds TCC_EA_WRREQ=2048 by 5272417 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-o-proj-m4096-0cc316", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789136430.514651, "occurrences": 14, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789132137.6060753, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m16-459d54", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789132791.4445062, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-qkv-proj-m4096-6c7a89", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789138667.2550852, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789135394.507227, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-down-proj-m16-ef7a94", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789135508.7280478, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789135528.1958854, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=470953 exceeds TCC_EA_WRREQ=98306 by 372647 (tolerance=10)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789136081.0842927, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=344107 exceeds TCC_EA_WRREQ=98311 by 245796 (tolerance=10)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp4-shared-gate-up-proj-m4096-090f5c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789136127.2467434, "occurrences": 16, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_o_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-o-proj-m4096-77d822", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789140064.84449, "occurrences": 16, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789132504.0746229, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=501680 exceeds TCC_EA_WRREQ=166490 by 335190 (tolerance=17)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-qkv-proj-m4096-dd9efa", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789132754.4875505, "occurrences": 17, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789137425.2445297, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=3225480 exceeds TCC_EA_WRREQ=524323 by 2701157 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789137794.0980163, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=1261079 exceeds TCC_EA_WRREQ=528582 by 732497 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789138280.041125, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=767466 exceeds TCC_EA_WRREQ=527379 by 240087 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789138781.138933, "occurrences": 13, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-down-proj-m4096-947961", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789141443.28619, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=574264 exceeds TCC_EA_WRREQ=524581 by 49683 (tolerance=53)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789139107.1333132, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=294741 exceeds TCC_EA_WRREQ=49154 by 245587 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789139471.6755168, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=201028 exceeds TCC_EA_WRREQ=49167 by 151861 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-hy3-tp8-shared-gate-up-proj-m4096-5ded23", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789139970.2548556, "occurrences": 16, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789134919.2977605, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789135288.4808311, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=587565 exceeds TCC_EA_WRREQ=997 by 586568 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789135452.7673776, "occurrences": 13, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789135829.0098934, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=74179 exceeds TCC_EA_WRREQ=7680 by 66499 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789136153.620334, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=74844 exceeds TCC_EA_WRREQ=7680 by 67164 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/m16_wqkv_a", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-m16-wqkv-a-4cd0a6", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789138714.0286825, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=75366 exceeds TCC_EA_WRREQ=15360 by 60006 (tolerance=8)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789140522.2268214, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=5050866 exceeds TCC_EA_WRREQ=372092 by 4678774 (tolerance=38)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789140582.3983114, "occurrences": 16, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-and-indexer-qk-m4096-b2c46e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789141530.007221, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=626040 exceeds TCC_EA_WRREQ=368550 by 257490 (tolerance=37)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789140409.2600608, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=920763 exceeds TCC_EA_WRREQ=295514 by 625249 (tolerance=30)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789140535.4843085, "occurrences": 16, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_003/minimax_tp4_qkv_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it003-candidate-minimax-tp4-qkv-proj-m4096-57b022", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789140912.1798205, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=960030 exceeds TCC_EA_WRREQ=295153 by 664877 (tolerance=30)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789140429.8178713, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=1090623 exceeds TCC_EA_WRREQ=335954 by 754669 (tolerance=34)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789140536.0103114, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789140894.4246855, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=1424548 exceeds TCC_EA_WRREQ=335872 by 1088676 (tolerance=34)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-fused-qkv-a-proj-m4096-56f6eb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789141912.0270984, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=1318442 exceeds TCC_EA_WRREQ=335872 by 982570 (tolerance=34)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/glm_tp8_shared_down_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-glm-tp8-shared-down-proj-m4096-05d388", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789140138.2558713, "occurrences": 14, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789141898.3435605, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-o-proj-m16-2e777f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789142691.0489917, "occurrences": 15, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789139945.7383735, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-hy3-tp4-qkv-proj-m16-c4abeb", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789140535.1161458, "occurrences": 16, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_profile", "ts": 1789142855.762183, "occurrences": 1, "raw": "invalid hipprof memory request counters: TCC_EA_WRREQ_64B=7406861 exceeds TCC_EA_WRREQ=217669 by 7189192 (tolerance=22)"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-11-1-b01955b3:iteration_004/minimax_tp4_shared_gate_up_proj_m4096", "run_id": "ahe-test-9-11-1-b01955b3-it004-candidate-minimax-tp4-shared-gate-up-proj-m4096-b5fba4", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789142861.8794122, "occurrences": 17, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-17-1-aaea0725:iteration_001/glm_tp8_fused_qkv_a_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-glm-tp8-fused-qkv-a-proj-m4096-2a6130", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789671546.3719437, "occurrences": 11, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-o-proj-m16-3768c0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789654520.1850443, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_o_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-o-proj-m16-3768c0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789669614.5742202, "occurrences": 12, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-qkv-proj-m16-d5c6e1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789644941.539075, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-17-1-aaea0725:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-hy3-tp4-qkv-proj-m16-d5c6e1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789645907.31042, "occurrences": 6, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test-9-17-1-aaea0725:iteration_001/minimax_tp8_qkv_proj_m4096", "run_id": "ahe-test-9-17-1-aaea0725-it001-candidate-minimax-tp8-qkv-proj-m4096-d93662", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789656192.9979718, "occurrences": 8, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_001/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-m16-wqkv-a-b0cfda", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789918291.7892172, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_001/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-m16-wqkv-a-b0cfda", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789919032.6085968, "occurrences": 6, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp4-shared-down-proj-m4096-5a4f0b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789920504.6527975, "occurrences": 9, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789918324.321239, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-qkv-proj-m16-b003a0", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789943030.612511, "occurrences": 5, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-shared-down-proj-m16-65601d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789918274.9310284, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_001/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it001-candidate-minimax-tp8-shared-down-proj-m16-65601d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789918921.8807068, "occurrences": 13, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_002/glm_tp8_o_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it002-candidate-glm-tp8-o-proj-m4096-987747", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789950341.680983, "occurrences": 12, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it002-candidate-hy3-tp4-o-proj-m16-18694f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789949169.9395714, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_002/hy3_tp4_o_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it002-candidate-hy3-tp4-o-proj-m16-18694f", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789949920.701323, "occurrences": 14, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_002/minimax_tp4_o_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it002-candidate-minimax-tp4-o-proj-m4096-208840", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789950753.3398697, "occurrences": 12, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_002/minimax_tp8_o_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it002-candidate-minimax-tp8-o-proj-m4096-2a09fc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789950160.4548075, "occurrences": 12, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_003_generalization/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it003-candidate-m16-wqkv-a-f5487a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789959956.1181443, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_003_generalization/m16_wqkv_a", "run_id": "ahe-test9-20-1-3f9b399d-it003-candidate-m16-wqkv-a-f5487a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789962802.6353812, "occurrences": 5, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_003_generalization/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-20-1-3f9b399d-it003-candidate-minimax-tp4-shared-down-proj-m4096-6c8629", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789961768.8448505, "occurrences": 8, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_003_generalization/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it003-candidate-minimax-tp8-qkv-proj-m16-74da05", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789960026.6161206, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_003_generalization/minimax_tp8_qkv_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it003-candidate-minimax-tp8-qkv-proj-m16-74da05", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789961575.8606138, "occurrences": 9, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_003_generalization/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it003-candidate-minimax-tp8-shared-down-proj-m16-66f487", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789959951.2396994, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-20-1-3f9b399d:iteration_003_generalization/minimax_tp8_shared_down_proj_m16", "run_id": "ahe-test9-20-1-3f9b399d-it003-candidate-minimax-tp8-shared-down-proj-m16-66f487", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1789961137.7096386, "occurrences": 13, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp4-qkv-proj-m16-292420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790069352.8018866, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp4_qkv_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp4-qkv-proj-m16-292420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790119644.569472, "occurrences": 6, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp8_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp8-shared-down-proj-m4096-931a67", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790072487.1284773, "occurrences": 5, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_001/hy3_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-hy3-tp8-shared-gate-up-proj-m4096-4fd75d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790067048.9322712, "occurrences": 6, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_001/minimax_tp8_o_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-minimax-tp8-o-proj-m16-85bd16", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790065852.3803318, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_001/minimax_tp8_o_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it001-candidate-minimax-tp8-o-proj-m16-85bd16", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790066474.6530178, "occurrences": 13, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_002/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-glm-tp8-shared-gate-up-proj-m4096-f5d1a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790125651.208742, "occurrences": 10, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_qkv_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-qkv-proj-m4096-4a2913", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790127121.7768092, "occurrences": 8, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-shared-down-proj-m16-ae1f1c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790124924.4558432, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_002/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-hy3-tp4-shared-down-proj-m16-ae1f1c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790126122.6879797, "occurrences": 12, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_002/minimax_tp4_shared_down_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it002-candidate-minimax-tp4-shared-down-proj-m4096-678a97", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790126080.0019906, "occurrences": 5, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_003/glm_tp8_kv_b_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-glm-tp8-kv-b-proj-m4096-8135a8", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790153423.632187, "occurrences": 11, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_003/glm_tp8_shared_gate_up_proj_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-glm-tp8-shared-gate-up-proj-m4096-4b3831", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790155538.5657268, "occurrences": 5, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-hy3-tp4-shared-down-proj-m16-e92ad1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790158152.485497, "occurrences": 1, "raw": "skip PMC for scalar bootstrap; first establish a validated DUMMA/current-best implementation"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_003/hy3_tp4_shared_down_proj_m16", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-hy3-tp4-shared-down-proj-m16-e92ad1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790160596.6828232, "occurrences": 2, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "MEAS-PMC-UNAVAILABLE", "task_id": "test9-22-1-fd2ebf70:iteration_003/minimax_tp8_qkv_proj_and_indexer_qk_m4096", "run_id": "ahe-test9-22-1-fd2ebf70-it003-candidate-minimax-tp8-qkv-proj-and-indexer-qk-m4096-e9ad3b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_pmc_skipped", "ts": 1790160422.476507, "occurrences": 2, "raw": "current best is unchanged; reuse matching prior counters and reserve fresh PMC for accepted-best or late ISA decisions"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-2e64c51b", "run_id": "dcu-kernel-auto-opt-2e64c51b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785678713.3673904, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-2e64c51b", "run_id": "dcu-kernel-auto-opt-2e64c51b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785679709.0106897, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-2e64c51b", "run_id": "dcu-kernel-auto-opt-2e64c51b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785681438.257903, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-2e64c51b", "run_id": "dcu-kernel-auto-opt-2e64c51b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1785681899.906194, "occurrences": 2, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-2e64c51b", "run_id": "dcu-kernel-auto-opt-2e64c51b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785682429.370898, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-2e64c51b", "run_id": "dcu-kernel-auto-opt-2e64c51b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785682713.1190863, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-2e64c51b", "run_id": "dcu-kernel-auto-opt-2e64c51b", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785690368.0639462, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-602ee1ee", "run_id": "dcu-kernel-auto-opt-602ee1ee", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_failed", "ts": 1785214036.8488905, "occurrences": 1, "raw": "worker_3-m16_wq_b-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-602ee1ee", "run_id": "dcu-kernel-auto-opt-602ee1ee", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_failed", "ts": 1785214283.8955948, "occurrences": 1, "raw": "worker_1-m16_wqkv_a-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-602ee1ee", "run_id": "dcu-kernel-auto-opt-602ee1ee", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_failed", "ts": 1785216535.1924798, "occurrences": 1, "raw": "worker_2-m2_wq_b-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785752202.81039, "occurrences": 1, "raw": "worker_0-tp4_wqkv_a_m3072-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1785752290.4722316, "occurrences": 3, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-7eafe2cc", "run_id": "dcu-kernel-auto-opt-7eafe2cc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785241993.7244773, "occurrences": 1, "raw": "worker_2-m16_wqkv_a-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-7eafe2cc", "run_id": "dcu-kernel-auto-opt-7eafe2cc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785242057.2822657, "occurrences": 1, "raw": "worker_3-m16_wq_b-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8b6136de", "run_id": "dcu-kernel-auto-opt-8b6136de", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785637213.4563422, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8b6136de", "run_id": "dcu-kernel-auto-opt-8b6136de", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785638353.8058178, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8b6136de", "run_id": "dcu-kernel-auto-opt-8b6136de", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785638984.5288863, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8b6136de", "run_id": "dcu-kernel-auto-opt-8b6136de", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785639573.4790523, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8b6136de", "run_id": "dcu-kernel-auto-opt-8b6136de", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785639681.3858476, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8b6136de", "run_id": "dcu-kernel-auto-opt-8b6136de", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785639724.1639957, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8b6136de", "run_id": "dcu-kernel-auto-opt-8b6136de", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785640268.0321515, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785218616.919345, "occurrences": 1, "raw": "worker_0-m2_wqkv_a-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785218899.5068572, "occurrences": 1, "raw": "worker_1-m2_wq_b-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785219308.795733, "occurrences": 1, "raw": "worker_3-m16_wq_b-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785219557.0726678, "occurrences": 1, "raw": "worker_3-m16_wq_b-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785219855.3941998, "occurrences": 1, "raw": "worker_3-m16_wq_b-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785219856.302024, "occurrences": 1, "raw": "worker_1-m2_wq_b-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785219930.967331, "occurrences": 1, "raw": "worker_0-m2_wqkv_a-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785219986.278926, "occurrences": 1, "raw": "worker_2-m16_wqkv_a-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1785220158.6208894, "occurrences": 1, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-8c1f2eac", "run_id": "dcu-kernel-auto-opt-8c1f2eac", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785220374.7020555, "occurrences": 1, "raw": "worker_2-m16_wqkv_a-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-9f40c2a1", "run_id": "dcu-kernel-auto-opt-9f40c2a1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785695259.782681, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-9f40c2a1", "run_id": "dcu-kernel-auto-opt-9f40c2a1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785698014.4737105, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-9f40c2a1", "run_id": "dcu-kernel-auto-opt-9f40c2a1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785699013.5962834, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-9f40c2a1", "run_id": "dcu-kernel-auto-opt-9f40c2a1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785701364.7046554, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-9f40c2a1", "run_id": "dcu-kernel-auto-opt-9f40c2a1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785703029.1371691, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-9f40c2a1", "run_id": "dcu-kernel-auto-opt-9f40c2a1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785705033.3837757, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-9f40c2a1", "run_id": "dcu-kernel-auto-opt-9f40c2a1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1785705335.7150187, "occurrences": 1, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-a473ffe1", "run_id": "dcu-kernel-auto-opt-a473ffe1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785209445.19346, "occurrences": 1, "raw": "worker_2-bootstrap-attempt1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-a473ffe1", "run_id": "dcu-kernel-auto-opt-a473ffe1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_bootstrap_attempt_failed", "ts": 1785210594.02065, "occurrences": 1, "raw": "worker_2-bootstrap-attempt3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-aafa5d91", "run_id": "dcu-kernel-auto-opt-aafa5d91", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785741521.010724, "occurrences": 1, "raw": "worker_0-tp4_wqkv_a_m3072-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-aafa5d91", "run_id": "dcu-kernel-auto-opt-aafa5d91", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785741851.1183147, "occurrences": 1, "raw": "worker_2-tp4_wo_b_m3072-iter1-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785306060.7760763, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785307297.6600785, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785309151.6776214, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785309678.5611196, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785310116.848132, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785311145.2140827, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785311295.3343465, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785311832.9900515, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785313629.9057655, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785314789.78325, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c7e73766", "run_id": "dcu-kernel-auto-opt-c7e73766", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785316221.8836966, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785694764.7083492, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785695019.7403889, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785695903.7978048, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785696015.0340383, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785696473.760561, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785696798.750096, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785697471.3845587, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785697798.8515074, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785698464.2135072, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785699358.686229, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785699461.3709233, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785700109.8193066, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785700666.6433258, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785701459.6623068, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785702376.3469975, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785702453.7041368, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785702864.4909053, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785703366.6988125, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785703450.7409902, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785704262.4465907, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785704357.4672914, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785704443.745174, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785705440.9602735, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1785705743.2965493, "occurrences": 4, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785705786.2262526, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785708791.0393379, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785710644.0325222, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785716225.0011766, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785719648.9995668, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785722427.3769863, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785723124.5307915, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785725904.8684738, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-c9d1c1dc", "run_id": "dcu-kernel-auto-opt-c9d1c1dc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785726783.9051843, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-cc86e2b2", "run_id": "dcu-kernel-auto-opt-cc86e2b2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1786028705.140466, "occurrences": 1, "raw": "worker_0-tp4_wqkv_a_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-cc86e2b2", "run_id": "dcu-kernel-auto-opt-cc86e2b2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1786030963.3745854, "occurrences": 4, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-cc86e2b2", "run_id": "dcu-kernel-auto-opt-cc86e2b2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1786034478.4419405, "occurrences": 1, "raw": "worker_0-tp4_wqkv_a_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-cc86e2b2", "run_id": "dcu-kernel-auto-opt-cc86e2b2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1786037683.0724137, "occurrences": 1, "raw": "worker_1-tp4_indexer_wq_b_m4096-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-d4dd2dc5", "run_id": "dcu-kernel-auto-opt-d4dd2dc5", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "orchestrator_error", "ts": 1785087740.7330818, "occurrences": 1, "raw": "RuntimeError('worker_2-m2_wqkv_a-iter2 failed: killed (see events log tail)')"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785396964.018002, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785397673.8623278, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785397923.2068164, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785398850.2579331, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785399366.6658313, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785399543.332932, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785399752.8946035, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785399786.2239237, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785400450.916607, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785401145.036267, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785401791.139172, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785401844.1630776, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785401952.2812314, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785402479.5573993, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eab76420", "run_id": "dcu-kernel-auto-opt-eab76420", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785404364.1950455, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785650831.3871984, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785651033.3752317, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785651867.6005561, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785651892.088693, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785652183.878433, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785652586.8926105, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785653344.701774, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785653649.917625, "occurrences": 1, "raw": "worker_1-m16_wq_b-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785653978.849791, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785654039.392312, "occurrences": 1, "raw": "worker_0-m16_wqkv_a-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785654163.734898, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785655178.184114, "occurrences": 1, "raw": "worker_3-m16_shared_gate_up-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785655405.2460268, "occurrences": 1, "raw": "worker_2-m16_wo_b-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eba3825d", "run_id": "dcu-kernel-auto-opt-eba3825d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785660157.8734443, "occurrences": 1, "raw": "worker_3-m16_shared_down-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eddb316e", "run_id": "dcu-kernel-auto-opt-eddb316e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785245699.2290893, "occurrences": 1, "raw": "worker_1-m2_wq_b-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eddb316e", "run_id": "dcu-kernel-auto-opt-eddb316e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785246233.4911096, "occurrences": 1, "raw": "worker_2-m16_wqkv_a-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-eddb316e", "run_id": "dcu-kernel-auto-opt-eddb316e", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1785246318.5458882, "occurrences": 1, "raw": "worker_3-m16_wq_b-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-fa8de7cc", "run_id": "dcu-kernel-auto-opt-fa8de7cc", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1785780009.6061301, "occurrences": 3, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "dcu-kernel-auto-opt-feb64434", "run_id": "dcu-kernel-auto-opt-feb64434", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1785840236.9333205, "occurrences": 2, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787536187.2094517, "occurrences": 1, "raw": "worker_0-glm_tp4_fused_qkv_a_proj_m4096-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787536967.2763832, "occurrences": 1, "raw": "worker_1-glm_tp4_q_b_proj_m4096-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787539318.6971748, "occurrences": 1, "raw": "worker_0-glm_tp4_fused_qkv_a_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787539365.9335814, "occurrences": 1, "raw": "worker_1-glm_tp4_q_b_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787539693.5323207, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_gate_up_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787540268.9951494, "occurrences": 1, "raw": "worker_1-glm_tp4_q_b_proj_m4096-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787541120.7198853, "occurrences": 1, "raw": "worker_0-glm_tp4_fused_qkv_a_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787541172.0263088, "occurrences": 1, "raw": "worker_1-glm_tp4_q_b_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787541244.8393183, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_gate_up_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787541723.5324538, "occurrences": 1, "raw": "worker_0-glm_tp4_fused_qkv_a_proj_m4096-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787542626.6315298, "occurrences": 1, "raw": "worker_0-glm_tp4_fused_qkv_a_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787543690.6820245, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_gate_up_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787544363.0086777, "occurrences": 1, "raw": "worker_0-glm_tp4_fused_qkv_a_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787545266.1108036, "occurrences": 1, "raw": "worker_0-glm_tp4_fused_qkv_a_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787545633.0751517, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_gate_up_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787545981.6971357, "occurrences": 1, "raw": "worker_1-glm_tp4_q_b_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787546327.7769544, "occurrences": 1, "raw": "worker_2-glm_tp4_o_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787546884.7507277, "occurrences": 1, "raw": "worker_1-glm_tp4_q_b_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787547230.8998313, "occurrences": 1, "raw": "worker_2-glm_tp4_o_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787547880.7428956, "occurrences": 1, "raw": "worker_0-glm_tp4_fused_qkv_a_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787548133.975832, "occurrences": 1, "raw": "worker_2-glm_tp4_o_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787549299.5353236, "occurrences": 1, "raw": "worker_1-glm_tp4_kv_b_proj_m4096-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787550504.972055, "occurrences": 1, "raw": "worker_2-glm_tp4_o_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787550969.8848507, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_down_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787551656.416713, "occurrences": 1, "raw": "worker_1-glm_tp4_kv_b_proj_m4096-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787552168.247675, "occurrences": 1, "raw": "worker_2-glm_tp4_o_proj_m4096-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787552348.1158488, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_down_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787552559.4900336, "occurrences": 1, "raw": "worker_1-glm_tp4_kv_b_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787553251.2089329, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_down_proj_m4096-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787554094.5196767, "occurrences": 1, "raw": "worker_2-glm_tp4_o_proj_m4096-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787555100.4657931, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_down_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787556499.341717, "occurrences": 1, "raw": "worker_1-glm_tp4_kv_b_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787556613.2382526, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_down_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787558957.1187959, "occurrences": 1, "raw": "worker_3-glm_tp4_shared_down_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787559765.6071377, "occurrences": 1, "raw": "worker_1-glm_tp4_kv_b_proj_m4096-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "run_id": "glm5-2-dsh-tp4-m4096-1-85cb689c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787560668.7511544, "occurrences": 1, "raw": "worker_1-glm_tp4_kv_b_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787995365.6610422, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m16-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787996846.4616838, "occurrences": 1, "raw": "worker_2-glm_tp8_o_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787998458.912014, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787998549.1925256, "occurrences": 1, "raw": "worker_2-glm_tp8_o_proj_m16-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787999378.6317968, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_gate_up_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787999545.9583588, "occurrences": 1, "raw": "worker_2-glm_tp8_o_proj_m16-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788001160.9519424, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_gate_up_proj_m16-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788001644.7794504, "occurrences": 1, "raw": "worker_2-glm_tp8_o_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788001844.221995, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m16-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788006899.9661834, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m16-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788009193.9203446, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m16-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788010362.5115619, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_down_proj_m16-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788012141.0951092, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788013542.6756968, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m16-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "run_id": "glm5-2-dsh-tp8-m16-test1-cb4d262c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788019430.148392, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_down_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788107943.754969, "occurrences": 1, "raw": "worker_1-glm_tp8_q_b_proj_m4096-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788108846.803183, "occurrences": 1, "raw": "worker_1-glm_tp8_q_b_proj_m4096-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788110286.7079377, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_gate_up_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788112436.7132661, "occurrences": 1, "raw": "worker_2-glm_tp8_o_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788113460.2819118, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_gate_up_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788113838.6608334, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788114741.8068755, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788115951.1147518, "occurrences": 1, "raw": "worker_2-glm_tp8_o_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788116498.8250256, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788116999.5121062, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_gate_up_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788117902.656899, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_gate_up_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788118447.4135957, "occurrences": 1, "raw": "worker_1-glm_tp8_q_b_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788118805.7853494, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_gate_up_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788118963.8368917, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788119350.561405, "occurrences": 1, "raw": "worker_1-glm_tp8_q_b_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788119726.5515392, "occurrences": 1, "raw": "worker_2-glm_tp8_o_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788120253.7194333, "occurrences": 1, "raw": "worker_1-glm_tp8_q_b_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788120666.11223, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_gate_up_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788120685.7136302, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788120724.8631632, "occurrences": 1, "raw": "worker_2-glm_tp8_o_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788121588.8768733, "occurrences": 1, "raw": "worker_0-glm_tp8_fused_qkv_a_proj_m4096-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788121665.4604042, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_down_proj_m4096-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788121915.2489867, "occurrences": 1, "raw": "worker_1-glm_tp8_q_b_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788125463.5608711, "occurrences": 1, "raw": "worker_3-glm_tp8_shared_down_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788130841.3778317, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788133309.5794642, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788134212.7798412, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788136467.2958114, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m4096-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788137164.644181, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m4096-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "run_id": "glm5-2-dsh-tp8-m4096-1-e6a280a2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788139133.9862375, "occurrences": 1, "raw": "worker_1-glm_tp8_kv_b_proj_m4096-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787574457.0351176, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter6-repair2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787574519.4238868, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m16-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787578934.7088852, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787581335.4273891, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787585835.510433, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "run_id": "hy3-dsh-tp4-m16-1-7f1fb1d1", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787586738.7050598, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787680818.3226888, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m16-iter3-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787687563.3196526, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787688466.3875751, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m16-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787691970.3891685, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787692873.5204475, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787693776.6603217, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787694679.7959783, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787695582.960271, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter22 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp4-m16-2-368c654c", "run_id": "hy3-dsh-tp4-m16-2-368c654c", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787696486.1178405, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m16-iter23 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787728737.053402, "occurrences": 1, "raw": "worker_2-tp8_wo_b_m16-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787732128.9851558, "occurrences": 1, "raw": "worker_0-tp8_wqkv_a_m16-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787732657.478598, "occurrences": 1, "raw": "worker_2-tp8_wo_b_m16-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787733032.0391212, "occurrences": 1, "raw": "worker_0-tp8_wqkv_a_m16-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787735737.0466871, "occurrences": 1, "raw": "worker_0-tp8_wqkv_a_m16-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787736640.1629536, "occurrences": 1, "raw": "worker_0-tp8_wqkv_a_m16-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787738660.0880985, "occurrences": 1, "raw": "worker_1-tp8_wq_b_m16-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787738996.109836, "occurrences": 1, "raw": "worker_2-tp8_wo_b_m16-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-1-adf021ab", "run_id": "hy3-dsh-tp8-m16-1-adf021ab", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787739227.9300969, "occurrences": 1, "raw": "worker_0-tp8_wqkv_a_m16-iter15-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787745508.2111301, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787745796.9800684, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787746700.0844424, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787746882.2559333, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787750228.9554598, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787750376.2004251, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787753305.7298133, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787754208.8615506, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787754710.0493317, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787755283.1153228, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787755613.115135, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787757007.6671262, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787757314.2477255, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787758217.3385515, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787759033.4604352, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-2-dc53295a", "run_id": "hy3-dsh-tp8-m16-2-dc53295a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787759120.399777, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788885806.4156728, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788886533.9887652, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788887402.6489935, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788887671.5530374, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788888240.3840954, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788890298.3398073, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788891722.9052222, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m16-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788892293.5052817, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter11-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788892896.264094, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788893289.0204368, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788894192.1290386, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788894428.7939332, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788896611.6824756, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788897127.8991883, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788897849.4519126, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788898360.495783, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m16-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788899263.6607034, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788899596.2453494, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788900630.4452195, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788901533.6095986, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788901989.0253177, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m16-iter22 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788902058.313712, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter24 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788903111.1802857, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter23 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788904516.1826916, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter24 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788904781.4449365, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter25 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788905667.3695593, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m16-iter26 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m16-9-8-0161e718", "run_id": "hy3-dsh-tp8-m16-9-8-0161e718", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788907104.0389645, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter30 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787762771.159863, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787763741.0450513, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787763764.8166113, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787764212.4243498, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787765672.5227344, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787766174.5641525, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787768089.2998128, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787768585.5110054, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787770381.8509474, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter10-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787771199.7347968, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787771260.0590007, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787771530.73313, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787772898.92155, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787773482.4843204, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787774436.821065, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787774633.235294, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787774669.3775158, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787775354.5286813, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787775630.6607025, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787776257.5698168, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787779570.203486, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "run_id": "hy3-dsh-tp8-m4096-1-0ebb994d", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787780301.0895889, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter22 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787330133.3220432, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m4096-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787330971.1762033, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_down_proj_m4096-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787331874.27763, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_down_proj_m4096-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787332265.5458734, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787332480.0610049, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787333168.6630106, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m4096-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787334071.7691936, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787334164.938052, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_down_proj_m4096-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787335133.1992915, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787335731.0086303, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787335961.7311177, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_down_proj_m4096-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787336859.4715426, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787337690.041209, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787337762.4851205, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787338665.5476255, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787339585.669471, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_down_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787340578.3541567, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787341460.314264, "occurrences": 1, "raw": "worker_3-hy3_tp4_shared_down_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787341481.4151182, "occurrences": 1, "raw": "worker_2-hy3_tp4_shared_gate_up_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787343735.429806, "occurrences": 1, "raw": "worker_1-hy3_tp4_o_proj_m4096-iter19-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787402945.4193828, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787403848.4562702, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787404751.4952207, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787408188.1735258, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787409986.52433, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "run_id": "hy3-tp4-m4096-dsh-test1-bfdf3002", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787412447.6514893, "occurrences": 1, "raw": "worker_0-hy3_tp4_qkv_proj_m4096-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788436675.5133948, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788439208.3227084, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788440111.44199, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788440146.205755, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788440714.2033908, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788441049.3268147, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788441952.4428375, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788442106.0947404, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788443553.5201573, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788443718.3186297, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788444456.6441672, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788445312.3637762, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788446215.4761727, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788447118.5783646, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788447531.0283144, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788447816.864694, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788448434.1408262, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m16-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788448434.9915369, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788449338.0394247, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788449437.4890194, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788453496.9464698, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788454205.3961008, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m16-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788455203.6741943, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788455421.1124237, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m16-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788457913.4200406, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788458816.5303583, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788459153.8792963, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788459793.4936898, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788460056.967801, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788460405.7716377, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788461604.956796, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788462217.927129, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788462695.6420248, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788463121.1026068, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter22 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788463598.7602267, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788464024.2703912, "occurrences": 1, "raw": "worker_2-hy3_tp8_shared_gate_up_proj_m4096-iter23 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1788464326.7370625, "occurrences": 3, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788465174.549022, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788465351.553597, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788466077.700759, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788466100.0254426, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788466254.6872842, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788466980.8516192, "occurrences": 1, "raw": "worker_3-hy3_tp8_shared_down_proj_m4096-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788467003.1923087, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788467800.1800528, "occurrences": 1, "raw": "worker_1-hy3_tp8_o_proj_m4096-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "run_id": "hy3-tp8-dsh-m16-and-m4096-2-f7fa9503", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1788467906.3548787, "occurrences": 1, "raw": "worker_0-hy3_tp8_qkv_proj_m4096-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789411504.116792, "occurrences": 1, "raw": "worker_0-hy3_tp8_indexer_wq_b_m16-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789412407.0681405, "occurrences": 1, "raw": "worker_0-hy3_tp8_indexer_wq_b_m16-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789413310.0446332, "occurrences": 1, "raw": "worker_0-hy3_tp8_indexer_wq_b_m16-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789414212.9901197, "occurrences": 1, "raw": "worker_0-hy3_tp8_indexer_wq_b_m16-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789415115.974221, "occurrences": 1, "raw": "worker_0-hy3_tp8_indexer_wq_b_m16-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789416018.960905, "occurrences": 1, "raw": "worker_0-hy3_tp8_indexer_wq_b_m16-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_indexer_wq_b_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-indexer-wq-b-m16-75f702", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1789416321.3114793, "occurrences": 2, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_o_proj_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-o-proj-m16-68c35a", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_skill_fallback", "ts": 1789410880.2223895, "occurrences": 2, "raw": "killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m16-ahe-v2-ea5b8069:iteration_004/hy3_tp8_wqkv_a_m16", "run_id": "ahe-hy3-tp8-m16-ahe-v2-ea5b8069-it004-candidate-hy3-tp8-wqkv-a-m16-efecfd", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1789460376.0947528, "occurrences": 1, "raw": "worker_2-hy3_tp8_wqkv_a_m16-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791314153.8422494, "occurrences": 1, "raw": "worker_6-hy3_tp8_shared_gate_up_proj_m4096-iter6-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791314154.4497643, "occurrences": 1, "raw": "worker_7-hy3_tp8_shared_down_proj_m4096-iter14-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "run_id": "hy3-tp8-m8-16-4096-2-9dcfce15", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791314155.209133, "occurrences": 1, "raw": "worker_3-hy3_tp8_o_proj_m4096-iter8-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787822350.7097085, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m16-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787822746.7566803, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m16-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787824016.469317, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787824365.2615306, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787824919.538211, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m16-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787825268.3037443, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m16-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787826171.3742833, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m16-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787832574.6248436, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m16-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787834173.2962184, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m16-iter20 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787834590.8736877, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787835957.8345978, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m16-iter22 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787840983.9606674, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m16-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787844968.4187193, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m16-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787849389.084162, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m16-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787850292.164017, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m16-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787851195.2506964, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m16-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-dsh-tp8-m16-1-78402260", "run_id": "minimax-dsh-tp8-m16-1-78402260", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787852098.33462, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m16-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791397395.8810244, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m8-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791399481.1912277, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m8-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791420876.466791, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m8-iter18 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimax-tp8-m8-16-4096-df6fb501", "run_id": "minimax-tp8-m8-16-4096-df6fb501", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1791426238.2863076, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter7 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787459751.4385138, "occurrences": 1, "raw": "worker_2-minimax_tp4_o_proj_m4096-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787460654.464638, "occurrences": 1, "raw": "worker_2-minimax_tp4_o_proj_m4096-iter4 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787469167.1212335, "occurrences": 1, "raw": "worker_2-minimax_tp4_o_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787471430.38305, "occurrences": 1, "raw": "worker_0-minimax_tp4_qkv_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787471817.145112, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787474023.3792024, "occurrences": 1, "raw": "worker_0-minimax_tp4_qkv_proj_m4096-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787474043.1319911, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter8 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787474926.3836458, "occurrences": 1, "raw": "worker_0-minimax_tp4_qkv_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787475753.04715, "occurrences": 1, "raw": "worker_1-minimax_tp4_qkv_proj_and_indexer_qk_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787475778.8675523, "occurrences": 1, "raw": "worker_3-minimax_tp4_shared_gate_up_proj_m4096-iter9-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "run_id": "minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787475829.4404302, "occurrences": 1, "raw": "worker_0-minimax_tp4_qkv_proj_m4096-iter10 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787907996.9177577, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m4096-iter2-repair1 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787908081.1306436, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter2 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787908984.2398531, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter3 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787909161.127291, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787910539.184811, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787911383.6412964, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m4096-iter6 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787912141.3910704, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m4096-iter9 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787913747.6932247, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter11 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787914284.8972495, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787914746.5505393, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787915744.091096, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787916526.1992366, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787916703.231273, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787916741.7065165, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787916916.8810394, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter13 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787917738.8032758, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787917914.092557, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter14 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787918736.6431787, "occurrences": 1, "raw": "worker_0-minimax_tp8_qkv_proj_m4096-iter16 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787918912.9777703, "occurrences": 1, "raw": "worker_1-minimax_tp8_qkv_proj_and_indexer_qk_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787919137.9455497, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m4096-iter17 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787919494.2187412, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m4096-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787920932.4940386, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m4096-iter19 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787922066.4702458, "occurrences": 1, "raw": "worker_2-minimax_tp8_o_proj_m4096-iter22 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787922648.3070118, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_gate_up_proj_m4096-iter21 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787925758.560179, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m4096-iter5 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787930563.3638773, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m4096-iter12 failed: killed (see events log tail)"} +{"class_id": "PROC-AGENT-KILLED", "task_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "run_id": "minimaxm3-dsh-tp8-m4096-1-b0482833", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_iteration_failed", "ts": 1787933253.9964492, "occurrences": 1, "raw": "worker_3-minimax_tp8_shared_down_proj_m4096-iter15 failed: killed (see events log tail)"} +{"class_id": "PROC-NO-WORKER", "task_id": "dcu-kernel-auto-opt-7892abc2", "run_id": "dcu-kernel-auto-opt-7892abc2", "task_type": "dcu-kernel-auto-opt", "source": "timeline.jsonl", "event": "worker_restart_failed", "ts": 1785746459.4578338, "occurrences": 1, "raw": "RuntimeError('worker bootstrap has fewer than one successful GPU worker')"} diff --git a/metainfer/tasks/dcu_kernel_auto_opt/form.yaml b/metainfer/tasks/dcu_kernel_auto_opt/form.yaml index ae870e84..064b3117 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/form.yaml +++ b/metainfer/tasks/dcu_kernel_auto_opt/form.yaml @@ -153,7 +153,7 @@ - label: "AI automatic" description: "The control plane deterministically balances exact shapes across available GPUs" - label: "Manual by GPU" - description: "Use four GPU cards below to explicitly choose where every shape is optimized" + description: "Use up to four lanes below and pick the card each one runs on (any card this machine exposes)" - key: shape_scope question: "Choose whether this task optimizes the complete API workload or only a selected subset." @@ -173,6 +173,9 @@ required: false form: textarea default: "" + # The catalog renders full operator signatures; in a half-width column every + # row was clipped, which made picking a shape guesswork. + full_row: true override_component: "shape-input" override_module: "app/dkao-shape-input" diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/api_contracts.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/api_contracts.py index fe06262b..feb5669f 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/api_contracts.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/api_contracts.py @@ -95,8 +95,16 @@ def resolve_operator_api( ) # Reference variant HIP code lives in the plugin's variant/ directory; # fall back to the legacy location next to the API file. The fine-grained - # variant TREE (variant/////.hip) is - # staged alongside the legacy single file so agents can navigate both. + # variant TREE is staged alongside the legacy single file so agents can + # navigate both. + # + # Variants are stored per hardware profile + # (variant//////.hip) because a + # median measured on one machine is meaningless on the other. Only the + # CURRENT machine's profile is staged, and it is staged one level shallower + # (``references//...``) so the agent-facing paths the prompts + # advertise do not change. A pre-split tree (families directly under + # variant/) is still staged as-is so an unmigrated checkout keeps working. variant_root = Path(__file__).resolve().parents[1] / "variant" legacy = next( (path for path in (variant_root / W8A8_VARIANTS_FILENAME,) @@ -107,11 +115,16 @@ def resolve_operator_api( variant_sources: tuple = () if legacy is not None: variant_sources += (legacy,) - # every subdirectory of variant/ is a fine-grained variant family tree - # (e.g. int8w8a8-gemm/hy3/TP4/M4096/o_proj.hip) if variant_root.is_dir(): + from . import variant_store as _variant_store + + staged_root = _variant_store.profile_root() + if not staged_root.is_dir(): + # Legacy layout: every subdirectory of variant/ is a family tree. + staged_root = variant_root variant_sources += tuple( - path for path in sorted(variant_root.iterdir()) if path.is_dir() + path for path in sorted(staged_root.iterdir()) + if path.is_dir() and path.name not in _variant_store.hw_profile.PROFILES ) return OperatorAPIContract( operator=operator, diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/config.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/config.py index b32662d6..49cedb0f 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/config.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/config.py @@ -9,6 +9,8 @@ import yaml +from metainfer.orchestrator import gpu_inventory + MOCK_MODE = "Mock (no GPU)" LEGACY_SMOKE_MODE = "Real agents + DCU (smoke harness)" SMOKE_MODE = "Infrastructure smoke (not operator optimization)" @@ -283,6 +285,40 @@ def _parse_shape_config(raw: Any) -> Dict[str, Any]: return parsed +def detected_hardware_label(answers: Mapping[str, Any]) -> str: + """The board this task runs on -- detected, not taken from the form. + + ``target_hardware`` is a form field, and it was wrong in exactly the way + that matters: every task launched by HE on the air board carried + ``K500SM_AI / gfx928`` (the liquid card), so generated skills, task records + and reports named the wrong machine -- and the hardware-dependent knowledge + they fed was filed under the wrong profile. The resolver's label is + authoritative; the form value survives only as a fallback for a machine + whose fingerprint is not registered yet. + """ + try: + from metainfer.orchestrator import hw_profile + + detected = hw_profile.label(fallback="") + except Exception: # pragma: no cover - config must load regardless + detected = "" + return detected or str(answers.get("target_hardware") or "Other DCU") + + +def _gpu_range_error(gpu: int) -> str: + """One message for an unusable card number, shared by both parsers. + + ``gpu`` is a host HCU ordinal (``hy-smi`` ``DCU[n]`` == the worker's + ``HIP_VISIBLE_DEVICES=n``), so the allowed set is this machine's detected + device list: eight entries on the air board, four on liquid. + """ + listed = ",".join(str(d) for d in gpu_inventory.devices()) + return ( + f"gpu must be one of {listed} " + f"({gpu_inventory.count()} HCU on this machine), got {gpu}" + ) + + def load_config(req: Mapping[str, Any]) -> OptimizerConfig: answers = _answers(req) agent_framework = resolve_agent_framework(req) @@ -381,6 +417,9 @@ def load_config(req: Mapping[str, Any]) -> OptimizerConfig: if not assignments_omitted: if not isinstance(raw_assignments, Mapping) or not raw_assignments: raise ValueError("shape_config.assignments must be a non-empty object") + # At most four workers (unchanged): the worker count is policy, not a + # hardware fact. Which four *cards* they may use is discovered instead -- + # see gpu_inventory. if isinstance(raw_assignments, Mapping) and len(raw_assignments) > 4: raise ValueError("worker29 MVP supports at most four workers") @@ -402,8 +441,10 @@ def load_config(req: Mapping[str, Any]) -> OptimizerConfig: if not isinstance(item, Mapping): raise ValueError(f"assignment {worker_id!r} must be an object") gpu = int(item.get("gpu", -1)) - if gpu not in range(4): - raise ValueError(f"{worker_id}: gpu must be one of 0,1,2,3") + if not gpu_inventory.is_addressable(gpu): + raise ValueError( + f"{worker_id}: {_gpu_range_error(gpu)}" + ) if gpu in seen_gpus: raise ValueError(f"physical GPU {gpu} is assigned more than once") seen_gpus.add(gpu) @@ -471,11 +512,32 @@ def load_config(req: Mapping[str, Any]) -> OptimizerConfig: assignments, lease_info = lease_gpus_for_task( assignments, answers, production=production) assignment_mode = "scheduled" + elif assignment_mode == "manual": + # Occupy + manual is the mode where the operator's placement is + # authoritative, and the generate phase honours it by **name**: every + # worker it creates is `worker_` and it rejects anything else + # (``gen_and_opt_pipeline._require_valid_child_assignments``), which is + # also how harness_evolve names its children. An assignment that does + # not follow the rule used to die inside generate_kernel_repo, minutes + # in; refusing it here says what to fix instead. + invalid = { + item.worker_id: item.gpu + for item in assignments + if item.worker_id != f"worker_{item.gpu}" + } + if invalid: + expected = { + f"worker_{item.gpu}": item.gpu for item in assignments + } + raise ValueError( + "manual assignments must name each worker after the card it " + f"runs on (worker_): {invalid} -> expected {expected}" + ) return OptimizerConfig( operator=str(answers.get("operator") or "Custom operator"), dtype=str(answers.get("dtype") or "Other"), - hardware=str(answers.get("target_hardware") or "Other DCU"), + hardware=detected_hardware_label(answers), kernel_language=str(answers.get("kernel_language") or "HIP C++"), claude_model=claude_model, execution_mode=mode, @@ -628,8 +690,8 @@ def validate_gpu_assignment( if not isinstance(item, dict): raise ValueError(f"{worker_id}: must be an object") gpu = int(item.get("gpu", -1)) - if gpu not in range(4): - raise ValueError(f"{worker_id}: gpu must be 0-3, got {gpu}") + if not gpu_inventory.is_addressable(gpu): + raise ValueError(f"{worker_id}: {_gpu_range_error(gpu)}") if gpu in seen_gpus: raise ValueError(f"GPU {gpu} assigned to more than one worker") seen_gpus.add(gpu) diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/final_layout_gate.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/final_layout_gate.py new file mode 100644 index 00000000..06ace406 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/final_layout_gate.py @@ -0,0 +1,641 @@ +"""Packed-weight layout gate for the final merge of a W8A8 GEMM task. + +Why this file exists +-------------------- +``prepare_weight`` is frozen at ``(raw_weight, weight_scale)``: it has **no M**, +because in the real model one weight is packed once and every M of that weight +reuses the image. The dispatch therefore cannot give the M=8 and the M=4096 GEMM +different weight bytes, while the worker contract explicitly lets each shape +bring its own packed layout ("may change only its HIP implementation **and the +matching GEMM interpretation** to test packed layouts"). + +When the members of one ``(N, K)`` family disagree, the merged artifact packs +with one member and interprets with another, and every element of the output is +wrong. That failure mode has landed twice: + +* ``hy3-tp8-m8-16-4096-2-9dcfce15`` -- m8/m16 shared a layout, m4096 changed it; + the three pack sources were byte-identical, so nothing static could see it. +* ``minimax-tp8-m8-16-4096-df6fb501`` -- m8 emitted a k32-blocked image without + the K-octet swizzle, m4096 emitted the swizzled one; 99.9% of the m4096 output + was wrong while its latency stayed exactly the worker's number (the cheap + kernel ran and produced garbage). + +Both were found only by the *full* serial validation, i.e. after a task had +already spent hours optimizing: the merge is the first place the two layouts +meet, and it had no check of any kind. + +What the gate measures +---------------------- +For each ``(N, K)`` family it runs **each member's own pack kernel**, in a child +process, on the family's **real weight shape** filled with a byte-counter image, +and hashes the packed bytes: + + fingerprint = SHA256(pack_member(counter_image_of_this_weight)) + +Two facts make this the right measurement: + +* It is **behavioural**: the hy3 pack sources were byte-identical yet disagreed, + so comparing source or symbol names would have missed that incident entirely. +* It must run at the **real (N, K)**: probed at a small canonical shape the two + df6fb501 packers agree (both wrote the same 32 KB block), while at the real + ``(1280, 6144)`` they differ -- the swizzle only engages on the full k32 grid. + Measured: 0.9 ms per 7.9 MB image, 8.3 ms for a 235 MB one, and never inside a + timed region. + +Interpretation: **equal fingerprints means byte-identical images**, so those +members are mutually interchangeable and may share one pack route. Different +fingerprints are treated as *incompatible*: the gate cannot prove they are +readable by each other, and the observed failure was exactly a reader that +accepted only its own image. That makes the verdict conservative -- a family +only pays an extra batch when its members really emit different bytes. + +The plan +-------- +A batch is a set of shapes merged into one extension. Inside a batch every +``(N, K)`` family has exactly one pack route, taken from the family's smallest +M, so a batch is sound exactly when all members it contains emit the same image +(:func:`family_groups`). Members that cannot share an image are grouped so that +each group is self-consistent, and the groups are packed into the fewest batches +(:func:`pack_batches`); the merged pipeline then validates one batch at a time, each +with its own dispatch and prebuilt set. + +If the probe cannot run (no objects on disk, no GPU, a crashed child) the gate +reports no conflict -- but it must not pretend the family agreed either. With +nothing measured, agreement is **unproven**, so every unmeasured multi-member +family is split by member (:func:`unproven_groups`, applied per family by +:func:`agreed_groups`): a blind gate then costs an extra batch instead of merging +two layouts into one pack route. A probe failure still never fails a task. + +The probe child is spawned through +:func:`metainfer.orchestrator.interpreter.python_executable`, never +``sys.executable``: inside a WebUI-launched orchestrator ``sys.executable`` is +``''``, and ``subprocess.run([''])`` raises +``PermissionError: [Errno 13] Permission denied: ''`` -- which the gate used to +absorb as "no conflict". See ``wiki-metainfer/gotchas/ +layout-gate-probe-blind-under-webui.md``. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import shutil +import subprocess +import tempfile +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple + +from metainfer.orchestrator.interpreter import python_executable + +__all__ = [ + "FamilyProbe", + "LayoutCheck", + "MemberProbe", + "UNRESOLVED", + "agreed_groups", + "check", + "family_groups", + "pack_batches", + "plan", + "unproven_groups", +] + +#: Sentinel for "this member's pack could not be measured". +UNRESOLVED = "unresolved" + + +def _counter_image(k: int, n: int) -> bytes: + """The probe weight: a byte counter. + + A counter -- not zeros, not a constant -- is what makes the image sensitive + to any in-image permutation (the k32 blocks, the K-octet swizzle, a row + stride): two layouts can only compare equal if they place every byte + identically. + """ + return bytes(i & 0xFF for i in range(k * n)) + + +_CHILD_CODE = r""" +import ctypes, hashlib, json, sys +from pathlib import Path + +import torch + +import metainfer.tasks.dcu_kernel_auto_opt.orchestrator.final_layout_gate as gate + +spec_path, out_path = sys.argv[1], sys.argv[2] +spec = json.loads(Path(spec_path).read_text()) + +# One GPU context for every family: a child per family would spend ~16 s each +# just initialising torch/HIP, against ~1 ms of actual packing. +torch.cuda.init() +stream = ctypes.c_void_p(torch.cuda.current_stream().cuda_stream) + +report = {"families": {}, "errors": {}} +for key, family in spec["families"].items(): + n, k = family["N"], family["K"] + raw = gate._materialize(gate._counter_image(k, n)) + zero = torch.zeros(n, dtype=torch.float32, device="cuda") + entry = {"fingerprints": {}, "errors": {}} + for shape_id, item in family["members"].items(): + try: + library = gate._load_probe_library( + Path(item["object"]), item["pack_symbol"] + ) + packed = torch.zeros(k * n, dtype=torch.int8, device="cuda") + packed_scale = torch.zeros(n, dtype=torch.float32, device="cuda") + gate._pack_fn(library, item["pack_symbol"])( + ctypes.c_void_p(raw.data_ptr()), + ctypes.c_void_p(zero.data_ptr()), + ctypes.c_void_p(packed.data_ptr()), + ctypes.c_void_p(packed_scale.data_ptr()), + k, n, stream, + ) + torch.cuda.synchronize() + entry["fingerprints"][shape_id] = hashlib.sha256( + packed.cpu().view(torch.uint8).numpy().tobytes() + ).hexdigest() + except Exception as exc: # noqa: BLE001 - one member must not kill it + entry["errors"][shape_id] = f"{type(exc).__name__}: {exc}"[:200] + report["families"][key] = entry + +Path(out_path).write_text(json.dumps(report)) +""" + + +def _materialize(blob: bytes, device: str = "cuda"): + """Wrap raw bytes as a device tensor (int8 bit pattern preserved).""" + import torch # noqa: PLC0415 - probe only + + return torch.frombuffer(bytearray(blob), dtype=torch.int8).to(device) + + +def _load_probe_library(object_path: Path, symbol: str): + """Link one namespaced prebuilt object into a loadable library.""" + import ctypes # noqa: PLC0415 - probe only + + scratch = Path(tempfile.mkdtemp(prefix="w8a8-layout-probe-")) + library_path = scratch / f"{symbol}.so" + result = subprocess.run( + [ + "gcc", "-shared", "-fPIC", "-o", str(library_path), + str(object_path), "-L/opt/dtk/lib", "-lamdhip64", + ], + capture_output=True, + text=True, + ) + if result.returncode != 0: + raise RuntimeError( + "probe link failed: " + + (result.stderr or result.stdout or "").strip()[-500:] + ) + library = ctypes.CDLL(str(library_path)) + # Keep the scratch dir referenced so the .so outlives this call. + library._dsh_probe_scratch = str(scratch) + return library + + +def _pack_fn(library, symbol: str): + """The trusted pack ABI: ``(raw, scale, packed, packed_scale, k, n, stream)``.""" + import ctypes # noqa: PLC0415 - probe only + + function = getattr(library, symbol) + function.restype = None + function.argtypes = [ + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, + ctypes.c_void_p, ctypes.c_int, ctypes.c_int, ctypes.c_void_p, + ] + return function + + +# --------------------------------------------------------------------------- # +# Results +# --------------------------------------------------------------------------- # + + +@dataclass +class MemberProbe: + """One member's packed image for the family's weight.""" + + shape_id: str + pack_symbol: str + fingerprint: str = UNRESOLVED + error: str = "" + + +@dataclass +class FamilyProbe: + """One ``(N, K)`` family, probed.""" + + n: int + k: int + members: List[MemberProbe] = field(default_factory=list) + #: ``shape_id -> M``: a family's pack route is its **smallest M**. + m_of: Dict[str, int] = field(default_factory=dict) + + def member(self, shape_id: str) -> MemberProbe: + for item in self.members: + if item.shape_id == shape_id: + return item + raise KeyError(shape_id) + + @property + def resolved(self) -> bool: + return any(item.fingerprint != UNRESOLVED for item in self.members) + + @property + def images(self) -> Dict[str, List[str]]: + """``fingerprint -> [shape_id, ...]``; unmeasured members kept together.""" + grouped: Dict[str, List[str]] = {} + for item in self.members: + grouped.setdefault(item.fingerprint, []).append(item.shape_id) + return {key: sorted(value) for key, value in sorted(grouped.items())} + + def as_json(self) -> Dict[str, Any]: + return { + "n": self.n, + "k": self.k, + "resolved": self.resolved, + "distinct_images": len(self.images), + "images": self.images, + "members": { + item.shape_id: { + "pack_symbol": item.pack_symbol, + "fingerprint": item.fingerprint, + "error": item.error, + } + for item in self.members + }, + } + + +@dataclass +class LayoutCheck: + """Probe result: one :class:`FamilyProbe` per ``(N, K)``.""" + + families: List[FamilyProbe] = field(default_factory=list) + probed: bool = False + probe_error: Optional[str] = None + + def family(self, n: int, k: int) -> Optional[FamilyProbe]: + for item in self.families: + if (item.n, item.k) == (n, k): + return item + return None + + @property + def conflicts(self) -> List[FamilyProbe]: + """Families whose members emit more than one distinct image.""" + return [ + family for family in self.families + if family.resolved and len(family.images) > 1 + ] + + @property + def ok(self) -> bool: + return not self.conflicts + + @property + def unproven(self) -> List[FamilyProbe]: + """Families whose packed image was never measured at all. + + Non-empty while :attr:`probed` can still be true: one family's members can + all fail to probe (a link error, say) while another family measures fine. + Those families must still be split by member -- the global ``probed`` flag + is not enough to decide per family. + """ + return [family for family in self.families if not family.resolved] + + def as_json(self) -> Dict[str, Any]: + return { + "probed": self.probed, + "probe_error": self.probe_error, + "conflicts": [ + {**item.as_json(), "groups": family_groups(item)} + for item in self.conflicts + ], + "families": [item.as_json() for item in self.families], + } + + +# --------------------------------------------------------------------------- # +# Probe driver +# --------------------------------------------------------------------------- # + + +def _probe_families( + families: Sequence[FamilyProbe], + artifacts: Sequence[Dict[str, Any]], + source_dir: Path, +) -> Dict[str, Any]: + """Fingerprint every family's members in one child process; never raises. + + Returns ``{"fingerprints": {shape_id: sha256}, "errors": {...}, + "probe_error": str}``. Unmeasured members are simply absent from + ``fingerprints``. + """ + by_shape = { + str(artifact["shape_id"]): artifact for artifact in artifacts + } + spec_families: Dict[str, Any] = {} + for family in families: + members: Dict[str, Dict[str, str]] = {} + for member in family.members: + artifact = by_shape.get(member.shape_id) + if artifact is None: + continue + object_path = source_dir / str(artifact.get("final_object") or "") + if not object_path.is_file(): + continue + members[member.shape_id] = { + "object": str(object_path), + "pack_symbol": member.pack_symbol, + } + if members: + spec_families[f"{family.n}x{family.k}"] = { + "N": family.n, + "K": family.k, + "members": members, + } + missing = { + member.shape_id: ( + "pack object missing: " + + str(source_dir / str( + (by_shape.get(member.shape_id) or {}).get("final_object") or "" + )) + ) + for family in families + for member in family.members + if member.shape_id not in { + shape_id + for entry in spec_families.values() + for shape_id in entry["members"] + } + } + if not spec_families: + return {"fingerprints": {}, "errors": missing, + "probe_error": "no pack object is available to probe"} + + scratch = Path(tempfile.mkdtemp(prefix="w8a8-family-probe-")) + spec_path = scratch / "spec.json" + out_path = scratch / "report.json" + spec_path.write_text( + json.dumps({"families": spec_families}), encoding="utf-8" + ) + env = dict(os.environ) + repo_root = str(Path(__file__).resolve().parents[3]) + env["PYTHONPATH"] = os.pathsep.join( + [repo_root, env.get("PYTHONPATH", "")] + ).strip(os.pathsep) + failure = "" + report: Dict[str, Any] = {} + interpreter = python_executable() + try: + result = subprocess.run( + [interpreter, "-c", _CHILD_CODE, str(spec_path), str(out_path)], + capture_output=True, + text=True, + timeout=1800, + env=env, + ) + if result.returncode != 0 or not out_path.is_file(): + failure = (result.stderr or result.stdout or "").strip()[-300:] + else: + report = json.loads(out_path.read_text(encoding="utf-8")) + except (OSError, ValueError, subprocess.SubprocessError) as exc: + # Name the interpreter: ``PermissionError: [Errno 13] Permission + # denied: ''`` (an empty sys.executable) says nothing on its own. + failure = f"{type(exc).__name__}: {exc} (interpreter={interpreter!r})"[:300] + finally: + shutil.rmtree(scratch, ignore_errors=True) + + fingerprints: Dict[str, str] = {} + errors: Dict[str, str] = dict(missing) + for entry in (report.get("families") or {}).values(): + fingerprints.update(entry.get("fingerprints") or {}) + errors.update(entry.get("errors") or {}) + if failure: + for family in families: + for member in family.members: + errors.setdefault(member.shape_id, failure) + return { + "fingerprints": fingerprints, + "errors": errors, + "probe_error": failure, + } + + +def check( + artifacts: Sequence[Dict[str, Any]], + source_dir: Path, + *, + probe: bool = True, + probe_families: Optional[Callable[..., Dict[str, Any]]] = None, +) -> LayoutCheck: + """Probe every ``(N, K)`` family of one merge. + + ``probe_families`` is injectable so the gate can be exercised without a GPU. + """ + grouped: Dict[Tuple[int, int], List[Dict[str, Any]]] = {} + for artifact in artifacts: + shape = artifact.get("shape") or {} + grouped.setdefault((int(shape["N"]), int(shape["K"])), []).append( + artifact + ) + result = LayoutCheck() + if not grouped: + return result + + families: List[FamilyProbe] = [] + for key, members in sorted(grouped.items()): + ordered = sorted(members, key=lambda item: str(item["shape_id"])) + families.append(FamilyProbe( + n=key[0], + k=key[1], + members=[ + MemberProbe( + shape_id=str(item["shape_id"]), + pack_symbol=str(item.get("pack_symbol") or ""), + error="" if probe else "probe disabled", + ) + for item in ordered + ], + m_of={ + str(item["shape_id"]): int( + (item.get("shape") or {}).get("M") or 0 + ) + for item in ordered + }, + )) + result.families = families + if not probe: + return result + + runner = probe_families or _probe_families + payload = runner( + [family for family in families], + artifacts, + source_dir, + ) + fingerprints = payload.get("fingerprints") or {} + errors = payload.get("errors") or {} + for family in families: + for member in family.members: + digest = fingerprints.get(member.shape_id) + member.fingerprint = str(digest) if digest else UNRESOLVED + member.error = str(errors.get(member.shape_id) or "") + result.probed = any(family.resolved for family in families) + if not result.probed: + error = payload.get("probe_error") or "no family could be probed" + result.probe_error = str(error) + return result + + +# --------------------------------------------------------------------------- # +# Planning +# --------------------------------------------------------------------------- # + + +def family_groups(family: FamilyProbe) -> List[List[str]]: + """The groups of members that can share one pack route. + + Grouping is by packed image: members that emit byte-identical images are + interchangeable, and members that emit different images get their own group + (a different image is not proof of incompatibility, but it is the shape of + the incident, and the merge cannot prove the reader accepts it). + + Groups come out smallest-M first: that is the order the dispatcher's pack + route follows, so the leading group is the one it will actually serve. + """ + def by_m(shape_id: str): + return (family.m_of.get(shape_id, 0), shape_id) + + return sorted( + (sorted(members) for members in family.images.values()), + key=lambda group: by_m(group[0]), + ) + + +def unproven_groups(family: FamilyProbe) -> List[List[str]]: + """The groups to use when this family's images were never measured. + + A blind probe proves nothing, so every member of a multi-member family gets + its own group: the merge then gives each M its own pack route, which is the + layout the shape's own kernel was tuned and validated against. Single-member + families are unaffected, so a task whose shapes never shared a weight pays + nothing. + + Same ordering contract as :func:`family_groups` (smallest M first), so batch + packing stays deterministic and batch 0 still leads with the decode shapes. + """ + if len(family.members) <= 1: + return [[member.shape_id for member in family.members]] + + def by_m(member: MemberProbe): + return (family.m_of.get(member.shape_id, 0), member.shape_id) + + return [[member.shape_id] for member in sorted(family.members, key=by_m)] + + +def agreed_groups(family: FamilyProbe) -> List[List[str]]: + """The grouping rule :func:`plan` uses once at least one family is blind. + + Per family: measured families keep the fingerprint rule, unmeasured ones fall + back to :func:`unproven_groups`. This is why the decision cannot be made from + the single global ``probed`` flag -- a probe run can measure every family but + one, and that one family would otherwise still share a pack route. + """ + if family.resolved: + return family_groups(family) + return unproven_groups(family) + + +def pack_batches( + artifacts: Sequence[Dict[str, Any]], + families: Sequence[FamilyProbe], + *, + groups: Callable[[FamilyProbe], List[List[str]]] = family_groups, +) -> List[List[Dict[str, Any]]]: + """Pack the groups into the fewest batches. + + A batch may hold any number of families but at most one group per family, so + within a batch every ``(N, K)`` has a single, self-consistent pack route. + Groups are placed largest-first into the earliest batch that still lacks + their family: minimal batch count, deterministic result, and the published + batch 0 leads with the decode shapes. + + ``groups`` is the grouping rule: :func:`family_groups` when the probe + measured the images, :func:`unproven_groups` when it could not. + """ + by_shape = { + str(artifact["shape_id"]): artifact for artifact in artifacts + } + rows: List[Tuple[Tuple[int, int], List[str], int]] = [] + for family in families: + for group in groups(family): + first_m = min( + (family.m_of.get(shape_id, 0) for shape_id in group), + default=0, + ) + rows.append(((family.n, family.k), group, first_m)) + if not rows: + return [list(artifacts)] + rows.sort(key=lambda item: (-len(item[1]), item[0], item[2], item[1][0])) + + batches: List[List[str]] = [] + seen: List[set] = [] + for key, group, _first_m in rows: + target = next( + ( + index for index, families_seen in enumerate(seen) + if key not in families_seen + ), + None, + ) + if target is None: + batches.append([]) + seen.append(set()) + target = len(batches) - 1 + batches[target].extend(group) + seen[target].add(key) + + return [ + [by_shape[shape_id] for shape_id in sorted(batch) if shape_id in by_shape] + for batch in batches + ] + + +def plan( + artifacts: Sequence[Dict[str, Any]], + source_dir: Path, + *, + probe: bool = True, +) -> Tuple[LayoutCheck, List[List[Dict[str, Any]]]]: + """Probe the families and return the batches the merge should validate. + + Three outcomes: + + * members of a family emit **different** images (:attr:`LayoutCheck.conflicts`) + -- one batch per self-consistent group; + * the probe **measured nothing** (``probed`` is false: no objects, no GPU, a + crashed child, or ``probe=False``) -- every multi-member family is split by + member, because agreement is unproven and a shared pack route would be a + guess. Unlike a measured conflict this is *not* reported as one + (:attr:`LayoutCheck.ok` stays true), so a broken probe still cannot fail a + task: it can only cost an extra batch; + * every family agrees on one image -- the single historical batch. + + A probe run can also leave *some* families blind while others measure fine + (:attr:`LayoutCheck.unproven`); those families get the same conservative + treatment through :func:`agreed_groups`. + """ + result = check(artifacts, source_dir, probe=probe) + if result.conflicts: + return result, pack_batches(artifacts, result.families) + if result.unproven: + return result, pack_batches( + artifacts, result.families, groups=agreed_groups + ) + return result, [list(artifacts)] diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gen_and_opt_pipeline.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gen_and_opt_pipeline.py index 3e0bc0cf..d821f320 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gen_and_opt_pipeline.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gen_and_opt_pipeline.py @@ -49,9 +49,14 @@ ) from .gpu_binding import bind_worker_gpu from .harness_io import harness_root, load_manifest, seed_workspace +from .gpu_probe import run_gpu_probe from .real_pipeline import _run, _safe, _status from .result_store import SCHEMA_VERSION, write_json -from .skill_store import generate_merged_skill, generate_worker_skill +from .skill_store import ( + generate_merged_skill, + generate_worker_skill, + sync_skill_libraries, +) from .w8a8_pipeline import ( RealW8A8OptimizationPipeline, W8A8Runner, @@ -63,6 +68,7 @@ snapshot_accepted_kernel_artifact, ) from . import gate_policy as _gates +from . import final_layout_gate from .validation_budget import ( resolve_bench_kwargs, resolve_validation_scope, ) @@ -75,12 +81,34 @@ _MAX_SYNTHESIS_RETRIES = 3 # µs-scale decode kernels are one-sided noisy on shared GPUs: transient # co-tenant load / clock / thermal state only inflates the median, never -# deflates it. When the final performance gate (final <= 1.05 x worker best) -# trips with no source change, re-measure up to _PERF_GATE_MAX_RETRIES times, -# waiting _PERF_GATE_RETRY_INTERVAL_S between attempts, and accept the best -# (min) median; only fail when every attempt still exceeds the gate. +# deflates it. When the final performance gate (final <= _perf_gate_limit(worker +# best), i.e. best + max(5%, _PERF_GATE_MIN_SLACK_US)) trips with no source +# change, re-measure up to _PERF_GATE_MAX_RETRIES times, waiting +# _PERF_GATE_RETRY_INTERVAL_S between attempts, and accept the best (min) +# median; only fail when every attempt still exceeds the gate. _PERF_GATE_MAX_RETRIES = 3 _PERF_GATE_RETRY_INTERVAL_S = 300 + +#: Absolute floor under the acceptance window: "5%, or this, whichever is wider". +#: +#: A percentage alone is wrong at µs scale. 5% of a 9.6 µs decode kernel is +#: 0.48 µs, which is below what this box can resolve: one and the same accepted +#: object was measured 9.638 µs during EXPLORE and 10.866/10.998 µs in the +#: serial validation (+12.7%/+14.1%), so a 5%-only window turns the machine's +#: own measurement spread into a failed task. The floor fixes the small end and +#: leaves everything else alone: the absolute term only wins below a best of +#: 40 µs (where 5% < 2 µs), so every larger shape keeps exactly the old window +#: — and because the crossover falls out of the two terms there is no band +#: boundary to straddle. +_PERF_GATE_MIN_SLACK_US = 2.0 + + +def _perf_gate_limit(best_median: float) -> float: + """Largest serial median accepted for one shape against its worker best.""" + return float(best_median) + max( + 0.05 * float(best_median), _PERF_GATE_MIN_SLACK_US + ) + _PMC_PLAN = { "script": "profile_pmc.sh", "mode": "hipprof_pmc_csv", @@ -309,16 +337,17 @@ def _validate_generate_scaffold( # device 0 happened to survive). bind_worker_gpu(probe_env, probe_gpu) probe_env["PYTHONDONTWRITEBYTECODE"] = "1" - probe_result = _run( - [ - "python3", str(harness), - "--source", str(source), - "--m", "1", "--n", "1", "--k", "1", - "--probe", - ], - cwd=source, + # A broken ROCm/amdsmi handshake makes this probe fail for a few seconds at + # a time ("shm_open: Permission denied"). One such blip must not kill a + # multi-hour task, so the probe is retried here; the assertions below are + # unchanged, so a genuinely unusable device still fails the preflight. + probe_result = run_gpu_probe( + harness=harness, + source=source, env=probe_env, + runner=_run, timeout=120, + state_dir=state_dir, ) gpu_probe = _last_json_object(probe_result.stdout) if gpu_probe.get("visible_devices") != 1: @@ -461,7 +490,8 @@ def _final_synthesis_prompt( - wavefront=64, blockDim a multiple of 64, current stream only; - DUMMA INT8 is m16n16k32 with int32 accumulation on gfx928; installed DTK uses `` and namespace `du::dumma`; -- final per-shape median must stay within 5% of that worker's measured best; +- final per-shape median must stay within 5% of that worker's measured best + (or 2 µs, whichever is wider, for shapes under 40 µs); - do not run Docker, the harness, compilation, benchmarks, or environment probes; trusted control plane validation runs after you return. {failure_block} @@ -484,6 +514,11 @@ def _namespace_prebuilt_object( shape_id: str, ) -> Dict[str, Any]: """Give every defined host symbol a shape namespace without recompiling.""" + # ``subprocess`` resolves ``cwd`` against the *process* directory, so a + # relative artifact path would be looked up inside the wrong tree. Every + # path handed to nm/objcopy is absolute for that reason. + source_object = Path(source_object).resolve() + destination_object = Path(destination_object).resolve() destination_object.parent.mkdir(parents=True, exist_ok=True) shutil.copy2(source_object, destination_object) nm_output = _run( @@ -537,51 +572,267 @@ def _namespace_prebuilt_object( } +def _artifact_families( + artifacts: list[Dict[str, Any]], +) -> list[tuple[tuple[int, int], list[Dict[str, Any]]]]: + """Group artifacts by weight family ``(N, K)``, members sorted by ``M``. + + One operator requested at several M values produces several artifacts that + share ``(N, K)`` -- Hy3 TP8 qkv_proj is one weight at M=8, 16 and 4096 -- so + ``(N, K)`` identifies a *weight* and ``M`` picks the kernel for one request. + Only M=2/8/16/4096 are ever in a catalog. + """ + families: Dict[tuple, list] = {} + for artifact in artifacts: + shape = artifact["shape"] + families.setdefault((int(shape["N"]), int(shape["K"])), []).append( + artifact + ) + return [ + (pair, sorted(members, key=lambda item: int(item["shape"]["M"]))) + for pair, members in sorted(families.items()) + ] + + +def _dispatch_family_plan( + artifacts: list[Dict[str, Any]], +) -> list[Dict[str, Any]]: + """What the rendered dispatch will do, for the task timeline. + + Cheap to record and the first thing worth looking at when a final + validation disagrees with the worker numbers. + """ + plan = [] + for (n, k), members in _artifact_families(artifacts): + plan.append({ + "n": n, + "k": k, + "pack_from": members[0]["shape_id"], + "gemm_route": [ + {"m": int(item["shape"]["M"]), "shape_id": item["shape_id"]} + for item in sorted( + members, key=lambda entry: -int(entry["shape"]["M"]) + ) + ], + }) + return plan + + +def plan_layout_batches( + artifacts: list[Dict[str, Any]], + source: Path, +) -> tuple["final_layout_gate.LayoutCheck", list[list[Dict[str, Any]]]]: + """Probe the ``(N, K)`` families and plan the validation batches. + + Returns the probe result (recorded in the timeline, so a final validation + that disagrees with worker numbers can be read off the event stream) and the + batches to validate -- one containing everything when every family shares a + usable packed image, otherwise one per group that has to pack for itself. + """ + return final_layout_gate.plan(artifacts, source) + + +def sync_prebuilt_artifacts( + source: Path, + artifacts: list[Dict[str, Any]], +) -> None: + """Make the staged ``prebuilt`` set exactly this batch's artifacts. + + A merged extension that carries objects for shapes it never dispatches is + not wrong, but it is not the artifact either: stashing them keeps every + batch's extension minimal and its build key a pure function of what it + actually links. The cache source dir (when the runner already staged one) + is synced too, because it is a snapshot the loader would otherwise keep + serving with the previous dispatch. + """ + prebuilt = source / "prebuilt" + prebuilt.mkdir(parents=True, exist_ok=True) + wanted = { + str(item["final_object"]).split("/", 1)[-1] for item in artifacts + } + # The stash lives *beside* prebuilt, never inside it: a nested stash is + # itself matched by ``prebuilt/*.o`` and would be shuffled on every call. + stash = source / ".prebuilt_unused" + for path in sorted(prebuilt.glob("*.o")): + if path.name not in wanted: + stash.mkdir(parents=True, exist_ok=True) + path.replace(stash / path.name) + for name in wanted: + stashed = stash / name + if stashed.is_file() and not (prebuilt / name).exists(): + stashed.replace(prebuilt / name) + _sync_prebuilt_into_compile_cache(source, wanted) + + +def _sync_prebuilt_into_compile_cache(source: Path, wanted: set) -> None: + raw = os.environ.get("METAINFER_W8A8_COMPILE_SOURCE_DIR", "").strip() + if not raw: + return + cache = Path(raw) + cache_prebuilt = cache / "prebuilt" + if not cache_prebuilt.is_dir(): + return + try: + if cache.resolve() == (source / "prebuilt").resolve(): + return + except OSError: + return + for path in sorted(cache_prebuilt.glob("*.o")): + if path.name not in wanted: + path.unlink() + for name in wanted: + origin = source / "prebuilt" / name + destination = cache_prebuilt / name + if origin.is_file() and not destination.is_file(): + try: + os.link(origin, destination) + except OSError: + shutil.copy2(origin, destination) + + +def _layout_batch_source( + source: Path, + workspace_dir: Path, + batch_index: int, + batch_count: int, +) -> Path: + """Where one layout batch's extension is rendered. + + Batch 0 *is* the published tree, and its dispatch has to be in place before + anything compiles against it: batching used to rewrite batch 0's dispatch + only after validation, and the first benchmark then died with + ``FileNotFoundError: compile inputs are missing: .../w8a8_dispatch.cpp``. + Later batches never touch ``final/source``; they get their own copy under + ``final/layout_batches/batchN``. + """ + if batch_count <= 1 or batch_index == 0: + return source + return workspace_dir / "final" / "layout_batches" / f"batch{batch_index}" + + +def _copy_final_source(source: Path, destination: Path) -> None: + """Copy the merged extension tree so one batch can rewrite its own dispatch. + + ``prebuilt`` and the caches are deliberately left behind: every batch copies + in exactly the objects it dispatches. + """ + destination.mkdir(parents=True, exist_ok=True) + for entry in sorted(source.iterdir()): + if entry.name in {"prebuilt", "cache", ".git", "__pycache__"}: + continue + target = destination / entry.name + if entry.is_dir(): + shutil.copytree( + entry, target, + ignore=shutil.ignore_patterns("__pycache__"), + dirs_exist_ok=True, + ) + elif entry.is_file(): + shutil.copy2(entry, target) + + def _render_prebuilt_dispatch( artifacts: list[Dict[str, Any]], ) -> str: - """Render control-plane C++ dispatch; this file contains no HIP kernel.""" + """Render control-plane C++ dispatch; this file contains no HIP kernel. + + Routing keys on the **whole shape** ``(m, n, k)``. Keying on ``(n, k)`` + alone collapsed every M variant of one operator onto a single artifact: the + three M variants of one weight share ``(n, k)``, so the M=4096 prefill ran + whichever member the shape-id sort put first -- the M=16 kernel. That is + numerically correct but 2.5x slower (1026 us vs 410 us measured), which is + what failed the final performance gate; see + ``error_catalog/04-kernel-quality.md``. + + Inside one ``(n, k)`` family the routes form a descending-M chain, so the + exact shape always wins and an M nobody optimized falls to the largest tuned + M that does not exceed it -- the conservative direction, since small-tile + kernels are the proven-generic end (the M=16 kernel handled M=4096 rows + correctly) -- or to the family's smallest M when the request is below every + tuned value. A family with a single member therefore routes exactly as it + did before. ``artifacts[0]`` stays the last resort. + + ``launch_pack_w8a8_weight`` cannot see ``m``: its ABI is ``(k, n)``, and in + the real model one packed weight feeds every M of that weight. Packing is + therefore one route per family, taken from the family's smallest M. That is + only sound when the family agrees on the packed layout, and the final + validation is what proves it: it packs and then benches every shape through + these routes, so a disagreement fails correctness instead of producing a + plausible wrong number. + """ + if not artifacts: + raise RuntimeError("cannot render dispatch without worker artifacts") + + families = _artifact_families(artifacts) + seen_shapes: set[tuple[int, int, int]] = set() + declarations: list[str] = [] gemm_routes: list[str] = [] pack_routes: list[str] = [] - seen_pairs: set[tuple[int, int]] = set() - for artifact in artifacts: - launch = artifact["launch_symbol"] - pack = artifact["pack_symbol"] - shape = artifact["shape"] - n = int(shape["N"]) - k = int(shape["K"]) - declarations.extend([ - f'extern "C" void {launch}(', - " const int8_t*, const int8_t*, const float*, const float*,", - " void*, void*, int64_t, int, int, int, hipStream_t);", - f'extern "C" void {pack}(', - " const int8_t*, const float*, int8_t*, float*,", - " int, int, hipStream_t);", - "", - ]) - pair = (n, k) - if pair in seen_pairs: - continue - seen_pairs.add(pair) - gemm_routes.extend([ + for (n, k), members in families: + for artifact in members: + shape = artifact["shape"] + key = (int(shape["M"]), n, k) + if key in seen_shapes: + raise RuntimeError( + "two artifacts claim the same shape " + f"(m, n, k)={key}: " + + ", ".join( + item["shape_id"] for item in members + if int(item["shape"]["M"]) == key[0] + ) + ) + seen_shapes.add(key) + declarations.extend([ + f'extern "C" void {artifact["launch_symbol"]}(', + " const int8_t*, const int8_t*, const float*, const float*,", + " void*, void*, int64_t, int, int, int, hipStream_t);", + f'extern "C" void {artifact["pack_symbol"]}(', + " const int8_t*, const float*, int8_t*, float*,", + " int, int, hipStream_t);", + "", + ]) + + # Packed layout is a property of the weight, not of M (see docstring). + pack_routes.extend([ f" if (n == {n} && k == {k}) {{", - f" {launch}(a, b, x_scale, weight_scale, out, workspace,", - " workspace_bytes, m, n, k, stream);", + f" {members[0]['pack_symbol']}(raw_weight, weight_scale, " + "packed_weight,", + " packed_weight_scale, k, n, stream);", " return;", " }", ]) - pack_routes.extend([ - f" if (n == {n} && k == {k}) {{", - f" {pack}(raw_weight, weight_scale, packed_weight,", - " packed_weight_scale, k, n, stream);", + + routed_ms = [int(item["shape"]["M"]) for item in members] + gemm_routes.append( + f" // {n}x{k}: optimized at M=" + + ", ".join(str(item) for item in routed_ms) + ) + gemm_routes.append(f" if (n == {n} && k == {k}) {{") + for artifact in sorted( + members, key=lambda entry: -int(entry["shape"]["M"]) + ): + gemm_routes.extend([ + f" if (m >= {int(artifact['shape']['M'])}) {{", + f" {artifact['launch_symbol']}(a, b, x_scale, " + "weight_scale, out, workspace,", + " workspace_bytes, m, n, k, stream);", + " return;", + " }", + ]) + gemm_routes.extend([ + f" {members[0]['launch_symbol']}(a, b, x_scale, weight_scale, " + "out, workspace,", + " workspace_bytes, m, n, k, stream);", " return;", " }", ]) - if not artifacts: - raise RuntimeError("cannot render dispatch without worker artifacts") - fallback_launch = artifacts[0]["launch_symbol"] - fallback_pack = artifacts[0]["pack_symbol"] + + # Last resort for a request whose (n, k) matches no family at all. Taken from + # the sorted families rather than ``artifacts[0]`` so the rendered file does + # not depend on the order the artifacts were collected in. + fallback_launch = families[0][1][0]["launch_symbol"] + fallback_pack = families[0][1][0]["pack_symbol"] return "\n".join([ "// Generated by the trusted control plane. Contains no HIP kernel.", "#include ", @@ -639,17 +890,21 @@ def _final_performance_gate( µs-scale decode kernels are one-sided noisy on shared GPUs: transient co-tenant load / clock / thermal state only inflates the median, never - deflates it (decode skill §5). When ``median > 1.05 x best`` with no - source change, re-measure with the same protocol up to ``max_retries`` - times, waiting ``retry_interval_s`` between attempts, and accept the best - (min) median. Raise only when every attempt still exceeds the gate. + deflates it (decode skill §5). When ``median`` exceeds the acceptance + window (``_perf_gate_limit``: the worker best plus 5%, or plus + ``_PERF_GATE_MIN_SLACK_US`` µs when that is wider) with no source change, + re-measure with the same protocol up to ``max_retries`` times, waiting + ``retry_interval_s`` between attempts, and accept the best (min) median. + Raise only when every attempt still exceeds the gate. Returns the metrics dict of the accepted (best) measurement. """ + limit = _perf_gate_limit(best_median) + slack = max(0.05 * float(best_median), _PERF_GATE_MIN_SLACK_US) final_median = float(metrics.get("median_us") or float("inf")) accepted = metrics retries_left = max_retries - while final_median > best_median * 1.05 and retries_left > 0: + while final_median > limit and retries_left > 0: retries_left -= 1 attempt = max_retries - retries_left store.append_timeline( @@ -657,8 +912,12 @@ def _final_performance_gate( { "shape_id": shape_id, "attempt": attempt, + # The running best so far (the value this retry has to beat), + # not the measurement the retry is about to take. "median_us": final_median, "best_median_us": best_median, + "limit_us": limit, + "slack_us": slack, "wait_s": retry_interval_s, }, ) @@ -672,10 +931,11 @@ def _final_performance_gate( final_median = min( final_median, float(retry.get("median_us") or float("inf")) ) - if final_median > best_median * 1.05: + if final_median > limit: raise RuntimeError( f"performance regressed for {shape_id}: final " f"{final_median:.3f} us vs worker best {best_median:.3f} us " + f"(limit {limit:.3f} us = best + {slack:.3f} us) " f"after {max_retries} re-measures" ) return accepted @@ -841,6 +1101,17 @@ def _prepare_worktrees( staged_contract = stage_operator_api(contract, seed) staged_references = stage_operator_references(contract, seed) + # Skills are knowledge, so they are bundled per hardware profile and the + # agent must read THIS board's edition: sync now, before any worker is + # spawned, so a library still holding the other board's edition (a fresh + # container, a machine whose board changed) is corrected first. Recorded + # in the timeline because it decides which numbers the agent can read. + skills_sync = sync_skill_libraries(workspace_dir=self.workspace_dir) + self.store.append_timeline("skills_profile_synced", { + "hw_profile": skills_sync.get("hw_profile"), + "seeded": skills_sync.get("seeded"), + "quarantined": skills_sync.get("quarantined"), + }) staged_operator_files = [staged_contract, *staged_references] staged_operator_relatives = [ str(path.relative_to(seed)) for path in staged_operator_files @@ -2036,6 +2307,187 @@ def run_lane( ) return baseline, dict(sorted(workers.items())) + def _validate_layout_batch( + self, + *, + source: Path, + batch: List[Dict[str, Any]], + batch_index: int, + batch_count: int, + layout_check: "final_layout_gate.LayoutCheck", + serial_gpu: int, + config: OptimizerConfig, + optimized_ids: set, + best_by_shape: Dict[str, Dict[str, Any]], + validation: Dict[str, Dict[str, Any]], + validation_shapes: List[Dict[str, Any]], + validation_scope: str, + bench_kwargs: Dict[str, int], + ) -> Dict[str, Any]: + """Render, link and serially validate one layout batch. + + With a single batch this is exactly the historical merge. With several + (packed layouts disagreed inside a family) each batch is a self-contained + extension: its own ``dispatch``, its own ``prebuilt`` set, its own + sequential validation -- so every ``(N, K)`` in a batch is packed and + interpreted by the same member, and nothing is ever handed an image its + own kernel cannot read. + + Batch 0 renders directly into ``final/source`` (so the published tree is + a batch that was actually validated); later batches live under + ``final/layout_batches/batchN``. + + Every batch is benchmarked against **its own** tree (``batch_source``): + the shared runner used to keep pointing at ``final/source``, so a later + batch's shapes were served by batch 0's dispatch -- in + glm-tp8-m8-16-4096-2663a78f the M=4096 shape fell to batch 0's m16 route, + which is numerically correct and 2.28x slower, and the perf gate read that + as a regression of an artifact that was never measured. + """ + batch_source = _layout_batch_source( + source, self.workspace_dir, batch_index, batch_count + ) + if batch_source != source: + shutil.rmtree(batch_source, ignore_errors=True) + _copy_final_source(source, batch_source) + # Shrink the tree to exactly this batch (its objects *and* its dispatch) + # before anything compiles against it. + sync_prebuilt_artifacts(batch_source, batch) + + dispatch_path = batch_source / "csrc" / "w8a8_dispatch.cpp" + dispatch_path.parent.mkdir(parents=True, exist_ok=True) + dispatch_path.write_text( + _render_prebuilt_dispatch(batch), encoding="utf-8" + ) + # The benchmark must build and load *this* batch's extension: the caches + # (references, torch extensions, logs) stay under ``final/``, but the tree + # under test is ``batch_source``. Reusing one runner across batches left + # every later batch measured through batch 0's dispatch. + runner = W8A8Runner( + self.workspace_dir / "final", serial_gpu, source=batch_source, + ) + self.store.append_timeline( + "final_artifact_link_launch", + { + "artifacts": [ + { + "shape_id": item["shape_id"], + "worker_id": item["worker_id"], + "object_sha256": item["object_sha256"], + } + for item in batch + ], + "gpu": serial_gpu, + # Which kernel serves which request, decided above. Recorded + # because a final validation that disagrees with the worker + # numbers is usually a routing question, not a kernel one. + "dispatch_families": _dispatch_family_plan(batch), + "layout_batch": { + "index": batch_index, + "count": batch_count, + "source": str(batch_source), + }, + # Which member's pack kernel the family's weight image really + # comes from, for every (N, K) family. + "layout_groups": layout_check.as_json(), + }, + ) + # Only the shapes this batch carries: an extension that does not + # contain a shape cannot dispatch it, so validating one would fall + # through to the last-resort route and report a failure that says + # nothing about this batch. (Measured: batch 0 of df6fb501 lists 7 + # artifacts, so it validates 7 shapes, not the task's 10.) + batch_shape_ids = { + str(item["shape_id"]) for item in batch + } + validation_shapes = [ + shape for shape in validation_shapes + if str(shape["id"]) in batch_shape_ids + ] + self.store.append_timeline( + "final_validation_plan", + { + "scope": validation_scope, + "shapes": [str(shape["id"]) for shape in validation_shapes], + "bench": dict(bench_kwargs), + "layout_batch": { + "index": batch_index, + "count": batch_count, + }, + }, + ) + + try: + for shape in validation_shapes: + shape_id = str(shape["id"]) + params = { + key: value for key, value in shape.items() + if key != "id" + } + if shape_id in optimized_ids: + metrics = runner.benchmark(params, **bench_kwargs) + else: + # Regression shapes stay on the light sampling; only the + # replay count follows the configured budget. + regression_kwargs = {"warmups": 2, "samples": 3} + if "replays_per_sample" in bench_kwargs: + regression_kwargs["replays_per_sample"] = ( + bench_kwargs["replays_per_sample"] + ) + metrics = runner.benchmark(params, **regression_kwargs) + validation[shape_id] = metrics + if not metrics.get("passed"): + raise RuntimeError( + f"correctness failed for {shape_id}: " + f"{json.dumps(metrics)}" + ) + if shape_id not in optimized_ids: + continue + best_median = float( + best_by_shape[shape_id].get("median_us") + or float("inf") + ) + metrics = _final_performance_gate( + shape_id=shape_id, + best_median=best_median, + metrics=metrics, + benchmark=lambda: runner.benchmark( + params, **bench_kwargs + ), + max_retries=_PERF_GATE_MAX_RETRIES, + retry_interval_s=_PERF_GATE_RETRY_INTERVAL_S, + store=self.store, + ) + validation[shape_id] = metrics + except Exception as exc: + self.store.append_timeline( + "final_artifact_link_rejected", + { + "error": str(exc), + "layout_batch": { + "index": batch_index, + "count": batch_count, + "shapes": [ + str(item["shape_id"]) for item in batch + ], + }, + }, + ) + raise RuntimeError( + "Final prebuilt-object W8A8 validation failed: " + f"{exc}" + ) from exc + + return { + "schema_version": SCHEMA_VERSION, + "strategy": "link_exact_accepted_worker_objects", + "hip_recompiled_by_main": False, + "hip_rewritten_by_main": False, + "dispatch_language": "C++", + "dispatch": str(dispatch_path), + "artifacts": batch, + } + def _synthesize_final_candidate( self, config: OptimizerConfig, @@ -2098,7 +2550,6 @@ def _synthesize_final_candidate( "serial validation GPU must be one of the completed worker " f"GPUs {sorted(assigned_gpus)}, got {serial_gpu}" ) - runner = W8A8Runner(root, serial_gpu) best_by_shape = { shape_id: workers[assignment.worker_id]["shapes"][shape_id][ "metrics" @@ -2184,35 +2635,78 @@ def _synthesize_final_candidate( ), }) artifacts.sort(key=lambda item: str(item["shape_id"])) - dispatch_path = source / "csrc" / "w8a8_dispatch.cpp" - dispatch_path.write_text( - _render_prebuilt_dispatch(artifacts), encoding="utf-8" - ) - proposal = { - "schema_version": SCHEMA_VERSION, - "strategy": "link_exact_accepted_worker_objects", - "hip_recompiled_by_main": False, - "hip_rewritten_by_main": False, - "dispatch_language": "C++", - "artifacts": artifacts, - } - write_json(source / "artifact_manifest.json", proposal) - self.store.append_timeline( - "final_artifact_link_launch", - { - "artifacts": [ - { - "shape_id": item["shape_id"], - "worker_id": item["worker_id"], - "object_sha256": item["object_sha256"], - } - for item in artifacts - ], - "gpu": serial_gpu, - }, - ) - validation: Dict[str, Dict[str, Any]] = {} + # A ``(N, K)`` weight is packed once and interpreted by every M of that + # weight, but the frozen ``prepare_weight`` ABI has no M -- so the + # family's members must agree on the packed image. Fingerprint each + # member's pack kernel and, when they disagree, validate one layout + # batch at a time instead of handing every M to one shared image. + layout_check, batches = plan_layout_batches(artifacts, source) + if layout_check.conflicts: + # The families disagree about the packed image: pack them per layout + # instead. Nothing is re-optimized and no kernel is rewritten -- + # each batch simply gets the packing its own GEMM expects. + offending = ", ".join( + f"{family.n}x{family.k} (" + + "; ".join( + f"{sorted(group)}" + for group in final_layout_gate.family_groups(family) + ) + + ")" + for family in layout_check.conflicts + ) + self.store.append_timeline( + "final_layout_conflict", + { + "reason": ( + "members of one (N, K) family pack the weight " + "differently; the frozen prepare_weight ABI has no M, " + "so the merge cannot share one image" + ), + "conflicts": offending, + "batches": [ + sorted(str(item["shape_id"]) for item in batch) + for batch in batches + ], + "layout_groups": layout_check.as_json(), + }, + ) + elif layout_check.unproven: + # Some family's packed image was never measured, so "no conflict" is + # not a finding there -- it is the absence of a measurement. Say so + # loudly: a silent single batch here is exactly how a packed-layout + # conflict reached serial validation in glm-tp8-m8-16-4096-2663a78f + # (the probe child was spawned with an empty ``sys.executable``). + multi_member = [ + f"{family.n}x{family.k}" + for family in layout_check.unproven + if len(family.members) > 1 + ] + self.store.append_timeline( + "final_layout_probe_failed", + { + "reason": ( + "the packed-image probe could not measure every member " + "of a (N, K) family, so agreement there is unproven" + ), + "probe_error": layout_check.probe_error, + "unproven_families": [ + f"{family.n}x{family.k}" + for family in layout_check.unproven + ], + "multi_member_families": multi_member, + "fallback": ( + "split_unproven_members" if len(batches) > 1 + else "single_batch" + ), + "batch_count": len(batches), + "batches": [ + sorted(str(item["shape_id"]) for item in batch) + for batch in batches + ], + "layout_groups": layout_check.as_json(), + }, + ) _answers = self.req.get("answers") if not isinstance(_answers, dict): _answers = self.req @@ -2226,65 +2720,48 @@ def _synthesize_final_candidate( validation_shapes = _validation_shape_list( optimized_shapes, fallback_shapes, validation_scope ) - self.store.append_timeline( - "final_validation_plan", - { - "scope": validation_scope, - "shapes": [str(shape["id"]) for shape in validation_shapes], - "bench": dict(bench_kwargs), - }, - ) - try: - for shape in validation_shapes: - shape_id = str(shape["id"]) - params = { - key: value for key, value in shape.items() - if key != "id" - } - if shape_id in optimized_ids: - metrics = runner.benchmark(params, **bench_kwargs) - else: - # Regression shapes stay on the light sampling; only the - # replay count follows the configured budget. - regression_kwargs = {"warmups": 2, "samples": 3} - if "replays_per_sample" in bench_kwargs: - regression_kwargs["replays_per_sample"] = ( - bench_kwargs["replays_per_sample"] - ) - metrics = runner.benchmark(params, **regression_kwargs) - validation[shape_id] = metrics - if not metrics.get("passed"): - raise RuntimeError( - f"correctness failed for {shape_id}: " - f"{json.dumps(metrics)}" - ) - if shape_id not in optimized_ids: - continue - best_median = float( - best_by_shape[shape_id].get("median_us") - or float("inf") - ) - metrics = _final_performance_gate( - shape_id=shape_id, - best_median=best_median, - metrics=metrics, - benchmark=lambda: runner.benchmark( - params, **bench_kwargs - ), - max_retries=_PERF_GATE_MAX_RETRIES, - retry_interval_s=_PERF_GATE_RETRY_INTERVAL_S, - store=self.store, - ) - validation[shape_id] = metrics - except Exception as exc: - self.store.append_timeline( - "final_artifact_link_rejected", - {"error": str(exc)}, + + validation: Dict[str, Dict[str, Any]] = {} + proposals: List[Dict[str, Any]] = [] + original_source = source + for batch_index, batch in enumerate(batches): + proposal = self._validate_layout_batch( + source=original_source, + batch=batch, + batch_index=batch_index, + batch_count=len(batches), + layout_check=layout_check, + serial_gpu=serial_gpu, + config=config, + optimized_ids=optimized_ids, + best_by_shape=best_by_shape, + validation=validation, + validation_shapes=validation_shapes, + validation_scope=validation_scope, + bench_kwargs=bench_kwargs, ) - raise RuntimeError( - "Final prebuilt-object W8A8 validation failed: " - f"{exc}" - ) from exc + proposals.append(proposal) + proposal = proposals[0] if proposals else {} + if len(proposals) > 1: + # The published tree is batch 0, which was rendered and validated + # there; every other batch stays complete under + # final/layout_batches/batchN and is listed here. + proposal = { + **proposal, + "layout_batches": [ + { + "index": index, + "artifacts": [ + item["shape_id"] for item in artifact_batch + ], + "dispatch": item.get("dispatch"), + } + for index, (item, artifact_batch) in enumerate( + zip(proposals, batches) + ) + ], + } + write_json(original_source / "artifact_manifest.json", proposal) changed_paths = sorted( line[3:].strip() @@ -2398,7 +2875,7 @@ def run(self, *, dry_run: bool = False) -> Dict[str, Any]: 5. SYNTHESIZE → VALIDATE → REPORT. """ task_id = str(self.req.get("task_id", "task")) - self.store.init_or_resume(task_id) + self.store.init_or_resume(task_id, "dcu-kernel-auto-opt") self.store.update_run( finished=False, final_status=None, diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_binding.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_binding.py index cef27317..6195d7fd 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_binding.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_binding.py @@ -3,21 +3,47 @@ DTK accepts either HIP_VISIBLE_DEVICES or ROCR_VISIBLE_DEVICES. Setting both to the same non-zero physical index applies filtering twice and can hide the device, so this task standardizes on HIP_VISIBLE_DEVICES only. + +The number handed over is a host HCU ordinal: ``hy-smi`` prints it as +``DCU[n]``, and it is the same index the measurement gate reads that card's +state with (see :mod:`metainfer.orchestrator.gpu_inventory`). The air board +exposes eight of them, the liquid board four, so the allowed set is discovered +rather than hard-coded -- and callers that know their task's own placement pass +it in as ``allowed``, which makes a wrong index fail here instead of silently +running on a card the task never asked for (possibly one somebody else is +using). """ from __future__ import annotations -from typing import Dict, MutableMapping +from typing import Dict, Iterable, MutableMapping, Optional + +from metainfer.orchestrator import gpu_inventory def bind_worker_gpu( - env: MutableMapping[str, str], physical_gpu: int + env: MutableMapping[str, str], + physical_gpu: int, + *, + allowed: Optional[Iterable[int]] = None, ) -> Dict[str, str]: - if physical_gpu not in range(4): - raise ValueError("physical_gpu must be one of 0,1,2,3") + """Point ``HIP_VISIBLE_DEVICES`` at one card. + + ``allowed`` is the task's own device set when the caller knows it; without + it every card this machine exposes is acceptable (current behaviour). + """ + if not gpu_inventory.is_addressable(physical_gpu, allowed): + if allowed is None: + listed = ",".join(str(d) for d in gpu_inventory.devices()) + else: + listed = ",".join(str(int(d)) for d in allowed) + raise ValueError( + f"physical_gpu must be one of {listed or ''}, " + f"got {physical_gpu}" + ) env.pop("ROCR_VISIBLE_DEVICES", None) - env["HIP_VISIBLE_DEVICES"] = str(physical_gpu) - return {"HIP_VISIBLE_DEVICES": str(physical_gpu)} + env["HIP_VISIBLE_DEVICES"] = str(int(physical_gpu)) + return {"HIP_VISIBLE_DEVICES": str(int(physical_gpu))} def hide_gpus_from_control_plane( diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_preflight.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_preflight.py index d5a92e4b..edf56703 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_preflight.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_preflight.py @@ -45,10 +45,14 @@ DEFAULT_POWER_SUSPECT_W = 230.0 KFD_PROC_ROOT = Path("/sys/class/kfd/kfd/proc") -#: Driver sysfs used when ``hy-smi`` cannot report HCU%/VRAM. -DRM_ROOT = Path("/sys/class/drm") -#: DRM drivers that mean "this card is a Hygon HCU". -_HCU_DRM_DRIVERS = ("hycu", "amdgpu", "hydcu") +from metainfer.orchestrator.gpu_inventory import ( + DRM_ROOT, + HCU_DRM_DRIVERS, + hcu_device_dirs, +) + +#: Kept for callers/tests that referenced the old private name. +_HCU_DRM_DRIVERS = HCU_DRM_DRIVERS #: ``N/A`` (or ``-``) where a number is expected: the device is present but the #: management interface cannot be read. Such a field is recorded as unknown #: instead of dropping the whole row, which used to make every card look idle. @@ -128,26 +132,12 @@ def read_sysfs_gpu_state( filtered to the HCU driver and ordered by card index (``card1`` is the first HCU on the K500SM_AI machines, since ``card0`` is the BMC VGA), which matches the ``hy-smi``/KFD device numbering the tasks use. + + The enumeration itself now lives in + :mod:`metainfer.orchestrator.gpu_inventory`, so this index space and the + one the task form offers the operator cannot drift apart. """ - cards: List[Path] = [] - try: - entries = list(drm_root.iterdir()) - except OSError: - return {} - for entry in sorted(entries, key=lambda p: p.name): - match = re.fullmatch(r"card(\d+)", entry.name) - if not match: - continue - device = entry / "device" - try: - uevent = (device / "uevent").read_text(encoding="utf-8") - except OSError: - continue - if not any(driver in uevent for driver in _HCU_DRM_DRIVERS): - continue - if not (device / "mem_info_vram_total").is_file(): - continue - cards.append(device) + cards: List[Path] = list(hcu_device_dirs(drm_root)) out: Dict[int, Dict[str, float]] = {} for index, device in enumerate(cards): row: Dict[str, float] = {} diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_probe.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_probe.py new file mode 100644 index 00000000..ce4a065c --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/gpu_probe.py @@ -0,0 +1,491 @@ +"""Run the trusted probe subprocess, surviving a transiently broken GPU stack. + +``w8a8_bench.py --probe`` initialises a device, so it goes through exactly the +same ROCm/amdsmi startup as a measurement. When *that* startup fails, the failure +is environmental, not the candidate's: + + RuntimeError: ... --probe failed (3): shm_open: Permission denied + UserWarning: Can't initialize amdsmi - Error code: 32 + +That came out of the cheap, implementation-free part of Generate (the scaffold +preflight), where the probe answers only "is one device visible and is HIP Graph +available". Two real observations shaped this module: + +* the failure is *transient and concurrent*. The identical command, run by hand + minutes later on the same machine, the same repository and each of the four + GPUs, printed the expected JSON with an empty stderr — yet a task that + initialises four devices at once failed every time, with three different + low-level symptoms across reports (``shm_open: Permission denied``, + ``shm_open: Invalid argument``, ``fchmod: Bad file descriptor``, always next to + ``Can't initialize amdsmi - Error code: 32``). The ROCm/amdsmi startup is a + shared-memory handshake over ``/dev/shm/rocm_smi_card*``, and concurrent + starters race on those segments; +* one such blip used to kill a multi-hour task, because a single failed probe + raised immediately. + +So the probe is (a) *serialised* across processes with a container-wide lock, so +the stack's handshake is never started by several processes at once, and (b) +retried a couple of times with a growing wait, long enough for a broken +handshake to heal. It is **always attempted**: waiting out a busy card is the +measurement gate's job (it knows the device set and waits up to 24 h), and +short-circuiting that here would stop runs that used to proceed on a shared +machine. Whether the card was busy is recorded, so the failure message can say +which of the two problems it was. Retrying cannot weaken the check: the same +two assertions (exactly one visible device, HIP Graph available) are applied to +whatever the successful attempt printed. When every attempt fails, the error is +enriched with what an operator would otherwise have to reproduce by hand, and a +diagnostic snapshot is written next to the task's other evidence. + +One failure does **not** heal by waiting: the same ``shm_open: Permission +denied`` pair also appears when the *launcher* runs inside a file sandbox that +makes everything outside its workspace read-only — ``/dev/shm`` included. ROCm +and amdsmi bring the stack up through ``/dev/shm`` rendezvous files +(``rocm_smi_card*``), so such a process tree can never see a device, and +retrying only repeats the denial. That case is therefore *measured* before the +first attempt (see :func:`shm_write_check`), reported as its own cause, and +never retried: the fix belongs to whoever started the WebUI/orchestrator, not to +the candidate kernel. +""" + +from __future__ import annotations + +import errno +import fcntl +import glob +import json +import os +import shutil +import subprocess +import tempfile +import time +from contextlib import contextmanager +from pathlib import Path +from typing import Any, Callable, Dict, List, Optional + +#: ``shm_open: Permission denied`` / ``Can't initialize amdsmi`` — the fragment +#: amdsmi's C library prints to stderr when its shared-memory handshake breaks. +#: Matched to *explain* the failure; the retry never depends on the diagnosis. +TRANSIENT_GPU_MARKERS = ("shm_open", "amdsmi", "hsa_", "rocr", "fchmod", + "rocm_smi", "bad file descriptor") + +#: ``/dev/shm`` files that *may* be cleaned, and only when an operator asks for +#: it. Deleting them is **off by default**: they are live rendezvous files, and a +#: process that opened one keeps a handle to the now-unlinked inode, so the next +#: ``fchmod``/``shm_open`` on it fails with "Bad file descriptor" / "Invalid +#: argument" — precisely the failure that was being investigated. Cleaning them +#: can only be justified on a quiet machine, so it is opt-in. +SHM_GLOBS = ("/dev/shm/rocm_smi_card*",) +CLEAN_SHM_ENV = "METAINFER_GPU_PROBE_CLEAN_SHM" + +#: Where POSIX shared memory lives on Linux (``shm_open("/x")`` is +#: ``open("/dev/shm/x")``). ROCm needs to *write* here, so a read-only +#: ``/dev/shm`` means no device, ever. +SHM_DIR = "/dev/shm" + +#: Errnos that mean "this process may not write there" — i.e. confinement, a +#: deterministic condition no wait can clear. Anything else (``ENOSPC``, +#: ``EMFILE``, …) is reported but still retried like an ordinary transient +#: failure, so a false negative here can only cost time, never a verdict. +SHM_DENIED_ERRNOS = (errno.EACCES, errno.EPERM, errno.EROFS) + +#: Waits before each retry. The handshake that breaks here is repaired by a +#: *later, quieter* start, so the waits are generous: one attempt plus these is +#: at most ~2.5 minutes, still inside the ordinary task budget. +RETRY_WAITS_SECONDS = (5.0, 20.0) + +#: One lock for the whole container: only one process may start the GPU stack at +#: a time. This is the actual fix for the concurrent-init race — retrying inside +#: a storm of starters only reproduces the storm. +PROBE_LOCK_PATH = "/tmp/metainfer-gpu-probe.lock" + +#: An operator can disable the serialisation (e.g. to prove it was the cause). +PROBE_LOCK_ENV = "METAINFER_GPU_PROBE_LOCK" + + +def probe_attempts(wait_seconds: Optional[tuple] = None, + ) -> int: + """How many probe attempts a caller should make.""" + waits = RETRY_WAITS_SECONDS if wait_seconds is None else tuple(wait_seconds) + return len(waits) + 1 + + +def shm_write_check(directory: str = SHM_DIR) -> Dict[str, Any]: + """Can **this** process create a POSIX shared-memory object? + + Returns ``{"ok": bool, ...}`` and never raises. The check creates (and + immediately removes) one throwaway file, which is exactly what + ``shm_open(name, O_CREAT)`` does internally, so a denial here is the denial + the GPU stack would hit a moment later. + + The failure this exists for: the MetaInfer WebUI (or any orchestrator) was + started from a shell that runs under a file sandbox — Landlock/bwrap-style + confinement grants writes to a couple of roots (the workspace, ``/tmp``) and + makes the rest of the filesystem read-only, ``/dev/shm`` included. Every + orchestrator the launcher spawns inherits that domain, and a child cannot + drop it, so the probe fails identically at every attempt while the very same + command run from an unrestricted shell succeeds. Naming it separately is the + difference between "the container's GPU stack is broken" (misleading: it + sends the operator to the wrong machine) and "restart the WebUI outside the + sandbox" (the actual fix). + """ + try: + fd, path = tempfile.mkstemp(prefix=".metainfer-shm-check-", + dir=directory) + except OSError as exc: + return { + "ok": False, + "dir": str(directory), + "errno": exc.errno, + "error": exc.strerror, + "denied": exc.errno in SHM_DENIED_ERRNOS, + } + try: + os.close(fd) + except OSError: + pass + finally: + try: + os.unlink(path) + except OSError: + pass + return {"ok": True, "dir": str(directory), "errno": None, "error": None, + "denied": False} + + +def _shm_denied(check: Dict[str, Any]) -> bool: + """Whether a failed :func:`shm_write_check` proves confinement. + + Only the "you may not write here" errnos qualify; a full ``/dev/shm`` or an + exhausted fd table is a different problem and keeps the ordinary retry path. + """ + return bool(check) and not check.get("ok", True) and bool( + check.get("denied") or check.get("errno") in SHM_DENIED_ERRNOS) + + +def clean_stale_rocm_shm(*, min_age_seconds: float = 60.0, + now: Optional[float] = None, + force: bool = False) -> List[str]: + """Remove idle amdsmi shared-memory files; return the names removed. + + **Off unless ``force`` (or ``METAINFER_GPU_PROBE_CLEAN_SHM=1``).** Unlinking + a file another process already opened leaves that process holding a handle to + a dead inode, and its next ``fchmod`` on it fails with + ``Bad file descriptor`` — the cleanup was creating the symptom it was meant + to clear. Only an operator who knows the machine is quiet should ask for it. + """ + if not force and str(os.environ.get(CLEAN_SHM_ENV, "")).strip().lower() \ + not in {"1", "on", "true", "yes"}: + return [] + clock = time.time() if now is None else float(now) + removed: List[str] = [] + for pattern in SHM_GLOBS: + for name in glob.glob(pattern): + path = Path(name) + try: + if clock - path.stat().st_mtime < float(min_age_seconds): + continue + path.unlink() + removed.append(path.name) + except OSError: + continue + return removed + + +def gpu_probe_diagnostics(*, command: List[str], attempts: int, + env: Optional[Dict[str, str]] = None, + waits: Optional[List[str]] = None, + state_dir: Optional[Path] = None, + attempts_performed: Optional[int] = None, + now: Optional[float] = None) -> Dict[str, Any]: + """Everything an operator needs to explain a probe that will not start.""" + diag: Dict[str, Any] = { + "ts": time.time() if now is None else float(now), + "attempts": attempts, + "command": list(command), + "uid": os.getuid(), + "gid": os.getgid(), + # The child's own environment, not ours: the probe is pinned to a + # device by HIP_VISIBLE_DEVICES, and that is the first thing to check. + "visible_devices_env": (env or os.environ).get("HIP_VISIBLE_DEVICES"), + # Why each attempt did not produce a device: "card was busy" (waiting) + # versus "the stack did not start" (retrying). + "waits": list(waits or []), + # Whether this process tree may write /dev/shm at all. When it may not, + # no attempt can ever see a device and no retry is spent on it. + "dev_shm_write": shm_write_check(), + "dev_shm": [], + } + if attempts_performed is not None: + # Fewer than the budget when the failure was provably deterministic. + diag["attempts_performed"] = int(attempts_performed) + diag["admission"] = device_admission(env or {}) + try: + for entry in sorted(Path("/dev/shm").iterdir()): + try: + info = entry.lstat() + except OSError: + continue + diag["dev_shm"].append({ + "name": entry.name, + "mode": oct(info.st_mode & 0o777), + "uid": info.st_uid, + "size": info.st_size, + "mtime": info.st_mtime, + }) + except OSError as exc: + diag["dev_shm_error"] = repr(exc) + for fd_name, limit_file in (("limits", "/proc/self/limits"),): + try: + diag[fd_name] = Path(limit_file).read_text( + encoding="utf-8", errors="replace") + except OSError as exc: + diag[fd_name] = repr(exc) + if state_dir is not None: + try: + target = Path(state_dir) / "gpu_probe_failure.json" + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(json.dumps(diag, ensure_ascii=False, indent=2), + encoding="utf-8") + diag["written_to"] = str(target) + except OSError: + pass + return diag + + +def _looks_transient(exc: BaseException) -> bool: + """Whether this probe failure is worth another attempt. + + Two classes qualify, and both describe the *container*, not the candidate: + + * the GPU stack's handshake broke (``shm_open`` / ``amdsmi`` / a HIP runtime + error) — the failure observed in production; + * the probe never returned, so device initialisation is wedged. + + Everything else (a missing module, a traceback from the harness) fails + immediately: retrying it would only delay a real report by 25 seconds. + """ + if isinstance(exc, subprocess.TimeoutExpired): + return True + text = f"{exc} {getattr(exc, 'stderr', '') or ''}".lower() + return any(marker in text for marker in TRANSIENT_GPU_MARKERS) + + +def _probe_lock_enabled() -> bool: + """Serialisation is on unless the operator turns it off explicitly.""" + value = str(os.environ.get(PROBE_LOCK_ENV, "")).strip().lower() + return value not in {"0", "off", "false", "no"} + + +@contextmanager +def gpu_probe_lock(*, timeout_seconds: float = 600.0): + """Let one process at a time start the GPU stack. + + ``flock`` on a file in ``/tmp`` is container-wide and costs nothing when + uncontended; when contended, the waiters queue instead of racing on the + amdsmi shared-memory handshake. On any problem with the lock file itself the + probe simply proceeds unlocked: a lock must never be the reason a task + cannot measure. + """ + if not _probe_lock_enabled(): + yield False + return + handle = None + acquired = False + try: + handle = open(PROBE_LOCK_PATH, "a+") + deadline = time.time() + float(timeout_seconds) + while True: + try: + fcntl.flock(handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + acquired = True + break + except OSError: + if time.time() >= deadline: + break + time.sleep(0.5) + yield acquired + except OSError: + yield False + finally: + if handle is not None: + try: + if acquired: + fcntl.flock(handle.fileno(), fcntl.LOCK_UN) + except OSError: + pass + try: + handle.close() + except OSError: + pass + + +def _admission_enabled() -> bool: + """The GPU admission switch, shared with the measurement gate. + + ``METAINFER_GPU_PREFLIGHT=0`` turns device admission off everywhere; the + probe must honour the same switch rather than invent its own policy. + """ + try: + from .gpu_preflight import preflight_enabled + return bool(preflight_enabled()) + except Exception: # noqa: BLE001 - fail open, like the gate does not + return True + + +def _target_device(env: Dict[str, str]) -> Optional[int]: + raw = str((env or {}).get("HIP_VISIBLE_DEVICES") or "").strip() + if not raw: + return None + try: + return int(raw.split(",")[0]) + except (TypeError, ValueError): + return None + + +def device_admission(env: Dict[str, str]) -> Dict[str, Any]: + """Is the device this probe would use actually free right now? + + Measured evidence, across three reports: a probe started while another job + was running on the card failed *every* attempt — ``shm_open: Permission + denied`` / ``Invalid argument``, ``fchmod: Bad file descriptor``, always next + to ``Can't initialize amdsmi - Error code: 32`` — while the same command on a + quiet card passed first try. Starting the ROCm stack on a busy card is what + breaks it, so a busy card is waited out instead of being hammered. + """ + if not _admission_enabled(): + return {"checked": False, "usable": True, "reason": "admission disabled"} + device = _target_device(env) + if device is None: + return {"checked": False, "usable": True, "reason": "no device pinned"} + try: + from .gpu_preflight import check_gpu, sample_gpu_state, total_vram_gb + state = sample_gpu_state([device], samples=1).get(device) or {} + verdict = check_gpu(device, state, total_gb=total_vram_gb(device)) + except Exception as exc: # noqa: BLE001 - never block on our own bookkeeping + return {"checked": False, "usable": True, + "reason": f"state unreadable: {exc!r}"} + reasons = list(verdict.get("reasons") or []) + return { + "checked": True, + "usable": bool(verdict.get("usable")), + "device": device, + "vram_percent": state.get("vram_max_percent", + state.get("vram_percent")), + "util_percent": state.get("util_max_percent", + state.get("util_percent")), + "reason": "; ".join(str(r) for r in reasons) or "card is idle", + } + + +def _record_recovered(state_dir: Optional[Path], attempt: int) -> None: + """Note that a retry (not the first attempt) got a device. + + Not an error: it is the evidence that the stack was momentarily unusable, + which is what tells an operator the serialisation/retry earned its keep. + """ + if state_dir is None: + return + try: + target = Path(state_dir) / "gpu_probe_recovered.jsonl" + target.parent.mkdir(parents=True, exist_ok=True) + with target.open("a", encoding="utf-8") as fh: + fh.write(json.dumps({"ts": time.time(), "attempt": attempt}) + "\n") + except OSError: + pass + + +def run_gpu_probe(*, harness: Path, source: Path, env: Dict[str, str], + runner: Callable[..., "subprocess.CompletedProcess[str]"], + timeout: int = 120, + state_dir: Optional[Path] = None, + waits: Optional[tuple] = None, + ) -> "subprocess.CompletedProcess[str]": + """Run the device probe, retrying a broken GPU stack. + + ``runner`` is the caller's own subprocess helper (``real_pipeline._run``), + injected so this module stays free of import cycles and is trivial to test. + Raises the last failure with diagnostics attached when every attempt fails. + """ + command = [ + "python3", str(harness), + "--source", str(source), + "--m", "1", "--n", "1", "--k", "1", + "--probe", + ] + wait_list = tuple(RETRY_WAITS_SECONDS if waits is None else waits) + attempts = len(wait_list) + 1 + # Measure the determinism of the environment *before* spending attempts on + # it: a process tree that cannot write /dev/shm cannot bring the GPU stack + # up, so "retry" would be the same experiment with the same answer. + shm = shm_write_check() + confined = _shm_denied(shm) + last_error: Optional[BaseException] = None + performed = 0 + for attempt in range(1, attempts + 1): + performed = attempt + try: + # Hold the container-wide lock for the whole (short) probe: the + # failure being defended against is *concurrent* device startup. + with gpu_probe_lock() as held: + result = runner(command, cwd=source, env=env, timeout=timeout) + if attempt > 1: + _record_recovered(state_dir, attempt) + return result + except (RuntimeError, subprocess.TimeoutExpired) as exc: # noqa: PERF203 + last_error = exc + transient = _looks_transient(exc) + if attempt >= attempts or not transient or confined: + # Nothing left to try, a failure that is not the transient stack + # problem this module exists for, or a condition that waiting + # cannot change: fail with evidence. + break + # A plain wait, and nothing else: touching /dev/shm here is what + # produced the "Bad file descriptor" failures. + time.sleep(wait_list[attempt - 1]) + diag = gpu_probe_diagnostics(command=command, attempts=attempts, env=env, + state_dir=state_dir, + attempts_performed=performed) + # The probe is always attempted, whatever the cards are doing: waiting for a + # quiet card is DKAO's measurement gate's job, and pre-empting it here would + # stop runs that used to proceed fine. The verdict is reported so a *busy* + # card can be told apart from a broken stack in the failure message. + busy = bool(diag.get("admission", {}).get("checked")) and not \ + diag["admission"].get("usable", True) + if confined: + # Deterministic and unfixable from inside the task: say what it is and + # who has to change it, instead of blaming the container's GPU stack. + hint = ( + "This process tree may not write to " + str(shm.get("dir")) + + f" (shm_open: {shm.get('error')}, errno {shm.get('errno')}). ROCm and " + "amdsmi bring the GPU stack up through /dev/shm rendezvous files, so " + "no device can appear while that path is read-only; the retry budget " + "was not spent because waiting cannot change a permission. The usual " + "cause is the launcher — the MetaInfer WebUI, or the shell that " + "started the orchestrator — running inside a file sandbox that only " + "allows writes under its workspace and /tmp. Restart it outside that " + "sandbox (an unrestricted shell) and relaunch the task; a child " + "process cannot drop the sandbox by itself." + ) + elif not shm.get("ok", True): + # Not a permission problem, but /dev/shm is still unusable: the GPU + # stack cannot come up either way, and the errno says why. + hint = ( + "This process tree cannot create shared memory in " + f"{shm.get('dir')} (shm_open: {shm.get('error')}, errno " + f"{shm.get('errno')}), which is where ROCm and amdsmi keep their " + "rendezvous files: no device can appear until /dev/shm is usable " + "(free space, fd limits, a mounted tmpfs)." + ) + elif busy: + hint = ( + "The device was busy with another job while these attempts ran " + f"({diag['admission'].get('reason')}), so this may be contention rather " + "than a broken container: try again, or free the card." + ) + else: + hint = "This is the container's GPU stack, not the kernel under test." + raise RuntimeError( + f"GPU probe failed after {performed} attempt(s) " + f"({type(last_error).__name__}): {last_error}\n{hint} " + "Diagnostics: " + json.dumps(diag, ensure_ascii=False)[:1500] + ) from last_error diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/guidance.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/guidance.py index 63c56104..fb57755c 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/guidance.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/guidance.py @@ -5,6 +5,7 @@ import fcntl import json import os +import re import time import uuid from contextlib import contextmanager @@ -12,11 +13,17 @@ from typing import Any, Dict, Iterator +#: Default lane names, used when a caller has no plan yet. WORKER_IDS = tuple(f"worker_{index}" for index in range(4)) +#: A worker is named after the card it runs on (``worker_6`` is GPU 6), so ids +#: are no longer restricted to 0..3: an eight-card machine may place its four +#: workers on any cards. Callers that own a plan already check the id against +#: that plan's assignments; this is the shape check underneath. +_WORKER_ID_RE = re.compile(r"worker_\d+") def _validate_worker(worker_id: str) -> None: - if worker_id not in WORKER_IDS: + if not _WORKER_ID_RE.fullmatch(str(worker_id)): raise ValueError(f"unknown worker: {worker_id}") diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/orchestrator.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/orchestrator.py index ccf9056d..8d74875c 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/orchestrator.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/orchestrator.py @@ -77,7 +77,6 @@ def run_with_requirements( extra_add_dirs=[workspace_dir], snapshot_file=state_dir / "agents.json", max_concurrent=4, - state_dir=state_dir, ) restore = install_subagent_shutdown_handlers(manager, pid_file=pid_file) try: diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/pipeline.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/pipeline.py index e4e876a4..df81b1e7 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/pipeline.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/pipeline.py @@ -44,7 +44,7 @@ def _phase(self, phase: str, **payload: Any) -> None: def run(self, *, dry_run: bool = False) -> Dict[str, Any]: task_id = str(self.req.get("task_id", "task")) - self.store.init_or_resume(task_id) + self.store.init_or_resume(task_id, "dcu-kernel-auto-opt") self.store.update_run(finished=False, final_status=None, notes=[]) started = time.time() try: diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/prompts.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/prompts.py index cc56ee10..40d9e318 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/prompts.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/prompts.py @@ -283,6 +283,62 @@ def w8a8_round_strategy( return decision +def hw_profile_guidance() -> str: + """Board-specific guidance, or ``""`` on any profile but the air board. + + Deliberately empty for liquid: the requirement is that the liquid prompt + stays byte-identical, and it is the air board whose memory behaviour the + generic guidance does not cover. Variants, the fixed baseline and the + staged reference tree are all per profile, so the agent has to know which + board produced the numbers it is reading (see + ``wiki-metainfer/knowledge-packs.md`` and + ``gotchas/air-machine-no-address-hashing.md``). + """ + try: + from metainfer.orchestrator import hw_profile + + resolution = hw_profile.resolve() + except Exception: # pragma: no cover - prompts must build regardless + return "" + profile = resolution.profile + if profile is None or profile.cooled != "air": + return "" + return f""" +## Hardware profile of this machine: AIR-COOLED ({profile.label}) + +This board has **no HBM address hashing**. When many threads read strided — a few +rows per pass, rows 4 KiB apart — the requests collapse onto the same memory +channels, so concurrent-read bandwidth is roughly half of the liquid board's +(370 vs 627 GB/s measured) and the operators that read most (MoE, decode) run +**1.06-1.40x slower** on the M=8 family; the gap is operator dependent (it is +widest on small-K, read-heavy shapes and smallest on large-K compute shapes). + +What that means for your rounds: + +- **Prefer longer, contiguous reads** (chunked or contiguous weight layouts, + merged/wider vector loads, one block resident and reused) over occupancy, + tile-size or LDS micro-tuning: the strided-read penalty is not something those + levers can buy back. +- Clock, NUMA, affinity and co-tenant effects **do not** explain this lane's + numbers; do not spend rounds on them. +- The staged `references/` variants and the fixed Triton baseline for this task + were measured **on this board** (each variant header carries + `hw_profile=air-k100ai machine=... measured_on=...`). Never compare against + numbers from the liquid board, and do not present a speedup computed against + another profile's baseline as a result. +- **One staged file is the exception**: the legacy flat bundle + `references/w8a8_gemm_variants.hip` predates the per-profile split and every + measurement in it comes from the **liquid** board. Read it for code ideas + only; its bandwidth/latency/occupancy figures do not hold on this board and + must be re-measured here before you rely on them. +- Treat the four air-side techniques (MoE block re-ordering, matmul solution + table re-selection, larger GDN convolution block parameters, decode dense + projection packing) as **reference ideas**, not a checklist; choose whichever + your measurements justify. + +""" + + def generate_kernel_prompt( *, operator: str, @@ -424,7 +480,7 @@ def generate_kernel_prompt( at least one shape. Use up to four GPUs; a subset task may use fewer when its operator-family grouping does not provide four independent units of work. Do not change any other file. -""" +{hw_profile_guidance()}""" def bootstrap_worker_prompt( @@ -578,7 +634,7 @@ def bootstrap_worker_prompt( ## Correctness-first bootstrap strategy {bootstrap_strategy} - +{hw_profile_guidance()} Include ``, use `hip_bfloat16`, and use its supported float conversion; alternatively use the existing proven manual uint16 store without switching type families. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/real_pipeline.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/real_pipeline.py index dc36db91..546d6529 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/real_pipeline.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/real_pipeline.py @@ -195,7 +195,7 @@ def _phase(self, phase: str, **payload: Any) -> None: def run(self, *, dry_run: bool = False) -> Dict[str, Any]: task_id = str(self.req.get("task_id", "task")) - self.store.init_or_resume(task_id) + self.store.init_or_resume(task_id, "dcu-kernel-auto-opt") self.store.update_run( finished=False, final_status=None, diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/recover_serial_validate.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/recover_serial_validate.py index 0d9f0645..596e7abe 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/recover_serial_validate.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/recover_serial_validate.py @@ -16,8 +16,10 @@ from __future__ import annotations import argparse +import dataclasses import json import os +import re import time from pathlib import Path @@ -39,8 +41,51 @@ def _load_json(path: Path) -> dict: return value if isinstance(value, dict) else {} +def select_shape_ids( + config, + *, + m_values: str = "", + shape_ids: str = "", +) -> "set[str] | None": + """Shapes one merge-and-validate batch covers, or ``None`` for all of them. + + A batch exists because a **weight** may only be packed once: the frozen API's + ``prepare_weight(raw_weight, weight_scale)`` has no ``M``, so every M variant + of one ``(N, K)`` must share a packed layout, while the worker contract lets + each shape's kernel bring its own. Merging one M at a time -- the way DKAO + tasks were always submitted before multi-M catalogs -- keeps every ``(N, K)`` + unique inside a batch, so each shape is served by its own kernel *and* its own + packing. See ``error_catalog/04-kernel-quality.md``. + """ + known = {str(shape.id): shape for shape in config.shapes.values()} + wanted: "set[str]" = set() + if m_values.strip(): + wanted_m = { + int(token) for token in re.split(r"[,\s]+", m_values.strip()) if token + } + for shape_id, shape in known.items(): + try: + value = int((shape.params or {}).get("M")) + except (TypeError, ValueError): + continue + if value in wanted_m: + wanted.add(shape_id) + if not wanted: + raise SystemExit(f"no shape has M in {sorted(wanted_m)}") + if shape_ids.strip(): + wanted |= { + token for token in re.split(r"[,\s]+", shape_ids.strip()) if token + } + if not wanted: + return None + unknown = sorted(wanted - set(known)) + if unknown: + raise SystemExit(f"unknown shape id(s): {unknown}") + return wanted + + def reconstruct_workers( - workspace_dir: Path, config + workspace_dir: Path, config, wanted: "set[str] | None" = None ) -> dict[str, dict]: """Rebuild ``workers[worker_id] = {"shapes": {shape_id: {...}}}`` from the accepted artifact manifests written by the (possibly restarted) lanes.""" @@ -50,6 +95,8 @@ def reconstruct_workers( root = workspace_dir / "workers" / assignment.worker_id lane_shapes: dict[str, dict] = {} for shape_id in assignment.shape_ids: + if wanted is not None and shape_id not in wanted: + continue manifest_path = root / "accepted" / shape_id / "manifest.json" if not manifest_path.is_file(): continue @@ -72,6 +119,20 @@ def reconstruct_workers( return workers +def restrict_to_batch(config, completed_assignments): + """The config as this batch sees it: only the batch's lanes and shapes. + + ``_synthesize_final_candidate`` re-derives the completed assignments from + ``config.assignments`` instead of taking them as an argument, so a batch that + only reconstructed some shapes has to be visible there too -- otherwise it + looks up a shape's metrics under a lane this batch never rebuilt (KeyError on + the first shape of another M). + """ + return dataclasses.replace( + config, assignments=list(completed_assignments) + ) + + def main() -> int: parser = argparse.ArgumentParser( prog="dcu-kernel-auto-opt-recover-serial-validate" @@ -87,6 +148,21 @@ def main() -> int: "resolved from agent_framework." ), ) + parser.add_argument( + "--m-values", + default="", + help=( + "Comma-separated M values to merge and validate in this batch " + "(e.g. '8'). One M per batch keeps every (N, K) unique, which is " + "what the frozen pack-once-per-weight API requires; empty means " + "every accepted shape." + ), + ) + parser.add_argument( + "--shape-ids", + default="", + help="Explicit shape ids for this batch (comma-separated).", + ) args = parser.parse_args() req = _load_json(args.requirements) @@ -104,17 +180,39 @@ def main() -> int: workspace = args.workspace_dir baseline = _load_json(workspace / "shared_baseline" / "results.json") initial_metrics = dict(baseline.get("shapes") or {}) - workers = reconstruct_workers(workspace, config) + wanted = select_shape_ids( + config, m_values=args.m_values, shape_ids=args.shape_ids + ) + if wanted is not None: + initial_metrics = { + key: value + for key, value in initial_metrics.items() + if key in wanted + } + workers = reconstruct_workers(workspace, config, wanted=wanted) if not workers: raise RuntimeError( "no accepted artifacts found under workers/*/accepted; " "cannot recover serial validation" ) - completed_assignments = [ - assignment - for assignment in config.assignments - if assignment.worker_id in workers - ] + completed_assignments = [] + for assignment in config.assignments: + shape_ids = [ + shape_id for shape_id in assignment.shape_ids + if wanted is None or shape_id in wanted + ] + if assignment.worker_id in workers and shape_ids: + completed_assignments.append( + dataclasses.replace(assignment, shape_ids=shape_ids) + ) + if not completed_assignments: + raise RuntimeError("no completed worker covers this batch") + print( + f"batch: M={args.m_values or 'all'} shapes=" + f"{sorted(s for w in workers.values() for s in w['shapes'])}", + flush=True, + ) + batch_config = restrict_to_batch(config, completed_assignments) print( f"recovered workers: {sorted(workers)} " f"({sum(len(w['shapes']) for w in workers.values())} shapes)", @@ -146,7 +244,7 @@ def main() -> int: ) try: merged_skill = pipeline._author_merged_skill( - config, completed_assignments + batch_config, completed_assignments ) except Exception as exc: # noqa: BLE001 - skill is documentation only print( @@ -161,7 +259,7 @@ def main() -> int: "reason": "recovery: skill synthesis agent unavailable", } synthesis = pipeline._synthesize_final_candidate( - config, workers, initial_metrics, task_id + batch_config, workers, initial_metrics, task_id ) validation = synthesis["validation"] worker_validation = { diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/skill_store.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/skill_store.py index 1c28ab7c..cf9cf27e 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/skill_store.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/skill_store.py @@ -36,9 +36,19 @@ def _write_candidate( source: str, content: str, ) -> Dict[str, Any]: + """Write one pending skill, stamped with this board. + + A worker/merged skill is **operator knowledge** -- its numbers were measured + on this machine -- so it carries the edition marker from the moment it is + written. That makes its provenance visible in the UI, and it means the + marker survives publication into the live library even if nothing else + stamps it. + """ target = pending_root / name target.mkdir(parents=True, exist_ok=True) - (target / "SKILL.md").write_text(content, encoding="utf-8") + (target / "SKILL.md").write_text( + stamp_edition(content, active_profile_id()), encoding="utf-8" + ) manifest = { "name": name, "kind": kind, @@ -268,29 +278,357 @@ def ccb_skills_root() -> Path: ) -def bundled_skills_root() -> Path: - """Skills bundled with this task plugin (seed content). +def plugin_skills_dir() -> Path: + """The plugin's vendored skill library, one directory per hardware profile. - The canonical skills this plugin curates (the INT8 W8A8 GEMM family) are - vendored under ``metainfer/tasks/dcu_kernel_auto_opt/skills/`` so a fresh - checkout carries them; ``seed_bundled_skills()`` installs any that are - missing from the live dsh library. + ``skills///SKILL.md``. Skills are knowledge, and knowledge + here is hardware dependent (an air board's kernel numbers are 1.06-1.70x off + the liquid board's, per operator), so each profile gets its own edition of + the same skill names -- see ``wiki-metainfer/knowledge-packs.md``. """ return Path(__file__).resolve().parents[1] / "skills" +def bundled_profile_dirs() -> Dict[str, Path]: + """``{profile_id: dir}`` for every profile the plugin bundles.""" + root = plugin_skills_dir() + found: Dict[str, Path] = {} + if not root.is_dir(): + return found + for child in sorted(root.iterdir()): + if not child.is_dir() or child.name.startswith("."): + continue + if (child / "SKILL.md").is_file(): + continue # itself a skill: this is the flat layout + if any((skill / "SKILL.md").is_file() for skill in child.glob("*/")): + found[child.name] = child + return found + + +def active_profile_id() -> str: + """This machine's hardware profile, or ``""`` when it cannot be resolved. + + Lazy import: ``skill_store`` must stay importable when the DKAO plugin is + loaded somewhere the resolver is not available (tests, a bare checkout). + """ + try: + from metainfer.orchestrator import hw_profile + + return hw_profile.profile_id() or "" + except Exception: # noqa: BLE001 - an unknown machine is a normal state + return "" + + +def _board_label() -> str: + """Human-readable board label (``K100_AI / gfx928``), or ``""``. + + Used only for prompt/prose; the *id* is what decides ownership. + """ + try: + from metainfer.orchestrator import hw_profile + + return hw_profile.label(fallback="") + except Exception: # noqa: BLE001 - an unknown machine is a normal state + return "" + + +def bundled_skills_root() -> Path: + """Seed content for **this machine's** profile. + + Falls back to the root itself for a pre-split checkout (skills stored flat), + so an unmigrated tree still seeds instead of silently seeding nothing. + """ + root = plugin_skills_dir() + profile = active_profile_id() + if profile: + candidate = root / profile + if candidate.is_dir(): + return candidate + return root + + +def skills_manifest_path() -> Path: + """Which profile's edition is installed, and with what content. + + Lives next to the library (never inside it): the skill provider treats every + direct child of the skills root as a skill, so bookkeeping files must stay + out of that directory. + """ + return dsh_skills_root().parent / ".metainfer-skills.json" + + +def skills_archive_root() -> Path: + """Where the other profile's editions are parked, one directory per profile.""" + return dsh_skills_root().parent / ".metainfer-skill-libraries" + + +def read_skills_manifest() -> Dict[str, Any]: + try: + payload = json.loads(skills_manifest_path().read_text(encoding="utf-8")) + except (OSError, ValueError): + return {} + return payload if isinstance(payload, dict) else {} + + +def _write_skills_manifest(profile: str, skills: Dict[str, str]) -> None: + path = skills_manifest_path() + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps({ + "hw_profile": profile, + "skills": skills, # name -> sha256(SKILL.md) as installed + "updated_at": time.strftime("%Y-%m-%d %H:%M:%S"), + }, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + + +def _skill_fingerprint(skill_dir: Path) -> str: + import hashlib + + skill_md = skill_dir / "SKILL.md" + try: + return hashlib.sha256(skill_md.read_bytes()).hexdigest()[:16] + except OSError: + return "" + + +#: Editions carry ```` as the first body +#: line. Ownership by marker (not by content hash) is what makes a swap safe: +#: an operator may edit or a round may publish into an edition, and it still +#: belongs to the board it was written for. +_EDITION_RE = re.compile(r"") + + +def skill_edition(skill_dir: Path) -> str: + """The profile an installed skill declares, or ``""`` when unmarked.""" + try: + text = (skill_dir / "SKILL.md").read_text(encoding="utf-8", errors="replace") + except OSError: + return "" + match = _EDITION_RE.search(text) + return match.group(1) if match else "" + + +def is_profile_owned(skill_dir: Path) -> bool: + """True when a skill declares a board, i.e. it is somebody's *operator* + knowledge. + + The marker is the split's only ownership signal and it means "operator + findings measured on ". A skill **without** one is shared knowledge + -- the DSH-provided ``hygon-dcu-kernel`` / ``dumma-dcu`` and friends describe + gfx928 itself, not this board's numbers -- so both boards may read it and no + swap may move it. That distinction is the whole point of the marker: only + operator findings split per board. + """ + return bool(skill_edition(skill_dir)) + + +def _edition_marker(profile: str) -> str: + return f"" + + +def stamp_edition(text: str, profile: str) -> str: + """Return ``text`` carrying exactly one ``metainfer-edition`` marker. + + Written as the first body line (right after the YAML frontmatter), the layout + the vendored editions use. Any previous marker is dropped, so a merge can + never leave the *other* board's declaration behind -- ownership is decided by + this marker and nothing else. + + ``profile == ""`` returns the text unchanged: a machine whose board cannot be + resolved must not claim one (``seed_bundled_skills`` refuses to seed in that + state for the same reason). + """ + if not profile: + return text + match = re.match(r"^(---\n.*?\n---\n)", text, flags=re.DOTALL) + head = match.group(1) if match else "" + body = _EDITION_RE.sub("", text[len(head):]).strip("\n") + marker = _edition_marker(profile) + if not head: + return f"{marker}\n\n{body}\n" if body else f"{marker}\n" + return f"{head}\n{marker}\n\n{body}\n" if body else f"{head}\n{marker}\n" + + +def _stamp_skill_file(skill_dir: Path, profile: str) -> bool: + """Stamp ``/SKILL.md`` in place; ``True`` when it changed.""" + skill_md = skill_dir / "SKILL.md" + try: + original = skill_md.read_text(encoding="utf-8", errors="replace") + except OSError: + return False + stamped = stamp_edition(original, profile) + if stamped == original: + return False + skill_md.write_text(stamped, encoding="utf-8") + return True + + +def _marked_skill_names(root: Path | None) -> set: + """Names in ``root`` that declare a board (see ``is_profile_owned``).""" + if root is None or not root.is_dir(): + return set() + names = set() + for skill in sorted(root.glob("*/")): + if (skill / "SKILL.md").is_file() and is_profile_owned(skill): + names.add(skill.name) + return names + + +def _bundled_fingerprints() -> Dict[str, Dict[str, str]]: + """``{profile: {skill_name: fingerprint}}`` over every bundled edition.""" + out: Dict[str, Dict[str, str]] = {} + for profile, directory in bundled_profile_dirs().items(): + out[profile] = { + skill.name: _skill_fingerprint(skill) + for skill in sorted(directory.glob("*/")) + if (skill / "SKILL.md").is_file() + } + return out + + +def _managed_skill_names(root: Path | None = None) -> set: + """Skill names this plugin's split owns, in ``root`` too. + + Ownership: what the plugin vendors (any profile's edition), what the manifest + records, and -- the part that makes fusion safe -- **anything that declares a + board**, wherever it sits. A fused or published operator skill is one of + ours even though the plugin never vendored it, so it has to be swept by a + profile swap like the vendored ones. Shared knowledge (no marker) stays out + of this set on purpose: both boards read it and nothing may move it. + """ + names = set() + for skills in _bundled_fingerprints().values(): + names.update(skills) + names.update(read_skills_manifest().get("skills") or {}) + names.update(_marked_skill_names(dsh_skills_root())) + names.update(_marked_skill_names(root)) + return names + + +def quarantine_foreign_skills(*, root: Path | None = None) -> Dict[str, Any]: + """Move the *other* board's editions out of a live library. + + Requirement (operator, 2026-10-06): the air board must read air skills, and + the two libraries stay separated. + + Scope (operator, 2026-10-07): **only operator knowledge splits.** A skill + that declares a board is this board's measured findings and is swapped like + the vendored editions; a skill without a marker is shared knowledge + (``hygon-dcu-kernel``, ``dumma-dcu``, ...) which both boards read and which + is never moved. That is what ``_managed_skill_names(root)`` encodes. + + Ownership is decided in this order: + + 1. the ```` marker inside the skill -- + authoritative, and stable even after the edition is edited or published + into, so a swap never leaves the other board's knowledge behind; + 2. failing that, the content fingerprint: only a skill whose ``SKILL.md`` + matches another profile's bundled edition byte-for-byte is recognised as + that board's (this is how the pre-marker libraries are classified). + + Anything else is the operator's/history's own business and is reported, not + moved: the live library is authoritative once a human has touched it. + + Moved, never deleted: ``~/.dsh/.metainfer-skill-libraries//``. + """ + root = root or dsh_skills_root() + profile = active_profile_id() + bundled = _bundled_fingerprints() + moved: list = [] + foreign_kept: list = [] + local: list = [] + current: list = [] + if not root.is_dir(): + return {"quarantined": moved, "foreign_unmodified": foreign_kept, + "local": local, "current": current, "profile": profile} + for name in sorted(_managed_skill_names(root=root)): + live = root / name + if not (live / "SKILL.md").is_file(): + continue + declared = skill_edition(live) + if declared == profile and declared: + current.append(name) # already this board's + continue + if declared: + owner = declared # the marker is decisive + else: + fingerprint = _skill_fingerprint(live) + if profile and fingerprint == (bundled.get(profile) or {}).get(name): + current.append(name) + continue # this profile's edition + owner = next( + (other for other, skills in bundled.items() + if other != profile and skills.get(name) == fingerprint), + None, + ) + if owner is None: + local.append(name) # edited/published here + continue + archive = skills_archive_root() / owner / name + archive.parent.mkdir(parents=True, exist_ok=True) + if archive.exists(): + shutil.rmtree(archive) + shutil.move(str(live), str(archive)) + moved.append({"name": name, "from_profile": owner, "archived_at": str(archive)}) + return {"quarantined": moved, "foreign_unmodified": foreign_kept, + "local": local, "current": current, "profile": profile} + + +def _restore_archived_owned_skills(root: Path, profile: str) -> list[str]: + """Move **this board's own** operator skills back out of the archive. + + A swap parks the other board's editions; when the library comes back to this + board, the operator skills this board wrote (fused or published, so *not* + vendored) have to come back too -- otherwise an air -> liquid -> air round + trip would silently drop merged knowledge out of the live library, and the + agent would read a library that is missing its own findings. + + Vendored names are deliberately skipped: those are the plugin's to seed, and + an older parked copy must not shadow a newer edition. Shared knowledge (no + marker) was never parked in the first place. + """ + archive = skills_archive_root() / profile + if not archive.is_dir(): + return [] + vendored: set = set() + for skills in _bundled_fingerprints().values(): + vendored.update(skills) + restored: list[str] = [] + for entry in sorted(archive.glob("*/")): + if entry.name in vendored or not (entry / "SKILL.md").is_file(): + continue + if (root / entry.name).exists(): + continue # a newer local copy wins + shutil.move(str(entry), str(root / entry.name)) + restored.append(entry.name) + return restored + + def seed_bundled_skills() -> Dict[str, Any]: - """Copy bundled plugin skills into the dsh library when missing. + """Install this profile's edition, after parking the other board's. + + Order matters: the foreign edition has to leave the library **before** the + seed step, or the (identical) skill name would be considered already + installed and the agent would keep reading the other board's numbers. - Idempotent and never overwrites: the live library is the authoritative - copy once an operator publishes/fuses over it, so only absent skills are - seeded. Mirrors to ccb are handled by ``sync_skill_libraries()``. + Within one profile the old behaviour is kept: an existing skill is never + overwritten, so a published or hand-edited skill survives. """ - source = bundled_skills_root() root = dsh_skills_root() + profile = active_profile_id() + source = bundled_skills_root() + quarantine = quarantine_foreign_skills(root=root) + if not profile: + # An unidentifiable machine gets no hardware-dependent knowledge: both + # editions are wrong somewhere. Say so instead of guessing. + return {"added": [], "root": str(root), "profile": "", + "quarantined": quarantine["quarantined"], + "reason": "hardware profile unresolved; refusing to seed " + "hardware-dependent skills"} if not source.exists(): - return {"added": [], "root": str(root)} + return {"added": [], "root": str(root), "profile": profile, + "quarantined": quarantine["quarantined"]} root.mkdir(parents=True, exist_ok=True) + restored = _restore_archived_owned_skills(root, profile) added: list[str] = [] for src_dir in sorted(source.glob("*/")): if not (src_dir / "SKILL.md").is_file(): @@ -303,7 +641,20 @@ def seed_bundled_skills() -> Dict[str, Any]: src_dir, dst, ignore=shutil.ignore_patterns("*.bak-*") ) added.append(name) - return {"added": added, "root": str(root)} + profile_dir = bundled_profile_dirs().get(profile) + bundled_names = ( + {skill.name for skill in profile_dir.glob("*/")} if profile_dir else set() + ) + installed = { + skill.name: _skill_fingerprint(skill) + for skill in sorted(root.glob("*/")) + if (skill / "SKILL.md").is_file() and skill.name in bundled_names + } + _write_skills_manifest(profile, installed) + return {"added": added, "root": str(root), "profile": profile, + "restored": restored, + "quarantined": quarantine["quarantined"], + "local_edits_kept": quarantine["local"]} def existing_skills_root() -> Path: @@ -396,13 +747,15 @@ def _write_sync_summary(workspace_dir: Path, summary: Dict[str, Any]) -> None: def sync_skill_libraries(*, workspace_dir: Path | None = None) -> Dict[str, Any]: - """Seed bundled plugin skills, then mirror dsh into ccb (one-way). + """Seed this profile's skills, then mirror dsh into ccb (one-way). First installs any bundled plugin skills (metainfer/.../dcu_kernel_auto_opt/ - skills) that are missing from the dsh library, then copies skills that are - new or changed in dsh into ccb; never deletes ccb-only skills. A ccb - SKILL.md is backed up before being overwritten so a bad mirror can be - rolled back. Idempotent and safe to call repeatedly. + skills/) that are missing from the dsh library -- after parking the + *other* profile's editions, so this machine never reads the other board's + numbers (``quarantine_foreign_skills``) -- then copies skills that are new or + changed in dsh into ccb; never deletes ccb-only skills. A ccb SKILL.md is + backed up before being overwritten so a bad mirror can be rolled back. + Idempotent and safe to call repeatedly. """ seeded = seed_bundled_skills() source = dsh_skills_root() @@ -412,6 +765,9 @@ def sync_skill_libraries(*, workspace_dir: Path | None = None) -> Dict[str, Any] skipped: list[str] = [] if source.exists(): target.mkdir(parents=True, exist_ok=True) + # The mirror must be profile-clean too, or a ccb agent would read the + # other board's edition of a skill the dsh library no longer has. + ccb_quarantine = quarantine_foreign_skills(root=target) for src_dir in sorted(source.glob("*/")): src_skill = src_dir / "SKILL.md" if not src_skill.is_file(): @@ -428,13 +784,21 @@ def sync_skill_libraries(*, workspace_dir: Path | None = None) -> Dict[str, Any] updated.append(name) else: skipped.append(name) + else: + ccb_quarantine = {"quarantined": []} summary = { "source": str(source), "target": str(target), + "hw_profile": seeded.get("profile", ""), + "quarantined": list(seeded.get("quarantined") or []) + + list(ccb_quarantine.get("quarantined") or []), "added": added, "updated": updated, "skipped": skipped, "seeded": sorted(seeded.get("added") or []), + # This board's own operator skills that came back out of the archive + # (a swap away and back must not lose fused findings). + "restored": sorted(seeded.get("restored") or []), "ccb_only": sorted(_skill_name_set(target) - _skill_name_set(source)), "ts": time.time(), } @@ -444,7 +808,12 @@ def sync_skill_libraries(*, workspace_dir: Path | None = None) -> Dict[str, Any] def rollback_skill(name: str, *, workspace_dir: Path) -> Dict[str, Any]: - """Restore the latest SKILL.md backup in the dsh library, then re-mirror.""" + """Restore the latest SKILL.md backup in the dsh library, then re-mirror. + + The restored body is re-stamped with this board: a backup taken before the + edition markers existed (or before fusion stamped them) must not come back + unmarked, or it would drop out of the profile split. + """ skill_dir = dsh_skills_root() / name if not skill_dir.is_dir(): raise FileNotFoundError(f"skill not found in dsh library: {name}") @@ -454,6 +823,7 @@ def rollback_skill(name: str, *, workspace_dir: Path) -> Dict[str, Any]: latest = backups[-1] shutil.copy2(latest, skill_dir / "SKILL.md") latest.unlink() + _stamp_skill_file(skill_dir, active_profile_id()) sync = sync_skill_libraries(workspace_dir=workspace_dir) return { "name": name, @@ -501,7 +871,21 @@ def _build_fuse_prompt( existing: list[Dict[str, Any]], pending_content: str, decision_path: Path, + *, + profile: str = "", + board_label: str = "", ) -> str: + """The fusion agent's instructions. + + Two things the agent cannot infer and must be told, because getting them + wrong files one board's measurements under the other: + + * **which board this is** -- the library holds that board's editions and the + write lands there, so only findings measured here may be folded in; + * **which existing skills are this board's operator knowledge** (marked) and + which are shared knowledge both boards read (unmarked, never merged into). + """ + board = f"{profile} ({board_label})" if board_label else (profile or "unknown") lines = [ "You are the skill-fusion agent for DCU kernel optimization.", "", @@ -510,13 +894,36 @@ def _build_fuse_prompt( "is mirrored to Claude Code automatically, so this one decision applies", "to both frameworks.", "", - "## Existing skills in the library (name, description, body excerpt)", + "## This machine's board", + "", + f"This machine is **{board}**, and the library holds that board's editions", + "(``~/.dsh/skills``). Air boards lack HBM address hashing, so the same", + "kernel measures 1.06-1.70x apart between boards -- a number measured on", + "one board is wrong on the other. Fold in only material measured on this", + "board; nothing else is offered to you.", + "", + "Each existing skill below is labelled with its ownership:", + "", + "* ``operator knowledge []`` -- board-specific measured findings,", + " i.e. the same kind of material as the new skill. Merging is allowed", + " here (and only here).", + "* ``shared knowledge [no board]`` -- describes gfx928/DTK itself and is", + " read by **both** boards. Never merge board-specific numbers into one:", + " a refusal is raised server-side. If the new material is really generic", + " the operator will promote it separately.", + "", + "## Existing skills in the library (name, ownership, body excerpt)", "", ] if not existing: lines.append("(the library is empty — add a new skill)") for item in existing: - lines.append(f"### {item.get('name')}") + declared = str(item.get("edition") or "") + ownership = ( + f"operator knowledge [{declared}]" if declared + else "shared knowledge [no board]" + ) + lines.append(f"### {item.get('name')} — {ownership}") lines.append(f"description: {item.get('description') or ''}") body = item.get("content") or "" if body.count("---\n") >= 2: @@ -544,8 +951,11 @@ def _build_fuse_prompt( "", '- action "new": name must be a short kebab-case id (<=63 chars) that does', " not collide with an existing skill; content is the complete body.", - '- action "merge": name must be EXACTLY an existing skill\'s name; content is', - " the complete UPDATED body after folding the new material in.", + '- action "merge": name must be EXACTLY an existing skill\'s name **that is', + " labelled operator knowledge**; content is the complete UPDATED body after", + " folding the new material in.", + "- A skill files the board automatically: write the body only, no", + " ``metainfer-edition`` marker (the server stamps the machine's board).", "- Preserve all existing working knowledge; add the new evidence/rules where", " they fit.", "- The body must be concise, actionable Markdown for a kernel-optimization", @@ -596,14 +1006,32 @@ def _parse_fuse_decision(text: str) -> Dict[str, Any]: } -def _apply_fuse_decision(decision: Dict[str, Any]) -> Dict[str, Any]: - """Apply a parsed fusion decision into the dsh library.""" +def _apply_fuse_decision( + decision: Dict[str, Any], *, profile: str +) -> Dict[str, Any]: + """Apply a parsed fusion decision into the dsh library. + + Everything written here is **this board's operator knowledge**, so it is + stamped with ``profile`` -- the marker is what makes a later profile swap + move it (``quarantine_foreign_skills``) instead of leaving this board's + numbers in the other board's library. + + A merge into shared knowledge is refused rather than stamped: ``hygon-dcu- + kernel`` and friends carry no board and are read by both boards, so folding + this board's measured numbers into one would either mislabel shared truth or + silently un-share it. Board-specific findings belong in a ``new`` skill. + """ root = dsh_skills_root() root.mkdir(parents=True, exist_ok=True) action = decision["action"] name = decision["name"] content = decision["content"] description = decision["description"] + if not profile: + raise RuntimeError( + "hardware profile unresolved: refusing to file operator knowledge " + "that cannot be attributed to a board" + ) if action == "new": target = root / name @@ -613,23 +1041,48 @@ def _apply_fuse_decision(decision: Dict[str, Any]) -> Dict[str, Any]: "choose merge or a new name" ) target.mkdir(parents=True, exist_ok=True) - body = _frontmatter( - name, description or f"{name} optimization skill" - ) + content + "\n" + body = stamp_edition( + _frontmatter(name, description or f"{name} optimization skill") + + content + "\n", + profile, + ) (target / "SKILL.md").write_text(body, encoding="utf-8") - return {"action": "new", "name": name, "path": str(target / "SKILL.md")} + return { + "action": "new", "name": name, "path": str(target / "SKILL.md"), + "edition": profile, + } target = root / name if not (target / "SKILL.md").is_file(): raise FileNotFoundError( f"merge target skill {name} does not exist in the dsh library" ) + declared = skill_edition(target) + if not declared: + raise ValueError( + f"refusing to merge {profile!r} findings into {name!r}: that skill " + "is shared knowledge (no metainfer-edition marker), so both boards " + "read it and this board's numbers must not be folded into it. Ask " + "for a new skill instead." + ) + if declared != profile: + raise ValueError( + f"refusing to merge into {name!r}: it is the {declared!r} edition " + f"while this machine is {profile!r}; the library still holds the " + "other board's knowledge -- run 「同步 Skill 库」 first" + ) previous = (target / "SKILL.md").read_text(encoding="utf-8", errors="replace") backup = _backup_skill(target) - # Keep the existing frontmatter (the dir name is the skill identity); - # only the body is replaced. + # Keep the existing frontmatter (the dir name is the skill identity) and the + # board; only the body is replaced. Re-stamping also repairs an edition + # whose marker the agent failed to reproduce. fm = re.match(r"^(---\n.*?\n---\n)", previous, flags=re.DOTALL) - body = (fm.group(1) if fm else _frontmatter(name, description or f"{name} optimization skill")) + content + "\n" + body = stamp_edition( + (fm.group(1) if fm else _frontmatter( + name, description or f"{name} optimization skill" + )) + content + "\n", + profile, + ) (target / "SKILL.md").write_text(body, encoding="utf-8") diff = "".join(difflib.unified_diff( previous.splitlines(keepends=True), @@ -643,6 +1096,7 @@ def _apply_fuse_decision(decision: Dict[str, Any]) -> Dict[str, Any]: "path": str(target / "SKILL.md"), "backup": str(backup) if backup else None, "diff": diff, + "edition": profile, } @@ -659,7 +1113,17 @@ def fuse_skill( The agent scans the existing dsh library, writes a JSON decision (new file vs merge into an existing skill), the decision is applied to the dsh library, and the result is mirrored to the ccb library. + + The result is **this board's** operator knowledge: it is stamped with the + machine's edition so a later profile swap carries it along, and a machine + whose board cannot be resolved refuses to fuse at all. """ + profile = active_profile_id() + if not profile: + raise RuntimeError( + "hardware profile unresolved: refusing to fuse operator knowledge " + "that cannot be attributed to a board" + ) pending = workspace_dir / "skills" / "pending" / skill_name if not (pending / "SKILL.md").is_file(): raise FileNotFoundError(f"pending skill not found: {skill_name}") @@ -667,13 +1131,17 @@ def fuse_skill( encoding="utf-8", errors="replace" ) + library_root = dsh_skills_root() existing = [ { "name": item["name"], "description": _frontmatter_description(item["content"]), "content": item["content"], + # Ownership, so the agent knows which targets are this board's + # operator knowledge and which are shared and untouchable. + "edition": skill_edition(library_root / item["name"]), } - for item in _skill_entries(dsh_skills_root(), status="existing") + for item in _skill_entries(library_root, status="existing") ] fuse_dir = workspace_dir / "skills" / "fuse" @@ -682,7 +1150,10 @@ def fuse_skill( decision_path.unlink(missing_ok=True) prompt_file = fuse_dir / "prompt.txt" prompt_file.write_text( - _build_fuse_prompt(existing, pending_content, decision_path), + _build_fuse_prompt( + existing, pending_content, decision_path, + profile=profile, board_label=_board_label(), + ), encoding="utf-8", ) @@ -722,7 +1193,7 @@ def fuse_skill( decision = _parse_fuse_decision( decision_path.read_text(encoding="utf-8", errors="replace") ) - applied = _apply_fuse_decision(decision) + applied = _apply_fuse_decision(decision, profile=profile) sync = sync_skill_libraries(workspace_dir=workspace_dir) outcome = { **applied, @@ -747,9 +1218,13 @@ def fuse_skill( def list_skill_library(workspace_dir: Path) -> Dict[str, Any]: pending_root = workspace_dir / "skills" / "pending" + manifest = read_skills_manifest() return { "existing_root": str(dsh_skills_root()), "ccb_mirror_root": str(ccb_skills_root()), + # Which board's editions this library holds (see ``seed_bundled_skills``). + "hw_profile": active_profile_id(), + "installed_profile": manifest.get("hw_profile", ""), "existing": _skill_entries(dsh_skills_root(), status="existing"), "pending": _skill_entries(pending_root, status="pending"), "fuse_status": read_fuse_status(workspace_dir), @@ -759,7 +1234,18 @@ def list_skill_library(workspace_dir: Path) -> Dict[str, Any]: def publish_skill(workspace_dir: Path, name: str) -> Dict[str, Any]: """Publish one exact pending directory into the dsh library, then mirror - the result to the ccb library. Never overwrites an existing skill.""" + the result to the ccb library. Never overwrites an existing skill. + + The published skill is operator knowledge measured on this machine, so it is + stamped with this board's edition: without the marker a profile swap would + leave it in the other board's library (see ``_managed_skill_names``). + """ + profile = active_profile_id() + if not profile: + raise ValueError( + "hardware profile unresolved: refusing to publish operator " + "knowledge that cannot be attributed to a board" + ) if _slug(name) != name: raise ValueError("invalid skill name") source = workspace_dir / "skills" / "pending" / name @@ -773,6 +1259,7 @@ def publish_skill(workspace_dir: Path, name: str) -> Dict[str, Any]: shutil.copytree( source, target, ignore=shutil.ignore_patterns("manifest.json") ) + _stamp_skill_file(target, profile) published = workspace_dir / "skills" / "published" / name published.parent.mkdir(parents=True, exist_ok=True) shutil.move(str(source), str(published)) @@ -781,5 +1268,6 @@ def publish_skill(workspace_dir: Path, name: str) -> Dict[str, Any]: "name": name, "path": str(target), "status": "existing", + "edition": profile, "synced_to_ccb": {"added": sync["added"], "updated": sync["updated"]}, } diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/variant_promote.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/variant_promote.py index daa6366d..393350d5 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/variant_promote.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/variant_promote.py @@ -26,6 +26,7 @@ derive_variant_meta, variant_path, ) +from .w8a8_baselines import trusted_baseline_us #: shape-id prefix -> the model label used by the variant taxonomy. MODEL_LABEL_BY_PREFIX = { @@ -75,7 +76,15 @@ def accepted_kernel_for(workspace_dir: Path, def _baseline_us(workspace_dir: Path, shape_id: str, shape_params: Dict[str, Any]) -> Optional[float]: - """Baseline for the recorded speedup: task table first, fixed table next.""" + """Baseline for the recorded speedup: task table first, fixed table next. + + Either source is accepted **only** when the number provably came from this + machine's hardware profile (:func:`trusted_baseline_us`). A task report + written before the profile split -- or one that fell back to the reference + table -- carries the other board's number, and recording a ratio against it + would file a cross-machine comparison as a same-machine fact. The variant + is still written in that case; only ``baseline_us``/``speedup`` are omitted. + """ report = workspace_dir / "final_report.json" if report.is_file(): try: @@ -84,14 +93,15 @@ def _baseline_us(workspace_dir: Path, shape_id: str, except (OSError, ValueError): initial = {} entry = initial.get(shape_id) - if isinstance(entry, dict) and entry.get("median_us"): - return float(entry["median_us"]) - if isinstance(entry, (int, float)): - return float(entry) + if entry is not None: + trusted = trusted_baseline_us(entry) + if trusted is not None: + return trusted try: from .w8a8_baselines import fixed_triton_graph_baseline params = dict(shape_params or {}) - return float(fixed_triton_graph_baseline(shape_id, params)["median_us"]) + record = fixed_triton_graph_baseline(shape_id, params) + return trusted_baseline_us(record) except Exception: # noqa: BLE001 - baseline is advisory return None diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/variant_store.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/variant_store.py index 879a1c38..4b49e823 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/variant_store.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/variant_store.py @@ -1,24 +1,34 @@ """Fine-grained implementation variants for agent reference. Variants are accepted kernels stored as a directory tree that mirrors the -form taxonomy: +form taxonomy, **under a hardware-profile directory**: - variant/////.hip + variant//////.hip -e.g. ``variant/int8w8a8-gemm/hy3/TP4/M4096/o_proj.hip``. +e.g. ``variant/air-k100ai/int8w8a8-gemm/hy3/TP4/M4096/o_proj.hip``. + +Why the profile level exists: two machines share one chip and one codebase but +differ in board firmware behaviour (air-cooled boards have no HBM address +hashing, so read-heavy operators are 40-87% slower there). A variant's +``median_us`` is only meaningful against the machine that produced it, so +air-measured and liquid-measured kernels must never share a leaf. The profile +id comes from :mod:`hw_profile` (detected, never typed in); see +``wiki-metainfer/knowledge-packs.md``. Each leaf file carries a machine-parseable header: // @@variant shape= commit= added= // median_us=.. p90_us=.. tops=.. bandwidth_gb_s=.. speedup=.. baseline_us=.. // source= + // hw_profile= machine= measured_on= Semantics (kept in sync with prompts.py): variants are evidence, not policy — agents may read and adapt the file matching their (operator/dtype, model, TP, M, operator) family, but must not be locked into it, and any reuse requires full re-validation. The tree is staged read-only into generated kernel -repositories under ``references/variants/`` and is not part of the build. +repositories under ``references//`` (only the profile of the machine +that stages it) and is not part of the build. """ from __future__ import annotations @@ -29,6 +39,35 @@ from pathlib import Path from typing import Any, Dict, List, Optional +from metainfer.orchestrator import hw_profile + +#: Directory name shown for entries that predate the profile split (a legacy +#: ``variant//...`` layout). Never written to; it exists so an +#: unmigrated checkout is *visible* in the index instead of silently +#: masquerading as the current machine's profile. +UNMIGRATED_PROFILE = "unmigrated" + + +def profile_ids() -> tuple: + """Registered profile ids, in a stable order.""" + return tuple(sorted(hw_profile.PROFILES)) + + +def current_profile_id() -> str: + """The profile of *this* machine, or ``UNMIGRATED_PROFILE`` if unresolved.""" + try: + return hw_profile.profile_id() or UNMIGRATED_PROFILE + except hw_profile.HardwareProfileError: + return UNMIGRATED_PROFILE + + +def _split_profile(rel_parts: tuple) -> tuple: + """Return ``(profile, rest)`` for one tree-relative path tuple.""" + if rel_parts and rel_parts[0] in hw_profile.PROFILES: + return rel_parts[0], rel_parts[1:] + return UNMIGRATED_PROFILE, rel_parts + + _HEADER_RE = re.compile(r"(?m)^// @@variant shape=(\S+)(.*?)\n(?:.*?\n)*?// @@end") # Captures the whole header block (first line tail + ``// key=value`` lines) # up to ``// @@end`` for structured parsing of metrics/source/commit. @@ -112,22 +151,37 @@ def variant_root() -> Path: return Path(__file__).resolve().parents[1] / "variant" -def variant_path(meta: Dict[str, Any]) -> Path: - """The leaf file path for one specific operator's variant.""" +def variant_path(meta: Dict[str, Any], *, profile: Optional[str] = None) -> Path: + """The leaf file path for one specific operator's variant. + + ``profile`` defaults to the profile detected for *this* machine: variants + are per-hardware evidence, so writing must land in the current profile's + subtree (an unresolved machine gets ``unmigrated``, which is visible in the + index rather than silently wrong). + """ + profile = profile or current_profile_id() tp = f"TP{meta['tp']}" if meta.get("tp") is not None else "TP?" m = f"M{meta['m']}" if meta.get("m") is not None else "M?" return ( - variant_root() / meta["family"] / meta["model"] / tp / m + variant_root() / profile / meta["family"] / meta["model"] / tp / m / f"{meta['operator_name']}.hip" ) +def profile_root(profile: Optional[str] = None) -> Path: + """The per-profile subtree that staging copies into agent workspaces.""" + return variant_root() / (profile or current_profile_id()) + + def section_header( meta: Dict[str, Any], *, commit: str, metrics: Dict[str, Any], source_task: str = "", + profile: Optional[str] = None, + machine: str = "", + measured_on: str = "", ) -> str: lines = [ f"// @@variant shape={meta['shape']} commit={commit or '?'} " @@ -151,10 +205,27 @@ def section_header( lines.append("// " + " ".join(metric_parts)) if source_task: lines.append(f"// source={source_task}") + # Which machine produced these numbers. Read by the variant index and the + # WebUI so a liquid variant can never be mistaken for an air measurement. + lines.append( + "// hw_profile={} machine={} measured_on={}".format( + profile or current_profile_id(), + machine or _hostname(), + measured_on or time.strftime("%Y-%m-%d"), + ) + ) lines.append("") return "\n".join(lines) +def _hostname() -> str: + import os + try: + return os.uname().nodename + except AttributeError: # pragma: no cover - non-POSIX + return "unknown" + + def add_variant( *, meta: Dict[str, Any], @@ -164,6 +235,8 @@ def add_variant( source_task: str = "", backup: bool = True, reject_slower_than_existing: bool = False, + profile: Optional[str] = None, + require_profile: bool = False, ) -> Dict[str, Any]: """Write one specific operator's accepted kernel into the variant tree. @@ -172,13 +245,28 @@ def add_variant( and ``metrics`` carry a ``median_us``, a candidate that is strictly slower than the existing variant is rejected with ``ValueError`` (equal or faster may replace). + + The leaf lives under the detected hardware profile, so a kernel measured on + one machine can never overwrite another machine's variant. Pass + ``require_profile=True`` at user-facing write boundaries to fail fast when + the machine cannot be identified (instead of writing into ``unmigrated``). """ - target = variant_path(meta) + if require_profile: + detected = hw_profile.require().id + if profile is not None and profile != detected: + raise ValueError( + f"refusing cross-profile variant write: requested profile " + f"{profile!r} but this machine is {detected!r}" + ) + profile = detected + profile = profile or current_profile_id() + target = variant_path(meta, profile=profile) target.parent.mkdir(parents=True, exist_ok=True) if target.exists() and reject_slower_than_existing: - _reject_slower_replacement(target, metrics) + _reject_slower_replacement(target, metrics, profile=profile) header = section_header( - meta, commit=commit, metrics=metrics, source_task=source_task + meta, commit=commit, metrics=metrics, source_task=source_task, + profile=profile, ) content = header + kernel_source.rstrip() + "\n// @@end\n" action = "updated" if target.exists() else "added" @@ -187,19 +275,33 @@ def add_variant( backup_path = target.with_name(f"{target.name}.bak-{int(time.time() * 1000)}") shutil.copy2(target, backup_path) target.write_text(content, encoding="utf-8") + # The reader caches the index; a promotion must be visible at once rather + # than after the cache expires (see list_variant_index). + invalidate_variant_index() return { "action": action, "shape": meta["shape"], "path": str(target), "backup": str(backup_path) if backup_path else None, + "profile": profile, "meta": meta, } -def _reject_slower_replacement(target: Path, metrics: Dict[str, Any]) -> None: +def _reject_slower_replacement( + target: Path, metrics: Dict[str, Any], *, profile: Optional[str] = None +) -> None: """Raise ValueError when the candidate is strictly slower than the - existing variant's recorded median (both must be known).""" + existing variant's recorded median (both must be known). + + Only meaningful within one profile: the recorded median was measured on a + specific machine, so a mismatch means the numbers are not comparable and + the guard is skipped rather than applied to the wrong baseline. + """ existing = _parse_variant_header(target.read_text(encoding="utf-8", errors="replace")) + existing_profile = existing.get("hw_profile") + if profile and existing_profile and str(existing_profile) != str(profile): + return existing_median = existing.get("median_us") new_median = metrics.get("median_us") if existing_median is None or new_median is None: @@ -214,13 +316,59 @@ def _reject_slower_replacement(target: Path, metrics: Dict[str, Any]) -> None: ) +# The index is a full walk + parse of every variant kernel (tens of files, a +# few hundred KB of .hip), which measures at ~0.4s. It is also read by *every* +# DKAO detail view on a timer, and each of those reads is synchronous inside +# the single WebUI event loop -- three open task tabs were enough to keep the +# whole UI (including the task list) waiting behind it. The library only +# changes when a round promotes a kernel, so the index is cached and the cache +# is invalidated by a cheap signature (file count + newest mtime) instead of +# being rebuilt per request. +_INDEX_CACHE: Dict[str, Any] = {} +_INDEX_TTL_S = 60.0 + + +def _tree_signature(root: Path) -> tuple: + count = 0 + newest = 0.0 + for file in root.rglob("*.hip"): + try: + st = file.stat() + except OSError: + continue + count += 1 + newest = max(newest, st.st_mtime) + return (count, round(newest, 3), str(root)) + + def list_variant_index() -> List[Dict[str, Any]]: """Walk the variant tree and return the section index (path-derived - taxonomy + header fields).""" - out: List[Dict[str, Any]] = [] + taxonomy + header fields). + + Cached: see ``_INDEX_CACHE``. A caller that just added a variant must call + ``invalidate_variant_index()`` (the write path does), and the signature + check catches any change made outside this process. + """ root = variant_root() if not root.exists(): - return out + return [] + signature = _tree_signature(root) + cached = _INDEX_CACHE.get("entry") + if cached and cached[0] == signature \ + and (time.monotonic() - cached[1]) < _INDEX_TTL_S: + return cached[2] + out = _build_variant_index(root) + _INDEX_CACHE["entry"] = (signature, time.monotonic(), out) + return out + + +def invalidate_variant_index() -> None: + """Drop the cached index (called by every write path).""" + _INDEX_CACHE.clear() + + +def _build_variant_index(root: Path) -> List[Dict[str, Any]]: + out: List[Dict[str, Any]] = [] # skip the legacy single-file variant (kept for historical reference) for file in sorted(root.rglob("*.hip")): if file.name.startswith("w8a8_gemm_variants"): @@ -229,13 +377,23 @@ def list_variant_index() -> List[Dict[str, Any]]: header = _parse_variant_header(text) shape = header.get("shape") or file.stem rel = file.relative_to(root).parts + # variant//////.hip (pre-split trees + # have no profile level and report UNMIGRATED_PROFILE instead). + path_profile, rest = _split_profile(rel) out.append({ "path": str(file.relative_to(root)), + # Layout profile (where the file lives) and the recorded profile + # (what the header claims) are reported separately: a disagreement + # means someone moved a file by hand. + "profile": path_profile, + "hw_profile": header.get("hw_profile", ""), + "machine": header.get("machine", ""), + "measured_on": header.get("measured_on", ""), "shape": shape, - "family": rel[0] if len(rel) > 0 else "", - "model": rel[1] if len(rel) > 1 else "", - "tp": rel[2] if len(rel) > 2 else "", - "m": rel[3] if len(rel) > 3 else "", + "family": rest[0] if len(rest) > 0 else "", + "model": rest[1] if len(rest) > 1 else "", + "tp": rest[2] if len(rest) > 2 else "", + "m": rest[3] if len(rest) > 3 else "", "operator": file.stem, "commit": header.get("commit", ""), "added": header.get("added", ""), @@ -254,7 +412,9 @@ def list_variant_index() -> List[Dict[str, Any]]: _HEADER_NUMERIC_FIELDS = frozenset({ "median_us", "p90_us", "tops", "bandwidth_gb_s", "speedup", "baseline_us", }) -_HEADER_TEXT_FIELDS = frozenset({"commit", "added", "source"}) +_HEADER_TEXT_FIELDS = frozenset({ + "commit", "added", "source", "hw_profile", "machine", "measured_on", +}) def _parse_variant_header(text: str) -> Dict[str, Any]: @@ -302,6 +462,7 @@ def backfill_variant_baselines() -> Dict[str, Any]: ) from metainfer.tasks.dcu_kernel_auto_opt.orchestrator.w8a8_baselines import ( # noqa: PLC0415 fixed_triton_graph_baseline, + trusted_baseline_us, ) tables: Dict[tuple, Dict[str, tuple]] = { @@ -356,7 +517,7 @@ def backfill_variant_baselines() -> Dict[str, Any]: k, n = kn m = int(m_dir[1:]) try: - baseline = fixed_triton_graph_baseline( + record = fixed_triton_graph_baseline( header.get("shape") or file.stem, { "tp_size": int(tp[2:]), @@ -364,7 +525,10 @@ def backfill_variant_baselines() -> Dict[str, Any]: "N": n, "K": k, }, - ).get("median_us") + ) + # Only backfill from this machine's own profile: the reference table + # belongs to the other board, so the ratio would not be reproducible. + baseline = trusted_baseline_us(record) except ValueError: baseline = None if baseline is None: diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/w8a8_baselines.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/w8a8_baselines.py index 0aacc03d..5d8c122e 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/w8a8_baselines.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/w8a8_baselines.py @@ -1,335 +1,274 @@ -"""Control-plane-owned W8A8 Triton Graph comparison baselines.""" +"""Profile-scoped Triton W8A8 Graph baselines. + +Baselines are **hardware dependent**, so they are stored per hardware profile: + + baselines/.json + +Background (see ``wiki-metainfer/knowledge-packs.md``): the two machines share +one chip and one codebase, but the air-cooled board has no HBM address hashing, +so its concurrent strided reads lose about half the bandwidth. Measured on +2026-09-29 against 15 TP8 M=8 shapes, the air Triton baseline is **1.06x-1.40x +slower** than the liquid one, and the gap is *operator dependent* (worst on +small-K, read-heavy ``shared_down_proj`` shapes; smallest on large-K compute +bound ones). A single scaling factor would therefore be wrong, and comparing an +air measurement against a liquid baseline silently overstates the gap. + +Lookup order for a shape: + +1. the machine's own profile table -> ``profile_mismatch=False`` +2. the reference (liquid) table -> ``profile_mismatch=True``, and the + record says so; callers that record knowledge (variant speedups, backfills) + must **not** turn this into a "speedup" +3. neither table -> :class:`BaselineUnavailable` + +Records keep the historical field set (``median_us``, ``baseline_us``, +``baseline_kind``, ``case``, ``tp_size``, ``shape``, ``timing_scope``, +``distribution_stats_available``, plus the TP8 distribution fields) and add the +provenance: ``baseline_profile``, ``baseline_machine``, ``baseline_measured_on``, +``profile_mismatch``, ``profile_note``. + +To measure a new profile (or extend one), run the recorded protocol and write +the result into its JSON: ``baseline/bench_triton_tp8_m4096.py`` and +``baseline/bench_triton_tp8_m8.py`` (three independent passes, per-shape median). +""" from __future__ import annotations -from typing import Any, Dict, Mapping - - -# User-supplied, fixed decode/prefill Graph measurements. These values are -# comparison targets, not measurements of the child-generated bootstrap HIP -# source. Latencies are stored in microseconds. -_TRITON_GRAPH_BASELINES_US = { - (4, 2, 1536, 4096): ("wqkv_a", 66.924), - (4, 16, 1536, 4096): ("wqkv_a", 66.597), - (4, 2, 8192, 1024): ("wq_b", 47.181), - (4, 16, 8192, 1024): ("wq_b", 48.049), - (4, 2, 4096, 2048): ("wo_b", 54.453), - (4, 16, 4096, 2048): ("wo_b", 54.617), - (4, 2, 1024, 4096): ("shared_gate_up", 60.389), - (4, 16, 1024, 4096): ("shared_gate_up", 60.965), - (4, 2, 4096, 512): ("shared_down", 18.975), - (4, 16, 4096, 512): ("shared_down", 21.372), - # M=3072 prefill baselines supplied on 2026-08-03. - (4, 3072, 1536, 4096): ("wqkv_a", 9488.0), - (4, 3072, 8192, 1024): ("wq_b", 15437.0), - (4, 3072, 4096, 2048): ("wo_b", 14680.0), - (4, 3072, 1024, 4096): ("shared_gate_up", 5985.0), - (4, 3072, 4096, 512): ("shared_down", 4199.0), - # M=4096 prefill baselines measured on 2026-08-06 (worker29, gfx928) with - # lmslim int8_utils.matmul_kernel (the W8A8 Triton baseline), GPU events, - # CUDA Graph replay, hot cache, warmups=10, samples=20, launches/sample=5. - # The (4096, 8192, 1024) entry also covers indexer.wq_b, which shares the - # same logical (K, N) shape as wq_b. - (4, 4096, 1536, 4096): ("wqkv_a", 13247.253), - (4, 4096, 8192, 1024): ("wq_b", 20590.225), - (4, 4096, 4096, 2048): ("wo_b", 19881.949), - (4, 4096, 1024, 4096): ("shared_gate_up", 8790.192), - (4, 4096, 4096, 512): ("shared_down", 5545.821), - # Model-catalog baselines measured on 2026-08-12 (worker29, gfx928) with - # lmslim int8_utils.matmul_kernel (the W8A8 Triton baseline), GPU events, - # CUDA Graph replay, hot cache, warmups=10, samples=20, launches/sample=5. - # Config follows int8_utils.matmul_int8 defaults per M (decode M<=32 uses - # BM16/BN32/BK256; M>1024 uses BM256/BN256/BK64). Covers the DeepSeek TP1 - # and TP8 M=2 additions plus the Hy3 (Hunyuan 3), MiniMax M3 and GLM5.2 - # catalogs. Keys are shared across models when (tp, M, N, K) coincide, - # e.g. (1, *, 8192, 4096) is DeepSeek wo_b / Hy3 o_proj and (4, *, 8192, - # 1024) covers indexer.wq_b. - (1, 2, 1536, 4096): ("wqkv_a", 78.046), - (1, 2, 2624, 6144): ("fused_qkv_a_proj", 96.623), - (1, 2, 3072, 4096): ("shared_gate_up_proj", 81.854), - (1, 2, 4096, 1536): ("shared_down_proj", 47.839), - (1, 2, 4096, 2048): ("shared_down_proj", 62.351), - (1, 2, 4096, 4096): ("shared_gate_up_proj", 109.998), - (1, 2, 4096, 6144): ("shared_gate_up_proj", 151.045), - (1, 2, 4096, 8192): ("wo_b", 194.748), - (1, 2, 6144, 2048): ("shared_down_proj", 66.098), - (1, 2, 6144, 3072): ("shared_down_proj", 92.174), - (1, 2, 6144, 6144): ("shared_gate_up_proj", 155.341), - (1, 2, 6144, 8192): ("o_proj", 190.044), - (1, 2, 6144, 16384): ("o_proj", 371.539), - (1, 2, 8192, 1024): ("wq_b", 55.599), - (1, 2, 9216, 6144): ("qkv_proj", 232.987), - (1, 2, 9856, 6144): ("qkv_proj_and_indexer_qk", 216.684), - (1, 2, 10240, 4096): ("qkv_proj", 175.532), - (1, 2, 16384, 2048): ("q_b_proj", 145.087), - (1, 2, 28672, 512): ("kv_b_proj", 82.624), - (1, 2, 32768, 1024): ("wq_b", 155.197), - (1, 16, 1536, 4096): ("wqkv_a", 80.574), - (1, 16, 2624, 6144): ("fused_qkv_a_proj", 98.463), - (1, 16, 3072, 4096): ("shared_gate_up_proj", 82.638), - (1, 16, 4096, 1536): ("shared_down_proj", 49.439), - (1, 16, 4096, 2048): ("shared_down_proj", 65.567), - (1, 16, 4096, 4096): ("shared_gate_up_proj", 110.606), - (1, 16, 4096, 6144): ("shared_gate_up_proj", 151.685), - (1, 16, 4096, 8192): ("wo_b", 205.644), - (1, 16, 6144, 2048): ("shared_down_proj", 67.346), - (1, 16, 6144, 3072): ("shared_down_proj", 91.838), - (1, 16, 6144, 6144): ("shared_gate_up_proj", 156.653), - (1, 16, 6144, 8192): ("o_proj", 192.412), - (1, 16, 6144, 16384): ("o_proj", 358.435), - (1, 16, 8192, 1024): ("wq_b", 56.847), - (1, 16, 9216, 6144): ("qkv_proj", 235.083), - (1, 16, 9856, 6144): ("qkv_proj_and_indexer_qk", 219.564), - (1, 16, 10240, 4096): ("qkv_proj", 171.069), - (1, 16, 16384, 2048): ("q_b_proj", 146.719), - (1, 16, 28672, 512): ("kv_b_proj", 83.504), - (1, 16, 32768, 1024): ("wq_b", 158.989), - (1, 3072, 1536, 4096): ("wqkv_a", 9795.766), - (1, 3072, 2624, 6144): ("fused_qkv_a_proj", 28944.543), - (1, 3072, 3072, 4096): ("shared_gate_up_proj", 21047.242), - (1, 3072, 4096, 1536): ("shared_down_proj", 11363.852), - (1, 3072, 4096, 2048): ("shared_down_proj", 14921.346), - (1, 3072, 4096, 4096): ("shared_gate_up_proj", 29325.629), - (1, 3072, 4096, 6144): ("shared_gate_up_proj", 43830.049), - (1, 3072, 4096, 8192): ("wo_b", 58484.25), - (1, 3072, 6144, 2048): ("shared_down_proj", 22320.748), - (1, 3072, 6144, 3072): ("shared_down_proj", 33673.799), - (1, 3072, 6144, 6144): ("shared_gate_up_proj", 67255.209), - (1, 3072, 6144, 8192): ("o_proj", 90545.355), - (1, 3072, 6144, 16384): ("o_proj", 182662.598), - (1, 3072, 8192, 1024): ("wq_b", 15305.526), - (1, 3072, 9216, 6144): ("qkv_proj", 103111.029), - (1, 3072, 9856, 6144): ("qkv_proj_and_indexer_qk", 110813.373), - (1, 3072, 10240, 4096): ("qkv_proj", 74301.431), - (1, 3072, 16384, 2048): ("q_b_proj", 59796.738), - (1, 3072, 28672, 512): ("kv_b_proj", 29183.929), - (1, 3072, 32768, 1024): ("wq_b", 62264.438), - (4, 2, 768, 4096): ("shared_gate_up_proj", 71.711), - (4, 2, 1024, 6144): ("shared_gate_up_proj", 108.278), - (4, 2, 1536, 6144): ("shared_gate_up_proj", 109.822), - (4, 2, 2304, 6144): ("qkv_proj", 113.134), - (4, 2, 2560, 4096): ("qkv_proj", 77.854), - (4, 2, 2560, 6144): ("qkv_proj_and_indexer_qk", 111.79), - (4, 2, 2624, 6144): ("fused_qkv_a_proj", 98.756), - (4, 2, 4096, 384): ("shared_down_proj", 21.664), - (4, 2, 6144, 512): ("shared_down_proj", 23.553), - (4, 2, 6144, 768): ("shared_down_proj", 33.407), - (4, 2, 6144, 2048): ("o_proj", 65.135), - (4, 2, 6144, 4096): ("o_proj", 110.501), - (4, 2, 7168, 512): ("kv_b_proj", 25.009), - (4, 16, 768, 4096): ("shared_gate_up_proj", 73.262), - (4, 16, 1024, 6144): ("shared_gate_up_proj", 108.422), - (4, 16, 1536, 6144): ("shared_gate_up_proj", 113.006), - (4, 16, 2304, 6144): ("qkv_proj", 113.422), - (4, 16, 2560, 4096): ("qkv_proj", 80.19), - (4, 16, 2560, 6144): ("qkv_proj_and_indexer_qk", 112.27), - (4, 16, 2624, 6144): ("fused_qkv_a_proj", 98.708), - (4, 16, 4096, 384): ("shared_down_proj", 23.216), - (4, 16, 6144, 512): ("shared_down_proj", 24.977), - (4, 16, 6144, 768): ("shared_down_proj", 34.063), - (4, 16, 6144, 2048): ("o_proj", 67.535), - (4, 16, 6144, 4096): ("o_proj", 112.198), - (4, 16, 7168, 512): ("kv_b_proj", 27.793), - (4, 3072, 768, 4096): ("shared_gate_up_proj", 4988.476), - (4, 3072, 1024, 6144): ("shared_gate_up_proj", 9708.601), - (4, 3072, 1536, 6144): ("shared_gate_up_proj", 14650.233), - (4, 3072, 2304, 6144): ("qkv_proj", 22634.012), - (4, 3072, 2560, 4096): ("qkv_proj", 16984.091), - (4, 3072, 2560, 6144): ("qkv_proj_and_indexer_qk", 25396.649), - (4, 3072, 2624, 6144): ("fused_qkv_a_proj", 28947.94), - (4, 3072, 4096, 384): ("shared_down_proj", 3366.413), - (4, 3072, 6144, 512): ("shared_down_proj", 6270.013), - (4, 3072, 6144, 768): ("shared_down_proj", 8900.2), - (4, 3072, 6144, 2048): ("o_proj", 22307.655), - (4, 3072, 6144, 4096): ("o_proj", 44517.157), - (4, 3072, 7168, 512): ("kv_b_proj", 7308.89), - (4, 4096, 768, 4096): ("shared_gate_up_proj", 6678.282), - (4, 4096, 1024, 6144): ("shared_gate_up_proj", 13036.336), - (4, 4096, 1536, 6144): ("shared_gate_up_proj", 19776.488), - (4, 4096, 2304, 6144): ("qkv_proj", 31556.192), - (4, 4096, 2560, 4096): ("qkv_proj", 24045.12), - (4, 4096, 2560, 6144): ("qkv_proj_and_indexer_qk", 35918.768), - (4, 4096, 2624, 6144): ("fused_qkv_a_proj", 40458.051), - (4, 4096, 4096, 384): ("shared_down_proj", 4381.289), - (4, 4096, 6144, 512): ("shared_down_proj", 8243.047), - (4, 4096, 6144, 768): ("shared_down_proj", 11759.473), - (4, 4096, 6144, 2048): ("o_proj", 29702.473), - (4, 4096, 6144, 4096): ("o_proj", 58712.341), - (4, 4096, 7168, 512): ("kv_b_proj", 9702.386), - (8, 2, 384, 4096): ("shared_gate_up_proj", 64.511), - (8, 2, 512, 4096): ("shared_gate_up_proj", 72.687), - (8, 2, 512, 6144): ("shared_gate_up_proj", 97.606), - (8, 2, 768, 6144): ("shared_gate_up_proj", 104.351), - (8, 2, 1280, 4096): ("qkv_proj", 73.535), - (8, 2, 1280, 6144): ("qkv_proj", 110.638), - (8, 2, 1536, 4096): ("wqkv_a", 74.767), - (8, 2, 1536, 6144): ("qkv_proj_and_indexer_qk", 110.19), - (8, 2, 2048, 2048): ("q_b_proj", 40.371), - (8, 2, 2624, 6144): ("fused_qkv_a_proj", 95.43), - (8, 2, 3584, 512): ("kv_b_proj", 17.873), - (8, 2, 4096, 192): ("shared_down_proj", 16.144), - (8, 2, 4096, 256): ("shared_down_proj", 19.344), - (8, 2, 4096, 1024): ("wq_b_or_wo_b", 36.271), - (8, 2, 6144, 256): ("shared_down_proj", 16.881), - (8, 2, 6144, 384): ("shared_down_proj", 21.968), - (8, 2, 6144, 1024): ("o_proj", 37.407), - (8, 2, 6144, 2048): ("o_proj", 64.948), - (8, 2, 8192, 1024): ("wq_b", 56.239), - (8, 16, 384, 4096): ("shared_gate_up_proj", 66.527), - (8, 16, 512, 6144): ("shared_gate_up_proj", 99.318), - (8, 16, 768, 6144): ("shared_gate_up_proj", 104.527), - (8, 16, 1280, 4096): ("qkv_proj", 74.782), - (8, 16, 1280, 6144): ("qkv_proj", 110.478), - (8, 16, 1536, 6144): ("qkv_proj_and_indexer_qk", 111.486), - (8, 16, 2048, 2048): ("q_b_proj", 42.035), - (8, 16, 2624, 6144): ("fused_qkv_a_proj", 97.99), - (8, 16, 3584, 512): ("kv_b_proj", 18.833), - (8, 16, 4096, 192): ("shared_down_proj", 17.312), - (8, 16, 6144, 256): ("shared_down_proj", 18.177), - (8, 16, 6144, 384): ("shared_down_proj", 23.152), - (8, 16, 6144, 1024): ("o_proj", 38.655), - (8, 16, 6144, 2048): ("o_proj", 66.98), - (8, 3072, 384, 4096): ("shared_gate_up_proj", 3009.348), - (8, 3072, 512, 6144): ("shared_gate_up_proj", 4355.572), - (8, 3072, 768, 6144): ("shared_gate_up_proj", 7087.451), - (8, 3072, 1280, 4096): ("qkv_proj", 8489.847), - (8, 3072, 1280, 6144): ("qkv_proj", 12553.842), - (8, 3072, 1536, 6144): ("qkv_proj_and_indexer_qk", 14614.569), - (8, 3072, 2048, 2048): ("q_b_proj", 6821.235), - (8, 3072, 2624, 6144): ("fused_qkv_a_proj", 29185.017), - (8, 3072, 3584, 512): ("kv_b_proj", 3685.827), - (8, 3072, 4096, 192): ("shared_down_proj", 2043.847), - (8, 3072, 6144, 256): ("shared_down_proj", 3593.156), - (8, 3072, 6144, 384): ("shared_down_proj", 4925.667), - (8, 3072, 6144, 1024): ("o_proj", 11664.713), - (8, 3072, 6144, 2048): ("o_proj", 22305.402), - # Model-catalog TP8 M=4096 large-prefill baselines measured on 2026-08-27 - # (worker29, gfx928) with the same protocol as the 2026-08-12 catalog: - # lmslim int8_utils.matmul_kernel (the W8A8 Triton baseline), GPU events, - # CUDA Graph replay, hot cache, warmups=10, samples=20, launches/sample=5, - # BM256/BN256/BK64 (matmul_int8 default for M>1024). Covers the Hy3 / - # MiniMax M3 / GLM5.2 TP8 operators at M=4096 (see - # MODEL_TP8_EXTRA_OPTIMIZATION_M_VALUES in the operator API contract). - (8, 4096, 1280, 4096): ("qkv_proj", 19381.383), - (8, 4096, 4096, 1024): ("o_proj", 14151.792), - (8, 4096, 384, 4096): ("shared_gate_up_proj", 16712.231), - (8, 4096, 4096, 192): ("shared_down_proj", 10762.616), - (8, 4096, 1280, 6144): ("qkv_proj", 41000.618), - (8, 4096, 1536, 6144): ("qkv_proj_and_indexer_qk", 35604.698), - (8, 4096, 6144, 1024): ("o_proj", 19897.517), - (8, 4096, 768, 6144): ("shared_gate_up_proj", 31882.257), - (8, 4096, 6144, 384): ("shared_down_proj", 22389.162), - (8, 4096, 2624, 6144): ("fused_qkv_a_proj", 68661.417), - (8, 4096, 2048, 2048): ("q_b_proj", 14385.571), - (8, 4096, 3584, 512): ("kv_b_proj", 6939.939), - (8, 4096, 6144, 2048): ("o_proj", 54775.797), - (8, 4096, 512, 6144): ("shared_gate_up_proj", 21254.589), - (8, 4096, 6144, 256): ("shared_down_proj", 17700.262), - # Model-catalog TP8 M=8 short-decode baselines measured on 2026-09-20 - # (worker29, gfx928) with the same protocol as the 2026-08-12 catalog: - # lmslim int8_utils.matmul_kernel (the W8A8 Triton baseline), GPU events, - # CUDA Graph replay, hot cache, warmups=10, samples=20, launches/sample=5, - # BM16/BN32/BK256 (matmul_int8 default for M<=32). Covers the Hy3 / - # MiniMax M3 / GLM5.2 TP8 operators at M=8 (see - # MODEL_TP8_EXTRA_OPTIMIZATION_M_VALUES in the operator API contract). - # Each value is the per-shape median of three independent passes - # (tools/baseline/bench_triton_tp8_m8.py -> tp8_m8_graph_run{1,2,3}.json, - # merged view in tp8_m8_graph_merged.json). The median keeps the estimate - # robust where a single pass was disturbed by a co-tenant on the shared - # card (e.g. minimax shared_down_proj pass 2: 33.760 vs 19.968/19.984). - # Cross-pass spread is <= 1.6% for 12 of 15 shapes, 5.8%/6.8% for - # glm52 q_b_proj / shared_down_proj. - (8, 8, 1280, 4096): ("qkv_proj", 66.432), - (8, 8, 4096, 1024): ("o_proj", 31.472), - (8, 8, 384, 4096): ("shared_gate_up_proj", 57.008), - (8, 8, 4096, 192): ("shared_down_proj", 14.592), - (8, 8, 1280, 6144): ("qkv_proj", 98.865), - (8, 8, 1536, 6144): ("qkv_proj_and_indexer_qk", 99.473), - (8, 8, 6144, 1024): ("o_proj", 32.960), - (8, 8, 768, 6144): ("shared_gate_up_proj", 92.928), - (8, 8, 6144, 384): ("shared_down_proj", 19.984), - (8, 8, 2624, 6144): ("fused_qkv_a_proj", 87.200), - (8, 8, 2048, 2048): ("q_b_proj", 36.704), - (8, 8, 3584, 512): ("kv_b_proj", 15.808), - (8, 8, 6144, 2048): ("o_proj", 58.864), - (8, 8, 512, 6144): ("shared_gate_up_proj", 88.448), - (8, 8, 6144, 256): ("shared_down_proj", 15.136), -} - - -# TP8 hot-cache measurements supplied in milliseconds and normalized here to -# microseconds. Graph median is the optimization comparison target; eager and -# both P90 values are retained as measurement evidence. -_TRITON_TP8_BASELINES_US = { - (8, 16, 1536, 4096): ("wqkv_a", 97.660, 98.404, 66.591, 67.347), - (8, 16, 4096, 1024): ("wq_b_or_wo_b", 95.284, 96.092, 32.137, 35.969), - (8, 16, 8192, 1024): ("indexer.wq_b", 96.204, 96.796, 48.274, 50.794), - (8, 16, 512, 4096): ("shared_gate_up", 96.056, 97.788, 59.210, 59.962), - (8, 16, 4096, 256): ("shared_down", 94.540, 95.740, 14.849, 18.001), - (8, 3072, 1536, 4096): ("wqkv_a", 9423.243, 9446.877, 9422.539, 9486.942), - (8, 3072, 4096, 1024): ("wq_b_or_wo_b", 7637.209, 8033.128, 7617.880, 7686.395), - (8, 3072, 8192, 1024): ("indexer.wq_b", 15461.000, 15528.571, 15466.648, 15958.569), - (8, 3072, 512, 4096): ("shared_gate_up", 2775.526, 2780.134, 2740.757, 2751.685), - (8, 3072, 4096, 256): ("shared_down", 2414.377, 2424.569, 2370.583, 2376.343), -} +import json +from functools import lru_cache +from pathlib import Path +from typing import Any, Dict, Mapping, Optional, Tuple + +from metainfer.orchestrator import hw_profile as _hw_profile + +BASELINES_DIR = Path(__file__).resolve().parents[1] / "baselines" + +#: Every measurement made before 2026-09-25 came from this board. It is the +#: fallback *reference* (never silently presented as a local measurement). +REFERENCE_PROFILE = "liquid-k500sm" + + +class BaselineUnavailable(ValueError): + """No baseline exists for this shape in any profile. + + Subclasses ``ValueError`` on purpose: existing call sites already guard the + baseline lookup with ``except ValueError`` and degrade to "no baseline". + """ + + +@lru_cache(maxsize=8) +def _load_profile(profile: str) -> Dict[str, Any]: + """Read one profile's table file (missing file = empty table, not an error).""" + path = BASELINES_DIR / f"{profile}.json" + if not path.is_file(): + return {"graph": {}, "tp8": {}, "meta": {"hw_profile": profile}} + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + payload = None + if not isinstance(payload, dict): + return {"graph": {}, "tp8": {}, "meta": {"hw_profile": profile}} + return { + "graph": payload.get("graph") or {}, + "tp8": payload.get("tp8") or {}, + "meta": payload, + } + + +def reload_tables() -> None: + """Drop the per-process table cache (after adding measurements).""" + _load_profile.cache_clear() + + +def profiles_with_baselines() -> Dict[str, int]: + """How many shapes each profile has measured (for the UI / diagnostics).""" + counts: Dict[str, int] = {} + if not BASELINES_DIR.is_dir(): + return counts + for path in sorted(BASELINES_DIR.glob("*.json")): + table = _load_profile(path.stem) + counts[path.stem] = len(table["graph"]) + len(table["tp8"]) + return counts + + +def _key(shape: Mapping[str, Any]) -> Tuple[int, int, int, int]: + try: + return ( + int(shape.get("tp_size", 4)), + int(shape["M"]), + int(shape["N"]), + int(shape["K"]), + ) + except (KeyError, TypeError, ValueError) as exc: + raise ValueError(f"invalid W8A8 shape: {shape}") from exc + + +def _key_str(key: Tuple[int, int, int, int]) -> str: + return "%d,%d,%d,%d" % key + + +def current_profile() -> Optional[str]: + """The detected profile of this machine, or None when it is unknown.""" + try: + return _hw_profile.profile_id() + except _hw_profile.HardwareProfileError: + return None + + +def _measurement(table: Dict[str, Any], key: str) -> Tuple[Optional[tuple], str]: + """(measurement, kind) from one profile table, preferring the TP8 rows.""" + found = table["tp8"].get(key) + if found is not None: + return found, "tp8" + return table["graph"].get(key), "graph" + + +def baseline_origin(record: Any) -> Tuple[str, bool]: + """``(profile, mismatch)`` of a baseline *number*, whatever produced it. + + Two shapes of record exist in this workspace: + + 1. records from :func:`fixed_triton_graph_baseline`, which carry + ``baseline_profile`` / ``profile_mismatch`` (i.e. proven provenance); + 2. records written before the profile split, e.g. a task report whose + ``initial_metrics`` came from the hardcoded table + (``source: user_supplied_fixed_table``) -- a bare number or a dict with no + provenance field at all. + + Shape 2 is **not** neutral: the hardcoded table *is* the reference board's + data, so such a number is attributed to :data:`REFERENCE_PROFILE` rather + than being trusted as "this machine's". That attribution is what stops a + pre-split task on the air board from recording a cross-profile speedup + (measured case: an air variant carried ``baseline_us=19.98 speedup=1.967`` + from the liquid table while this board's own value for the same shape is + 25.936 us). + """ + if isinstance(record, Mapping): + profile = record.get("baseline_profile") + if profile: + return str(profile), bool(record.get("profile_mismatch")) + return REFERENCE_PROFILE, False + + +def trusted_baseline_us( + record: Any, + *, + profile: Optional[str] = None, + allow_reference_profile: bool = False, +) -> Optional[float]: + """``median_us`` of ``record``, but only when it provably came from ``profile``. + + Returns ``None`` when the number belongs to another board (or to an + unknown one) -- callers that *record knowledge* (variant promotion, header + backfill, reports) must then omit ``baseline_us``/``speedup`` instead of + filing a ratio that cannot be reproduced here. A machine whose profile + cannot be resolved has nothing provably local, so it gets ``None`` too. + + ``allow_reference_profile=True`` is the explicit opt-in used when the + reference number is only displayed, never recorded as a same-machine + comparison. + """ + if isinstance(record, Mapping): + median = record.get("median_us") + if median is None: + median = record.get("baseline_us") + else: + median = record + if median is None: + return None + used_profile, mismatch = baseline_origin(record) + current = profile if profile is not None else current_profile() + if current is None: + return None + if used_profile == current and not mismatch: + return float(median) + if allow_reference_profile and used_profile == REFERENCE_PROFILE: + return float(median) + return None def fixed_triton_graph_baseline( shape_id: str, shape: Mapping[str, Any], *, - bootstrap_metrics: Mapping[str, Any] | None = None, + bootstrap_metrics: Optional[Mapping[str, Any]] = None, + hw_profile: Optional[str] = None, + allow_reference_profile: bool = True, ) -> Dict[str, Any]: """Return the immutable Graph baseline for one exact M/N/K shape. + Resolution is per hardware profile; when this machine's profile has not + measured the shape, the reference table is used only if + ``allow_reference_profile`` is set, and the returned record is flagged with + ``profile_mismatch=True`` so no caller can mistake it for a same-machine + comparison. + ``bootstrap_metrics`` remains separate because it describes the current child HIP source used for PMC, rollback and iterative improvement. """ - try: - # Legacy callers predate TP-aware shape metadata and are TP4-only. - tp_size = int(shape.get("tp_size", 4)) - key = ( - tp_size, - int(shape["M"]), - int(shape["N"]), - int(shape["K"]), - ) - except (KeyError, TypeError, ValueError) as exc: - raise ValueError( - f"invalid W8A8 shape for fixed baseline: {shape_id}: {shape}" - ) from exc - tp8_measurement = _TRITON_TP8_BASELINES_US.get(key) - fixed_measurement = _TRITON_GRAPH_BASELINES_US.get(key) - if tp8_measurement is None and fixed_measurement is None: - raise ValueError( + profile = hw_profile if hw_profile is not None else current_profile() + key = _key(shape) + key_str = _key_str(key) + lookup_profile = profile or REFERENCE_PROFILE + + table = _load_profile(lookup_profile) + measurement, kind = _measurement(table, key_str) + used_profile = lookup_profile + mismatch = False + used_meta = table["meta"] + + if measurement is None and lookup_profile != REFERENCE_PROFILE: + reference = _load_profile(REFERENCE_PROFILE) + if allow_reference_profile: + measurement, kind = _measurement(reference, key_str) + if measurement is not None: + mismatch = True + used_profile = REFERENCE_PROFILE + used_meta = reference["meta"] + + if measurement is None: + raise BaselineUnavailable( "no fixed Triton Graph baseline for " - f"{shape_id} with TP={key[0]}, M={key[1]}, " - f"N={key[2]}, K={key[3]}" + f"{shape_id} with TP={key[0]}, M={key[1]}, N={key[2]}, K={key[3]} " + f"(profile {lookup_profile!r} has no entry, and the " + f"{REFERENCE_PROFILE!r} reference has none either)" ) - if tp8_measurement is not None: - case, eager_median_us, eager_p90_us, latency_us, graph_p90_us = ( - tp8_measurement - ) + if kind == "tp8": + case, eager_median_us, eager_p90_us, latency_us, graph_p90_us = measurement + distribution = True else: - assert fixed_measurement is not None - case, latency_us = fixed_measurement + case, latency_us = measurement + distribution = False + eager_median_us = eager_p90_us = graph_p90_us = None record: Dict[str, Any] = { "median_us": latency_us, "baseline_us": latency_us, "baseline_kind": "triton_graph", "case": case, - "tp_size": tp_size, + "tp_size": key[0], "shape": {"M": key[1], "N": key[2], "K": key[3]}, - "source": "user_supplied_fixed_table", + "source": f"profile_table:{used_profile}", "timing_scope": ( "prefill_graph_replay" if key[1] > 16 else "decode_graph_replay" ), - "distribution_stats_available": tp8_measurement is not None, + "distribution_stats_available": distribution, + # Provenance: which machine produced this number. + "baseline_profile": used_profile, + "baseline_machine": used_meta.get("machine"), + "baseline_measured_on": used_meta.get("measured_on"), + "profile_mismatch": mismatch, } - if tp8_measurement is not None: + if mismatch: + record["profile_note"] = ( + f"no {lookup_profile!r} baseline for this shape; using the " + f"{REFERENCE_PROFILE!r} reference. speedup against it is NOT a " + "same-machine comparison (air boards have no HBM address hashing, " + "so the reference is 1.06x-1.40x faster depending on the shape)." + ) + if distribution: record.update( { "p90_us": graph_p90_us, diff --git a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/w8a8_pipeline.py b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/w8a8_pipeline.py index ac9068a0..16c28ef5 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/w8a8_pipeline.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/orchestrator/w8a8_pipeline.py @@ -6,6 +6,7 @@ import hashlib import os import shutil +import subprocess import threading import time from concurrent.futures import ThreadPoolExecutor, as_completed @@ -357,6 +358,7 @@ def _round_strategy_text( parse_memory_traffic_csv, parse_pmc_csv, ) +from .gpu_probe import run_gpu_probe from .real_pipeline import _last_json, _run, _safe, _status from .contention import discard_contaminated_round from .result_store import SCHEMA_VERSION, append_jsonl, write_json @@ -1083,24 +1085,43 @@ def _check_required_files(repo: Path) -> bool: class W8A8Runner: - def __init__(self, worker_root: Path, gpu: int) -> None: - self.worker_root = worker_root + def __init__(self, worker_root: Path, gpu: int, *, + allowed: Optional[Sequence[int]] = None, + source: Optional[Path] = None) -> None: + # Absolute from here on: the harness is launched as a subprocess with a + # ``cwd`` of its own source dir, so a relative harness path would be + # resolved *against that dir* and looked up twice. (df6fb501 recovery: + # ``can't open file '.../final/source/nodes/worker29/.../w8a8_bench.py'``.) + self.worker_root = Path(worker_root).resolve() self.gpu = int(gpu) - self.source = worker_root / "source" + # ``source`` overrides the tree under test; ``worker_root`` still owns the + # caches (references, torch extensions, logs). Layout batching renders + # each batch into its own tree (``final/layout_batches/batchN``), and + # benchmarking a batch's shapes against *another* batch's tree validates + # the wrong extension: in glm-tp8-m8-16-4096-2663a78f the M=4096 shape of + # a later batch fell through batch 0's M-chain to the m16 kernel -- correct + # but 2721 us, against the 1191 us its own kernel measures. + self.source = ( + Path(source).resolve() if source is not None + else self.worker_root / "source" + ) staged_harness = self.source / "w8a8_bench.py" self.harness = ( - staged_harness if staged_harness.is_file() else HARNESS + staged_harness if staged_harness.is_file() else Path(HARNESS) ) + self.harness = Path(self.harness).resolve() self.env = dict(os.environ) - bind_worker_gpu(self.env, gpu) + # The task's own device set, when the caller knows it: a card outside it + # is a bug here, not something to run on somebody else's device. + bind_worker_gpu(self.env, gpu, allowed=allowed) self.env.update({ "MAX_JOBS": "2", "PYTHONDONTWRITEBYTECODE": "1", "PYTORCH_ROCM_ARCH": "gfx928", - "TORCH_EXTENSIONS_DIR": str(worker_root / "cache" / "torch"), - "TRITON_CACHE_DIR": str(worker_root / "cache" / "triton"), - "XDG_CACHE_HOME": str(worker_root / "cache" / "xdg"), - "TMPDIR": str(worker_root / "cache" / "tmp"), + "TORCH_EXTENSIONS_DIR": str(self.worker_root / "cache" / "torch"), + "TRITON_CACHE_DIR": str(self.worker_root / "cache" / "triton"), + "XDG_CACHE_HOME": str(self.worker_root / "cache" / "xdg"), + "TMPDIR": str(self.worker_root / "cache" / "tmp"), }) for key in ( "TORCH_EXTENSIONS_DIR", "TRITON_CACHE_DIR", "XDG_CACHE_HOME", @@ -1185,10 +1206,14 @@ def _prepare_compile_cache(self) -> Dict[str, Any]: return metadata def probe(self) -> Dict[str, Any]: - result = _run([ - "python3", str(self.harness), "--source", str(self.source), - "--m", "1", "--n", "1", "--k", "1", "--probe", - ], cwd=self.source, env=self.env, timeout=60) + # Same transient-stack protection as the Generate preflight: a worker's + # probe is the first thing that touches the device, and the container's + # GPU stack can be briefly unusable. + result = run_gpu_probe( + harness=self.harness, source=self.source, env=self.env, + runner=_run, timeout=60, + state_dir=getattr(self, "state_dir", None), + ) return _last_json(result.stdout) def benchmark( @@ -1230,14 +1255,45 @@ def benchmark( command.append("--skip-correctness") build = self._prepare_compile_cache() self._ensure_gate("benchmark") - result = _run( - command, cwd=self.source, env=self.env, - timeout=_BENCHMARK_TIMEOUT_S, + metrics = _last_json( + self._run_harness_logged(command, timeout=_BENCHMARK_TIMEOUT_S) ) - metrics = _last_json(result.stdout) metrics["compile_cache_key"] = build["build_key"] return metrics + def _run_harness_logged(self, command: list[str], *, timeout: int) -> str: + """Run the benchmark with its output on disk instead of a pipe. + + ``subprocess.run(capture_output=True)`` reads the pipe until EOF, and a + *grandchild* that inherited it keeps EOF away for as long as it lives: + the timeout then applies to the direct child only and the caller blocks + indefinitely (measured: a merge-and-validate sat 14 minutes past its last + timeline row, with the recovery process sleeping and no benchmark + process of its own). Files cannot deadlock, and the harness log is worth + keeping next to the run anyway. + """ + logs = self.worker_root / "logs" + logs.mkdir(parents=True, exist_ok=True) + log_path = logs / "bench_harness.log" + with log_path.open("w", encoding="utf-8") as sink: + completed = subprocess.run( + command, + cwd=self.source, + env=self.env, + stdout=sink, + stderr=subprocess.STDOUT, + text=True, + timeout=timeout, + check=False, + ) + output = log_path.read_text(encoding="utf-8", errors="replace") + if completed.returncode != 0: + raise RuntimeError( + f"{' '.join(command)} failed ({completed.returncode}): " + f"{output[-2000:]}" + ) + return output + def _reference_cache_path(self, m: int, n: int, k: int) -> Path: """Path of the exact int64 reference cache for one (M, N, K).""" return ( @@ -1390,7 +1446,7 @@ def _phase(self, phase: str, **payload: Any) -> None: def run(self, *, dry_run: bool = False) -> Dict[str, Any]: task_id = str(self.req.get("task_id", "task")) - self.store.init_or_resume(task_id) + self.store.init_or_resume(task_id, "dcu-kernel-auto-opt") self.store.update_run( current_iteration=0, finished=False, @@ -1613,7 +1669,10 @@ def run_one( assignment: WorkerAssignment, ) -> Dict[str, Dict[str, Any]]: root = self.workspace_dir / "workers" / assignment.worker_id - runner = W8A8Runner(root, assignment.gpu) + runner = W8A8Runner( + root, assignment.gpu, + allowed=[item.gpu for item in config.assignments], + ) _status( root, assignment, state="building", iteration=0, shape_id=None, @@ -1988,7 +2047,10 @@ def _run_worker( ) -> Dict[str, Any]: root = self.workspace_dir / "workers" / assignment.worker_id source = root / "source" - runner = W8A8Runner(root, assignment.gpu) + runner = W8A8Runner( + root, assignment.gpu, + allowed=[item.gpu for item in config.assignments], + ) probe = runner.probe() result: Dict[str, Any] = { "worker_id": assignment.worker_id, diff --git a/metainfer/tasks/dcu_kernel_auto_opt/reports/_build_concise.py b/metainfer/tasks/dcu_kernel_auto_opt/reports/_build_concise.py new file mode 100644 index 00000000..8b4004c9 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/reports/_build_concise.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +"""Concise operator-performance table extracted from the DKAO run records. + +No benchmarking: everything is read from the task records +(``final_report.json`` for finally-validated runs, ``plan.json`` + +``shared_baseline/results.json`` + ``workers/*/result.json`` for runs that only +finished the parallel-exploration phase). + +Columns: 模型 / TP / 算子 / M / N / K / 基线µs / 性能µs / 加速比 / 提升% / 带宽GB/s / TOPS / 验收状态 +""" +from __future__ import annotations + +import csv +import glob +import json +import os + +WS_ROOT = "/root/zth_agent/MetaInfer/nodes/worker29/workspaces" +OUT_DIR = os.path.dirname(os.path.abspath(__file__)) + +STATE_OK = "已最终验收" +STATE_WORKER = "仅worker验收" + +# (task_id, model label, TP) ; one canonical run per model×TP×M group +RUNS = [ + ("hy3-dsh-tp4-m16-2-368c654c", "Hy3", 4), + ("hy3-tp4-m4096-dsh-test1-bfdf3002", "Hy3", 4), + ("hy3-dsh-tp8-m4096-1-0ebb994d", "Hy3", 8), + ("hy3-dsh-tp8-m16-9-8-0161e718", "Hy3", 8), + ("minimax-dsh-tp8-m16-1-78402260", "MiniMax-M3", 8), + ("minimaxm3-dsh-tp4-m4096-1-0c2f84a9", "MiniMax-M3", 4), + ("minimaxm3-dsh-tp8-m4096-1-b0482833", "MiniMax-M3", 8), + ("glm5-2-dsh-tp8-m4096-1-e6a280a2", "GLM5.2", 8), + ("glm5-2-dsh-tp4-m4096-1-85cb689c", "GLM5.2", 4), + ("glm5-2-dsh-tp8-m16-test1-cb4d262c", "GLM5.2", 8), +] + + +def algorithmic_bytes(m: int, n: int, k: int) -> int: + """Same formula as the trusted harness: int8 A + int8 B + fp32 scales + bf16 out.""" + return m * k + k * n + 4 * (m + n) + 2 * m * n + + +def load_json(path, default=None): + try: + with open(path, encoding="utf-8") as handle: + return json.load(handle) + except (OSError, ValueError): + return default + + +def row_from_validated(task_id, model, tp): + rep = load_json(os.path.join(WS_ROOT, task_id, "final_report.json")) or {} + if rep.get("status") != "success": + return [] + cfg = rep.get("config") or {} + target = (rep.get("final_target") or {}).get("shapes") or {} + validation = rep.get("final_validation") or {} + rows = [] + for shape in cfg.get("shapes") or []: + sid = shape.get("id") + info = target.get(sid) + if not info: + continue + record = validation.get(sid) or {} + dims = record.get("shape") or {} + m = dims.get("M", shape.get("M")) + n = dims.get("N", shape.get("N")) + k = dims.get("K", shape.get("K")) + base = info.get("baseline_us") + perf = info.get("final_us") + if not (base and perf): + continue + rows.append(_row(model, tp, shape.get("operator"), m, n, k, base, perf, + record.get("p90_us"), STATE_OK)) + return rows + + +def row_from_worker(task_id, model, tp): + ws = os.path.join(WS_ROOT, task_id) + plan = load_json(os.path.join(ws, "plan.json")) + if not plan: + return [] + baseline = (load_json(os.path.join(ws, "shared_baseline", "results.json")) or {}).get("shapes") or {} + best = {} + for wid in sorted(glob.glob(os.path.join(ws, "workers", "worker_*"))): + result = load_json(os.path.join(wid, "result.json")) or {} + for sid, rec in (result.get("shapes") or {}).items(): + metrics = rec.get("metrics") or {} + median = metrics.get("median_us") + if median is None: + continue + if sid not in best or median < best[sid]["perf"]: + best[sid] = {"perf": median, "p90": metrics.get("p90_us"), + "bytes": metrics.get("algorithmic_bytes")} + rows = [] + for shape in plan.get("shapes") or []: + sid = shape.get("id") + rec = best.get(sid) + if not rec: + continue + entry = baseline.get(sid) or {} + base = entry.get("baseline_us") or entry.get("median_us") + if not base: + continue + rows.append(_row(model, shape.get("tp_size", tp), shape.get("operator"), + shape.get("M"), shape.get("N"), shape.get("K"), + base, rec["perf"], rec.get("p90"), STATE_WORKER)) + return rows + + +def _row(model, tp, operator, m, n, k, base, perf, p90, state): + total_bytes = algorithmic_bytes(int(m), int(n), int(k)) + return { + "模型": model, + "TP": int(tp), + "算子": operator, + "M": int(m), "N": int(n), "K": int(k), + "基线µs": round(float(base), 3), + "性能µs": round(float(perf), 3), + "加速比": round(float(base) / float(perf), 3), + "提升%": round((float(base) / float(perf) - 1.0) * 100.0, 1), + "带宽GB/s": round(total_bytes / (float(perf) * 1e-6) / 1e9, 1), + "TOPS": round(2.0 * int(m) * int(n) * int(k) / (float(perf) * 1e-6) / 1e12, 2), + "P90µs": round(float(p90), 3) if p90 else "", + "验收状态": state, + } + + +def main(): + rows = [] + for task_id, model, tp in RUNS: + validated = row_from_validated(task_id, model, tp) + if validated: + rows.extend(validated) + print(f"{task_id:42s} {model:11s} TP{tp} validated rows={len(validated)}") + continue + worker = row_from_worker(task_id, model, tp) + rows.extend(worker) + print(f"{task_id:42s} {model:11s} TP{tp} worker-only rows={len(worker)}") + + rows.sort(key=lambda r: (r["模型"], r["TP"], r["M"], r["算子"])) + print(f"\nTOTAL rows = {len(rows)} " + f"({sum(1 for r in rows if r['验收状态'] == STATE_OK)} validated, " + f"{sum(1 for r in rows if r['验收状态'] == STATE_WORKER)} worker-only)") + + path = os.path.join(OUT_DIR, "dkao_operators_concise.csv") + with open(path, "w", newline="", encoding="utf-8-sig") as handle: + writer = csv.DictWriter(handle, fieldnames=list(rows[0].keys())) + writer.writeheader() + writer.writerows(rows) + print("wrote", path) + + print("\n%-11s %2s %-22s %6s %8s %9s %8s %8s %8s" % ( + "模型", "TP", "算子", "M", "基线µs", "性能µs", "加速比", "带宽", "TOPS")) + for r in rows: + print("%-11s %2d %-22s %6d %9.2f %9.2f %7.2fx %8.1f %8.2f" % ( + r["模型"], r["TP"], r["算子"], r["M"], r["基线µs"], r["性能µs"], + r["加速比"], r["带宽GB/s"], r["TOPS"])) + + +if __name__ == "__main__": + main() diff --git a/metainfer/tasks/dcu_kernel_auto_opt/reports/_make_concise_xlsx.py b/metainfer/tasks/dcu_kernel_auto_opt/reports/_make_concise_xlsx.py new file mode 100644 index 00000000..f73eeecb --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/reports/_make_concise_xlsx.py @@ -0,0 +1,144 @@ +#!/usr/bin/env python3 +# -*- coding: utf-8 -*- +"""Build the concise operator table workbook from dkao_operators_concise.csv.""" +from __future__ import annotations + +import csv +import os + +from openpyxl import Workbook +from openpyxl.styles import Alignment, Border, Font, PatternFill, Side +from openpyxl.utils import get_column_letter + +OUT_DIR = os.path.dirname(os.path.abspath(__file__)) +CSV = os.path.join(OUT_DIR, "dkao_operators_concise.csv") + +HEAD = ["模型", "TP", "算子", "M", "N", "K", "基线µs", "性能µs", + "加速比", "提升%", "带宽GB/s", "TOPS", "验收状态"] +NUMFMT = {"基线µs": "0.00", "性能µs": "0.00", "加速比": "0.00", + "提升%": "0", "带宽GB/s": "0.0", "TOPS": "0.0"} + +thin = Side(style="thin", color="D0D0D0") +border = Border(left=thin, right=thin, top=thin, bottom=thin) +head_fill = PatternFill("solid", fgColor="1F4E78") +head_font = Font(bold=True, color="FFFFFF") +alt_fill = PatternFill("solid", fgColor="EAF1F8") +warn_fill = PatternFill("solid", fgColor="FFF2CC") + + +def read_rows(): + rows = list(csv.DictReader(open(CSV, encoding="utf-8-sig"))) + for r in rows: + for k in ("TP", "M", "N", "K"): + r[k] = int(r[k]) + for k in ("基线µs", "性能µs", "加速比", "提升%", "带宽GB/s", "TOPS"): + r[k] = float(r[k]) + return rows + + +def main(): + rows = read_rows() + wb = Workbook() + + # ---- sheet 1: concise operator table ---- + ws = wb.active + ws.title = "算子性能" + ws.append(HEAD) + for c in range(1, len(HEAD) + 1): + cell = ws.cell(row=1, column=c) + cell.fill, cell.font = head_fill, head_font + cell.alignment = Alignment(horizontal="center", vertical="center") + cell.border = border + for i, r in enumerate(rows, start=2): + for j, key in enumerate(HEAD, start=1): + cell = ws.cell(row=i, column=j, value=r[key]) + cell.border = border + if key in NUMFMT: + cell.number_format = NUMFMT[key] + cell.alignment = Alignment(horizontal="right") + elif key in ("TP", "M", "N", "K"): + cell.alignment = Alignment(horizontal="right") + else: + cell.alignment = Alignment(horizontal="left") + if r["验收状态"] != "已最终验收": + cell.fill = warn_fill + elif i % 2 == 0: + cell.fill = alt_fill + widths = {"模型": 12, "TP": 5, "算子": 24, "M": 7, "N": 7, "K": 7, + "基线µs": 11, "性能µs": 11, "加速比": 9, "提升%": 9, + "带宽GB/s": 11, "TOPS": 8, "验收状态": 13} + for j, key in enumerate(HEAD, start=1): + ws.column_dimensions[get_column_letter(j)].width = widths[key] + ws.freeze_panes = "A2" + ws.auto_filter.ref = f"A1:{get_column_letter(len(HEAD))}{ws.max_row}" + ws.row_dimensions[1].height = 22 + + # ---- sheet 2: by model x TP ---- + ws2 = wb.create_sheet("按模型×TP汇总") + head2 = ["模型", "TP", "算子数", "平均加速比", "最高加速比", "平均带宽GB/s", + "平均TOPS", "平均性能µs", "其中仅worker验收"] + ws2.append(head2) + for c in range(1, len(head2) + 1): + cell = ws2.cell(row=1, column=c) + cell.fill, cell.font = head_fill, head_font + cell.alignment = Alignment(horizontal="center") + cell.border = border + groups = {} + for r in rows: + groups.setdefault((r["模型"], r["TP"]), []).append(r) + for i, ((model, tp), rr) in enumerate(sorted(groups.items()), start=2): + sp = [x["加速比"] for x in rr] + vals = [model, tp, len(rr), sum(sp) / len(sp), max(sp), + sum(x["带宽GB/s"] for x in rr) / len(rr), + sum(x["TOPS"] for x in rr) / len(rr), + sum(x["性能µs"] for x in rr) / len(rr), + sum(1 for x in rr if x["验收状态"] != "已最终验收")] + ws2.append(vals) + for j in range(1, len(head2) + 1): + cell = ws2.cell(row=i, column=j) + cell.border = border + if j in (4, 5, 6, 7, 8): + cell.number_format = "0.00" + cell.alignment = Alignment(horizontal="right") + elif j in (2, 3, 9): + cell.alignment = Alignment(horizontal="right") + if i % 2 == 0: + for j in range(1, len(head2) + 1): + ws2.cell(row=i, column=j).fill = alt_fill + for j, w in enumerate([12, 5, 8, 11, 11, 13, 10, 12, 18], start=1): + ws2.column_dimensions[get_column_letter(j)].width = w + ws2.freeze_panes = "A2" + + # ---- sheet 3: notes ---- + ws3 = wb.create_sheet("说明") + notes = [ + ["指标", "含义"], + ["硬件/数据类型", "worker29（j20r1n08 / 10.8.160.17 容器 zth_meta）4×K500SM_AI gfx928；INT8 W8A8 GEMM（HIP C++ + DUMMA），输出 bf16"], + ["基线µs", "固定 Triton(CUDA-graph replay) 基线中位延迟；与优化 kernel 同协议测量"], + ["性能µs", "最优 variant 的中位延迟（每个 (模型×TP×算子×M) 取记录中的最优已验收结果）"], + ["加速比 / 提升%", "加速比 = 基线µs ÷ 性能µs；提升% = (基线µs ÷ 性能µs − 1) × 100"], + ["带宽GB/s", "算法最小字节数 ÷ 性能µs = (M·K + K·N + 4(M+N) + 2·M·N) ÷ t；非实测 HBM 流量"], + ["TOPS", "逻辑 INT8 运算率 = 2·M·N·K ÷ t"], + ["验收状态", "『已最终验收』= 该任务串行最终验收全部 shape 通过（final_report.json status=success）；『仅worker验收』= 该任务未产出成功 final_report，取 worker 并行探索阶段已验收最优 median（黄色行）"], + ["覆盖范围", f"共 {len(rows)} 个算子实例：Hy3（TP4 M16/M4096、TP8 M16/M4096）、MiniMax-M3（TP4 M4096、TP8 M16/M4096）、GLM5.2（TP4 M4096、TP8 M16/M4096）"], + ["未覆盖", "DeepSeek-V4 默认 42-shape 的早期任务报告在本地已被清空（0 字节），无可用数据；Hy3 TP8 M16+M4096 合并任务、GLM 旧的任务快照因重复或被更新版本覆盖未纳入"], + ["数据来源", "从各任务工作区记录直接提取（final_report.json / plan.json / shared_baseline/results.json / workers/*/result.json），未做任何新的性能测试"], + ] + for i, pair in enumerate(notes, start=1): + ws3.append(pair) + for j in (1, 2): + cell = ws3.cell(row=i, column=j) + cell.alignment = Alignment(wrap_text=True, vertical="top") + if i == 1: + cell.fill, cell.font = head_fill, head_font + ws3.column_dimensions["A"].width = 18 + ws3.column_dimensions["B"].width = 110 + + path = os.path.join(OUT_DIR, "dkao_operators_concise.xlsx") + wb.save(path) + print("wrote", path) + print("sheets:", wb.sheetnames, "| data rows:", len(rows)) + + +if __name__ == "__main__": + main() diff --git a/metainfer/tasks/dcu_kernel_auto_opt/reports/dkao_operators_concise.csv b/metainfer/tasks/dcu_kernel_auto_opt/reports/dkao_operators_concise.csv new file mode 100644 index 00000000..cd13d5bd --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/reports/dkao_operators_concise.csv @@ -0,0 +1,50 @@ +模型,TP,算子,M,N,K,基线µs,性能µs,加速比,提升%,带宽GB/s,TOPS,P90µs,验收状态 +GLM5.2,4,fused_qkv_a_proj,4096,2624,6144,40458.051,960.1,42.139,4113.9,65.4,137.56,961.375,仅worker验收 +GLM5.2,4,kv_b_proj,4096,7168,512,9702.386,430.997,22.511,2151.1,149.7,69.76,431.527,仅worker验收 +GLM5.2,4,o_proj,4096,6144,4096,58712.341,1550.283,37.872,3687.2,59.5,132.98,1569.931,仅worker验收 +GLM5.2,4,q_b_proj,4096,4096,2048,19881.949,620.323,32.051,3105.1,81.2,110.78,621.137,仅worker验收 +GLM5.2,4,shared_down_proj,4096,6144,512,8243.047,350.98,23.486,2248.6,158.5,73.42,351.66,仅worker验收 +GLM5.2,4,shared_gate_up_proj,4096,1024,6144,13036.336,410.155,31.784,3078.4,97.2,125.66,411.251,仅worker验收 +GLM5.2,8,fused_qkv_a_proj,16,2624,6144,97.99,41.966,2.335,133.5,388.8,12.29,42.09,仅worker验收 +GLM5.2,8,kv_b_proj,16,3584,512,18.833,8.607,2.188,118.8,229.1,6.82,9.766,仅worker验收 +GLM5.2,8,o_proj,16,6144,2048,66.98,24.005,2.79,179.0,534.8,16.77,24.019,仅worker验收 +GLM5.2,8,q_b_proj,16,2048,2048,42.035,14.121,2.977,197.7,304.6,9.5,14.144,仅worker验收 +GLM5.2,8,shared_down_proj,16,6144,256,18.177,8.239,2.206,120.6,218.2,6.11,9.122,仅worker验收 +GLM5.2,8,shared_gate_up_proj,16,512,6144,99.318,10.818,9.181,818.1,301.6,9.31,10.87,仅worker验收 +GLM5.2,8,fused_qkv_a_proj,4096,2624,6144,68661.417,938.071,73.194,7219.4,67.0,140.79,939.324,已最终验收 +GLM5.2,8,kv_b_proj,4096,3584,512,6939.939,134.756,51.5,5050.0,247.3,111.55,134.94,已最终验收 +GLM5.2,8,o_proj,4096,6144,2048,54775.797,755.802,72.474,7147.4,94.4,136.38,756.358,已最终验收 +GLM5.2,8,q_b_proj,4096,2048,2048,14385.571,215.436,66.774,6577.4,136.4,159.49,215.749,已最终验收 +GLM5.2,8,shared_down_proj,4096,6144,256,17700.262,273.626,64.688,6368.8,193.7,47.09,274.465,已最终验收 +GLM5.2,8,shared_gate_up_proj,4096,512,6144,21254.589,272.711,77.938,7693.8,119.3,94.49,272.997,已最终验收 +Hy3,4,o_proj,16,4096,2048,54.617,15.558,3.51,251.0,550.8,17.25,15.603,已最终验收 +Hy3,4,qkv_proj,16,2560,4096,80.19,27.723,2.893,189.3,383.9,12.1,27.861,已最终验收 +Hy3,4,shared_down_proj,16,4096,384,23.216,9.129,2.543,154.3,189.1,5.51,9.246,已最终验收 +Hy3,4,shared_gate_up_proj,16,768,4096,73.262,11.451,6.398,539.8,282.8,8.79,11.477,已最终验收 +Hy3,4,o_proj,4096,4096,2048,19881.949,803.898,24.732,2373.2,62.7,85.48,805.136,已最终验收 +Hy3,4,qkv_proj,4096,2560,4096,24045.12,756.726,31.775,3077.5,63.8,113.51,760.159,已最终验收 +Hy3,4,shared_down_proj,4096,4096,384,4381.289,214.838,20.393,1939.3,171.0,59.97,215.117,已最终验收 +Hy3,4,shared_gate_up_proj,4096,768,4096,6678.282,220.712,30.258,2925.8,118.9,116.76,221.538,已最终验收 +Hy3,8,o_proj,16,4096,1024,32.137,11.044,2.91,191.0,394.6,12.15,11.163,仅worker验收 +Hy3,8,qkv_proj,16,1280,4096,74.782,19.297,3.875,287.5,277.5,8.69,19.323,仅worker验收 +Hy3,8,shared_down_proj,16,4096,192,17.312,8.671,1.997,99.7,108.1,2.9,9.144,仅worker验收 +Hy3,8,shared_gate_up_proj,16,384,4096,66.527,12.488,5.327,432.7,132.3,4.03,12.534,仅worker验收 +Hy3,8,o_proj,4096,4096,1024,14151.792,282.851,50.033,4903.3,148.4,121.48,283.359,已最终验收 +Hy3,8,qkv_proj,4096,1280,4096,19381.383,318.457,60.86,5986.0,102.1,134.87,326.052,已最终验收 +Hy3,8,shared_down_proj,4096,4096,192,10762.616,171.812,62.642,6164.2,204.6,37.5,172.16,已最终验收 +Hy3,8,shared_gate_up_proj,4096,384,4096,16712.231,115.35,144.883,14388.3,186.5,111.7,115.534,已最终验收 +MiniMax-M3,4,o_proj,4096,6144,2048,29702.473,752.485,39.473,3847.3,94.8,136.99,753.655,已最终验收 +MiniMax-M3,4,qkv_proj,4096,2304,6144,31556.192,1561.908,20.204,1920.4,37.3,74.25,1577.306,已最终验收 +MiniMax-M3,4,qkv_proj_and_indexer_qk,4096,2560,6144,35918.768,1089.015,32.983,3198.3,56.8,118.32,1102.644,已最终验收 +MiniMax-M3,4,shared_down_proj,4096,6144,768,11759.473,634.156,18.543,1754.3,91.8,60.95,634.634,已最终验收 +MiniMax-M3,4,shared_gate_up_proj,4096,1536,6144,19776.488,962.347,20.55,1955.0,49.1,80.33,965.787,已最终验收 +MiniMax-M3,8,o_proj,16,6144,1024,38.655,11.773,3.283,228.3,554.6,17.1,12.421,已最终验收 +MiniMax-M3,8,qkv_proj,16,1280,6144,110.478,33.196,3.328,232.8,241.3,7.58,33.329,已最终验收 +MiniMax-M3,8,qkv_proj_and_indexer_qk,16,1536,6144,111.486,39.737,2.806,180.6,241.4,7.6,39.853,已最终验收 +MiniMax-M3,8,shared_down_proj,16,6144,384,23.152,9.341,2.478,147.8,276.9,8.08,10.176,已最终验收 +MiniMax-M3,8,shared_gate_up_proj,16,768,6144,104.527,17.319,6.035,503.5,279.7,8.72,17.501,已最终验收 +MiniMax-M3,8,o_proj,4096,6144,1024,19897.517,394.505,50.437,4943.7,154.3,130.64,394.903,已最终验收 +MiniMax-M3,8,qkv_proj,4096,1280,6144,41000.618,474.805,86.353,8535.3,91.7,135.69,475.417,已最终验收 +MiniMax-M3,8,qkv_proj_and_indexer_qk,4096,1536,6144,35604.698,519.467,68.541,6754.1,90.9,148.82,520.489,已最终验收 +MiniMax-M3,8,shared_down_proj,4096,6144,384,22389.162,311.009,71.989,7098.9,174.6,62.14,311.135,已最终验收 +MiniMax-M3,8,shared_gate_up_proj,4096,768,6144,31882.257,311.935,102.208,10120.8,116.0,123.92,312.368,已最终验收 diff --git a/metainfer/tasks/dcu_kernel_auto_opt/reports/dkao_operators_concise.xlsx b/metainfer/tasks/dcu_kernel_auto_opt/reports/dkao_operators_concise.xlsx new file mode 100644 index 00000000..3c9aca63 Binary files /dev/null and b/metainfer/tasks/dcu_kernel_auto_opt/reports/dkao_operators_concise.xlsx differ diff --git a/metainfer/tasks/dcu_kernel_auto_opt/server/routes.py b/metainfer/tasks/dcu_kernel_auto_opt/server/routes.py index fb56ba0d..f37acbd4 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/server/routes.py +++ b/metainfer/tasks/dcu_kernel_auto_opt/server/routes.py @@ -2,7 +2,9 @@ from __future__ import annotations +import hashlib import json +import math import os import re import subprocess @@ -12,7 +14,7 @@ from pathlib import Path from typing import Any, Dict -from fastapi import APIRouter, HTTPException, Request +from fastapi import APIRouter, HTTPException, Request, Response from metainfer.server._helpers import ( require_task_type, @@ -39,12 +41,16 @@ rollback_skill, sync_skill_libraries, ) +from metainfer.orchestrator import hw_profile from ..orchestrator.variant_store import ( add_variant, derive_variant_meta, list_variant_index, ) -from ..orchestrator.w8a8_baselines import fixed_triton_graph_baseline +from ..orchestrator.w8a8_baselines import ( + fixed_triton_graph_baseline, + trusted_baseline_us, +) PLUGIN_TYPE = "dcu-kernel-auto-opt" @@ -98,18 +104,29 @@ def _ahe_child_entry(state_dir_raw: str): """Build a read-only entry for a harness_evolve child's artifact dir. Raises 404 unless ``state_dir`` is the ``state/`` of a question inside a - *registered* harness_evolve task's ``children/iteration_NNN//`` - tree. The registry is what makes the caller trustworthy: without it any - path on the box could be read through these routes. + *registered* harness_evolve task's ``children///`` tree. + The registry is what makes the caller trustworthy: without it any path on + the box could be read through these routes. """ from metainfer.server import tasks as _tasks state_dir = Path(state_dir_raw) parts = state_dir.parts - # .../children/iteration_NNN//state - # parts[-1] = "state", parts[-2] = , parts[-3] = iteration_NNN + # .../children///state + # parts[-1] = "state", parts[-2] = , parts[-3] = + # + # The attempt directory is not always a bare ``iteration_NNN``: HE gives a + # retry ``_a`` and the generalization retake ``_generalization``, and + # those runs are just as viewable as the first pass -- a stricter pattern + # here 404'd every request the mirror made for the retake's round, so the + # drill-down showed nothing but errors. HE owns the naming rule, so ask it + # rather than keeping a second copy of the pattern that can drift. + from metainfer.tasks.harness_evolve.orchestrator.rounds import ( + iteration_of_dir_name, + ) + if state_dir.name != "state" or len(parts) < 3 \ - or not re.fullmatch(r"iteration_\d+", parts[-3]): + or not iteration_of_dir_name(parts[-3]): raise HTTPException(404, f"not an AHE child directory: {state_dir}") question_dir = state_dir.parent iteration_dir = question_dir.parent @@ -164,6 +181,59 @@ def _require_writable(entry) -> None: "iteration; steer it from its own task page") +def _skill_profile_blocked_reason(entry) -> str: + """Why this task's skills may not be fused/published here, or ``""``. + + Fusion and publishing write into the skill library of **this machine's** + profile (``~/.dsh/skills``, mirrored to Claude), while the pending skill's + numbers were measured on the task's own board. Filing one board's findings + under the other's edition is exactly the mix-up the profile split exists to + prevent -- air boards have no HBM address hashing, so the same code measures + 1.06-1.70x apart per operator -- so the two boards must be the same. A task + with no record is refused for the same reason a variant promotion is: there + is no way to tell whose numbers these are. + + One rule, two callers: the write guard below (409) and ``GET /skills``, + which reports it so the page hides the buttons instead of letting them fail. + Deliberately non-raising: an unresolvable machine blocks the write rather + than crashing the *read* that explains why. + """ + detected = hw_profile.profile_id() or "" + from metainfer.server.tools.backfill_task_hw_profile import recorded_profile + + task_profile, _source = recorded_profile(state_dir_for(entry)) + if not detected: + return ( + "cannot establish this machine's hardware profile, so its skill " + "library has no board to file findings under; resolve the profile " + "before publishing or fusing skills" + ) + if not task_profile: + return ( + f"this task has no recorded hardware profile, so its skill cannot " + f"be filed under {detected!r}: classify it first with " + "`python3 -m metainfer.server.tools.backfill_task_hw_profile --apply`" + ) + if task_profile != detected: + return ( + f"refusing to write a {task_profile!r} task's skill into the " + f"{detected!r} skill library: its findings were measured on " + f"{task_profile!r} and would be read as {detected!r} knowledge " + f"(1.06-1.70x apart, per operator). Do it on the {task_profile!r} " + "machine instead." + ) + return "" + + +def _require_task_board_matches_library(entry) -> None: + """Refuse to file a task's skill under another board's edition. + + The visible half is ``SkillLibrary``'s hidden buttons; this is the half a + hand-made request cannot bypass (same shape as ``add_variant_route``'s 409). + """ + reason = _skill_profile_blocked_reason(entry) + if reason: + raise HTTPException(status_code=409, detail=reason) def _latest_gpu_lease(state_dir: Path) -> Dict[str, Any]: @@ -416,11 +486,65 @@ def _read_bootstrap_attempts( return attempts +def localize_experiment_baseline( + experiment: Dict[str, Any], + shape_id: str, + shape_params: Optional[Dict[str, Any]] = None, +) -> Dict[str, Any]: + """Show a round's speedup against **this board's** baseline. + + A round records ``speedup = baseline_us / median_us`` as it ran. For a task + that predates the hardware-profile split that ``baseline_us`` is the + hardcoded reference board's number (``user_supplied_fixed_table``), so the + page would show e.g. ``shared_down_proj 1.967x`` next to a variant that + correctly says ``超基线 2.55x`` -- two ratios for one kernel, one of them + cross-board. This rewrites the *displayed* numbers when the recorded + baseline is not provably this board's, and keeps the original under + ``recorded_*`` so nothing is hidden (the historical decision is untouched: + report and iteration files are never modified). + """ + recorded = experiment.get("baseline_us") + if recorded is None: + return experiment + if trusted_baseline_us(recorded) is not None: + return experiment # provably this board's: as recorded + if not shape_params: + return experiment + try: + record = fixed_triton_graph_baseline(shape_id, dict(shape_params)) + except ValueError: + return experiment + local = trusted_baseline_us(record) + median = experiment.get("median_us") + if median is None: + median = (experiment.get("metrics") or {}).get("median_us") + if local is None or median in (None, 0): + return experiment + out = dict(experiment) + out["baseline_us"] = local + out["speedup"] = float(local) / float(median) + out["recorded_baseline_us"] = recorded + out["recorded_speedup"] = experiment.get("speedup") + out["baseline_profile"] = record.get("baseline_profile") + out["baseline_note"] = ( + f"speedup recomputed against this board's baseline ({local} us); the run " + f"itself used {recorded} us from another hardware profile" + ) + return out + + def read_worker_lanes( workspace_dir: Path, state_dir: Path | None = None ) -> Dict[str, Any]: """Return exactly four worker lanes with their full iteration history.""" plan = _load(workspace_dir / "plan.json", {}) or {} + shape_params_by_id = { + str(item.get("id")): { + key: item[key] for key in ("M", "N", "K", "tp_size") if key in item + } + for item in (plan.get("shapes") or []) + if isinstance(item, dict) and item.get("id") + } max_iterations = int(plan.get("max_iterations") or 0) assignment_by_worker = { str(item.get("worker_id")): item @@ -435,7 +559,7 @@ def read_worker_lanes( continue name = str(agent.get("name") or "") match = re.match( - r"^(worker_[0-3])-(?:.+-iter\d+|skill|bootstrap-attempt\d+)$", + r"^(worker_\d+)-(?:.+-iter\d+|skill|bootstrap-attempt\d+)$", name, ) if not match: @@ -448,9 +572,21 @@ def read_worker_lanes( >= float(current.get("started_at") or 0) ): agent_by_worker[worker_id] = agent - lanes = [] + # Lanes are the task's own workers, in plan order. They used to be the fixed + # `worker_0..3`, which is wrong on an eight-card machine where manual mode + # names each worker after the card it runs on (`worker_6`). The detail view + # still always shows four rows, so a shorter plan is padded with the + # historical lane names that are not already taken -- which reproduces the + # old listing exactly for a plan that stays on cards 0..3. + worker_ids = list(assignment_by_worker) for index in range(4): - worker_id = f"worker_{index}" + if len(worker_ids) >= 4: + break + candidate = f"worker_{index}" + if candidate not in worker_ids: + worker_ids.append(candidate) + lanes = [] + for worker_id in worker_ids: worker_root = workspace_dir / "workers" / worker_id assignment = assignment_by_worker.get(worker_id, {}) assigned_shapes = assignment.get("shapes") or [] @@ -463,6 +599,18 @@ def read_worker_lanes( if runs_root.exists(): for path in sorted(runs_root.glob("*/experiments.jsonl")): experiments.extend(_read_jsonl(path)) + # Display each round's speedup against THIS board's baseline (see + # ``localize_experiment_baseline``): the recorded ratio of a pre-split + # task is computed from another profile's number and must not be shown + # next to a variant that uses this board's. + experiments = [ + localize_experiment_baseline( + item, + str(item.get("shape_id") or ""), + shape_params_by_id.get(str(item.get("shape_id") or "")), + ) + for item in experiments + ] experiments.sort( key=lambda item: ( float(item.get("timestamp") or 0), @@ -558,9 +706,13 @@ def read_worker_lanes( active_iteration["max_repairs"] = int( status.get("max_repairs") or 4 ) + # A lane without an assignment shows the card its name claims, which is + # how the fixed 0..3 rows used to be labelled. + digits = re.fullmatch(r"worker_(\d+)", worker_id) + fallback_gpu = int(digits.group(1)) if digits else None lanes.append({ "worker_id": worker_id, - "gpu": assignment.get("gpu", index), + "gpu": assignment.get("gpu", fallback_gpu), "assigned": bool(assignment), "assigned_shapes": assigned_shapes, "state": lane_state or ( @@ -595,37 +747,289 @@ def read_worker_lanes( return {"workers": lanes} +def _stat_token(path: Path) -> str: + try: + st = path.stat() + except OSError: + return "-" + return f"{int(st.st_mtime)}:{st.st_size}" + + +def _state_signature(state_dir: Path, workspace_dir: Path) -> str: + """Tiny fingerprint of everything the detail view renders. + + The detail page polls while a run is alive. Fetching its two heavy + payloads (``/summary`` ~0.6 MB, ``/iterations`` ~5.7 MB on a finished + multi-worker task) every 3 s was the single biggest cost of keeping a task + open: ~6 MB per poll, re-parsed and re-compared in the browser. This + signature costs a few dozen ``stat`` calls, so the client can skip both + unless something actually moved. + + **Completeness is a correctness requirement, not an optimisation.** The + signature answers "may I reuse what I already have?", so every file the two + payloads are derived from has to be in here. It used to cover only + ``iteration.json`` / ``status.json`` / ``best.json``, which was blind to the + two feed files the lanes are really built from: + + * ``workers/*/runs/*/experiments.jsonl`` -- the rounds themselves. A + worker that uses the ``runs/`` layout (which is what a real run writes; + ``iterations/`` is the older layout) had *no* covered input, so the + gate reported "unchanged" while rounds were being appended and the page + froze on a stale round list. + * ``state_dir/timeline.jsonl`` -- appended for the whole run, and it is + where ``/summary``'s ``gpu_leases`` comes from. + + ``_read_jsonl``-backed feeds are tracked by (mtime, size); a truncated or + rewritten file changes size, an appended one changes both. + """ + tokens = [ + _stat_token(state_dir / "run.json"), + _stat_token(state_dir / "requirements.json"), + _stat_token(state_dir / "timeline.jsonl"), + _stat_token(state_dir / "measurement_gate.jsonl"), + _stat_token(state_dir / "agents.json"), + _stat_token(workspace_dir / "final_report.json"), + _stat_token(workspace_dir / "plan.json"), + ] + workers = workspace_dir / "workers" + if workers.is_dir(): + # Every glob below is sorted: the digest must depend on file state + # only, never on directory-iteration order (see + # test_signature_is_stable_and_content_addressed). + for pattern in ( + "*/status.json", + "*/bootstrap_progress.json", + "*/bootstrap_result.json", + "*/result.json", + "*/best.json", + "*/iterations/*/iteration.json", + "*/runs/*/experiments.jsonl", + ): + for path in sorted(workers.glob(pattern)): + tokens.append(_stat_token(path)) + digest = hashlib.sha1("|".join(tokens).encode("utf-8")).hexdigest() + return digest[:20] + + +# --------------------------------------------------------------------------- # +# Memoised heavy payloads +# --------------------------------------------------------------------------- # +# ``/summary`` and ``/iterations`` are pure functions of (state_dir, +# workspace_dir, view), and the signature above is a complete, cheap +# fingerprint of exactly those inputs. So the signature doubles as the cache +# key *and* the validator: a poll whose signature has not moved is answered +# without rebuilding the body at all. +# +# That ordering is the whole point. The old arrangement built the payload +# first and hashed it afterwards, which meant an unchanged poll still paid the +# full rebuild -- measured on a 5.6 MB lane payload: 43-58 ms reading the +# per-round files, 31-35 ms of ``json.dumps``, 3.8 ms of sha1. Content-hashing +# could only ever save the transfer, never that CPU, and it silently gave up +# once a body passed the middleware's size cap (a 20 MB task therefore re-sent +# 20 MB on every single poll). + +_PAYLOAD_CACHE: Dict[Any, bytes] = {} +_PAYLOAD_CACHE_BYTES = 0 +_PAYLOAD_CACHE_MAX_ENTRIES = 6 +_PAYLOAD_CACHE_MAX_BYTES = 64 * 1024 * 1024 + + +def _cached_payload(key: Any, build) -> bytes: + """Serialise one heavy payload once per signature. + + Bounded by entry count *and* total bytes, evicting oldest-first: these + payloads are megabytes, so an unbounded cache would accumulate every task + ever opened in the WebUI's lifetime. + """ + global _PAYLOAD_CACHE_BYTES + hit = _PAYLOAD_CACHE.get(key) + if hit is not None: + return hit + payload = build() + _PAYLOAD_CACHE[key] = payload + _PAYLOAD_CACHE_BYTES += len(payload) + while _PAYLOAD_CACHE and ( + len(_PAYLOAD_CACHE) > _PAYLOAD_CACHE_MAX_ENTRIES + or _PAYLOAD_CACHE_BYTES > _PAYLOAD_CACHE_MAX_BYTES + ): + oldest = next(iter(_PAYLOAD_CACHE)) + _PAYLOAD_CACHE_BYTES -= len(_PAYLOAD_CACHE.pop(oldest)) + return payload + + +def _json_safe(value: Any) -> Any: + """Replace non-finite floats with ``None``, recursively. + + These routes used to return a plain dict and let FastAPI serialise it. + Because they were annotated ``-> Dict[str, Any]``, FastAPI inferred a + response model and ran the payload through Pydantic, which encodes ``inf`` + and ``nan`` as JSON ``null``. Now that the handlers return a ``Response`` + (so the ETag can be decided *before* the body is built) that step is gone, + and it has to be reproduced by hand -- exactly: + + * real round files do contain non-finite floats: ``experiments.jsonl`` is + written by ``json.dumps`` with its default ``allow_nan=True``, and one + workspace here holds ``"Infinity"`` for a candidate that never measured; + * emitting a bare ``Infinity`` (i.e. ``allow_nan=True``) would produce a + body the browser's ``JSON.parse`` rejects outright, breaking the page; + * and ``allow_nan=False`` alone raises, i.e. a 500 on a real task. + + Only floats are touched, so the common case returns the original container + rather than a copy. + """ + if isinstance(value, float): + return value if math.isfinite(value) else None + if isinstance(value, dict): + return {k: _json_safe(v) for k, v in value.items()} + if isinstance(value, (list, tuple)): + return [_json_safe(v) for v in value] + return value + + +def _json_bytes(payload: Any) -> bytes: + """The bytes FastAPI's (Pydantic-backed) JSONResponse would have produced.""" + return json.dumps( + _json_safe(payload), + ensure_ascii=False, allow_nan=False, separators=(",", ":"), + ).encode("utf-8") + + +def _revalidated( + request: Request, etag: str, build +) -> Response: + """Answer 304 from the fingerprint, or build the body and tag it. + + ``build`` is only called when the client does not already hold these bytes, + so the expensive path is reachable exactly once per state change. + """ + if request.headers.get("if-none-match") == etag: + return Response(status_code=304, headers={"ETag": etag}) + return Response( + content=build(), media_type="application/json", headers={"ETag": etag} + ) + + +# Fields the lane payload carries for the *agent* pipeline, but that no browser +# code reads. Verified across every plugin's ``static/`` tree: ``candidate_isa`` +# (the dominant one at ~35 KiB per round), ``inline_asm_source``, +# ``profile_evidence`` and ``isa_optimization`` have zero references in the +# frontend, which is why ``?view=lanes`` drops them. The orchestrator reads all +# four straight from the round files on disk, so its path is untouched. +# +# This is a denylist on purpose: anything not named here survives, so a field +# this list has not caught up with is preserved rather than silently dropped. +_LANE_HEAVY_EXPERIMENT_FIELDS = frozenset({ + "candidate_isa", + "inline_asm_source", + "profile_evidence", + "isa_optimization", +}) + + +def _trim_lane_payload(payload: Dict[str, Any]) -> Dict[str, Any]: + """The list view's cut of ``/iterations``: same lanes, minus dead weight. + + Measured on this machine: 19.67 MiB -> 5.95 MiB (-70%) for the heaviest + task, 5.59 MiB -> 2.17 MiB (-61%) for a typical finished one. + """ + workers = [] + for lane in payload.get("workers") or []: + if not isinstance(lane, dict): + workers.append(lane) + continue + trimmed = {k: v for k, v in lane.items() if k != "latest"} + trimmed["experiments"] = [ + {k: v for k, v in experiment.items() + if k not in _LANE_HEAVY_EXPERIMENT_FIELDS} + if isinstance(experiment, dict) else experiment + for experiment in (lane.get("experiments") or []) + ] + workers.append(trimmed) + return {"workers": workers} + + +def _build_summary(workspace_dir: Path, state_dir: Path) -> bytes: + """Everything ``/summary`` derives from disk, as response bytes.""" + workers = [] + workers_root = workspace_dir / "workers" + if workers_root.exists(): + for status_path in sorted(workers_root.glob("*/status.json")): + workers.append(_load(status_path, {})) + return _json_bytes({ + "run": read_run(state_dir), + "plan": _load(workspace_dir / "plan.json", None), + "workers": workers, + "report": _load(workspace_dir / "final_report.json", None), + # production mode: which queue rank this task took and which devices it + # holds, so the operator can tell a real run from a harness round + "execution_profile": _execution_profile(state_dir), + "gpu_leases": _latest_gpu_lease(state_dir), + "gate": _gate_status(state_dir), + }) + + +def _build_iterations( + workspace_dir: Path, state_dir: Path, lanes_view: str +) -> bytes: + """Everything ``/iterations`` derives from disk, as response bytes.""" + lanes = read_worker_lanes(workspace_dir, state_dir) + if lanes_view == "lanes": + lanes = _trim_lane_payload(lanes) + return _json_bytes(lanes) + + def build_router(plugin) -> APIRouter: router = APIRouter() + @router.get("/state-signature") + def state_signature(task_id: str, request: Request) -> Dict[str, Any]: + """Cheap change detector for the detail view (see _state_signature).""" + entry = _entry_for(request, task_id) + require_task_type(entry, PLUGIN_TYPE) + return { + "signature": _state_signature( + state_dir_for(entry), workspace_dir_for(entry) + ) + } + @router.get("/summary") - def summary(task_id: str, request: Request) -> Dict[str, Any]: + def summary(task_id: str, request: Request) -> Response: entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) state_dir = state_dir_for(entry) workspace_dir = workspace_dir_for(entry) - workers = [] - workers_root = workspace_dir / "workers" - if workers_root.exists(): - for status_path in sorted(workers_root.glob("*/status.json")): - workers.append(_load(status_path, {})) - return { - "run": read_run(state_dir), - "plan": _load(workspace_dir / "plan.json", None), - "workers": workers, - "report": _load(workspace_dir / "final_report.json", None), - # production mode: which queue rank this task took and which devices - # it holds, so the operator can tell a real run from a harness round - "execution_profile": _execution_profile(state_dir), - "gpu_leases": _latest_gpu_lease(state_dir), - "gate": _gate_status(state_dir), - } + signature = _state_signature(state_dir, workspace_dir) + etag = '"%s-summary"' % signature + + def build() -> bytes: + return _cached_payload( + ("summary", str(workspace_dir), str(state_dir), signature), + lambda: _build_summary(workspace_dir, state_dir), + ) + + return _revalidated(request, etag, build) @router.get("/iterations") - def iterations(task_id: str, request: Request) -> Dict[str, Any]: + def iterations(task_id: str, request: Request, view: str = "full") -> Response: entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) - return read_worker_lanes(workspace_dir_for(entry), state_dir_for(entry)) + state_dir = state_dir_for(entry) + workspace_dir = workspace_dir_for(entry) + # Only two cuts exist; anything else falls back to the full body, so a + # stray query string can never invent a third cache entry. + lanes_view = "lanes" if view == "lanes" else "full" + signature = _state_signature(state_dir, workspace_dir) + # The view is part of the tag: two cuts of one state must not collide. + etag = '"%s-%s"' % (signature, lanes_view) + + def build() -> bytes: + return _cached_payload( + ("iterations", str(workspace_dir), str(state_dir), + signature, lanes_view), + lambda: _build_iterations(workspace_dir, state_dir, lanes_view), + ) + + return _revalidated(request, etag, build) @router.post("/workers/{worker_id}/guidance") async def submit_guidance( @@ -657,7 +1061,9 @@ def restart_worker(task_id: str, worker_id: str, request: Request) -> Dict[str, entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) _require_writable(entry) - if not re.fullmatch(r"worker_[0-3]", worker_id): + # The worker name carries the card (`worker_6`), so any worker_; the + # plan lookup below is what actually pins it to this task. + if not re.fullmatch(r"worker_\d+", worker_id): raise HTTPException(status_code=400, detail="invalid worker id") state_dir = state_dir_for(entry) workspace_dir = workspace_dir_for(entry) @@ -778,6 +1184,20 @@ def state_graph(task_id: str, request: Request) -> Dict[str, Any]: @router.get("/skills") def skills(task_id: str, request: Request) -> Dict[str, Any]: + """The skill library, plus the hardware-profile context of a write. + + Two profiles travel in this payload and they are different things + (compare ``variants_index``): + + * ``hw_profile`` / ``installed_profile`` -- the board **this machine** + is, i.e. whose edition the library holds and where a write lands; + * ``task_hw_profile`` -- the board **this task's numbers** were measured + on. Fusion and publishing fold the task's pending skill into the + library of this machine's board, so the two must agree; + ``write_blocked_reason`` carries the server's verdict (the same string + its 409 returns) so the page can hide the buttons rather than offer + ones that fail. + """ entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) workspace_dir = workspace_dir_for(entry) @@ -791,6 +1211,18 @@ def skills(task_id: str, request: Request) -> Dict[str, Any]: "Infrastructure-smoke findings are unrelated to the selected " "operator and cannot be published as optimization skills." ) + from metainfer.server.tools.backfill_task_hw_profile import recorded_profile + + task_profile, task_profile_source = recorded_profile(state_dir_for(entry)) + library["task_hw_profile"] = task_profile + library["task_hw_profile_source"] = task_profile_source + library["write_blocked_reason"] = _skill_profile_blocked_reason(entry) + library["profiles"] = [ + {"id": profile.id, "label": profile.label, "cooled": profile.cooled} + for profile in sorted( + hw_profile.PROFILES.values(), key=lambda item: item.id + ) + ] return library @router.post("/skills/{skill_name}/publish") @@ -798,6 +1230,7 @@ def publish(task_id: str, skill_name: str, request: Request) -> Dict[str, Any]: entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) _require_writable(entry) + _require_task_board_matches_library(entry) workspace_dir = workspace_dir_for(entry) plan = _load(workspace_dir / "plan.json", {}) or {} if plan.get("execution_mode") in {LEGACY_SMOKE_MODE, SMOKE_MODE}: @@ -821,7 +1254,13 @@ def publish(task_id: str, skill_name: str, request: Request) -> Dict[str, Any]: @router.post("/skills/sync") def sync_skills(task_id: str, request: Request) -> Dict[str, Any]: - """Manually mirror the authoritative dsh library into the ccb library.""" + """Manually mirror the authoritative dsh library into the ccb library. + + Deliberately *not* behind the task-board guard: this only re-seeds this + machine's own profile and mirrors it, so it cannot file one board's + findings under the other -- it is the repair action, not a write of task + knowledge. + """ entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) _require_writable(entry) @@ -832,10 +1271,16 @@ def sync_skills(task_id: str, request: Request) -> Dict[str, Any]: async def fuse_skill_route(task_id: str, request: Request) -> Dict[str, Any]: """Trigger the main-agent fusion of one pending skill into the dsh library (mirrored to ccb afterwards). Runs in the background; poll - ``GET /skills`` ``fuse_status`` for progress.""" + ``GET /skills`` ``fuse_status`` for progress. + + The task's own board must be this machine's: the fusion agent folds the + pending skill into the library of *this* profile, so a liquid task's + findings fused on the air machine would be read as air knowledge. + """ entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) _require_writable(entry) + _require_task_board_matches_library(entry) workspace_dir = workspace_dir_for(entry) state_dir = state_dir_for(entry) plan = _load(workspace_dir / "plan.json", {}) or {} @@ -906,21 +1351,81 @@ def rollback(task_id: str, skill_name: str, request: Request) -> Dict[str, Any]: @router.get("/variants") def variants_index(task_id: str, request: Request) -> Dict[str, Any]: """Return the fine-grained variant index (which shapes are already - captured in the shared variant library).""" + captured in the variant library), plus the hardware-profile context. + + Variants are stored per hardware profile: the air-cooled board has no + HBM address hashing, so a median measured there is 40-87% slower on + read-heavy operators and must never be compared with a liquid one. + + Two profiles travel in this payload, and they are different things: + + * ``hw_profile`` -- the board **this machine** is (what a write lands in); + * ``task_hw_profile`` -- the board **this task's numbers** were measured + on. A task page lists the variants of the task's own board: opening a + liquid task on the air machine must show the liquid library, not an + empty column (and promoting from it must be refused server-side). + """ entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) - return {"variants": list_variant_index()} + from metainfer.server.tools.backfill_task_hw_profile import recorded_profile + + task_profile, task_profile_source = recorded_profile(state_dir_for(entry)) + return { + "variants": list_variant_index(), + "hw_profile": hw_profile.snapshot(), + "task_hw_profile": task_profile, + "task_hw_profile_source": task_profile_source, + "profiles": [ + {"id": profile.id, "label": profile.label, "cooled": profile.cooled} + for profile in sorted( + hw_profile.PROFILES.values(), key=lambda item: item.id + ) + ], + } @router.post("/variants") async def add_variant_route(task_id: str, request: Request) -> Dict[str, Any]: """Add one optimized shape's accepted kernel into the shared variant library, organized by operator type / model / TP / specific operator. - Replaces an existing section for the same shape (with a file backup).""" + Replaces an existing section for the same shape (with a file backup). + + The numbers in that kernel were measured on one specific board, and the + write lands in *this* machine's profile, so the two must be the same: + promoting a liquid task's kernel on the air machine would file liquid + measurements under the air library (``variant/air-k100ai/...``). A task + whose profile is unknown is refused for the same reason -- there is no + way to tell whether its medians belong here. + """ entry = _entry_for(request, task_id) require_task_type(entry, PLUGIN_TYPE) _require_writable(entry) state_dir = state_dir_for(entry) workspace_dir = workspace_dir_for(entry) + detected = hw_profile.require().id + from metainfer.server.tools.backfill_task_hw_profile import recorded_profile + + task_profile, task_profile_source = recorded_profile(state_dir) + if not task_profile: + raise HTTPException( + status_code=409, + detail=( + f"this task has no recorded hardware profile, so its numbers " + f"cannot be filed under {detected!r}: classify it first with " + "`python3 -m metainfer.server.tools.backfill_task_hw_profile " + "--apply`" + ), + ) + if task_profile != detected: + raise HTTPException( + status_code=409, + detail=( + f"refusing to promote a {task_profile!r} task's kernel into " + f"the {detected!r} library: its median was measured on " + f"{task_profile!r} and is not comparable here " + f"(1.06-1.70x apart, per operator). Promote it on the " + f"{task_profile!r} machine instead." + ), + ) try: body = await request.json() except Exception: # noqa: BLE001 - malformed body @@ -955,23 +1460,25 @@ async def add_variant_route(task_id: str, request: Request) -> Dict[str, Any]: # table first (final_report); when the task stopped before REPORT, # fall back to the shared fixed Triton baseline table so the variant # still records a meaningful baseline_us/speedup. + # + # Either source counts only when the number provably came from *this* + # machine's profile (trusted_baseline_us): a pre-split task report, or a + # lookup that fell back to the reference board, carries the other + # board's number -- the kernel is still recorded, the ratio is not. baseline = None initial = _load( workspace_dir / "final_report.json", {} ).get("initial_metrics") or {} b = initial.get(shape_id) - if isinstance(b, dict): - baseline = b.get("median_us") - elif b is not None: - baseline = b + if b is not None: + baseline = trusted_baseline_us(b) if baseline is None: shape_params = dict(manifest.get("shape") or {}) if meta.get("tp") is not None: shape_params["tp_size"] = meta["tp"] try: - baseline = fixed_triton_graph_baseline( - shape_id, shape_params - ).get("median_us") + record = fixed_triton_graph_baseline(shape_id, shape_params) + baseline = trusted_baseline_us(record) except ValueError: baseline = None if baseline is not None and metrics.get("median_us"): @@ -991,7 +1498,13 @@ async def add_variant_route(task_id: str, request: Request) -> Dict[str, Any]: # Hard guard: a strictly slower candidate may not replace the # existing variant for the same shape (equal/faster may). reject_slower_than_existing=True, + # This is the user-facing write boundary: the leaf must land in + # the profile of the machine that measured it. An unidentifiable + # machine fails here instead of writing into "unmigrated". + require_profile=True, ) + except hw_profile.HardwareProfileError as exc: + raise HTTPException(status_code=409, detail=str(exc)) from exc except ValueError as exc: raise HTTPException(status_code=409, detail=str(exc)) from exc return {"ok": True, **result} diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-decode/SKILL.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-decode/SKILL.md new file mode 100644 index 00000000..bca66fa8 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-decode/SKILL.md @@ -0,0 +1,99 @@ +--- +name: int8-w8a8-gemm-decode +description: > + INT8 W8A8 GEMM decode kernels (M<=32) on the AIR-COOLED Hygon K100_AI + (gfx928, no HBM address hashing): split-K partial kernels with fused atomic + combine, small-M zero-pad path, and a µs-scale acceptance protocol with noise + tolerance. Air-board edition: baselines in references/ are kme6's own. Load + together with int8-w8a8-gemm-foundations. For M>32 use int8-w8a8-gemm-prefill. +--- + + + +# INT8 W8A8 GEMM — Decode (M <= 32) — air board (K100_AI / DEV-AI, gfx928) + +This board has **no HBM address hashing**: many threads reading strided (a few +rows per pass, rows 4 KiB apart) collapse onto the same memory channels, so +concurrent-read bandwidth is roughly half the liquid board's. On the TP8 M=8 +decode family — where both boards were measured with the same script and +protocol — that costs **1.06-1.40x**, and the loss tracks how fragmented the read +is: `shared_down_proj` K=256 loses 1.399x, `qkv_proj` K=6144 only 1.059x. + +> The liquid edition of this skill is a ledger of M=2 / M=16 kernels measured on +> `K500SM_AI / DEV-HPC`. **Nothing in it is validated on this board**, and its +> latencies must not be quoted here. It lives in `skills/liquid-k500sm/` and is +> not installed on this machine. Your numbers come from +> `references/air-measured-baselines.md` or your own rounds. + +## 1. Where to start (this board) + +1. Read `references/air-measured-baselines.md`: TP8 M=8 (15 shapes, with the + reference board's value and the ratio) and TP8 M=16 (air values only). +2. Pick the shape's **air** baseline from `baselines/air-k100ai.json` via + `fixed_triton_graph_baseline(...)`; that number is what a promoted variant's + `speedup` is computed against. A record with `profile_mismatch=True` is the + other board's and must never become a speedup. +3. Reuse this board's accepted kernels from `variant/air-k100ai/` if one matches + your (operator, model, TP, M) family — same-board evidence, still requiring a + full re-validation. + +## 2. Kernel shape for M <= 32 *(method, carried from the liquid edition)* + +- Decode is **launch/latency-bound**: a µs-scale measurement where a few extra + instructions and one extra barrier are visible, and where noise is a real + threat to acceptance (see §4). +- The family that worked: **split-K partial kernels with a fused atomic + combine**, small tiles, and a zero-padded small-M path for M below the tile. +- Keep a simple scalar int8/int32 fallback for unmatched shapes; it is also the + bootstrap path and the correctness anchor. +- Prefer direct fragment epilogues (no accumulator LDS round trip, one fewer + barrier) once the lane ownership is verified. +- **On this board, spend the effort on the read pattern first**: fewer, longer + contiguous chunks per thread (merged/wider vector loads, contiguous weight + layout, one resident block reusing its data) beat occupancy or tile + micro-tuning, because the strided-read penalty is not something those levers + can buy back. + +## 3. Boundary: 16 < M <= 32 + +An extrapolation gap on both boards. Route by measurement: keep a +decode-compatible arm and a prefill-compatible arm until one is measured for the +exact shape, and do not assume the M<=16 recipe scales. + +## 4. Acceptance protocol (µs-scale noise) *(method)* + +1. Correctness first, on the exact shape, extreme and random int8 values, and + the M tails. +2. Graph capture -> update static input contents -> replay -> compare again. +3. Fair unprofiled GPU-event timing with the protocol in the foundations skill + (warmups 50, samples 50-100, launches/sample 20, median + p90 + min). +4. **Noise discipline**: this board is a shared host. Check the device is + actually free before a timed run (`VRAM <= 90%` and `HCU == 0`), take + three independent passes, and compare medians. A candidate inside the + pass-to-pass dispersion of the baseline is **not** an improvement; the + measured dispersion of this board's baseline table is ~0.7% median, with a + tail on the tiny shapes. +5. A speedup is only meaningful against **this board's** baseline entry for the + same exact (TP, M, N, K), and the promotion gates re-check it server-side. + +The task's own harness (`w8a8_bench.py`, `profile_pmc.sh`, the acceptance +thresholds in the task record) owns the authoritative protocol; this skill only +explains what tends to fail. + +## 5. Rejected ideas worth knowing *(method, from the liquid ledger)* + +The liquid edition lists decode ideas that were measured and failed there +(among them: pretending an allocation removal is kernel acceleration, and +treating PMC time as the score). Those *reasoning* traps are board independent +and are kept in the foundations skill's guardrails; the **numbers** behind them +are not, so re-measure before reuse. + +## 6. What to publish + +A decode round on this board produces: +- an accepted kernel promoted into `variant/air-k100ai///TP/M/` + (the promote path stamps `hw_profile=air-k100ai machine=kme6 measured_on=...` + and computes `speedup` against the air baseline, or omits the ratio when only + the other board's baseline exists); +- optionally a skill published into this air edition — with the shape, the air + baseline it beat, and the measurement protocol. Never a liquid figure. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-decode/references/air-measured-baselines.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-decode/references/air-measured-baselines.md new file mode 100644 index 00000000..0750014a --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-decode/references/air-measured-baselines.md @@ -0,0 +1,81 @@ +# Air-board (K100_AI / DEV-AI) measured Triton Graph baselines + +Generated from `metainfer/tasks/dcu_kernel_auto_opt/baselines/air-k100ai.json` +(profile `air-k100ai`, machine `kme6`, measured 2026-09-29, per-shape median of +three independent passes of `lmslim int8_utils.matmul_kernel` = the production +W8A8 Triton baseline; GPU events, CUDA-Graph replay, hot cache). + +Look any shape up with the same resolver the task uses (never by eye): + +```python +from metainfer.tasks.dcu_kernel_auto_opt.orchestrator.w8a8_baselines import ( + fixed_triton_graph_baseline, +) +rec = fixed_triton_graph_baseline("", + {"tp_size": 8, "M": 8, "N": 1280, "K": 6144}) +rec["median_us"], rec["baseline_profile"], rec["profile_mismatch"] +# -> 104.736, 'air-k100ai', False (profile_mismatch=True would mean the +# number is the OTHER board's: it must then never become a speedup) +``` + +**These are the numbers for this board.** The liquid board's equivalents live in +`baselines/liquid-k500sm.json` and are NOT transferable: the ratio is 1.06-1.70x +and differs per operator (see the ratio column, and the reading rules below). + +## TP=8, M=8 (decode; this board vs the reference board) + +This is the one family where the two sides were measured with the same script and protocol, so the ratio is quotable **as a direction** (read-heavy, small-K shapes suffer most). It is still not a conversion factor for other shapes. + +| operator | N | K | air median µs | air p90 µs | liquid µs | air/liquid | +|---|---:|---:|---:|---:|---:|---:| +| `qkv_proj_and_indexer_qk` | 1536 | 6144 | **106.032** | — | 99.473 | 1.066 | +| `qkv_proj` | 1280 | 6144 | **104.736** | — | 98.865 | 1.059 | +| `shared_gate_up_proj` | 768 | 6144 | **100.336** | — | 92.928 | 1.080 | +| `shared_gate_up_proj` | 512 | 6144 | **97.584** | — | 88.448 | 1.103 | +| `fused_qkv_a_proj` | 2624 | 6144 | **94.768** | — | 87.2 | 1.087 | +| `qkv_proj` | 1280 | 4096 | **80.768** | — | 66.432 | 1.216 | +| `o_proj` | 6144 | 2048 | **66.96** | — | 58.864 | 1.138 | +| `shared_gate_up_proj` | 384 | 4096 | **63.824** | — | 57.008 | 1.120 | +| `q_b_proj` | 2048 | 2048 | **43.84** | — | 36.704 | 1.194 | +| `o_proj` | 4096 | 1024 | **39.936** | — | 31.472 | 1.269 | +| `o_proj` | 6144 | 1024 | **39.52** | — | 32.96 | 1.199 | +| `shared_down_proj` | 6144 | 384 | **25.936** | — | 19.984 | 1.298 | +| `kv_b_proj` | 3584 | 512 | **21.408** | — | 15.808 | 1.354 | +| `shared_down_proj` | 6144 | 256 | **21.168** | — | 15.136 | 1.399 | +| `shared_down_proj` | 4096 | 192 | **20.128** | — | 14.592 | 1.379 | + +## TP=8, M=16 (decode) + +Reference rows for this family come from other sessions/images, so no ratio is shown: use the air column only. + +| operator | N | K | air median µs | air p90 µs | +|---|---:|---:|---:|---:| +| `qkv_proj_and_indexer_qk` | 1536 | 6144 | **107.56** | — | +| `qkv_proj` | 1280 | 6144 | **106.732** | — | +| `shared_gate_up_proj` | 768 | 6144 | **99.984** | — | +| `fused_qkv_a_proj` | 2624 | 6144 | **96.848** | — | +| `shared_gate_up_proj` | 512 | 6144 | **94.112** | — | +| `wqkv_a` | 1536 | 4096 | **79.084** | 79.936 | +| `qkv_proj` | 1280 | 4096 | **77.368** | — | +| `shared_gate_up` | 512 | 4096 | **69.92** | 70.576 | +| `shared_gate_up_proj` | 384 | 4096 | **66.552** | — | +| `o_proj` | 6144 | 2048 | **63.532** | — | +| `indexer.wq_b` | 8192 | 1024 | **61.708** | 62.448 | +| `q_b_proj` | 2048 | 2048 | **44.604** | — | +| `wq_b_or_wo_b` | 4096 | 1024 | **41.012** | 41.696 | +| `o_proj` | 6144 | 1024 | **39.528** | — | +| `shared_down_proj` | 6144 | 384 | **25.804** | — | +| `shared_down_proj` | 6144 | 256 | **21.804** | — | +| `shared_down` | 4096 | 256 | **21.056** | 22.736 | +| `kv_b_proj` | 3584 | 512 | **20.964** | — | +| `shared_down_proj` | 4096 | 192 | **20.02** | — | + +## Reading rules + +1. **The air column is the baseline** for a same-board speedup. It is what + `baseline_us` in a promoted variant is computed against. +2. The ratio column answers "why is this board slower here", not "what will + this kernel do after optimization" -- never multiply a liquid number by it. +3. Coverage: this profile has 219 shapes (209 Graph + 10 TP8), aligned + one-for-one with the reference table. Any shape missing from it is + **unmeasured on this board**; do not substitute the liquid value. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-foundations/SKILL.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-foundations/SKILL.md new file mode 100644 index 00000000..3356c483 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-foundations/SKILL.md @@ -0,0 +1,211 @@ +--- +name: int8-w8a8-gemm-foundations +description: > + Shared foundations for INT8 W8A8 GEMM on the AIR-COOLED Hygon K100_AI + (gfx928, no HBM address hashing): operator/layout contract, DUMMA and + epilogue rules, Graph-safe PyTorch bindings, SGLang integration, fair + benchmarking, correctness gates. This is the air-board edition -- its + measured numbers are kme6's. Load it together with int8-w8a8-gemm-decode + (M<=32) or int8-w8a8-gemm-prefill (M>32). +--- + + + +# INT8 W8A8 GEMM — Shared Foundations (air board: K100_AI / DEV-AI, gfx928) + +**Which board this edition is for.** `kme6` / `air-k100ai`: K100_AI on the AI +firmware line (`DEV-AI`), where **HBM address hashing is off**. Concurrent +strided reads collapse onto the same channels, so read-heavy operators run +1.06–1.70x slower than on the liquid board (`K500SM_AI / DEV-HPC`) and the gap +is per-operator. Everything below marked *(method)* is hardware independent and +was carried over from the liquid edition; everything marked *(air measured)* is +this board's own data. + +> **Never quote another board's numbers here.** The liquid edition of this skill +> family contains liquid measurements (and, in its `references/`, deep ISA +> walkthroughs written against them). Those are archived under +> `skills/liquid-k500sm/` and are **not** installed on this machine. If you need +> a baseline, read `baselines/air-k100ai.json` (or the decode/prefill editions' +> `references/air-measured-baselines.md`) -- a number that is not in this board's +> table is simply **unmeasured here**. + +## 0. Phase routing (read first) *(method)* + +| M | Skill to load | Kernel family | Bottleneck | +|---|---|---|---| +| **M <= 32 (decode)** | `int8-w8a8-gemm-decode` | split-K partial + fused combine, small tiles | launch/latency-bound, µs-scale | +| **M > 32 (prefill)** | `int8-w8a8-gemm-prefill` | 2D M-tile, staged LDS, no split-K | LDS/compute-bound, ms-scale | + +Recipes do not transfer across that boundary, and **none of the liquid edition's +validated recipes counts as validated here** -- they are starting points to +re-measure. M in (32, 3072) is an unmeasured gap on this board: route by +measurement, keep one arm per regime. + +## 1. Operator contract *(method)* + +```text +x_q[M,K] int8 @ weight[K,N] int8 + -> accumulator[M,N] int32 + -> accumulator * x_scale[M,1] * weight_scale[N,1].T + -> out[M,N] bf16 +``` + +The timed GEMM includes integer accumulation, both scales, the bf16 conversion +and the output writeback. It excludes activation quantization, weight +preprocessing, allocation, output clearing, JIT and Graph capture. + +All target K values are divisible by 128 and all N values by 64. + +## 2. Shape and layout contract *(method)* + +```text +hidden states x[M,K] bf16, stride=(K,1) -> per_token_quant_int8 +x_q[M,K] int8, stride=(K,1); x_scale[M,1] fp32 contiguous + +checkpoint weight[N,K] int8 contiguous, stride=(K,1) + -> layer.weight = weight.t() (Triton fallback layout) + -> one-time contiguous() -> HIP weight[K,N], stride=(N,1) +``` + +- Keep the original transposed view for a safe fallback; register the contiguous + HIP buffer separately. **Never** replace `layer.weight.data` with the copy -- + that changes the fallback layout and was observed to regress TTFT. +- Packed, shape-specific weight layouts are allowed as long as packing is + one-time, outside the timed region and outside Graph capture, keeps the same + byte count and buffer (graph-stable addresses), and correctness is checked + against the raw logical `[K,N]` weight. + +## 3. DUMMA and epilogue rules *(method)* + +gfx928 wavefront = 64, so block sizes are multiples of 64. INT8 primitive: + +```text +int8 x int8 -> int32, m16n16k32 +``` + +1. `du_fill_fragment(acc, 0)` before the K loop. +2. Load A/B with `du_load_matrix_sync`, accumulate with `du_mma_sync`. +3. `du_store_matrix_sync` takes an accumulator fragment, not a raw array. +4. When materializing C in LDS, pass a scalar pointer (`&smem_c[0][0]`) and + synchronize before other lanes read it. +5. The direct epilogue ownership on gfx928 is `row = lane & 15`, + `col_mod4 = lane >> 4`, `frag.x[i]` -> columns `col_mod4 + 4*i`. Re-derive it + if you change architecture or fragment type. + +Direct fragment epilogues remove an accumulator LDS round trip and a barrier -- +prefer them once verified. + +### 3.1 ISA as an acceptance test, not a design tool *(method)* + +Save the exact code object used for timing, find the exact template symbol, then +check load/MMAC/wait/LDS-store order and read VGPR/SGPR/LDS/scratch/spill from +the metadata. Reject a candidate whose compiler removed a required load, moved a +wait before MMAC, spilled prefetched values, or silently changed cache +semantics. Use raw inline asm only for a minimal, already-verified operation. +Re-run exact correctness, Graph replay, median and P90 after every ISA change. + +## 4. Graph-safe PyTorch interface *(method)* + +```python +gemm_out(x_q, packed_weight, x_scale, weight_scale, out) +gemm_out_optimized(x_q, packed_weight, x_scale, weight_scale, out, workspace) +gemm_out_prefill(x_q, packed_weight, x_scale, weight_scale, out) +``` + +Build/load the extension and allocate output/workspace **before** capture; +launch on `at::cuda::getCurrentCUDAStream(device)`. Inside capture: no stream +creation, no sync, no allocation, no autotune, no weight preprocessing, no CPU +reads of GPU values, no address changes. The launcher is shape validation + +static dispatch + async launches only. + +## 5. SGLang integration *(method; paths differ on this board)* + +```text +CompressedTensorsW8A8Int8.process_weights_after_loading() + -> prepare_dcu_w8a8_layer(): keep transposed fallback, create contiguous + HIP weight + workspace once +CompressedTensorsW8A8Int8.apply_weights() + -> per_token_quant_int8(x) -> try_dcu_w8a8(M<=32: gemm_out_optimized / + M>32: gemm_out_prefill / unsupported: None) -> Triton fallback +``` + +The liquid edition lists `/workspace/...` container paths from that board; on +this machine confirm the actual locations before relying on them. New-path +enablement is opt-in (`SGLANG_USE_ZTH_W8A8_GEMM=1` + `PYTHONPATH`), and any +unsupported condition must return `None` and keep the stock SGLang path. + +## 6. Fair benchmarking *(method)* + +Keep three scopes apart: (1) the existing Python operator, which may include +allocation and extra fill kernels; (2) the fair `gemm_out` with preallocated +output, timing every kernel needed for the final bf16 output; (3) PMC samples, +which are **diagnostic only** -- profiled time is never the score. + +```text +excluded: input quantization, weight preprocessing, output allocation/clear +included: GEMM, scale + bf16 epilogue, split-K combine +``` + +Use unprofiled GPU events; report eager and Graph separately; never compare HIP +Graph against Triton eager. Identical warmups, samples, launches/sample, inputs, +output ownership and cache policy. Stable hot-cache protocol: warmups 50, +samples 50-100, launches/sample 20, report median + p90 + min. + +Derived TOPS/bandwidth are logical rates (`2*M*N*K / median`), not measured HBM +traffic; use PMC `vmem_read` for real global reads. + +## 7. Correctness and acceptance (shared) *(method)* + +1. Compare against the reference for every requested shape. +2. Extreme int8 values and random values. +3. M = 1/2/4/8/16, tails such as 65, and the actual prefill M. +4. Exact bf16 equality when the int32 accumulation order is identical; + otherwise define and justify a tolerance. +5. Capture Graph, update static input contents, replay, compare again. +6. Verify current-stream behaviour and absence of hidden synchronization. +7. Fair unprofiled median/P90, then PMC to explain it. +8. SGLang fallback tests plus a representative end-to-end workload. + +After a killed round, restore the accepted source to its recorded digest before +the next experiment. A proposal that fails to compile is a candidate/infra +failure -- never evidence that templates/DUMMA/INT8 are unsupported. + +## 8. Guardrails *(method)* + +- Do not mutate the original transposed weight into contiguous storage. +- Do not count deleting `torch.zeros` as kernel acceleration. +- Do not use PMC time as final latency. +- No JIT/autotune/allocation/preprocessing/CPU reads/synchronization in capture. +- Do not call logical bytes/time "physical HBM bandwidth". +- A source change with stale profile data is not evidence. +- For M >= 3072 the CPU int64 exact reference is extremely slow; the control + plane pre-seeds it once per shape (`w8a8_bench.py --prepare-reference`) -- + never run a large-M correctness check without the reference cache present. + +## 9. This board's own facts *(air measured)* + +- **Baseline table**: `baselines/air-k100ai.json` -- 219 shapes (209 Graph + 10 + TP8), aligned one-for-one with the reference board's, per-shape median of + three independent passes (pass-to-pass median dispersion 0.72%). +- **Where the pain is**: on the TP8 M=8 family (same script and protocol on both + sides) this board is 1.06-1.40x slower, worst on the small-K read-heavy shapes + (`shared_down_proj` K=256: 1.399x; K=192: 1.379x) and mildest on the large-K + compute-bound ones (`qkv_proj` K=6144: 1.059x). See + `int8-w8a8-gemm-decode/references/air-measured-baselines.md`. +- **What that implies**: spend rounds on **longer contiguous reads** (chunked or + contiguous weight layouts, merged/wider vector loads, one resident block + reused) rather than on occupancy/tile/LDS micro-tuning, and do not spend any + round on clocks, NUMA, affinity or co-tenant theories -- those cannot buy back + a strided-read penalty. +- **Variant library**: `variant/air-k100ai/...` (this board's accepted kernels + only, each stamped `hw_profile=air-k100ai machine=kme6 measured_on=...`). The + liquid tree is a different library; its numbers are not comparable. + +## 10. Open questions for this board + +- Which shapes in the M=3072 / M=4096 prefill families recover most from + contiguous-read restructuring? (No validated air recipe exists yet.) +- Does the M=8 decode penalty shrink once the kernel reads fewer, longer + contiguous chunks? Measure both against the air baseline table. +- Anything found by a round here belongs in `variant/air-k100ai/` and, once + published, in this air edition of the skill family -- not in the liquid one. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-prefill/SKILL.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-prefill/SKILL.md new file mode 100644 index 00000000..8c796139 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-prefill/SKILL.md @@ -0,0 +1,87 @@ +--- +name: int8-w8a8-gemm-prefill +description: > + INT8 W8A8 GEMM prefill kernels (M>32) on the AIR-COOLED Hygon K100_AI + (gfx928, no HBM address hashing): 2D M-tile staged-LDS DUMMA kernels, no + split-K, occupancy/LDS tuning, and this board's M=3072 / M=4096 baselines. + Air-board edition. Load together with int8-w8a8-gemm-foundations. For M<=32 + use int8-w8a8-gemm-decode. +--- + + + +# INT8 W8A8 GEMM — Prefill (M > 32) — air board (K100_AI / DEV-AI, gfx928) + +Prefill on this board has **no validated recipe yet**: every "accepted +architecture" ledger in the liquid edition was validated on `K500SM_AI / +DEV-HPC`, and this board's memory behaviour differs (no HBM address hashing -> +strided concurrent reads lose about half the bandwidth). Treat any inherited +recipe as a hypothesis to re-measure, never as a result. + +> Baseline numbers for this board: `references/air-measured-baselines.md` +> (TP8 M=3072 and M=4096) and `baselines/air-k100ai.json`. **The liquid +> edition's latencies are not quotable here** and — for M=4096 — not even the +> cross-board ratios are readable (the reference rows are single-pass and one +> was visibly disturbed by a co-tenant, so they even look like "air is faster"). +> The liquid edition lives in `skills/liquid-k500sm/` and is not installed here. + +## 1. Where to start + +1. Read `references/air-measured-baselines.md` and take the **air** baseline for + the exact shape via `fixed_triton_graph_baseline(...)`. +2. Reuse this board's kernels from `variant/air-k100ai/` where the family + matches; nothing in the liquid tree counts as evidence here. +3. Because this board loses most on fragmented reads, make the **read pattern** + the first design variable: stage through LDS with longer contiguous chunks, + keep the resident block's data reused across K rather than re-reading HBM, + and prefer merged/wider vector loads. Occupancy and LDS micro-tuning come + after that, not instead of it. + +## 2. Kernel shape for M > 32 *(method, carried)* + +- Prefill is **LDS/compute-bound** at ms scale: 2D M-tile kernels with staged + LDS and **no split-K** (split-K's combine cost dominates here). +- Block sizes are multiples of the 64-lane wavefront; the INT8 primitive is + `m16n16k32`. +- Keep a scalar fallback for unmatched shapes and for the small-M API cases. +- Shape-specific packed weight layouts are allowed (one-time, out of the timed + region and out of Graph capture, same byte count, graph-stable addresses, + correctness checked against the raw logical `[K,N]` weight). + +## 3. Boundary: 32 < M < 3072 + +Unmeasured on this board. Route by measurement and keep one arm per regime +(staged prefill vs decode-compatible) until an exact-shape measurement exists. + +## 4. Acceptance protocol *(method)* + +1. Correctness on the exact shape (extreme + random int8), Graph capture -> + content update -> replay -> compare. +2. Fair unprofiled GPU-event timing; median + p90; three independent passes. +3. **Large-M reference cache**: the CPU int64 exact reference takes minutes to + hours at these shapes. The control plane pre-seeds it once per shape + (`w8a8_bench.py --prepare-reference`) — never run a large-M correctness check + without it, and never use a CPU int64 GEMM as an ad-hoc reference (no BLAS + path for int64: a single shape took 76 s in this workspace's own tooling). + A chunked fp32 GEMM with int32 accumulation is exact here, because + `127*127*chunk < 2**24`. +4. Confirm the device was actually idle for the timed run (`VRAM <= 90%`, + `HCU == 0`); this is a shared host and a co-tenant can move a ms-scale number + by tens of percent. +5. A promoted variant's `speedup` must come from this board's baseline entry for + the same exact (TP, M, N, K); otherwise the promote path omits the ratio. + +## 5. Rejected / guardrail evidence *(method)* + +The liquid ledger's rejected ideas are worth reading for the *reasoning* traps +(claiming an allocation removal as kernel acceleration, using profiled time as +the score, accepting a change with stale profile data). Their numbers are the +liquid board's: re-measure before you reuse or refute any of them here. + +## 6. What to publish + +A prefill round on this board produces accepted kernels under +`variant/air-k100ai///TP/M/` (stamped with this board and +its measurement date) and, optionally, a skill published into this air edition: +include the shape, the air baseline it beat, the protocol, and the resource +report. Do not import liquid figures into it. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-prefill/references/air-measured-baselines.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-prefill/references/air-measured-baselines.md new file mode 100644 index 00000000..7fa62040 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-gemm-prefill/references/air-measured-baselines.md @@ -0,0 +1,78 @@ +# Air-board (K100_AI / DEV-AI) measured Triton Graph baselines + +Generated from `metainfer/tasks/dcu_kernel_auto_opt/baselines/air-k100ai.json` +(profile `air-k100ai`, machine `kme6`, measured 2026-09-29, per-shape median of +three independent passes of `lmslim int8_utils.matmul_kernel` = the production +W8A8 Triton baseline; GPU events, CUDA-Graph replay, hot cache). + +Look any shape up with the same resolver the task uses (never by eye): + +```python +from metainfer.tasks.dcu_kernel_auto_opt.orchestrator.w8a8_baselines import ( + fixed_triton_graph_baseline, +) +rec = fixed_triton_graph_baseline("", + {"tp_size": 8, "M": 8, "N": 1280, "K": 6144}) +rec["median_us"], rec["baseline_profile"], rec["profile_mismatch"] +# -> 104.736, 'air-k100ai', False (profile_mismatch=True would mean the +# number is the OTHER board's: it must then never become a speedup) +``` + +**These are the numbers for this board.** The liquid board's equivalents live in +`baselines/liquid-k500sm.json` and are NOT transferable: the ratio is 1.06-1.70x +and differs per operator (see the ratio column, and the reading rules below). + +## TP=8, M=3072 (chunked prefill) + +| operator | N | K | air median µs | air p90 µs | +|---|---:|---:|---:|---:| +| `fused_qkv_a_proj` | 2624 | 6144 | **41974.5** | — | +| `o_proj` | 6144 | 2048 | **35178.5** | — | +| `indexer.wq_b` | 8192 | 1024 | **23560.9** | 23584.6 | +| `qkv_proj_and_indexer_qk` | 1536 | 6144 | **19368** | — | +| `o_proj` | 6144 | 1024 | **18312.3** | — | +| `qkv_proj` | 1280 | 6144 | **16478.8** | — | +| `wqkv_a` | 1536 | 4096 | **13980.8** | 14037.7 | +| `qkv_proj` | 1280 | 4096 | **11585.8** | — | +| `wq_b_or_wo_b` | 4096 | 1024 | **11440.1** | 11474.1 | +| `q_b_proj` | 2048 | 2048 | **10070.1** | — | +| `shared_gate_up_proj` | 768 | 6144 | **9610.36** | — | +| `shared_down_proj` | 6144 | 384 | **7942.94** | — | +| `shared_gate_up_proj` | 512 | 6144 | **6326.73** | — | +| `shared_down_proj` | 6144 | 256 | **6042.75** | — | +| `kv_b_proj` | 3584 | 512 | **5548.28** | — | +| `shared_gate_up` | 512 | 4096 | **4422.59** | 4432.99 | +| `shared_gate_up_proj` | 384 | 4096 | **4411.61** | — | +| `shared_down` | 4096 | 256 | **4100.32** | 4130.3 | +| `shared_down_proj` | 4096 | 192 | **3478.65** | — | + +## TP=8, M=4096 (chunked prefill) + +⚠️ For this family the *reference* rows are single-pass and one of them was visibly disturbed by a co-tenant, so cross-board ratios are meaningless here (they even read as "air is faster"). Use the air column only, and prefer re-measuring both sides on one image before quoting any gap. + +| operator | N | K | air median µs | air p90 µs | +|---|---:|---:|---:|---:| +| `fused_qkv_a_proj` | 2624 | 6144 | **57764.7** | — | +| `o_proj` | 6144 | 2048 | **45946.1** | — | +| `qkv_proj_and_indexer_qk` | 1536 | 6144 | **28560.4** | — | +| `o_proj` | 6144 | 1024 | **23849.3** | — | +| `qkv_proj` | 1280 | 6144 | **22144** | — | +| `qkv_proj` | 1280 | 4096 | **16439.4** | — | +| `o_proj` | 4096 | 1024 | **16403.5** | — | +| `q_b_proj` | 2048 | 2048 | **14312.8** | — | +| `shared_gate_up_proj` | 768 | 6144 | **12796.7** | — | +| `shared_down_proj` | 6144 | 384 | **10430.3** | — | +| `shared_gate_up_proj` | 512 | 6144 | **8613.86** | — | +| `shared_down_proj` | 6144 | 256 | **7961.13** | — | +| `kv_b_proj` | 3584 | 512 | **7722.22** | — | +| `shared_gate_up_proj` | 384 | 4096 | **5961.59** | — | +| `shared_down_proj` | 4096 | 192 | **4644.43** | — | + +## Reading rules + +1. **The air column is the baseline** for a same-board speedup on this board. +2. Do not quote a cross-board ratio for the prefill families: the reference + rows come from different sessions, images and (for M=4096) a single pass. +3. M=3072 / M=4096 prefill kernels for this board have **no validated recipe + yet** -- every "accepted architecture" ledger in the liquid edition was + validated on the other board. Re-derive here and publish your own. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-quantized-gemm-optimization/SKILL.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-quantized-gemm-optimization/SKILL.md new file mode 100644 index 00000000..5dc70a6f --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/air-k100ai/int8-w8a8-quantized-gemm-optimization/SKILL.md @@ -0,0 +1,40 @@ +--- +name: int8-w8a8-quantized-gemm-optimization +description: > + Router for the INT8 W8A8 GEMM skill family on the AIR-COOLED Hygon K100_AI + (gfx928, no HBM address hashing). Load int8-w8a8-gemm-foundations plus the + phase skill: int8-w8a8-gemm-decode (M<=32) or int8-w8a8-gemm-prefill (M>32). + This router carries no tuning content itself. +--- + + + +# INT8 W8A8 GEMM — skill family router (air board: K100_AI / DEV-AI) + +The installed editions of this family are this board's (`air-k100ai`, machine +`kme6`). The liquid board's editions are a **separate library**, kept under +`skills/liquid-k500sm/` in the plugin and not installed on this machine: their +measurements were taken with HBM address hashing on and are 1.06-1.70x apart +from this board's, per operator. + +| M | Load | +|---|---| +| M <= 32 (decode) | `int8-w8a8-gemm-decode` + `int8-w8a8-gemm-foundations` | +| M > 32 (prefill) | `int8-w8a8-gemm-prefill` + `int8-w8a8-gemm-foundations` | + +- **int8-w8a8-gemm-foundations** — operator/layout contract, DUMMA and epilogue + rules, Graph-safe interface, SGLang integration, fair benchmarking, shared + correctness gates, this board's measured facts. +- **int8-w8a8-gemm-decode** — split-K partial + fused combine, small-M paths, + µs-scale acceptance with noise tolerance, and this board's TP8 M=8 / M=16 + baselines in `references/`. +- **int8-w8a8-gemm-prefill** — 2D M-tile staged-LDS kernels, no split-K, + occupancy/LDS acceptance, and this board's TP8 M=3072 / M=4096 baselines. + +Routing boundary: **M <= 32 -> decode, M > 32 -> prefill**; the in-between range +is an unmeasured gap on this board -- route by measurement and keep one arm per +regime. + +No tuning content lives in this router. Every number you rely on must come from +this board: `baselines/air-k100ai.json`, `variant/air-k100ai/`, or your own +rounds. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/SKILL.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/SKILL.md new file mode 100644 index 00000000..32b902d4 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/SKILL.md @@ -0,0 +1,388 @@ +--- +name: int8-w8a8-gemm-decode +description: > + INT8 W8A8 GEMM decode kernels for M<=32 on Hygon K500SM_AI/gfx928: split-K + partial kernels with fused atomic combine, small-M zero-pad path, and a + µs-scale acceptance protocol with noise tolerance. Load together with + int8-w8a8-gemm-foundations (contract, layout, DUMMA rules, Graph safety, + benchmarking). For M>32 use int8-w8a8-gemm-prefill. +--- + + + +> **Edition: LIQUID board (`liquid-k500sm`, K500SM_AI / DEV-HPC, HBM address +> hashing ON).** Every measurement in this edition was taken there. The air +> board (`K100_AI / DEV-AI`, no hashing) runs these operators 1.06-1.70x slower +> with a per-operator ratio, so **these numbers must not be used on the air +> board** and this edition is not installed on it -- the air machine gets +> `skills/air-k100ai/` instead. Kept for the liquid board and for provenance. +# INT8 W8A8 GEMM — Decode (M <= 32) — gfx928 / SGLang TP4 + +Worker-29 lineage, DTK 26.04; K500SM_AI/gfx928, 120 CUs, wavefront 64, +64 KiB LDS/CU, 65,536 VGPRs/CU. Validated evidence covers M<=16; M in +(16, 32] is a boundary gap — reuse the M=16 recipes below and measure (see +§3). This skill is the decode half of the phase split; load +`int8-w8a8-gemm-foundations` for the operator/layout contract, DUMMA and +epilogue rules, Graph-safe interface, SGLang integration, and shared +guardrails. Baselines are fixed user-supplied Triton CUDA-graph numbers that +were never re-measured in this lineage — do not treat any baseline as optimal; +"speedup" is always vs that fixed table. + +For the four Hy3 TP4 M=16 shapes (2026-08-23, hy3-dsh-tp4-m16-1-7f1fb1d1) the +source of truth is the authoritative control-plane fact ledger plus each +worker's `runs//experiments.jsonl` — pending worker SKILL.md tables may +be stale and must not override the ledger. **All four shapes have accepted +candidates; no plateau was proven anywhere (`plateau=false`): every accepted +number is best-found, not a proven optimum.** A shape whose worker timed out / +produced no accepted candidate stays **unoptimized** — never infer success, a +speedup, or optimality from a failed or timed-out worker. + +Operator: `x_q[M,K] int8 @ W[K,N] int8 -> int32 -> * x_scale[M,1] * +weight_scale[N,1].T -> bf16[M,N]`. Timed region = GEMM + scales + bf16 +epilogue + split-K combine inside the CUDA Graph; excludes quantization, +weight packing/preprocessing, allocation, JIT, capture. Protocol: unprofiled +CUDA-graph replay, 100 warmup / 30 samples x 100 replays, hot cache; report +median AND p90; Graph vs Graph only. + +## 1. M = 16 decode (validated) + +Use gfx928 INT8 DUMMA: + +```cpp +DUFragment +DUFragment +DUFragment +``` + +Best validated TP4 dispatch: + +| K,N | Path | +|---|---| +| 4096,1536 | non-uniform split-K=10, 4 waves, stage-K=64, double-buffer prefetch | +| 4096,1024 | split-K=16 | +| 1024,8192 | two-wave staged kernel, K-stage=128 | +| 2048,4096 | two-wave staged kernel, K-stage=128 | +| 512,4096 | two-wave staged kernel, K-stage=128 | + +The non-split kernel computes a 16x32 output tile with two wavefronts. It +loads A/B cooperatively, performs m16n16k32 DUMMA, and writes each accumulator +fragment directly from registers through a fused scale/bf16 epilogue. + +Split-K requirements: + +- Allocate int32 workspace before Graph capture. +- Every launch overwrites every partial tile; no workspace clear is needed. +- Measure partial GEMM plus combine together. +- Do not report only the main kernel as total operator latency. +- Do not restrict split-K to powers of two. Sweep block count near an integer + multiple of the physical CU count, while keeping every K boundary aligned + to the DUMMA/staging unit. + +For the fully validated `M=16,K=4096,N=1536` optimization sequence, byte +accounting, ISA evidence, rejected variants, and the 570 GB/s result, read +[references/wqkv-a-m16-split10-570gbps.md](references/wqkv-a-m16-split10-570gbps.md). + +### 1.1 Hy3 TP4 M=16 shapes (authoritative, 2026-08-23) + +Each accepted kernel is selected by an exact (m,n,k) shape guard placed +**before** generic tiled/scalar paths so decode, M=3072/4096 prefill, and +generic arms stay untouched. `HIP_KERNEL_NAME(templated<...>)` + +`hipLaunchKernelGGL` are valid on this toolchain and used by every accepted +kernel here. Scalar bootstrap kernels (qkv 476.8 us, o_proj 511.4 us, +gate_up 331.0 us, down_proj 39.4 us) are correctness-first placeholders, not +baselines. + +| shape_id | M,N,K | Accepted kernel (final) | Best median / p90 us | baseline us | speedup | +|---|---|---|---|---|---| +| hy3_tp4_qkv_proj_m16 | 16,2560,4096 | `w8a8_dumma_m16_n64_splitk_partial_kernel<6, FUSED=true>` (iter 23) | 26.046 / 26.090 | 80.19 | 3.08x | +| hy3_tp4_o_proj_m16 | 16,4096,2048 | `w8a8_gemm_m16_dumma_packedb_kernel` (iter 16) | 18.221 / 18.242 | 54.617 | 3.00x | +| hy3_tp4_shared_gate_up_proj_m16 | 16,768,4096 | `m16_dumma_runtime_splitk_lds_staged_colmajor_B` (iter 8) | 11.310 / 11.325 | 73.262 | 6.48x | +| hy3_tp4_shared_down_proj_m16 | 16,4096,384 | `w8a8_dumma_m16n16k32_sk2_kernel` (iter 15) | 10.848 / 11.088 | 23.216 | 2.14x | + +Accepted architectures (reusable recipes): + +- **qkv (N=2560,K=4096)**: block-N=64, 256 thr = 4 waves (one 16x16 + quadrant/wave), grid = 40 tiles x split-K=6 = **240 blocks = 2 blocks/CU**; + stage-K=64 double-buffered A+B LDS (10,240 B/block); one 16-B cooperative + load/lane/stage; load-bearing tid-linear B lane roles (`b_krow = tid>>2`, + `b_nchunk = tid&3` = 16 unique 128-B lines per wave-load); one + `__syncthreads`/stage; non-uniform 64-aligned K slicing ([0,K) exactly once, + preserves bit-exact int32 order); **fused last-arrival combine tail** — + monotonic global counters (`arrived % SPLIT_K == SPLIT_K-1`), + `__threadfence` + `atomicAdd`, last arriver sums the L2-hot planes + ascending, scales, stores one 8-B bf16; counters in the last 160 B of the + 16-plane contract workspace, one-time async memset before first launch → + **one kernel launch per replay**. Accepted chain: 476.8 (scalar) → 140.63 → + 131.71 → 124.92 → 54.21 → 36.88 → 31.52 → 28.48 → 26.33 → **26.05** us. +- **o_proj (N=4096,K=2048)**: 256 blocks x 128 thr = 2 waves, one 16x16 N + tile/block; in-block split-K=2, 8 uniform stages of K=128 (4 m16n16k32 + steps, 8 dwordx2 loads)/wave; **register-only K-loop transport with depth-1 + prefetch** (next stage's 8 dwordx2 loads issued before the current 4-MMAC + burst, rotated in after; compiler wait lands at next fill); packed-B + fragment-slot layout `packed[n_tile][k_step][lane][8]` produced once + out-of-timed-region for (k,n)==(2048,4096); A stays logical row-major + (32 KiB, L2-hot); LDS = two 1 KiB int32 partial planes only; one END-of-K + barrier; 44 VGPR/24 SGPR/2048 B LDS, 0 spills. +- **gate_up (N=768,K=4096)**: 16x64 block tile, 2 waves x 2 N-tiles/wave + sharing one A fragment, 12 N-groups; runtime split-K 1..16 default **16** → + 192 blocks (1.6/CU, all 120 CUs); whole 256-row split slice staged once + (A 16x272, B 64x272; `kStagedRowStride=272=256+16` bank skew), batched + 16-B/thread loads → `ds_write_b128`, one barrier, then **zero-barrier + LDS-only K loop** (unroll 2: 2 ds_read2_b32 + 2 ds_read2_b64 + 4 v_mmac); + packed `[N,K]` n-major weight for (k,n)==(4096,768) with `col_major` B + + one 8-B LDS read per fragment; S=16 combine specialization = 96 blocks x + 128 thr with all 16 plane loads issued into a register array before the + ascending sum; GEMM 123 VGPR/68 SGPR/25,856 B LDS; combine in the timed + Graph. +- **down_proj (N=4096,K=384)**: grid = N/16 = 256 blocks x 128 thr = 2 waves, + split-K=2 along K (K-half per wave, 6 unrolled steps); A staged 16x192 + row-major, 208-B padded stride (192+16), dwordx4 loads → ds_write_b128; B + staged n-major from packed `[N][K]` transpose `P[n*384+k]=W[k*4096+n]`, + 208-B padded column stride, one ds_read_b64 per step (was 8 ds_read_u8 + + ~11 VALU); **fused per-block LDS combine**: wave 0 publishes int32 partial + to `s_part`, one barrier, wave 1 adds s=0 then s=1 (exact int32 order) and + emits scaled bf16 from registers; scales prefetched into LDS `s_scale[32]` + at kernel top with stores **deferred until after both staging loops** + (removes prologue vmcnt window); 43 VGPR/18 SGPR/14,464 B LDS. + +Cross-shape rules that held (measured): + +1. **M=16 grid parallelism is scarce — split-K is the occupancy lever.** + Sweep non-power-of-two split counts near integer multiples of 120 CUs + ({2,3,4,5,6,8,9,10,12,16}); winners were 6 (qkv), 2 (o_proj, down_proj), + 16 (gate_up). Keep every K boundary aligned to the staging/DUMMA unit. +2. **Direct global fragment loads are poison first**: library byte-load + + reassembly + per-load `vmcnt` before each mmac cost 1.6-3x. Fix: one + aligned 8/16-B vector load per lane per fragment/step, via LDS staging + (gate_up, down_proj, qkv) or register-only transport (o_proj). +3. **Pack the cold once-read weight (B) outside the timed region/Graph** + into the fragment's layout ([N,K] n-major transpose, per-tile/ + fragment-slot packs) so each fragment is one contiguous dwordx2/b64 read; + leave small L2-hot A logical. Packing alone was small; it is the + prerequisite for vectorized B loads. +4. **Zero in-loop barriers for decode**; one END-of-K barrier for the LDS + int32 partial-plane combine is cheap and order-independent (bit-identical). +5. **Split-K combine is part of the operator wall** — never time the partial + kernel alone. Fuse it where it wins (qkv last-arrival tail; down_proj + per-block LDS) but **not universally**: gate_up's atomic-accumulator + + finalize fusion regressed 43.8%. +6. **LDS bank skew**: pad row/column strides to 16-B-aligned non-power-of-two + values (272=256+16, 208=192+16, 80/72/68/144 elsewhere); strides ≡ 0 + (mod 128 B) alias every row onto one bank phase. +7. **Exact int32 accumulation order (k-ascending per split, ascending split + sum) is preserved across all accepted variants → outputs are bit-identical + (0 mismatches).** +8. **Launch geometry before micro-opt** (down_proj: 256x1-wave 68.9 us → + 128x2-wave 15.4 us); verify the exact code object of the timed run + (digest-matched), not a stale sibling; source-level prefetch is not + overlap without ISA proof (`global_load_dwordx4 -> v_mmac -> vmcnt(0) -> + ds_write_b128`). + +### 1.2 Fallback routing (must stay byte-identical and building) + +- qkv: FUSED=false partial + quadrant combine kernel for workspaces < 16 + planes; generic scalar fallback for every other (m,n,k) including paired + M=2. +- o_proj: scalar fallback decodes the packed-B layout for (k,n)==(2048,4096); + M=2 fallback exact. +- gate_up: scalar fallback decodes the [N,K] pack for (k,n)==(4096,768); + identity copy otherwise; generic single-buffer tiled arms kept untuned. +- down_proj: unsplit two-wave DUMMA (grid 128) for other M=16 shapes; generic + scalar with a `b_transposed` flag for M=2. +- Routing rule: exact-shape guard → tuned kernel; any other shape → generic + tiled path or scalar fallback, never the tuned kernel. + +## 2. 1 <= M < 16 + +The implemented compatibility path zero-pads A to 16 rows in LDS and masks +output rows. It is correct and Graph-safe, but performance must be measured +for each M. Do not assume the M=16 dispatch is optimal for M=1/2/4/8. + +For the validated `M=2,K=4096,N=1536` split-K=8 route, ISA/PMC evidence, +bandwidth accounting, failed B-staging/prefetch/finalize variants, the other +M=2 baselines, and acceptance gates, read +[references/m2-decode-search.md](references/m2-decode-search.md). Do not route +other M=2 shapes to a custom kernel until that reference's exactness and +repeated Graph speedup gates pass; retain Triton otherwise. + +## 3. Boundary: 16 < M <= 32 + +No accepted candidate has been measured in (16, 32]. Reuse the M=16 recipe +(split-K + fused combine family) and measure per exact M before committing a +route. Keep a decode-compatible arm and a staged prefill arm for the same +(K,N) until measured; the decode direct-B-from-global pattern does not +transfer to larger M, and prefill's no-split-K rule does not apply here. + +## 4. Validated evidence + +### M=2 decode, 2026-08-03 + +Fair preallocated-output baseline, hot cache, 50 warmups, 50 samples, +20 launches/sample; exact output check passed: + +| Case | Eager ms | Graph ms | +|---|---|---:|---:| +| wqkv_a | 0.099411 | 0.066924 | +| wq_b / indexer.wq_b | 0.095939 | 0.047181 | +| wo_b | 0.096399 | 0.054453 | +| shared gate_up_proj | 0.096759 | 0.060389 | +| shared down_proj | 0.094487 | 0.018975 | + +For `wqkv_a (K=4096,N=1536)`, the accepted HIP dispatch is split-K=8, +StageK=512, two waves/block sharing one padded A stage, direct row-major B +loads, direct partial stores to `[8,2,N]`, and a 64-thread vectorized combine. +Three final Graph median/P90 runs were `20.004/21.592`, `20.024/20.448`, and +`19.944/20.336 us`; changed-input Graph replay was exact bf16. This is about +3.35x faster than the 66.924 us Triton Graph baseline. The other four rows +remain baseline-only and require their own search. + +### M=16 optimized HIP Graph + +Formal hot-cache run: 50 warmups, 100 samples, 20 launches/sample. +Split-K results include the combine kernel: + +| Case | HIP path | HIP Graph ms | Triton Graph ms | Speedup | +|---|---|---|---|---:|---:| +| wqkv_a | split-K=10 w4 stage64 prefetch | **0.017344** | 0.066597 | **3.84x** | +| wq_b | staged2 K128 | 0.030972 | 0.048049 | 1.55x | +| wo_b | staged2 K128 | 0.050917 | 0.054617 | 1.07x | +| shared gate_up_proj | split-K=16 | 0.018440 | 0.060965 | 3.31x | +| shared down_proj | staged2 K128 | 0.014568 | 0.021372 | 1.47x | + +The updated `wqkv_a` row uses 100 warmups, 300 samples, and 20 Graph replays +per event sample. Its P90 is 0.017616 ms and exact bf16 replay with changed +input contents passed. The other rows retain their earlier protocol and must +not be interpreted as having been rerun in the same session. + +### Hy3 TP4 M=16 (2026-08-23, hy3-dsh-tp4-m16-1-7f1fb1d1) + +The authoritative table (accepted kernels, median/p90, baselines, recipes, +accepted chains) lives in §1.1. Earlier drafts of this table listed down_proj +as (16, 6144, 768) — that was stale; the ledger and worker_3 evidence confirm +`hy3_tp4_shared_down_proj_m16 = (16, 4096, 384)` (`w8a8_dumma_m16n16k32_sk2_ +kernel`, 10.848 us median / 11.088 us p90, 2.14x vs the 23.216 us baseline). + +## 5. Decode acceptance: µs-scale noise protocol + +Decode kernels are µs-scale, so acceptance gates are dominated by measurement +noise, not kernel quality: + +- Report median AND p90; never min. Compare Graph vs Graph only. +- The `final <= worker best + max(5%, 2 µs)` gate can trip on thermal/GPU-state + drift at µs scale. The 2 µs floor (below a 40 µs best, where 5% is under the + box's own measurement resolution) covers the sub-40 µs band; above it the + window is the plain 5%. Observed failures that predate the floor: + `hy3_tp4_shared_gate_up_proj_m16` final 13.009 us + vs worker best 11.310 us (15% gap) and `minimax_tp8_qkv_proj_m16` final + 73.611 us vs worker best 33.202 us (2.2x gap) despite unchanged kernels — + both tasks stopped on the gate. The control plane re-measures on a gate + failure: up to 3 re-measures, 300 s apart (`_PERF_GATE_MAX_RETRIES=3`, + `_PERF_GATE_RETRY_INTERVAL_S=300`, `_PERF_GATE_MIN_SLACK_US=2.0`), accepting + the best (min) median; only an every-attempt failure is reported as a + regression. +- Split-K candidates: measure partial GEMM **plus combine together**; never + accept on partial-kernel time alone. +- Expect small speedups: the decode launch/latency floor (~5-10 us) and the + already-decent Triton decode baselines cap headroom at roughly 3-6x. + A flat or small gain can still be the right answer at M<=32. +- Protocol: unprofiled CUDA-graph replay, 100 warmup / 30 samples x 100 + replays, hot cache; report median AND p90 (p90 guard vs current best); + Graph vs Graph only. Timed region = GEMM + scales + bf16 epilogue + + split-K combine inside the Graph (see intro). +- One smallest focused change per round. Only after acceptance explain with + hipprof PMC (`--pmc --pmc-type 3`) and require predicted counters to move, + else the mechanism is ambiguous. `lds_wait ≈ lds_instructions` is NOT a + bottleneck signal (true in accepted kernels too). Profiled duration is + never the score. Derived TOPS/GB/s are logical rates (`2MNK/median`), not + measured HBM traffic. +- After a killed/aborted round, restore the accepted source to its recorded + digest before the next experiment. + +### 5.1 Integration and correctness gates (Graph-safe) + +- Build/load the extension before capture; preallocate output and workspace + before capture; launch on `at::cuda::getCurrentCUDAStream(device)`; no + allocation/sync/JIT/preprocessing/CPU reads inside capture; replay may + update contents, never addresses/shapes. Route M<=16 through + `gemm_out_optimized`, M>16 through the prefill path; unsupported + conditions return None to keep the original SGLang path (foundations holds + the shared Graph-safe contract). +- Correctness: exact bf16 vs reference (0 mismatches expected — int32 order + preserved), extreme/random int8, M tails, Graph capture + changed contents + + replay. Paired M=2 fallback validation must decode the packed layout: + qkv 531 us, o_proj 789.4 us, gate_up 0 mismatch, down_proj 85.9 us — all + mismatch 0. + +## 6. Rejected / guardrail evidence (decode) + +- Do not assume power-of-two split-K. The validated wqkv_a winner uses + non-uniform split10; split4 and split16 both regressed for different reasons. +- Do not fuse split-K finalization merely to remove a Graph node. Atomic + last-arriver finalization regressed, and DTK 26.04 failed to capture the + cooperative-launch candidate into the Graph. (Exception: the hy3 qkv_proj_m16 + winner's fused monotonic-arrival combine is accepted on this lineage — it + passed exact Graph replay.) +- Do not assume fewer tiles per block is better: for decode, insufficient + block count can leave roughly 120 CUs underfilled. +- Historical M=2 trials rejected uint16/uint32 reinterpret vectorization, + manual ILP/unrolling, row fusion, and collapsing grid parallelism for two + shapes. Re-profile before revisiting. +- For M=2 wqkv_a, final ISA contains `v_mmac_i32_16x16x32_i8`, but the + row-major B fragment loader expands each K=512 split to 128 + `global_load_ubyte` instructions. Coalesced B-to-LDS staging regressed to + 32.712 us, paired B-fragment prefetch to 20.488 us, uneven split10 to + 21.381 us, and atomic last-arriver finalize to 21.892 us. Do not replace + the accepted route with these merely because their source looks more + parallel or more vectorized. +- Do not describe the M=2 event-derived 316 GB/s logical tensor rate or + 326 GB/s workspace-inclusive program rate as peak HBM. PMC reported about + 98,560 external 64B read requests (about 6.31 MB) for the main kernel, but + the real compute path also executes a fixed M16 DUMMA tile: 14 of 16 rows + are padding. A 500+ GB/s pure-memory proxy is not a valid compute baseline. +- Hy3 TP4 per-shape rejected (measured; do not re-litigate without new + evidence): + - **qkv** (flat/regressed): packed-B + A-only LDS (55.7), LDS bank skew ldm + 64→68 (36.8), split-K 12 (40.0), stage-K 128 (42.1), deeper prefetch + (32.4), block-N=128 split-K=12 (32.1), per-64-column packed B (26.5), + split-K 9 (28.9), cold-plane L2 warm (28.4). Killed rounds 12/15/20/21 + (infra — prove nothing). + - **o_proj**: in-block split-K=4 on the byte-load kernel (94.2), grid + split-K + combine kernel (94.6), in-loop-barrier A-only LDS (209.3), + depth-2 prefetch (20.16), 8-wave uniform (19.60), stage-K=128 at 6 waves + imbalanced (19.33), epilogue plane bank-skew (19.39), 1-wave occupancy + cliff (19.60), stage-K=256 (19.17). No compile failures in this lineage. + - **gate_up**: finer one-wave 16x32 grid (13.62), packed A-fragment loader — + lighter ISA regressed 44% (20.19; ISA counts are not a performance + proxy), CU-aligned split-K=10 120 blocks (12.99), split-K=8 24 N-groups + (13.23), nontemporal glc/slc B loads (13.11), the whole combine CU-spread + family (13.0-20.7), atomic last-arriver + finalize 3-launch tail (20.11), + exact 2-blocks/CU 240-block grid (15.22). Killed round 7 (agent timeout — + proves nothing). + - **down_proj**: register prefetch depth-2 (flat — compiler kept vmcnt + before mmac), split-K 4/3 occupancy probes (15.0/13.6), stride-4 bank + repack (capacity-invalid, caught by correctness and reverted), four + per-8-k-group B planes (flat 11.72). More co-residency was monotonically + worse at this shape (latency/issue-bound, not occupancy-starved). All 15 + rounds built and passed correctness. +- A compile error proves only that candidate failed to compile (e.g. gate_up + iter 5's undeclared `kPackedK`, fixed in-round), never that DUMMA, INT8, a + HIP API, templates, or inline asm are unsupported. Killed rounds (missing + proposal, ~900 s timeout, restore failures) are infra/agent failures that + prove nothing. +- Do not transfer any winner (split-K default, stage size, layout, occupancy) + to another M / (K,N) / TP size / architecture without re-measuring; decode + vs prefill (M=3072/4096) conclusions do not cross M regimes. + +## 7. Next optimization questions (decode) + +- M=2: keep the validated wqkv_a dispatch; execute the phased search for the + other four `(K,N)` families and add static routes only after they clear the + Graph acceptance gate. +- M=1/4/8: build per-M dispatch only if Graph latency improves. +- M in (16, 32]: measure the M=16 recipe per exact M before committing a route + (see §3); the decode acceptance protocol in §5 applies. +- Hy3 TP4 M=16: all four shapes are **optimized** (accepted, correct, + Graph-passed) with `plateau=false` — accepted numbers are best-found, not + proven floors; continue per-shape HIP search. Baselines (80.19 / 54.617 / + 73.262 / 23.216 us) are fixed user-supplied Triton Graph numbers, never + re-measured, **never optimal**. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/references/m2-decode-search.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/references/m2-decode-search.md new file mode 100644 index 00000000..3187997c --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/references/m2-decode-search.md @@ -0,0 +1,166 @@ +# M=2 decode search and validated wqkv_a route on gfx928 + +Use this reference only for `M=2` W8A8 decode. Optimize Graph replay first; +use eager timing only as a secondary launch/binding check. + +## Fair Triton Graph baselines + +| K,N | Graph baseline | +|---|---:| +| 4096,1536 | 66.924 us | +| 1024,8192 | 47.181 us | +| 2048,4096 | 54.453 us | +| 4096,1024 | 60.389 us | +| 512,4096 | 18.975 us | + +Do not choose scalar or DUMMA from M alone. Padded DUMMA wastes 14 of 16 +rows; scalar code gives up matrix throughput. Benchmark both under the same +Graph scope. + +## Candidate A: specialized padded DUMMA + +1. Start with one wave per N16 tile. +2. Keep `A[16,K_STAGE]` in LDS, but initialize padded rows 2..15 once outside + the K loop and copy only the two real rows per stage. +3. Sweep `K_STAGE={64,128,256}` and reject resource/occupancy regressions. +4. Load packed row-major B directly first; add B LDS only after measured reuse + or load-stall evidence. +5. Store only accumulator rows 0 and 1 through the verified direct epilogue. + Lanes with `(lane&15)<2` write; all lanes still execute DUMMA. +6. Fuse scales, bf16 conversion, and output writeback. + +## Candidate B: two-row skinny dot + +Try only when padded DUMMA lacks a robust win. Let one thread own N columns, +compute two int32 accumulators, and reuse each B value across the two rows. +Keep adjacent lanes on adjacent N columns. Start with one wave and 64 columns +per block. Do not use packed casts or `sdot4` until DTK ISA and an appropriate +load-time packed weight layout are verified. Reject row fusion if reduced grid +parallelism outweighs B reuse. + +## Starting grids + +| K,N | N16 blocks | First candidates | +|---|---:|---| +| 4096,1536 | 96 | 1-wave DUMMA; split-K=2/4 | +| 1024,8192 | 512 | 1-wave; 2-wave A reuse; no split initially | +| 2048,4096 | 256 | 1-wave; no split initially | +| 4096,1024 | 64 | 1-wave; split-K=2/4/8 | +| 512,4096 | 256 | 1-wave, stage64/128; avoid split initially | + +For split-K, store only `[split_k,2,N] int32`, not 16 padded rows. Include the +combine kernel in total timing. + +## Validated wqkv_a route: M=2, K=4096, N=1536 + +Worker29, K500SM_AI/gfx928, DTK 26.04, preallocated output/workspace, 50 +warmups, 100 samples, and 20 operations/event produced these retained Graph +results: + +| Candidate | Median us | P90 us | Status | +|---|---:|---:|---| +| generic M<16 padded DUMMA | 197.585 | 197.793 | baseline | +| direct M2 Stage64 | 92.066 | 92.146 | retained | +| direct M2 Stage128 | 69.781 | 70.225 | retained | +| direct M2 Stage256 | 41.625 | 42.161 | retained | +| direct M2 Stage512 | 40.213 | 40.624 | best no-workspace route | +| direct M2 Stage1024 | 40.884 | 41.368 | rejected | +| direct M2 Stage2048 | 41.480 | 41.905 | rejected | +| split2 Stage512 | 26.564 | 26.736 | retained | +| split4 Stage256 | 22.684 | 23.312 | retained | +| split8 Stage512, one wave, combine256 | 20.544 | 21.360 | superseded | +| split8 Stage512, two waves, combine64 | 20.280 | 20.696 | selected architecture | +| B coalesced global-to-LDS staging | 32.712 | 32.912 | rejected | +| paired B-fragment prefetch | 20.488 | 20.856 | rejected | +| uneven split10 | 21.381 | 21.681 | rejected | +| fused atomic last-arriver finalize | 21.892 | 22.008 | rejected | + +After static dispatch integration, three independent selected-path Graph +median/P90 runs were: + +```text +20.004 / 21.592 us +20.024 / 20.448 us +19.944 / 20.336 us +``` + +This clears the 66.924 us Triton Graph baseline by about 3.35x. Exact bf16 +comparison passed for every retained stage and for split8 after changing all +captured input tensor contents before Graph replay. + +### Selected algorithm + +1. Split the 128 K32 MMAC tiles uniformly eight ways; launch + `96 N16 tiles x 8 splits`. +2. Put two adjacent N16 waves in each block. Let both waves share one + `A[16,512]` LDS stage. +3. Zero padded A rows 2..15 once per block and copy only rows 0..1 with + aligned 16-byte HIP vector loads. +4. Load row-major B directly through `du_load_matrix_sync`; execute 16 + `m16n16k32` MMAC instructions per split. +5. Store only two accumulator rows into `[8,2,N] int32` workspace using the + verified direct fragment lane mapping. +6. Use a separate 64-thread vectorized combine kernel for scaling and bf16 + output. Allocate output and workspace before Graph capture. + +The public no-workspace fallback uses direct Stage512. The optimized Graph +API uses split8/two-wave/combine64. + +### ISA and PMC evidence + +The exact accepted code object confirms: + +```text +16 v_mmac_i32_16x16x32_i8 +128 global_load_ubyte +16 ds_read2_b32 +2 ds_write_b128 +1 global_load_dwordx4 +4 global_store_dword +77 s_waitcnt +2 s_barrier +``` + +PMC reports 64 architecture VGPR, 32 SGPR, 8192 bytes LDS, zero scratch, and +about 98,560 external 64-byte read requests per main-kernel invocation +(approximately 6.31 MB). The byte loads are not proof that DUMMA is absent: +the MMAC is present. They arise from the row-major B fragment mapping, where +each lane's eight K values are separated by leading dimension N. + +Do not write raw MMAC merely to satisfy an inline-assembly goal. DUMMA already +emits the target instruction. Raw VMEM is also not justified here: previous +gfx928 raw global-load constraints require independent correctness proof, and +the B elements needed by one lane are not a contiguous dword. Prefer a +prepacked weight experiment only when one-time preprocessing and an explicit +packed-layout contract are allowed. + +### Bandwidth labels + +At about 19.99 us, unique logical tensor bytes imply roughly 316 GB/s; +including split-workspace writes and reads implies roughly 326 GB/s. Label +both as event-derived program rates, not physical peak HBM bandwidth. The +kernel performs real computation and the fixed DUMMA M16 tile pads 14 of 16 +rows, so comparison with a 500-800 GB/s pure-memory proxy is invalid. + +To pursue 500+ GB/s without removing computation, change one of the contracts: + +- prepack B into a fragment/lane-friendly layout outside timed execution; +- batch/fuse multiple M2 requests so the physical M16 tile has more live rows; +- fuse adjacent operators so split/combine and launch costs are amortized. + +Remeasure fallback layout and end-to-end TTFT after any weight preprocessing. + +## Experiment and acceptance order + +```text +Triton baseline -> generic padded DUMMA -> M2-specialized DUMMA +-> stage/wave sweep -> selected split-K -> skinny dot if needed +-> PMC explanation -> SGLang Graph replay +``` + +Accept a dispatch only after exact bf16 random/extreme tests, changed-content +Graph replay, at least 5% median Graph improvement in three independent runs, +no material P90 regression, no scratch/spill, combine-inclusive timing, and no +eager/end-to-end regression that erases the Graph gain. Otherwise retain +Triton. Historical vectorization/ILP failures apply only to those measured +implementations, not to all tiny-M kernels. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/references/wqkv-a-m16-split10-570gbps.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/references/wqkv-a-m16-split10-570gbps.md new file mode 100644 index 00000000..04ed0206 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-decode/references/wqkv-a-m16-split10-570gbps.md @@ -0,0 +1,116 @@ +# wqkv_a M16 split10: 570 GB/s logical request bandwidth + +## Contents + +- Evidence and metric contract +- Final kernel structure +- Why non-uniform split-K=10 wins +- Logical-byte derivation +- Optimization sequence and rejected evidence + +## Evidence and metric contract + +This case was validated on worker29 K500SM_AI/gfx928, DTK 26.04: + +```text +M=16, K=4096, N=1536 +int8 A x int8 B -> int32 -> scales -> bf16 +``` + +Use unprofiled GPU events with CUDA/HIP Graph, 100 warmups, 300 samples, and +20 operator calls per event sample. Time partial plus combine. Exact eager, +Graph, and changed-input-at-captured-address bf16 checks passed. + +The reported 570.0 GB/s is `known program-level global requests / event time`. +It is not a hardware-counter measurement of physical HBM traffic. Hot Graph +replay can hit caches. To claim HBM bandwidth, use a cache-cold ring whose +footprint exceeds cache and verify HBM bytes with appropriate PMC counters. + +## Final kernel structure + +- Use block-N=64: four wave64 compute four 16x16 tiles and share A. +- Use stage-K=64 and two LDS buffers. +- Issue stage `i+1` aligned 16-byte A/B global loads before the two current + `v_mmac_i32_16x16x32_i8` operations. +- Place `s_waitcnt vmcnt(0)` immediately before writing the alternate LDS + buffer. Keep the cross-wave barrier; the validated synchronization variant + uses a raw first `s_barrier` and compiler-managed second barrier. +- Write split-major int32 partials and use a separate 256-thread combine + kernel for ten partials, scales, bf16 conversion, and output. + +Do not infer a working software pipeline from HIP source alone. Require final +ISA to show next-stage `global_load_dwordx4` before current MMAC and its wait +near `ds_write_b128` into alternate LDS. + +Final partial-kernel metadata: + +```text +40 VGPR, 27 SGPR, 10240 B LDS +private/scratch 0, VGPR spill 0, SGPR spill 0 +workgroup 256, wavefront 64 +``` + +## Why non-uniform split-K=10 wins + +K contains 64 stage-K=64 units. Divide them as `4x7 + 6x6`, so every split +boundary remains stage/DUMMA aligned. With `N/64=24` N tiles: + +```text +partial blocks = 24 * 10 = 240 +device CUs = 120 +grid batches = exactly 2 blocks/CU +``` + +This avoids the 192-block split8 grid's second batch containing only 72 +blocks. It also avoids split16's extra workspace/combine traffic. Generalize +the method, not the literal split: scan +`ceil(N/blockN) * splitK` around integer multiples of the target CU count, +while measuring block lifetime, residency, partial bytes, and combine cost. + +## Logical-byte derivation + +```text +B read once 4096*1536 = 6,291,456 B +A reread for each of 24 N blocks 16*4096*24 = 1,572,864 B +10 partial planes written 10*16*1536*4 = 983,040 B +10 partial planes read 10*16*1536*4 = 983,040 B +bf16 output 16*1536*2 = 49,152 B +fp32 scales (16+1536)*4 = 6,208 B +total 9,885,760 B +``` + +At the default optimized API Graph median of 17.344 us: + +```text +9,885,760 B / 17.344 us = 570.0 GB/s +2*M*N*K / 17.344 us = 11.608 INT8 TOPS +P90 = 17.616 us +``` + +The 300-sample direct tuner result was 17.432 us median / 17.888 us P90. +The same-session split8/w4 baseline was 18.312 us / 18.512 us. + +## Optimization sequence and rejected evidence + +1. Reduce stage-K to 64 to shorten wait/barrier granularity. +2. Double-buffer LDS and overlap next-stage VMEM with current MMAC. +3. Move from three to four waves: A logical reads fall from 32 copies to 24, + and the partial kernel uses a 16x64 block tile. +4. Use the validated raw-first-barrier synchronization mode. +5. Replace uniform split8 with stage-aligned non-uniform split10 to match the + 120-CU grid. + +Retain these negative results as search boundaries: + +| Candidate | Graph median | Reason | +|---|---:|---| +| waves8, split8 | 18.848 us | only 96 blocks; insufficient CU coverage | +| waves4, split4 | 22.338 us | only 96 blocks despite lower partial traffic | +| waves3/4, split16 | 20.604--22.144 us | excess partial/combine traffic | +| atomic last-arriver fused finalize | 18.864 us | atomic and tail imbalance erase launch saving | +| cooperative grid finalize | unusable in Graph | DTK 26.04 captured an empty Graph | + +Do not optimize only the partial kernel. Separately measure partial and +combine for diagnosis, then accept only full-operator Graph timing. Preserve +working and rejected symbols so later DTK versions can revisit Graph capture +or synchronization without losing evidence. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-foundations/SKILL.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-foundations/SKILL.md new file mode 100644 index 00000000..2f5f8fd0 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-foundations/SKILL.md @@ -0,0 +1,348 @@ +--- +name: int8-w8a8-gemm-foundations +description: > + Shared foundations for INT8 W8A8 GEMM on Hygon K500SM_AI/gfx928: operator + and layout contracts, DUMMA/epilogue rules, Graph-safe PyTorch bindings, + SGLang integration, fair benchmarking, and correctness gates. Always load + this together with the phase skill: int8-w8a8-gemm-decode (M<=32) or + int8-w8a8-gemm-prefill (M>32). +--- + + + +> **Edition: LIQUID board (`liquid-k500sm`, K500SM_AI / DEV-HPC, HBM address +> hashing ON).** Every measurement in this edition was taken there. The air +> board (`K100_AI / DEV-AI`, no hashing) runs these operators 1.06-1.70x slower +> with a per-operator ratio, so **these numbers must not be used on the air +> board** and this edition is not installed on it -- the air machine gets +> `skills/air-k100ai/` instead. Kept for the liquid board and for provenance. + +# INT8 W8A8 GEMM — Shared Foundations (gfx928 / SGLang TP4) + +This skill records the implementation and evidence validated on worker29 with +DTK 26.04. It supersedes generic rules when the exact TP4 shapes below match. +Do not transfer measured latency or routing to another architecture, TP size, +DTK version, or weight layout without rerunning the benchmark. + +## 0. Phase routing (read first) + +The INT8 W8A8 GEMM skill family is split by execution phase because the two +regimes have non-transferable recipes, different bottlenecks, and different +acceptance protocols: + +| M | Skill to load | Kernel family | Bottleneck | +|---|---|---|---| +| **M <= 32 (decode)** | `int8-w8a8-gemm-decode` | split-K partial + fused combine, small tiles | launch/latency-bound, µs-scale | +| **M > 32 (prefill)** | `int8-w8a8-gemm-prefill` | 2D M-tile, staged LDS, no split-K | LDS/compute-bound, ms-scale | + +Load **this foundations skill together with the phase skill**. Recipes do NOT +transfer across the boundary: the M=16 decode kernel's direct-B-from-global +win does not transfer to prefill, and prefill's "no split-K" rule does not +apply to decode. M in (32, 128] is a measured-boundary gap (validated decode +covers M<=16, validated prefill starts at M=3072): route by measurement and +keep both a staged prefill arm and a decode-compatible arm until measured. + +## 1. Operator Contract + +```text +x_q[M,K] int8 @ weight[K,N] int8 + -> accumulator[M,N] int32 + -> accumulator * x_scale[M,1] * weight_scale[N,1].T + -> out[M,N] bf16 +``` + +The timed GEMM includes integer accumulation, both scales, bf16 conversion, +and output writeback. It excludes activation quantization, weight +preprocessing, tensor allocation, output clearing, JIT compilation, and +Graph capture. + +Target TP4 shapes: + +| Layer | K | N | +|---|---|---:| +| wqkv_a | 4096 | 1536 | +| wq_b / indexer.wq_b | 1024 | 8192 | +| wo_b | 2048 | 4096 | +| shared gate_up_proj | 4096 | 1024 | +| shared down_proj | 512 | 4096 | + +All target K values are divisible by 128 and all N values by 64. + +The M=4096 chunked-prefill family adds four fixed-M shapes (see +`int8-w8a8-gemm-prefill`): `hy3_tp4_qkv_proj_m4096` (M,N,K = 4096,2560,4096), +`hy3_tp4_o_proj_m4096` (4096,4096,2048), +`hy3_tp4_shared_gate_up_proj_m4096` (4096,768,4096), and +`hy3_tp4_shared_down_proj_m4096` (4096,4096,384). + +## 2. Shape and Layout Contract + +SGLang data flow: + +```text +hidden states x[M,K] bf16, stride=(K,1) + -> per_token_quant_int8 +x_q[M,K] int8, stride=(K,1) +x_scale[M,1] fp32, contiguous +``` + +Checkpoint and runtime weight flow: + +```text +checkpoint weight[N,K] int8, contiguous, stride=(K,1) + -> layer.weight = weight.t() +SGLang Triton weight[K,N], non-contiguous, stride=(1,K) + -> one-time contiguous() after loading +HIP weight[K,N], contiguous, stride=(N,1) +``` + +Keep both layouts: + +- Preserve `layer.weight` as the original transposed view for safe fallback. +- Register a non-persistent contiguous HIP buffer once after checkpoint load. +- Never replace `layer.weight.data` with the contiguous copy. Doing so changes + the Triton fallback layout and was observed to regress TTFT. +- Weight preprocessing is allowed by the task and must remain outside timed + execution and Graph capture. + +For the M=4096 lineage the packed weight layout is shape-specific and opaque +(o_proj: `[N/64, K, 64]` int8 panels; gate_up: `[K,N]→[N,K]` transpose; +down_proj: n-major `packed[n][k]`; qkv: n-major `packed[n][k]` for +(k,n)==(4096,2560)); packing is one-time, out-of-timed-region, out-of-Graph, +keeps the same byte count and buffer (graph-stable addresses unchanged), and +correctness is always checked against the raw logical `[K,N]` weight. + +## 3. DUMMA and Epilogue Rules + +gfx928 uses wavefront=64. Block size must be a multiple of 64. + +For INT8, the supported primitive is: + +```text +int8 x int8 -> int32, m16n16k32 +``` + +API rules: + +1. `du_fill_fragment(acc, 0)` before the K loop. +2. Load matrix A/B with `du_load_matrix_sync`. +3. Accumulate with `du_mma_sync`. +4. `du_store_matrix_sync` accepts an accumulator fragment, not a raw array. +5. If materializing C in LDS, pass a scalar pointer such as + `&smem_c[0][0]` and synchronize before other lanes read it. +6. A direct epilogue may use the verified gfx928 accumulator ownership: + `row=lane&15`, `col_mod4=lane>>4`, `frag.x[i]` maps to columns + `col_mod4+4*i`. Re-derive this mapping when changing architecture or + fragment type. + +Prefer direct fragment epilogues when they are verified: they remove the +accumulator LDS round trip and one barrier. For the final non-split `wo_b` +kernel, the resource report was 40 VGPR, 32 SGPR, 6144 bytes LDS, and zero +scratch. + +### 3.1 ISA use for memory-compute overlap + +Use ISA as an acceptance test after HIP/DUMMA architecture choices have been +measured, not as a substitute for tile and stage exploration: + +1. Save the exact gfx928 code object used by the timing run. +2. Locate the exact template symbol; do not disassemble a stale sibling. +3. Check load/MMAC/wait/LDS-store order in the steady-state stage. +4. Read VGPR, SGPR, LDS, scratch, and spill fields from code-object metadata. +5. Reject a candidate if the compiler removes a required load, moves the wait + before MMAC, spills prefetched values, or changes cache semantics silently. +6. Use raw inline assembly only for a minimal operation whose constraint and + hazard behavior is already verified on the current DTK/gfx928 toolchain. +7. Re-run exact correctness, Graph replay with changed contents, median, and + P90 after every ISA-level change. + +For large-M prefill, prefer HIP/DUMMA for loads and MMAC. The accepted raw asm +is limited to `s_waitcnt vmcnt(0)` before alternate-LDS writes and the +validated LDS-ready sequence `s_waitcnt lgkmcnt(0); s_barrier`. Do not copy +raw global-load/store or raw MMAC templates without a lane-layout microtest. + +## 4. Graph-Safe PyTorch Interface + +Preferred API: + +```python +gemm_out(x_q, packed_weight, x_scale, weight_scale, out) +gemm_out_optimized( + x_q, packed_weight, x_scale, weight_scale, out, workspace +) +gemm_out_prefill(x_q, packed_weight, x_scale, weight_scale, out) +``` + +Requirements: + +- Build/load the extension before capture. +- Allocate output and workspace before capture for the low-level interface. +- Launch on `at::cuda::getCurrentCUDAStream(device)`. +- Do not create streams, synchronize, allocate device memory, autotune, + preprocess weights, inspect GPU values on the CPU, or change tensor + addresses inside capture. +- The launcher should contain only shape validation, static dispatch, and + asynchronous kernel launches. +- A Graph replay may update tensor contents but not captured addresses or + shapes. + +Python exposure: + +```text +w8a8_gemm.py + -> torch.utils.cpp_extension.load (before capture) + -> TORCH_LIBRARY / TORCH_LIBRARY_IMPL registration + -> torch.ops.zth_w8a8.gemm_out* + -> HIP launcher on current PyTorch stream +``` + +## 5. SGLang Integration + +Integration flow: + +```text +CompressedTensorsW8A8Int8.process_weights_after_loading() + -> prepare_dcu_w8a8_layer() + -> preserve transposed fallback weight + -> create contiguous HIP weight/workspace once + +CompressedTensorsW8A8Int8.apply_weights() + -> per_token_quant_int8(x) + -> try_dcu_w8a8(...) + M<=32: gemm_out_optimized + M>32 : gemm_out_prefill + unsupported: return None + -> existing Triton fallback +``` + +Files on worker29: + +```text +/workspace/int8w8a8gemm/w8a8_gemm.py +/workspace/int8w8a8gemm/csrc/bindings.cpp +/workspace/int8w8a8gemm/csrc/w8a8_gemm_hip.hip +/workspace/sglang-v0.5.10_dpsk_v4/python/sglang/srt/layers/quantization/dcu_w8a8_gemm.py +/workspace/sglang-v0.5.10_dpsk_v4/python/sglang/srt/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_int8.py +``` + +Enable before server start: + +```bash +export PYTHONPATH="/workspace/int8w8a8gemm:${PYTHONPATH:-}" +export SGLANG_USE_ZTH_W8A8_GEMM=1 +``` + +Unsupported conditions must return `None` and retain the original SGLang +path. Current custom path excludes non-TP4 target K/N shapes, bias, non-bf16 +output, wrong dtype/rank/scale shape, non-contiguous HIP buffers, and weights +that were not prepared. + +## 6. Fair Benchmarking + +Keep three scopes separate: + +1. Existing Python operator: may include `torch.zeros`, allocation, and extra + fill kernels. +2. Fair `gemm_out`: preallocated output; times all kernels needed to produce + final bf16 output. +3. PMC kernel sample: diagnostic only; profiler timing is not the final score. + +Formal fair baseline: + +```text +input quantization excluded +weight preprocessing excluded +output allocation/clear excluded +GEMM included +scale + bf16 epilogue included +split-K combine included +``` + +Use unprofiled GPU Events for acceptance. Report eager and Graph separately: + +```text +Triton eager vs HIP eager +Triton Graph vs HIP Graph +``` + +Never compare HIP Graph against Triton eager. Use identical warmups, samples, +launches/sample, inputs, output ownership, and cache policy. + +Recommended stable hot-cache protocol: + +```text +warmups=50 +samples=50 or 100 +launches_per_sample=20 +median + p90 + min +``` + +Use PMC after timing to explain VGPR/LDS/scratch, bank conflicts, MFMA use, +and cache behavior. Do not treat lower VGPR or bank conflicts alone as a +performance win. `lds_wait ≈ lds_instructions` alone is not a bottleneck +signal (true in accepted kernels too). Profiled durations are perturbed — +never use PMC time as the score. Derived TOPS / bandwidth numbers are +`2*M*N*K / median` style logical rates, not measured HBM traffic; use PMC +vmem_read for real global reads. Phase-specific acceptance protocols live in +the decode / prefill skills. + +## 7. Correctness and Acceptance (shared) + +Before accepting a variant: + +1. Compare against the reference for all five TP4 shapes. +2. Test extreme int8 values and random values. +3. Test M=1/2/4/8/16, M tails such as 65, and actual prefill M=3072. +4. Require exact bf16 equality when comparing implementations with identical + int32 accumulation order; otherwise define and justify tolerance. +5. Capture Graph, update static input contents, replay, and compare again. +6. Verify current stream behavior and absence of hidden synchronization. +7. Run fair unprofiled median/P90. +8. Then use hipprof/PMC to explain the result. +9. Run SGLang fallback tests and a representative end-to-end workload. + +After a killed/aborted round, restore the accepted source to its recorded +digest before the next experiment. `hipLaunchKernelGGL` + +`HIP_KERNEL_NAME(templated<...>)` is valid and used by every accepted kernel; +compile failures of a proposal are candidate/infra failures, never evidence +that templates/DUMMA/INT8 are unsupported. + +Relevant tests: + +```bash +HIP_VISIBLE_DEVICES=0 python /workspace/int8w8a8gemm/tests/test_correctness_graph.py --full --m 16 +HIP_VISIBLE_DEVICES=0 python /workspace/int8w8a8gemm/tests/test_sglang_integration.py +HIP_VISIBLE_DEVICES=0 python /workspace/int8w8a8gemm/tests/benchmark_prefill.py --case tp4_decode --m 3072 +``` + +## 8. Shared Guardrails (rejected evidence) + +- Do not mutate the original SGLang transposed weight into contiguous storage; + it silently changes fallback performance. +- Do not count deletion of `torch.zeros` as GEMM kernel acceleration. +- Do not use PMC time as final latency. +- Do not perform JIT, autotuning, allocation, preprocessing, CPU reads, or + synchronization during Graph capture. +- Do not call event-derived logical bytes/time "physical HBM bandwidth". + Report it as program-level logical request bandwidth unless PMC and a + cache-cold protocol prove actual HBM bytes. +- A source change with unchanged/stale profile data is not evidence. Confirm + the code object/source digest and rerun unprofiled timing. +- Reference cache: for M>=3072 the CPU int64 exact reference takes ~10+ + minutes (torch.mm int64 at ~0.2 GFLOPS). The control plane pre-seeds it + once per shape (`w8a8_bench.py --prepare-reference`, 3600s budget) before + the timed benchmark; never re-run a correctness check without the reference + cache present, or the 900s subprocess budget is blown. + +## 9. End-to-end evidence + +TP4 test service with `num_hidden_layers=5`, chunked prefill size 3072, +16 prompts, 20k input, 64 output, max concurrency 16, warmup requests 0: + +| Metric | Original | HIP decode+prefill | Change | +|---|---:|---:|---:| +| duration | 79.03 s | 52.20 s | -33.9% | +| mean TTFT | 41.81 s | 27.09 s | -35.2% | +| mean TPOT | 590.3 ms | 398.1 ms | -32.6% | +| input throughput | 4049 tok/s | 6130 tok/s | 1.51x | + +This is a five-layer validation service, not full 43-layer model performance. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-prefill/SKILL.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-prefill/SKILL.md new file mode 100644 index 00000000..e6c02655 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-prefill/SKILL.md @@ -0,0 +1,769 @@ +--- +name: int8-w8a8-gemm-prefill +description: > + INT8 W8A8 GEMM prefill kernels for M>32 on Hygon K500SM_AI/gfx928: 2D M-tile + staged-LDS DUMMA kernels, no split-K, occupancy/LDS tuning, and the M=3072 / + M=4096 validated evidence. Load together with int8-w8a8-gemm-foundations + (contract, layout, DUMMA rules, Graph safety, benchmarking). For M<=32 use + int8-w8a8-gemm-decode. +--- + + + +> **Edition: LIQUID board (`liquid-k500sm`, K500SM_AI / DEV-HPC, HBM address +> hashing ON).** Every measurement in this edition was taken there. The air +> board (`K100_AI / DEV-AI`, no hashing) runs these operators 1.06-1.70x slower +> with a per-operator ratio, so **these numbers must not be used on the air +> board** and this edition is not installed on it -- the air machine gets +> `skills/air-k100ai/` instead. Kept for the liquid board and for provenance. +# INT8 W8A8 GEMM — Prefill (M > 32) — gfx928 / SGLang TP4 + TP8 + +Worker-29 lineage, DTK 26.04; K500SM_AI/gfx928, 120 CUs, wavefront 64, +64 KiB LDS/CU, 65,536 VGPRs/CU. Validated evidence covers M=3072 (TP4) and +M=4096 in three lineages: TP4 (`hy3_tp4_*_m4096`, §2), TP8 +(`hy3_tp8_*_m4096`, §3) and GLM-5.2 TP8 (`glm5-2-dsh-tp8-m4096-1`, §4). +M in (32, 3072) [TP4] / (32, 4096) [TP8 / GLM-5.2] are +measured-boundary gaps — reuse the staged recipes below and measure. This +skill is the prefill half of the phase split; load `int8-w8a8-gemm-foundations` +for the operator/layout contract, DUMMA and epilogue rules, Graph-safe +interface, SGLang integration, and shared guardrails. For each M=4096 lineage +the source of truth is its authoritative control-plane fact ledger (TP4: +2026-08-22 re-verified values; TP8: 2026-08-27; GLM-5.2: +`glm5-2-dsh-tp8-m4096-1-e6a280a2`): pending worker SKILL.md +tables may be stale and must not override the ledger. Baselines are fixed +user-supplied Triton CUDA-graph numbers that were never re-measured — do not +treat any baseline as measured or optimal; "speedup" is always vs that fixed +table. + +## 1. M = 3072 prefill (validated) + +For `M=3072,K=4096,N=1536`, use the validated large-M path: + +```text +block tile = 64x64 +K stage = 64 +threads = 256 = 4 wavefronts +wave ownership = one 32x32 quadrant per wave +DUMMA unit = m16n16k32 +LDS = two buffers of A[64,64] + B[64,64] = 16 KB total +prefetch = two 16-byte vectors/thread into VGPR +epilogue = direct fragment -> scale -> bf16 store +tail M = zero-filled A loads + masked stores +``` + +Launch grid: + +```text +grid.x = N / 64 +grid.y = ceil(M / 64) +``` + +Issue stage `i+1` global loads before the stage `i` MMAC sequence. Keep the +prefetched vectors in VGPR, execute the current eight MMAC instructions, wait +for VMEM immediately before writing the alternate LDS buffer, then use an +LDS-ready barrier. Do not infer this overlap from source order: require the +gfx928 ISA to show `global_load_dwordx4 -> v_mmac_i32_* -> vmcnt(0) -> +ds_write_b128` in that order. + +Current optimized prefill constraints are `M>32`, `K%64==0`, `N%64==0`, +contiguous inputs/scales/output, bf16 output, and no bias. + +Do not use split-K for large M unless profiling proves grid parallelism is +insufficient. MxN already supplies many blocks, and reduction overhead is +normally unnecessary. + +For the full M=3072 experiment table, ISA procedure, resource evidence, +bandwidth accounting, and rejected stage sizes, read +[references/wqkv-a-m3072-prefetch-isa.md](references/wqkv-a-m3072-prefetch-isa.md). + +## 2. M = 4096 chunked prefill — TP4 (hy3_tp4_*_m4096) + +Validated on the worker-29 lineage, one worker/GPU per shape ID, DTK 26.04. +Operator contract is identical to the foundations skill; weight +packing/preprocessing runs outside the timed region and outside Graph +capture. **Source of truth = the authoritative control-plane fact ledger** +(2026-08-22 re-verified values); pending worker SKILL.md tables may be stale +and must not override the ledger. + +| shape_id | M,N,K | Worker | Status | Accepted best (median / p90 us) | TOPS | vs baseline | +|---|---|---|---|---|---|---| +| hy3_tp4_qkv_proj_m4096 | 4096, 2560, 4096 | worker_0 | optimized (iter 18) | 756.73 / 760.16 | 113.51 | 31.78x (24045.12) | +| hy3_tp4_o_proj_m4096 | 4096, 4096, 2048 | worker_1 | optimized (iter 10) | 803.90 / 805.14 | 85.48 | 24.73x (19881.949) | +| hy3_tp4_shared_gate_up_proj_m4096 | 4096, 768, 4096 | worker_2 | optimized (iter 13) | 220.71 / 221.54 | 116.76 | 30.26x (6678.282) | +| hy3_tp4_shared_down_proj_m4096 | 4096, 4096, 384 | worker_3 | optimized (iter 14) | 214.84 / 215.12 | 59.97 | 20.39x (4381.289) | + +> The **2026-08-22 re-verified values**: each accepted kernel object was +> re-benchmarked with correctness ON (mismatch=0, max_abs_error=0.0) using the +> same protocol (100 warmup / 30 samples × 100 replays, CUDA-graph replay, hot +> cache). TOPS = 2MNK/median; bandwidth = algorithmic bytes / median (not +> measured HBM traffic). Speedup = fixed user-supplied Triton baseline / +> median. Iteration-chain values in §6 are the historical in-run measurements +> of the same accepted kernels. + +- All four shapes have accepted candidates: **no GPU worker is unavailable and + no shape in this lineage is unoptimized**. Rule for any future shape: a + shape whose worker timed out or produced no accepted candidate stays + **unoptimized** — never infer success, a speedup, or optimality from a + failed/timed-out worker, and never call its baseline optimal when no + candidate was accepted. +- M=4096 prefill: **no split-K** in any accepted candidate — the M×N + output-tile grid already dwarfs 120 CUs, and the 16 MiB workspace has no + useful int32 partial-plane capacity. +- Routing: exact `(m,n,k)` guard per shape → tuned DUMMA kernel; every other + shape falls back. Place the exact-shape guard before generic tiled paths and + keep it shape-exact so decode (M≤32), M=3072, and generic arms are + untouched. Each source carries a generic scalar int8/int32 fallback that + decodes the packed layout when the `(k,n)` matches: o_proj `(2048,4096)`, + qkv `(4096,2560)`, down_proj n-major, gate_up `(4096,768)`; gate_up also + keeps generic single-buffer tiled arms (`<64,128,128>`, `<128,64,128>`, + `<64,64,128>`) that were never re-tuned. Untuned paths must remain + byte-identical to the accepted source and must still build/pass. +- All speedups are vs the fixed user-supplied Triton CUDA-graph baseline table + above; those baselines were never re-measured and must not be called + optimal. +- Decode (M≤32), M=3072, and any other shape are outside this lineage and + must keep their own validated routes/fallbacks. + +## 3. M = 4096 chunked prefill — TP8 (hy3_tp8_*_m4096) + +Validated on the worker-29 lineage (hy3-dsh-tp8-m4096-1), one worker/GPU per +shape ID, DTK 26.04. **Source of truth = the authoritative control-plane fact +ledger (2026-08-27)**; pending worker SKILL.md tables may be stale and must +not override the ledger (e.g. down_proj's draft names round 8 as final while +the ledger also accepted rounds 10/12/13; gate_up's draft calls round 11 "not +accepted" though the ledger accepts it). All four TP8 M=4096 prefill shapes +have accepted candidates (4/4); no GPU worker is unavailable. ~30 rounds in +this lineage ended as `compile_or_agent_failure` (killed ~900 s, missing +proposal, restore/resume failures): these are candidate/infra failures and +prove nothing — never infer success, a speedup, or optimality from a +timed-out/failed worker, and a compile error is never evidence that DUMMA, +INT8, a HIP API, templates, or inline asm are unsupported. Recipes do NOT +transfer across TP size: the TP4 numbers in §2 are a different lineage with +different shapes. + +### 3.1 Routing and accepted evidence (authoritative ledger) + +| shape_id | M,N,K | accepted best (iter) | median / p90 us | logical TOPS | speedup vs baseline | +|---|---|---|---|---|---| +| hy3_tp8_qkv_proj_m4096 | 4096,1280,4096 | 6 | 418.22 / 431.55 | 102.7 | 46.34x (19381.383) | +| hy3_tp8_o_proj_m4096 | 4096,4096,1024 | 12 | 322.99 / 325.38 | 106.4 | 43.82x (14151.792) | +| hy3_tp8_shared_gate_up_proj_m4096 | 4096,384,4096 | 18 | 115.95 / 116.23 | 111.1 | 144.13x (16712.231) | +| hy3_tp8_shared_down_proj_m4096 | 4096,4096,192 | 10 | 170.80 / 171.03 | 37.7 | 63.01x (10762.616) | + +All four: correctness passed (mismatch 0 / max_abs_error 0.0), Graph capture ++ changed-contents replay passed. No proven optimum on any shape (no plateau +established): accepted numbers are the best found, not ceilings. No split-K +in any accepted candidate — the MxN tile grid dwarfs 120 CUs; the only +measured split (gate_up S=2) cost ~40-70 us of combine. + +### 3.2 TP8 pack layouts and fallback guards (lineage-specific) + +Foundations records the TP4 layouts; the TP8 layouts below are distinct and +shape-specific. Checkpoint weight is [N,K] (stride K); SGLang Triton view is +W.t() — preserve it for fallback and register one one-time contiguous HIP +buffer; never replace `layer.weight.data` with the contiguous copy. Packing +is one-time, out-of-timed-region, out-of-Graph, same byte count, same +graph-stable buffer; correctness is always checked against the raw logical +[K,N] weight: + +- qkv (k=4096,n=1280): n-major `packed[n][k]` +- o_proj (k=1024,n=4096): n-major `packed[n*K+kk] == raw[kk*N+n]` +- gate_up (k=4096,n=384): [N,K] transpose (exact (K,N)==(4096,384) only) +- down_proj (k=192,n=4096): n-major `packed[n][k]` + +Validate every pack kernel with a byte-exact simulation of the index +relation. o_proj iter 10's mis-vectorized transpose silently left most of +the buffer unwritten (mismatch 16,775,923 / max_abs_error 86.5): a fast but +wrong measurement is not a win; the iter-11 repair (per-thread 16-byte +gather from 16 raw rows + aligned int4 store) restored mismatch 0. + +Fallback routing (must stay byte-identical and building): exact-shape guard +first -> tuned DUMMA kernel; every other shape -> generic tiled arms / scalar +fallback (`w8a8_gemm_scalar_fallback_kernel`), which decodes the packed +layout when the (k,n) matches and is otherwise identity. Guards (unchanged +across the lineage): qkv `(m >= 128 && n == 1280 && k == 4096)`; o_proj +`(m,n,k) == (4096,4096,1024)`; gate_up `(m == 4096 && k == 4096 && n == 384)`; +down_proj `(m >= 128 && k == 192 && n % 128 == 0)`. M tails and paired shapes +must still pass. Occupancy: verify from the exact code object (vgpr_count, +sgpr, LDS bytes, scratch/spills); `__launch_bounds__` minBlocks is a hint +only. Math: blockDim multiple of 64; blocks x LDS <= 64 KiB; VGPR x threads x +blocks <= 65,536. + +### 3.3 Accepted architectures (per shape) + +- **qkv — `w8a8_dumma_prefill_packedb_tiled_kernel<64,128>` (iter 6)**: + 64x128 tile, 256 thr = 4 waves, 32x64 quadrant/wave = 8 m16n16k32 int32 + acc; grid (N/128)x(M/64) = 10x64 = 640 blocks; single-buffered 64-K LDS + stage (A[64,64] stride 68 B odd-word, B[128,64] n-major stride 80 B, + 14,592 B/block), 2 __syncthreads/stage, peeled final stage (127 + barriers/block); hoisted 12 fragment loads (4 ds_read2_b32 A + 4 + ds_read2_b64 B via load_b_frag8) then 16-MMAC burst; register-scale + coalesced epilogue (4x4 shuffle transpose, 2+4 scale loads prefetched, one + 8-byte store/lane/fragment); 2 blocks/CU, VGPR-bound (arch 96 / code-object + 89 / sgpr 26 / no spills). Chain: 448.86 (iter 1) -> 418.22 (iter 6). +- **o_proj — `w8a8_dumma_128x64x64_kernel` (iter 12)**: 128x64 tile, 256 thr + = 4 waves, 64x32 quadrant/wave = 8 acc; grid dim3(64,32) = 2048 blocks; + single-buffered 64-K stage A[128,80]+B[64,80] = 15,360 B, 2 + __syncthreads/stage (32/block); 3 int4 global loads/thread/stage -> + ds_write_b128 -> barrier -> 6 ds_read2_b64 -> 16 v_mmac; n-major [N,K] pack + (repaired iter-11 gather) + load_fragment8 on BOTH operands (one 8-byte LDS + read per fragment, zero reassembly VALU); coalesced 8-byte epilogue (iter 6: + 4x4 __shfl_xor transpose, ds_bpermute lowering accepted at the block tail); + 3 blocks/CU per the ledger (worker code-object re-verification: 61 VGPR/22 + SGPR/0 scratch — confirm residency from the exact object). Chain: 419.05 + (iter 6) -> 348.27 (iter 11) -> 322.99 (iter 12). +- **gate_up — `w8a8_gemm_prefill_tiled_kernel_w8<64,128,64>` (iter 18)**: + 64x128 tile, 512 thr = 8 waves, 32x32 quadrant/wave = 4 acc, 512 + v_mmac/wavefront; grid 3x64 = 192 blocks; K stage 64 double-buffered, one + __syncthreads/stage (65/block); B packed transposed [N,K] (exact + (K,N)==(4096,384) only), col_major fragments via load_fragment8 (one + contiguous 8-byte LDS read); LDS 30,720 B/block, 2 blocks/CU, code object + 53 VGPR/26 SGPR/0 scratch. Chain: 169.46 (iter 2) -> 118.47 (iter 9: w4, + 256 thr/4 waves/32x64 quadrant/8 acc — biggest single win) -> 117.50 + (iter 13, load-all-then-MMAC-all burst) -> 115.95 (iter 18, w8); iters + 11/14/15 also accepted (117.6-117.7). +- **down_proj — `w8a8_dumma_prefill_kernel<64,128,64>` (iter 10)**: 64x128 + tile, 512 thr = 8 waves, 32x32 quadrant/wave = 4 acc; grid (N/128, M/64) = + 2048 blocks; K=192 in 3 x kStageK=64 double-buffered stages, one + barrier/stage + prologue (4 dynamic; iter-12 variant drops the dead last + barrier -> 3); A stride 88 B (kLdsPad 24), B n-major packed stride 72 B + (kLdsPadB 8), ds_write2_b64 staging; 29,696 B/block, 2 blocks/CU = 16 + resident waves, 55-57 VGPR; load_frag8 (ds_read_b64, merged to 4 + ds_read2_b64/wave-stage) replacing du_load_matrix_sync; fused + direct-fragment epilogue with iter-10 register-batched weight_scale loads + (16 -> 8 scale loads/thread); left-associative ((float)acc * xs) * ws fp32, + __float2bfloat16. Chain: 237.78 (iter 1) -> 234.23 (iter 4) -> 230.91 + (iter 6) -> 171.19 (iter 8, load_frag8, -25.9%) -> 170.80 (iter 10); + iters 12/13 also accepted (170.93/170.94). + +### 3.4 Rejected ideas (measured, correct where noted; do not re-litigate) + +- qkv: 64x64 flip 509.14; 128x64 466.05; k8 swizzle 451.86 (neutral — + conflicts not binding); explicit load_a_frag8 561.88 (CONFOUNDED: VGPR + 96->78 silently moved 2->3 blocks/CU) and 896.63 (pinned — the library + loader's "dead" identity VALU is load-bearing latency fill; do NOT remove + A-side VALU here); 512-thread blocks 481.55; one-stage-ahead staging + prefetch 1097.63 (DTK scattered 34 vmcnt waits); double-buffer/1-barrier + 457.14; A stride 68->80 768.34 (odd 17-word stride load-bearing); B + st-pair merge 659.43; tail-guard consolidation 448.17. +- o_proj: iter-10 pack bug 327.36 (CORRECTNESS FAILED — never count a fast + wrong result); 128x128 854.05; 512 thr 491.51; B stride 80->72 719.73 / + A-stride 80->72 437.15 (conflict floor not on the critical path); epilogue + scale hoisting 741.15; double-buffer prefetch 891.29 (confounded: real vgpr + 85 + 30,720 B = 2 blocks/CU, 50% occupancy loss); prefetch + + __launch_bounds__(256,4) 659.5 (15 spills + 48-B scratch — + register-blocked in pure HIP); last-barrier drop 491.30 (flat/unstable + window). +- gate_up: BM=32 grid 768 586.23 (doubled B staging); 128x64 flip 181.44; + 64x64 3 blocks/CU 223.90; stage-top publish 181.21 and 121.40 at w4 + (+2.5%); split-K S=2 240.19; deeper-MMAC-pipe prediction REFUTED (halving + per-wavefront MMAC count gave only +2.17% — ~2-deep per-SIMD pipe). +- down_proj: 2-stage prefetch + 1 block/CU 617.07 (co-residency load-bearing); + staging thread remap 182.74 (-6.3%: 4x L2 requests); ws sharing 16->8 + 172.17 (p90 guard fail); scales staged in LDS 179.52 (-4.9%); split-3 + infeasible by arithmetic (LDS 89,088 B > 64 KiB, VGPR 86,016 > 65,536). +- Killed rounds (qkv 2,10,13,15,16,20; o_proj 2,9,13,15,22; gate_up 1,5,12, + 16,17; down_proj 2,5,7,11,15,17) prove nothing about the toolchain. + +### 3.5 Cross-shape rules that held (measured) + +1. Pack B once into the exact (k,n) layout (n-major / [N,K]) outside + timing+Graph; together with 8-byte fragment loads this is the dominant + lever on every shape. +2. `load_fragment8` (one ds_read2_b64 per fragment, identical x[0..7] bytes = + identical v_mmac operand) is the winning fragment-load form — EXCEPT qkv's + A operand, where the library row-major loader's identity VALU is + load-bearing latency fill (removal regressed even with residency pinned; + qkv's B-side load_b_frag8 is fine). +3. LDS bank skew: 16-byte-aligned non-power-of-two row strides (68/72/80/88); + strides ≡ 0 (mod 128 B) alias every row onto one bank phase. When + stride % 16 == 8, stage with two int64 halves (ds_write2_b64), not + ds_write_b128. +4. Residency is per-shape and load-bearing: qkv 2 blocks/CU (VGPR-bound), + o_proj 3-4, gate_up 2, down_proj 2 = 16 waves/CU. Verify the exact code + object before trusting any delta; occupancy confounds invalidate verdicts. +5. These kernels are LDS-latency/issue-bound, not HBM- or MMAC-bound (MMAC + pipe ~2-3%): bank-conflict elimination, barrier removal, deeper register + prefetch, and occupancy changes stop paying once strides and pack layout + are fixed. +6. Epilogue: coalesced 8-byte stores (4x fewer vmem_write) and scale-load + hoisting/register batching are the winning post-MMAC moves; staging scales + in LDS lost (down_proj -4.9%). +7. Bit-identical discipline: preserve k0-outer/kk-inner int32 order and the + element-to-slot fragment mapping; variants then pass mismatch 0 / + max_abs_error 0.0 with no tolerance debate. +8. Recipes do not transfer across M regimes or per-shape constants: tile + aspect (down_proj 64x128 wins at K=192 vs 128x64 at TP4 K=384), stage + size (64 saturates), and occupancy must be re-measured per exact shape. + +Acceptance for this lineage follows the shared protocol in §7 plus the +TP8-specific notes there. + +## 4. M = 4096 chunked prefill — GLM-5.2 TP8 (glm5-2-dsh-tp8-m4096-1) + +Synthesis of the `glm5-2-dsh-tp8-m4096-1-e6a280a2` lineage: 4 workers, 6 +exact-shape IDs, all INT8 W8A8 GEMM prefill (M=4096) on Hygon K500SM_AI / +gfx928 (120 CUs, wavefront 64, 64 KiB LDS/CU, 65,536 VGPR/CU, DTK 26.04). +**Source of truth = the authoritative control-plane fact ledger**; pending +worker SKILL.md tables may be stale and must not override it. All 6/6 exact +M=4096 shapes are optimized (accepted, correct, Graph-passed); no GPU worker +is unavailable (`{}`). Baselines are fixed user-supplied Triton CUDA-graph +numbers that were never re-measured — never call them optimal, and no accepted +candidate is a proven optimum (`plateau=false` everywhere: accepted numbers +are best-found, not ceilings). ~30 rounds across the lineage ended +`compile_or_agent_failure` (agent killed ~900 s, missing proposal, restore +failures): these prove nothing — a compile error is evidence only that that +candidate failed to compile, never that DUMMA, INT8, a HIP API, templates, or +inline asm are unsupported. `hipLaunchKernelGGL(HIP_KERNEL_NAME(templated<...>))` +is valid and used by every accepted kernel. + +### 4.1 Routing and fallback guards + +Exact-shape guard first -> tuned DUMMA kernel; every other `(m,n,k)` falls +through to generic tiled arms / scalar fallback. Guards (unchanged across the +lineage): fused_qkv_a `m >= 128 && n == 2624 && k == 6144`; kv_b +`k == 512 && n == 3584 && m >= 64 && m % 64 == 0`; q_b exact `(k,n) == +(2048,2048)`; o_proj exact `(m,n,k) == (4096,6144,2048)`; shared_down exact +`(m,n,k) == (4096,6144,256)`; shared_gate_up exact `(m,n,k) == +(4096,512,6144)`. Untuned arms (generic single-buffered `<64,128,128>` / +`<128,64,128>` / `<64,64,128>` tiled instantiations, pack kernels, scalar +fallback) must stay byte-identical and still build/pass; M tails and paired +API shapes keep working. New mechanisms are added as defaulted template +parameters behind `if constexpr` so retention instantiations are +token-identical. Operator contract: no bias, no workspace (`(void)workspace` — +no combine pass exists), **no split-K** (MxN tile grid dwarfs 120 CUs on every +shape; the only measured split in the TP8 family cost ~40-70 us of combine); +scale order left-associative `(float(dot) * x_scale[row]) * weight_scale[col]`, +then `__float2bfloat16` (plain RNE without exec-masked inf/NaN fixup is +accepted where bit-identical for finite inputs). + +### 4.2 Pack layouts (shape-specific, validated byte-exact) + +Checkpoint weight is `[N,K]` (stride K); the SGLang Triton view is `W.t()` — +preserve it for fallback, register one one-time contiguous HIP buffer, never +replace `layer.weight.data`. Every pack is one-time, out-of-timed, out-of-Graph, +same byte count, same graph-stable buffer, and must be validated by a +byte-exact simulation of the index relation before timing (a fast kernel +reading an incompletely written pack is a correctness failure, never a win; +the sibling lineage's mis-vectorized transpose left 16.7M elements unwritten). +The scalar fallback must decode the pack for the exact (k,n). + +| shape (K, N) | pack layout | +|---|---| +| fused_qkv_a (6144, 2624) | B-panel `packed[(n>>6)*(k*64) + (kk>>4)*1024 + (n&63)*16 + (kk&15)]` — 41 panels of [K,64], 16-B aligned contiguous k-runs/column | +| kv_b (512, 3584) | tile-contiguous stage-major `packed[(((k0*(n/128)+nt)*8+kc)*128+row)*8+b]` — each (stage, n-tile) B tile one contiguous 8192-B region | +| q_b (2048, 2048) | swizzled 64-k-stage-major plane `[kc][n][8]` — lane-linear 8-byte fragment reads, zero LDS bank conflicts | +| o_proj (2048, 6144) | n-major `packed[n*K+kk] == raw[kk*N+n]` | +| shared_down (256, 6144) | panel `[N/128][K/64][128][64]` (n-major 64-k-byte rows/panel) for fully coalesced 128-B staging lines | +| shared_gate_up (6144, 512) | `[K,N] -> [N,K]` n-major transpose | + +### 4.3 Accepted evidence (authoritative ledger; median / p90 us, CUDA-graph replay) + +| shape_id | M, N, K | accepted best (iter) | kernel | median / p90 us | logical TOPS | speedup vs fixed baseline | +|---|---|---|---|---|---|---| +| glm_tp8_fused_qkv_a_proj_m4096 | 4096, 2624, 6144 | 5 | `w8a8_dumma_prefill_packedb_db_kernel<128,64,64>` | 937.58 / 939.48 | 140.86 | 73.23x (68661.417) | +| glm_tp8_kv_b_proj_m4096 | 4096, 3584, 512 | 17 | `w8a8_dumma_prefill_tile_kernel<64,128,32,32,true,24>` | 137.40 / 137.85 | 109.40 | 50.51x (6939.939) | +| glm_tp8_q_b_proj_m4096 | 4096, 2048, 2048 | 7 | `w8a8_dumma_prefill_128x128_kernel` | 220.14 / 220.50 | 156.08 | 65.35x (14385.571) | +| glm_tp8_o_proj_m4096 | 4096, 6144, 2048 | 9 | `w8a8_dumma_256x64x64_packedb_kernel` | 759.92 / 760.81 | 135.64 | 72.08x (54775.797) | +| glm_tp8_shared_down_proj_m4096 | 4096, 6144, 256 | 5 (9 also accepted) | `w8a8_dumma_prefill_tiled_kernel<64,128,64,...>` | 278.44 / 279.19 | 46.28 | 63.57x (17700.262) | +| glm_tp8_shared_gate_up_proj_m4096 | 4096, 512, 6144 | 16 | `w8a8_dumma_prefill_tiled_kernel<64,64,64,true,true,true,true,true>` | 273.83 / 275.35 | 94.11 | 77.62x (21254.589) | + +All six: correctness passed (mismatch 0 / max_abs_error 0.0), Graph capture + +changed-contents replay passed. No split-K anywhere; exactly one kernel per +GEMM; workspace untouched. + +### 4.4 Accepted architectures (condensed recipes) + +- **fused_qkv_a (iter 5)**: 128x64 tile, 256 thr = 4 waves, 64x32 quadrant/ + wave = 8 int32 acc; grid 41x32 = 1312 blocks (M=4096=32x128, N=2624=41x64 + exact fit — no tail); 96 ascending 64-K stages, double-buffered LDS + `a_tile[2][128*72] + b_tile[2][64*72]` = 27,648 B/block -> **2 blocks/CU = + 8 waves/CU**; 93 VGPR / 31 SGPR / 0 spills; stage-(s+1) loads prefetched + into 3 int4 staging VGPR at stage top, flush after MMAC burst, ONE + barrier/stage; B-panel pack. Chain: 5697.3 (iter 1) -> 2276.9 (iter 2, + n-major pack) -> 1070.2 (iter 3, double buffer) -> **937.6** (iter 5, + B-panel pack). +- **kv_b (iter 17)**: 64x128 tile, 512 thr = 8 waves, 32x32 quadrant/wave = + 4 int32 acc; grid 28x64 = 1792 blocks; K=512 = 8 x 64-K stages, + single-buffered (two barriers/stage); A stride 88 (kALdsPad=24, zero- + conflict reads) + B tile-contiguous stage-major, ds_write2_b64 staging; + LDS 13,824 B/block -> **2 blocks/CU = 16 waves/CU**, 61 VGPR; + `kPrefetchNext` issues stage-s+1 grouped A+B loads AFTER the s MMAC burst, + before the trailing barrier (8-VGPR payload live only across barrier + + back-edge — never carry staging payloads across the burst). Chain: + 209.41 -> 169.26 (128x64, 3 blocks/CU) -> 149.54 (64x128 flip, 16 waves/CU) + -> 146.54 (tile-contiguous pack) -> 142.70 (plain RNE bf16) -> 139.57 + (grouped A+B staging) -> **137.40** (consolidation of prefetch + A skew). +- **q_b (iter 7)**: 128x128 tile, 256 thr = 4 waves, 64x64 quadrant/wave = + 16 int32 acc; grid 16x32 = 512 blocks; single-buffered 64-K stage + A[128,80] + B plane `[kc][n][8]` = 18,432 B/block -> **1 block/CU = 4 + waves/CU** (best occupancy on this shape; more resident waves regressed); + 141 VGPR; `load_frag8` on BOTH operands (iter 7, -11%); register-batched + coalesced epilogue (iter 6, -9.1%). Chain: 310.22 (n-major pack) -> 275.79 + (128x128) -> 272.06 (swizzled plane) -> 247.28 (epilogue) -> **220.14** + (A load_frag8). +- **o_proj (iter 9)**: 256x64 tile, 256 thr = 4 waves, 128x32 quadrant/wave = + 16 int32 acc; grid 96x16 = 1536 blocks; single-buffered 64-K stage + A[256,88] + B[64,72] = 27,136 B/block -> **2 blocks/CU = 8 waves/CU**; + 151 VGPR / 34 SGPR; 5 `global_load_dwordx4` -> 5 `ds_write2_b64` (strides + 8 mod 16) -> 10 `ds_read2_b64` -> 32 v_mmac per wave/stage; n-major pack + + `load_frag8` B; register-batched scales epilogue (iter 6, +9.0%). Chain: + 1149.26 (128x64 DB) -> 848.53 (256x64, +35.4% biggest win) -> 839.96 (pack + + load_frag8) -> 770.49 (epilogue batching) -> **759.92** (A stride 88 + conflict-free, +1.4%). +- **shared_down (iter 5; iter 9 also accepted)**: 64x128 tile, 512 thr = 8 + waves, 32x32 quadrant/wave = 4 int32 acc; grid 48x64 = 3072 blocks; K=256 = + 4 x 64-K stages single-buffered (2 barriers/stage) — single buffer beat + double buffer on this K=256 shape (~1.7%); LDS A[64,88] + B[128,72] = + 14,848 B/block -> 2 blocks/CU; panel pack (fully coalesced 128-B staging + lines) + load_frag8 + kSkew8 strides. Iter 9 adds kFragReuse + + `__launch_bounds__(512,3)` (3 blocks/CU = 24 waves/CU at 42 VGPR) — + essentially tied (278.52/279.11), accepted per ledger. Chain: 286.53 -> + 281.65 -> **278.44**. +- **shared_gate_up (iter 16)**: 64x64 tile, 256 thr = 4 waves, 32x32 + quadrant/wave = 4 int32 acc; grid 8x64 = 512 blocks; K=6144 = 96 x 64-K + stages double-buffered, ONE barrier/stage (97/block); LDS A[64,88] + + B[64,72] x2 = 20,480 B/block -> **3 blocks/CU = 12 waves/CU**, 54 VGPR; + `kPayload` lead-2 loop-carried payload: stage s+2 global loads issued at + the top of iteration s (no wait), payload published into the idle LDS + buffer before the stage-s MMAC burst, vmcnt wait lands AFTER the 8 x v_mmac + burst immediately before the barrier (removes the exposed prefetch wait); + `load_frag8` (iter 10, -9.1%, body 175->118 instr) + kSkew8 strides + (iter 11, conflicts 9.44M->3.15M). Chain: 432.52 -> 368.76 -> 331.64 + (double buffer) -> 301.57 (load_frag8) -> 293.30 (skew) -> **273.83** + (payload). + +### 4.5 Rejected ideas (measured; do not re-litigate without new evidence) + +- **fused_qkv_a**: `<128,128,64>` square tile, 1 block/CU 3491.7 (occupancy + halving dominates); branch-free bf16 RNE epilogue 2554.8 (epilogue NOT the + bottleneck — "ISA-neutral by construction" must be measured); prefetch + distance 2 via two staging sets 4992.1 (register blowup); BM=64, 3 blocks/CU + 1148.0; StageK 128 / 512 thr / 1 block/CU 1055.5; rasterization flip + `dim3(32,41)` 3354.3; bottom-of-stage prefetch issue 3233.0; load-all-then- + MMAC-all 938.8 (neutral, p90 guard fail). +- **kv_b**: double-buffered one-barrier stage 197.24 (28,672 B -> 2 blocks/CU + occupancy loss dominates VMEM-stall removal); double-buffer pipelines with + the 8-VGPR payload carried across the burst (72/72/68 VGPR -> 1 block/CU) + 153-156; 64x64 tile + minBlocks=5 probe (vgpr 48 honored, 20 waves/CU) + 491.10; ISA-guided dead last-stage barrier removal 137.87 (p90 guard fail). +- **q_b**: 512-thread w8 variant 256.09 (+50% per-stage LDS read bytes); + `#pragma unroll 1` -> 2 blocks/CU 266.57 (occupancy REFUTED on this shape — + 1 block/CU = 4 waves/CU stays best); loop-carried payload + stage-top + publish 221.57 (+0.65%, p90 fail — VMEM exposure refuted at 1 wave/SIMD); + A stride 88 skew 223.73 (conflict floor not on the critical path here); + 128-K stage depth 357.77 (stage 64 saturates). +- **o_proj**: 64x128 tile 1256.05 (per-byte B ds_read_u8 class); direct-A from + global 1715.91 (cooperative A+B staging load-bearing); one-stage-ahead + prefetch 786.08 (-2.0%, falsified — VMEM latency already hidden by + co-residency); load-all-then-MMAC-all 760.88 (flat, DTK already hoists); + wave-private staging + barrier removal 973.16 (-21.9%, 2x staging traffic); + disjoint 64x64 quadrants 796.73 (-4.6% despite -20% LDS fragment-read + traffic — LDS-read-pipe hypothesis falsified). +- **shared_down**: 64x64 tile 297.15 (+33% A staging traffic); kRNEEpilogue + branch-free RNE 320.88 (+15.2% — epilogue VALU not binding); kSPayload + register payload 281.19 (p90 fail — load-wait relocation alone does not pay + at K=256); kStageK=128 279.61; kBatchEpilogue scale hoisting 284.61; + kLoadPair back-to-back staging loads 281.24. +- **shared_gate_up**: 32x64 tile / 2-wave blocks 749.87 (confounded, wave + count per CU load-bearing); q64 128-thr 32x64 quadrant 629.02 (+90%); + fragment-prefetch pipeline (persistent fragments across the K loop) 337.97; + 64x128 w8 / 512 thr 395.39 (-16.1%, loses grid blocks and tail); kStageK=32 + / 6 blocks/CU 447.37 (-34.4% — more occupancy with phase-locked co-resident + blocks does not pay). +- Killed rounds (fused_qkv_a 9,10,12,15,17,18; kv_b 11,14,15,18,19,21; q_b + 1,2,12-15,17; o_proj 7,11,13,14; shared_down 1,7; shared_gate_up 5,9,13,14, + 15,17) are infra/agent failures — no evidence about hardware, APIs, or + candidate ideas. + +### 4.6 Cross-shape rules that held (measured) + +1. **Per-shape B packing once, outside timing+Graph, into the shape-exact + layout** is the dominant lever on every shape; together with 8-byte + fragment loads it removes the per-byte LDS read + reassembly VALU class. + Validate packs byte-exact; the scalar fallback must decode the pack for + the exact (k,n). +2. **`load_frag8` wins** (identical operand bytes -> bit-identical int32) + except where the library A-side identity VALU is load-bearing latency fill + (verify per shape). +3. **LDS bank skew**: 16-byte-aligned non-power-of-two row strides + (68/72/80/88/136); `stride % 16 == 8` -> `ds_write2_b64` halves. Fix + conflicts only when they sit on the critical path (payoff is per-shape: + +0.6% kv_b, -1.6% q_b, +1.4% o_proj). +4. **64-K stage depth saturates** (128-K lost on q_b 357.77 and fused_qkv_a + 1055.5; 32-K lost on gate_up 447.37). Double buffering wins only when + residency is preserved (fused_qkv_a, gate_up); it lost whenever it cost + blocks/CU (kv_b, down_proj). +5. **Residency is per-shape and load-bearing**: fused_qkv_a/o_proj 2 + blocks/CU = 8 waves/CU; kv_b/down_proj 2 blocks/CU = 16 waves/CU; gate_up + 3 blocks/CU = 12 waves/CU; q_b 1 block/CU = 4 waves/CU. Verify from the + exact code object; occupancy confounds invalidate verdicts. +6. These kernels are **LDS-latency/issue-bound, not HBM- or MMAC-bound** + (MMAC pipe ~2-3%). Once pack layout + strides + pipeline are fixed, + barrier removal, occupancy probes, epilogue-scale hoisting and load-pair + micro-scheduling stop paying. +7. **No split-K** for M=4096; launcher is a pure dispatch on the caller + stream; packing/scales stay out of timing and Graph. +8. Recipes/constants do NOT transfer across shapes or M regimes: tile aspect + (down_proj 64x128 at K=256 vs q_b 128x128 at K=2048 vs o_proj 256x64 at + N=6144), occupancy, stage depth and pipeline form must be re-measured per + exact (M,N,K). + +### 4.7 Boundary gaps + +- 32 < M < 4096: **no accepted candidate measured** in this lineage. Reuse + the staged recipes above and measure per exact M before committing a route; + keep both a staged prefill arm and a decode-compatible arm until measured. +- M <= 32 (decode): **UNOPTIMIZED in this lineage** — do not claim any decode + speedup from this lineage, never infer success from the prefill evidence, + and route M <= 32 work to `int8-w8a8-gemm-decode` + + `int8-w8a8-gemm-foundations` (split-K partial kernels + fused combine, + µs-scale noise-tolerant acceptance). Decode conclusions do not cross M + regimes (the M=16 direct-global-operand win regressed 7.9x on M=4096 + gate_up in the sibling lineage). +- No plateau proven on any of the six shapes; continue per-shape HIP search + with the §7 protocol (one bounded mechanism per round, falsifiable + prediction, p90 guard vs current best, exact code-object verification). + Next candidates: fused_qkv_a's p90-failed 938.84 us load-all-then-MMAC-all + variant and kv_b's shadow mechanisms (stride skew / prefetch, ~138.2-138.8 + us) are within ~1% of the accepted best. Revalidate on the full model / + 8-card deployment before claiming production gains; never present the fixed + Triton baselines as measured or optimal. + +## 5. Boundary: 32 < M < 3072 (TP4) / 32 < M < 4096 (TP8 / GLM-5.2) + +No accepted candidate has been measured in (32, 3072) for TP4 or (32, 4096) +for TP8 or the GLM-5.2 lineage. Reuse the staged recipes above (2D M-tile, +64-K double-buffered stage, no split-K) and measure per exact M before +committing a route. Keep the M×N output-tile grid well above 120 CUs; if it +is not, split-K may become necessary (measure it — do not assume either way). +TP8's optimized evidence is exactly M=4096 (hy3 4/4 shapes, §3; GLM-5.2 6/6 +shapes, §4); do not extend those accepted numbers to other M values. + +## 6. Validated evidence (TP4) + +### M=3072 prefill microbenchmark + +Exact bf16 comparison passed for all five TP4 shapes, using the real +transposed Triton weight view and a preprocessed contiguous HIP weight: + +| Case | Triton ms | HIP ms | Speedup | +|---|---|---:|---:|---:| +| wqkv_a | 9.487898 | 0.857572 | 11.06x | +| wq_b | 15.437104 | 1.149457 | 13.43x | +| wo_b | 14.680174 | 1.102976 | 13.31x | +| shared gate_up_proj | 5.985047 | 0.623089 | 9.61x | +| shared down_proj | 4.198581 | 0.303484 | 13.83x | + +The very large gain is layout-sensitive. It must not be claimed for a Triton +baseline already using an equally optimized packed layout. + +The later wqkv_a ISA retune supersedes the `0.857572 ms` HIP row above for the +exact shape `(3072,1536,4096)`: the selected stage64 double-buffer prefetch +kernel measured `0.598983 ms` median and `0.605783 ms` P90 in an alternating +thermal-state trial, or `64.534 INT8 TOPS`, with exact bf16 output. A sustained +100-warmup/300-sample run measured `0.615996 ms` median and `0.621276 ms` P90, +showing device frequency/thermal drift that must be reported rather than +hidden by the best short run. + +### M=4096 chunked prefill (hy3_tp4_*_m4096) + +Accepted candidates per shape; baselines are the fixed user-supplied Triton +table in §2. Numbers below are the **2026-08-22 re-verified values**; the +iteration chains are the historical in-run measurements of the same accepted +kernels. + +**hy3_tp4_qkv_proj_m4096 (worker_0)** — final accepted iter 18 (kernel +`w8a8_dumma_prefill_64x128_packedb_kernel`): 64x128 tile, 256 threads = 4 +waves, 32x64 quadrant/wave = 8 m16n16k32 int32 accumulators; grid +(N/128)x(M/64) = 20x64 = 1280 blocks; single-buffered 64-K stage with 2 +`__syncthreads`/stage; A staged row-major stride 68 B, B staged n-major stride +80 B; LDS 14,592 B/block, **3 blocks/CU** (arch_vgpr 80); one-time n-major +pack `packed[n][k]` for (k,n)==(4096,2560) outside the timed region; coalesced +4-bf16-per-lane epilogue stores. Accepted chain (us): 1296.50 (iter 1) → +1108.58 (iter 2, double-buffered 64-K) → 880.99 (iter 3, single-buffer +control) → 770.14 (iter 5, n-major packed B + 3 blocks/CU) → 721.87 (iter 6, +coalesced epilogue stores) → **756.73** (iter 18, explicit A-fragment loader +replacing the library `du_load_matrix_sync` row_major byte-reassembly VALU). +Plateau NOT proven (iter 19's reason: require 8 valid HIP rounds within ±2% of +best — continue HIP-only work). + +**hy3_tp4_o_proj_m4096 (worker_1)** — final accepted iter 10 (kernel +`w8a8_dumma_128x64x64_packed_kernel`): weight packed once into `[N/64, K, 64]` +int8 panels (each 64-K stage = one coalesced 4 KiB stream); 128x64 tile, 256 +threads = 4 waves, 64x32 quadrant/wave = eight m16n16k32 int32 accumulators; +grid dim3(64,32) = 2048 blocks; 64-K double-buffered LDS, A and B both +80-byte row strides (16-byte-aligned, five bank phases), 30,720 B/block, 2 +blocks/CU; loop-carried int4 register payload published into the idle buffer +at the **top** of the stage (before the MMAC burst) with global prefetch one +full extra stage ahead; one `__syncthreads`/stage; fused fragment→scale→bf16 +epilogue. Accepted chain (us): 981.53 (iter 1) → 834.14 (iter 3, A LDS stride +64→80; PMC: conflicts 77.6M→31.5M, 8-way→2-way) → **803.90** (iter 10, +stage-top publish). + +**hy3_tp4_shared_gate_up_proj_m4096 (worker_2)** — final accepted iter 13 +(`w8a8_gemm_prefill_tiled_kernel<64,128,64,2>`): 64x128 tile, 512 threads = 8 +waves, 32x32 quadrant/wave = 4 accumulators; grid dim3(6,64) = 384 blocks; +K stage 64 double-buffered, one barrier/stage (65/block), 30,720 B LDS, 2 +blocks/CU; `pack_weight` transposes [K,N]→[N,K] for this shape only (outside +timed region); B staged [N,K], `col_major` fragments; iter 13's +`load_fragment8` writes the 8 consecutive bytes straight into fragment +storage (one ds_read2_b64 per fragment), removing the per-byte mask/OR +reassembly (VALU 16.08M→6.25M). Accepted chain (us): 436.21 (iter 1) → +392.58 (iter 3, double-buffered K) → 280.05 (iter 10, [N,K] pack + col_major) +→ **220.71** (iter 13, load_fragment8). Bottleneck chain was LDS-wait-bound +throughout: bank-skew → software pipeline → B-fragment aliasing → +byte-reassembly VALU. + +**hy3_tp4_shared_down_proj_m4096 (worker_3)** — final accepted iter 14 +(`w8a8_dumma_prefill_kernel<128,64,64>`): 128x64 tile, 512 threads = 8 waves, +32x32 quadrant/wave; grid (N/64, M/128) = 2048 blocks; K stage 64 +double-buffered, 29,696 B/block (A stride 80, B stride 72), **2 blocks/CU at +57 VGPR — first clean 2-block residency at constant tile** (VGPR-bound; +`__launch_bounds__` minBlocks was ignored at 74 VGPR); iter 14 packs weight +**n-major `packed[n][k]`** so B staging is 2× ds_write_b64/thread (was 16× +ds_write_b8 + ~12 VALU extracts); kk body 2×-unrolled with two independent +fragment sets, all 8 ds_read2 before the 8 v_mmac. Accepted chain (us): +266.74 → 259.23 (double-buffer + VGPR prefetch) → 255.46 (n-major B in LDS + +col_major) → 245.29 (stage 32→64) → 236.39 (unrolled kk) → **214.84** (n-major +pack). This lineage ended `plateau=false` — the accepted 214.84 us is the best +found, not a proven optimum. + +## 7. Prefill acceptance protocol (M=4096 lineages) + +Unprofiled CUDA-graph replay, 100 warmup / 30 samples × 100 replays, hot +cache; report median AND p90 (p90 guard vs current best), never min; compare +Graph vs Graph only; re-verified runs must have correctness ON (mismatch=0, +max_abs_error=0.0). PMC (`hipprof --pmc --pmc-type 3`) explains results after +acceptance and must move the predicted counters, else the mechanism is +ambiguous. Report thermal/frequency drift explicitly (see the M=3072 +wqkv_a retune) rather than hiding it with the best short run. + +Additionally for M=4096: + +- Exact bf16 vs reference (mismatch 0); preserve the exact int32 accumulation + order (k0-outer, kk-inner) and the element-to-slot fragment mapping so + variants are bit-identical. +- Graph capture + changed input contents + replay must pass; after any change, + re-capture the Graph with changed input contents and replay again. A Graph + replay may update contents, never addresses/shapes. M tails and the scalar + fallback must still pass. +- Verify resources from the exact code object + launch record (arch_vgpr vs + code-object vgpr_count, LDS bytes, scratch/spills, grid/workgroup); + `__launch_bounds__` minBlocks is a hint and can be ignored (down_proj + iter 7). Occupancy math: blockDim multiple of 64; 2-3 blocks × LDS ≤ 64 KiB; + VGPR × threads × blocks ≤ 512 KiB. +- Use the pre-seeded CPU int64 exact reference (it is slow); never run + correctness without the reference cache present. +- Report thermal/frequency drift (max samples ~1.5x median are common; + implied clock ~1.45 GHz) instead of hiding it with the best short run. +- `lds_wait ~= lds_instructions` alone is not a bottleneck signal (true in + accepted kernels too); profiled time is never the score — PMC must move the + predicted counters. + +TP8-specific notes (hy3_tp8_*_m4096 and GLM-5.2 glm_tp8_*_m4096): + +- Restore the accepted source to its recorded digest before the next + experiment; make ONE bounded mechanism per round with a falsifiable + prediction (expected direction + expected PMC/ISA deltas; a >2% regression + falsifies the mechanism). +- Verify the exact code object of the timed run (digest-matched, never a + stale sibling symbol); audit the exact template symbol for load -> MMAC -> + wait -> LDS-store order, VGPR/LDS/scratch/spills, no removed required + loads, no spilled prefetches. `__launch_bounds__` minBlocks is a hint, not + a contract. +- Report thermal/frequency drift: max samples ~1.5-2.1x median are routine + (GLM-5.2 lineage), not just the ~1.5x seen on TP4. +- Continue per-shape HIP search — no TP8 shape has a proven optimum (qkv's + plateau rule of 8 valid HIP rounds within ±2% was never met; down_proj's + best is best-found; all six GLM-5.2 shapes are `plateau=false`). + +## 8. Rejected / guardrail evidence (TP4 prefill) + +TP8 M=4096 rejected ideas live in §3.4. TP4 rejected evidence: + +**M=4096 (hy3_tp4_*_m4096) rejected ideas** (measured, correct where noted — +do not repeat without new evidence): + +- qkv: 128x64 tile flip (1152.10); one-stage-earlier global prefetch (725.77, + −0.54% — global-load latency already hidden by co-residency); minBlocks=4 / + 72→64 VGPR squeeze (749.80 — 4 spill slots, confounded); 512-thread 8-wave + blocks (733.01, −1.52%); B stride 80→72 conflict fix (752.79 — B conflicts + sit at the ~4-cycle 32-bank pigeonhole floor); A stride 68→80 (732.59); + sync-group retile (920.53); iter 19 accepted-build variant (713.84, p90 + guard failed). +- o_proj: 128x128 tile/1 block per CU (840.44, loses 2nd resident block); + 64x128 tile flip (959.38); register-rotated kk fragment prefetch (845.00); + kStageK 64→32 / 3 blocks per CU (866.60); four-A-fragment WAR removal + (830.13, flat); per-wave staging with zero barriers (967.17 — duplicated + staging traffic swamps barrier savings); dual loop-carried payloads / + two-stage lead (827.76, flat); eight A fragments at stage top (872.61). +- gate_up: direct-A arm with B-only staging (3444.26 — 7.9x slower; the + M=16 decode kernel's direct-B-from-global win does **not** transfer across M + regimes); 128x128 tile / 1024 threads (519.86, 5 VGPR spill slots + + 192-block grid); kStage 64→32 / 3 blocks per CU twice (407.85, 413.60 — + more occupancy with phase-locked co-resident blocks does not pay). +- down_proj: K stage 64→128 (363.76 — stage-depth axis saturates at 64); + 64x64 active tile occupancy probe (270.34 — confounded: shrunk tile doubles + per-output B staging); B-staging "register diet" (304.22 — flat batched + staging issue order is load-bearing; the restructure caused scratch spills). +- Common pattern: these kernels are LDS-latency/issue-bound, not HBM-bound; + barrier removal, occupancy changes, and deeper register prefetch all failed + once LDS strides and pack layout were fixed. Do not re-litigate without + profiling the exact shape. +- Killed rounds produced no candidate (`compile_or_agent_failure`: agent + killed ~900 s, missing proposal.json, restore/resume failures): qkv + 8,9,10,15,17,20; o_proj 2,5,9,11,12,13,18,19; gate_up 5-7,9,11,14,15; + down_proj 3,4,6,8,13,15. These are **infra/agent failures and prove + nothing** — not that DUMMA, INT8, a HIP API, templates, or inline asm are + unsupported. After a killed round, restore the accepted source to its + recorded digest before the next experiment. +- **LDS bank-skew rule**: pad row strides to 16-byte-aligned non-power-of-two + values (e.g., 80 = 64+16, 72 = 64+8, 68 = 64+4, 144 = 128+16); strides ≡ 0 + (mod 128 B) alias every row onto one bank phase (up to 8-16-way fragment + conflicts). +- For compute prefill, report unique API bytes and tiled global requests + separately. The M=3072 winner has only about 47.30 GB/s of unique API bytes + but about 1.024 TB/s of tiled A/B+output requests; cache can serve repeated + tile traffic, so neither number alone is physical HBM bandwidth. +- Do not assume a deeper Graph helps large-M compute. For the M=3072 winner, + one node beat 4/8/16-node cold-ring Graphs because launch overhead was + already negligible relative to the roughly 0.6 ms kernel. +- M=4096 baselines are the fixed user-supplied Triton table values; they were + not re-measured and must not be called optimal. All four shapes now have + accepted candidates; a future shape whose worker timed out or produced no + accepted candidate stays unoptimized — never infer a speedup or optimality + for it. + +## 9. Next optimization questions (prefill) + +- Prefill shapes other than wqkv_a: repeat the stage32/64/128 and single/double + LDS search; do not copy the wqkv_a stage64 dispatch without measurement. +- Prefill tile search: compare 64x64 against larger/smaller tiles while + preserving enough grid blocks and auditing VGPR/LDS/scratch in the exact + code object. +- Measure hot and production-like cache behavior separately. +- Revalidate on the full model/8-card deployment before claiming production + DeepSeek-V4 gains. +- M=4096 per-shape search should continue: **no proven optimum** in either + lineage — the control plane ended `plateau=false` for TP4 qkv and + down_proj, o_proj/gate_up explicitly disclaim optimality, and no TP8 shape + established a plateau. Accepted numbers are the best found, not proven + ceilings. For TP4 qkv, iter 19's plateau rule (8 valid HIP rounds within + ±2% of best) was not met — continue HIP-only work on that shape. TP4 + down_proj's 214.84 us is the best found, not a proven optimum; TP8 qkv + (iter 6), o_proj (iter 12), gate_up (iter 18), down_proj (iter 10) are + likewise best-found. +- TP8 (hy3_tp8_*_m4096): continue per-shape HIP search with the §7 protocol + (one bounded mechanism per round, falsifiable prediction); boundary + 32 < M < 4096: reuse the staged recipes and measure per exact M before + committing a route. +- GLM-5.2 TP8 M=4096 (glm5-2-dsh-tp8-m4096-1, §4): continue per-shape HIP + search on all six shapes with the §7 protocol (no plateau proven) — e.g. + fused_qkv_a's p90-failed 938.84 us load-all-then-MMAC-all variant and + kv_b's shadow mechanisms (stride skew / prefetch, ~138.2-138.8 us) are + within ~1% of the accepted best. GLM-5.2 decode (M <= 32) is unoptimized in + that lineage: run the decode skill's split-K + fused-combine family on the + GLM-5.2 TP8 shapes. Revalidate on the full model / 8-card deployment before + claiming production gains. +- M in (32, 3072): reuse the staged recipes and measure (see §5). diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-prefill/references/wqkv-a-m3072-prefetch-isa.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-prefill/references/wqkv-a-m3072-prefetch-isa.md new file mode 100644 index 00000000..d4c75624 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-gemm-prefill/references/wqkv-a-m3072-prefetch-isa.md @@ -0,0 +1,144 @@ +# wqkv_a M=3072: DUMMA prefetch and ISA evidence + +## Contents + +- [Scope and result](#scope-and-result) +- [Selected pipeline](#selected-pipeline) +- [Why stage64 won](#why-stage64-won) +- [ISA audit](#isa-audit) +- [Bandwidth accounting](#bandwidth-accounting) +- [Reproduction and artifacts](#reproduction-and-artifacts) + +## Scope and result + +Validated on worker29, K500SM_AI/gfx928, DTK 26.04: + +```text +M=3072, K=4096, N=1536 +int8 A x int8 B -> int32 -> x_scale * weight_scale -> bf16 +``` + +Every accepted candidate matched the full bf16 torch float32-GEMM reference +exactly and passed HIP Graph replay. The selected result was: + +```text +tile 64x64 +workgroup 256 threads / four wave64 +stage-K 64 +LDS 16 KiB, two alternating 8 KiB buffers +VGPR / SGPR 63 / 43 +scratch / spills 0 / 0 +median / P90 598.983 / 605.783 us +INT8 throughput 64.534 TOPS +``` + +A longer 100-warmup/300-sample run measured 615.996 us median and 621.276 us +P90. Preserve both the alternating comparison and sustained result; the gap +is thermal/frequency drift, not a correctness change. + +## Selected pipeline + +Each block owns a 64x64 output tile. Four waves each own one 32x32 quadrant, +represented by four m16n16 accumulators. For every stage: + +```text +issue next A/B global_load_dwordx4 into VGPR +load current A/B fragments from current LDS buffer +execute eight v_mmac_i32_16x16x32_i8 per wave +s_waitcnt vmcnt(0) +write prefetched int4 vectors to alternate LDS with ds_write_b128 +s_waitcnt lgkmcnt(0) +s_barrier +swap current/alternate LDS buffers +``` + +Each thread carries two 16-byte next-stage vectors for stage64. Keep those +registers live only across the current MMAC group; extending their lifetime +raises VGPR pressure without improving overlap. + +## Why stage64 won + +| Candidate | Median us | P90 us | INT8 TOPS | Diagnosis | +|---|---:|---:|---:|---| +| original stage128, single LDS | 841.748 | 844.148 | 45.922 | no VMEM/MMAC overlap | +| double LDS stage32 | 819.212 | 832.112 | 47.185 | twice as many stage barriers | +| double LDS stage64 | 602.054 | 610.554 | 64.205 | best latency/occupancy balance | +| double LDS stage128 | 883.311 | 885.871 | 43.761 | 85 VGPR + 32 KiB LDS pressure | +| stage64 + raw LDS-ready barrier | 598.983 | 605.783 | 64.534 | selected | + +Do not infer that shallower stages are always better. stage32 reduced LDS and +prefetch registers but doubled synchronization frequency. stage128 doubled +work per barrier but expanded LDS and live prefetch state enough to regress. + +One-node Graph was best. Ordinary stage64 measured approximately +603.035/610.055/613.035/614.690 us per call at Graph depths 1/4/8/16. + +## ISA audit + +Use the code object extracted from the same JIT extension that produced the +timing. Match the exact symbol containing `ILi64ELb1E`; `Li64` is stage64 and +`Lb1` is the accepted raw-ready-barrier specialization. + +Required steady-state evidence: + +1. A next-stage `global_load_dwordx4` appears before the current MMAC group. +2. Eight `v_mmac_i32_16x16x32_i8` sites appear before the load's + `s_waitcnt vmcnt(0)`. +3. `ds_write_b128` consumes the prefetched VGPR tuple after the VMEM wait. +4. `s_waitcnt lgkmcnt(0); s_barrier` protects the LDS-ready boundary. +5. Metadata reports 63 VGPR, 43 SGPR, 16,384 bytes LDS, zero private segment, + and zero spills. + +Reject source-only claims. In the pure-memory precursor, LLVM commoned two +nominal A reads; timing looked faster, but ISA showed only five loads instead +of the required six. An opaque VGPR index was needed to retain the second +ordinary GLOBAL load. The general rule is to count required memory sites and +inspect their cache modifiers before accepting bytes/time. + +Use raw asm only after proving a compiler limitation or synchronization +redundancy. Do not hand-write raw global memory or raw MMAC here: tuple +constraints, EXEC/VCC/SCC effects, cache flags, and fragment lane ownership +must be independently verified. + +## Bandwidth accounting + +Keep these metrics separate: + +```text +unique API bytes/call 28,329,984 +tiled A/B requests 603,979,776 +output stores 9,437,184 +tiled operand+output request bytes 613,416,960 +``` + +At 598.983 us, unique API bytes give about 47.30 GB/s, while tiled +operand+output requests give about 1.024 TB/s. The first reflects useful tensor +footprint; the second shows that the global request pipeline remains busy +while MMAC executes. Repeated tile requests can hit cache, so neither is a +physical-HBM claim without PMC plus a cache-cold protocol. + +The pure-memory M=3072 proxy reached 805.651 GB/s using 56,659,968 explicit +read+write bytes per call and a 16-node cold-ring Graph. Do not compare that +copy-program metric directly with real GEMM's unique-byte metric. + +## Reproduction and artifacts + +```bash +cd /workspace/ISA_test_codex/wqkv_a_compute_tuning +HIP_VISIBLE_DEVICES=0 python benchmark_prefill_m3072.py \ + --stage-k=65 --warmups=50 --samples=150 --nodes=1 --check +``` + +`stage-k=65` is only the retained tuning identifier for stage64 plus the raw +LDS-ready barrier. The public `gemm_out_prefill(...)` dispatches the same +kernel. + +Artifacts: + +```text +/workspace/ISA_test_codex/wqkv_a_compute_tuning/w8a8_gemm_compute.hip +/workspace/ISA_test_codex/wqkv_a_compute_tuning/M3072_RESULTS.md +/workspace/ISA_test_codex/wqkv_a_compute_tuning/wqkv_a_m3072_compute_gfx928.co +/workspace/ISA_test_codex/wqkv_a_compute_tuning/wqkv_a_m3072_compute_stage64_raw.isa +/workspace/ISA_test_codex/wqkv_a_compute_tuning/wqkv_a_m3072_compute.notes +``` diff --git a/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-quantized-gemm-optimization/SKILL.md b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-quantized-gemm-optimization/SKILL.md new file mode 100644 index 00000000..d16680c7 --- /dev/null +++ b/metainfer/tasks/dcu_kernel_auto_opt/skills/liquid-k500sm/int8-w8a8-quantized-gemm-optimization/SKILL.md @@ -0,0 +1,47 @@ +--- +name: int8-w8a8-quantized-gemm-optimization +description: > + Router for the INT8 W8A8 GEMM skill family on Hygon K500SM_AI/gfx928. + This skill is split by execution phase into three skills — load + int8-w8a8-gemm-foundations plus the phase skill: int8-w8a8-gemm-decode + (M<=32) or int8-w8a8-gemm-prefill (M>32). This router exists for backward + compatibility; it carries no tuning content itself. +--- + + + +> **Edition: LIQUID board (`liquid-k500sm`, K500SM_AI / DEV-HPC, HBM address +> hashing ON).** Every measurement in this edition was taken there. The air +> board (`K100_AI / DEV-AI`, no hashing) runs these operators 1.06-1.70x slower +> with a per-operator ratio, so **these numbers must not be used on the air +> board** and this edition is not installed on it -- the air machine gets +> `skills/air-k100ai/` instead. Kept for the liquid board and for provenance. + +# INT8 W8A8 GEMM — skill family router + +The former single "int8-w8a8-quantized-gemm-optimization" skill was split by +execution phase (2026-08-23) because decode and prefill have non-transferable +recipes, different bottlenecks, and different acceptance protocols. Load the +following instead: + +| M | Load | +|---|---| +| M <= 32 (decode) | `int8-w8a8-gemm-decode` + `int8-w8a8-gemm-foundations` | +| M > 32 (prefill) | `int8-w8a8-gemm-prefill` + `int8-w8a8-gemm-foundations` | + +- **int8-w8a8-gemm-foundations** — operator/layout contract, DUMMA and + epilogue rules, Graph-safe PyTorch interface, SGLang integration, fair + benchmarking, shared correctness gates and guardrails, E2E evidence. +- **int8-w8a8-gemm-decode** — split-K partial kernels + fused combine, + M=1..16 validated recipes, M=16 decode, M > 32 -> prefill**. M in (32, 128] +(decode side) and (32, 3072) (prefill side) are measured-boundary gaps — route +by measurement and keep both an arm per regime until measured. + +No tuning content lives in this router; always load the foundations skill plus +the matching phase skill. diff --git a/metainfer/tasks/dcu_kernel_auto_opt/static/dkao-detail.js b/metainfer/tasks/dcu_kernel_auto_opt/static/dkao-detail.js index 49e378f7..b026d588 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/static/dkao-detail.js +++ b/metainfer/tasks/dcu_kernel_auto_opt/static/dkao-detail.js @@ -1,5 +1,14 @@ import { html } from "htm/preact"; import { useCallback, useEffect, useState } from "preact/hooks"; +// Shell-owned plumbing shared by every auto-refreshing panel: +// pollWhileVisible — stop polling while the tab is hidden, and let a panel +// that describes finished history switch itself off (the AHE page mounts +// this view once per iteration, so a finished round used to keep pulling +// megabytes that could not change); +// fetchJsonCached — coalesce the same URL across mounts and skip unchanged +// bodies, so a poll that changes nothing costs a header exchange. +import { fetchJsonCached, UNCHANGED } from "app/api"; +import { usePollWhileVisible as pollWhileVisible } from "app/utils"; async function fetchJson(url) { const response = await fetch(url, { cache: "no-store" }); @@ -32,9 +41,94 @@ async function postJson(url, body) { // // Every component below takes `ctx = { taskId, stateDir, readOnly }` so both // the standalone page and the embedded one share one code path. -export function apiBase({ taskId, stateDir }) { - const base = `/api/dcu-kernel-auto-opt/${encodeURIComponent(taskId || "")}`; - return stateDir ? `${base}?state_dir=${encodeURIComponent(stateDir)}` : base; +// +// The state dir is a *query parameter*, so it must stay at the END of the +// URL: every endpoint below is `/` and FastAPI routes on the +// path alone. Handing callers a base that already carries `?state_dir=…` +// turns `${base}/summary` into `…?state_dir=/summary`, i.e. the endpoint +// is appended to the query value -- no route matches and the embedded view +// 404s on every lane (the standalone page is unaffected, since it passes no +// state dir). ``apiUrl`` therefore builds the whole URL and always appends +// the query last; keep it the single place that knows this rule. +export function apiBase({ taskId }) { + return `/api/dcu-kernel-auto-opt/${encodeURIComponent(taskId || "")}`; +} + +export function apiUrl(ctx, endpoint = "") { + const base = apiBase(ctx); + const query = ctx && ctx.stateDir + ? `state_dir=${encodeURIComponent(ctx.stateDir)}` + : ""; + if (!query) return `${base}${endpoint}`; + // `endpoint` may carry its own query string already (e.g. + // "/iterations?view=lanes"), so choose the separator rather than assuming + // there is no "?" yet. + const sep = endpoint.includes("?") ? "&" : "?"; + return `${base}${endpoint}${sep}${query}`; +} + +// The variant that may be shown as "current" for one shape: **this machine's +// own hardware profile only**. A variant measured on the other board is not +// comparable (air boards have no HBM address hashing, so read-heavy operators +// are 1.06-1.70x off), so a task on this board must show 无 -- exactly as the +// page did before the profile split -- rather than another profile's number. +// Writes are guarded server-side (``require_profile``); this is the read side +// of the same rule. ``index`` is a ``Map(shape -> variant record)``. +export function currentProfileVariant(index, shape, hwProfileId) { + const variant = index ? index.get(shape) : null; + if (!variant || !variant.profile || !hwProfileId) return null; + return variant.profile === hwProfileId ? variant : null; +} + +// The variant library that belongs to **this task**: the board the task's +// numbers were measured on, falling back to this machine's board only when the +// task carries no record of its own. +// +// This is not the same question as "which board is this machine": the task list +// lets an operator open any task, including one from the other board, and that +// page must show *that* board's library. Filtering by the machine's profile +// made every liquid task look empty here. +export function displayProfile(taskHwProfile, machineHwProfile) { + return taskHwProfile || machineHwProfile || null; +} + +// Whether this page may write into the variant library. The write lands in +// *this machine's* profile, so it is only meaningful when the task's numbers +// came from this board; the server enforces the same rule (409) -- this is the +// visible half, so the button is not offered at all. +export function canPromote(taskHwProfile, machineHwProfile) { + if (!machineHwProfile) return false; + return taskHwProfile === machineHwProfile; +} + +// The same rule for the skill library: fusion and publishing write into +// ``~/.dsh/skills`` (this machine's edition, mirrored to Claude) while the +// pending skill's numbers were measured on the task's board, so a write may only +// be offered when the two agree -- a liquid task's findings fused on the air +// machine would be read as air knowledge. The server refuses the same thing with +// a 409; this is the visible half, so the button is not offered at all. +export function canWriteSkills(taskHwProfile, machineHwProfile) { + return canPromote(taskHwProfile, machineHwProfile); +} + +// ``Map(shape -> variant record)`` for ONE board, built from the raw index the +// route returns. +// +// Filtering here -- not only at render time -- is what makes the button and the +// "当前" column agree with reality: the route lists *both* profiles' records for +// the same shape (air first, liquid later), so a Map built straight from that +// list keeps the **liquid** record for every shared shape. Every row then still +// read "当前: 无" after this board's variant was added, and the same rows kept +// offering "加入 variant" instead of "更新 variant". +export function profileVariantIndex(variants, hwProfileId) { + const index = new Map(); + if (!hwProfileId) return index; + for (const variant of variants || []) { + if (!variant || !variant.shape) continue; + if (variant.profile !== hwProfileId) continue; + index.set(variant.shape, variant); + } + return index; } function ctxOf(props) { @@ -360,9 +454,8 @@ function WorkerLane({ lane, ctx, onSaved }) { setSaving(true); setMessage(""); try { - const base = apiBase(ctx); await postJson( - `${base}/workers/${encodeURIComponent(lane.worker_id)}/guidance`, + apiUrl(ctx, `/workers/${encodeURIComponent(lane.worker_id)}/guidance`), { text: text.trim() }, ); setText(""); @@ -382,9 +475,9 @@ function WorkerLane({ lane, ctx, onSaved }) { setRestarting(true); setMessage(""); try { - const base = apiBase(ctx); await postJson( - `${base}/workers/${encodeURIComponent(lane.worker_id)}/restart`, {}, + apiUrl(ctx, `/workers/${encodeURIComponent(lane.worker_id)}/restart`), + {}, ); setMessage(`Restarted on GPU ${lane.gpu}; sibling workers were not interrupted.`); await onSaved(); @@ -496,7 +589,7 @@ function WorkerLane({ lane, ctx, onSaved }) { `; } -function SkillFile({ skill, canPublish, canFuse, onPublish, onFuse, busy }) { +function SkillFile({ skill, canPublish, canFuse, onPublish, onFuse, busy, pending }) { return html`
@@ -527,7 +620,7 @@ function SkillFile({ skill, canPublish, canFuse, onPublish, onFuse, busy }) { onPublish(skill); }} >${busy === `publish-${skill.name}` ? "Adding…" : "Add to existing"} - ` : html`existing`} + ` : (pending ? null : html`existing`)} 
${skill.content}
@@ -564,10 +657,25 @@ function SkillLibrary({ ctx, onClose }) { const [library, setLibrary] = useState(null); const [error, setError] = useState(""); const [busy, setBusy] = useState(""); + // Writes land in *this machine's* edition of the library, so the task's own + // board must be the same one (``write_blocked_reason`` is the server's verdict + // and outranks this comparison; the server also returns 409). + const machineProfile = library?.hw_profile || ""; + const taskProfile = library?.task_hw_profile || ""; + const canWrite = Boolean(library) && !ctx.readOnly + && !library.write_blocked_reason + && canWriteSkills(taskProfile, machineProfile); + const profileName = (id) => { + if (!id) return "未知"; + const found = (library?.profiles || []).find((item) => item.id === id); + if (!found) return id; + if (found.cooled === "air") return "风冷"; + if (found.cooled === "liquid") return "液冷"; + return found.label || id; + }; const load = useCallback(async () => { try { - const base = apiBase(ctx); - setLibrary(await fetchJson(`${base}/skills`)); + setLibrary(await fetchJson(apiUrl(ctx, "/skills"))); setError(""); } catch (err) { setError(String(err)); @@ -585,9 +693,8 @@ function SkillLibrary({ ctx, onClose }) { if (!window.confirm(`Add "${skill.name}" to the existing skill library (dsh, mirrored to Claude)?`)) return; setBusy(`publish-${skill.name}`); try { - const base = apiBase(ctx); await postJson( - `${base}/skills/${encodeURIComponent(skill.name)}/publish`, {}, + apiUrl(ctx, `/skills/${encodeURIComponent(skill.name)}/publish`), {}, ); await load(); } catch (err) { @@ -602,8 +709,8 @@ function SkillLibrary({ ctx, onClose }) { )) return; setBusy(`fuse-${skill.name}`); try { - const base = apiBase(ctx); - await postJson(`${base}/skills/fuse`, { skill_name: skill.name }); + await postJson(apiUrl(ctx, "/skills/fuse"), + { skill_name: skill.name }); await load(); } catch (err) { setError(String(err)); @@ -614,8 +721,7 @@ function SkillLibrary({ ctx, onClose }) { const syncNow = async () => { setBusy("sync"); try { - const base = apiBase(ctx); - await postJson(`${base}/skills/sync`, {}); + await postJson(apiUrl(ctx, "/skills/sync"), {}); await load(); } catch (err) { setError(String(err)); @@ -627,8 +733,8 @@ function SkillLibrary({ ctx, onClose }) { if (!window.confirm(`Restore the last backup of skill "${name}"?`)) return; setBusy(`rollback-${name}`); try { - const base = apiBase(ctx); - await postJson(`${base}/skills/${encodeURIComponent(name)}/rollback`, {}); + await postJson(apiUrl(ctx, `/skills/${encodeURIComponent(name)}/rollback`), + {}); await load(); } catch (err) { setError(String(err)); @@ -649,7 +755,7 @@ function SkillLibrary({ ctx, onClose }) { ${error ? html`
${error}
` : null} - ${readOnly ? html` + ${ctx.readOnly ? html`
只读镜像 —— 该 DKAO 运行是 harness_evolve 的一轮考题，由 HE 编排器调度； 本页不提供引导 / 重启 / 发布等写操作。 @@ -662,6 +768,13 @@ function SkillLibrary({ ctx, onClose }) { ? ` (${library.quarantined_count} pending files quarantined)` : ""}
` : null} + ${!ctx.readOnly && library && !canWrite ? html` +
+ ${taskProfile + ? `不可写入：此任务是【${profileName(taskProfile)}档】，本机是【${profileName(machineProfile)}档】 —— 融合/发布只会写进本机档的 skill 库，会把别档的数值当成这一档的知识。请在【${profileName(taskProfile)}档】的机器上操作。` + : "不可写入：此任务没有档位记录（先跑一次 backfill_task_hw_profile）。"} +
+ ` : null} <${SkillFuseStatus} status=${library?.fuse_status} onRollback=${ctx.readOnly ? null : rollback} /> ${!library ? html`

Loading skills…

` : html` @@ -682,8 +795,9 @@ function SkillLibrary({ ctx, onClose }) { ${(library.pending || []).map((skill) => html` <${SkillFile} skill=${skill} - canPublish=${!ctx.readOnly} - canFuse=${!ctx.readOnly} + pending=${true} + canPublish=${canWrite} + canFuse=${canWrite} busy=${busy} onPublish=${publish} onFuse=${fuse} @@ -709,13 +823,40 @@ export default function DcuKernelAutoOptDetail(props) { const refresh = useCallback(async () => { if (!taskId) return; try { - const base = apiBase(ctx); + // Poll the cheap change detector first. fetchJsonCached returns + // UNCHANGED when the body is byte-identical, so an unchanged signature + // short-circuits the heavy payloads (/summary ~0.6 MB + /iterations + // ~5.7 MB on a multi-worker task). Before this gate the panel pulled + // ~6 MB every 3 s, which is what made the page feel stuck. + // + // The gate and the server agree by construction: /iterations is tagged + // with the very same signature, so "signature unchanged" and "the lane + // payload is unchanged" are now the same statement -- and it covers the + // round feeds (workers/*/runs/*/experiments.jsonl) that decide whether + // the list is actually current. + const signature = await fetchJsonCached(apiUrl(ctx, "/state-signature")); + if (signature === UNCHANGED) return; const [summary, graph, iterations] = await Promise.all([ - fetchJson(`${base}/summary`), - fetchJson(`${base}/state-graph`), - fetchJson(`${base}/iterations`), + fetchJsonCached(apiUrl(ctx, "/summary")), + fetchJsonCached(apiUrl(ctx, "/state-graph")), + // `view=lanes` drops the per-round fields no browser code reads + // (candidate_isa and friends): 19.7 -> 6.0 MB on the heaviest task. + // The server still serves the full body by default, so this is the + // list view opting into a smaller cut, not a contract change. + fetchJsonCached(apiUrl(ctx, "/iterations?view=lanes")), ]); - setRuntime({ summary, graph, lanes: iterations.workers || [] }); + // 304 for all three: nothing to re-render, and no state churn. + if (summary === UNCHANGED && graph === UNCHANGED + && iterations === UNCHANGED) { + setError(null); + return; + } + setRuntime((prev) => ({ + summary: summary === UNCHANGED ? prev.summary : summary, + graph: graph === UNCHANGED ? prev.graph : graph, + lanes: iterations === UNCHANGED ? prev.lanes + : (iterations.workers || []), + })); setError(null); } catch (err) { setError(String(err)); @@ -723,60 +864,139 @@ export default function DcuKernelAutoOptDetail(props) { }, [ctx.taskId, ctx.stateDir]); useEffect(() => { refresh(); }, [refresh]); - useEffect(() => { - const timer = setInterval(refresh, 3000); - return () => clearInterval(timer); - }, [refresh]); + + // A finished run is immutable history: keep the panel live only while the + // run can still move, and never while the tab is in the background. + const innerRun = runtime.summary && runtime.summary.run; + pollWhileVisible(refresh, 3000, { + enabled: !innerRun || !innerRun.finished, + }); // ---- variant library (加入 variant 按钮) ------------------------------ // const [variantBusy, setVariantBusy] = useState(""); const [variantMsg, setVariantMsg] = useState(""); const [variantIndex, setVariantIndex] = useState(null); // null = loading + // Two different profiles are in play, and conflating them was a bug: + // * this machine's board (``hwProfileId``) -- where a write would land; + // * the board **this task** was measured on (``taskHwProfileId``) -- whose + // library this page must show. + // Variants live per profile (variant//…) because a median measured + // on an air board is not comparable with one measured on a liquid board: air + // boards lack HBM address hashing, so read-heavy operators are 40-87% slower + // there. + const [currentProfile, setCurrentProfile] = useState(null); + const [taskProfile, setTaskProfile] = useState(null); + const [variantProfiles, setVariantProfiles] = useState(null); + const hwProfileId = currentProfile ? currentProfile.hw_profile : null; + const taskHwProfileId = taskProfile ? taskProfile.task_hw_profile : null; + // Which library the rows describe: the task's own board. + const shownProfileId = displayProfile(taskHwProfileId, hwProfileId); + const mayPromote = canPromote(taskHwProfileId, hwProfileId); + const profileName = (id) => { + if (!id) return "未知"; + if (id === "unmigrated") return "未分档"; + const p = variantProfiles ? variantProfiles.get(id) : null; + if (!p) return id; + if (p.cooled === "air") return "风冷"; + if (p.cooled === "liquid") return "液冷"; + return p.label || id; + }; + // Display lookup: the task's own board -- see ``currentProfileVariant``. + const variantFor = (shape) => currentProfileVariant( + variantIndex, shape, shownProfileId + ); const loadVariants = useCallback(async () => { if (!taskId) return; try { - const base = apiBase(ctx); - const data = await fetchJson(`${base}/variants`); - // Map shape -> variant record (incl. median_us/speedup/source) so the - // update flow can compare the existing variant against this task. - setVariantIndex(new Map((data.variants || []).map((v) => [v.shape, v]))); + const data = await fetchJsonCached(apiUrl(ctx, "/variants")); + if (data === UNCHANGED) return; + // Map shape -> variant record (incl. median_us/speedup/source/profile) + // so the update flow can compare the existing variant against this task. + // Only the shown board's records go in (see ``profileVariantIndex``): the + // list holds both boards' records for the same shape. + const machine = data.hw_profile ? data.hw_profile.hw_profile : null; + const shown = displayProfile( + data.task_hw_profile, machine + ); + setVariantIndex(profileVariantIndex(data.variants, shown)); + setVariantProfiles(new Map((data.profiles || []).map((p) => [p.id, p]))); + setCurrentProfile(data.hw_profile || null); + setTaskProfile({ + task_hw_profile: data.task_hw_profile || "", + task_hw_profile_source: data.task_hw_profile_source || "", + }); } catch (err) { /* non-fatal */ } }, [ctx.taskId, ctx.stateDir]); useEffect(() => { loadVariants(); }, [loadVariants]); + // The library changes only when a round promotes a kernel, so no reason to + // ask every 3s; a minute keeps the "已有 variant" hint honest. + pollWhileVisible(loadVariants, 60000); + // "加入 variant" follows the liquid board's mechanism: this profile's existing + // variant is replaced only by a **faster** candidate, and a slower one is not + // added at all. The server enforces the same rule (``add_variant``'s + // ``reject_slower_than_existing``); checking it here as well means the + // operator gets the verdict immediately instead of after a round trip, and a + // refusal reads as a decision rather than as an error. const addVariant = async (shape) => { - const existing = variantIndex ? variantIndex.get(shape) : null; + if (!mayPromote) { + // The write would land in this machine's profile while the numbers came + // from the task's board (server-side 409 for the same reason). + setVariantMsg( + taskHwProfileId + ? `未加入：此任务是【${profileName(taskHwProfileId)}档】，` + + `本机是【${profileName(hwProfileId)}档】—— 它的数字不能写进本机档。` + : "未加入：此任务没有档位记录，先跑一次 " + + "`python3 -m metainfer.server.tools.backfill_task_hw_profile --apply`。" + ); + return; + } + const existing = variantFor(shape); + const candidate = bestRoundPerShape(runtime.lanes).get(shape) || {}; + const newUs = candidate.median_us; + const oldUs = existing ? existing.median_us : null; + const profileLabel = `${profileName(hwProfileId)}档`; + if (existing && newUs != null && oldUs != null && newUs > oldUs) { + setVariantMsg( + `未加入：本次 ${metric(newUs)} µs 慢于本档已有 variant 的` + + ` ${metric(oldUs)} µs —— ${profileLabel}保留更快的那一条。` + ); + return; + } if (existing) { - const candidate = bestRoundPerShape(runtime.lanes).get(shape) || {}; - const newUs = candidate.median_us; - const oldUs = existing.median_us; - let verdict = ""; - if (newUs != null && oldUs != null) { - verdict = newUs < oldUs - ? "（本次更快）" - : newUs > oldUs ? "（本次更慢）" : "（持平）"; - } + const verdict = newUs == null || oldUs == null + ? "" + : newUs < oldUs ? "（本次更快）" : "（持平）"; const ok = window.confirm( `该算子已有 variant：median ${oldUs ?? "?"} µs` + `（来源 ${existing.source || "?"}）。\n` + `本次候选：median ${newUs ?? "?"} µs ${verdict}。\n` - + `用本次候选替换？旧文件会备份（.bak-*）。` + + "用本次候选替换？旧文件会备份（.bak-*）。" ); if (!ok) return; } setVariantBusy(shape); setVariantMsg(""); try { - const base = apiBase(ctx); - const res = await postJson(`${base}/variants`, { shape_id: shape }); + const res = await postJson(apiUrl(ctx, "/variants"), + { shape_id: shape }); const meta = res.meta || {}; const tp = meta.tp != null ? `tp${meta.tp}` : "tp?"; setVariantMsg( `已加入 variant: ${meta.type} | ${meta.model} | ${tp} | ${meta.operator}` - + `（${res.action === "updated" ? "更新已有分区" : "新增分区"}）` + + `（${res.action === "updated" ? "更替本档更慢的那一条" : "新增"}）` ); await loadVariants(); } catch (err) { - setVariantMsg(`加入 variant 失败: ${String(err)}`); + const detail = String(err && err.message ? err.message : err); + setVariantMsg( + // The server refuses a slower candidate (the guard above may not see + // it: the candidate number shown here is the lane's best round, the + // server reads the accepted kernel's own manifest). + /slower variant replacement/i.test(detail) + ? `未加入：服务端判定本次候选不优于${profileLabel}已有 variant` + + `（保留更快的那一条）。` + : `加入 variant 失败: ${detail}` + ); } finally { setVariantBusy(""); } @@ -803,8 +1023,7 @@ export default function DcuKernelAutoOptDetail(props) { setRenameBusy(true); setRenameMsg(""); try { - const base = apiBase(ctx); - const res = await postJson(`${base}/rename-repo`, { + const res = await postJson(apiUrl(ctx, "/rename-repo"), { new_name: target.trim(), }); setRenameMsg(`已重命名: ${res.old_name} → ${res.new_name}`); @@ -821,6 +1040,12 @@ export default function DcuKernelAutoOptDetail(props) { ); const bestRoundEntries = Array.from(bestRounds.entries()) .sort(([a], [b]) => (shapeOrder.get(a) ?? 1e9) - (shapeOrder.get(b) ?? 1e9)); + // Rounds whose speedup had to be recomputed against this board's baseline + // (the run itself used another hardware profile's number). Shown as a + // footnote rather than silently swapped numbers. + const rebasedRows = bestRoundEntries.filter( + ([, item]) => item.recorded_speedup != null + ); return html`
${error ? html`
${error}
` : null} @@ -930,6 +1155,15 @@ export default function DcuKernelAutoOptDetail(props) { Best optimization round per shape accepted “best” round from each worker lane +

+ 本机硬件档：${profileName(hwProfileId)} + ${hwProfileId ? html`（${hwProfileId}${currentProfile && currentProfile.machine ? " @ " + currentProfile.machine : ""}）` : html`（未识别：写入会被拒绝，见 wiki-metainfer/knowledge-packs.md §3）`} + ${shownProfileId && shownProfileId !== hwProfileId ? html` +
此任务硬件档：${profileName(shownProfileId)} + （${shownProfileId}；下面的 variant 与超基线数字都是这一档测的，**不可**与本机档比较，本机也不能把它的结果写进本机档） + ` : null} + ${shownProfileId ? html`　variant 按档存放，跨档数字不可比较` : null} +

@@ -941,38 +1175,56 @@ export default function DcuKernelAutoOptDetail(props) { - + `)}
ShapeWorkerBest roundPerformanceSpeedupINT8 TOPSAlgorithmic BWVariant
${item.worker_id} ${item.iteration ?? "—"} ${metric(item.median_us)} µs${metric(item.speedup, 3)}×${metric(item.speedup, 3)}×${item.recorded_speedup != null + ? html` *` + : null} ${metric(item.metrics.logical_tops ?? item.metrics.tflops, 3)} ${metric(item.metrics.algorithmic_bandwidth_gb_s ?? item.metrics.bandwidth_gb_s, 2)} GB/s - ${readOnly ? null : html` + ${readOnly ? null : !mayPromote ? html` +
+ ${taskHwProfileId + ? `不可写入：此任务是【${profileName(taskHwProfileId)}档】，本机是【${profileName(hwProfileId)}档】` + : "不可写入：此任务没有档位记录（先跑一次 backfill_task_hw_profile）"} +
+ ` : html` `} ${(() => { - const v = variantIndex ? variantIndex.get(shape) : null; + const v = variantFor(shape); if (!v) { return html`
当前: 无
`; } - if (v.speedup != null) { - return html`
当前: 超基线 ${metric(v.speedup, 2)}x
`; - } - if (v.median_us != null) { - return html`
当前: ${metric(v.median_us)} µs
`; - } - return html`
当前: —
`; + const value = v.speedup != null + ? html`超基线 ${metric(v.speedup, 2)}x` + : v.median_us != null ? html`${metric(v.median_us)} µs` : "—"; + return html`
当前: ${value}
`; })()}
+ ${rebasedRows.length ? html` +

+ * 带 * 的 speedup 已按【本机档】baseline 重算（与 variant 同一把尺， + 所以"当前: 超基线"应与它一致）。本任务运行时用的判据是另一档的 baseline： + ${rebasedRows.map(([shape, item]) => html` + + ${shape} ${item.recorded_speedup != null ? metric(item.recorded_speedup, 3) : "—"}× + （${metric(item.recorded_baseline_us)} µs） + + `)} + —— 两档数字不可比较；历史报告文件未被改动。 +

+ ` : null} ${variantMsg ? html`

${variantMsg}

` : null} ` : null} ${report ? html` diff --git a/metainfer/tasks/dcu_kernel_auto_opt/static/dkao-shape-input.js b/metainfer/tasks/dcu_kernel_auto_opt/static/dkao-shape-input.js index a0240b49..fa4a3ad6 100644 --- a/metainfer/tasks/dcu_kernel_auto_opt/static/dkao-shape-input.js +++ b/metainfer/tasks/dcu_kernel_auto_opt/static/dkao-shape-input.js @@ -24,9 +24,13 @@ function parseCsvInts(raw) { var W8A8_M_VALUES = [2, 16, 3072]; // Large-prefill boundary added on 2026-08-06 for TP4; extended to the // Hy3 / MiniMax M3 / GLM5.2 TP8 catalogs on 2026-08-27 via per-topology -// mValues, plus the M=8 short-decode boundary on 2026-09-20. DeepSeek TP1/TP8 -// keep the three original M values. The backend validates shapes against the -// contract (MODEL_TP8_EXTRA_OPTIMIZATION_M_VALUES = (8, 4096)). +// mValues, plus the M=8 short-decode boundary on 2026-09-20. DeepSeek TP8 +// joined them on 2026-10-11 (same five M values; measured on the air board). +// What does NOT change: the DeepSeek-only *default* workload +// (DEFAULT_W8A8_SHAPES -> contract DEFAULT_OPTIMIZATION_SHAPES, 42 shapes, +// M=2/16/3072) and its serial-validation fallback scope. The backend validates +// every submitted shape against the contract +// (MODEL_TP8_EXTRA_OPTIMIZATION_M_VALUES = (8, 4096)). var W8A8_TP4_M_VALUES = [2, 16, 3072, 4096]; var W8A8_MODEL_TP8_M_VALUES = [2, 8, 16, 3072, 4096]; @@ -38,17 +42,6 @@ var MODEL_WORKLOADS = { "DeepSeek V4 Flash": { idPrefix: "", topologies: [ - { - tp_size: 1, - operators: [ - { operator: "wqkv_a", K: 4096, N: 1536 }, - { operator: "wq_b", K: 1024, N: 32768 }, - { operator: "indexer.wq_b", K: 1024, N: 8192 }, - { operator: "wo_b", K: 8192, N: 4096 }, - { operator: "shared_gate_up_proj", K: 4096, N: 4096 }, - { operator: "shared_down_proj", K: 2048, N: 4096 }, - ], - }, { tp_size: 4, operators: [ @@ -62,6 +55,9 @@ var MODEL_WORKLOADS = { }, { tp_size: 8, + // DeepSeek TP8 covers the same five M values as the other model + // catalogs since 2026-10-11 (M=8 short decode, M=4096 large prefill). + mValues: W8A8_MODEL_TP8_M_VALUES, operators: [ { operator: "wqkv_a", K: 4096, N: 1536 }, { operator: "wq_b", K: 1024, N: 4096 }, @@ -76,15 +72,6 @@ var MODEL_WORKLOADS = { "Hy3 (Hunyuan 3)": { idPrefix: "hy3_", topologies: [ - { - tp_size: 1, - operators: [ - { operator: "qkv_proj", K: 4096, N: 10240 }, - { operator: "o_proj", K: 8192, N: 4096 }, - { operator: "shared_gate_up_proj", K: 4096, N: 3072 }, - { operator: "shared_down_proj", K: 1536, N: 4096 }, - ], - }, { tp_size: 4, operators: [ @@ -109,16 +96,6 @@ var MODEL_WORKLOADS = { "MiniMax M3": { idPrefix: "minimax_", topologies: [ - { - tp_size: 1, - operators: [ - { operator: "qkv_proj", K: 6144, N: 9216 }, - { operator: "qkv_proj_and_indexer_qk", K: 6144, N: 9856 }, - { operator: "o_proj", K: 8192, N: 6144 }, - { operator: "shared_gate_up_proj", K: 6144, N: 6144 }, - { operator: "shared_down_proj", K: 3072, N: 6144 }, - ], - }, { tp_size: 4, operators: [ @@ -145,17 +122,6 @@ var MODEL_WORKLOADS = { "GLM5.2": { idPrefix: "glm_", topologies: [ - { - tp_size: 1, - operators: [ - { operator: "fused_qkv_a_proj", K: 6144, N: 2624 }, - { operator: "q_b_proj", K: 2048, N: 16384 }, - { operator: "kv_b_proj", K: 512, N: 28672 }, - { operator: "o_proj", K: 16384, N: 6144 }, - { operator: "shared_gate_up_proj", K: 6144, N: 4096 }, - { operator: "shared_down_proj", K: 2048, N: 6144 }, - ], - }, { tp_size: 4, operators: [ @@ -186,7 +152,14 @@ var MODEL_WORKLOADS = { function modelCatalog(modelLabel) { var model = MODEL_WORKLOADS[modelLabel] || MODEL_WORKLOADS["DeepSeek V4 Flash"]; var shapes = []; - (model.topologies || []).forEach(function (topology) { + // The trusted API contract only defines TP 4 and TP 8 shards + // (int8_w8a8_gemm_api.py: "tp_size must be 4 or 8"), so a topology the + // contract would reject is never offered: picking one used to fail the task + // in `prepare` with "shape ... is outside the fixed API contract". + var CONTRACT_TP_SIZES = [4, 8]; + (model.topologies || []).filter(function (topology) { + return CONTRACT_TP_SIZES.indexOf(topology.tp_size) !== -1; + }).forEach(function (topology) { // Per-topology M values override the defaults; TP4 and the Hy3/MiniMax/ // GLM TP8 catalogs cover the M=8 short-decode and M=4096 large-prefill // boundaries. @@ -209,8 +182,17 @@ function modelCatalog(modelLabel) { return shapes; } -// Backward-compatible DeepSeek-only default catalog. -var DEFAULT_W8A8_SHAPES = modelCatalog("DeepSeek V4 Flash"); +// Backward-compatible DeepSeek-only default catalog: it mirrors the contract's +// DEFAULT_OPTIMIZATION_SHAPES (42 shapes; TP8 M=2/16/3072, TP4 additionally +// M=4096). The TP8 M=8 / M=4096 boundaries above are *panel* affordances for +// "Selected shapes only", so they are filtered out here -- a task submitted with +// an empty shape field must still ask for exactly the contract's default set. +// (test_shape_catalog_panel.py checks this list against the contract.) +var DEFAULT_W8A8_SHAPES = modelCatalog("DeepSeek V4 Flash").filter( + function (shape) { + return shape.M !== 8 && !(shape.tp_size === 8 && shape.M === 4096); + } +); function currentShapeId(catalog, rawId, modelLabel) { if (catalog.some(function (shape) { return shape.id === rawId; })) { @@ -269,18 +251,105 @@ function parseShapeRecords(raw) { return shapes; } +// ---- machine GPU inventory ------------------------------------------------- +// +// Which cards this machine exposes is a *detected* machine fact (eight on the +// air board, four on liquid) and lives server-side in +// metainfer.orchestrator.gpu_inventory, surfaced through the shell's +// /api/sys-shell/hw-profile. We fetch it once per page and cache the promise, +// because a form control that silently assumed "0..3" is exactly how four of +// kme6's eight cards became unselectable. +// +// A card number here is a host HCU ordinal: the same number hy-smi prints as +// DCU[n] and the worker exports as HIP_VISIBLE_DEVICES=n. +var DEFAULT_LANES = 4; +var DEFAULT_GPU_DEVICES = [0, 1, 2, 3]; +var _gpuDevicesPromise = null; + +function fetchGpuDevices() { + if (_gpuDevicesPromise) return _gpuDevicesPromise; + _gpuDevicesPromise = fetch("/api/sys-shell/hw-profile", { cache: "no-store" }) + .then(function (response) { + return response.ok ? response.json() : null; + }) + .then(function (payload) { + var devices = payload && payload.gpus && payload.gpus.devices; + if (!Array.isArray(devices) || !devices.length) return DEFAULT_GPU_DEVICES; + return devices + .map(function (item) { return parseInt(item, 10); }) + .filter(function (item) { return Number.isFinite(item) && item >= 0; }); + }) + .catch(function () { return DEFAULT_GPU_DEVICES; }); + return _gpuDevicesPromise; +} + +function useGpuDevices() { + var _b = useState(DEFAULT_GPU_DEVICES), devices = _b[0], setDevices = _b[1]; + useEffect(function () { + var alive = true; + fetchGpuDevices().then(function (list) { + if (alive && list && list.length) setDevices(list); + }); + return function () { alive = false; }; + }, []); + return devices; +} + +// Lane i uses cards[i]; the default is the historical 0,1,2,3 layout, so a +// four-card board renders byte-identical to before. +function fillLaneCards(partial, devices) { + var pool = (devices && devices.length ? devices : DEFAULT_GPU_DEVICES).slice(); + var used = {}; + var out = []; + for (var lane = 0; lane < DEFAULT_LANES; lane++) { + var card = partial && partial[lane] != null ? partial[lane] : null; + if (card != null) used[card] = true; + out.push(card); + } + for (var index = 0; index < DEFAULT_LANES; index++) { + if (out[index] != null) continue; + var pick = null; + for (var probe = 0; probe < pool.length; probe++) { + if (!used[pool[probe]]) { pick = pool[probe]; break; } + } + if (pick == null) pick = index; + used[pick] = true; + out[index] = pick; + } + return out; +} + +function defaultLaneCards(devices) { + return fillLaneCards(null, devices); +} + function parseManualOwners(raw) { + // Returns {owners: shapeId -> lane, laneCards: [card per lane]}. + // + // A worker is named after the card it runs on (`worker_6: {gpu: 6, ...}`), + // which is the contract the pipeline validates, so the *lane* is positional: + // lanes are laid out in the order the entries appear. The card itself is read + // from `gpu:` (falling back to the number in the worker name, which is what + // the same entry looked like before lanes and cards could differ). var owners = {}; - if (!raw) return owners; - var re = /worker_(\d+):\s*\{[^}]*shapes:\s*\[([^\]]*)\]/g, match; + var laneCards = []; + if (!raw) return { owners: owners, laneCards: laneCards }; + var re = /worker_(\d+):\s*\{([^}]*)\}/g, match; while ((match = re.exec(raw)) !== null) { - var gpu = parseInt(match[1], 10); - match[2].split(",").forEach(function (item) { + var body = match[2]; + var gpuMatch = /gpu:\s*(\d+)/.exec(body); + var card = gpuMatch ? parseInt(gpuMatch[1], 10) : parseInt(match[1], 10); + var shapesMatch = /shapes:\s*\[([^\]]*)\]/.exec(body); + if (!shapesMatch) continue; + var lane = laneCards.length; + if (lane >= DEFAULT_LANES) break; + laneCards.push(card); + shapesMatch[1].split(",").forEach(function (item) { var id = item.trim().replace(/^["']|["']$/g, ""); - if (id) owners[id] = gpu; + if (id) owners[id] = lane; }); } - return owners; + return { owners: owners, laneCards: laneCards }; } function operatorGroupedOwners(shapes) { @@ -351,16 +420,21 @@ function serializeShapesHeader(mode, scope, shapes, modelLabel) { return lines; } -function serializeManualAssignment(shapes, owners, scope, modelLabel) { +function serializeManualAssignment(shapes, owners, scope, modelLabel, laneCards) { + var cards = laneCards || DEFAULT_GPU_DEVICES; var lines = serializeShapesHeader("manual", scope, shapes, modelLabel); lines.push("assignments:"); - for (var gpu = 0; gpu < 4; gpu++) { + for (var lane = 0; lane < DEFAULT_LANES; lane++) { var ids = shapes - .filter(function (shape) { return owners[shape.id] === gpu; }) + .filter(function (shape) { return owners[shape.id] === lane; }) .map(function (shape) { return shape.id; }); if (ids.length) { + // Worker name == card: `_require_valid_child_assignments` and the + // generate-phase prompt both require worker_N on GPU N, so the lane's + // chosen card names the worker here rather than the lane index. + var card = cards[lane] != null ? cards[lane] : lane; lines.push( - " worker_" + gpu + ": {gpu: " + gpu + + " worker_" + card + ": {gpu: " + card + ", shapes: [" + ids.join(", ") + "]}" ); } @@ -393,6 +467,15 @@ function TpFilterBar(_a) { `; } +// Full-text hover hint for a shape row. The row no longer clips its text (see +// ``.dkao-shape-choice`` in dkao.css), but the tooltip keeps the whole +// signature readable at a glance without having to scan the wrapped line. +function shapeTitle(shape) { + return "TP=" + shape.tp_size + " · " + shape.operator + + " · M=" + shape.M + " · N=" + shape.N + " · K=" + shape.K + + " · " + shape.id; +} + function ShapeCatalog(_a) { var shapes = _a.shapes, selected = _a.selected, onToggle = _a.onToggle, readOnly = _a.readOnly; @@ -424,7 +507,8 @@ function ShapeCatalog(_a) { ${visible.map(function (shape) { var checked = Boolean(selected[shape.id]); return html` -