diff --git a/metainfer/tasks/harness_evolve/FLOW.md b/metainfer/tasks/harness_evolve/FLOW.md new file mode 100644 index 00000000..383df151 --- /dev/null +++ b/metainfer/tasks/harness_evolve/FLOW.md @@ -0,0 +1,252 @@ +# HE 流程（固定协议） + +> **这是 harness_evolve（AHE 外层循环）的规范文本，不是设计笔记。** +> 代码必须与本文一致；改协议 = 改本文 + 改代码 + 改契约测试 +> （`tests/test_flow_contract.py` 会检查本文存在且关键约束仍在）。 +> 实现位置见文末「代码对应」。 + +--- + +## 0. 边界：HE 只发起 DKAO 任务 + +**HE 的唯一职责：把这一轮的算子清单交给 DKAO，等它交回测量结果，据此判门、决定下一步。** + +| 事项 | 归属 | +|---|---| +| 取题 / 组卷（这一轮测哪些算子） | **HE** | +| 把每个算子起成一个 DKAO 子任务、传 harness 快照、等它跑完、收测量结果 | **HE** | +| 判性能门 / 泛化门、决定 analyze→evolve 还是待批 | **HE** | +| 每轮题数 = 4 的倍数（一波铺满 4 张 HCU） | **HE**（只做这个尺寸决定） | +| 卡干不干净（VRAM ≤ 90% 且 HCU == 0）、等卡、闸门超时 | **DKAO**（`ensure_measurement_gate`） | +| 卡被抢导致的污染测量作废 / 回滚 | **DKAO**（contamination 处理） | +| 内核编译、基准、正确性、PMC、最终验收 | **DKAO** | +| 子任务崩溃/工具链缺失等"环境欠一次测量"的重试 | **HE**（重发一次 DKAO，见 §5） | + +HE **不做**的事（照这条检查任何新代码）： + +- 不租卡（不调用 GPU broker）、不读卡状态、不因为"卡忙"等待或否决； +- 不 `SIGSTOP`/冻结子进程（那是早前为"卡被抢"设计的机制，已废弃：卡的事是 DKAO 的事）； +- 不判 `measurement_suspect`、不写污染窗口； +- 不因为自身对环境的判断而终止一轮——能终止一轮的只有"子任务没交回可用测量"这件事本身。 + +HE 为每道题只做一个**布局决定**：设备 `= index % 4`（第 i 道题绑第 i%4 张卡）， +它只保证"一波不超过 4 道题、一题一张卡"，不代表 HE 认为那张卡空闲。 +把每道题**立刻**发给 DKAO；卡不空时，是 DKAO 的子进程在闸门那里等。 + +--- + +## 1. 取题与组卷 + +### 1.1 每轮题数固定为 4 的倍数 + +- 机器 4 张 HCU（`GPU_COUNT = 4`），**一波 = 4 题**；一轮题数只能是 4 的倍数。 +- 表单字段 `per_round_budget`（Round budget）只提供 4 的倍数选项；CLI/接口传非 4 倍数时**向上取整** + （3→4、5→8、7→8），合法值原样保留（4→4、12→12）；非法/空 → 默认 4（`OPERATORS_PER_GATE`）。 +- 题数**只按配置抽**，不按空闲卡数缩水：卡忙是 DKAO 的事，缩题数会让一轮考不出结论。 + （历史上那个 `fit_questions_to_devices` 反向开关已移除。） + +### 1.2 抽题规则 + +从注册池（`pool_source`，真实实测池）中抽，**随机**，且： + +- 只抽**可判**算子：既有 variant 又有可用基线的才抽（否则算不出 win/loss，只会稀释比例）； + 可判算子不足一波（4 个）时**直接停任务**（`final_status=no_question_pool`，事件 + `variant_pool_too_small`）——缩到 3 题会改变门的含义（"≥75% 赢"在 3 题上等于"必须全胜"）； +- **保证同时含 decode 与 prefill**：至少 1 个 decode（`M ≤ 32`）和 1 个 prefill（`M > 32`）； + 其余名额随机填满——一轮只考一类 shape 会把更难、更有价值的那一半漏掉； +- 第 1 轮（baseline）：随机抽 4 题 = **A1**，这 4 题**同时固化为本轮的泛化考卷**（只写一次，之后永不替换）； +- 第 2 轮起（performance）：**重新随机抽 4 题 = A_i**，并**排除考卷**（`A_i ≠ A1`，也不同于上一轮的题）； +- **抽题可复现**：随机种子由 `(task_id, iteration, stage)` 派生（表单 `round_seed` 可显式钉住）， + 并写进该轮的 `variant_round_selection` 事件（`seed` / `count` / `judgeable_in_pool`）， + 这样"为什么抽到这 4 题"事后能查、那一轮也能重跑。 + +--- + +## 2. 轮次与阶段 + +一次"通过"叫一个 stage，记录在 `runs/iteration_NNN/input/stage.json`： + +| stage | 轮次编号 | 做什么 | 判哪门 | 消耗轮次 | +|---|---|---|---|---| +| **baseline** | iteration 1 | 抽 A1（4 题）交 DKAO 实测；这 4 题固化为考卷 | 都不判 | 0 | +| **performance** | iteration 2 起 | 抽新题 A_i（排除考卷）交 DKAO 实测 | 只判性能门 | 1.0 | +| **generalization** | 紧随其后的那一趟（"第 2.5 轮"） | 同一版 harness **再发一次 DKAO**，题目固定为 A1 | 只判泛化门 | 0.5 | + +> "紧随其后的那一趟"= 性能门通过后**立刻**用同一版 harness 再发一次评测（占下一趟 pass， +> 有自己的 `runs/iteration_NNN` 目录），不是在同一趟里重测；批准时记录的是**赢下性能门的那一轮编号**。 + +- **第 1 轮不判门**：它只负责"量出基线"。它测出来的更快 kernel **直接写进 variant 池** + （阈值 `promote_min_improvement_percent`，默认 3%），不需要人工批准——"kernel 是事实，harness 才要批"。 + 若第 1 轮有任一算子没拿到可用测量 → 整个 HE 任务立刻停（`baseline_round_failed`），不留半截证据。 +- **一轮的完整生命周期**（第 2 轮起）： + 1. 抽新题 A_i（4 的倍数、含 decode+prefill、排除考卷）→ 一题一个 DKAO 子任务； + 2. 判**性能门**（§3）；不过 → 回退到上一好 harness → analyze → evolve 产出新 harness → 下一轮重新随机抽题； + 3. 过了性能门 → 用**同一版 harness**再发一次 DKAO，题目 = 固化考卷 A1 → 判**泛化门**（§4）； + 4. 两门都过 → 写 `pending_promotion.json`，任务停在 `awaiting_approval`，**等人工批准**才接入生产 DKAO； + 任何一门不过 → 同上走 analyze→evolve→下一轮。 + +--- + +## 3. 性能门（performance stage） + +对手是**每个算子当前的 variant**（该算子目前的最优解：成绩 + 内核）。 + +- **赢** = 严格快于 variant（含 2% 噪声带；平局不算赢）； +- 通过条件：**≥75% 的题赢**（4 题即至少 3 题），**且没赢的那些不得低于 variant 的 80%**。 + +## 4. 泛化门（generalization stage） + +**仅在性能门通过后开**。不抽新题：**重考第 1 轮那份固化考卷 A1**，同一版 harness 再发一次 DKAO。 + +- 通过条件：**≥50% 的题赢**，同样 **80% 地板**（任一题低于 variant 的 80% → 不过）； +- **平局不算输**：子任务以 `best_known` 热身起跑，考卷上的算子正是第 1 轮刚写进池的 kernel， + 重考大概率复现同一个 kernel；把"复现自己"判成失败等于要求搜索策略每轮都超越自己。 + 泛化门要的是"无败场 + 真退步就否决"。 + +### 门的三种结论：PASS / FAIL / **INCOMPLETE** + +门是对 **harness** 的断言，所以只有在这轮真的产出了证据时才允许下结论： + +| 结论 | 何时 | 循环怎么走 | +|---|---|---| +| **PASS** | 全部题都测到，且满足比例 + 地板 | 性能门过 → 冻结重考；泛化门过 → 写 `pending_promotion`，停 `awaiting_approval` 等批准 | +| **FAIL** | 已测到的部分**已经决定输**：有题低于 80% 地板，或"已赢 + 未测"仍达不到所需胜场数 | 回退上一好 harness → analyze → evolve → 下一轮重新抽题 | +| **INCOMPLETE** | 有题没测到，而结果**仍然开放** | **本轮不下结论**：记 `round_incomplete`，不写候选、不改判 harness；下一趟重新测量（泛化门缺测时**保留**冻结的重考请求，考卷仍欠一个判决） | + +这条分界很重要：过去任何缺测都直接判 FAIL，等于把"机器没给数"记成"harness 退步"， +既让循环去修一个没坏的 harness，也会在泛化重考时把候选白白花掉。 + +## 5. 测量没拿到怎么办（唯一允许的重发） + +- 子任务**崩溃/工具链缺失/无 report** → HE 认为"环境欠一次测量"，**重发一次 DKAO** + （重发是全新的一次 DKAO、独立子任务目录；上限 `env_retry_attempts`，默认 3 次尝试）。 +- 子任务**被 DKAO 自己的闸门拒测**（exit 75 / `gate_blocked`）→ **不重发**：DKAO 已经在它那张卡上 + 等满自己的预算（30 分钟一次 × 48 = 24 小时）才给的否决，HE 没有可换的卡，重发等于无视它的结论。 + 该轮判 `round_incomplete`（cause `gpu_gate_blocked`）→ **终态**，等人工决定（不会被自动 resume）。 +- 某一轮**有题缺测但结果仍开放**（见 §3 的 INCOMPLETE）→ 该轮不下结论，事件 `round_incomplete` + （`gate` / `unmeasured` / `measured`），循环按 §3 的表继续。 +- 每次尝试都写独立目录（`children/iteration_NNN`、`..._a2`、`..._generalization`）， + 失败那次的日志原样留档；DKAO 会把子任务工作区的 `main` 绑定到本次 kernel repo，共用目录会让重试秒退。 + +--- + +## 6. 判据与产物 + +| 文件 | 内容 | +|---|---| +| `variant_table.json` | 逐算子 variant + 固化考卷（`groups.generalization_ids`） | +| `runs/iteration_NNN/input/stage.json` | 这次是哪个 stage + `round_cost` | +| `.../baseline_round.json` | 第 1 轮：判定 + 入池结果 | +| `.../performance_gate.json` / `.../generalization_gate.json` | 两门的逐题判定（wins/needed/floor） | +| `.../decision.json` | 本轮 system decision | +| `pending_generalization.json` | 冻结的重考请求 | +| `pending_promotion.json` / `promoted_harness.json` | 待批候选 / 已接入版本 | +| `children/<尝试目录>/<算子>/` | 每个算子一次 DKAO 运行（独立目录） | + +批准后的动作：`approve` 把候选 harness 接入 DKAO（备份上一版）、用**写泛化门那次重考**的成绩更新 variant、 +把该轮算子内核按"最好者胜"写进生产池；任务结束（`promoted:<版本>`）。**只有被接受那一轮的优化进入生产。** + +--- + +## 7. 代码对应（改协议时一并改） + +| 协议条目 | 实现 | +|---|---| +| 题数 = 4 的倍数、一波 4 题 | `orchestrator/rounds.py`: `GPU_COUNT` / `OPERATORS_PER_GATE` / `normalized_per_gate` | +| 取题（随机 / 可判 / decode+prefill / 排除考卷） | `orchestrator/rounds.py`: `sample_operators` / `regime_of` | +| 轮次与阶段、考卷固化 | `orchestrator/rounds.py`: `select_round_questions` / `record_paper` / `round_stage` | +| 两门判据（75% / 50% + 80% 地板 + 2% 噪声带） | `orchestrator/decision_engine.py`（`gate_verdict`）+ `rounds.py: evaluate_gate` | +| 每轮把算子清单发给 DKAO、设备 = index % 4 | `orchestrator/adapters/eval.py`: `DkaoCliEvaluator.evaluate` / `_run_wave` | +| 环境相关的重发与终态 | `adapters/eval.py`: `_env_failed` / `_stop_on_incomplete_round` | +| 表单题数只给 4 的倍数 | `form.yaml`: `per_round_budget` | +| 契约测试 | `tests/test_flow_contract.py` | + +**环境相关的判据不在本文件**：卡干净与否、等多久、污染怎么作废，全部写在 DKAO 侧 +（`dcu_kernel_auto_opt/orchestrator/gpu_preflight.py`、`w8a8_pipeline.py` 的测量闸门、 +`contention.py`；DKAO 侧的报错归档见 `dcu_kernel_auto_opt/error_catalog/`）。 + +--- + +## 8. 旧路径（suite / legacy）：保留，但必须自报家门 + +新任务**一律**用本文件的固定协议（配一个实测池文件）。旧的 suite/legacy 路径仍然保留 +（离线套件、2026-09-15 之前的老实验），但它**不固化考卷、不判两门、不按 4 的倍数取题**—— +它的轮次不可能产出可接入 DKAO 的 harness。所以从 2026-09-22 起，它必须自己说出来： + +| 说给谁看 | 形式 | +|---|---| +| 盯着进程的人 | orchestrator 日志一条 `WARNING: running the LEGACY protocol (…). New tasks must use a measured pool file — see FLOW.md` | +| 任务的审计记录 | timeline 事件 `legacy_protocol_selected`（含 `reason` / `pool_source` / `pool_file` / `policy`） | +| 页面上的人 | 实验目录写 `protocol.json`，任务详情页据此在顶部显示黄色横幅「⚠ 本任务未按固定协议运行」 | + +**触发条件**（任一即判定为旧路径）：`round_questions: legacy|suite|off`；没配池文件（只用内联 suite）； +池文件路径不存在；用了 `builtin` 假池。 + +> 结论性的取舍（要不要彻底删掉这条路径）见 `README.md` 的「现状与边界」；在删除之前， +> 这条告警是防止它被误用的唯一保障。 + +--- + +## 9. 停止与续跑（operator 操作说明） + +**停止 = 暂停，不是杀进程。** 这两件事在本系统里必须分得很清：一个进程"只是没了"和"崩了"在判据上 +完全一样，而 supervisor / janitor / `should_launch_resume` 的存在就是为了把崩掉的跑起来——所以 +"只杀进程的停止"会在几秒到 5 分钟内被自动复活。 + +### 9.1 点按钮（推荐） + +任务详情页（HE 页面）状态机下方那一行：**「停止任务」**（琥珀色边框，仅在 `running` 时出现）。 + +一次点击按固定顺序做完六步（代码：`server/routes.py: stop`）： + +| # | 动作 | 为什么必须在这个位置 | +|---|---|---| +| 1 | 写 `workspace/operator_stop.json` | **标记先落地**：janitor 一旦先看到"没人在跑"，就会自动拉起它 | +| 2 | 停 supervisor（写 `supervisor.stop` + SIGTERM） | 它存在的意义就是"把死掉的实验拉起来"，必须先停它 | +| 3 | 杀掉本轮所有 DKAO 子任务（按 cmdline 匹配 workspace 路径） | 子任务各自 `start_new_session`，父进程死了它们还活着 | +| 4 | 杀掉 HE 编排器（`_run_alive`：`orchestrator.pid` → `resume.pid` → cmdline 扫描） | 最后杀，避免它在子任务写到一半时被打断 | +| 5 | **再扫一次子任务** | 编排器把"子任务消失"当环境失败，会在第 3~4 步之间发一波新的（`env_retry_attempts`）；漏掉它，页面写着"已停止"而新子任务还在卡上跑几小时 | +| 6 | `run.json` 写 `finished: true` / `final_status: stopped_by_operator` | 让所有自动路径收手；`finished` 同时是 `/resume` 的前置条件 | + +**关键点**：运行中的编排器**自己不读** `operator_stop.json`——只有 janitor（`_operator_stopped`）和 +supervisor 读它，CLI 只在"刻意重启"时删它。所以标记只防"自动复活"，**不会**让活着的进程停下来， +第 4 步不能省。 + +### 9.2 停止后一定不会被自动拉起 + +janitor 每 300 s 扫一遍，命中标记就跳过；`stopped_by_operator` 也在 `TERMINAL_STATUSES` 里， +`should_launch_resume` 直接返回 False。想确认，看 WebUI 日志（`/root/zth_agent/webui-8765.log`）： + +``` +[he-janitor] {'task': '', 'action': 'skip', 'reason': 'stopped by operator'} +``` + +### 9.3 续跑 + +页面上「续跑到该轮次上限」，或 `POST /api/harness-evolve//resume {"target": N}`。 +`_launch_resume` 会**先删掉 operator-stop 标记**再起进程，所以续跑不会被自己的标记挡住。 + +被停掉的那一轮**重跑**：抽题种子由 `(task_id, iteration, stage)` 派生 → 抽到的还是**同样那批题**； +那一轮没有判完，所以它的成绩**不会**写回 variant（写回只发生在"轮次判完"或"harness 被人工批准"时）。 +已经跑完的子任务证据留在磁盘上，但同样不计入。 + +### 9.4 不要做什么 + +| 做法 | 后果 | +|---|---| +| `kill -9 <编排器 pid>`（任何"只杀进程"） | 看起来像崩溃 → janitor 在 ≤5 分钟内 `supervisor+resume` 自动拉起；子任务在各自 session 里**不会**被杀，变成孤儿继续占卡 | +| shell 任务页的 **Kill** 按钮 | 它会按路径杀整棵树（比裸 kill 干净），但写的是 `final_status: killed`，**不在终态集合里**；没有 operator-stop 标记时，janitor 仍会按"未达目标轮数"把它拉起来 | +| shell 任务页的 **Reset** | 那是真删除：日志 / 迭代产物 / 运行记录全清，只留 `requirements.json`，不可恢复 | +| 手删 `operator_stop.json` | 等于解除暂停保护：下一次 janitor tick 就会把它当崩溃恢复 | + +### 9.5 代码对应 + +| 事项 | 位置 | +|---|---| +| 停止的六步顺序 | `server/routes.py: stop` | +| 宽口径活性判断（orchestrator.pid → resume.pid → cmdline） | `server/routes.py: _run_alive`（窄口径 `_resume_running` 只认 `resume.pid`，New Task 起的任务没有它） | +| 子任务定位（各自独立 session，靠 cmdline 认领） | `server/routes.py: _live_dkao_children` | +| 信号发送（先 `killpg` 再 `kill`，僵尸算已停） | `server/routes.py: _terminate_pid` | +| 终态集合 / 自动复活判据 | `server/routes.py: TERMINAL_STATUSES` / `should_launch_resume` | +| 续跑并清除标记 | `server/routes.py: _launch_resume`、`orchestrator/cli.py: _clear_operator_stop` | +| 契约测试 | `tests/test_operator_stop.py` | diff --git a/metainfer/tasks/harness_evolve/README.md b/metainfer/tasks/harness_evolve/README.md new file mode 100644 index 00000000..85b645e0 --- /dev/null +++ b/metainfer/tasks/harness_evolve/README.md @@ -0,0 +1,197 @@ +# harness_evolve（v0）— AHE 外层循环（headless） + +把 dcu_kernel_auto_opt 任务实例当评测单元跑 AHE evaluate→analyze→evolve 的 +"第一版可用"实现。设计依据： + +- **协议（固定，先读这个）：[`FLOW.md`](FLOW.md)** —— 每轮题数、取题规则、两门判据、 + HE 与 DKAO 的边界；代码必须与它一致（`tests/test_flow_contract.py` 守着） +- `MetaInfer/docs/dkao_harness_eval_protocol.md`（评测协议 v2） +- `MetaInfer/docs/ahe_dkao_integration_plan.md`（接入架构） +- 官方参考：`/root/zth_agent/ahe-ref`（MIT） + +## 变体轮次协议（固定，规范文本见 [`FLOW.md`](FLOW.md)） + +**协议以 [`FLOW.md`](FLOW.md) 为准**：每轮题数固定为 4 的倍数（4 张 HCU 一波铺满）、 +第 1 轮随机抽 A1 并**固化为泛化考卷**、性能门 75% + 80% 地板、泛化门（仅性能门通过后开） +50% + 80% 地板、过关后等人工批准；**HE 只发起 DKAO 任务，不管理环境**。 +本节只留实现要点；与 `FLOW.md` 不一致时以 `FLOW.md` + 契约测试为准。 + +**停止 / 续跑怎么操作**（按钮做了什么、为什么"裸杀进程"会被自动复活、哪些做法不可逆）： +见 [`FLOW.md`](FLOW.md) §9「停止与续跑（operator 操作说明）」。一句话：**页面的「停止任务」= 暂停**， +产物保留、不会被自动拉起，之后用「续跑到该轮次上限」接着跑；**直接 kill 进程或 shell 的 Kill 按钮 +都不等于暂停**。 + +语义（先统一）： + +- **baseline**：每个算子固定的 Triton 基线（`baseline_us`），永不变。 +- **variant**：每个算子**当前的最优解**（成绩 + 内核），是候选必须打败的对手； + **只在人工批准接入某一版 harness 时更新**（`variant_table.json`）。 +- **harness**：搜索策略（planner/gates）。它只影响 DKAO 怎么找内核，不参与测量。 + + +**阶段（stage）**：一次通过循环叫一个 stage，`runs/iteration_NNN/input/stage.json` 记录它是哪一个： + +| stage | 轮次编号 | 做什么 | 判哪门 | 预算 | +|---|---|---|---|---| +| **baseline** | iteration 1 | 随机抽 4 题（= `A1`），交给 DKAO 实测；这 4 题**固化为泛化考卷** | **都不判** | 0 | +| **performance** | iteration 2 起 | 重新随机抽 4 题（`A2`，排除考卷），交给 DKAO 实测 | **只判性能门** | 1.0 | +| **generalization** | 与上手**同一编号** | 过关后**再用同一版 harness 发动一次 DKAO**，题目固定为 `A1` | **只判泛化门** | 0.5 | + +- **第 1 轮是基线轮**：它不判任何门、不产生 `pending_promotion.json`，但**它测出来的更快 kernel + 直接写入 variant 池**（`promote_kernels_for_round`，阈值与生产一致：`promote_min_improvement_percent`， + 默认 3%；写池前自动备份池文件并追加 history）。这是"kernel 结果是事实，harness 才需要人批准"的分离。 + 第 1 轮若**没能对每个算子都拿到可用测量**（子任务崩溃 / DKAO 自己的 GPU 门拒测 / 无 report） + → 整个 HE 任务**立即停止**（`final_status=baseline_round_failed`），不留半截证据。 +- **性能门**：随机抽的那 4 题**必须真正快于**各自 variant（严格优于，含 2% 噪声带）——≥75% WIN + （4 题即至少 3 题），且没赢的不得低于 variant 的 80%。平局**不算赢**。 +- **泛化门**：重考考卷那 4 题，≥50% WIN，同样 80% 地板。**平局不算输**：DKAO 子任务是 + `warm_start: best_known` 起跑，考卷上的算子正是基线轮刚写进池的 kernel，重考大概率复现同一个 + kernel（实测 delta = 0.0%）；把它判成失败等于要求"搜索策略每轮都比自己上一轮更快"。 + 所以泛化门要求的是**无败场 + 真的退步就否决**（任一算子低于 variant 的 80% → 不过）。 +- **过关后**：写 `pending_promotion.json`，任务停在 `awaiting_approval`，**等你人工批准**才把 harness + 接进生产 DKAO。任何一门不过 → 回退到"当前已发布/上一好 harness"、analyze→evolve、进入下一轮 + 重新抽题（上限 `max_rounds`，默认 10；泛化轮算 0.5 轮，所以一个"过关再重考"的完整循环消耗 1.5 轮）。 +- **harness 与 DKAO 的边界**：HE 不执行任何评测本身，它只对每个算子**起一个 DKAO 子任务** + （`dcu_kernel_auto_opt.orchestrator.cli run`），把当前 harness 快照（`METAINFER_HARNESS_ROOT`）和 + 本轮算子清单交过去；测量、GPU 门、contamination 全在 DKAO 内。泛化轮就是**再发一次**这样的任务， + 只是题目固定为考卷 —— 代码上只是再调一次 `evaluator.evaluate()`，DKAO 一行不用改。 + 每一个"评估尝试"都有独立的子任务目录（`children/iteration_NNN`、重试 `..._a2`、重考 `..._generalization`）， + 保证重考是"重新发动一次 DKAO"、重试也是新的一次 DKAO，而不是接着上一批继续跑。 + 这一点是必须的：DKAO 会把子任务工作区的 `main` 符号链接绑死到它本次要用的 kernel repo， + 复用同一个目录会让重试**秒退**（`workspace main already points to ...`）， + 于是环境一抖就烧完整个重试预算、看起来像永久失败。重试目录独立后， + 失败那次尝试的日志与状态也原样留在旁边作为证据。 + +**环境（卡）不是 HE 的事**：卡被占用、闸门怎么等、污染测量怎么作废，全部由 DKAO 的子进程 +自己处理（`ensure_measurement_gate` / contamination），判据文本见 `FLOW.md` §0 与 §5。 +HE 过去在父层做过的三件事都已移除：复查卡状态、`SIGSTOP` 冻结子任务、 +按空闲卡数缩小题数（`fit_questions_to_devices`）——冻结并不会把卡让给别人， +缩题数只会让一轮考不出结论。 + +**机制门**（宽松）：每轮对上一轮 `change_manifest` 声明的机制做 grading，结果写进 +`decision.json.mechanism_gate` 与时间线（`mechanism_gate` 事件）；它**不否决**已通过两门的候选， +但"晋升了却说不清为什么"会留痕。第 1 轮（baseline）没有门可 grade，其 `mechanism_gate` 为空。 + +**批准后的动作**： +- `approve`：把候选 harness 发布进 DKAO（备份上一版 + `promoted_harness.json`）、 + 用**被接受那一轮**（写泛化门的那次重考）的成绩更新 variant、把该轮算子的内核写入生产池 + （逐算子"最好者胜"：低于池内 `best_known_us` 才写入，先备份池文件），任务结束 + （`final_status=promoted:<版本>`）。**只有被接受那一轮的优化进入生产，其余轮次丢弃。** +- `deny`：丢弃候选，恢复迭代。 +- `rollback-harness --to last-good|h-N|seed`：把生产版本回退。 + +命令行： + +```bash +python -m metainfer.tasks.harness_evolve.orchestrator.cli pending --state-dir --workspace-dir +python -m metainfer.tasks.harness_evolve.orchestrator.cli approve --state-dir --workspace-dir [--by NAME] +python -m metainfer.tasks.harness_evolve.orchestrator.cli deny --state-dir --workspace-dir [--reason TEXT] +python -m metainfer.tasks.harness_evolve.orchestrator.cli rollback-harness --state-dir --workspace-dir --to last-good +``` + +WebUI：`GET /promotion` 看待批准候选与生产版本，`POST /promotion/approve|deny` 做决定。 + +产物：`variant_table.json`（逐算子 variant + 泛化考卷）、`variant_history.jsonl`（谁把哪个 +算子刷成了多少）、`runs/iteration_NNN/input/stage.json`（这次是哪个 stage）、 +`.../baseline_round.json`（第 1 轮：判定 + 入池结果）、`.../performance_gate.json`、 +`.../generalization_gate.json`、`.../decision.json`、`pending_generalization.json`（冻结的重考请求）、 +`pending_promotion.json`、`promoted_harness.json`、 +`promotion_log.jsonl`；内核入库会更新池文件的 `best_known_us`/`history`，池文件备份在 +`/promoted/.<时间戳>.yaml`。 +DKAO 子任务目录：`children/<尝试目录>/<算子>`，其中尝试目录为 `iteration_NNN`（首次）、 +`iteration_NNN_a2`（环境失败后的第 2 次尝试，最多 `env_retry_attempts` 次）、 +`iteration_NNN_generalization`（泛化重考）；任何两个尝试都不共用目录。 + +回滚修复（本轮）：`workspace_revision` 现在记录**本轮自己的** commit（此前读的是上一轮 +manifest 的字段，恒为空）；`champion_override.json` 缺失时不再抛异常。 + +## HE 不管理 GPU（2026-09-22 起，operator 规则） + +**HE 只发起 DKAO 任务，卡的事归 DKAO。** 判据文本见 [`FLOW.md`](FLOW.md) §0。 + +- HE 不租卡、不读卡状态、不因为"卡忙"等待或否决、不冻结子进程。 +- 卡干不干净由 **DKAO 子进程**在测量闸门里判（VRAM ≤ 90% 且 HCU == 0；30 分钟一次、 + 最长 24 小时），写在子任务的 `state/measurement_gate.jsonl`——那是唯一存在闸门判定的地方。 +- HE 侧只剩每轮的**设备布局**记录：`runs/iteration_NNN/input/benchmark/gpu_preflight.json` + （哪道题绑了哪张卡、`index % 4`）。它是布局快照，不是占用检查。 +- 子任务被 DKAO 闸门拒测（exit 75）→ HE **不重发**，本轮判 `round_incomplete` + （cause `gpu_gate_blocked`，终态，等人工决定）；子任务崩溃 / 工具链缺失 → HE 重发一次 + （`env_retry_attempts`）。 +- 逃生阀：`METAINFER_GPU_PREFLIGHT=0` 关的是 **DKAO 侧**的闸门。 + +## 运行（dry-run，离线可跑通闭环） + +```bash +# 1) 写 requirements.json（answers 含 suite_yaml / max_iterations / execution_mode） +# answers.execution_mode: dry-run（默认，确定性 fixture）| dkao-cli（需 worker29） +# 2) headless 运行（框架契约） +python3 -m metainfer.tasks.harness_evolve.orchestrator.cli run \ + requirements.json --state-dir --workspace-dir +``` + +## 目录布局（每轮双代际，同 ahe-ref 语义） + +``` +/ # = AHE experiment root +├── workspace/ # 可演化 harness（从 dcu harness_default 播种） +├── runs/iteration_NNN/ +│ ├── input/workspace/ # 本轮评测快照 +│ ├── input/benchmark/results.json +│ ├── input/diff.json # vs 上轮（flipped/regressed） +│ ├── input/change_evaluation.json # 上轮 change_manifest 的归因（verdicts） +│ ├── input/analysis/overview.md +│ └── evolve/change_manifest.json # 本轮 Evolve 输出 +├── best_ever.json # 超过则更新；低于则自动回滚 workspace +├── iteration_scores.jsonl +└── report.md +``` + +## Pool v2（考题协议 v2，2026-09-09） + +- 新增 `orchestrator/pool.py`（注册池 + τ_family 自动判过 + 分层抽样/heldout）与 + `orchestrator/round_plan.py`（单角色出卷 + 三条护栏：池内 / 与上轮重叠 ≥60% / budget）。 +- 表单新增 `pool_source`（`builtin` | 池 yaml 路径）与 `per_round_budget`；置空 = 旧 suite 模式（行为不变）。 +- pool 模式下每轮 evaluate 只跑 round_plan 选中的题，`evolve/round_plan.json` 输出下轮考卷， + 决策（best_ever/回滚/归因）用重叠对口径；详见 `MetaInfer/docs/dkao_suite_protocol_v2.md`。 + +## 真实接入 v1（2026-09-09） + +- `decision_engine.py`：BASELINE / PROMOTE / PROMOTE_UNEXPLAINED / CONFIRM_REQUIRED / NO_SIGNAL / SPECIALIZE / REJECT / REJECT_OVERFIT；性能门优先、机制门解释、held-out 可专用化。 +- `DkaoCliEvaluator`：每题一个真实 DKAO 子任务，并发<=4，强制 `METAINFER_KERNEL_REPOS=/root/zth_agent/ahe-kernel-repos`；解析 final_report 与 child/repo/性能元数据。 +- `AgentEvolver`：单角色 DSH Agent 同时修改 harness、写完整 change_manifest、出下一轮 round_plan；确认轮保持候选字节不变。 +- 前端：每轮显示 system decision、child task/repo、median/p90、change attribution。 + +- 实测注册池构建：`python3 -m metainfer.tasks.harness_evolve.orchestrator.pool_builder` + 只读扫描历史 final_report → `/root/zth_agent/ahe-kernel-repos/registered_pool.yaml`（34 实例）。 +- dkao-cli 模式拒绝 builtin 假池；必须使用实测池文件。 +- planner 策略已数据化：`dcu_kernel_auto_opt/harness_default/planner_policy.yaml`（wired:true）， + AgentEvolver 改它即可改变下一轮状态→方案选择。 + +## 现状与边界（2026-09-22） + +运行方式与判据：**协议见 [`FLOW.md`](FLOW.md)**（固定文本，契约测试 `tests/test_flow_contract.py` 守着）。 + +- ✅ **变体轮次协议是默认路径**：每轮 4 的倍数题、第 1 轮固化泛化考卷、性能门 75% + 泛化门 50% + （各带 80% 地板）、过关后 `awaiting_approval` 等人工批准。实测状态：5 个 HE 任务里 + 1 个（`test-9-17-1-07bb5cd8`）两门通过并写出 `pending_promotion.json`，**至今未被批准**； + 全库尚无 `promoted_harness.json` —— 循环还没真正闭合过一次。 +- ✅ `dkao-cli` evaluator 是生产路径（不再是骨架）：每题一个真实 DKAO 子任务、并发 ≤4、 + `METAINFER_KERNEL_REPOS=/root/zth_agent/ahe-kernel-repos`、harness 快照经 + `METAINFER_HARNESS_ROOT` 传给子任务。 +- ✅ `AgentEvolver` 已实现（`evolve_mode: agent`，DSH Agent 改 harness 并写 change_manifest / + round_plan）；`evolve_mode: dry-run` 仍可用于离线闭环。 +- ✅ **HE 不管理 GPU**：不租卡、不读卡、不冻结子进程、不按空闲卡缩题（细节见 `FLOW.md` §0 与 + 下面「HE 不管理 GPU」节）。 +- ✅ 门有三种结论 **PASS / FAIL / INCOMPLETE**：缺测且结果仍开放时不给 harness 判词（`FLOW.md` §3）。 +- ⚠️ **测量噪声未标定**：门用 2% 噪声带、子任务用 `bench_profile=quick`，但"同一版 harness + 反复重测同一批题"的噪声基线还没有实测数据；判据基准（variant/pool 的 `best_known_us`） + 直接来自 DKAO 的测量，因此噪声会直接影响门的结论。 +- ⚠️ **待批候选没有提醒机制**：`awaiting_approval` 是终态，不批就一直停着（上面那个候选已躺 5 天）。 +- ⚠️ **旧路径（suite / legacy）保留但会自报家门**：它不判两门、不固化考卷，因此不可能产出可接入的 + harness；现在它会在日志里打 WARNING、在 timeline 写 `legacy_protocol_selected`、并在任务页面顶部 + 显示黄色横幅（细节见 `FLOW.md` §8）。彻底删掉这条路径仍是**未决**：删掉会一起失去 suite 模式 + （离线 dry-run 闭环）与"Evolve Agent 出下一轮考卷"（`round_plan.json`，默认协议下这份 plan 目前 + 只写不用）；在删除之前，这条告警就是防止它被误用的保障。 +- ⚠️ `harness_source` 默认指向 dcu_kernel_auto_opt 的 `harness_default/` 种子； + 指向 AHE 演化出的 workspace（git 仓库）即可继续"harness 自动进化"。 +- ⚠️ 历史文档 `docs/ahe_dkao_integration_plan.md` 等描述的是**父层管卡**的旧设计，已作废。 diff --git a/metainfer/tasks/harness_evolve/__init__.py b/metainfer/tasks/harness_evolve/__init__.py new file mode 100644 index 00000000..e711c2a3 --- /dev/null +++ b/metainfer/tasks/harness_evolve/__init__.py @@ -0,0 +1,13 @@ +"""harness_evolve — AHE outer-loop task type for MetaInfer. + +Drives dcu_kernel_auto_opt task instances as evaluation units, per +``MetaInfer/docs/dkao_harness_eval_protocol.md`` (evaluation protocol v2) and +``MetaInfer/docs/ahe_dkao_integration_plan.md``. + +This is the "first usable version": a headless orchestrator running the AHE +iteration layout with pluggable evaluators/evolvers (dry-run default; DKAO-CLI +and AgentEvolver reserved for worker29 validation), plus read-only WebUI routes. +""" + +from .orchestrator import plugin as _task_plugin # noqa: F401 +from .server import plugin as _web_plugin # noqa: F401 diff --git a/metainfer/tasks/harness_evolve/form.yaml b/metainfer/tasks/harness_evolve/form.yaml new file mode 100644 index 00000000..726b92b7 --- /dev/null +++ b/metainfer/tasks/harness_evolve/form.yaml @@ -0,0 +1,209 @@ +# harness_evolve（AHE outer loop）——真实测评任务表单。 +# +# 考卷不由人工填写：每轮考题由单角色 Evolve Agent 的 round_plan 从注册池选择， +# 控制面强制池内、重叠>=60%、预算封顶。评测只用真实 DKAO（dkao-cli）， +# 无 dry-run 虚拟测试选项。 + +- key: pool_source + question: "考题注册池 YAML（已接入、有实测基线的 GEMM 算子；dkao-cli 禁止使用内置假池）" + header: "Question pool" + required: false + form: file + default: "/root/zth_agent/ahe-kernel-repos/registered_pool.yaml" + +# 协议见 FLOW.md §1.1：题数固定为 4 的倍数（4 张 HCU 一波铺满），所以这里只给 +# 4 的倍数选项；CLI/接口传非倍数时由 normalized_per_gate 向上取整。 +- key: per_round_budget + question: "每轮考题数（固定为 4 的倍数：4 张 HCU 一波铺满）" + header: "Round budget" + required: false + form: select + default: "4" + options: + - label: "4" + description: "默认：一波 4 题，正好铺满 4 张卡" + - label: "8" + description: "两波共 8 题" + - label: "12" + description: "三波共 12 题" + - label: "16" + description: "四波共 16 题" + - label: "20" + description: "五波共 20 题（池子不够时按池子大小取）" + +# NOTE on the two numbers below — they are different layers and used to read +# alike, which is exactly how a run once ended up doing rounds nobody asked for: +# default_rounds = one question, inside DKAO: how many optimization rounds its +# worker may try (this is DKAO's own "Max iters"). +# rounds = the whole HE task: how many iterations it may run, one +# iteration = one round. +# The select field's options are the on-screen text (the renderer shows neither +# the question text nor per-option descriptions for a select), so the option +# labels stay plain numbers and the meaning lives in the question text. +- key: default_rounds + question: "【单题内部】每道考题交给 DKAO 后，它的 worker 最多做几轮内核优化——这是单题预算，不是任务轮次（下面 Iterations 才是）。含义：1=只做一次基线验证；3=快速试 3 轮；5=默认，与生产 DKAO 的 Max iters 一致；10/20=更充分但更慢。该值原样传给 DKAO 的 Max iters。" + header: "Max iters" + required: false + form: select + default: "5" + options: + - label: "1" + description: "只做一次基线验证（单题不迭代）" + - label: "3" + description: "快速试 3 轮" + - label: "5" + description: "默认：单题最多 5 轮，与生产 DKAO 一致" + - label: "10" + description: "单题最多 10 轮，更充分但更慢" + - label: "20" + description: "单题最多 20 轮，最慢" + +- key: rounds + question: "【任务上限，唯一的轮次字段】最多跑几轮，1 轮 = 1 次迭代。第 1 轮是基线轮（只测量、不判门）；之后每一轮 = 重抽一组新题判性能门，过关则用同一版 harness 重考考卷判泛化门（重考不额外占一轮）。任何一轮两门全过就立刻停下、等你批准接入生产；比如 Rounds=10 而第 5 轮就过了，任务就在第 5 轮结束。轮次用尽仍未过关则以 rounds_exhausted 结束。" + header: "Rounds" + required: false + form: number + default: "10" + +- key: harness_source + question: "Harness 起点：dcu_default = 内置种子；或指向已演化 workspace 的绝对路径" + header: "Harness src" + required: false + form: text + default: "dcu_default" + +- key: execution_mode + question: "评测方式（真实 DKAO 专用）" + header: "Eval mode" + required: false + form: select + default: "dkao-cli" + options: + - label: "dkao-cli" + description: "真实 DKAO GPU 评测（worker29 容器 + dsh agent）" + +- key: ahe_repo_root + question: "AHE 专用 kernel 仓库根（与人工 /root/zth_agent/kernel-repos 隔离）" + header: "AHE repos" + required: false + form: file + default: "/root/zth_agent/ahe-kernel-repos" + +- key: child_timeout_minutes + question: "单个 DKAO 考题子任务超时（分钟；0=不限）" + header: "Child timeout" + required: false + form: number + default: "0" + +- key: planner_enabled + question: "DKAO 子任务是否启用 planner（METAINFER_PLANNER=1）：只有开启后，被测 harness 的 planner_policy 才真正生效" + header: "Planner" + required: false + form: select + default: "true" + options: + - label: "true" + description: "注入 METAINFER_PLANNER=1：轮次指令由 harness 的 planner_policy/catalog 渲染（HE 演化 planner 的前提）" + - label: "false" + description: "使用 DKAO 默认菜单，planner_policy 改动不生效（仅用于对照/回退）" + +- key: validate_scope + question: "子任务最终验证范围：task = 只验证本题 shape（快）；api = 全量 API shape 回归（慢，生产默认）" + header: "Validate scope" + required: false + form: select + default: "task" + options: + - label: "task" + description: "只做本题 shape 的串行验证（HE 每题只有 1 个 shape，省去 42 个回归 shape 的固定开销）" + - label: "api" + description: "全量 API shape 验证（含未优化 shape 的回归）" + +- key: bench_profile + question: "benchmark 采样密度：quick = warmups30/samples30/replays50（稳定档）；full = 资产默认(100/30/100)" + header: "Bench profile" + required: false + form: select + default: "quick" + options: + - label: "quick" + description: "较快且稳定（warmups30/samples30/replays50），兼顾可比性" + - label: "full" + description: "资产默认的高密度采样" + +- key: mechanism_policy + question: "机制门严格度：lenient = 性能显著提升即可晋级（未验证记 PROMOTE_UNEXPLAINED，仅「矛盾证据」才要求复测）；strict = 无机制证据即要求复测；off = 不做机制判定" + header: "Mechanism gate" + required: false + form: select + default: "lenient" + options: + - label: "lenient" + description: "宽松：性能优先，机制命中/部分命中/未观测都可晋级；只有明确矛盾（如 harness 未生效、planner 被关闭）才复测一轮" + - label: "strict" + description: "保守：机制未验证就要求复测一轮" + - label: "off" + description: "关闭机制判定（只看性能）" + +- key: promote_kernels + question: "本轮评测出更快且正确的算子时，自动替换 DKAO 共享 variant（写前备份、可回滚）" + header: "Kernel promotion" + required: false + form: select + default: "true" + options: + - label: "true" + description: "自动把比现有 variant 快 ≥ 阈值的成品写回 DKAO variant 树" + - label: "false" + description: "只判定不写入（promotion.json 仍记录每题结果）" + +- key: promote_min_improvement_percent + question: "自动替换要求的最小提升幅度（相对现有 variant 的 median，%）" + header: "Promote threshold" + required: false + form: number + default: "3" + +- key: evolve_mode + question: "Evolve 模式：agent = 真实单角色 Evolve Agent 改 harness 并出下一轮考卷；dry-run 仅供调试占位" + header: "Evolve mode" + required: false + form: select + default: "agent" + options: + - label: "agent" + description: "真实 DSH Evolve Agent（改 harness + round_plan）" + - label: "dry-run" + description: "调试占位：只写 manifest、不改 harness" + +- key: evolve_model + question: "真实 Evolve Agent 模型 id（dsh 情况下可选；deepseek-flash-4.1 = 当前默认，deepseek-v4-flash = 旧 pinned 构建）" + header: "Evolve model" + required: false + form: select + default: "deepseek/deepseek-flash" + options: + - label: "deepseek/deepseek-flash" + description: "DeepSeek Flash 4.1（表单标签 deepseek-flash-4.1）——当前默认" + - label: "deepseek/deepseek-v4-flash-0731" + description: "DeepSeek V4 Flash 旧 pinned 构建（复现历史实验时选它）" + +- key: evolve_timeout_seconds + question: "真实 Evolve Agent 单轮超时（秒）" + header: "Evolve timeout" + required: false + form: number + default: "3600" + +- key: agent_framework + question: "DKAO 子任务的 Agent framework" + header: "Agent fw" + required: false + form: select + default: "dsh" + options: + - label: "dsh" + description: "DeepSeek Harness" + - label: "ccb" + description: "Claude Code" diff --git a/metainfer/tasks/harness_evolve/orchestrator/__init__.py b/metainfer/tasks/harness_evolve/orchestrator/__init__.py new file mode 100644 index 00000000..e219a487 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/__init__.py @@ -0,0 +1,7 @@ +"""harness_evolve orchestrator package.""" + +from metainfer.orchestrator.tasks import register + +from .plugin import PLUGIN + +register(PLUGIN) diff --git a/metainfer/tasks/harness_evolve/orchestrator/adapters/__init__.py b/metainfer/tasks/harness_evolve/orchestrator/adapters/__init__.py new file mode 100644 index 00000000..8eb218d7 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/adapters/__init__.py @@ -0,0 +1 @@ +"""Evaluation and evolve adapters for harness_evolve (v0).""" diff --git a/metainfer/tasks/harness_evolve/orchestrator/adapters/eval.py b/metainfer/tasks/harness_evolve/orchestrator/adapters/eval.py new file mode 100644 index 00000000..9270459f --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/adapters/eval.py @@ -0,0 +1,834 @@ +"""Evaluation adapters for harness_evolve. + +Real evaluator contract: +- every question becomes one real dcu_kernel_auto_opt child task; +- artificial/production kernel repos are READ-ONLY seeds; +- all child output repos are forced under the isolated AHE repo root + (default /root/zth_agent/ahe-kernel-repos); +- per-child final_report / worker rounds / token budget are normalized for the + parent AHE benchmark and frontend. +""" + +from __future__ import annotations + +import concurrent.futures +import json +import os +import re +import subprocess +import sys +import time +import uuid + +from pathlib import Path +from typing import Any, Dict, List + +import yaml + +from ..config import ExperimentConfig, InstanceSpec +from ..process_control import ChildProcess, ProcessRegistry, spawn +from ..rounds import ( + CHILDREN_DIR, GPU_COUNT, STAGE_GENERALIZATION, STAGE_PERFORMANCE, + device_for_index, stage_dir_name, +) + + +def _round_stage(cfg: Any) -> str: + """Which stage this evaluation is: the pipeline sets ``answers[round_stage]``. + + Missing means "an old/legacy run": the performance stage is the shape the + adapter had before the stages existed, so that is the fallback. + """ + answers = getattr(cfg, "answers", None) or {} + try: + return str(answers.get("round_stage") or STAGE_PERFORMANCE) + except AttributeError: + return STAGE_PERFORMANCE + +#: The exit code DKAO's in-harness measurement gate uses when it has waited its +#: whole budget (``METAINFER_GATE_MAX_WAITS``, 48 x 30 min = 24 h) without ever +#: seeing a clean device (VRAM <= 90% and HCU == 0). Mirrors +#: ``metainfer/tasks/dcu_kernel_auto_opt/assets/w8a8_bench.py::GATE_EXIT_CODE``; +#: duplicated on purpose so the HE adapter does not import the child's asset +#: module just for a number, and asserted against it in the tests. +GATE_EXIT_CODE = 75 + + +class Evaluator: + name = "base" + + def evaluate( + self, + cfg: ExperimentConfig, + workspace: Path, + iteration: int, + instances: "List[InstanceSpec] | None" = None, + ) -> Dict[str, Dict[str, Any]]: + raise NotImplementedError + + +class DryRunEvaluator(Evaluator): + """Deterministic offline evaluator (tests / CI / protocol dry runs).""" + + name = "dry-run" + + def evaluate(self, cfg, workspace, iteration, instances=None): + targets = list(instances) if instances else list(cfg.suite) + fixture: Dict[str, Any] = dict(cfg.answers.get("dry_fixture") or {}) + # ``dry_fixture_all`` sets the default for questions the fixture does + # not name, so an offline run can be steered wholesale (used by the + # variant-round end-to-end test). + default_pass = cfg.answers.get("dry_fixture_all") + results: Dict[str, Dict[str, Any]] = {} + for idx, inst in enumerate(targets): + override = fixture.get(inst.id) + if override is None: + override = default_pass + passed = bool(override) if override is not None else (idx != 0) + threshold = float(inst.pass_median_us_le or 1.0) + scale = 0.8 if passed else 1.4 + median = round(threshold * scale, 3) + results[inst.id] = { + "status": "completed", + "passed": passed, + "correctness_ok": True, + "p90_ok": True, + "median_us": median, + "p90_us": median * 1.01, + "rounds_used": inst.budget_rounds, + "tokens": 0, + "reason": "dry-run fixture", + "child_task_id": f"dry-{iteration}-{inst.id}", + "repo_path": None, + "harness_revision": _harness_revision(workspace), + } + return results + + +class DkaoCliEvaluator(Evaluator): + """Real DKAO evaluator (one isolated child task per question).""" + + name = "dkao-cli" + + def __init__(self) -> None: + # live child handles, so a question can be terminated while it runs + # (see process_control.py). Freezing a child because "its" device was + # taken over is gone with the rest of HE's GPU management: the child's + # own gate never measures on a busy card in the first place. + self.children = ProcessRegistry() + + def evaluate(self, cfg, workspace, iteration, instances=None): + targets = list(instances) if instances else list(cfg.suite) + if not targets: + return {} + stage = _round_stage(cfg) + repo_root = Path( + str(cfg.answers.get("ahe_repo_root") or + "/root/zth_agent/ahe-kernel-repos") + ).expanduser().resolve() + repo_root.mkdir(parents=True, exist_ok=True) + # One DKAO child per question, under this *attempt's* own directory. + # Each attempt is a different DKAO run, so none of them may share a + # child workspace: DKAO binds that workspace to the exact kernel repo it + # was launched with, and a second run pointed at the same directory dies + # on its own ownership check before it does any work. + children_root = cfg.exp_dir / CHILDREN_DIR + children_root.mkdir(parents=True, exist_ok=True) + # HE does not manage GPU occupancy. Every question this round is handed + # to its own DKAO child immediately, pinned to a device by the fixed + # ``index % GPU_COUNT`` rotation; whether that device is free is DKAO's + # business, not ours. The child's measurement gate + # (``ensure_measurement_gate``: VRAM <= 90% and HCU == 0, 30 min per + # check, at most 24 h) waits for the card, and the child's own + # contamination handling discards a round measured on a shared device. + # HE therefore never takes a lease, never pre-checks the cards and never + # freezes a child for a device it does not own — it only records the + # round's device layout (``_save_preflight``) for audit. + # + # The wave exists for one reason only: not to launch more children than + # there are devices (a wave is at most GPU_COUNT questions), so N + # questions land as N / 4 sequential waves with one question per device + # in each. + layout = self._device_layout(cfg, iteration, targets) + self._save_preflight(cfg, iteration, layout) + self._append_timeline(cfg, "gpu_layout_fixed", { + "iteration": iteration, + "gpu_count": GPU_COUNT, + "devices": layout["devices"], + "managed_by": layout["managed_by"], + }) + + results: Dict[str, Dict[str, Any]] = {} + pending = list(targets) + max_attempts = int(cfg.answers.get("env_retry_attempts") or 3) + attempt = 0 + gate_blocked: List[str] = [] + while pending: + wave_size = min(GPU_COUNT, len(pending)) + wave = pending[:wave_size] + attempt += 1 + child_root = (children_root + / stage_dir_name(stage, iteration, attempt=attempt)) + child_root.mkdir(parents=True, exist_ok=True) + if attempt > 1: + # The retry owns a fresh directory, so the failed attempt's + # evidence -- its logs above all -- stays where it is. + self._append_timeline(cfg, "env_retry_attempt", { + "iteration": iteration, "stage": stage, "attempt": attempt, + "dir": child_root.name, + "questions": [inst.id for inst in wave], + }) + wave_results = self._run_wave(cfg, workspace, iteration, wave, + repo_root, child_root, + wave_size) + results.update(wave_results) + # A child that produced no usable number (build/toolchain/crash) is + # retried: the environment owes us a measurement. Two things are NOT + # retried: + # - a device the child's own gate refused to measure on (exit 75): + # DKAO already waited its whole 24 h budget on that card. HE has + # no cards to reassign, so relaunching would only ignore a verdict + # it has no standing to overrule. The round stops instead. + # - a contention window: gone entirely — HE no longer detects or + # reacts to contention, DKAO's gate and contamination handling do. + gave_up = [inst for inst in pending + if (wave_results.get(inst.id) or {}).get("gate_blocked")] + gate_blocked.extend(inst.id for inst in gave_up) + retry = [inst for inst in pending + if inst not in gave_up + and self._env_failed(wave_results.get(inst.id) or {})] + done = [inst for inst in pending if inst not in retry] + pending = [inst for inst in pending if inst not in done] + if not retry: + break + if attempt >= max_attempts: + # The budget counts attempts, not retries: with + # ``env_retry_attempts: 2`` a question runs at most twice. + self._append_timeline(cfg, "env_retry_exhausted", { + "iteration": iteration, "attempts": attempt, + "pending": [inst.id for inst in retry], + }) + break + pending = retry + self._stop_on_incomplete_round(cfg, iteration, targets, results, + reason=("gpu_gate_blocked" if gate_blocked + else "environment_failed")) + return results + + def _save_preflight(self, cfg: ExperimentConfig, iteration: int, + layout: Dict[str, Any]) -> None: + """Record the round's device *layout* next to the iteration's evidence. + + This is a layout record, not an occupancy check: it says which device + each question was handed to and that HE took no lease. Whether that + device was free at the time is answered by the child's own + ``measurement_gate.jsonl`` — the only place a gate verdict exists now. + """ + stage = _round_stage(cfg) + name = "gpu_preflight.json" if stage != STAGE_GENERALIZATION else \ + "gpu_preflight_generalization.json" + try: + bench_dir = (cfg.exp_dir / "runs" / f"iteration_{iteration:03d}" + / "input" / "benchmark") + bench_dir.mkdir(parents=True, exist_ok=True) + (bench_dir / name).write_text( + json.dumps(layout, ensure_ascii=False, indent=2), + encoding="utf-8") + except OSError: + pass + + @classmethod + def _device_layout(cls, cfg: ExperimentConfig, iteration: int, + targets: List[Any]) -> Dict[str, Any]: + """The fixed ``index % GPU_COUNT`` device assignment for one round.""" + devices = [device_for_index(idx) for idx, _ in enumerate(targets)] + return { + "schema": "he-gpu-layout/1", + "enabled": False, + "managed_by": "dcu_kernel_auto_opt", + "reason": ("harness_evolve does not manage GPU occupancy; DKAO's " + "measurement gate (VRAM<=90% and HCU==0) decides when a " + "device may be measured on"), + "leases": [], + "gpu_count": GPU_COUNT, + "iteration": iteration, + "devices": { + str(getattr(inst, "id", f"question_{idx}")): gpu + for idx, (inst, gpu) in enumerate(zip(targets, devices)) + }, + "assigned_devices": sorted(set(devices)), + } + + def _run_wave(self, cfg: ExperimentConfig, workspace: Path, iteration: int, + wave: List[Any], repo_root: Path, + child_root: Path, + wave_size: int) -> Dict[str, Dict[str, Any]]: + """Run one wave: one DKAO child per question, one question per device. + + The device is chosen by the fixed ``index % GPU_COUNT`` rotation rather + than by a reading of the cards: HE hands the task over, DKAO's own + measurement gate decides when the card is clean enough to time a + kernel. Nothing here watches the devices — a round that DKAO's gate + holds up is visible in the child's ``measurement_gate.jsonl``, not in a + verdict HE invents. + """ + results: Dict[str, Dict[str, Any]] = {} + if not wave: + return results + with concurrent.futures.ThreadPoolExecutor(max_workers=wave_size) as ex: + futures = { + ex.submit( + self._run_one, cfg, workspace, iteration, inst, + device_for_index(idx), + repo_root, child_root, None, + ): inst.id + for idx, inst in enumerate(wave) + } + for fut in concurrent.futures.as_completed(futures): + iid = futures[fut] + try: + results[iid] = fut.result() + except Exception as exc: # noqa: BLE001 + import traceback as _traceback + tb_text = _traceback.format_exc(limit=8) + results[iid] = { + "status": "exception", + "passed": False, + "correctness_ok": False, + "p90_ok": False, + "median_us": None, + "p90_us": None, + "reason": ( + f"evaluator exception: {exc}\n{tb_text}" + ), + "child_task_id": None, + "repo_path": None, + "harness_revision": _harness_revision(workspace), + } + return results + + #: statuses that mean "no usable measurement because the environment broke". + #: ``partial_worker_result`` is deliberately absent: the child may have + #: produced a valid measurement whose only problem is that final synthesis + #: did not land. Treating that as an environment failure requeued questions + #: that had already measured, and the retry then collided with the child's + #: own kernel repo — which is how a round ended up with zero medians. + ENV_FAILURE_STATUSES = ("missing_report", "exception", "timeout") + + #: statuses that mean "the workers measured, the round did not finish" + PARTIAL_STATUSES = ("partial_worker_result",) + + #: reason fragments that mean "the toolchain/environment broke", i.e. the + #: question deserves a retry rather than being counted as a bad kernel + ENV_FAILURE_HINTS = ("compile", "build failed", "ninja", "hipcc", + "undefined template", "no space left", + "outofmemory", "out of memory", "timed out", + "timeout", "killed") + + def _env_failed(self, result: Dict[str, Any], + *, retry: bool = False) -> bool: + """A failure caused by the environment (not by the candidate kernel). + + The rule is "no usable number => the environment owes us a measurement": + a child that could not be built, timed out or crashed is retried (30 + minutes of rest, up to 3 attempts). A child that *did* measure is not + retried — its number is evidence even when the run ended early (for + example ``partial_worker_result``: the workers measured, final synthesis + did not land). + + ``retry=True`` marks a re-measurement of a question that already ran + once, where even a measured partial is an environment problem: its + device was taken over mid-run, so the number describes a shared card. + """ + if not isinstance(result, dict): + return False + if result.get("passed") is True: + return False + status = str(result.get("status") or "").lower() + measured = result.get("median_us") not in (None, "") + if status in self.ENV_FAILURE_STATUSES: + return True + if measured: + # a real number: only a retry treats a partial finish as an + # environment problem, and only because the device was shared + return bool(retry and status in self.PARTIAL_STATUSES) + reason = str(result.get("reason") or "").lower() + if any(hint in reason for hint in self.ENV_FAILURE_HINTS): + return True + return True + + def _request_stop(self, cfg: ExperimentConfig, reason: str, + **fields: Any) -> None: + """Ask the outer loop to stop the whole experiment (terminal state).""" + try: + (cfg.exp_dir / "stop_requested.json").write_text( + json.dumps({"reason": reason, **fields}, ensure_ascii=False, + indent=2), encoding="utf-8") + except OSError: + pass + + #: statuses that mean "the round ended without a measurement" + INCOMPLETE_STATUSES = ENV_FAILURE_STATUSES + ("rejected", "failed") + + def _incomplete_ids(self, targets: List[Any], + results: Dict[str, Dict[str, Any]]) -> List[str]: + """Questions the round never measured (no number, or a broken run). + + A round that ran out of retries used to be recorded as "these + questions just failed", and the decisions downstream then compared + against whatever numbers happened to exist. That silently weakens the + evidence for a promotion, so the caller stops the run instead. + """ + out: List[str] = [] + for inst in targets: + iid = getattr(inst, "id", str(inst)) + res = results.get(iid) + if not isinstance(res, dict) or res.get("passed") is True: + continue + if not self._is_measured(res): + out.append(iid) + return out + + def _is_measured(self, result: Dict[str, Any]) -> bool: + """True when the result carries a usable number of its own.""" + if not isinstance(result, dict): + return False + median = result.get("median_us") + if median in (None, ""): + return False + status = str(result.get("status") or "").lower() + return status not in self.ENV_FAILURE_STATUSES + + def _stop_on_incomplete_round(self, cfg: ExperimentConfig, iteration: int, + targets: List[Any], + results: Dict[str, Dict[str, Any]], + *, reason: str) -> bool: + """True when the round must stop instead of deciding on partial data.""" + missing = self._incomplete_ids(list(targets), results or {}) + if not missing: + return False + unmeasured = sorted(set(missing)) + self._append_timeline(cfg, "round_incomplete", { + "iteration": iteration, "reason": reason, + "unmeasured": unmeasured, + "measured": sorted(set(results or {}) - set(unmeasured)), + }) + self._request_stop(cfg, "round_incomplete", iteration=iteration, + cause=reason, unmeasured=unmeasured) + return True + + def _append_timeline(self, cfg: ExperimentConfig, event: str, + payload: Dict[str, Any]) -> None: + try: + from metainfer.tasks.harness_evolve.orchestrator.state import ( + append_timeline, + ) + append_timeline(cfg.state_dir, event, payload) + except Exception: # noqa: BLE001 - progress records are best effort + pass + + def _run_one( + self, cfg: ExperimentConfig, workspace: Path, iteration: int, + inst: InstanceSpec, index: int, repo_root: Path, child_root: Path, + preflight: Optional[Dict[str, Any]] = None, + gpu_override: "int | None" = None, + ) -> Dict[str, Any]: + # Device layout: the machine exposes 4 GPUs; spread the concurrent + # children over 0..3 in submission order and repeat cyclically, so a + # 4-multiple question set keeps every card busy exactly once per wave. + # (worker id == GPU id is enforced by DKAO manual assignments, hence + # _build_dkao_requirements names the worker worker_.) + gpu = (int(gpu_override) if gpu_override is not None + else device_for_index(index)) + suffix = uuid.uuid4().hex[:6] + child_id = _slug( + f"ahe-{cfg.task_id}-it{iteration:03d}-candidate-{inst.id}-{suffix}" + ) + repo_name = child_id + state_dir = child_root / inst.id / "state" + work_dir = child_root / inst.id / "workspace" + state_dir.mkdir(parents=True, exist_ok=True) + work_dir.mkdir(parents=True, exist_ok=True) + req = self._build_dkao_requirements( + cfg, inst, iteration, child_id, repo_name, gpu, workspace + ) + req_path = child_root / inst.id / "requirements.json" + req_path.parent.mkdir(parents=True, exist_ok=True) + req_path.write_text(json.dumps(req, ensure_ascii=False, indent=2), + encoding="utf-8") + log_path = child_root / inst.id / "orchestrator-external.log" + + cmd = [ + _python_executable(), "-m", + "metainfer.tasks.dcu_kernel_auto_opt.orchestrator.cli", "run", + str(req_path), "--state-dir", str(state_dir), + "--workspace-dir", str(work_dir), + ] + env = _child_env(cfg, repo_root, workspace, + child_state_dir=state_dir) + timeout_min = float(cfg.answers.get("child_timeout_minutes") or 0) + timeout = timeout_min * 60 if timeout_min > 0 else None + started = time.time() + # Spawn (not run): the parent keeps a handle so it can terminate the + # child on a timeout. Freezing a child for "its" device is gone with the + # rest of HE's GPU management — that device is DKAO's to arbitrate. + child = spawn(cmd, cwd=str(Path(__file__).resolve().parents[5]), + env=env, log_path=log_path) + self.children.add(inst.id, child) + rc, timed_out = 0, False + try: + if timeout: + rc = child.proc.wait(timeout=timeout) + else: + rc = child.proc.wait() + except subprocess.TimeoutExpired: + child.terminate(reason=f"child timeout after {timeout:.0f}s") + rc, timed_out = 124, True + finally: + self.children.drop(inst.id) + try: + child.proc.wait(timeout=30) + except Exception: # noqa: BLE001 - already dead or unkillable + pass + terminated_by = child.terminate_reason + duration = time.time() - started + report_path = work_dir / "final_report.json" + report = _load_json(report_path) or {} + normalized = ( + _normalize_report(inst, report) + if report else _normalize_workspace_result(inst, work_dir) + ) + normalized.update({ + "child_task_id": child_id, + "parent_ahe_task_id": cfg.task_id, + "ahe_iteration": iteration, + "candidate_role": "candidate", + "repo_path": str(repo_root / repo_name), + "repo_root": str(repo_root), + "harness_revision": _harness_revision(workspace), + "child_state_dir": str(state_dir), + "child_workspace_dir": str(work_dir), + "log_path": str(log_path), + "duration_s": round(duration, 3), + "returncode": rc, + "timed_out": timed_out, + "gpu": gpu, + "terminated_by": terminated_by or None, + }) + # No ``gpu_preflight`` / ``measurement_suspect`` stamping here any more: + # HE takes no reading of the device it handed out, so it has no verdict + # to attach. The child records every admission check it made in + # ``/measurement_gate.jsonl``, which is what the HE + # detail page reads when it shows why a question is waiting. + # + # ``GATE_EXIT_CODE`` (75) is the child's *own* gate saying "I waited 48 + # x 30 min and no card ever became clean". That is an authoritative + # verdict, not a flaky environment: HE does not repeal it by launching + # the same child again, so it is marked and the round stops. + if rc == GATE_EXIT_CODE: + normalized["gate_blocked"] = True + normalized.setdefault("status", "environment_failed") + normalized["reason"] = ( + f"DKAO measurement gate gave up on device " + f"{gpu} after its full wait budget" + ) + # + # final_report.json does not carry a canonical rounds field (its shape + # changed over DKAO revisions), so fall back to the authoritative + # per-attempt records the workers wrote. + counted = _worker_rounds(work_dir) + if counted and int(normalized.get("rounds_used") or 0) < counted: + normalized["rounds_used"] = counted + if rc != 0 and normalized.get("status") == "missing_report": + normalized["reason"] = _tail(log_path) + return normalized + + def _build_dkao_requirements( + self, cfg: ExperimentConfig, inst: InstanceSpec, iteration: int, + child_id: str, repo_name: str, gpu: int, workspace: Path, + ) -> Dict[str, Any]: + s = inst.shape + shape_yaml = { + "model": str(s.get("model", "AHE registered GEMM")), + "assignment_mode": "manual", + "shape_scope": "subset", + "shapes": [{ + "id": inst.id, + "M": int(s["M"]), "N": int(s["N"]), "K": int(s["K"]), + "tp_size": int(s.get("tp_size", 8)), + "operator": str(s["operator"]), + }], + "assignments": { + # DKAO's manual mode requires worker_N to map to GPU N + # (gen_and_opt_pipeline validates worker_id == gpu). Name the + # single worker after the assigned GPU so parallel children + # spread across devices 0..3 without tripping the check. + f"worker_{gpu}": {"gpu": gpu, "shapes": [inst.id]}, + }, + } + return { + "task_id": child_id, + "task_type": "dcu-kernel-auto-opt", + "label": child_id, + "operator": "Quantized GEMM", + "kernel_language": "HIP C++", + "target_hardware": "K500SM_AI / gfx928", + "dtype": "INT8 W8A8", + "agent_framework": cfg.agent_framework, + # The child DKAO task runs on the same DSH model generation as the + # AHE evolve agent (agent_model is a DKAO form label, resolved to an + # id by DKAO's own orchestrator/config.py). + "agent_model": "deepseek-flash-4.1", + "execution_mode": "Generate & optimize (auto-create kernel repo)", + "target_repo_path": repo_name, + "model": str(s.get("model", "AHE registered GEMM")), + "shape_assignment_mode": "Manual by GPU", + "shape_scope": "Selected shapes only", + "shape_config": yaml.safe_dump(shape_yaml, sort_keys=False), + "correctness_ref": "PyTorch eager", + "perf_target": "AHE automatic baseline-relative criterion", + "max_iterations": str(inst.budget_rounds), + "minimum_improvement_percent": 0, + # Parent/trace metadata: ignored by old DKAO code, preserved in req. + "parent_ahe_task_id": cfg.task_id, + "ahe_iteration": iteration, + "harness_revision": _harness_revision(workspace), + "candidate_role": "candidate", + } + + +def _normalize_workspace_result(inst: InstanceSpec, work_dir: Path) -> Dict[str, Any]: + """Best-effort evidence when final synthesis/serial validate fails. + + Partial worker metrics are visible in the AHE frontend but never pass the + system gate (no final validation = candidate cannot be promoted). + """ + for path in sorted((work_dir / "workers").glob("*/result.json")): + data = _load_json(path) or {} + shape = (data.get("shapes") or {}).get(inst.id) or {} + metrics = shape.get("metrics") or {} + if not metrics: + continue + median, p90 = metrics.get("median_us"), metrics.get("p90_us") + correctness = bool( + metrics.get("graph_capture_passed") is True + and metrics.get("correctness_passed_in_precheck") is True + ) + rounds_path = path.parent / "runs" / inst.id / "experiments.jsonl" + rounds = 0 + if rounds_path.is_file(): + rounds = len(rounds_path.read_text( + encoding="utf-8", errors="replace" + ).splitlines()) + return { + "status": "partial_worker_result", + "passed": False, + "correctness_ok": correctness, + "p90_ok": bool(p90 is not None), + "median_us": float(median) if median is not None else None, + "p90_us": float(p90) if p90 is not None else None, + "rounds_used": rounds, + "tokens": None, + "new_best_known": False, + "reason": "final report missing; worker best preserved as evidence", + } + return { + "status": "missing_report", "passed": False, + "correctness_ok": False, "p90_ok": False, + "median_us": None, "p90_us": None, "rounds_used": 0, + "tokens": None, "reason": "final_report.json and worker result missing", + } + + +def _normalize_report(inst: InstanceSpec, report: Dict[str, Any]) -> Dict[str, Any]: + if not report: + return { + "status": "missing_report", "passed": False, + "correctness_ok": False, "p90_ok": False, + "median_us": None, "p90_us": None, "rounds_used": 0, + "tokens": None, "reason": "final_report.json missing", + } + final = (report.get("final_validation") or {}).get(inst.id) or {} + worker = (report.get("worker_validation") or {}).get(inst.id) or {} + metrics = final or (worker.get("metrics") or {}) + correctness = bool( + metrics.get("passed") is True + or ( + metrics.get("graph_capture_passed") is True + and (metrics.get("mismatch_count") in (None, 0)) + and report.get("status") == "success" + ) + ) + median = metrics.get("median_us") + p90 = metrics.get("p90_us") + return { + "status": str(report.get("status") or "unknown"), + "passed": bool(report.get("status") == "success" and correctness), + "correctness_ok": correctness, + "p90_ok": bool(p90 is not None), + "median_us": float(median) if median is not None else None, + "p90_us": float(p90) if p90 is not None else None, + "rounds_used": _count_rounds(report, inst.id), + "tokens": None, + "new_best_known": False, + "reason": "final_report parsed", + "final_report_status": report.get("status"), + } + + +def _count_rounds(report: Dict[str, Any], shape_id: str) -> int: + # final report does not currently expose a canonical rounds_used field; + # worker result/details remain available through child_workspace_dir. + workers = report.get("workers") or {} + for worker in workers.values(): + shape = (worker.get("shapes") or {}).get(shape_id) if isinstance(worker, dict) else None + if isinstance(shape, dict): + try: + return int(shape.get("iterations") or shape.get("rounds_used") or 0) + except (TypeError, ValueError): + pass + return 0 + + +def _harness_revision(workspace: Path) -> str: + manifest = _load_json(workspace / "manifest.json") + if not manifest and (workspace / "manifest.yaml").is_file(): + try: + manifest = yaml.safe_load( + (workspace / "manifest.yaml").read_text(encoding="utf-8") + ) or {} + except (OSError, ValueError): + manifest = {} + return str((manifest or {}).get("revision") or "unknown") + + +def _load_json(path: Path) -> Dict[str, Any] | None: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + return value if isinstance(value, dict) else None + + +def _slug(value: str) -> str: + value = re.sub(r"[^a-z0-9]+", "-", value.lower()).strip("-") + return value[:120] or f"ahe-child-{uuid.uuid4().hex[:8]}" + + +def _tail(path: Path, size: int = 2000) -> str: + try: + text = path.read_text(encoding="utf-8", errors="replace") + except OSError: + return "child failed; log unavailable" + return text[-size:] + + +def _truthy(value: Any, default: bool = False) -> bool: + if value is None: + return default + if isinstance(value, bool): + return value + return str(value).strip().lower() not in {"0", "false", "no", "off", ""} + + +def _child_env(cfg: ExperimentConfig, repo_root: Path, + workspace: Path, + child_state_dir: Optional[Path] = None) -> Dict[str, str]: + """Environment for one DKAO child task. + + ``METAINFER_PLANNER=1`` is what makes the evaluated harness' planner + actually drive DKAO's round instructions; without it the harness snapshot + is inert for planning decisions. The form switch (default on) keeps that + auditable. + + ``METAINFER_CHILD_STATE_DIR`` points the child's own measurement gate at a + state directory, so every "gate blocked / gate ok" check it makes while + iterating is auditable next to the round's other evidence. + """ + env = os.environ.copy() + env["METAINFER_KERNEL_REPOS"] = str(repo_root) + env["METAINFER_HARNESS_ROOT"] = str(workspace) + if child_state_dir is not None: + env["METAINFER_CHILD_STATE_DIR"] = str(child_state_dir) + if _truthy(cfg.answers.get("planner_enabled"), default=True): + env["METAINFER_PLANNER"] = "1" + else: + env.pop("METAINFER_PLANNER", None) + + # Validation budget: one question = one shape, so AHE defaults to the + # task-scoped serial validation with the quick bench profile (production + # DKAO tasks keep the full API sweep unless their form says otherwise). + answers = cfg.answers + scope = str(answers.get("validate_scope") or "task").strip().lower() + env["METAINFER_VALIDATE_SCOPE"] = scope if scope in {"api", "task"} else "task" + profile = str(answers.get("bench_profile") or "quick").strip().lower() + quick = {} + if profile in {"quick", "fast"}: + try: + from metainfer.tasks.dcu_kernel_auto_opt.orchestrator.validation_budget import ( # noqa: E501 + QUICK_BENCH, + ) + quick = dict(QUICK_BENCH) + except Exception: # noqa: BLE001 + quick = {"warmups": 10, "samples": 20, "replays_per_sample": 20} + explicit = { + "METAINFER_BENCH_WARMUPS": answers.get("bench_warmups"), + "METAINFER_BENCH_SAMPLES": answers.get("bench_samples"), + "METAINFER_BENCH_REPLAYS": answers.get("bench_replays"), + } + if quick: + env["METAINFER_BENCH_WARMUPS"] = str(quick["warmups"]) + env["METAINFER_BENCH_SAMPLES"] = str(quick["samples"]) + env["METAINFER_BENCH_REPLAYS"] = str(quick["replays_per_sample"]) + for name, value in explicit.items(): + if value not in (None, ""): + try: + env[name] = str(int(value)) + except (TypeError, ValueError): + pass + return env + + +def _worker_rounds(work_dir: Path) -> int: + """Rounds actually attempted, from each worker's experiments.jsonl.""" + total = 0 + workers = work_dir / "workers" + if not workers.is_dir(): + return 0 + for exp_file in workers.glob("*/runs/*/experiments.jsonl"): + try: + total += len(exp_file.read_text( + encoding="utf-8", errors="replace").splitlines()) + except OSError: + continue + return total + + +def _python_executable() -> str: + """A reliable python binary for spawning DKAO child orchestrators. + + ``sys.executable`` can be ``''`` when the WebUI launcher renames argv[0] + to a friendly process name (``metainfer-orchestrator``) via Popen's + ``executable=`` split: CPython then cannot resolve its own path, and + spawning a child with the empty string fails with + ``PermissionError [Errno 13] Permission denied: ''``. Prefer the explicit + ``METAINFER_PYTHON`` override, then a valid ``sys.executable``, then + ``python3`` on PATH. + """ + import shutil + + env_py = str(os.environ.get("METAINFER_PYTHON") or "").strip() + if env_py: + return env_py + exe = str(sys.executable or "").strip() + if exe and os.path.isfile(exe) and os.access(exe, os.X_OK): + return exe + for cand in ("python3", "python"): + found = shutil.which(cand) + if found: + return found + return "/usr/bin/python3" + + +def build_evaluator(mode: str) -> Evaluator: + return DkaoCliEvaluator() if mode == "dkao-cli" else DryRunEvaluator() diff --git a/metainfer/tasks/harness_evolve/orchestrator/attribution.py b/metainfer/tasks/harness_evolve/orchestrator/attribution.py new file mode 100644 index 00000000..440ff0c3 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/attribution.py @@ -0,0 +1,134 @@ +"""Attribution / flip / rollback helpers (ported semantics from ahe-ref).""" + +from __future__ import annotations + +import json +import shutil +from pathlib import Path +from typing import Any, Dict, List + + +def pass_rate(results: Dict[str, Dict[str, Any]]) -> Dict[str, Any]: + items = list(results.values()) + n = len(items) + n_pass = sum(1 for r in items if r.get("passed") is True) + return { + "n_total": n, + "n_pass": n_pass, + "pass_rate": (n_pass / n) if n else 0.0, + } + + +def diff_results( + prev: Dict[str, Dict[str, Any]] | None, + curr: Dict[str, Dict[str, Any]], +) -> Dict[str, Any]: + """Cross-iteration diff (bool-level; three-state noise lives in real eval). + + flipped: fail -> pass regressed: pass -> fail unchanged otherwise. + """ + prev = prev or {} + flipped: List[str] = [] + regressed: List[str] = [] + for inst_id, r in curr.items(): + p_ok = bool((prev.get(inst_id) or {}).get("passed") is True) + c_ok = bool(r.get("passed") is True) + if not p_ok and c_ok: + flipped.append(inst_id) + elif p_ok and not c_ok: + regressed.append(inst_id) + return { + "flipped": sorted(flipped), + "regressed": sorted(regressed), + "net": len(flipped) - len(regressed), + } + + +def evaluate_changes(manifest: Dict[str, Any], diff: Dict[str, Any]) -> Dict[str, Any]: + """change_manifest -> per-change verdicts (schema mirrors ahe-ref).""" + flipped_set = set(diff.get("flipped", [])) + regressed_set = set(diff.get("regressed", [])) + all_predicted = set() + all_risk = set() + evaluations: List[Dict[str, Any]] = [] + + for chg in manifest.get("changes", []): + predicted = list(chg.get("predicted_fixes", []) or []) + risks = list(chg.get("risk_tasks", []) or []) + all_predicted.update(predicted) + all_risk.update(risks) + actually_fixed = [t for t in predicted if t in flipped_set] + still_failed = [t for t in predicted if t not in flipped_set] + risk_realized = [t for t in risks if t in regressed_set] + n_fixed = len(actually_fixed) + n_pred = len(predicted) + n_risk = len(risk_realized) + + if n_risk > 0 and n_fixed == 0: + verdict = "HARMFUL" + elif n_risk > 0 and n_fixed > 0: + verdict = "MIXED" + elif n_pred and n_fixed == n_pred: + verdict = "EFFECTIVE" + elif n_fixed > 0: + verdict = "PARTIALLY_EFFECTIVE" + else: + verdict = "INEFFECTIVE" + + evaluations.append({ + "change_id": chg.get("id", "unknown"), + "description": chg.get("description", ""), + "files": list(chg.get("files", []) or []), + "predicted_fixes": predicted, + "actually_fixed": actually_fixed, + "still_failed": still_failed, + "predicted_risks": risks, + "risk_realized": risk_realized, + "hit_rate": f"{n_fixed}/{n_pred}" if n_pred else "0/0", + "verdict": verdict, + }) + + unattributed = sorted( + t for t in regressed_set if t not in (all_predicted | all_risk) + ) + return { + "evaluating_iteration": int(manifest.get("iteration", 0)), + "change_evaluations": evaluations, + "unattributed_regressions": unattributed, + "summary": ", ".join( + f"{e['change_id']}: {e['verdict']}" for e in evaluations + ), + } + + +def copy_tree(src: Path, dst: Path) -> None: + dst.mkdir(parents=True, exist_ok=True) + for item in src.iterdir(): + if item.name == ".git": + continue + target = dst / item.name + if item.is_dir(): + shutil.copytree(item, target, dirs_exist_ok=True) + else: + shutil.copy2(item, target) + + +def replace_tree(src: Path, dst: Path) -> None: + """Replace dst's non-git content with src (true rollback semantics).""" + dst.mkdir(parents=True, exist_ok=True) + for item in list(dst.iterdir()): + if item.name == ".git": + continue + if item.is_dir(): + shutil.rmtree(item) + else: + item.unlink() + copy_tree(src, dst) + + +def save_json(path: Path, payload: Dict[str, Any]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2), + encoding="utf-8") + tmp.replace(path) diff --git a/metainfer/tasks/harness_evolve/orchestrator/champion_selection.py b/metainfer/tasks/harness_evolve/orchestrator/champion_selection.py new file mode 100644 index 00000000..da5e72f2 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/champion_selection.py @@ -0,0 +1,211 @@ +"""Generation-level champion selection for harness_evolve. + +The decision engine compares the candidate harness against a *champion*. Using +"the last generation that happened to be PROMOTEd" is wrong: a generation can +hold the best measured result and still not be promoted in its own round (its +change needed confirmation, or only one question won). If the next generation +is worse but does get promoted, that worse harness would take the champion +slot. + +This module keeps one record per evaluated generation and picks the champion by +*measured merit*: + +1. generations whose verdict was REJECT / REJECT_OVERFIT are disqualified + (they were proven not to generalise or to regress); +2. more passing questions wins (absolute count on each generation's own set); +3. otherwise, on the questions the two generations share, more WINs wins; +4. otherwise the generation with the better cross-shape median wins; +5. ties keep the earlier generation (stability over novelty). + +The champion is therefore "the best harness we have measured so far", which is +what every later round must beat — even if that generation was never promoted +at the time. +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Mapping, Optional + +GENERATIONS_FILE = "generations.jsonl" +_REJECTED_VERDICTS = {"REJECT", "REJECT_OVERFIT"} +_PENDING_VERDICTS = {"CONFIRM_REQUIRED"} + + +@dataclass +class Generation: + iteration: int + verdict: str = "" + results: Dict[str, Dict[str, Any]] = field(default_factory=dict) + snapshot_dir: str = "" + workspace_revision: str = "" + pass_count: int = 0 + rejected: bool = False + pending: bool = False + + def __post_init__(self) -> None: + # the verdict decides qualification; keep the two flags in sync + if self.verdict in _REJECTED_VERDICTS: + self.rejected = True + if self.verdict in _PENDING_VERDICTS: + self.pending = True + + def to_dict(self) -> Dict[str, Any]: + return { + "iteration": self.iteration, + "verdict": self.verdict, + "pass_count": self.pass_count, + "rejected": self.rejected, + "pending": self.pending, + "snapshot_dir": self.snapshot_dir, + "workspace_revision": self.workspace_revision, + } + + @classmethod + def from_dict(cls, data: Mapping[str, Any], + results: Optional[Mapping[str, Any]] = None) -> "Generation": + verdict = str(data.get("verdict") or "") + return cls( + iteration=int(data.get("iteration") or 0), + verdict=verdict, + results=dict(results or {}), + snapshot_dir=str(data.get("snapshot_dir") or ""), + workspace_revision=str(data.get("workspace_revision") or ""), + pass_count=int(data.get("pass_count") or 0), + rejected=bool(data.get("rejected", verdict in _REJECTED_VERDICTS)), + pending=bool(data.get("pending", verdict in _PENDING_VERDICTS)), + ) + + +def _median(row: Mapping[str, Any]) -> Optional[float]: + try: + value = float(row.get("median_us")) + except (TypeError, ValueError): + return None + return value if value > 0 else None + + +def load_generations(exp_dir: Path) -> List[Generation]: + """Every evaluated generation, oldest first, with its measured results.""" + exp = Path(exp_dir) + path = exp / GENERATIONS_FILE + records: List[Dict[str, Any]] = [] + if path.is_file(): + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict): + records.append(row) + out: List[Generation] = [] + for row in sorted(records, key=lambda r: int(r.get("iteration") or 0)): + iteration = int(row.get("iteration") or 0) + results_path = (exp / "runs" / f"iteration_{iteration:03d}" / "input" + / "benchmark" / "results.json") + results: Dict[str, Any] = {} + if results_path.is_file(): + try: + results = json.loads(results_path.read_text(encoding="utf-8")) or {} + results = results.get("results", results) + except (OSError, ValueError): + results = {} + out.append(Generation.from_dict(row, results)) + return out + + +def record_generation(exp_dir: Path, generation: Generation) -> None: + """Append (or replace) one generation record.""" + exp = Path(exp_dir) + path = exp / GENERATIONS_FILE + rows: List[Dict[str, Any]] = [] + if path.is_file(): + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict) and int(row.get("iteration") or 0) != generation.iteration: + rows.append(row) + rows.append(generation.to_dict()) + rows.sort(key=lambda r: int(r.get("iteration") or 0)) + path.write_text( + "\n".join(json.dumps(r, ensure_ascii=False) for r in rows) + "\n", + encoding="utf-8") + + +def _pass_count(results: Mapping[str, Mapping[str, Any]]) -> int: + return sum(1 for row in results.values() if row.get("passed") is True) + + +def _wins(a: Mapping[str, Mapping[str, Any]], + b: Mapping[str, Mapping[str, Any]], noise_percent: float) -> int: + """How many questions generation ``a`` beats generation ``b`` on.""" + wins = 0 + for iid in set(a) & set(b): + ma, mb = _median(a[iid]), _median(b[iid]) + if ma is None or mb is None: + continue + delta = (mb - ma) / mb * 100.0 # positive => a is faster + if delta > noise_percent: + wins += 1 + return wins + + +def compare_generations(a: Generation, b: Generation, *, + noise_percent: float = 2.0) -> str: + """Return ``\"a\"``, ``\"b\"`` or ``\"tie\"`` (a wins / b wins / no winner).""" + if a.rejected and not b.rejected: + return "b" + if b.rejected and not a.rejected: + return "a" + a_pass, b_pass = _pass_count(a.results), _pass_count(b.results) + if a_pass != b_pass: + return "a" if a_pass > b_pass else "b" + a_wins = _wins(a.results, b.results, noise_percent) + b_wins = _wins(b.results, a.results, noise_percent) + if a_wins != b_wins: + return "a" if a_wins > b_wins else "b" + # shared-question medians: compare the geometric mean of ratios + shared = [iid for iid in set(a.results) & set(b.results) + if _median(a.results[iid]) and _median(b.results[iid])] + if shared: + ratio = 1.0 + for iid in shared: + ratio *= _median(a.results[iid]) / _median(b.results[iid]) + ratio **= 1.0 / len(shared) + if ratio < 1.0 - noise_percent / 100.0: + return "a" + if ratio > 1.0 + noise_percent / 100.0: + return "b" + return "tie" + + +def best_generation(generations: List[Generation], *, + noise_percent: float = 2.0) -> Optional[Generation]: + """The champion: best measured generation among non-rejected ones.""" + candidates = [g for g in generations if not g.rejected and g.results] + if not candidates: + return None + best = candidates[0] + for gen in candidates[1:]: + winner = compare_generations(gen, best, noise_percent=noise_percent) + if winner == "a": + best = gen + return best + + +def summarize(generations: List[Generation]) -> List[Dict[str, Any]]: + """Cross-round review table: one row per generation.""" + champion = best_generation(generations) + rows: List[Dict[str, Any]] = [] + for gen in generations: + rows.append({ + **gen.to_dict(), + "champion": bool(champion and champion.iteration == gen.iteration), + "medians": {iid: _median(row) + for iid, row in sorted(gen.results.items())}, + }) + return rows diff --git a/metainfer/tasks/harness_evolve/orchestrator/cli.py b/metainfer/tasks/harness_evolve/orchestrator/cli.py new file mode 100644 index 00000000..d7cdf1a0 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/cli.py @@ -0,0 +1,462 @@ +"""CLI entry point for harness_evolve orchestrator. + +Framework contract:: + + python -m metainfer.tasks.harness_evolve.orchestrator.cli run \\ + --state-dir DIR --workspace-dir DIR + +Explicit publish gate:: + + python -m metainfer.tasks.harness_evolve.orchestrator.cli publish \\ + --exp-dir DIR [--harness] [--harness-target PATH] \\ + [--kernels] [--pool PATH] [--reason TEXT] +""" + +from __future__ import annotations + +import argparse +import json +import os +import signal +from contextlib import contextmanager +from pathlib import Path +from typing import Any, Iterator, Optional + +from metainfer.orchestrator._bootstrap import clear_pid_file, write_pid_file + +from .config import load_experiment_config +from .pipeline import completed_iterations, run_experiment +from .state import read_target_iterations, write_target_iterations +from .publish import check_publishable, publish_harness, publish_kernels + +_DEFAULT_POOL = Path("/root/zth_agent/ahe-kernel-repos/registered_pool.yaml") + + +@contextmanager +def _orchestrator_pid(state_dir: Path, task_id: str) -> Iterator[Path]: + """Own the task's ``orchestrator.pid`` for as long as this process runs. + + The WebUI decides whether a task is *running* (green dot, live kill + button) from this file and nothing else: ``Launcher.status()`` reads + ``{"pid", "started_at"}`` and checks that the pid is still the same + process. An orchestrator that never writes it looks stopped for its whole + life, which is exactly how harness_evolve used to behave while a run was + in progress. + + Written before any work starts and cleared on every exit path — normal + return, exception, or SIGTERM/SIGINT (the WebUI's Stop button) — so the + file never claims a dead pid. Signals use ``os._exit`` after clearing, so + an ``atexit`` handler would not be enough on its own. + + Covers both entry points the server uses: ``run`` (started from the New + Task form) and ``resume`` (started by the supervisor to continue a run). + """ + pid_file = state_dir / "orchestrator.pid" + + def _on_signal(signum, _frame): + try: + clear_pid_file(pid_file) + except Exception: # noqa: BLE001 - best effort while dying + pass + os._exit(143 if signum == signal.SIGTERM else 130) + + previous = (signal.getsignal(signal.SIGTERM), signal.getsignal(signal.SIGINT)) + write_pid_file(pid_file, task_id) + signal.signal(signal.SIGTERM, _on_signal) + signal.signal(signal.SIGINT, _on_signal) + try: + yield pid_file + finally: + signal.signal(signal.SIGTERM, previous[0]) + signal.signal(signal.SIGINT, previous[1]) + try: + clear_pid_file(pid_file) + except Exception: # noqa: BLE001 - never mask the real failure + pass + + +def clear_stop_request(exp_dir: Path, *, state_dir: Path | None = None) -> "list[str]": + """Drop a previous stop signal so a resumed round can actually run. + + A run that stopped on purpose (``round_incomplete`` — e.g. because a child's + DKAO gate never got a clean device) leaves ``stop_requested.json`` behind, + and the pipeline treats it as "stop now" after the next evaluate. A human + asking for another round has decided the condition is over, so the signal is + recorded in the timeline and then removed: without this, every resume would + stop again immediately. + """ + import time as _time + + cleared: list = [] + stop_path = Path(exp_dir) / "stop_requested.json" + if not stop_path.is_file(): + return cleared + try: + data = json.loads(stop_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + data = {} + try: + from .state import append_timeline + append_timeline(Path(state_dir or exp_dir), "stop_request_cleared", + {"reason": (data or {}).get("reason"), + "cleared_at": _time.time()}) + except Exception: # noqa: BLE001 - the audit row is best effort + pass + try: + stop_path.unlink() + cleared.append("stop_requested.json") + except OSError: + pass + return cleared + + +#: The round target is a small number of judged passes; the WebUI clamps its own +#: "target" input to this range, so the form's value is clamped the same way. +_TARGET_MIN, _TARGET_MAX = 1, 50 + + +def _configured_rounds(cfg: Any) -> int: + """The round limit the New Task form asked for (``rounds``). + + One round is one iteration: the baseline round is round 1, every later round + is a performance pass plus — when it wins — the generalization retake inside + that same iteration. ``max_iterations`` is the field's old name, still read + so runs created before the rename keep their limit. + """ + for key in ("rounds", "max_iterations"): + raw = cfg.answers.get(key) + if raw in (None, ""): + continue + try: + value = int(raw) + except (TypeError, ValueError): + continue + return max(_TARGET_MIN, min(_TARGET_MAX, value)) + return _TARGET_MAX // 5 # 10, the form's default + + +def _seed_rounds_from_form(cfg: Any, workspace_dir: Path) -> int: + """Pin the form's round limit where the loop and the WebUI both read it.""" + rounds = _configured_rounds(cfg) + write_target_iterations(Path(workspace_dir), rounds) + print(f"[harness_evolve] rounds = {rounds} " + f"(from the form's Rounds field)", flush=True) + return rounds + + +def run_with_requirements( + requirements_path: Path, + *, + state_dir: Path, + workspace_dir: Path, +) -> int: + state_dir.mkdir(parents=True, exist_ok=True) + task_id = _task_id_of(requirements_path, fallback=state_dir.parent.name) + with _orchestrator_pid(state_dir, task_id): + cfg = load_experiment_config(requirements_path, state_dir, workspace_dir) + _seed_rounds_from_form(cfg, workspace_dir) + report = run_experiment(cfg) + print(f"[harness_evolve] done -> {report}") + return 0 + + +def _clear_operator_stop(workspace_dir: Path) -> "list[str]": + """Drop the operator-stop marker (see the WebUI's Stop button). + + ``operator_stop.json`` is what keeps a deliberately stopped run from being + quietly restarted by the supervisor, the janitor sweep or a plain resume. + A resume is a human asking for the run to continue, so the marker must not + outlive it -- otherwise the run we are starting now would be treated as + already stopped. Idempotent: no marker, nothing to report. + """ + path = Path(workspace_dir) / "operator_stop.json" + try: + path.unlink() + return ["operator_stop.json"] + except OSError: + return [] + + +def _task_id_of(requirements_path: Path, *, fallback: str) -> str: + """The task id to stamp in the PID file, read from requirements.json. + + Best effort: the PID file must exist before anything else can fail, so a + missing or malformed requirements file falls back to the directory name + rather than raising. + """ + try: + data = json.loads(requirements_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return fallback + task_id = str(data.get("task_id") or "").strip() + return task_id or fallback + + +def resume_command(args: argparse.Namespace) -> int: + """Continue an existing experiment with one (or more) further rounds. + + The already-measured champion and previous round come back from disk, so + only the new generation of the harness is evaluated. + """ + state_dir = Path(args.state_dir) + workspace_dir = Path(args.workspace_dir) + req_path = (Path(args.requirements) if args.requirements + else state_dir / "requirements.json") + if not req_path.is_file(): + print(f"[resume] requirements not found: {req_path}", flush=True) + return 2 + cfg = load_experiment_config(req_path, state_dir, workspace_dir) + # A resumed run is just as "live" to the WebUI as a freshly started one, so + # it owns the same PID file (see _orchestrator_pid). + with _orchestrator_pid(state_dir, cfg.task_id): + def _another_driver() -> Optional[int]: + """PID of a *different* live orchestrator on this experiment. + + Two drivers on one experiment is how a round ends up measured twice + on the same four cards: the janitor's resume used to open iteration + N+1 while iteration N was still being evaluated. Nothing here counts + ourselves — the pid file this process just wrote is its own. + """ + try: + from .supervisor import task_process_alive + return task_process_alive(state_dir) + except Exception: # noqa: BLE001 - never block on our own bookkeeping + return None + + done = completed_iterations(cfg.exp_dir) + if done < 1: + # Two cases: the task is already running (started by the server) and + # we should simply stay out of the way, or nothing is running and we + # are the ones to start round 1. + live = _another_driver() + if live: + print(f"[resume] task already running (pid {live}); nothing to do", + flush=True) + return 0 + print("[resume] no completed iteration yet; starting iteration 1", + flush=True) + # ``target`` is optional on this path (programmatic callers build a + # bare Namespace), so read it defensively. + first_target = getattr(args, "target", None) + if first_target is not None: + write_target_iterations(cfg.exp_dir, max(1, int(first_target))) + elif read_target_iterations(cfg.exp_dir) is None: + _seed_rounds_from_form(cfg, cfg.exp_dir) + report = run_experiment(cfg, start_iteration=1, iterations_to_run=1) + print(f"[harness_evolve] started -> {report}") + return 0 + # Same guard on the "there is history" path: a completed pass on disk + # says nothing about whether the run that produced it is *still going*. + running = _another_driver() + if running: + print(f"[resume] task already running (pid {running}); nothing to do", + flush=True) + return 0 + explicit_target = getattr(args, "target", None) + if explicit_target is not None: + write_target_iterations(cfg.exp_dir, max(1, int(explicit_target))) + print(f"[resume] target iterations set to {int(explicit_target)}", + flush=True) + elif read_target_iterations(cfg.exp_dir) is None: + # No target was ever stored (a run created before the form's + # Iterations field was wired): fall back to it instead of running + # unbounded until the round budget trips. + _seed_rounds_from_form(cfg, cfg.exp_dir) + start = done + 1 + rounds = max(1, int(args.iterations or 1)) + cleared = clear_stop_request(cfg.exp_dir, state_dir=state_dir) + if cleared: + print(f"[resume] cleared previous stop signal: {cleared}", flush=True) + cleared += _clear_operator_stop(workspace_dir) + print(f"[resume] completed iterations: {done}; running " + f"iteration {start}..{start + rounds - 1}", flush=True) + champion_iteration = getattr(args, "champion_iteration", None) + if champion_iteration: + print(f"[resume] comparison baseline pinned to iteration " + f"{int(champion_iteration)}", flush=True) + report = run_experiment(cfg, start_iteration=start, + iterations_to_run=rounds, + champion_iteration=champion_iteration) + print(f"[harness_evolve] resumed -> {report}") + return 0 + + +def _dcu_harness_default() -> Path: + return ( + Path(__file__).resolve().parents[2] + / "dcu_kernel_auto_opt" / "harness_default" + ) + + +def _promotion_context(args: argparse.Namespace): + """Load the experiment config for the promotion commands.""" + state_dir = Path(args.state_dir) + workspace_dir = Path(args.workspace_dir) + req_path = state_dir / "requirements.json" + if not req_path.is_file(): + raise SystemExit(f"[promote] requirements not found: {req_path}") + return load_experiment_config(req_path, state_dir, workspace_dir) + + +def pending_command(args: argparse.Namespace) -> int: + from .promotion import promoted_harness, read_pending + + cfg = _promotion_context(args) + pending = read_pending(cfg.exp_dir) + promoted = promoted_harness(cfg.exp_dir) + print(json.dumps({"pending": pending, "promoted": promoted}, + ensure_ascii=False, indent=2)) + return 0 if pending else 1 + + +def approve_command(args: argparse.Namespace) -> int: + from .promotion import approve_promotion + + cfg = _promotion_context(args) + result = approve_promotion(cfg, cfg.exp_dir, approved_by=str(args.by)) + if not result.get("ok"): + print(f"[approve] refused: {result.get('errors')}") + return 2 + print(f"[approve] promoted {result['version']} -> " + f"{result['record']['production_dir']}") + print(f"[approve] variants updated: {result['variant_update']['updated']}") + print(f"[approve] previous version backed up at " + f"{result['record']['previous_backup']}") + return 0 + + +def deny_command(args: argparse.Namespace) -> int: + from .promotion import deny_promotion + + cfg = _promotion_context(args) + result = deny_promotion(cfg, cfg.exp_dir, reason=str(args.reason)) + if not result.get("ok"): + print(f"[deny] refused: {result.get('errors')}") + return 2 + print(f"[deny] discarded candidate {result['denied']}; " + "resume the experiment to keep iterating") + return 0 + + +def rollback_command(args: argparse.Namespace) -> int: + from .promotion import rollback_harness + + cfg = _promotion_context(args) + result = rollback_harness(cfg.exp_dir, to=str(args.to)) + if not result.get("ok"): + print(f"[rollback] refused: {result.get('errors')}") + return 2 + print(f"[rollback] restored from {result['restored_from']}") + print(f"[rollback] replaced tree backed up at " + f"{result['record']['replaced_backup']}") + return 0 + + +def publish_command(args: argparse.Namespace) -> int: + exp = Path(args.exp_dir).resolve() + if not (exp / "config_snapshot.json").is_file(): + print(f"[publish] not a harness_evolve experiment: {exp}", flush=True) + return 2 + did = False + if args.harness or args.all: + target = Path(args.harness_target or _dcu_harness_default()) + result = publish_harness(exp, target=target, reason=args.reason) + print(f"[publish][harness] ok={result['ok']} " + f"{result.get('published')} {result.get('errors') or ''}", + flush=True) + if not result.get("ok"): + return 3 + did = True + if args.kernels or args.all: + pool = Path(args.pool or _DEFAULT_POOL) + result = publish_kernels(exp, pool_path=pool, reason=args.reason) + print(f"[publish][kernels] ok={result['ok']} " + f"published={result.get('published')} " + f"updated={result.get('updated', [])} " + f"{result.get('errors') or ''}", flush=True) + if not result.get("ok"): + return 3 + did = True + if not did: + gate = check_publishable(exp) + print(f"[publish] check only: ok={gate.get('ok')} " + f"iteration={gate.get('iteration')} " + f"verdict={gate.get('verdict')} errors={gate.get('errors') or []}", + flush=True) + return 0 if gate.get("ok") else 3 + return 0 + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(prog="metainfer-harness-evolve") + sub = parser.add_subparsers(dest="command", required=True) + run_p = sub.add_parser("run") + run_p.add_argument("requirements", type=Path) + run_p.add_argument("--state-dir", type=Path, required=True) + run_p.add_argument("--workspace-dir", type=Path, required=True) + + res = sub.add_parser("resume", help="continue an existing experiment") + res.add_argument("--state-dir", type=Path, required=True) + res.add_argument("--workspace-dir", type=Path, required=True) + res.add_argument("--requirements", type=Path, default=None, + help="defaults to /requirements.json") + res.add_argument("--iterations", type=int, default=1, + help="how many further rounds to run (default 1)") + res.add_argument("--target", type=int, default=None, + help="total iterations to reach (dynamic; re-read every round)") + res.add_argument("--champion-iteration", type=int, default=None, + help="pin the comparison baseline to a finished iteration") + res.set_defaults(func=resume_command) + + pub = sub.add_parser("publish", help="explicit publish gate") + pub.add_argument("--exp-dir", type=Path, required=True) + pub.add_argument("--harness", action="store_true", + help="publish champion harness to production default seed") + pub.add_argument("--harness-target", type=Path, default=None, + help="override harness target (default: dcu harness_default)") + pub.add_argument("--kernels", action="store_true", + help="publish new best-known kernels into the registered pool") + pub.add_argument("--all", action="store_true") + pub.add_argument("--pool", type=Path, default=None) + pub.add_argument("--reason", type=str, default="explicit publish") + pub.set_defaults(func=publish_command) + + # Human decision on a candidate harness that passed both gates. Nothing is + # written to production DKAO until `approve` runs. + appr = sub.add_parser("approve", help="approve a pending harness promotion") + appr.add_argument("--state-dir", type=Path, required=True) + appr.add_argument("--workspace-dir", type=Path, required=True) + appr.add_argument("--by", type=str, default="operator") + appr.set_defaults(func=approve_command) + + deny = sub.add_parser("deny", help="reject a pending harness promotion") + deny.add_argument("--state-dir", type=Path, required=True) + deny.add_argument("--workspace-dir", type=Path, required=True) + deny.add_argument("--reason", type=str, default="operator") + deny.set_defaults(func=deny_command) + + roll = sub.add_parser("rollback-harness", + help="restore a previously promoted harness") + roll.add_argument("--state-dir", type=Path, required=True) + roll.add_argument("--workspace-dir", type=Path, required=True) + roll.add_argument("--to", type=str, default="last-good", + help="version label (h-2), last-good, or seed") + roll.set_defaults(func=rollback_command) + + pend = sub.add_parser("pending", help="show the pending promotion, if any") + pend.add_argument("--state-dir", type=Path, required=True) + pend.add_argument("--workspace-dir", type=Path, required=True) + pend.set_defaults(func=pending_command) + + args = parser.parse_args(argv) + if args.command == "run": + return run_with_requirements( + args.requirements, + state_dir=args.state_dir, + workspace_dir=args.workspace_dir, + ) + return args.func(args) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/metainfer/tasks/harness_evolve/orchestrator/config.py b/metainfer/tasks/harness_evolve/orchestrator/config.py new file mode 100644 index 00000000..aac57434 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/config.py @@ -0,0 +1,170 @@ +"""Suite and experiment configuration for harness_evolve.""" + +from __future__ import annotations + +import json +import os +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List + +import yaml + +from .rounds import normalized_per_gate + +#: default instance used by dry-run when the suite spec is empty +DRY_DEFAULT_INSTANCES = [ + { + "id": "hy3_tp8_qkv_m4096_w4", + "shape": {"model": "hy3", "tp_size": 8, "operator": "qkv_proj", + "M": 4096, "N": 1280, "K": 4096}, + "warm_start_from": "best_known", + "budget_rounds": 4, + "pass_median_us_le": 500.0, + }, + { + "id": "glm52_tp4_o_m16_w4", + "shape": {"model": "glm52", "tp_size": 4, "operator": "o_proj", + "M": 16, "N": 6144, "K": 2048}, + "warm_start_from": "best_known", + "budget_rounds": 4, + "pass_median_us_le": 60.0, + }, +] + + +@dataclass +class InstanceSpec: + id: str + shape: Dict[str, Any] + budget_rounds: int = 4 + warm_start_from: str = "best_known" + pass_median_us_le: float = 0.0 + + @classmethod + def from_dict(cls, d: Dict[str, Any]) -> "InstanceSpec": + return cls( + id=str(d["id"]), + shape=dict(d.get("shape") or {}), + budget_rounds=int(d.get("budget_rounds") or 4), + warm_start_from=str(d.get("warm_start_from") or "best_known"), + pass_median_us_le=float(d.get("pass_median_us_le") or 0.0), + ) + + def to_dict(self) -> Dict[str, Any]: + return { + "id": self.id, + "shape": self.shape, + "budget_rounds": self.budget_rounds, + "warm_start_from": self.warm_start_from, + "pass_median_us_le": self.pass_median_us_le, + } + + +@dataclass +class ExperimentConfig: + task_id: str + state_dir: Path + workspace_dir: Path + suite: List[InstanceSpec] + max_iterations: int = 3 + harness_source: str = "dcu_default" # dir path or "dcu_default" + execution_mode: str = "dry-run" # dry-run | dkao-cli + agent_framework: str = "dsh" + pool_source: str = "" # "" = legacy suite mode; builtin|path = pool v2 + per_round_budget: int = 0 # 0 = unlimited (legacy); pool budget cap + answers: Dict[str, Any] = field(default_factory=dict) + + @property + def exp_dir(self) -> Path: + return self.workspace_dir + + def dump_snapshot(self, path: Path) -> None: + path.write_text(json.dumps({ + "schema_version": 1, + "task_id": self.task_id, + "max_iterations": self.max_iterations, + "harness_source": self.harness_source, + "execution_mode": self.execution_mode, + "pool_source": self.pool_source, + "per_round_budget": self.per_round_budget, + "suite": [s.to_dict() for s in self.suite], + }, ensure_ascii=False, indent=2), encoding="utf-8") + + @property + def pool_mode(self) -> bool: + return bool(self.pool_source) and self.pool_source != "suite" + + +def load_experiment_config( + requirements_path: Path, state_dir: Path, workspace_dir: Path +) -> ExperimentConfig: + req = json.loads(requirements_path.read_text(encoding="utf-8")) + # The shell/server frontend flattens form answers onto the requirements + # top level (same contract dcu_kernel_auto_opt consumes via its + # ``_answers`` fallback). Accept either an ``answers`` dict (CLI/tests) or + # the flattened top level so a WebUI-launched experiment honors the form. + req_answers = req.get("answers") + answers = req_answers if isinstance(req_answers, dict) else req + task_id = str(req.get("task_id", "task")) + + suite_yaml = str(answers.get("suite_yaml") or "") + raw_instances: List[Dict[str, Any]] = [] + if suite_yaml.strip(): + data = yaml.safe_load(suite_yaml) + if isinstance(data, dict): + raw_instances = list(data.get("instances") or []) + elif isinstance(data, list): + raw_instances = list(data) + if not raw_instances: + raw_instances = DRY_DEFAULT_INSTANCES + + return ExperimentConfig( + task_id=task_id, + state_dir=Path(state_dir), + workspace_dir=Path(workspace_dir), + suite=[InstanceSpec.from_dict(d) for d in raw_instances], + max_iterations=int(answers.get("max_iterations") or 3), + harness_source=str(answers.get("harness_source") or "dcu_default"), + execution_mode=str(answers.get("execution_mode") or "dry-run"), + agent_framework=str(answers.get("agent_framework") or "dsh"), + pool_source=str(answers.get("pool_source") or ""), + per_round_budget=_normalized_budget(answers.get("per_round_budget")), + answers=dict(answers), + ) + + +def _normalized_budget(value: Any) -> int: + """This round's question count: always a multiple of 4 (FLOW.md §1.1). + + The machine has four HCUs, so a round is sized in whole waves; a request of + 5 or 7 questions would leave devices idle in the second wave, so it is + rounded up (never down) here, at the single point where the form/CLI answer + enters the run. ``0``/empty keeps the legacy meaning "no cap, use the whole + pool", which is why this cannot simply be ``normalized_per_gate`` (that one + maps 0 to the 4-question default). + """ + try: + count = int(value) + except (TypeError, ValueError): + return 0 + if count <= 0: + return 0 + return normalized_per_gate(count) + + +def harness_seed_dir(harness_source: str) -> Path: + """Resolve the harness seed directory. + + ``dcu_default`` maps to the dcu_kernel_auto_opt plugin's harness_default/. + An explicit path (absolute or relative to the experiment workspace) is used + as-is. Later this becomes the "champion workspace" an AHE experiment forks. + """ + if harness_source and harness_source != "dcu_default": + p = Path(harness_source).expanduser() + return p.resolve() + dcu_plugin = ( + Path(__file__).resolve().parents[2] + / "dcu_kernel_auto_opt" / "harness_default" + ) + return dcu_plugin.resolve() diff --git a/metainfer/tasks/harness_evolve/orchestrator/decision_engine.py b/metainfer/tasks/harness_evolve/orchestrator/decision_engine.py new file mode 100644 index 00000000..ce4d4b73 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/decision_engine.py @@ -0,0 +1,588 @@ +"""System-level harness decision engine. + +Separates the performance gate (did the agent system get better?) from the +mechanism gate (did it improve for the reason declared in change_manifest?). +True performance is primary: an unexplained but reproducible win may be +promoted; a mechanism miss alone never turns a real stable win into REJECT. + +Verdicts: + BASELINE first measured version, becomes champion + PROMOTE clear performance win + mechanism evidence hit + CONFIRM_REQUIRED clear performance win, mechanism absent/missed; repeat + PROMOTE_UNEXPLAINED repeated clear win but mechanism still missed/unknown + NO_SIGNAL inside noise; champion unchanged, candidate archived + SPECIALIZE target family wins but held-out regresses; routeable + REJECT correctness or primary-suite significant regression + REJECT_OVERFIT main improves but held-out regresses and not routeable +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Any, Dict, Iterable, List, Mapping, Optional + + +@dataclass(frozen=True) +class DecisionPolicy: + noise_percent: float = 2.0 + min_independent_wins: int = 2 + max_loss_ratio: float = 0.10 + max_heldout_losses: int = 0 + #: The performance gate is a *question-count* rule: the candidate passes + #: when at least this fraction of the counted questions (probe questions + #: excluded) are measurably faster than the champion. With the default 0.75 + #: that is "3 of 4 questions beat the current variant", regardless of how + #: far behind the remaining question is. + min_win_ratio: float = 0.75 + #: One hard red line inside that rule: a single question that is slower by + #: more than this percentage — or that fails correctness — still rejects the + #: candidate, so "three small wins plus one collapse" cannot be promoted. + hard_regression_percent: float = 30.0 + #: A round whose paired set is mostly unmeasurable (crashed children, + #: suppressed measurements) cannot support a verdict: it is recorded as + #: INCONCLUSIVE instead of REJECT. + min_valid_pair_ratio: float = 0.5 + #: How strictly the mechanism gate is enforced when a real performance win + #: has no (or only partial) mechanism evidence: + #: ``lenient`` (default) — unverified/partial evidence may still promote + #: (recorded as PROMOTE_UNEXPLAINED); only a *contradiction* forces + #: one confirmation round; + #: ``strict`` — anything short of a verified hit requires confirmation; + #: ``off`` — skip the mechanism gate entirely. + mechanism_policy: str = "lenient" + + +def _median(r: Mapping[str, Any]) -> Optional[float]: + try: + v = float(r.get("median_us")) + return v if v > 0 else None + except (TypeError, ValueError): + return None + + +_MISSING_STATUSES = {"missing_report", "exception", "timeout", "partial_worker_result"} + + +def _correctness_hard_fail(r: Mapping[str, Any]) -> bool: + return r.get("correctness_ok") is False or r.get("correctness_passed") is False + + +def _measurement_missing(r: Mapping[str, Any]) -> bool: + """True when there is no usable measurement at all. + + A crashed child (``missing_report``) or a timeout is an *environment* + failure: treating it as HARD_FAIL/LOSS turns infrastructure trouble into a + fake harness regression, which is how a run can be "rejected" five times + in a row while nothing was actually measured. + """ + if r.get("median_us") in (None, ""): + return True + return str(r.get("status") or "").strip().lower() in _MISSING_STATUSES + + +def compare_instances( + champion: Mapping[str, Mapping[str, Any]], + candidate: Mapping[str, Mapping[str, Any]], + *, + noise_percent: float, +) -> Dict[str, Any]: + """Paired comparison on ids present in both result sets.""" + ids = sorted(set(champion) & set(candidate)) + rows: Dict[str, Any] = {} + counts = {"WIN": 0, "LOSS": 0, "NO_SIGNAL": 0, "HARD_FAIL": 0, + "INVALID": 0} + pass_before = pass_after = 0 + + for iid in ids: + before, after = champion[iid], candidate[iid] + pass_before += int(before.get("passed") is True) + pass_after += int(after.get("passed") is True) + b, a = _median(before), _median(after) + if _measurement_missing(after): + verdict, delta = "INVALID", None + elif _correctness_hard_fail(after): + verdict, delta = "HARD_FAIL", None + elif b is None or a is None: + verdict, delta = "INVALID", None + else: + delta = (b - a) / b * 100.0 # positive = candidate faster + if delta > noise_percent: + verdict = "WIN" + elif delta < -noise_percent: + verdict = "LOSS" + else: + verdict = "NO_SIGNAL" + counts[verdict] += 1 + rows[iid] = { + "champion_median_us": b, + "candidate_median_us": a, + "delta_percent": round(delta, 4) if delta is not None else None, + "verdict": verdict, + "champion_passed": before.get("passed") is True, + "candidate_passed": after.get("passed") is True, + } + + return { + "overlap_ids": ids, + "per_instance": rows, + "counts": counts, + "pass_before": pass_before, + "pass_after": pass_after, + "delta_pass": pass_after - pass_before, + } + + +def _manifest_changes(manifest: Optional[Mapping[str, Any]]) -> List[Mapping[str, Any]]: + if not manifest: + return [] + return [x for x in manifest.get("changes", []) if isinstance(x, Mapping)] + + +def _mechanism_status( + manifest: Optional[Mapping[str, Any]], + mechanism_checks: Optional[Mapping[str, bool]], +) -> Dict[str, Any]: + changes = _manifest_changes(manifest) + declared = [c for c in changes if c.get("mechanism_signature")] + if not declared: + return {"status": "NOT_DECLARED", "hits": [], "misses": []} + if not mechanism_checks: + return { + "status": "NOT_CHECKED", + "hits": [], + "misses": [str(c.get("id", "unknown")) for c in declared], + "unobserved": [], + "contradicted": [], + } + hits, misses, unobserved, contradicted = [], [], [], [] + for c in declared: + cid = str(c.get("id", "unknown")) + raw = mechanism_checks.get(cid) + if raw is True or raw == "hit": + hits.append(cid) + elif raw == "partial": + hits.append(cid) # lenient: partial evidence counts + elif raw == "unobserved": + unobserved.append(cid) + elif raw == "contradicted": + contradicted.append(cid) + misses.append(cid) + else: + misses.append(cid) + if contradicted: + status = "MISS" + elif hits and not misses: + status = "HIT" + elif hits and misses: + status = "PARTIAL" + elif unobserved and not misses: + status = "UNOBSERVED" + else: + status = "MISS" + return {"status": status, "hits": hits, "misses": misses, + "unobserved": unobserved, "contradicted": contradicted} + + +def _collapse(row: Mapping[str, Any], limit_percent: float) -> bool: + """One question that regressed past the red line (or broke correctness).""" + delta = row.get("delta_percent") + try: + if delta is not None and float(delta) < -abs(float(limit_percent)): + return True + except (TypeError, ValueError): + pass + return row.get("verdict") == "HARD_FAIL" + + +def _routeable(manifest: Optional[Mapping[str, Any]]) -> bool: + for c in _manifest_changes(manifest): + scope = c.get("scope") or {} + if isinstance(scope, Mapping) and scope.get("routeable") is True: + return True + return False + + +def decide( + champion: Optional[Mapping[str, Mapping[str, Any]]], + candidate: Mapping[str, Mapping[str, Any]], + *, + manifest: Optional[Mapping[str, Any]] = None, + heldout_champion: Optional[Mapping[str, Mapping[str, Any]]] = None, + heldout_candidate: Optional[Mapping[str, Mapping[str, Any]]] = None, + mechanism_checks: Optional[Mapping[str, bool]] = None, + confirmation_reproduced: bool = False, + purposes: Optional[Mapping[str, str]] = None, + policy: DecisionPolicy = DecisionPolicy(), +) -> Dict[str, Any]: + """Return a complete, auditable system-level decision record. + + ``purposes`` labels each question (regression / repair / probe). Probe + questions are *generalisation* evidence only: they never enter the + pass-rate or win/loss pair that decides promotion, but a probe that + regresses blocks promotion, because it means the change does not + generalise beyond the family it was tuned on. + """ + if champion is None: + return { + "verdict": "BASELINE", + "performance_gate": {"status": "BASELINE"}, + "mechanism_gate": {"status": "NOT_APPLICABLE"}, + "heldout_gate": {"status": "NOT_APPLICABLE"}, + "action": "SET_CHAMPION", + } + + purposes = purposes or {} + probe_set = {i for i, p in purposes.items() if p == "probe"} + main_ids = [i for i in candidate if i not in probe_set] + if not main_ids: + main_ids = list(candidate) + champion_main = {k: v for k, v in champion.items() if k in main_ids} + candidate_main = {k: v for k, v in candidate.items() if k in main_ids} + + perf = compare_instances(champion_main, candidate_main, + noise_percent=policy.noise_percent) + counts = perf["counts"] + #: correctness/missing-measurement failures are always fatal: a comparison + #: missing one side is not evidence that the candidate is faster. + hard_fail = counts["HARD_FAIL"] > 0 or counts["INVALID"] > 0 + measured = counts["WIN"] + counts["LOSS"] + counts["NO_SIGNAL"] + # Question-count performance gate: >= min_win_ratio of the measured + # questions must be measurably faster than the champion. NO_SIGNAL (inside + # the noise band) is not a win, so it counts against the ratio. + win_ratio = (counts["WIN"] / measured) if measured else 0.0 + enough_wins = measured > 0 and win_ratio >= float(policy.min_win_ratio) + # Hard red line: one question that fell off a cliff still rejects. + collapses = [ + iid for iid, row in (perf.get("per_instance") or {}).items() + if _collapse(row, policy.hard_regression_percent) + ] + clear_improvement = bool(enough_wins and not hard_fail and not collapses) + + held = None + heldout_regression = False + if heldout_champion is not None and heldout_candidate is not None: + held = compare_instances( + heldout_champion, heldout_candidate, + noise_percent=policy.noise_percent, + ) + heldout_regression = ( + held["counts"]["HARD_FAIL"] > 0 + or held["counts"]["LOSS"] > policy.max_heldout_losses + or held["delta_pass"] < 0 + ) + + mechanism = _mechanism_status(manifest, mechanism_checks) + + #: Counted questions that produced a usable pairing (WIN/LOSS/NO_SIGNAL). + measurable = counts["WIN"] + counts["LOSS"] + counts["NO_SIGNAL"] + #: All counted questions inside the noise band: "no measurable difference" + #: rather than "worse". Drives both the gate status and the verdict. + indistinguishable = bool(measured) and counts["WIN"] == 0 and counts["LOSS"] == 0 + paired_total = len(perf["overlap_ids"]) + coverage = (measured / paired_total) if paired_total else 1.0 + inconclusive = bool(paired_total) and coverage < float( + policy.min_valid_pair_ratio) + + generalization: Dict[str, Any] = {"status": "NOT_RUN", "comparison": None, + "ids": sorted(probe_set)} + if probe_set: + probe_champion = {k: v for k, v in champion.items() if k in probe_set} + probe_candidate = {k: v for k, v in candidate.items() if k in probe_set} + if set(probe_champion) & set(probe_candidate): + probe_perf = compare_instances( + probe_champion, probe_candidate, + noise_percent=policy.noise_percent) + probe_counts = probe_perf["counts"] + generalization = { + "status": ("FAIL" if (probe_counts["HARD_FAIL"] > 0 + or probe_counts["LOSS"] > 0) else "PASS"), + "comparison": probe_perf, + "ids": sorted(probe_set), + } + + if inconclusive: + # Not a result: keep the champion, do not count it as a failure. + verdict, action = "INCONCLUSIVE", "KEEP_CHAMPION_ARCHIVE_CANDIDATE" + elif hard_fail or collapses: + verdict, action = "REJECT", "RESTORE_CHAMPION" + elif clear_improvement and heldout_regression: + if _routeable(manifest): + verdict, action = "SPECIALIZE", "ARCHIVE_SPECIALIZED" + else: + verdict, action = "REJECT_OVERFIT", "RESTORE_CHAMPION" + elif clear_improvement and generalization["status"] == "FAIL": + # A win inside the tuned families that breaks an untouched family is + # over-fitting, not progress: require one confirmation round. + if confirmation_reproduced: + verdict, action = "PROMOTE_UNEXPLAINED", "SET_CHAMPION" + else: + verdict, action = "CONFIRM_REQUIRED", "ARCHIVE_AND_CONFIRM" + elif clear_improvement: + mstatus = mechanism["status"] + policy_mode = str(policy.mechanism_policy or "lenient").lower() + if mstatus in {"HIT", "PARTIAL"}: + verdict, action = "PROMOTE", "SET_CHAMPION" + elif policy_mode == "off": + verdict, action = "PROMOTE_UNEXPLAINED", "SET_CHAMPION" + elif policy_mode == "lenient" and mstatus in {"UNOBSERVED", + "NOT_CHECKED", + "NOT_DECLARED"}: + # Performance is primary: an unverified win may still take the + # champion slot, but it is recorded as unexplained. + verdict, action = "PROMOTE_UNEXPLAINED", "SET_CHAMPION" + elif confirmation_reproduced: + verdict, action = "PROMOTE_UNEXPLAINED", "SET_CHAMPION" + else: + verdict, action = "CONFIRM_REQUIRED", "ARCHIVE_AND_CONFIRM" + else: + # Below the win ratio. Distinguish "we measured no difference" (an + # honest no-signal: keep the champion, try again) from "the candidate + # is measurably worse" (a real reject that counts toward the circuit + # breaker). A round where every counted question sat inside the noise + # band is the former; anything with LOSSes is the latter. + if indistinguishable: + verdict, action = "NO_SIGNAL", "KEEP_CHAMPION_ARCHIVE_CANDIDATE" + else: + verdict, action = "REJECT", "RESTORE_CHAMPION" + + performance_status = ( + "INCONCLUSIVE" if inconclusive else + "FAIL" if hard_fail or collapses else + "PASS" if clear_improvement else + "NO_SIGNAL" if indistinguishable else "FAIL" + ) + heldout_status = ( + "NOT_RUN" if held is None else + "FAIL" if heldout_regression else "PASS" + ) + record = { + "verdict": verdict, + "action": action, + "policy": { + "noise_percent": policy.noise_percent, + "min_win_ratio": policy.min_win_ratio, + "hard_regression_percent": policy.hard_regression_percent, + "min_independent_wins": policy.min_independent_wins, + "max_loss_ratio": policy.max_loss_ratio, + "max_heldout_losses": policy.max_heldout_losses, + "mechanism_policy": policy.mechanism_policy, + }, + "performance_gate": {"status": performance_status, + "measurable_pairs": measurable, + "paired_total": paired_total, + "coverage": round(coverage, 3), + # the question-count rule, in one line: + "win_ratio": round(win_ratio, 3), + "min_win_ratio": policy.min_win_ratio, + "wins_needed": _wins_needed(measured, policy), + "collapses": sorted(collapses), + **perf}, + "mechanism_gate": mechanism, + "generalization_gate": generalization, + "purpose_breakdown": { + iid: {"purpose": purposes.get(iid, "regression"), + "counted": iid not in probe_set} + for iid in sorted(set(champion) | set(candidate)) + }, + "paired_ids": sorted(set(champion_main) & set(candidate_main)), + "probe_ids": sorted(probe_set), + "heldout_gate": { + "status": heldout_status, + "comparison": held, + }, + "reason": _reason(verdict, perf, mechanism, heldout_regression, + win_ratio=win_ratio, + collapses=collapses, + policy=policy), + } + if generalization["status"] == "FAIL": + tail = ("promoted after reproduction" if verdict == "PROMOTE_UNEXPLAINED" + else ("rejected" if action == "RESTORE_CHAMPION" + else "repeat once")) + record["reason"] = ( + "performance improved inside the tuned families but an untouched " + f"family regressed (over-fitting): {tail}" + ) + return record + + +def _needed_wins(measured: int, ratio: float) -> int: + """Ceiling of ``measured * ratio``: 4 operators at 75% need 3 wins.""" + if measured <= 0: + return 0 + import math + return max(1, min(measured, int(math.ceil(measured * float(ratio))))) + + +def _wins_needed(measured: int, policy: DecisionPolicy) -> int: + """How many of the counted questions must win (round-level policy).""" + return _needed_wins(measured, policy.min_win_ratio) + + +#: The two harness gates, on the operator-count rule the operator specified. +#: Both compare the candidate against the *current variant* of each operator and +#: both refuse a candidate that lets any single operator fall below a floor, so +#: "many small wins plus one collapse" cannot pass. +PERFORMANCE_GATE_WIN_RATIO = 0.75 # 4 operators -> at least 3 must win +PERFORMANCE_GATE_FLOOR_PERCENT = 0.80 # every non-winning operator >= 80% of the variant +GENERALIZATION_GATE_WIN_RATIO = 0.50 # 4 operators -> at least 2 must win +GENERALIZATION_GATE_FLOOR_PERCENT = 0.80 + +#: How a tie (a difference inside the comparison's noise band) is scored. +#: ``"loss"``: only a strict win counts — the performance gate's rule, where +#: fresh operators must genuinely come out ahead. +#: ``"neutral"``: a tie is evidence of *holding the line*, not of failure — the +#: generalization gate's rule. The retake re-measures the paper on the same +#: harness, and DKAO warm-starts from the kernel the baseline round just put in +#: the pool, so re-finding that same kernel is the expected outcome; scoring it +#: as a loss would demand that a search policy out-optimize its own best result +#: every time. A real regression (below the floor) still fails the gate. +TIE_AS_LOSS = "loss" +TIE_AS_NEUTRAL = "neutral" + + +def gate_verdict(reference: Mapping[str, Mapping[str, Any]], + candidate: Mapping[str, Mapping[str, Any]], + *, + kind: str = "performance", + noise_percent: float = 2.0, + win_ratio: Optional[float] = None, + floor_percent: Optional[float] = None, + ties: Optional[str] = None, + ) -> Dict[str, Any]: + """Judge one round of a gate on the operator-count rule. + + ``kind`` is ``"performance"`` (>= 75% of operators must beat their variant) + or ``"generalization"`` (>= 50%). In both cases every operator that did not + win must still be at least ``floor_percent`` of its variant, and an operator + that produced no usable measurement fails the gate outright rather than + being counted as a loss. + + ``ties`` decides how a difference inside ``noise_percent`` is scored; it + defaults to the gate's own rule (performance: a tie is not a win; + generalization: a tie holds the line). Every operator that is genuinely + slower than its variant (and every one below the floor) blocks the gate. + """ + if win_ratio is None: + win_ratio = (PERFORMANCE_GATE_WIN_RATIO if kind == "performance" + else GENERALIZATION_GATE_WIN_RATIO) + if floor_percent is None: + floor_percent = (PERFORMANCE_GATE_FLOOR_PERCENT if kind == "performance" + else GENERALIZATION_GATE_FLOOR_PERCENT) + if ties is None: + ties = TIE_AS_LOSS if kind == "performance" else TIE_AS_NEUTRAL + comparison = compare_instances(reference, candidate, + noise_percent=noise_percent) + rows = comparison.get("per_instance") or {} + judged = ["WIN", "LOSS", "NO_SIGNAL"] + all_judged = [iid for iid, row in rows.items() + if row.get("verdict") in judged] + ties_ids = [iid for iid in all_judged + if rows[iid]["verdict"] == "NO_SIGNAL"] + losses = [iid for iid in all_judged if rows[iid]["verdict"] == "LOSS"] + # Neutral ties leave the denominator: they are neither a win to count nor a + # loss to answer for (the floor is what catches a real regression). + counted = ([iid for iid in all_judged if iid not in set(ties_ids)] + if ties == TIE_AS_NEUTRAL else all_judged) + wins = [iid for iid in counted if rows[iid]["verdict"] == "WIN"] + unmeasured = sorted( + iid for iid, row in rows.items() + if row.get("verdict") in {"INVALID", "HARD_FAIL"}) + too_slow = sorted( + iid for iid in all_judged + if _below_floor(rows[iid], floor_percent)) + measured = len(counted) + ratio = (len(wins) / measured) if measured else 0.0 + needed = _needed_wins(measured, win_ratio) + reasons: List[str] = [] + if unmeasured: + reasons.append("no usable measurement: " + ", ".join(unmeasured)) + if too_slow: + reasons.append( + f"below {floor_percent:.0%} of the variant: " + ", ".join(too_slow)) + if measured and len(wins) < needed: + reasons.append(f"{len(wins)}/{measured} operators beat the variant " + f"(needs {needed} = {win_ratio:.0%})") + all_tied = bool(all_judged) and len(ties_ids) == len(all_judged) + if all_tied: + # Every operator tied. Scored as holding the line (the generalization + # gate's rule) that is a pass on the floor requirement alone; scored as + # a loss it means the candidate proved nothing. + if ties == TIE_AS_LOSS: + reasons.append("no operator beat its variant (all tied)") + elif not measured and not unmeasured: + reasons.append("nothing comparable to the variant") + record = { + "kind": kind, + "status": "PASS" if not reasons else "FAIL", + "win_ratio": round(ratio, 3), + "required_win_ratio": win_ratio, + "floor_percent_of_variant": floor_percent, + "tie_policy": ties, + "operators": sorted(rows), + "wins": sorted(wins), + "losses": sorted(losses), + "ties": sorted(ties_ids), + "counted": measured, + "wins_needed": needed, + "unmeasured": unmeasured, + "below_floor": too_slow, + "reasons": reasons, + "comparison": comparison, + } + return record + + +def _below_floor(row: Mapping[str, Any], floor_percent: float) -> bool: + """True when a measured operator is worse than ``floor_percent`` of its variant.""" + delta = row.get("delta_percent") + try: + if delta is None: + return False + return float(delta) < (float(floor_percent) - 1.0) * 100.0 + except (TypeError, ValueError): + return False + + + +def _reason(verdict: str, perf: Mapping[str, Any], mechanism: Mapping[str, Any], + heldout_regression: bool, *, win_ratio: float = 0.0, + collapses: Optional[List[str]] = None, + policy: Optional[DecisionPolicy] = None) -> str: + policy = policy or DecisionPolicy() + counts = perf.get("counts") or {} + measured = (int(counts.get("WIN", 0)) + int(counts.get("LOSS", 0)) + + int(counts.get("NO_SIGNAL", 0))) + summary = (f"{counts.get('WIN', 0)}/{measured} counted questions faster " + f"than the champion (needs {_wins_needed(measured, policy)}, " + f"{policy.min_win_ratio:.0%})") + if verdict == "INCONCLUSIVE": + return ("too few usable measurements this round (crashed children or " + "suppressed readings); no verdict") + if verdict == "PROMOTE": + return f"clear performance improvement ({summary}); declared mechanism verified" + if verdict == "PROMOTE_UNEXPLAINED": + return (f"reproduced performance improvement ({summary}); mechanism " + "remains unverified") + if verdict == "CONFIRM_REQUIRED": + return (f"performance improved ({summary}) but mechanism is " + "absent/unverified; repeat once") + if verdict == "SPECIALIZE": + return ("target suite improved but held-out regressed; preserve as " + "routeable variant") + if verdict == "REJECT_OVERFIT": + return "target suite improved but held-out regression exceeded boundary" + if verdict == "REJECT": + if collapses: + return (f"hard red line crossed on {', '.join(sorted(collapses))} " + f"(> {policy.hard_regression_percent:.0f}% slower or " + f"incorrect); {summary}") + if int(counts.get("HARD_FAIL", 0)) or int(counts.get("INVALID", 0)): + return ("correctness failure or an unusable measurement in the " + "comparison; not a result we can trust") + return (f"performance gate failed: {summary} " + f"(needed {policy.min_win_ratio:.0%} of counted questions)") + if verdict == "NO_SIGNAL": + return (f"candidate is inside the noise band ({summary}); keep " + "champion and archive candidate") + return "first measured version establishes the champion baseline" diff --git a/metainfer/tasks/harness_evolve/orchestrator/evolve.py b/metainfer/tasks/harness_evolve/orchestrator/evolve.py new file mode 100644 index 00000000..5c11bd0f --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/evolve.py @@ -0,0 +1,272 @@ +"""Single-role Evolve adapters. + +The real AgentEvolver modifies the harness workspace and, in the same turn, +produces BOTH: + - _ahe_change_manifest.json (what changed / system prediction / mechanism) + - _ahe_round_plan.json (which registered questions to run next + why) +The pipeline moves these to runs/iteration_NNN/evolve/ after validation. +""" + +from __future__ import annotations + +import json +import os +import subprocess +from pathlib import Path +from typing import Any, Dict, Optional + +#: Model id the DSH Evolve Agent runs on by default. The form's `evolve_model` +#: field picks a label (see form.yaml) and the pipeline passes the id down to +#: ``bridge/dsh/dsh_agent.py --model``; this constant is the fallback used by +#: callers that were given no answer at all. +#: ``deepseek/deepseek-flash`` is the 4.1 Flash revision (form label +#: ``deepseek-flash-4.1``). ``deepseek/deepseek-v4-flash-0731`` is the older +#: pinned build, kept selectable for reproducing earlier experiments. +DEFAULT_EVOLVE_MODEL = "deepseek/deepseek-flash" +LEGACY_EVOLVE_MODEL = "deepseek/deepseek-v4-flash-0731" + + +def _empty_manifest(iteration: int) -> Dict[str, Any]: + return { + "schema_version": 1, + "iteration": iteration, + "changes": [], + "verification": { + "status": "pending", "fixes": [], "regressions": [], + "false_predictions": [], "verdict": None, + }, + "note": "dry-run evolve: no automatic changes", + } + + +class Evolver: + name = "base" + + def apply(self, workspace: Path, iteration: int, + query_context: Dict[str, Any]) -> Dict[str, Any]: + raise NotImplementedError + + +class DryRunEvolver(Evolver): + name = "dry-run" + + def apply(self, workspace, iteration, query_context): + fake = os.environ.get("HARNESS_EVOLVE_FAKE_CHANGE", "") in {"1", "true"} + manifest = _empty_manifest(iteration) + changed = False + summary_lines = [f"# Evolve summary — iteration {iteration} (dry-run)", "", + "No automatic harness change (dry-run evolver)."] + if fake: + manifest = { + "schema_version": 1, "iteration": iteration, + "changes": [{ + "id": f"chg-{iteration}", "type": "new", + "component": "gates", + "description": "synthetic change for attribution testing", + "files": ["gates.yaml"], + "failure_evidence": ["dry-run synthetic failure"], + "root_cause": "synthetic", "targeted_fix": "synthetic gates edit", + "failure_pattern": "synthetic", "predicted_fixes": [], + "risk_tasks": [], + "scope": {"expected_improve": ["synthetic"], + "expected_unchanged": [], "at_risk": [], + "routeable": False}, + "system_prediction": {"direction": "improve", + "expected_effect": ["pass-rate increase"]}, + "mechanism_signature": ["synthetic mechanism fires"], + }], + "verification": {"status": "pending", "fixes": [], + "regressions": [], "false_predictions": [], + "verdict": None}, + } + changed = True + summary_lines.append("Emitted one synthetic change (fake mode).") + return {"summary": "\n".join(summary_lines), "manifest": manifest, + "changed": changed, "round_plan": None} + + +class AgentEvolver(Evolver): + """Real single-role DSH evolve agent.""" + + name = "agent" + MANIFEST_NAME = "_ahe_change_manifest.json" + PLAN_NAME = "_ahe_round_plan.json" + + def __init__(self, *, model: str = DEFAULT_EVOLVE_MODEL, + timeout_seconds: float = 3600): + self.model = model + self.timeout_seconds = timeout_seconds + + def apply(self, workspace, iteration, query_context): + manifest_path = workspace / self.MANIFEST_NAME + plan_path = workspace / self.PLAN_NAME + manifest_path.unlink(missing_ok=True) + plan_path.unlink(missing_ok=True) + prompt = self._prompt(workspace, iteration, query_context) + wrapper = ( + Path(__file__).resolve().parents[2] / "dcu_kernel_auto_opt" + / "bridge" / "dsh" / "dsh_agent.py" + ) + cmd = [ + str(wrapper), "-p", "--output-format", "stream-json", + "--input-format", "text", "--verbose", + "--permission-mode", "bypassPermissions", + "--add-dir", str(workspace), "--model", self.model, + "--effort", "max", "--tools", "Read,Glob,Grep,Write", + "--disallowedTools", "Bash,Skill,WebFetch,WebSearch", + ] + try: + proc = subprocess.run( + cmd, input=prompt, text=True, capture_output=True, + timeout=self.timeout_seconds, env=os.environ.copy(), + ) + except subprocess.TimeoutExpired: + # A slow/failed evolve turn must not abort the whole experiment: + # the iteration's evaluation value is already recorded, so fall + # back to a no-change placeholder and let the pipeline finish + # (champion kept, report + finished still written). + return _evolve_failed( + iteration, + f"evolve agent timed out after {self.timeout_seconds:.0f}s", + ) + except Exception as exc: # noqa: BLE001 - never crash the outer loop + return _evolve_failed( + iteration, f"evolve agent crashed: {exc!r}" + ) + if proc.returncode != 0: + return _evolve_failed( + iteration, + f"evolve agent failed rc={proc.returncode}: " + f"{(proc.stderr or '')[-800:]}", + ) + manifest = _read_json(manifest_path) + plan = _read_json(plan_path) + missing = [ + name for name, value in + ((self.MANIFEST_NAME, manifest), (self.PLAN_NAME, plan)) + if value is None + ] + if missing: + return _evolve_failed( + iteration, + "evolve agent did not write " + ", ".join(missing), + ) + manifest_path.unlink(missing_ok=True) + plan_path.unlink(missing_ok=True) + summary = _last_result(proc.stdout) or "Evolve Agent completed." + return {"summary": summary, "manifest": manifest, "round_plan": plan, + "changed": bool(manifest.get("changes"))} + + def _prompt(self, workspace: Path, iteration: int, + context: Dict[str, Any]) -> str: + pool_ids = list(context.get("pool_ids") or []) + selected = list(context.get("selected") or []) + return f"""You are the SINGLE AHE Evolve Agent for DKAO INT8 W8A8 GEMM. + +Iteration: {iteration} +Harness workspace (the ONLY directory you may modify): {workspace} +Analysis overview: {context.get('overview')} +Previous results/diff are embedded below. + +Current selected questions: {json.dumps(selected)} +Registered pool ids (you may select ONLY these): {json.dumps(pool_ids)} +Results: {json.dumps(context.get('results') or {}, ensure_ascii=False)[:16000]} +Diff: {json.dumps(context.get('diff') or {}, ensure_ascii=False)} +Decision: {json.dumps(context.get('decision') or {}, ensure_ascii=False)[:8000]} + +Your two responsibilities in this ONE role: +1) Evidence-driven harness evolution: inspect workspace, make at most ONE logical + change to planner/gates/prompt/skill/memory; do not touch model config. Every + change needs failure_evidence, root_cause, targeted_fix, system_prediction, + scope (expected_improve/unchanged/at_risk/routeable), mechanism_signature. +2) Choose the NEXT round exam from registered pool ids. Give strategy + explore|exploit|balanced, rationale and selected ids. The machine assigns + each question a role (2 regression anchors / 1 repair / 1 cross-family + probe) and enforces pool membership, budget, family quota and >=60% + overlap: your picks are used where they fit a role and replaced where they + do not (the correction is recorded in the round rationale). Weights are + ignored — priority comes from the role, and the probe never counts toward + the pass-rate decision. + +Mechanism evidence (machine-checkable, not prose): every change MUST also +carry a structured `mechanism_check` describing how a robot can verify that the +mechanism actually ran. Allowed kinds: + {{"kind": "planner_plan_ids", "expect_any": ["memory_layout", "epilogue_fusion"], + "expect_order": ["architecture_explore", "memory_layout"]}} + {{"kind": "harness_revision"}} # candidate revision must be the one children load + {{"kind": "gate_events", "expect_any": ["gate_blocked"]}} + {{"kind": "gate_values", "expect_any": ["round_acceptance_improvement_percent"]}} +Use planner_plan_ids when the change edits planner_policy/planner_catalog, +gate_values when it edits gates.yaml (the runtime reads that file now), and +gate_events for gating behaviour that shows up in the child timeline. A change without mechanism_check stays +"unobserved" (allowed, but it cannot be verified as effective). + +Write strict JSON files INSIDE workspace: +- {manifest_path_for_prompt(workspace, self.MANIFEST_NAME)} + {{"schema_version":1,"iteration":{iteration},"changes":[...], + "verification":{{"status":"pending","fixes":[],"regressions":[], + "false_predictions":[],"verdict":null}}}} +- {manifest_path_for_prompt(workspace, self.PLAN_NAME)} + {{"iteration":{iteration},"strategy":"balanced","rationale":"...", + "selected":[...],"weights":{{}}}} + +Use file tools only. Do not compile, benchmark, run DKAO, edit outside workspace, +or invent pool ids. Finish after rereading both JSON files from disk. +""" + + +def manifest_path_for_prompt(workspace: Path, name: str) -> str: + return str(workspace / name) + + +def _evolve_failed(iteration: int, reason: str) -> Dict[str, Any]: + """No-change evolve placeholder used when the evolve turn fails. + + The outer loop keeps the iteration's evaluation result (champion + unchanged) and still writes evolve artifacts + report, instead of + crashing mid-experiment and losing the whole round. + """ + manifest = { + "schema_version": 1, + "iteration": iteration, + "changes": [], + "verification": { + "status": "evolve_failed", "fixes": [], "regressions": [], + "false_predictions": [], "verdict": None, + }, + "note": reason, + } + summary = ( + f"# Evolve summary — iteration {iteration} (failed)\n\n" + f"{reason}\n\nNo harness change was applied; " + "the champion is unchanged." + ) + return {"summary": summary, "manifest": manifest, "round_plan": None, + "changed": False} + + +def _read_json(path: Path) -> Optional[Dict[str, Any]]: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + return value if isinstance(value, dict) else None + + +def _last_result(stdout: str) -> str: + result = "" + for line in stdout.splitlines(): + try: + event = json.loads(line) + except ValueError: + continue + if event.get("type") == "result": + result = str(event.get("result") or "") + return result + + +def build_evolver(mode: str, *, model: str = DEFAULT_EVOLVE_MODEL, + timeout_seconds: float = 3600) -> Evolver: + if mode == "agent": + return AgentEvolver(model=model, timeout_seconds=timeout_seconds) + return DryRunEvolver() diff --git a/metainfer/tasks/harness_evolve/orchestrator/mechanism_checks.py b/metainfer/tasks/harness_evolve/orchestrator/mechanism_checks.py new file mode 100644 index 00000000..22976fb1 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/mechanism_checks.py @@ -0,0 +1,352 @@ +"""Mechanism gate: check whether a harness change was actually exercised. + +The Evolve Agent declares, per change, *why* it should work (a +``mechanism_signature``) and — since this module landed — how to verify that +the mechanism really ran (a structured ``mechanism_check``). This module +collects the candidate iteration's evidence and grades each declared change: + + ``hit`` the declared mechanism was observed + ``partial`` planner/agent activity observed, but not the declared plan + ``unobserved`` nothing to observe (no record) — *not* a contradiction + ``contradicted`` the change demonstrably did not take effect (e.g. the + candidate harness revision was never loaded, or the + planner produced no plan records although the change + requires it) + +Grades feed ``decision_engine``: under the default lenient policy a real +performance win may promote with ``hit``/``partial``/``unobserved`` evidence +(the latter recorded as PROMOTE_UNEXPLAINED), while a *contradiction* forces +one confirmation round so an implausible explanation cannot silently win. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any, Dict, List, Mapping, Optional, Tuple + +from .rounds import child_dir_map + +_PLAN_ID_KEYS = ("plan_id", "planner_plan_id", "plan") +_GATE_EVENT_HINTS = ("gate", "guard", "threshold", "reject", "blocked") + + +def _load_json(path: Path) -> Dict[str, Any]: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {} + return value if isinstance(value, dict) else {} + + +def _jsonl(path: Path) -> List[Dict[str, Any]]: + out: List[Dict[str, Any]] = [] + if not path.is_file(): + return out + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict): + out.append(row) + return out + + +def _candidate_gates(candidate_workspace: Optional[Path]) -> Dict[str, Any]: + """Gate values the candidate harness declares (its gates.yaml + wiring).""" + if candidate_workspace is None: + return {} + root = Path(candidate_workspace) + try: + import yaml + manifest = yaml.safe_load( + (root / "manifest.yaml").read_text(encoding="utf-8")) or {} + data = yaml.safe_load( + (root / "gates.yaml").read_text(encoding="utf-8")) or {} + except (OSError, ValueError): + return {} + components = manifest.get("components") or {} + gates_entry = components.get("gates") if isinstance(components, dict) else {} + return { + "gates": data.get("gates") if isinstance(data, dict) else None, + "wired": bool((gates_entry or {}).get("wired", True)), + } + + +def collect_candidate_evidence(exp: Path, iteration: int, + planner_enabled: Optional[bool] = None, + candidate_workspace: Optional[Path] = None, + ) -> Dict[str, Any]: + """Gather observable evidence from one iteration's DKAO children.""" + exp = Path(exp) + children_root = exp / "children" + snapshot_manifest = ( + exp / "runs" / f"iteration_{iteration:03d}" / "input" / "workspace" + / "manifest.yaml" + ) + candidate_revision = "" + try: + import yaml + candidate_revision = str( + (yaml.safe_load(snapshot_manifest.read_text(encoding="utf-8")) + or {}).get("revision") or "" + ) + except (OSError, ValueError): + candidate_revision = "" + + observed_gate_snapshots: List[Dict[str, Any]] = [] + agent_plan_ids: List[str] = [] + planner_plan_ids: List[str] = [] + harness_revisions: List[str] = [] + gate_events: List[str] = [] + children: Dict[str, Any] = {} + # An iteration can own several attempt directories (a first run, its + # environment retries, and the generalization retake): evidence is gathered + # from all of them, because a mechanism that fired in any attempt fired. + for child in (child_dir_map(children_root, iteration).values() + if children_root.is_dir() else []): + workspace = child / "workspace" + child_plans: List[str] = [] + for exp_file in (workspace / "workers").glob( + "*/runs/*/experiments.jsonl"): + for row in _jsonl(exp_file): + for key in _PLAN_ID_KEYS: + value = row.get(key) + if isinstance(value, str) and value.strip(): + child_plans.append(value.strip()) + break + agent_plan_ids.extend(child_plans) + + # Hard evidence: the plans the DKAO planner itself selected for + # this worker's rounds (written by _round_strategy_text). + for plans_file in (workspace / "workers").glob( + "*/planner_plans.jsonl"): + for row in _jsonl(plans_file): + value = row.get("plan_id") + if isinstance(value, str) and value.strip(): + planner_plan_ids.append(value.strip()) + + scaffold = _load_json(workspace / "main" / "scaffold_manifest.json") + harness = scaffold.get("harness") if isinstance(scaffold, dict) else {} + revision = "" + if isinstance(harness, dict): + revision = str(harness.get("revision") or "") + if revision: + harness_revisions.append(revision) + + gate_snapshot = _load_json(child / "state" / "gates_effective.json") + if gate_snapshot: + observed_gate_snapshots.append(gate_snapshot) + + for event in _jsonl(child / "state" / "timeline.jsonl"): + etype = str(event.get("type") or "") + if any(hint in etype for hint in _GATE_EVENT_HINTS): + gate_events.append(etype) + + children[child.name] = { + "agent_plan_ids": child_plans, + "harness_revision": revision, + } + + return { + "iteration": iteration, + "planner_enabled": planner_enabled, + "candidate_harness_revision": candidate_revision, + "observed_harness_revisions": sorted(set(harness_revisions)), + # ``plan_ids`` stays the agents' self-reported ids for backwards + # compatibility; ``planner_plan_ids`` is the authoritative record. + "plan_ids": agent_plan_ids, + "agent_plan_ids": agent_plan_ids, + "planner_plan_ids": planner_plan_ids, + "plan_id_source": ("planner" if planner_plan_ids + else ("agent" if agent_plan_ids else "none")), + "candidate_gates": _candidate_gates(candidate_workspace), + "observed_gates": observed_gate_snapshots, + "gate_events": sorted(set(gate_events)), + "children": children, + } + + +def _grade_harness_revision(change: Mapping[str, Any], + evidence: Mapping[str, Any]) -> Tuple[str, str]: + observed = list(evidence.get("observed_harness_revisions") or []) + expected = str(evidence.get("candidate_harness_revision") or "") + if not observed: + return "unobserved", "no child recorded which harness revision it loaded" + if expected and all(rev == expected for rev in observed): + return "hit", f"candidate harness revision {expected} was loaded" + if expected and all(rev != expected for rev in observed): + return ("contradicted", + f"children loaded {observed} but the candidate revision is " + f"{expected}: the change never took effect") + return "partial", f"harness revisions observed: {observed}" + + +def _grade_plan_ids(change: Mapping[str, Any], + evidence: Mapping[str, Any]) -> Tuple[str, str]: + check = change.get("mechanism_check") or {} + planner_ids = list(evidence.get("planner_plan_ids") or []) + agent_ids = list(evidence.get("agent_plan_ids") + or evidence.get("plan_ids") or []) + observed = planner_ids or agent_ids + expect_any = [str(x) for x in (check.get("expect_any") or [])] + expect_order = [str(x) for x in (check.get("expect_order") or [])] + if not expect_any and not expect_order: + return "unobserved", "mechanism_check declares no expected plan ids" + # A disabled planner is hard counter-evidence: a planner_policy change + # cannot have been consumed, whatever plan ids the agents reported for + # themselves (those come from the workers' own proposals). + if evidence.get("planner_enabled") is False: + return ("contradicted", + "the planner was disabled for this iteration, so a " + "planner_policy change could not have taken effect" + + (f" (agent-reported plan ids: {sorted(set(observed))})" + if observed else "")) + if not observed: + return ("unobserved", + "no planner plan records were produced this iteration") + if not planner_ids: + # Only the workers' own proposals mention plans; that cannot prove the + # planner drove the mandate, so it is soft evidence at best. + hits_any = [pid for pid in agent_ids + if pid in expect_any or pid in expect_order] + if hits_any: + return ("partial", + "plan ids match only agent self-reports (no planner " + f"record): {sorted(set(hits_any))}") + return ("partial", + f"agent-reported plan ids {sorted(set(agent_ids))} do not " + f"match the declared ones") + if expect_order: + idx = 0 + for pid in observed: + if idx < len(expect_order) and pid == expect_order[idx]: + idx += 1 + if idx == len(expect_order): + return "hit", f"observed expected plan order {expect_order}" + hit_any = [pid for pid in observed if pid in expect_any] + if hit_any: + return "hit", f"observed declared plan(s) {sorted(set(hit_any))}" + return ("partial", + f"planner produced plans {sorted(set(observed))} but none of the " + f"declared ones {sorted(set(expect_any + expect_order))}") + + +def _grade_gate_events(change: Mapping[str, Any], + evidence: Mapping[str, Any]) -> Tuple[str, str]: + check = change.get("mechanism_check") or {} + expected = [str(x) for x in (check.get("expect_any") or [])] + observed = list(evidence.get("gate_events") or []) + if not expected: + return "unobserved", "mechanism_check declares no expected gate events" + hits = [event for event in observed if event in expected] + if hits: + return "hit", f"observed declared gate event(s) {hits}" + return "unobserved", f"declared gate events absent (observed {observed})" + + +def _grade_gate_values(change: Mapping[str, Any], + evidence: Mapping[str, Any]) -> Tuple[str, str]: + """Did children actually run with the gate values the harness declares?""" + candidate = evidence.get("candidate_gates") or {} + declared = candidate.get("gates") + observed = list(evidence.get("observed_gates") or []) + if not candidate: + return "unobserved", "candidate harness gates could not be read" + if not candidate.get("wired", True): + return ("contradicted", + "gates.yaml is not wired in the candidate manifest, so its " + "values cannot reach the runtime") + if not observed: + return ("unobserved", + "children recorded no effective-gate snapshot for this run") + check = change.get("mechanism_check") or {} + paths = [str(p) for p in (check.get("expect_any") or [])] + for snapshot in observed: + seen = snapshot.get("gates") + if not isinstance(seen, Mapping) or declared is None: + continue + if paths: + mismatched = [] + for path in paths: + node_seen: Any = seen + node_declared: Any = declared + for part in path.split("."): + node_seen = node_seen.get(part) if isinstance(node_seen, Mapping) else None + node_declared = (node_declared.get(part) + if isinstance(node_declared, Mapping) else None) + if node_seen != node_declared: + mismatched.append(path) + if mismatched: + return ("contradicted", + f"children ran with different values for {mismatched}") + return "hit", f"children ran with the declared values for {paths}" + if dict(seen) == dict(declared): + return "hit", "children ran with the candidate gate values" + return ("contradicted", + "children ran with gate values that differ from the candidate") + return "unobserved", "no comparable gate snapshot" + + +_GRADERS = { + "harness_revision": _grade_harness_revision, + "planner_plan_ids": _grade_plan_ids, + "planner_plan_id_sequence": _grade_plan_ids, + "gate_events": _grade_gate_events, + "gate_values": _grade_gate_values, +} + + +def evaluate_changes( + manifest: Optional[Mapping[str, Any]], + evidence: Mapping[str, Any], +) -> Tuple[Dict[str, str], Dict[str, Any]]: + """Grade every declared change; returns (checks, detail).""" + changes = [c for c in ((manifest or {}).get("changes") or []) + if isinstance(c, Mapping)] + checks: Dict[str, str] = {} + detail: Dict[str, Any] = {"changes": []} + for change in changes: + cid = str(change.get("id") or "unknown") + check = change.get("mechanism_check") + declared = bool(change.get("mechanism_signature")) + if not declared: + checks[cid] = "unobserved" + detail["changes"].append({ + "id": cid, "grade": "unobserved", + "reason": "change declares no mechanism_signature", + }) + continue + if not isinstance(check, Mapping): + checks[cid] = "unobserved" + detail["changes"].append({ + "id": cid, "grade": "unobserved", + "reason": ("no structured mechanism_check declared; " + "signature kept as documentation only"), + }) + continue + kind = str(check.get("kind") or "").strip() + grader = _GRADERS.get(kind) + if grader is None: + checks[cid] = "unobserved" + detail["changes"].append({ + "id": cid, "grade": "unobserved", + "reason": f"unknown mechanism_check kind {kind!r}", + }) + continue + grade, reason = grader(change, evidence) + checks[cid] = grade + detail["changes"].append({ + "id": cid, "grade": grade, "kind": kind, "reason": reason, + "expect": {k: v for k, v in check.items() if k != "kind"}, + }) + detail["planner_plan_ids"] = list(evidence.get("planner_plan_ids") or []) + detail["agent_plan_ids"] = list(evidence.get("agent_plan_ids") or []) + detail["summary"] = { + "hit": [c for c, g in checks.items() if g == "hit"], + "partial": [c for c, g in checks.items() if g == "partial"], + "unobserved": [c for c, g in checks.items() if g == "unobserved"], + "contradicted": [c for c, g in checks.items() if g == "contradicted"], + } + return checks, detail diff --git a/metainfer/tasks/harness_evolve/orchestrator/phases.py b/metainfer/tasks/harness_evolve/orchestrator/phases.py new file mode 100644 index 00000000..8a5de058 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/phases.py @@ -0,0 +1,40 @@ +"""harness_evolve phases (state graph) for the shell.""" + +from __future__ import annotations + +from typing import Any, Dict, List + +PREPARE = "prepare" +EVALUATE = "evaluate" +ANALYZE = "analyze" +EVOLVE = "evolve" +REPORT = "report" +FINISHED = "finished" + +_ORDER = [PREPARE, EVALUATE, ANALYZE, EVOLVE, REPORT, FINISHED] +_LABELS = { + PREPARE: "Prepare", + EVALUATE: "Evaluate", + ANALYZE: "Analyze", + EVOLVE: "Evolve", + REPORT: "Report", + FINISHED: "Finished", +} + + +def graph_payload(current: str, **_: Any) -> Dict[str, Any]: + nodes = [ + {"id": phase, "label": _LABELS[phase], "is_terminal": phase == FINISHED} + for phase in _ORDER + ] + edges = [ + {"from": _ORDER[i], "to": _ORDER[i + 1], "label": "ok", + "outcomes": ["ok"]} + for i in range(len(_ORDER) - 1) + ] + return { + "current": current, + "nodes": nodes, + "edges": edges, + "terminal_nodes": [FINISHED], + } diff --git a/metainfer/tasks/harness_evolve/orchestrator/pipeline.py b/metainfer/tasks/harness_evolve/orchestrator/pipeline.py new file mode 100644 index 00000000..7fa6894d --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/pipeline.py @@ -0,0 +1,2422 @@ +"""harness_evolve main pipeline (AHE outer loop). + +Iteration layout (double generation, mirrors ahe-ref semantics): + runs/iteration_NNN/ + input/workspace/ snapshot of the workspace evaluated this loop + input/benchmark/results.json + input/benchmark/scored_ids.json (pool mode: ids scored this round) + input/diff.json vs previous loop's eval (overlap ids) + input/analysis/overview.md + evolve/change_manifest.json + evolve/round_plan.json (pool mode: next round's question set) + evolve/evolve_summary.md + evolve/workspace/ (only when the evolver reported a change) + +Root artifacts: best_ever.json, iteration_scores.jsonl, report.md. +Auto rollback: when an iteration's decision pass rate falls below best-ever, the +best snapshot is restored over the live workspace (workspace/). + +Two suite modes: +- legacy (pool_source=""): evaluate cfg.suite every round (v0 behaviour). +- pool v2 (pool_source="builtin"|): questions come from the registered + pool; the single Evolve role's round_plan picks next-round sets; machine + guardrails enforce in-pool + >=60% overlap + budget; decisions (best-ever / + rollback / attribution) are computed on the overlap pairs. +""" + +from __future__ import annotations + +import json +import subprocess +import sys +import time +import traceback +from pathlib import Path +from typing import Any, Dict, List, Optional + +from .adapters.eval import build_evaluator +from .champion_selection import ( + Generation, best_generation, load_generations, record_generation, +) +from .attribution import ( + copy_tree, diff_results, evaluate_changes, pass_rate, replace_tree, save_json, +) +from .config import ExperimentConfig, InstanceSpec, harness_seed_dir +from .decision_engine import DecisionPolicy, decide +from .evolve import DEFAULT_EVOLVE_MODEL, build_evolver +from .phases import ( + ANALYZE, EVALUATE, EVOLVE, FINISHED, PREPARE, REPORT, +) +from .pool import Pool, builtin_pool +from .promotion import ( + restore_promoted_harness, write_pending_promotion, +) +from .rounds import ( + GENERALIZATION_ROUND_COST, OPERATORS_PER_GATE, PERFORMANCE_ROUND_COST, + STAGE_BASELINE, STAGE_GENERALIZATION, STAGE_PERFORMANCE, + baseline_round_verdict, child_dir_map, clear_pending_generalization, + evaluate_gate, group_state, normalized_per_gate, + read_pending_generalization, record_paper, resolve_child_dir, round_stage, + select_round_questions, write_pending_generalization, +) +from .variant import ( + load_variant_table, pool_instances, record_variants, save_variant_table, +) +from .question_roles import select_questions +from .round_plan import ( + RoundPlan, enforce_guardrails, load_rotation, load_round_plan, + mark_selected, +) +from .state import ( + append_timeline, harness_revision, loaded_harness_revision, + read_target_iterations, set_run, write_harness_revision, +) + +BEST_EVER_FILE = "best_ever.json" +SCORES_FILE = "iteration_scores.jsonl" +#: This run's question plan (performance set + the fixed generalization paper). +ROUND_QUESTIONS_FILE = "round_questions.json" +REPORT_FILE = "report.md" +LAST_SCORED_FILE = "last_scored_ids.json" + + +def _uses_variant_rounds(cfg: ExperimentConfig) -> bool: + """Whether this run measures operators against the variant table. + + This is the **default protocol** (operator decision, 2026-09-15): one HE + run chases one promotable harness, every round draws fresh random operators + judged against their current variant, and round 1 fixes the generalization + paper. Pool mode without a real pool file, and legacy suite mode, keep the + old path; ``round_questions: legacy`` restores it explicitly. + """ + raw = str(cfg.answers.get("round_questions") or "").strip().lower() + if raw in {"legacy", "suite", "off", "0", "false"}: + return False + if raw in {"variant", "on", "1", "true", "yes"}: + return True + if not cfg.pool_mode: + return False + source = str(cfg.answers.get("question_pool") or cfg.pool_source or "").strip() + if not source or source == "builtin": + return False # nothing to draw operators from: stay on the old path + return Path(source).expanduser().is_file() + + +def _question_pool_path(cfg: ExperimentConfig) -> Optional[Path]: + """The pool file holding the operators a round may draw from.""" + raw = str(cfg.answers.get("question_pool") or cfg.pool_source or "").strip() + if not raw or raw == "builtin": + return None + path = Path(raw).expanduser() + return path if path.is_file() else None + + +def _round_rng(cfg: ExperimentConfig, iteration: int, + stage: str = "") -> "tuple[Any, str]": + """This round's draw RNG, plus the seed it came from. + + The draw is what the round is judged on, so "why these four operators?" has + to be answerable later. ``round_seed`` in the form pins it explicitly; with + no pin the seed is derived from the task id, the iteration and the stage, + which is still a fresh draw per round (A_i ≠ A_{i-1}) but reproducible when + somebody has to re-run one pass. + + Returns ``(rng, seed_label)``. + """ + import random + + pinned = cfg.answers.get("round_seed") + if pinned in (None, ""): + seed: Any = f"{cfg.task_id}:{iteration}:{stage or 'round'}" + return random.Random(str(seed)), str(seed) + try: + return random.Random(int(pinned) + int(iteration)), f"round_seed={pinned}+{iteration}" + except (TypeError, ValueError): + seed = f"{pinned}:{iteration}" + return random.Random(seed), seed + + +def _legacy_protocol_reason(cfg: ExperimentConfig) -> str: + """Why this run is not on the fixed protocol, in one sentence. + + ``FLOW.md`` describes the only protocol a new HE task may use (variant + rounds: a fixed generalization paper, two gates, rounds sized in waves of 4). + The older paths still exist for offline suite runs and for experiments + started before 2026-09-15 — but "opt-in" was *invisible*: nothing said the + run had left the protocol, so a pass that judges neither gate looked like any + other. This sentence is what the operator reads instead. + """ + raw = str(cfg.answers.get("round_questions") or "").strip().lower() + if raw in {"legacy", "suite", "off", "0", "false"}: + return f"round_questions={raw!r} was requested explicitly" + source = str(cfg.answers.get("question_pool") or cfg.pool_source or "").strip() + if not source: + return ("no question pool is configured, so the questions come from the " + "inline suite and neither gate is judged") + if source == "builtin": + return "the built-in demo pool is in use instead of a measured pool file" + path = Path(source).expanduser() + if not path.is_file(): + return f"the configured pool file does not exist: {source}" + return "this run is not on the variant-round protocol" + + +def _announce_legacy_protocol(cfg: ExperimentConfig, exp: Path) -> None: + """Say out loud that this run bypasses the fixed protocol. Never raises. + + Three places, because the three audiences are different: the orchestrator log + (whoever is watching the process), the task timeline (the run's own record, + and what the WebUI lists), and ``protocol.json`` (what the task page reads to + put a banner on screen instead of a run that merely looks quiet). + """ + reason = _legacy_protocol_reason(cfg) + payload = { + "protocol": "legacy", + "mode": "legacy", + "reason": reason, + "round_questions": cfg.answers.get("round_questions") or None, + "pool_source": cfg.pool_source or None, + "pool_file": (str(_question_pool_path(cfg)) if _question_pool_path(cfg) + else None), + "since": time.time(), + "policy": ("FLOW.md §1–§4 is the fixed protocol for new tasks: a frozen " + "generalization paper, the performance/generalization gates, " + "rounds sized in waves of 4. This run judges neither gate, so " + "its rounds cannot promote a harness."), + } + try: + append_timeline(cfg.state_dir, "legacy_protocol_selected", payload) + except Exception: # noqa: BLE001 - progress records are best effort + pass + try: + from .attribution import save_json + + save_json(Path(exp) / "protocol.json", payload) + except Exception: # noqa: BLE001 + pass + try: + print(f"[harness_evolve] WARNING: running the LEGACY protocol ({reason}). " + f"New tasks must use a measured pool file — see FLOW.md", + file=sys.stderr, flush=True) + except OSError: + pass + + +def _judgeable_ids(instances: Dict[str, Dict[str, Any]], + table: Dict[str, Any]) -> List[str]: + """Pool operators that can produce a win/loss (they have a variant).""" + from .variant import resolve_variant + + out: List[str] = [] + for iid in instances: + try: + if resolve_variant(table or {}, iid, instances=instances) is not None: + out.append(str(iid)) + except Exception: # noqa: BLE001 - a malformed row is simply not judgeable + continue + return sorted(out) + + +def _mechanism_gate_for_round(cfg: ExperimentConfig, exp: Path, + iteration: int) -> Dict[str, Any]: + """Grade the previous round's declared mechanism against what was observed. + + Round N evaluates the harness change that round N-1's evolve produced, so + the manifest to grade is the previous round's one. Round 1 tests the seed, + which declares nothing: the gate is then ``NOT_DECLARED``, which the lenient + policy allows. + """ + manifest = None + if iteration > 1: + manifest = _load_json( + exp / "runs" / f"iteration_{iteration - 1:03d}" / "evolve" + / "change_manifest.json") + if not manifest: + return {"status": "NOT_DECLARED", "hits": [], "misses": [], + "reason": "no previous-round change manifest to verify"} + try: + from .mechanism_checks import ( + collect_candidate_evidence, evaluate_changes, + ) + from .decision_engine import _mechanism_status + + evidence = collect_candidate_evidence( + exp, iteration, + planner_enabled=_truthy(cfg.answers.get("planner_enabled"), + default=True), + candidate_workspace=(exp / "runs" / f"iteration_{iteration:03d}" + / "input" / "workspace")) + checks, detail = evaluate_changes(manifest, evidence) + status = _mechanism_status(manifest, checks) + status["detail"] = detail + append_timeline(cfg.state_dir, "mechanism_gate", { + "iteration": iteration, "status": status.get("status"), + "hits": status.get("hits"), "misses": status.get("misses"), + "contradicted": status.get("contradicted"), + }) + return status + except Exception as exc: # noqa: BLE001 - the gate is advisory here + append_timeline(cfg.state_dir, "mechanism_check_failed", + {"iteration": iteration, "error": repr(exc)}) + return {"status": "NOT_CHECKED", "hits": [], "misses": [], + "error": repr(exc)} + + +def _seed_workspace(cfg: ExperimentConfig) -> Path: + workspace = cfg.exp_dir / "workspace" + if not (workspace / "manifest.yaml").exists(): + copy_tree(harness_seed_dir(cfg.harness_source), workspace) + if not (workspace / ".git").exists(): + subprocess.run(["git", "init"], cwd=workspace, check=True, + stdout=subprocess.DEVNULL) + subprocess.run(["git", "config", "user.name", "MetaInfer AHE"], + cwd=workspace, check=True) + subprocess.run(["git", "config", "user.email", "ahe@metainfer.local"], + cwd=workspace, check=True) + subprocess.run(["git", "add", "-A"], cwd=workspace, check=True) + subprocess.run(["git", "commit", "-m", "baseline: seed harness"], + cwd=workspace, check=True, stdout=subprocess.DEVNULL) + return workspace + + +def _head_revision(workspace: Path) -> str: + return subprocess.check_output( + ["git", "rev-parse", "HEAD"], cwd=workspace, text=True + ).strip() + + +def _current_revision(workspace: Path) -> str: + """HEAD of the live harness workspace, or ``""`` when it is not a repo. + + Used to stamp each round with *its own* revision: reading it from the + previous round's manifest left ``workspace_revision`` empty in every + generation record, which is what made the rollback machinery unable to + restore a precise revision. + """ + try: + return _head_revision(workspace) + except (OSError, subprocess.SubprocessError): + return "" + + +def _commit_workspace(workspace: Path, iteration: int) -> str: + """Commit one evolved harness generation; return the exact revision.""" + subprocess.run(["git", "add", "-A"], cwd=workspace, check=True) + diff = subprocess.run( + ["git", "diff", "--cached", "--quiet"], cwd=workspace + ) + if diff.returncode != 0: + subprocess.run( + ["git", "commit", "-m", f"evolve: iteration {iteration}"], + cwd=workspace, check=True, stdout=subprocess.DEVNULL, + ) + return subprocess.check_output( + ["git", "rev-parse", "HEAD"], cwd=workspace, text=True + ).strip() + + +def _load_json(path: Path) -> Optional[Dict[str, Any]]: + if not path.is_file(): + return None + try: + return json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + +def _load_jsonl(path: Path) -> List[Dict[str, Any]]: + out: List[Dict[str, Any]] = [] + if not path.is_file(): + return out + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict): + out.append(row) + return out + + +def _load_ids(path: Path) -> List[str]: + data = _load_json(path) + return list(data) if isinstance(data, list) else [] + + +def _save_scores(cfg: ExperimentConfig, entry: Dict[str, Any]) -> None: + with (cfg.exp_dir / SCORES_FILE).open("a", encoding="utf-8") as fh: + fh.write(json.dumps(entry, ensure_ascii=False) + "\n") + + +def _update_best_ever(cfg: ExperimentConfig, iteration: int, rate: float, + snapshot: Path) -> Dict[str, Any]: + best = _load_json(cfg.exp_dir / BEST_EVER_FILE) or {} + if not best or rate > float(best.get("pass_rate", -1.0)): + best = { + "iteration": iteration, + "pass_rate": rate, + "snapshot_dir": str(snapshot), + } + save_json(cfg.exp_dir / BEST_EVER_FILE, best) + return best + + +def _write_overview(input_dir: Path, iteration: int, + results: Dict[str, Dict[str, Any]], + diff: Optional[Dict[str, Any]]) -> None: + lines = [ + f"# Analysis overview — iteration {iteration}", + "", + "| instance | passed | median_us | reason |", + "|---|---:|---:|---|", + ] + for inst_id, r in sorted(results.items()): + lines.append( + f"| {inst_id} | {r.get('passed')} | {r.get('median_us')} | " + f"{r.get('reason')} |" + ) + if diff: + lines.append("") + lines.append(f"- flipped: {diff.get('flipped')}") + lines.append(f"- regressed: {diff.get('regressed')}") + analysis = input_dir / "analysis" + analysis.mkdir(parents=True, exist_ok=True) + (analysis / "overview.md").write_text("\n".join(lines) + "\n", + encoding="utf-8") + + +def _variant_instance_spec(entry: Dict[str, Any], *, budget_rounds: int) -> InstanceSpec: + """Build an :class:`InstanceSpec` straight from a pool entry (no Pool object).""" + contract = dict(entry.get("contract") or {}) + variant = entry.get("best_known_us") + baseline = entry.get("baseline_us") + threshold = variant if variant is not None else (baseline or 0.0) + return InstanceSpec.from_dict({ + "id": str(contract.get("id") or entry.get("id") or ""), + "shape": { + "model": str(contract.get("model") or ""), + "tp_size": int(contract.get("tp_size") or 8), + "operator": str(contract.get("operator") or ""), + "M": int(contract.get("M") or 0), + "N": int(contract.get("N") or 0), + "K": int(contract.get("K") or 0), + }, + "budget_rounds": int(budget_rounds), + "warm_start_from": "best_known", + "pass_median_us_le": float(threshold or 0.0), + }) + + +def _pool_instance_spec(inst: Any, budget_rounds: int = 4) -> InstanceSpec: + def _g(name: str, default: Any = "") -> Any: + value = getattr(inst, name, default) + return default if value is None else value + + return InstanceSpec.from_dict({ + "id": inst.id, + "shape": { + "model": _g("model"), + "tp_size": _g("tp_size", 8), + "operator": _g("operator"), + "M": _g("M", 0), + "N": _g("N", 0), + "K": _g("K", 0), + }, + "budget_rounds": budget_rounds, + "warm_start_from": "best_known", + # Dry evaluator uses this to synthesize median; choose the registered + # best/target scale so pool.auto_pass still yields first-fail/rest-pass. + "pass_median_us_le": ( + getattr(inst, "best_known_us", None) + or 0.5 * float(getattr(inst, "baseline_us", 1.0)) + ), + }) + + +def _harness_validate_error(workspace: Path) -> Optional[str]: + """Malformed harness component (bad YAML) must never be benchmarked. + + Returns the first offending file's error text, or None when valid. + """ + import yaml as _yaml + for path in sorted(workspace.rglob("*.yaml")) + sorted(workspace.rglob("*.yml")): + try: + _yaml.safe_load(path.read_text(encoding="utf-8")) + except Exception as exc: # noqa: BLE001 + return f"{path.name}: {exc}" + return None + + +def _wired_scope_violations(before: Path, after: Path) -> List[Dict[str, str]]: + """Components a harness change touched although the manifest says unwired. + + A component with ``wired: false`` is not read by the runtime, so changing + it cannot show any effect — the round would be spent for nothing. The + manifest is therefore load-bearing: it declares which components apply. + """ + import yaml as _yaml + + before = Path(before) + after = Path(after) + manifest_path = after / "manifest.yaml" + if not manifest_path.is_file(): + return [] + try: + manifest = _yaml.safe_load(manifest_path.read_text(encoding="utf-8")) or {} + except (OSError, ValueError): + return [] + components = manifest.get("components") + if not isinstance(components, dict): + return [] + out: List[Dict[str, str]] = [] + for name, entry in components.items(): + if not isinstance(entry, dict) or entry.get("wired", True): + continue + relative = str(entry.get("path") or "") + if not relative: + continue + old_file = before / relative + new_file = after / relative + try: + old_text = old_file.read_text(encoding="utf-8") if old_file.is_file() else None + new_text = new_file.read_text(encoding="utf-8") if new_file.is_file() else None + except OSError: + continue + if old_text != new_text: + out.append({"component": str(name), "path": relative}) + return out + + +def _load_pool(cfg: ExperimentConfig) -> Optional[Pool]: + if not cfg.pool_mode: + return None + if cfg.pool_source == "builtin": + if cfg.execution_mode == "dkao-cli": + raise ValueError( + "dkao-cli requires a measured registered pool YAML; " + "builtin contains dry-run fixture baselines only" + ) + return builtin_pool() + return Pool.from_yaml(Path(cfg.pool_source).expanduser()) + + +def _record_state(cfg: ExperimentConfig, phase: str, + iteration: Optional[int] = None, + event: Optional[str] = None, + **payload: Any) -> None: + """Mirror the outer-loop position into run.json + timeline (WebUI).""" + fields: Dict[str, Any] = {"current_phase": phase} + if phase == FINISHED: + fields["finished"] = True + fields["final_status"] = "success" + if iteration is not None: + fields["current_iteration"] = iteration + set_run(cfg.state_dir, **fields) + if event: + append_timeline(cfg.state_dir, event, + {"iteration": iteration, **payload}) + + +def completed_iterations(exp: Path) -> int: + """Highest iteration number that already has *end-of-pass* artifacts. + + "Done" is a judged pass: the variant protocol leaves ``baseline_round.json`` + (round 1), ``performance_gate.json`` / ``generalization_gate.json``, or a + ``decision.json`` behind — reading only ``results.json`` made this return 0 + for a run that had just finished three real rounds, and ``resume`` then + refused to continue (it believes nothing has been measured yet). + + ``input/stage.json`` is deliberately **not** in the list: it is written when + a pass *starts* (it records which stage this pass is). Counting it made a + round that had just begun indistinguishable from a finished one, so the + resume the sweep launches computed ``start = done + 1`` and opened the next + round in parallel with the one being measured. + """ + runs = exp / "runs" + if not runs.is_dir(): + return 0 + done = 0 + for d in sorted(runs.glob("iteration_*")): + if not d.is_dir(): + continue + input_dir = d / "input" + judged = any( + (input_dir / name).is_file() + for name in ("decision.json", "baseline_round.json", + "performance_gate.json", "generalization_gate.json") + ) or (input_dir / "benchmark" / "results.json").is_file() + if not judged: + continue + try: + done = max(done, int(d.name.split("_")[-1])) + except ValueError: + continue + return done + + +def _iteration_results(exp: Path, iteration: int) -> Optional[Dict[str, Any]]: + """Benchmark results recorded for one iteration (None when absent).""" + data = _load_json( + exp / "runs" / f"iteration_{iteration:03d}" + / "input" / "benchmark" / "results.json" + ) + if not data: + return None + results = data.get("results") + return dict(results) if isinstance(results, dict) else None + + +def _record_champion_generation(cfg: ExperimentConfig, exp: Path, *, + iteration: int, decision: Dict[str, Any], + decision_rate: Any, snapshot: Path, + live_workspace: Path, + results: Dict[str, Any], + current: Dict[str, Any]) -> Dict[str, Any]: + """Record this generation and update the champion. **Never raises.** + + Which generation the champion points at is what every later round compares + against, so losing this bookkeeping silently is not acceptable. It *used* to + be swallowed into a timeline row and nothing else (26 ``champion_record_failed + / AttributeError`` rows in one run on 2026-09-11), which left the run + comparing against a champion chain nobody could vouch for. + + A failure is now loud (stderr, i.e. the orchestrator log), carries its + traceback, and is followed by a rebuild of ``best_ever.json`` from the + generations already on disk so the chain survives the error. + + Returns ``{"best_ever", "recorded", "recovered", ...}``; the caller adopts + ``best_ever``. + """ + best = dict(current or {}) + outcome: Dict[str, Any] = {"iteration": iteration, "recorded": False, + "recovered": False, "best_ever": best} + try: + record_generation(exp, Generation( + iteration=iteration, + verdict=str(decision.get("verdict") or ""), + results=dict(results), + snapshot_dir=str(snapshot), + # This generation's own revision (not the previous round's + # manifest field, which was always empty here). + workspace_revision=_current_revision(live_workspace), + pass_count=sum(1 for r in results.values() + if isinstance(r, dict) and r.get("passed") is True), + )) + pinned = _load_json(exp / "champion_override.json") or {} + pinned_iter = int(pinned.get("champion_iteration") or 0) + measured_now = None + if pinned_iter: + # An explicit baseline (resume --champion-iteration) stays in force + # until it is changed: automation must not silently override a + # human's comparison choice. + pinned_results = _iteration_results(exp, pinned_iter) + best = { + "iteration": pinned_iter, + "pass_rate": (decision_rate + if pinned_iter == iteration else None), + "snapshot_dir": str( + exp / "runs" / f"iteration_{pinned_iter:03d}" + / "input" / "workspace"), + "workspace_revision": _current_revision(live_workspace), + "verdict": "MANUAL_BASELINE", + } + save_json(exp / BEST_EVER_FILE, best) + append_timeline(cfg.state_dir, "champion_pinned_held", { + "iteration": iteration, "champion_iteration": pinned_iter, + "pinned_results_present": bool(pinned_results), + }) + else: + measured_now = best_generation(load_generations(exp)) + if measured_now is not None and measured_now.results: + best = { + "iteration": measured_now.iteration, + "pass_rate": (decision_rate + if measured_now.iteration == iteration else None), + "snapshot_dir": measured_now.snapshot_dir or str(snapshot), + "workspace_revision": measured_now.workspace_revision, + "verdict": ("BEST_MEASURED" + if measured_now.iteration != iteration + else str(decision.get("verdict") or "")), + } + save_json(exp / BEST_EVER_FILE, best) + append_timeline(cfg.state_dir, "champion_best_measured", { + "iteration": iteration, + "champion_iteration": measured_now.iteration, + "verdict": best["verdict"], + }) + outcome.update({"recorded": True, "best_ever": best}) + return outcome + except Exception as exc: # noqa: BLE001 - never break the loop + append_timeline(cfg.state_dir, "champion_record_failed", { + "iteration": iteration, "error": repr(exc), + "traceback": traceback.format_exc()[-1500:], + }) + try: + print(f"[harness_evolve] champion bookkeeping failed at iteration " + f"{iteration}: {exc!r} — rebuilding best_ever from the " + f"generations on disk", file=sys.stderr, flush=True) + except OSError: + pass + outcome["error"] = repr(exc) + recovered = None + try: + recovered = best_generation(load_generations(exp)) + except Exception as exc2: # noqa: BLE001 - the fallback is best effort + append_timeline(cfg.state_dir, "champion_rebuild_failed", + {"iteration": iteration, "error": repr(exc2)}) + if recovered is not None and recovered.results: + best = { + "iteration": recovered.iteration, + "pass_rate": (decision_rate + if recovered.iteration == iteration else None), + "snapshot_dir": recovered.snapshot_dir or str(snapshot), + "workspace_revision": recovered.workspace_revision, + "verdict": "REBUILT_AFTER_ERROR", + } + try: + save_json(exp / BEST_EVER_FILE, best) + outcome["recovered"] = True + append_timeline(cfg.state_dir, "champion_rebuilt", { + "iteration": iteration, + "champion_iteration": recovered.iteration, + }) + except OSError: + pass + outcome["best_ever"] = best + return outcome + + +def _truthy(value: Any, default: bool = True) -> bool: + if value is None: + return default + if isinstance(value, bool): + return value + return str(value).strip().lower() not in {"0", "false", "no", "off", ""} + + +def _promote_iteration(cfg: ExperimentConfig, exp: Path, iteration: int, + results: Dict[str, Any], + instances: List[InstanceSpec], + *, freeze: bool = False, + ) -> Optional[Dict[str, Any]]: + """Promote this iteration's better-than-variant kernels into DKAO. + + A question that beats the existing variant for its shape by at least + ``promote_min_improvement_percent`` (default 3%) and did not fail + correctness becomes the new shared variant — i.e. DKAO gains a better + operator. Writes are backed up by the variant store and both the outcome + table and an append-only log are recorded for audit/rollback. + """ + if not _truthy(cfg.answers.get("promote_kernels"), default=True): + return None + children = exp / "children" + if not children.is_dir(): + return None + try: + from metainfer.tasks.dcu_kernel_auto_opt.orchestrator.variant_promote import ( # noqa: E501 + model_label_for, promote_variant, + ) + except Exception as exc: # noqa: BLE001 - promotion must never break the loop + return {"iteration": iteration, "enabled": True, "error": repr(exc), + "results": []} + threshold = float(cfg.answers.get("promote_min_improvement_percent") or 3.0) + outcomes: List[Dict[str, Any]] = [] + for inst in instances: + result = results.get(inst.id) or {} + # The question's newest attempt owns the workspace that produced the + # result (a retry runs in its own directory). + # The question's newest attempt owns the workspace that produced the + # result (a retry runs in its own directory). + child_dir = resolve_child_dir(children, iteration, inst.id) + workspace = (child_dir / "workspace") if child_dir else Path("") + if not workspace.is_dir(): + outcomes.append({"shape": inst.id, "action": "skipped", + "reason": "no child workspace for this question"}) + continue + shape = inst.shape or {} + answers = { + "operator": "Quantized GEMM", + "dtype": "INT8 W8A8", + "model": str(shape.get("model") or ""), + "tp_size": shape.get("tp_size"), + } + outcome = promote_variant( + workspace_dir=workspace, + answers=answers, + shape_id=inst.id, + source_task=f"{cfg.task_id}/iteration_{iteration:03d}/{inst.id}", + correctness_ok=result.get("correctness_ok"), + min_improvement_percent=threshold, + tp=shape.get("tp_size"), + m=shape.get("M"), + model_label=(model_label_for(inst.id, str(shape.get("model") or "")) + or None), + dry_run=freeze, + ) + outcome["ahe_passed"] = result.get("passed") + outcome["ahe_median_us"] = result.get("median_us") + outcomes.append(outcome) + if freeze: + with (exp / "promotion_candidates.jsonl").open( + "a", encoding="utf-8") as fh: + fh.write(json.dumps({ + "ts": time.time(), "iteration": iteration, + "shape": inst.id, "workspace": str(workspace), + "source_task": f"{cfg.task_id}/iteration_{iteration:03d}/{inst.id}", + "answers": answers, "tp": shape.get("tp_size"), + "m": shape.get("M"), + "model_label": (model_label_for( + inst.id, str(shape.get("model") or "")) or None), + "action": outcome.get("action"), + "ahe_passed": result.get("passed"), + "ahe_median_us": result.get("median_us"), + "variant_median_us": outcome.get("old_median_us"), + }, ensure_ascii=False) + "\n") + + promoted = [o["shape"] for o in outcomes + if o.get("action") in {"added", "updated"}] + payload = { + "iteration": iteration, + "enabled": True, + "frozen": bool(freeze), + "min_improvement_percent": threshold, + "promoted": promoted, + "results": outcomes, + "note": ("frozen: candidates recorded, promotion happens once at the end" + if freeze else None), + } + save_json(exp / "runs" / f"iteration_{iteration:03d}" / "promotion.json", + payload) + with (exp / "promotion_log.jsonl").open("a", encoding="utf-8") as fh: + for outcome in outcomes: + fh.write(json.dumps({ + "ts": time.time(), "iteration": iteration, + "task_id": cfg.task_id, **outcome, + }, ensure_ascii=False) + "\n") + append_timeline(cfg.state_dir, "kernel_promotion", { + "iteration": iteration, "promoted": promoted, + "min_improvement_percent": threshold, "frozen": bool(freeze), + }) + return payload + + +def _promote_baseline_kernels(cfg: ExperimentConfig, exp: Path, + variant_pool: Dict[str, Any], + results: Dict[str, Any]) -> Dict[str, Any]: + """Round 1's faster kernels go into the variant pool — as the new bar. + + A kernel result is a fact about the hardware, not a verdict about a + harness: the operator's rule is that the baseline round's improvements enter + the variant pool immediately, so every later gate (and DKAO's own + ``warm_start: best_known``) compares against what the baseline round + actually achieved. The *harness* still waits for approval — this only moves + kernels. + + Only operators the round actually measured are touched, only a real + improvement (>= ``promote_min_improvement_percent`` faster than the recorded + best) is written, and the pool file is backed up before being rewritten. + """ + if not results: + return {"ok": True, "updated": [], "promoted": [], + "skipped": "no measurements"} + pool_path = _question_pool_path(cfg) + if pool_path is None: + return {"ok": True, "updated": [], "promoted": [], + "skipped": "no question pool file"} + try: + from .promotion import promote_kernels_for_round + except Exception as exc: # noqa: BLE001 - promotion must never break the loop + return {"ok": False, "updated": [], "errors": [repr(exc)]} + measured = { + iid: row for iid, row in (results or {}).items() + if isinstance(row, dict) and row.get("correctness_ok") is not False + and row.get("median_us") is not None + } + # Same improvement rule as every other kernel write (production default 3%), + # so a measurement inside the noise band cannot churn the pool. + threshold = float(cfg.answers.get("promote_min_improvement_percent") or 3.0) + kept: Dict[str, Any] = {} + for iid, row in measured.items(): + old = (variant_pool.get(iid) or {}).get("best_known_us") + try: + new = float(row["median_us"]) + old_value = float(old) if old is not None else None + except (TypeError, ValueError): + continue + if new <= 0: + continue + if old_value is not None and \ + (old_value - new) / old_value * 100.0 < threshold: + continue + kept[iid] = row + if not kept: + append_timeline(cfg.state_dir, "baseline_kernels_promoted", { + "iteration": 1, "operators": [], "pool": str(pool_path), + "reason": f"no operator improved by >= {threshold:.2f}%", + }) + return {"ok": True, "updated": [], "promoted": [], + "pool": str(pool_path), + "reason": f"no operator improved by >= {threshold:.2f}%"} + version = "baseline:it1" + outcome = promote_kernels_for_round( + exp, pool_path, kept, version=version, iteration=1, + reason="baseline round: faster kernels enter the variant pool") + updated = outcome.get("updated") or [] + outcome["promoted"] = sorted(str(u["operator_id"]) for u in updated) + append_timeline(cfg.state_dir, "baseline_kernels_promoted", { + "iteration": 1, "operators": [u["operator_id"] for u in updated], + "pool": str(pool_path), "backup": outcome.get("backup"), + "min_improvement_percent": threshold, + }) + for change in updated: + # Keep the experiment's own view in step with the pool it just moved. + iid = str(change["operator_id"]) + entry = variant_pool.get(iid) + if isinstance(entry, dict): + entry["best_known_us"] = change["median_us"] + return outcome + + +_FAILING_VERDICTS = {"REJECT", "REJECT_OVERFIT", "INCONCLUSIVE"} +_RECOVERING_VERDICTS = {"PROMOTE", "PROMOTE_UNEXPLAINED", "NO_SIGNAL"} + + +def _next_fail_streak(verdict: str, streak: int) -> int: + """Consecutive failing rounds. + + INCONCLUSIVE counts: rounds where nothing could be measured are exactly the + situation that used to spin forever (a run reached 513 rounds with 510 + REJECTs while a shared GPU made every measurement unusable). + """ + verdict = str(verdict or "") + if verdict in _FAILING_VERDICTS: + return int(streak) + 1 + if verdict in _RECOVERING_VERDICTS: + return 0 + return int(streak) + + +def _stop_requested(exp: Path) -> Optional[Dict[str, Any]]: + """A request (from the evaluator) to stop the whole experiment. + + Raised when no device satisfied the gate for the configured number of + hourly checks: waiting forever would burn wall-clock time, so the task + stops with an auditable reason instead. + """ + data = _load_json(Path(exp) / "stop_requested.json") + return data or None + + +def _variant_policy(cfg: ExperimentConfig) -> str: + """``freeze`` (default): collect candidates, promote once at the end. + + Freezing keeps the reference implementation identical across rounds, so + every round compares under the same conditions. ``per_round`` restores the + old behaviour (write the variant as soon as a round beats it). + + Under the variant-round protocol this knob is bypassed on purpose: the + baseline round's faster kernels go into the variant pool immediately (the + operator's rule — a kernel result is a fact, while the *harness* is what + waits for approval). + """ + raw = str(cfg.answers.get("variant_policy") or "freeze").strip().lower() + return "per_round" if raw in {"per_round", "always", "each", "immediate"} \ + else "freeze" + + +def _seed_variant_entry(operators: Dict[str, Any], iid: str, + entry: Dict[str, Any]) -> None: + """Mirror one pool ``best_known`` into the experiment's variant table.""" + operators[iid] = { + "median_us": entry.get("best_known_us"), + "p90_us": None, + "correctness_ok": True, + "kernel_source": "pool:best_known", + "harness_version": None, + "source_iteration": None, + "family": entry.get("family") or "", + "contract": entry.get("contract") or {}, + "conditions": {"origin": "variant_pool_refresh"}, + "recorded_at": None, + } + + +def _reload_variant_baseline(cfg: ExperimentConfig, + variant_pool: Dict[str, Any], + variant_table: Dict[str, Any]) -> List[str]: + """Pull externally-updated ``best_known_us`` back into this run's view. + + The baseline round writes the kernels it beat the variant with straight + into the question pool. Every later round must therefore compare against — + and DKAO must warm-start from — that *new* number, not the one this process + read at startup. Returns the operators whose reference moved, so the gates + can say out loud that the bar was raised. + """ + path = _question_pool_path(cfg) + if path is None: + return [] + try: + fresh = pool_instances(path) + except (OSError, ValueError): + return [] + moved: List[str] = [] + operators = variant_table.setdefault("operators", {}) + for iid, entry in fresh.items(): + variant_pool[iid] = entry + best = entry.get("best_known_us") + recorded = (operators.get(iid) or {}).get("median_us") + try: + better = best is not None and (recorded is None + or float(best) < float(recorded)) + except (TypeError, ValueError): + better = False + if not better: + continue + moved.append(str(iid)) + _seed_variant_entry(operators, str(iid), entry) + if moved: + save_variant_table(cfg.exp_dir, variant_table) + append_timeline(cfg.state_dir, "variant_reference_refreshed", { + "operators": sorted(moved)}) + return sorted(moved) + + +def _baseline_stop(cfg: ExperimentConfig, exp: Path, *, iteration: int, + verdict: Dict[str, Any]) -> None: + """Round 1 could not measure the paper: the run stops right there. + + Round 1 judges nothing, so it has no gate to fail — but it owes one usable + measurement per operator: those kernels seed the pool every later round has + to beat, and the paper is the run's exam. If a child crashed, was blocked by + DKAO's own measurement gate or reported nothing usable, there is no honest + way to continue, so the task ends with an auditable reason instead of + inventing evidence. + """ + reason = "; ".join(verdict.get("reasons") or []) or "baseline round failed" + save_json(Path(exp) / "stop_requested.json", { + "iteration": iteration, "reason": "baseline_round_failed", + "detail": {"failed": verdict.get("failed"), + "measured": verdict.get("measured"), + "message": reason}, + }) + append_timeline(cfg.state_dir, "baseline_round_failed", { + "iteration": iteration, "failed": verdict.get("failed"), + "measured": verdict.get("measured"), "reason": reason, + }) + _record_state(cfg, FINISHED, iteration, event="baseline_round_failed", + failed=list(verdict.get("failed") or [])) + set_run(cfg.state_dir, final_status="baseline_round_failed") + + + +def _track_best_known(cfg: ExperimentConfig, exp: Path, iteration: int, + results: Dict[str, Any]) -> Dict[str, Any]: + """Per-shape best-known curve + explicit regression detection. + + Before this, a round whose kernel was slower than the known best only + silently failed to promote. Recording the curve makes "this round got + worse starting point" visible in the record and the UI. + """ + path = exp / "best_known.json" + data = _load_json(path) or {} + regressions: List[Dict[str, Any]] = [] + tolerance = 1.02 + for iid, row in (results or {}).items(): + if not isinstance(row, dict): + continue + median = row.get("median_us") + entry = data.setdefault(iid, {"best_median_us": None, + "best_iteration": None, "history": []}) + entry.setdefault("history", []).append({ + "iteration": iteration, "median_us": median, + "passed": row.get("passed"), + }) + entry["history"] = entry["history"][-40:] + try: + value = float(median) + except (TypeError, ValueError): + continue + if value <= 0: + continue + best = entry.get("best_median_us") + if best is None or value < float(best): + entry["best_median_us"] = value + entry["best_iteration"] = iteration + elif value > float(best) * tolerance: + regressions.append({ + "shape": iid, "iteration": iteration, "median_us": value, + "best_median_us": best, "best_iteration": entry.get("best_iteration"), + "delta_percent": round((value - float(best)) / float(best) * 100, 2), + }) + save_json(path, data) + if regressions: + with (exp / "regressions.jsonl").open("a", encoding="utf-8") as fh: + for row in regressions: + fh.write(json.dumps({"ts": time.time(), **row}, + ensure_ascii=False) + "\n") + append_timeline(cfg.state_dir, "kernel_regression", { + "iteration": iteration, "regressions": regressions, + }) + return {"regressions": regressions} + + +def _finalize_promotion(cfg: ExperimentConfig, exp: Path) -> Optional[Dict[str, Any]]: + """Promote the best kernel per shape once, after a frozen experiment.""" + if not _truthy(cfg.answers.get("promote_kernels"), default=True): + return None + candidates_path = exp / "promotion_candidates.jsonl" + if not candidates_path.is_file(): + return None + best: Dict[str, Dict[str, Any]] = {} + for line in candidates_path.read_text(encoding="utf-8", + errors="replace").splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if not isinstance(row, dict) or row.get("ahe_passed") is not True: + continue + if row.get("action") not in {"would_add", "would_update", "added", + "updated"}: + continue + current = best.get(str(row.get("shape"))) + if current is None or (row.get("ahe_median_us") or float("inf")) < ( + current.get("ahe_median_us") or float("inf")): + best[str(row.get("shape"))] = row + if not best: + payload = {"enabled": True, "frozen": True, "promoted": [], + "results": [], "note": "no eligible candidate"} + save_json(exp / "final_promotion.json", payload) + return payload + try: + from metainfer.tasks.dcu_kernel_auto_opt.orchestrator.variant_promote import ( # noqa: E501 + promote_variant, + ) + except Exception as exc: # noqa: BLE001 + return {"enabled": True, "error": repr(exc), "results": []} + threshold = float(cfg.answers.get("promote_min_improvement_percent") or 3.0) + outcomes: List[Dict[str, Any]] = [] + for shape_id, row in sorted(best.items()): + workspace = Path(str(row.get("workspace") or "")) + if not workspace.is_dir(): + outcomes.append({"shape": shape_id, "action": "skipped", + "reason": "candidate workspace is gone"}) + continue + outcome = promote_variant( + workspace_dir=workspace, + answers=row.get("answers") or {}, + shape_id=shape_id, + source_task=str(row.get("source_task") or ""), + correctness_ok=True, + min_improvement_percent=threshold, + tp=row.get("tp"), m=row.get("m"), + model_label=row.get("model_label"), + ) + outcome["ahe_median_us"] = row.get("ahe_median_us") + outcome["candidate_iteration"] = row.get("iteration") + outcomes.append(outcome) + promoted = [o["shape"] for o in outcomes + if o.get("action") in {"added", "updated"}] + payload = { + "enabled": True, "frozen": True, "min_improvement_percent": threshold, + "promoted": promoted, "results": outcomes, + "candidates_considered": sorted(best), + } + save_json(exp / "final_promotion.json", payload) + with (exp / "promotion_log.jsonl").open("a", encoding="utf-8") as fh: + for outcome in outcomes: + fh.write(json.dumps({"ts": time.time(), "iteration": -1, + "task_id": cfg.task_id, "final": True, + **outcome}, ensure_ascii=False) + "\n") + append_timeline(cfg.state_dir, "final_kernel_promotion", { + "promoted": promoted, "candidates": sorted(best), + }) + return payload + + +def _write_round_review(cfg: ExperimentConfig, exp: Path) -> Dict[str, Any]: + """Cross-round review: every generation, medians, champion and regressions.""" + from .champion_selection import best_generation, load_generations, summarize + + generations = load_generations(exp) + rows = summarize(generations) + champion = best_generation(generations) + review = { + "task_id": cfg.task_id, + "generations": rows, + "champion_iteration": champion.iteration if champion else None, + "champion_verdict": champion.verdict if champion else None, + "final_promotion": _load_json(exp / "final_promotion.json"), + "best_known": _load_json(exp / "best_known.json"), + "variant_policy": _variant_policy(cfg), + "pinned_baseline": _load_json(exp / "champion_override.json"), + } + save_json(exp / "round_review.json", review) + lines = ["# Cross-round review (AHE)", "", + f"- variant policy: **{review['variant_policy']}**", + f"- champion generation: **iteration {review['champion_iteration']}**" + f" ({review['champion_verdict']})", ""] + if rows: + shapes = sorted({iid for row in rows for iid in row.get("medians", {})}) + header = "| gen | verdict | pass | " + " | ".join(shapes) + " |" + lines += [header, "|" + "---|" * (3 + len(shapes))] + for row in rows: + medians = row.get("medians") or {} + cells = [] + for shape in shapes: + value = medians.get(shape) + cells.append(f"{value:.1f}" if isinstance(value, (int, float)) + else "-") + lines.append( + f"| {row['iteration']}{' ⭐' if row.get('champion') else ''} " + f"| {row.get('verdict')} | {row.get('pass_count')} | " + + " | ".join(cells) + " |") + lines.append("") + best_known = review["best_known"] or {} + if best_known: + lines.append("## Best known per shape") + for shape, entry in sorted(best_known.items()): + lines.append( + f"- `{shape}`: {entry.get('best_median_us')} us " + f"(iteration {entry.get('best_iteration')})") + lines.append("") + promotion = review["final_promotion"] or {} + if promotion: + lines.append("## Final promotion") + for outcome in promotion.get("results") or []: + lines.append( + f"- `{outcome.get('shape')}`: {outcome.get('action')} " + f"({outcome.get('reason') or outcome.get('improvement_percent') or ''})") + (exp / "round_review.md").write_text("\n".join(lines) + "\n", + encoding="utf-8") + append_timeline(cfg.state_dir, "round_review_written", { + "champion_iteration": review["champion_iteration"], + "generations": len(rows), + }) + return review + + +def _code_reload_pending(cfg: ExperimentConfig, + loaded: str) -> Optional[Dict[str, Any]]: + """Whether the harness code on disk changed since this process started. + + A round runs for hours, so an operator who fixes the loop (a new GPU gate, + a new retry policy) cannot expect the live process to honour it — the + modules are already imported. Rather than running one more round of stale + logic, the loop stops here: the recorded revision marker makes the restart + idempotent (a fresh process records its own revision and carries on), and + the supervisor/server resumes at the next iteration with the new code. + """ + current = harness_revision() + if not loaded or current == loaded: + return None + return { + "loaded_revision": loaded, + "on_disk_revision": current, + "resumes_at_iteration": None, # filled by the caller + } + + +def _rounds_used_on_disk(exp: Path) -> float: + """How many rounds this experiment has already spent. + + ``rounds_used`` lives in ``run.json`` (the WebUI reads it). When it is + missing — a run created before the field — it is derived from the round + artifacts, so a resumed run does not silently start with a fresh budget. + """ + stored = _load_json(Path(exp) / "run.json") or {} + raw = stored.get("rounds_used") + try: + if raw is not None: + return max(0.0, float(raw)) + except (TypeError, ValueError): + pass + used = 0.0 + for stage_file in (Path(exp) / "runs").glob("iteration_*/input/stage.json"): + data = _load_json(stage_file) or {} + try: + used += float(data.get("round_cost") or 0.0) + except (TypeError, ValueError): + continue + return used + + +#: A round is an iteration: the baseline round is round 1, and each later round +#: is one performance pass plus (when it wins) the generalization retake inside +#: the same iteration, so a retake never consumes a round of its own. +ROUNDS_DEFAULT = 10 +ROUNDS_MAX = 50 + +#: Cost guardrail, deliberately far above any real run: a judged pass costs 1.0 +#: and a retake 0.5, so ROUNDS_MAX rounds of steady retakes still cost well under +#: this. It exists only so a pathological loop cannot spend forever. +COST_GUARDRAIL = 100.0 + + +def _rounds_limit(cfg: ExperimentConfig) -> int: + """How many rounds (iterations) this task may run — the form's "Rounds". + + ``max_iterations`` is still honoured for runs whose requirements were built + before the form field was renamed; the default is ROUNDS_DEFAULT. + """ + for key in ("rounds", "max_iterations"): + raw = cfg.answers.get(key) + if raw in (None, ""): + continue + try: + return max(1, min(ROUNDS_MAX, int(raw))) + except (TypeError, ValueError): + continue + return ROUNDS_DEFAULT + + +def _rounds_used_on_disk(exp: Path) -> float: + """The judged-pass cost this experiment has already spent (1.0 / 0.5). + + Only used by the cost guardrail; the round *limit* is an iteration number. + """ + stored = _load_json(Path(exp) / "run.json") or {} + raw = stored.get("rounds_used") + try: + if raw is not None: + return max(0.0, float(raw)) + except (TypeError, ValueError): + pass + used = 0.0 + for stage_file in (Path(exp) / "runs").glob("iteration_*/input/stage.json"): + data = _load_json(stage_file) or {} + try: + used += float(data.get("round_cost") or 0.0) + except (TypeError, ValueError): + continue + return used + + +def run_experiment(cfg: ExperimentConfig, *, + start_iteration: int = 1, + iterations_to_run: Optional[int] = None, + champion_iteration: Optional[int] = None) -> Path: + """Run (or continue) the AHE outer loop. + + ``start_iteration`` > 1 resumes an existing experiment: the champion and + previous-round state are rebuilt from disk (best_ever + the iteration + artifacts) so a new generation of the harness can be evaluated against + the already-measured champion without re-running earlier rounds. + """ + exp = cfg.exp_dir + (exp / "runs").mkdir(parents=True, exist_ok=True) + cfg.dump_snapshot(exp / "config_snapshot.json") + # Record the revision of the code this process loaded, so a later round can + # tell "the operator changed the harness" from "nothing happened". + loaded_revision = write_harness_revision(cfg.state_dir) + _record_state( + cfg, PREPARE, + event="experiment_start" if start_iteration <= 1 else "experiment_resume", + start_iteration=start_iteration, + max_iterations=cfg.max_iterations, + execution_mode=cfg.execution_mode, + pool_source=cfg.pool_source, + harness_revision=loaded_revision, + ) + + live_workspace = _seed_workspace(cfg) + evaluator = build_evaluator(cfg.execution_mode) + evolver = build_evolver( + str(cfg.answers.get("evolve_mode") or "dry-run"), + model=str(cfg.answers.get("evolve_model") or DEFAULT_EVOLVE_MODEL), + timeout_seconds=float(cfg.answers.get("evolve_timeout_seconds") or 3600), + ) + + pool = _load_pool(cfg) + # FLOW.md §1.1: a round is sized in whole waves of 4 questions. The + # configured count is already normalized on load (config.py); this is the + # second call site, where "no cap" means "the whole pool" and the pool size + # itself is the only remaining source of a non-multiple count. + budget = cfg.per_round_budget or (len(pool.all_ids()) if pool else 0) + if budget > 0: + budget = normalized_per_gate(budget) + purposes: Dict[str, str] = {} + selection_notes: List[str] = [] + prev_selected: List[str] = _load_ids(exp / LAST_SCORED_FILE) + + prev_results: Optional[Dict[str, Any]] = None + champion_results: Optional[Dict[str, Any]] = None + pending_confirmation = False + best_ever: Dict[str, Any] = _load_json(exp / BEST_EVER_FILE) or {} + + if start_iteration > 1: + # Resume: the champion is whichever generation best_ever recorded; its + # measured results and the previous round's set come back from disk. + # ``champion_iteration`` lets the caller pin the comparison baseline + # explicitly (e.g. "compare iteration 3 against iteration 2"). + prev_results = _iteration_results(exp, start_iteration - 1) + # The champion is the best *measured* generation among the ones that + # were not rejected: a generation can hold the best result and still + # not have been promoted in its own round (it needed confirmation, or + # too few questions won). Comparing against "the last promoted + # generation" would let a worse harness take the champion slot. + measured = best_generation(load_generations(exp)) + champ_iter = int(champion_iteration + or (measured.iteration if measured else 0) + or best_ever.get("iteration") + or start_iteration - 1) + if champion_iteration is None and measured is not None and measured.results: + champion_results = dict(measured.results) + best_ever = { + "iteration": measured.iteration, + "pass_rate": None, + "snapshot_dir": measured.snapshot_dir or str( + exp / "runs" / f"iteration_{measured.iteration:03d}" + / "input" / "workspace"), + "workspace_revision": measured.workspace_revision, + "verdict": "BEST_MEASURED", + } + save_json(exp / BEST_EVER_FILE, best_ever) + else: + champion_results = _iteration_results(exp, champ_iter) + if champion_iteration is not None: + snapshot_dir = (exp / "runs" / f"iteration_{champ_iter:03d}" + / "input" / "workspace") + scores = [row for row in _load_jsonl(exp / SCORES_FILE) + if int(row.get("iteration") or 0) == champ_iter] + revision = "" + manifest = _load_json( + exp / "runs" / f"iteration_{champ_iter:03d}" / "evolve" + / "change_manifest.json") + if manifest: + revision = str(manifest.get("workspace_revision") or "") + best_ever = { + "iteration": champ_iter, + "pass_rate": (scores[-1].get("pass_rate") if scores else None), + "snapshot_dir": str(snapshot_dir), + "workspace_revision": revision, + "verdict": "MANUAL_BASELINE", + } + save_json(exp / BEST_EVER_FILE, best_ever) + save_json(exp / "champion_override.json", + {"champion_iteration": champ_iter, + "pinned_by": "resume --champion-iteration"}) + append_timeline(cfg.state_dir, "champion_baseline_pinned", + {"iteration": champ_iter}) + if not prev_selected: + prev_selected = _load_ids(exp / LAST_SCORED_FILE) + set_run(cfg.state_dir, finished=False, final_status=None) + + # Stop condition is re-read every round, so the WebUI can raise or lower + # the target while the loop is running ("run to iteration N"). The hard + # cap only bounds a runaway target. + HARD_ITERATION_CAP = 50 + #: One HE run chases one promotable harness and stops when it gets one. + #: The round budget keeps a run that never succeeds from going forever. + max_rounds = max(1.0, COST_GUARDRAIL) + + # Variant table: what every operator is measured against. Seeded from the + # existing pool (each operator's best known result) and only updated when a + # harness is accepted and promoted, so the reference never drifts. + variant_pool: Dict[str, Any] = {} + variant_table: Dict[str, Any] = {"operators": {}, "groups": {}} + if not _uses_variant_rounds(cfg): + _announce_legacy_protocol(cfg, exp) + if _uses_variant_rounds(cfg): + pool_path = _question_pool_path(cfg) + variant_pool = pool_instances(pool_path) if pool_path else {} + variant_table = load_variant_table(exp, pool_path=pool_path) + per_gate = normalized_per_gate( + cfg.answers.get("questions_per_round") + or cfg.answers.get("per_round_budget") + or OPERATORS_PER_GATE) + append_timeline(cfg.state_dir, "variant_table_ready", { + "operators": len(variant_table.get("operators") or {}), + "pool_operators": len(variant_pool), + "harness_version": variant_table.get("harness_version"), + "questions_per_gate": per_gate, + }) + if not variant_pool: + # Nothing to measure: fail loudly instead of spinning through the + # round budget. This happens when the pool file is missing/empty. + append_timeline(cfg.state_dir, "variant_pool_empty", { + "pool": str(_question_pool_path(cfg) or cfg.pool_source)}) + _record_state(cfg, FINISHED, start_iteration, + event="variant_pool_empty") + set_run(cfg.state_dir, final_status="no_question_pool") + return _final_report(cfg) + judgeable = _judgeable_ids(variant_pool, variant_table) + if len(judgeable) < per_gate: + # A round is a whole wave of 4 questions (FLOW.md §1.1). Drawing + # fewer would silently change what the gate means: with 3 measured + # operators "at least 75% win" becomes "all three must win". Say so + # and stop instead of judging a round the pool cannot fill. + append_timeline(cfg.state_dir, "variant_pool_too_small", { + "pool": str(_question_pool_path(cfg) or cfg.pool_source), + "judgeable": judgeable, + "judgeable_count": len(judgeable), + "questions_per_round": per_gate, + "policy": ("a round is sized in whole waves of 4; add judgeable " + "operators (variant + baseline) to the pool or lower " + "the round size"), + }) + _record_state(cfg, FINISHED, start_iteration, + event="variant_pool_too_small") + set_run(cfg.state_dir, final_status="no_question_pool") + return _final_report(cfg) + # ``iterations_to_run`` is the *current call's* batch (the server resumes one + # round at a time), not the run's bound: the bound is the form's "Rounds". + fallback_end = start_iteration + (iterations_to_run or cfg.max_iterations) - 1 + while_end = max(fallback_end, start_iteration) + HARD_ITERATION_CAP + iteration = start_iteration + #: Round budget. A pass that judges the performance gate costs a full + #: round; the generalization retake (the same harness handed to DKAO again + #: with the frozen paper) costs half a round — the operator calls it + #: "iteration 2.5". ``iteration`` therefore stays an integer (it names the + #: ``runs/iteration_NNN`` directory) while this counter is fractional. + # Reported cost of the judged passes so far (1.0 per performance pass, 0.5 + # per retake). It is bookkeeping for the guardrail below, not the run limit: + # the limit is an iteration number. + rounds_used = _rounds_used_on_disk(exp) + rounds_limit = _rounds_limit(cfg) + #: Round 1 could not measure the paper, so the run has no foundation and + #: already ended: the tail below must not overwrite that verdict. + baseline_stopped = False + while iteration <= while_end: + target = read_target_iterations(exp) + set_run(cfg.state_dir, target_iterations=target, max_rounds=max_rounds, + rounds_used=rounds_used) + # The form's "Rounds" is the limit; the WebUI target can raise or lower + # it for a live run. Either way the unit is a round = an iteration, so + # round 10 means the run may start iteration 10 and no further. + rounds_limit_now = max(1, int(target)) if target is not None else rounds_limit + # The variant protocol counts rounds here; the legacy/pool path keeps its + # historical "requested iterations" bound (its callers pass an explicit + # batch and its rounds are not the same unit). + if not _uses_variant_rounds(cfg) and iteration > fallback_end: + break + if iteration > rounds_limit_now: + append_timeline(cfg.state_dir, "rounds_reached", { + "iteration": iteration - 1, "rounds": rounds_limit_now, + "target": target, + "stage": "variant" if _uses_variant_rounds(cfg) else "legacy"}) + break + if _uses_variant_rounds(cfg) and rounds_used >= max_rounds: + # The cost guardrail tripped: far above any real run, it only stops + # a pathological loop. The run limit is the form's "Rounds". + append_timeline(cfg.state_dir, "round_budget_exhausted", { + "rounds_used": rounds_used, "max_rounds": max_rounds, + "rounds_limit": rounds_limit_now, + "reason": "cost guardrail tripped"}) + set_run(cfg.state_dir, final_status="rounds_exhausted", + rounds_used=rounds_used) + break + # A pending retake means the previous performance stage passed: this + # pass re-measures the frozen paper with the same harness bytes and + # judges the generalization gate. Nothing else happens in it. + pending_retake = (read_pending_generalization(exp) + if _uses_variant_rounds(cfg) else None) + stage = round_stage(iteration=iteration, pending=pending_retake) + #: Set when this pass produced its own gate decision (baseline and the + #: generalization retake do): the frozen-champion decision engine must + #: not run, and the loop steps on by itself. + own_decision = False + hist_index = len(load_generations(exp)) + iter_dir = exp / "runs" / f"iteration_{iteration:03d}" + input_dir = iter_dir / "input" + evolve_dir = iter_dir / "evolve" + + # --- select this round's instances --------------------------------- + # Variant-driven rounds: the operators come from the question pool + # (which already records each operator's current best), and they are + # judged against the variant table instead of a frozen champion. This + # is what makes "one HE run chases one promotable harness" work. + if _uses_variant_rounds(cfg): + # The round draws exactly the configured number of operators (a + # multiple of the 4 HCUs): the count comes from the New Task form + # and is never reduced to fit the cards. Whether a card is free is + # DKAO's business — the child's engine waits on its own admission + # gate — so a round is always the size the operator asked for. + fit = {"requested": per_gate, "count": per_gate, + "free_devices": None, "reduced": False} + draw_count = fit["count"] or per_gate + reference_moved: List[str] = [] + if stage == STAGE_PERFORMANCE: + # The baseline round may have raised the bar (its kernels go + # straight into the pool); every later round has to see it. + reference_moved = _reload_variant_baseline( + cfg, variant_pool, variant_table) + round_questions = select_round_questions( + iteration=iteration, table=variant_table, + instances=variant_pool, state={"groups": group_state(variant_table)}, + count=draw_count, rng=_round_rng(cfg, iteration, stage)[0], + pending=pending_retake) + selected = list(round_questions["performance_ids"]) + purposes = {iid: "performance" for iid in selected} + for iid in round_questions.get("generalization_ids") or []: + purposes.setdefault(iid, "generalization") + if stage == STAGE_GENERALIZATION: + purposes = {iid: "generalization" for iid in selected} + selection_notes = [ + { + STAGE_BASELINE: ("baseline round: measures the new run's " + "generalization paper; judges no gate"), + STAGE_PERFORMANCE: ("performance gate: fresh random " + "operators, judged against each " + "operator's variant"), + STAGE_GENERALIZATION: ("generalization retake: the same " + "harness handed to DKAO again with " + "the frozen paper"), + }.get(stage, "variant-driven round"), + f"questions={len(selected)} (requested {fit['requested']})"] + if reference_moved: + selection_notes.append( + "variant reference refreshed from the baseline round's " + "kernels: " + ", ".join(reference_moved)) + # Fresh pool data also decides what DKAO warm-starts from, so the + # retake and the performance stage always launch children against + # the current best-known kernel. + inst_budget = int(cfg.answers.get("default_rounds") or 5) + instances = [ + _variant_instance_spec({"id": i, **dict(variant_pool[i])}, + budget_rounds=inst_budget) + for i in selected if i in variant_pool] + save_json(input_dir / "benchmark" / "scored_ids.json", selected) + save_json(input_dir / "benchmark" / "purposes.json", { + "iteration": iteration, "stage": stage, + "purposes": purposes, + "families": [], "rationale": "variant-driven selection", + "notes": selection_notes, + "generalization_ids": round_questions.get("generalization_ids"), + "defines_paper": round_questions.get("defines_paper"), + }) + save_json(exp / ROUND_QUESTIONS_FILE, { + "iteration": iteration, "stage": stage, + "performance_ids": selected, + "generalization_ids": round_questions.get("generalization_ids"), + "defines_paper": round_questions.get("defines_paper"), + }) + save_json(input_dir / "stage.json", { + "iteration": iteration, "stage": stage, + "round_cost": { + # The baseline round judges nothing, so it costs nothing: + # the budget is spent on judged passes. + STAGE_BASELINE: 0.0, + STAGE_GENERALIZATION: GENERALIZATION_ROUND_COST, + }.get(stage, PERFORMANCE_ROUND_COST), + "rounds_used_before": rounds_used, + "retake_of_iteration": (pending_retake or {}).get("iteration"), + }) + prev_selected = list(selected) + if round_questions.get("defines_paper"): + # Persist the run's fixed generalization paper immediately: it + # is the exam for the rest of the run, so it must survive even + # if the baseline round is the last thing that happens. + variant_table["groups"] = record_paper( + variant_table, iteration=iteration, + generalization_ids=round_questions["generalization_ids"], + performance_ids=selected) + save_variant_table(exp, variant_table) + append_timeline(cfg.state_dir, "variant_round_selection", { + "iteration": iteration, "stage": stage, + "performance_ids": selected, + "generalization_ids": round_questions.get("generalization_ids"), + "defines_paper": round_questions.get("defines_paper"), + # Which draw this was: the seed is what makes "why these four + # operators?" reproducible after the fact. + "seed": _round_rng(cfg, iteration, stage)[1], + "count": draw_count, + "judgeable_in_pool": len(_judgeable_ids(variant_pool, variant_table)), + "rounds_used": rounds_used, + }) + elif pool is not None: + plan = None + if iteration > 1: + prev_dir = iter_dir.parent / f"iteration_{iteration - 1:03d}" + plan = load_round_plan(prev_dir / "evolve" / "round_plan.json") + # Role-based selection: regression anchors + repair + a + # cross-family generalisation probe, with family quota, overlap + # and least-recently-used rotation. The evolve agent's picks are + # honoured where they fit a role. + rotation = load_rotation(exp) + selection = select_questions( + pool=pool, + budget=budget or len(pool.all_ids()), + prev_selected=prev_selected, + prev_results=prev_results, + plan=plan, + rotation=rotation, + ) + selected = selection.selected + purposes = dict(selection.purposes) + selection_notes = list(selection.notes) + inst_budget = int(cfg.answers.get("default_rounds") or 5) + instances = [ + _pool_instance_spec(pool.get(i), budget_rounds=inst_budget) + for i in selected + ] + save_json(input_dir / "benchmark" / "scored_ids.json", selected) + save_json(input_dir / "benchmark" / "purposes.json", { + "iteration": iteration, + "purposes": purposes, + "families": selection.families, + "rationale": selection.rationale, + "notes": selection_notes, + }) + save_json(exp / LAST_SCORED_FILE, selected) + mark_selected(exp, selected, iteration) + append_timeline(cfg.state_dir, "question_selection", { + "iteration": iteration, + "selected": selected, + "purposes": purposes, + "families": selection.families, + "notes": selection_notes, + }) + prev_selected = list(selected) + else: + instances = list(cfg.suite) + + # --- safety net: a malformed evolved harness rolls back immediately --- + malformed = _harness_validate_error(live_workspace) + if malformed and iteration > 1: + best_dir = Path(str(best_ever.get("snapshot_dir", ""))) + if best_dir.is_dir(): + replace_tree(best_dir, live_workspace) + best_rev = best_ever.get("workspace_revision") + if best_rev: + subprocess.run( + ["git", "reset", "--hard", str(best_rev)], + cwd=live_workspace, check=True, + stdout=subprocess.DEVNULL, + ) + save_json(iter_dir / "rollback.json", { + "iteration": iteration, + "verdict": "REJECT", + "action": "RESTORE_CHAMPION", + "reason": f"harness malformed: {malformed}", + "restored_from": str(best_dir), + }) + save_json(input_dir / "decision.json", { + "iteration": iteration, + "verdict": "REJECT", + "action": "RESTORE_CHAMPION", + "reason": f"harness malformed: {malformed}", + }) + # end the experiment early: champion restored and recorded + break + + # --- snapshot + evaluate the live workspace ------------------------ + snapshot = input_dir / "workspace" + copy_tree(live_workspace, snapshot) + _record_state(cfg, EVALUATE, iteration, event="iteration_start", + stage=stage, + selected=list(selected) if pool is not None else + [i.id for i in instances]) + + # The evaluator hands every question to its own DKAO child. Which stage + # this is decides only where those children live on disk — the retake + # gets its own directory so it is a fresh DKAO job, not a continuation + # of the performance stage's optimization. DKAO itself is untouched. + cfg.answers["round_stage"] = stage + results = evaluator.evaluate(cfg, snapshot, iteration, instances=instances) + stop = _stop_requested(exp) + if stage == STAGE_BASELINE and _uses_variant_rounds(cfg): + # The baseline round is the run's foundation: it judges no gate, but + # it owes one usable measurement per operator (those kernels set the + # bar every later round has to clear, and the paper is the run's + # exam). Nothing — a crashed child, DKAO's own measurement gate + # refusing the cards, a missing report — may be papered over. + verdict = baseline_round_verdict(results, selected) + if stop or verdict["status"] != "PASS": + reason = (str(stop.get("reason")) if stop + else "; ".join(verdict.get("reasons") or [])) + append_timeline(cfg.state_dir, "round_failed", { + "iteration": iteration, "stage": stage, + "reason": reason or "baseline round failed", + "failed": verdict.get("failed"), + }) + save_json(input_dir / "benchmark" / "results.json", { + "iteration": iteration, "results": results, + "round_failed": reason or "baseline round failed", + }) + _record_state(cfg, ANALYZE, iteration, + event="baseline_round_failed") + _baseline_stop(cfg, exp, iteration=iteration, verdict=verdict) + baseline_stopped = True + break + if stop: + append_timeline(cfg.state_dir, "round_failed", { + "iteration": iteration, "stage": stage, + "reason": str(stop.get("reason") or "stop requested"), + "detail": {k: v for k, v in stop.items() if k != "iteration"}, + }) + try: + (exp / "stop_requested.json").unlink() + except OSError: + pass + save_json(input_dir / "benchmark" / "results.json", { + "iteration": iteration, "results": results, + "round_failed": str(stop.get("reason") or "stop requested"), + }) + if _uses_variant_rounds(cfg): + if stage == STAGE_GENERALIZATION: + # The retake is spent: a failed retake is a failed + # candidate, so the frozen request is cleared and the next + # pass is a normal round again. Its budget was already spent + # when it was launched. + clear_pending_generalization(exp) + else: + rounds_used += PERFORMANCE_ROUND_COST + _record_state(cfg, ANALYZE, iteration, + event="round_failed_retry", + rounds_used=rounds_used) + iteration += 1 + continue + _record_state(cfg, FINISHED, iteration, event="stop_requested", + **{k: v for k, v in stop.items() if k != "iteration"}) + set_run(cfg.state_dir, + final_status=str(stop.get("reason") or "stopped")) + break + + # --- variant stages: baseline, then performance, then generalization - + # Every comparison is against each operator's *current variant* (the + # best kernel the variant pool holds). Which gates run depends on the + # stage: + # baseline — judges nothing; its faster kernels go into the + # variant pool so later rounds have a real bar; + # performance — fresh random operators, performance gate only; + # generalization — the *same* harness handed to DKAO a second time + # with the frozen paper, generalization gate only. + # Only a generalization pass that wins parks the candidate for a human. + if _uses_variant_rounds(cfg): + perf_ids = list(selected) + perf_gate: Optional[Dict[str, Any]] = None + general_gate: Optional[Dict[str, Any]] = None + mechanism: Dict[str, Any] = {} + baseline_promotion: Optional[Dict[str, Any]] = None + reference_moved: List[str] = [] + + if stage == STAGE_BASELINE: + # The baseline round owes evidence, not a verdict. + baseline_verdict = baseline_round_verdict(results, perf_ids) + if baseline_verdict["status"] != "PASS": + _baseline_stop(cfg, exp, iteration=iteration, + verdict=baseline_verdict) + _record_state(cfg, ANALYZE, iteration, + event="baseline_round_failed") + break + # Faster kernels are a kernel fact, not a harness verdict: they + # enter the variant pool immediately, so every later gate — and + # DKAO's own warm start — compares against what this round + # achieved. Nothing about the *harness* moves here. + baseline_promotion = _promote_baseline_kernels( + cfg, exp, variant_pool, results) + record_variants( + exp, variant_table, results, + version=str(variant_table.get("harness_version") or "seed"), + iteration=iteration, + groups=variant_table.get("groups")) + reference_moved = sorted( + f"{u['operator_id']} ({u.get('previous_best_us')} -> " + f"{u.get('median_us')} us)" + for u in ((baseline_promotion or {}).get("updated") or [])) + save_json(input_dir / "baseline_round.json", { + "iteration": iteration, "stage": stage, + "verdict": baseline_verdict, + "pool_promotion": baseline_promotion, + "reference_moved": reference_moved, + }) + append_timeline(cfg.state_dir, "baseline_round_measured", { + "iteration": iteration, + "operators": perf_ids, + "pool_promoted": reference_moved, + "paper": perf_ids, + }) + + if stage != STAGE_BASELINE: + # One gate per stage. The performance stage judges the fresh + # draw; the generalization stage judges the paper it was + # launched to re-measure. + kind = ("generalization" if stage == STAGE_GENERALIZATION + else "performance") + gate = evaluate_gate(kind=kind, operator_ids=perf_ids, + results=results, table=variant_table, + instances=variant_pool) + gate["iteration"] = iteration + gate["stage"] = stage + if kind == "performance": + perf_gate = gate + save_json(input_dir / "performance_gate.json", gate) + # The round's own results, where every reader of this run + # expects them (the WebUI, resume, and the round review). + save_json(input_dir / "benchmark" / "results.json", + {"iteration": iteration, "stage": stage, + "results": results}) + else: + general_gate = gate + save_json(input_dir / "generalization_gate.json", gate) + save_json(input_dir / "benchmark" + / "generalization_results.json", + {"iteration": iteration, "results": results}) + save_json(input_dir / "benchmark" / "results.json", + {"iteration": iteration, "stage": stage, + "results": results}) + append_timeline(cfg.state_dir, f"{kind}_gate", { + "iteration": iteration, "stage": stage, + "status": gate["status"], "wins": gate["wins"], + "wins_needed": gate["wins_needed"], + "below_floor": gate["below_floor"], + "unmeasured": gate["unmeasured"], + "operators": gate["operators"], + **({"paper": perf_ids} + if kind == "generalization" else {}), + }) + + # Mechanism gate (observability, lenient): a harness change must say + # *why* it should help and produce evidence for it. A contradicted + # mechanism does not veto an accepted win — the counter gates do + # that — but it is recorded, so "we promoted it and cannot explain + # it" is visible instead of silent. + if stage != STAGE_BASELINE: + mechanism = _mechanism_gate_for_round(cfg, exp, iteration) + + if stage != STAGE_BASELINE and gate.get("status") == "INCOMPLETE": + # The round drew questions but did not measure all of them, and + # the missing ones still leave the outcome open. There is no + # verdict to reach: judging here would blame the harness for a + # number the machine never produced — and on the retake it would + # spend the candidate on it. Record it as an incomplete round and + # try again; a frozen retake stays frozen, because the paper is + # still owed a judgement. + append_timeline(cfg.state_dir, "round_incomplete", { + "iteration": iteration, + "stage": stage, + "gate": ("generalization" if stage == STAGE_GENERALIZATION + else "performance"), + "unmeasured": gate["unmeasured"], + "measured": sorted(set(results) - set(gate["unmeasured"])), + "reasons": gate["reasons"], + "rounds_used": rounds_used, + "policy": ("no harness verdict this round; the missing " + "questions are measured in a later pass"), + }) + if stage != STAGE_GENERALIZATION: + # A fresh round was spent, verdict or not. + rounds_used += PERFORMANCE_ROUND_COST + restore_promoted_harness(cfg, exp, live_workspace) + _record_state(cfg, ANALYZE, iteration, + event="round_incomplete", + stage=stage, + unmeasured=gate["unmeasured"], + rounds_used=rounds_used) + iteration += 1 + continue + + both_passed = bool( + stage == STAGE_GENERALIZATION and general_gate + and general_gate["status"] == "PASS") + if stage == STAGE_BASELINE: + verdict, action = "BASELINE_MEASURED", "ITERATE" + reason = ("baseline round: the paper was measured and the " + "faster kernels entered the variant pool; no gate is " + "judged this round") + elif both_passed: + verdict, action = "PROMOTE_CANDIDATE", "AWAIT_APPROVAL" + reason = ("performance and generalization gates passed; " + "waiting for human approval before publishing the " + "harness to DKAO") + elif stage == STAGE_GENERALIZATION: + verdict, action = "REJECT", "RESTORE_CHAMPION" + reason = ("generalization gate " + f"{(general_gate or {}).get('status')}") + else: + verdict, action = "REJECT", "RESTORE_CHAMPION" + reason = f"performance gate {(perf_gate or {}).get('status')}" + decision = { + "iteration": iteration, + "stage": stage, + "round_cost": { + STAGE_BASELINE: 0.0, + STAGE_GENERALIZATION: GENERALIZATION_ROUND_COST, + }.get(stage, PERFORMANCE_ROUND_COST), + "verdict": verdict, + "action": action, + "reason": reason, + "performance_gate": perf_gate, + "generalization_gate": general_gate, + "mechanism_gate": mechanism, + "baseline_promotion": baseline_promotion, + "variant_reference": { + iid: {"median_us": row.get("median_us"), + "source": row.get("variant_source")} + for iid, row in ((perf_gate or general_gate or {}) + .get("comparison", {}) + .get("per_instance") or {}).items() + }, + } + save_json(input_dir / "decision.json", decision) + # The baseline round answers to no gate, so the legacy decision + # engine (which compares against a frozen champion) has nothing to + # say about it and must not overwrite this verdict. + own_decision = stage == STAGE_BASELINE + score_row = pass_rate(results) + with (exp / SCORES_FILE).open("a", encoding="utf-8") as fh: + fh.write(json.dumps({ + **score_row, "iteration": iteration, "stage": stage, + "rounds_used": rounds_used, + "decision_pass_rate": score_row.get("pass_rate"), + "ts": time.time(), + }, ensure_ascii=False) + "\n") + _record_state(cfg, ANALYZE, iteration, event="iteration_decision", + stage=stage, + verdict=decision["verdict"], + action=decision["action"], + rounds_used=rounds_used, + performance_gate=(perf_gate or {}).get("status"), + generalization_gate=(general_gate or {}).get("status"), + mechanism_gate=mechanism.get("status")) + if both_passed: + # The harness that just proved itself is the one parked for + # approval: the revision frozen when the retake was scheduled, + # and the round number that *won* the performance gate — the + # retake is half a round, not a round of its own. + write_pending_promotion( + exp, iteration=int(pending_retake.get("iteration") + or iteration), + revision=(pending_retake.get("harness_revision") + or _head_revision(live_workspace)), + snapshot_dir=str(snapshot), + performance_gate=perf_gate or general_gate or {}, + generalization_gate=general_gate, + results=results) + set_run(cfg.state_dir, final_status="awaiting_approval", + rounds_used=rounds_used) + append_timeline(cfg.state_dir, "awaiting_approval", { + "iteration": iteration, + "performance_wins": (pending_retake.get("performance_wins") + or []), + "generalization_wins": general_gate["wins"], + }) + return _final_report(cfg) + if stage == STAGE_PERFORMANCE: + # The performance gate passed: do NOT re-measure the paper in + # this pass. Freeze the retake — same harness bytes, the paper's + # operators — and let the next pass hand that job to DKAO. + if (perf_gate or {}).get("status") == "PASS": + pending = write_pending_generalization( + exp, iteration=iteration, + harness_revision=_head_revision(live_workspace), + snapshot_dir=str(snapshot), + operator_ids=round_questions.get("generalization_ids") + or perf_ids, + performance_gate=perf_gate or {}) + # Both halves are spent the moment the retake is launched: + # the empty half is a launch in which the whole frozen + # document is retested. Accounting for it here (rather than + # when the retake is judged) also keeps the budget honest on + # the run's last pass, which ends inside the retake. + rounds_used += (PERFORMANCE_ROUND_COST + + GENERALIZATION_ROUND_COST) + append_timeline(cfg.state_dir, "generalization_retake_frozen", { + "iteration": iteration, + "operators": pending["operator_ids"], + "harness_revision": pending["harness_revision"], + "rounds_used": rounds_used, + }) + # This pass is spent: the retake owns the next one. It runs + # as its own stage, with its own DKAO launch of the frozen + # paper on the very same harness bytes. + iteration += 1 + continue + rounds_used += PERFORMANCE_ROUND_COST + restore_promoted_harness(cfg, exp, live_workspace) + iteration += 1 + continue + if stage == STAGE_GENERALIZATION: + # The retake did not win: the candidate is spent, so the frozen + # request is cleared and the next pass is a normal round again. + # Its budget was already spent when it was launched. + clear_pending_generalization(exp) + restore_promoted_harness(cfg, exp, live_workspace) + iteration += 1 + continue + # The baseline round judges nothing and keeps the iteration: the + # harness it just measured still has to evolve before the next + # round can test it. + iteration += 1 + continue + # Pool mode owns the pass criterion: turn real/dry measurements into an + # automatic family-baseline pass decision (no user threshold required). + if pool is not None: + for iid, result in results.items(): + median = result.get("median_us") + if iid in pool.instances and isinstance(median, (int, float)): + auto = pool.auto_pass( + iid, float(median), + correctness_ok=bool(result.get("correctness_ok")), + p90_ok=bool(result.get("p90_ok")), + ) + result["raw_evaluator_passed"] = result.get("passed") + result["passed"] = auto["passed"] + result["automatic_pass"] = auto + save_json(input_dir / "benchmark" / "results.json", { + "iteration": iteration, + "results": results, + }) + stats = pass_rate(results) + + # overlap decision stats (pool mode keeps decisions comparable) + overlap_stats = None + if prev_results is not None: + overlap = { + k: v for k, v in results.items() if k in prev_results + } + if overlap: + overlap_stats = pass_rate(overlap) + + decision_rate = ( + float(overlap_stats["pass_rate"]) + if pool is not None and overlap_stats is not None + else float(stats["pass_rate"]) + ) + _save_scores(cfg, { + "iteration": iteration, + **stats, + "overlap_pass_rate": ( + overlap_stats["pass_rate"] if overlap_stats is not None else None + ), + "decision_pass_rate": decision_rate, + "ts": time.time(), + }) + + diff = diff_results(prev_results, results) + save_json(input_dir / "diff.json", diff) + _write_overview(input_dir, iteration, results, diff) + + # --- change attribution for the PREVIOUS loop's manifest ----------- + prev_manifest = None + if iteration > 1: + prev_manifest = _load_json( + (iter_dir.parent / f"iteration_{iteration - 1:03d}" + / "evolve" / "change_manifest.json") + ) + if prev_manifest: + save_json( + input_dir / "change_evaluation.json", + evaluate_changes(prev_manifest, diff), + ) + + # --- system-level decision engine ---------------------------------- + # Performance is primary; mechanism evidence explains a win but cannot + # veto a repeated real improvement. Held-out inputs land in a later + # adapter iteration; current v1 writes NOT_RUN when absent. + # Mechanism gate: grade the previous loop's declared changes against + # what the candidate iteration actually did (planner plan ids, the + # harness revision children loaded, gate events). + mechanism_checks_map = None + if prev_manifest is not None: + try: + from .mechanism_checks import ( + collect_candidate_evidence, + evaluate_changes as evaluate_mechanism_checks, + ) + evidence = collect_candidate_evidence( + exp, iteration, + planner_enabled=_truthy( + cfg.answers.get("planner_enabled"), default=True), + candidate_workspace=snapshot, + ) + mechanism_checks_map, mechanism_detail = ( + evaluate_mechanism_checks(prev_manifest, evidence)) + save_json(input_dir / "mechanism_evidence.json", { + "iteration": iteration, + "evidence": evidence, + "checks": mechanism_checks_map, + "detail": mechanism_detail, + }) + except Exception as exc: # noqa: BLE001 - never break the loop + append_timeline(cfg.state_dir, "mechanism_check_failed", + {"iteration": iteration, "error": repr(exc)}) + + wired_violations: List[Dict[str, str]] = [] + try: + previous_workspace = ( + exp / "runs" / f"iteration_{iteration - 1:03d}" + / "input" / "workspace" + ) + if iteration > 1 and previous_workspace.is_dir(): + wired_violations = _wired_scope_violations( + previous_workspace, live_workspace) + if wired_violations: + append_timeline(cfg.state_dir, + "harness_wired_scope_violation", + {"iteration": iteration, + "components": wired_violations}) + except Exception: # noqa: BLE001 - advisory only + wired_violations = [] + + # No pairing is dropped for a shared device any more: HE does not watch + # the cards, and DKAO's admission gate refuses to time a kernel on a + # busy one in the first place (a blocked round produces no measurement + # at all, which the round handles as an environment failure). + comparison_candidate = results + comparison_champion = champion_results + + if not own_decision: + decision = decide( + comparison_champion, + comparison_candidate, + manifest=prev_manifest, + mechanism_checks=mechanism_checks_map, + confirmation_reproduced=pending_confirmation, + purposes=purposes, + policy=DecisionPolicy( + mechanism_policy=str( + cfg.answers.get("mechanism_policy") or "lenient"), + ), + ) + decision.update({ + "iteration": iteration, + "candidate_snapshot": str(snapshot), + "decision_pass_rate": decision_rate, + "wired_scope_violations": wired_violations, + }) + save_json(input_dir / "decision.json", decision) + champion_outcome = _record_champion_generation( + cfg, exp, iteration=iteration, decision=decision, + decision_rate=decision_rate, snapshot=snapshot, + live_workspace=live_workspace, results=results, + current=best_ever) + best_ever = champion_outcome.get("best_ever", best_ever) + _record_state(cfg, ANALYZE, iteration, event="iteration_decision", + verdict=decision["verdict"], action=decision["action"], + decision_pass_rate=decision_rate) + try: + if not _uses_variant_rounds(cfg): + # Legacy path only: promote this round's kernels immediately. + # Under the variant protocol nothing reaches production until a + # harness is accepted and approved, so a rejected round must not + # leave its kernels behind (operator rule: only the accepted + # round's optimizations survive). + _promote_iteration(cfg, exp, iteration, results, instances, + freeze=(_variant_policy(cfg) == "freeze")) + _track_best_known(cfg, exp, iteration, results) + except Exception as exc: # noqa: BLE001 - never break the outer loop + append_timeline(cfg.state_dir, "kernel_promotion_failed", + {"iteration": iteration, "error": repr(exc)}) + + # No circuit breaker: under the variant protocol a round that does not + # beat the current variants is a normal outcome, and the run's budget + # (max_rounds) is what bounds it. Stopping early on "three rejections" + # would kill a run that was still making progress on its own terms. + verdict = str(decision.get("verdict") or "") + + if own_decision: + # The variant-round protocol decides for itself (see the gates + # above): there is no frozen-champion action to take here. The + # baseline round measured the paper and moved the variant pool; a + # passing performance stage re-launches as the generalization + # retake; a failing gate already restored the promoted harness. + iteration += 1 + continue + + action = decision["action"] + if action == "SET_CHAMPION": + best_ever = { + "iteration": iteration, + "pass_rate": decision_rate, + "snapshot_dir": str(snapshot), + "workspace_revision": _head_revision(live_workspace), + "verdict": decision["verdict"], + } + save_json(exp / BEST_EVER_FILE, best_ever) + champion_results = dict(results) + pending_confirmation = False + elif action in {"RESTORE_CHAMPION", "KEEP_CHAMPION_ARCHIVE_CANDIDATE", + "ARCHIVE_SPECIALIZED"}: + if action == "ARCHIVE_SPECIALIZED": + copy_tree(snapshot, exp / "specialized" / f"iteration_{iteration:03d}") + best_dir = Path(str(best_ever.get("snapshot_dir", ""))) + if best_dir.is_dir(): + replace_tree(best_dir, live_workspace) + best_rev = best_ever.get("workspace_revision") + if best_rev: + subprocess.run( + ["git", "reset", "--hard", str(best_rev)], + cwd=live_workspace, check=True, + stdout=subprocess.DEVNULL, + ) + disposition = { + "iteration": iteration, + "verdict": decision["verdict"], + "action": action, + "champion_snapshot": str(best_dir), + "candidate_snapshot": str(snapshot), + "reason": decision["reason"], + } + save_json(iter_dir / "candidate_disposition.json", disposition) + # rollback.json is reserved for true reject/overfit, not no-signal. + if decision["verdict"] in {"REJECT", "REJECT_OVERFIT"}: + save_json(iter_dir / "rollback.json", disposition) + pending_confirmation = False + elif action == "ARCHIVE_AND_CONFIRM": + # Leave the candidate live for one confirmation round; no champion + # switch yet. A reproduced win can become PROMOTE_UNEXPLAINED. + save_json(iter_dir / "candidate_disposition.json", { + "iteration": iteration, + "verdict": decision["verdict"], + "action": action, + "candidate_snapshot": str(snapshot), + "reason": decision["reason"], + }) + pending_confirmation = True + + # --- analyze -> evolve --------------------------------------------- + # A confirmation round must retest the SAME candidate; do not let the + # Evolve Agent mutate it between the first unexplained win and repeat. + final_round = ( + (target is not None and iteration >= int(target)) + or (target is None and iteration >= fallback_end) + ) + if final_round: + # No next round exists to evaluate a new generation: evolving here + # would leave an unverified harness behind. + evolve_out = { + "summary": ( + f"# Final round — iteration {iteration}\n\n" + "Evolve skipped: the target round was reached, so no " + "unverified next generation is produced." + ), + "manifest": { + "schema_version": 1, + "iteration": iteration, + "changes": [], + "verification": {"status": "target_reached_skip_evolve"}, + }, + "round_plan": None, + "changed": False, + } + append_timeline(cfg.state_dir, "evolve_skipped_final_round", { + "iteration": iteration, "target": target, + "fallback_end": fallback_end, + }) + elif action == "ARCHIVE_AND_CONFIRM": + evolve_out = { + "summary": ( + f"# Confirmation scheduled — iteration {iteration}\n\n" + "Candidate kept byte-identical for one repeat evaluation." + ), + "manifest": { + "schema_version": 1, + "iteration": iteration, + "changes": [], + "verification": {"status": "confirmation_pending"}, + }, + "round_plan": None, + "changed": False, + } + else: + evolve_out = evolver.apply( + live_workspace, iteration, + query_context={ + "overview": str(input_dir / "analysis" / "overview.md"), + "results": results, + "diff": diff, + "decision": decision, + "selected": (list(selected) if pool is not None else + [i.id for i in instances]), + "pool_ids": (pool.all_ids() if pool is not None else + [i.id for i in instances]), + }, + ) + _record_state(cfg, EVOLVE, iteration, event="evolve_start", + evolve_mode=cfg.answers.get("evolve_mode")) + evolve_dir.mkdir(parents=True, exist_ok=True) + revision = _commit_workspace(live_workspace, iteration) + evolve_out["manifest"]["workspace_revision"] = revision + save_json(evolve_dir / "change_manifest.json", evolve_out["manifest"]) + (evolve_dir / "evolve_summary.md").write_text( + evolve_out["summary"] + "\n", encoding="utf-8" + ) + if pool is not None: + agent_plan = evolve_out.get("round_plan") + if isinstance(agent_plan, dict): + plan_obj = RoundPlan.from_dict(agent_plan) + # Validate agent output immediately. The next iteration applies + # the overlap/budget guardrail and can auto-fill if needed. + pool.assert_in_pool(plan_obj.selected) + else: + plan_obj = RoundPlan( + iteration=iteration, + strategy="balanced", + rationale="dry-run default: keep same scored set", + selected=list(selected), + ) + save_json(evolve_dir / "round_plan.json", plan_obj.to_dict()) + if evolve_out.get("changed"): + copy_tree(live_workspace, evolve_dir / "workspace") + + prev_results = results + iteration += 1 + + # Round boundary: apply a code change instead of running another round + # of stale logic. The supervisor/server restarts the process, which + # records its own revision and resumes at this iteration. + reload_reason = _code_reload_pending(cfg, loaded_revision) + if reload_reason is not None: + reload_reason["resumes_at_iteration"] = iteration + reload_reason["finished_iteration"] = iteration - 1 + append_timeline(cfg.state_dir, "code_reload_deferred", reload_reason) + _record_state( + cfg, PREPARE, iteration - 1, + event="code_reload_deferred", + loaded_revision=reload_reason["loaded_revision"], + on_disk_revision=reload_reason["on_disk_revision"], + resumes_at_iteration=iteration, + ) + set_run(cfg.state_dir, + final_status=f"code_reload_pending:{iteration}") + return _final_report(cfg) + + if baseline_stopped: + # The task already ended on the baseline round: report and stop without + # touching final_status (which ``_baseline_stop`` owns). + _record_state(cfg, REPORT, event="experiment_report") + return _final_report(cfg) + try: + # Variant-round protocol promotes kernels through the pool (the baseline + # round) and the approved round, never through the frozen-candidate + # sweep: running it here would promote kernels from rejected rounds. + if _variant_policy(cfg) == "freeze" and not _uses_variant_rounds(cfg): + _finalize_promotion(cfg, exp) + except Exception as exc: # noqa: BLE001 - reporting must still happen + append_timeline(cfg.state_dir, "final_promotion_failed", + {"error": repr(exc)}) + try: + _write_round_review(cfg, exp) + except Exception as exc: # noqa: BLE001 + append_timeline(cfg.state_dir, "round_review_failed", + {"error": repr(exc)}) + _record_state(cfg, REPORT, event="experiment_report") + report = _final_report(cfg) + _record_state(cfg, FINISHED, event="experiment_finished") + return report + + +def _final_report(cfg: ExperimentConfig) -> Path: + exp = cfg.exp_dir + lines = [ + "# harness_evolve report (AHE outer loop)", + "", + f"- task: {cfg.task_id}", + f"- iterations: {cfg.max_iterations}", + f"- execution_mode: {cfg.execution_mode}", + f"- suite size: {len(cfg.suite)}", + f"- pool_mode: {cfg.pool_mode}", + ] + best = _load_json(exp / BEST_EVER_FILE) or {} + if best: + lines.append( + f"- best-ever: iteration {best.get('iteration')} " + f"pass_rate {best.get('pass_rate')}" + ) + lines.append("") + lines.append("See runs/iteration_*/ for per-iteration artifacts.") + report = exp / REPORT_FILE + report.write_text("\n".join(lines) + "\n", encoding="utf-8") + return report diff --git a/metainfer/tasks/harness_evolve/orchestrator/plugin.py b/metainfer/tasks/harness_evolve/orchestrator/plugin.py new file mode 100644 index 00000000..094d62bd --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/plugin.py @@ -0,0 +1,11 @@ +"""Task plugin descriptor for harness_evolve.""" + +from metainfer.orchestrator.tasks.base import TaskPlugin + + +PLUGIN = TaskPlugin( + task_type="harness-evolve", + cli_module="metainfer.tasks.harness_evolve.orchestrator.cli", + phases_module="metainfer.tasks.harness_evolve.orchestrator.phases", + diagnostic_globs=("*.json", "*.jsonl", "*.md", "*.log"), +) diff --git a/metainfer/tasks/harness_evolve/orchestrator/pool.py b/metainfer/tasks/harness_evolve/orchestrator/pool.py new file mode 100644 index 00000000..132cbec0 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/pool.py @@ -0,0 +1,241 @@ +"""Registered-instance pool + automatic pass criteria (suite protocol v2). + +A "question" can only come from the pool: instances the user registered with a +frozen Triton baseline and a valid contract (they are the operators actually +integrated & measured, e.g. DeepSeek INT8 W8A8 GEMM family). AHE decides pass +automatically (``tau_family`` derived from each family's accepted history) and +picks per-round question sets via round_plan + machine guardrails (see +``docs/dkao_suite_protocol_v2.md``). +""" + +from __future__ import annotations + +import json +import statistics +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional, Set + +DEFAULT_TAU = 0.6 # cold-start default: must beat baseline by >= 1/DEFAULT_TAU? see auto_pass + + +def family_of(M: int, operator: str) -> str: + regime = "decode" if M <= 32 else "prefill" + return f"{regime}__{operator}" + + +@dataclass +class PoolInstance: + id: str + model: str + tp_size: int + operator: str + M: int + N: int + K: int + baseline_us: float + family: str = "" + best_known_us: Optional[float] = None + history: List[Dict[str, Any]] = field(default_factory=list) + last_selected_round: Optional[int] = None + + @property + def key(self) -> str: + return self.id + + def to_dict(self) -> Dict[str, Any]: + return { + "id": self.id, + "contract": { + "model": self.model, "tp_size": self.tp_size, + "operator": self.operator, "M": self.M, "N": self.N, + "K": self.K, + }, + "family": self.family or family_of(self.M, self.operator), + "baseline_us": self.baseline_us, + "best_known_us": self.best_known_us, + "history": self.history, + "last_selected_round": self.last_selected_round, + } + + +def _entry_to_instance(d: Dict[str, Any]) -> PoolInstance: + c = d.get("contract") or {} + baseline = d.get("baseline_us") + if not isinstance(baseline, (int, float)) or baseline <= 0: + raise ValueError(f"instance {d.get('id')} needs positive baseline_us") + inst = PoolInstance( + id=str(d["id"]), + model=str(c.get("model", "")), + tp_size=int(c.get("tp_size", 8)), + operator=str(c.get("operator", "")), + M=int(c.get("M", 0)), + N=int(c.get("N", 0)), + K=int(c.get("K", 0)), + baseline_us=float(baseline), + family=str(d.get("family") or family_of(int(c.get("M", 0)), + str(c.get("operator", "")))), + best_known_us=( + float(d["best_known_us"]) if d.get("best_known_us") else None + ), + history=list(d.get("history") or []), + last_selected_round=( + int(d["last_selected_round"]) if d.get("last_selected_round") is not None + else None + ), + ) + return inst + + +class Pool: + """Registered question pool (file-backed or built-in fixture).""" + + def __init__(self, instances: Optional[Dict[str, PoolInstance]] = None): + self.instances: Dict[str, PoolInstance] = instances or {} + + @classmethod + def from_yaml(cls, path: Path) -> "Pool": + import yaml + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + entries = data.get("instances") if isinstance(data, dict) else data + pool = cls() + for d in entries or []: + inst = _entry_to_instance(d) + pool.register(inst) + return pool + + def register(self, inst: PoolInstance) -> None: + if inst.id in self.instances: + raise ValueError(f"duplicate pool instance: {inst.id}") + self.instances[inst.id] = inst + + def get(self, inst_id: str) -> PoolInstance: + return self.instances[inst_id] + + def all_ids(self) -> List[str]: + return sorted(self.instances) + + def assert_in_pool(self, ids: List[str]) -> None: + unknown = [i for i in ids if i not in self.instances] + if unknown: + raise ValueError(f"instances not in pool: {unknown}") + + # -- per-family automatic target (tau) -------------------------------- + def family_ratios(self, family: str) -> List[float]: + ratios = [] + for inst in self.instances.values(): + if (inst.family or family_of(inst.M, inst.operator)) != family: + continue + for rec in inst.history: + if rec.get("accepted") is True and rec.get("median_us"): + try: + r = float(rec["median_us"]) / float(inst.baseline_us) + except (TypeError, ValueError, ZeroDivisionError): + continue + if 0 < r < 1.0: + ratios.append(r) + return ratios + + def tau_family(self, family: str, quantile: float = 0.6, + default: float = DEFAULT_TAU) -> float: + ratios = self.family_ratios(family) + if not ratios: + return default + ratios.sort() + idx = min(len(ratios) - 1, max(0, int(len(ratios) * quantile))) + return max(0.05, ratios[idx]) + + # -- auto pass --------------------------------------------------------- + def auto_pass(self, inst_id: str, median_us: float, *, + correctness_ok: bool, p90_ok: bool) -> Dict[str, Any]: + inst = self.get(inst_id) + tau = self.tau_family(inst.family or family_of(inst.M, inst.operator)) + target_us = tau * float(inst.baseline_us) + perf_ok = median_us <= target_us + passed = bool(correctness_ok and perf_ok and p90_ok) + return { + "instance": inst_id, + "passed": passed, + "tau_family": tau, + "target_us": round(target_us, 3), + "median_us": round(float(median_us), 3), + "correctness_ok": bool(correctness_ok), + "p90_ok": bool(p90_ok), + } + + # -- coverage / sampling helpers --------------------------------------- + def stratified_sample(self, budget: int, + exclude: Optional[Set[str]] = None) -> List[str]: + """Deterministic coverage-first sample: at least one per family/M-regime + bucket when budget allows, then fill by insertion order.""" + exclude = exclude or set() + by_family: Dict[str, List[str]] = {} + for iid in self.all_ids(): + if iid in exclude: + continue + inst = self.instances[iid] + fam = inst.family or family_of(inst.M, inst.operator) + by_family.setdefault(fam, []).append(iid) + picked: List[str] = [] + seen: Set[str] = set() + for fam in sorted(by_family): + if len(picked) >= budget: + break + iid = by_family[fam][0] + picked.append(iid) + seen.add(iid) + for iid in self.all_ids(): + if len(picked) >= budget: + break + if iid not in seen and iid not in exclude: + picked.append(iid) + seen.add(iid) + return picked + + def historical_ids(self) -> List[str]: + """Instances with at least one accepted history record (anchor set).""" + return [ + iid for iid, inst in self.instances.items() if inst.history + ] + + def heldout_ids(self, k: int, *, + exclude: Optional[Set[str]] = None) -> List[str]: + """Least-recently-selected instances (lightweight held-out probe).""" + exclude = exclude or set() + candidates = [ + (inst.last_selected_round if inst.last_selected_round is not None + else -10**9, iid) + for iid, inst in self.instances.items() + if iid not in exclude + ] + candidates.sort() + return [iid for _, iid in candidates[:k]] + + +# --------------------------------------------------------------------------- +# Built-in fixture pool (dry-run / tests). Real deployment loads a user pool +# file (registered operators with measured baselines). +# --------------------------------------------------------------------------- +def _default_instances() -> Dict[str, PoolInstance]: + rows = [ + ("dsv4_tp8_qkv_proj_m16", "deepseek-v4", 8, "qkv_proj", 16, 1280, 4096, 100.0, 45.0), + ("hy3_tp8_qkv_proj_m16", "hy3", 8, "qkv_proj", 16, 1280, 4096, 90.0, 40.0), + ("hy3_tp8_o_proj_m16", "hy3", 8, "o_proj", 16, 4096, 1024, 60.0, 22.0), + ("glm52_tp4_gate_up_m16", "glm52", 4, "shared_gate_up_proj", 16, 1024, 4096, 70.0, 30.0), + ("hy3_tp8_qkv_proj_m4096", "hy3", 8, "qkv_proj", 4096, 1280, 4096, 10000.0, 5000.0), + ("hy3_tp8_o_proj_m4096", "hy3", 8, "o_proj", 4096, 4096, 1024, 8000.0, 3000.0), + ("hy3_tp8_gate_up_m4096", "hy3", 8, "shared_gate_up_proj", 4096, 384, 4096, 7000.0, 2500.0), + ] + insts = {} + for iid, model, tp, op, m, n, k, base, best in rows: + hist = [{"accepted": True, "median_us": best}] + insts[iid] = PoolInstance( + id=iid, model=model, tp_size=tp, operator=op, M=m, N=n, K=k, + baseline_us=base, family=family_of(m, op), + best_known_us=best, history=hist, last_selected_round=None, + ) + return insts + + +def builtin_pool() -> Pool: + return Pool(_default_instances()) diff --git a/metainfer/tasks/harness_evolve/orchestrator/pool_builder.py b/metainfer/tasks/harness_evolve/orchestrator/pool_builder.py new file mode 100644 index 00000000..ab41f8b6 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/pool_builder.py @@ -0,0 +1,105 @@ +"""Build a measured AHE question pool from historical DKAO final reports. + +Sources are read-only (MetaInfer nodes/*/workspaces/*/final_report.json). +Output belongs under the isolated AHE root, e.g. +/root/zth_agent/ahe-kernel-repos/registered_pool.yaml. +""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any, Dict + +import yaml + +from .pool import family_of + + +def build_pool(meta_root: Path) -> Dict[str, Any]: + entries: Dict[str, Dict[str, Any]] = {} + for report_path in sorted(meta_root.glob("nodes/*/workspaces/*/final_report.json")): + try: + report = json.loads(report_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + continue + config = report.get("config") or {} + initial = report.get("initial_metrics") or {} + final = report.get("final_validation") or {} + for shape in config.get("shapes", []) or []: + sid = str(shape.get("id") or "") + if not sid: + continue + baseline = (initial.get(sid) or {}).get("baseline_us") + if not isinstance(baseline, (int, float)) or baseline <= 0: + continue + metrics = final.get(sid) or {} + median = metrics.get("median_us") + current = entries.get(sid) + best = ( + float(median) if isinstance(median, (int, float)) and median > 0 + else None + ) + history = [] + if best is not None: + history.append({ + "accepted": True, + "median_us": best, + "p90_us": metrics.get("p90_us"), + "source_report": str(report_path), + }) + row = { + "id": sid, + "contract": { + "model": str(config.get("model") or report.get("model") or + report_path.parent.name), + "tp_size": int(shape.get("tp_size", 8)), + "operator": str(shape.get("operator") or ""), + "M": int(shape.get("M", 0)), + "N": int(shape.get("N", 0)), + "K": int(shape.get("K", 0)), + }, + "family": family_of(int(shape.get("M", 0)), + str(shape.get("operator") or "")), + "baseline_us": float(baseline), + "best_known_us": best, + "history": history, + "last_selected_round": None, + } + if current is None: + entries[sid] = row + else: + # Keep the fixed baseline from the first valid report; merge + # histories and retain the fastest measured best-known. + current["history"].extend(history) + if best is not None and ( + current.get("best_known_us") is None + or best < current["best_known_us"] + ): + current["best_known_us"] = best + current["contract"] = row["contract"] + return { + "schema_version": 1, + "source": str(meta_root.resolve()), + "instances": [entries[k] for k in sorted(entries)], + } + + +def main(argv=None) -> int: + p = argparse.ArgumentParser() + p.add_argument("--meta-root", type=Path, + default=Path("/root/zth_agent/MetaInfer")) + p.add_argument("--out", type=Path, + default=Path("/root/zth_agent/ahe-kernel-repos/registered_pool.yaml")) + args = p.parse_args(argv) + data = build_pool(args.meta_root) + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(yaml.safe_dump(data, sort_keys=False, allow_unicode=True), + encoding="utf-8") + print(f"pool: {len(data['instances'])} instances -> {args.out}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/metainfer/tasks/harness_evolve/orchestrator/process_control.py b/metainfer/tasks/harness_evolve/orchestrator/process_control.py new file mode 100644 index 00000000..abc4d994 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/process_control.py @@ -0,0 +1,133 @@ +"""Child-process handles: spawn, watch, terminate. + +Why this exists +--------------- +A question runs as a real DKAO child task for hours, and HE has to be able to +stop that child when the round itself is over — most importantly when the child +outstays its timeout (``child_timeout_minutes``) and would otherwise hold a +device forever. + +* ``spawn`` starts the child in its own session, so the child and everything it + spawned (agent sessions, compilers, benchmark subprocesses) can be signalled + as one group; +* ``terminate`` escalates ``SIGTERM`` → ``SIGKILL`` on that group. + +What is deliberately **not** here any more: freezing (``SIGSTOP``/``SIGCONT``). +It existed to pause a child whose device had been taken over — exactly the +"HE manages the environment" coupling that FLOW.md §0 removed. Whether a card is +clean enough to measure on is DKAO's verdict, taken in the child's own +measurement gate, and the child's contamination handling deals with a round that +was measured on a shared device. A stopped process keeps its VRAM anyway, so +pausing never handed the card to anybody. +""" + +from __future__ import annotations + +import os +import signal +import subprocess +import threading +import time +from pathlib import Path +from typing import Any, Dict, List, Optional, Sequence + + +class ChildProcess: + """One child task under process control.""" + + def __init__(self, proc: "subprocess.Popen[Any]") -> None: + self.proc = proc + self.pid = int(proc.pid) + # start_new_session=True made the child a session/group leader, so the + # whole tree shares this pgid. + try: + self.pgid = os.getpgid(self.pid) + except OSError: + self.pgid = self.pid + self.terminated: bool = False + self.terminate_reason: str = "" + + # ------------------------------------------------------------- queries + def poll(self) -> Optional[int]: + return self.proc.poll() + + def alive(self) -> bool: + return self.proc.poll() is None + + # ------------------------------------------------------------- control + def _signal_group(self, sig: int) -> bool: + """Signal the whole process group; fall back to the child alone.""" + try: + os.killpg(self.pgid, sig) + return True + except OSError: + pass + try: + os.kill(self.pid, sig) + return True + except OSError: + return False + + def terminate(self, *, reason: str = "", grace_s: float = 20.0, + now: Optional[float] = None) -> bool: + """Stop the child for good (``SIGTERM`` then ``SIGKILL``).""" + if self.terminated: + return False + self.terminate_reason = reason + self._signal_group(signal.SIGTERM) + deadline = time.time() + max(0.0, float(grace_s)) + while self.alive() and time.time() < deadline: + time.sleep(0.2) + if self.alive(): + self._signal_group(signal.SIGKILL) + self.terminated = True + return True + + def state(self) -> Dict[str, Any]: + return { + "pid": self.pid, + "pgid": self.pgid, + "alive": self.alive(), + "terminated": self.terminated, + "terminate_reason": self.terminate_reason, + } + + +def spawn(command: Sequence[str], *, cwd: str, env: Dict[str, str], + log_path: Path) -> ChildProcess: + """Start a child task in its own session (so it can be signalled as a tree).""" + log = log_path.open("w", encoding="utf-8") + proc = subprocess.Popen( + list(command), cwd=cwd, env=env, stdout=log, + stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL, + text=True, start_new_session=True, + ) + log.close() # the child keeps its own descriptor + return ChildProcess(proc) + + +class ProcessRegistry: + """Live child handles, keyed by question id (thread-safe enough for us).""" + + def __init__(self) -> None: + self._children: Dict[str, ChildProcess] = {} + self._lock = threading.Lock() + + def add(self, question_id: str, child: ChildProcess) -> None: + with self._lock: + self._children[question_id] = child + + def get(self, question_id: str) -> Optional[ChildProcess]: + with self._lock: + return self._children.get(question_id) + + def drop(self, question_id: str) -> None: + with self._lock: + self._children.pop(question_id, None) + + def items(self) -> List[Any]: + with self._lock: + return list(self._children.items()) + + def state(self) -> Dict[str, Any]: + return {qid: child.state() for qid, child in self.items()} diff --git a/metainfer/tasks/harness_evolve/orchestrator/promotion.py b/metainfer/tasks/harness_evolve/orchestrator/promotion.py new file mode 100644 index 00000000..5b80aa00 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/promotion.py @@ -0,0 +1,416 @@ +"""Human-approved promotion of an evolved harness into production DKAO. + +The operator's rule: an HE run exists to produce one promotable harness, and +nothing is written to DKAO without a human decision. So a candidate that passes +both gates does **not** publish itself: + +1. the round writes ``pending_promotion.json`` (candidate revision, snapshot, + both gate records) and the run parks in ``awaiting_approval``; +2. ``approve`` copies the candidate harness into production (backing up the + previous version), records it, updates the variant table and finishes the + run; +3. ``deny`` throws the candidate away and the caller keeps iterating. + +``rollback_harness`` puts the previous production version back, which is what +makes an approval safe to make. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any, Dict, Optional + +from .attribution import copy_tree, replace_tree, save_json +from .state import append_timeline + +PENDING_FILE = "pending_promotion.json" +PROMOTED_FILE = "promoted_harness.json" +PROMOTION_LOG = "promotion_log.jsonl" + + +def _load_json(path: Path) -> Any: + try: + return json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + +def _ts() -> str: + return time.strftime("%Y%m%d-%H%M%S") + + +def _log(exp_dir: Path, **fields: Any) -> None: + row = {"ts": time.time(), **fields} + try: + Path(exp_dir).mkdir(parents=True, exist_ok=True) + with (Path(exp_dir) / PROMOTION_LOG).open("a", encoding="utf-8") as fh: + fh.write(json.dumps(row, ensure_ascii=False) + "\n") + except OSError: + pass + + +def version_for(iteration: int) -> str: + """The harness version label used in records (h-1 is the seed).""" + return f"h-{int(iteration)}" + + +def write_pending_promotion(exp_dir: Path, *, iteration: int, revision: str, + snapshot_dir: str, + performance_gate: Dict[str, Any], + generalization_gate: Dict[str, Any], + results: Dict[str, Any]) -> Path: + """Record a candidate that passed both gates and is awaiting approval.""" + exp_dir = Path(exp_dir) + payload = { + "schema_version": 1, + "created_at": time.time(), + "iteration": iteration, + "version": version_for(iteration), + "revision": revision, + "snapshot_dir": snapshot_dir, + "performance_gate": performance_gate, + "generalization_gate": generalization_gate, + "operator_results": { + iid: {k: row.get(k) for k in + ("median_us", "p90_us", "passed", "correctness_ok", + "status", "repo_path")} + for iid, row in (results or {}).items() if isinstance(row, dict) + }, + "awaiting": "human approval (approve / deny)", + } + save_json(exp_dir / PENDING_FILE, payload) + _log(exp_dir, action="pending_promotion", iteration=iteration, + version=payload["version"], revision=revision) + return exp_dir / PENDING_FILE + + +def read_pending(exp_dir: Path) -> Optional[Dict[str, Any]]: + data = _load_json(Path(exp_dir) / PENDING_FILE) + return data if isinstance(data, dict) else None + + +def clear_pending(exp_dir: Path) -> None: + try: + (Path(exp_dir) / PENDING_FILE).unlink() + except OSError: + pass + + +def promoted_harness(exp_dir: Path) -> Optional[Dict[str, Any]]: + data = _load_json(Path(exp_dir) / PROMOTED_FILE) + return data if isinstance(data, dict) else None + + +def _production_dir() -> Path: + from .config import harness_seed_dir + return harness_seed_dir("dcu_default") + + +def _versioned_backup(exp_dir: Path, version: str) -> Path: + """Keep a named copy of what production looked like before a promotion.""" + target = _production_dir() + backup_root = Path(exp_dir) / "promoted" + backup_root.mkdir(parents=True, exist_ok=True) + backup = backup_root / version + if target.is_dir() and not backup.exists(): + copy_tree(target, backup) + return backup + + +def promote_harness(exp_dir: Path, pending: Dict[str, Any], *, + approved_by: str = "operator") -> Dict[str, Any]: + """Write the approved candidate harness into production DKAO. + + The previous production tree is kept as a named copy (and a timestamped + one) so a promotion is always reversible. + """ + exp_dir = Path(exp_dir) + snapshot = Path(str(pending.get("snapshot_dir") or "")) + if not snapshot.is_dir(): + return {"ok": False, "errors": [f"candidate snapshot missing: {snapshot}"]} + target = _production_dir() + version = str(pending.get("version") or version_for(pending.get("iteration", 0))) + previous = promoted_harness(exp_dir) or {} + previous_version = previous.get("version") or "h-1" + backup = _versioned_backup(exp_dir, previous_version) + live_backup = target.parent / f"{target.name}.{_ts()}" + if target.is_dir(): + copy_tree(target, live_backup) + replace_tree(snapshot, target) + + record = { + "schema_version": 1, + "version": version, + "iteration": pending.get("iteration"), + "revision": pending.get("revision"), + "snapshot_dir": str(snapshot), + "production_dir": str(target), + "backup_dir": str(backup), + "previous_version": previous_version, + "previous_backup": str(live_backup), + "approved_by": approved_by, + "approved_at": time.time(), + "performance_gate": { + "status": (pending.get("performance_gate") or {}).get("status"), + "wins": (pending.get("performance_gate") or {}).get("wins"), + }, + "generalization_gate": { + "status": (pending.get("generalization_gate") or {}).get("status"), + "wins": (pending.get("generalization_gate") or {}).get("wins"), + "paper": (pending.get("generalization_gate") or {}).get("operators"), + }, + } + save_json(exp_dir / PROMOTED_FILE, record) + _log(exp_dir, action="promoted", version=version, + iteration=pending.get("iteration"), revision=pending.get("revision"), + production_dir=str(target), backup=str(backup), + previous_version=previous_version, approved_by=approved_by) + return {"ok": True, "record": record, "backup": str(backup), + "previous_backup": str(live_backup)} + + +def rollback_harness(exp_dir: Path, *, to: str = "last-good") -> Dict[str, Any]: + """Restore a previously promoted harness version into production. + + ``to`` accepts a version label (``h-2``), ``last-good`` (the version the + current one replaced) or ``seed`` (the built-in harness). + """ + exp_dir = Path(exp_dir) + current = promoted_harness(exp_dir) or {} + target = _production_dir() + if to == "seed": + source = harness_seed_source() + label = "seed" + elif to in {"last-good", "last_good", ""}: + source = Path(str(current.get("backup_dir") or "")) + label = str(current.get("previous_version") or "seed") + if not source.is_dir(): + source = harness_seed_source() + label = "seed" + else: + source = exp_dir / "promoted" / str(to) + label = str(to) + if not Path(source).is_dir(): + return {"ok": False, "errors": [f"no harness tree to restore: {source}"]} + live_backup = target.parent / f"{target.name}.{_ts()}" + if target.is_dir(): + copy_tree(target, live_backup) + replace_tree(Path(source), target) + record = { + "schema_version": 1, + "version": f"rollback:{label}", + "rolled_back_at": time.time(), + "production_dir": str(target), + "restored_from": str(source), + "replaced_backup": str(live_backup), + "replaced_version": current.get("version"), + } + save_json(exp_dir / PROMOTED_FILE, record) + _log(exp_dir, action="rollback", to=label, restored_from=str(source), + replaced_backup=str(live_backup)) + return {"ok": True, "record": record, "restored_from": str(source)} + + +def harness_seed_source() -> Path: + """The built-in seed harness that ships with the plugin (h-1).""" + here = Path(__file__).resolve().parents[2] + return here / "dcu_kernel_auto_opt" / "harness_default" + + +def restore_promoted_harness(cfg: Any, exp_dir: Path, + live_workspace: Path) -> Dict[str, Any]: + """Put the experiment workspace back to the promoted (production) harness. + + A rejected round must not leave its own edit live, and it must not fall all + the way back to the seed either: the reference for the next round is what + production currently runs. + """ + promoted = promoted_harness(exp_dir) + source: Optional[Path] = None + revision = "" + if promoted and promoted.get("snapshot_dir"): + candidate = Path(str(promoted["snapshot_dir"])) + if candidate.is_dir(): + source = candidate + revision = str(promoted.get("revision") or "") + if source is None: + source = harness_seed_source() + replace_tree(source, live_workspace) + if revision: + import subprocess + try: + subprocess.run(["git", "reset", "--hard", revision], + cwd=live_workspace, check=True, + stdout=subprocess.DEVNULL) + except (OSError, subprocess.CalledProcessError): + pass + append_timeline(cfg.state_dir, "workspace_restored_to_promoted", { + "from": str(source), + "revision": revision or None, + "version": (promoted or {}).get("version"), + }) + return {"restored_from": str(source), "revision": revision} + + +def _versioned_pool_backup(exp_dir: Path, pool_path: Path) -> Path: + """Copy the pool next to the experiment before rewriting production data.""" + import shutil + root = Path(exp_dir) / "promoted" + root.mkdir(parents=True, exist_ok=True) + backup = root / f"{pool_path.stem}.{_ts()}{pool_path.suffix}" + try: + shutil.copy2(pool_path, backup) + except OSError: + return pool_path + return backup + + +def promote_kernels_for_round(exp_dir: Path, pool_path: Path, + results: Dict[str, Any], *, + version: str, iteration: int, + reason: str = "harness promotion", + ) -> Dict[str, Any]: + """Write an accepted round's operator kernels into the production pool. + + Only the operators this round actually measured are touched, and only when + they beat what the pool already records — "best known wins", applied + surgically instead of rescanning every historical child report. The pool is + backed up before being rewritten. + """ + import yaml + from .variant import _num + + pool_path = Path(pool_path) + if not pool_path.is_file(): + return {"ok": False, "updated": [], + "errors": [f"pool file missing: {pool_path}"]} + try: + data = yaml.safe_load(pool_path.read_text(encoding="utf-8")) or {} + except (OSError, ValueError) as exc: # noqa: BLE001 + return {"ok": False, "updated": [], "errors": [str(exc)]} + instances = {str(i.get("id")): i for i in (data.get("instances") or []) + if isinstance(i, dict)} + updated: List[Dict[str, Any]] = [] + for iid, row in (results or {}).items(): + if not isinstance(row, dict): + continue + entry = instances.get(str(iid)) + if entry is None: + continue + median = _num(row.get("median_us")) + if median is None or row.get("correctness_ok") is False: + continue + best = _num(entry.get("best_known_us")) + if best is not None and median >= best: + continue + entry["best_known_us"] = float(median) + history = entry.setdefault("history", []) + if isinstance(history, list): + history.append({ + "accepted": True, + "median_us": float(median), + "p90_us": row.get("p90_us"), + "source_report": row.get("repo_path") or row.get("child_task_id"), + "harness_version": version, + "iteration": iteration, + }) + updated.append({"operator_id": str(iid), "median_us": float(median), + "previous_best_us": best}) + if not updated: + return {"ok": True, "updated": [], "pool": str(pool_path), + "reason": reason} + backup = _versioned_pool_backup(exp_dir, pool_path) + pool_path.write_text( + yaml.safe_dump(data, sort_keys=False, allow_unicode=True), + encoding="utf-8") + _log(exp_dir, action="kernels_promoted", version=version, + iteration=iteration, pool=str(pool_path), updated=updated, + backup=str(backup)) + return {"ok": True, "updated": updated, "pool": str(pool_path), + "backup": str(backup), "reason": reason} + + +def resolve_question_pool(exp_dir: Path) -> Optional[Path]: + """The pool file this experiment draws its operators from, if any.""" + exp_dir = Path(exp_dir) + for name in ("requirements.json", "config_snapshot.json"): + data = _load_json(exp_dir / name) + if not isinstance(data, dict): + continue + answers = (data.get("answers") + if isinstance(data.get("answers"), dict) else data) + for key in ("question_pool", "pool_source"): + raw = str((answers or {}).get(key) or "").strip() + if raw and raw != "builtin": + candidate = Path(raw).expanduser() + if candidate.is_file(): + return candidate + return None + + +def approve_promotion(cfg: Any, exp_dir: Path, *, + approved_by: str = "operator") -> Dict[str, Any]: + """Approve the pending candidate: promote it, update variants, finish run.""" + from .variant import load_variant_table, record_variants + + exp_dir = Path(exp_dir) + pending = read_pending(exp_dir) + if not pending: + return {"ok": False, "errors": ["no pending promotion"]} + result = promote_harness(exp_dir, pending, approved_by=approved_by) + if not result.get("ok"): + return result + version = result["record"]["version"] + table = load_variant_table(exp_dir) + # The candidate won on the operators it measured; those become the new + # variants (and therefore the reference for the next HE run). + variant_update = record_variants( + exp_dir, table, pending.get("operator_results") or {}, + version=version, iteration=int(pending.get("iteration") or 0), + groups={"generalization_ids": + (pending.get("generalization_gate") or {}).get("operators") or []}) + # ... and their kernels go into production, so the accepted round is the + # only one whose optimizations survive. + pool_path = resolve_question_pool(exp_dir) + if pool_path is not None: + kernel_update = promote_kernels_for_round( + exp_dir, pool_path, pending.get("operator_results") or {}, + version=version, iteration=int(pending.get("iteration") or 0)) + else: + kernel_update = {"ok": True, "updated": [], + "skipped": "no question pool configured"} + clear_pending(exp_dir) + append_timeline(cfg.state_dir, "harness_promoted", { + "version": version, + "iteration": pending.get("iteration"), + "approved_by": approved_by, + "variant_updates": variant_update.get("updated"), + "kernels_promoted": [u["operator_id"] + for u in kernel_update.get("updated", [])], + "backup": result.get("backup"), + }) + try: + from .state import set_run + set_run(cfg.state_dir, final_status=f"promoted:{version}") + except Exception: # noqa: BLE001 + pass + return {**result, "variant_update": variant_update, + "kernel_update": kernel_update, "version": version} + + +def deny_promotion(cfg: Any, exp_dir: Path, *, reason: str = "operator") -> Dict[str, Any]: + """Throw the pending candidate away; the run keeps iterating.""" + exp_dir = Path(exp_dir) + pending = read_pending(exp_dir) + if not pending: + return {"ok": False, "errors": ["no pending promotion"]} + clear_pending(exp_dir) + _log(exp_dir, action="denied", iteration=pending.get("iteration"), + version=pending.get("version"), reason=reason) + append_timeline(cfg.state_dir, "promotion_denied", { + "iteration": pending.get("iteration"), + "version": pending.get("version"), "reason": reason, + }) + return {"ok": True, "denied": pending.get("version")} diff --git a/metainfer/tasks/harness_evolve/orchestrator/publish.py b/metainfer/tasks/harness_evolve/orchestrator/publish.py new file mode 100644 index 00000000..74168872 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/publish.py @@ -0,0 +1,200 @@ +"""Publish gate: promote an AHE experiment's champion into production DKAO. + +Nothing is ever written to the production DKAO automatically. Publish only +happens through this explicit gate, and only when the requested evidence is on +disk: + +Harness publish (champion -> dcu_kernel_auto_opt/harness_default): + required: a promoted iteration (verdict PROMOTE / PROMOTE_UNEXPLAINED) with a + held-out gate that did NOT fail; target dir is backed up first and + a publish_log.jsonl record is appended for rollback. + +Kernel publish (new best-knowns -> registered pool): + scans AHE child DKAO runs for final-success shapes whose median beats the pool + best_known; the pool YAML is backed up, then best_known/history are updated. + +Every publish creates a timestamped backup of the destination before writing. +""" + +from __future__ import annotations + +import json +import shutil +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Dict, List, Optional, Tuple + +import yaml + +from .attribution import copy_tree, replace_tree, save_json + +PUBLISH_LOG = "publish_log.jsonl" +_PROMOTABLE = {"PROMOTE", "PROMOTE_UNEXPLAINED"} + + +def _ts() -> str: + return time.strftime("%Y%m%d-%H%M%S") + + +def _backup(path: Path, suffix: str) -> Path: + """Copy destination into a timestamped backup (keep production live).""" + if not path.exists(): + return path # nothing to back up yet + backup = path.parent / f"{path.name}.{suffix}-{_ts()}" + if path.is_dir(): + copy_tree(path, backup) + else: + backup.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(path, backup) + return backup + + +def _decision_at(exp: Path, iteration: int) -> Optional[Dict[str, Any]]: + path = exp / "runs" / f"iteration_{iteration:03d}" / "input" / "decision.json" + if not path.is_file(): + return None + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + return data if isinstance(data, dict) else None + + +def _best_ever(exp: Path) -> Optional[Dict[str, Any]]: + path = exp / "best_ever.json" + if not path.is_file(): + return None + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + return data if isinstance(data, dict) else None + + +def check_publishable(exp: Path) -> Dict[str, Any]: + """Validate the experiment carries promotable, non-overfit evidence.""" + best = _best_ever(exp) + if not best: + return {"ok": False, "errors": ["no best_ever.json (nothing promoted)"]} + iteration = int(best.get("iteration", 0)) + decision = _decision_at(exp, iteration) + if not decision: + return {"ok": False, "errors": [f"no decision.json at iteration {iteration}"]} + verdict = decision.get("verdict") + errors: List[str] = [] + if verdict not in _PROMOTABLE: + errors.append(f"iteration {iteration} verdict {verdict!r} is not promotable") + held = ((decision.get("heldout_gate") or {}).get("status")) + if held == "FAIL": + errors.append("held-out gate failed (overfit guard)") + snapshot = best.get("snapshot_dir") + if not snapshot or not Path(snapshot).is_dir(): + errors.append("best snapshot dir missing") + if errors: + return {"ok": False, "errors": errors} + return { + "ok": True, + "iteration": iteration, + "verdict": verdict, + "snapshot_dir": str(snapshot), + "workspace_revision": best.get("workspace_revision"), + } + + +def publish_harness(exp: Path, *, target: Path, reason: str = "explicit publish" + ) -> Dict[str, Any]: + """Copy the champion harness snapshot over the production default seed.""" + gate = check_publishable(exp) + if not gate["ok"]: + return {"ok": False, "errors": gate["errors"], "published": False} + source = Path(gate["snapshot_dir"]) + backup = _backup(target, "pre-publish") + replace_tree(source, target) + record = { + "ts": time.time(), + "kind": "harness", + "iteration": gate["iteration"], + "verdict": gate["verdict"], + "source_snapshot": str(source), + "workspace_revision": gate.get("workspace_revision"), + "target": str(target), + "backup": str(backup) if backup != target else None, + "reason": reason, + } + _append_log(exp, record) + return {"ok": True, "published": True, "record": record} + + +def _child_reports(exp: Path) -> List[Tuple[str, Dict[str, Any]]]: + reports: List[Tuple[str, Dict[str, Any]]] = [] + for path in sorted((exp / "children").glob("*/*/workspace/final_report.json")): + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + continue + if isinstance(data, dict): + reports.append((str(path), data)) + return reports + + +def publish_kernels(exp: Path, *, pool_path: Path, reason: str = "explicit publish" + ) -> Dict[str, Any]: + """Update pool best_known/history from final-success child DKAO runs.""" + if not pool_path.is_file(): + return {"ok": False, "published": False, + "errors": [f"pool file missing: {pool_path}"]} + try: + data = yaml.safe_load(pool_path.read_text(encoding="utf-8")) or {} + except (OSError, ValueError) as exc: + return {"ok": False, "published": False, "errors": [str(exc)]} + instances = {i["id"]: i for i in data.get("instances", [])} + updated: List[str] = [] + for src, report in _child_reports(exp): + if report.get("status") != "success": + continue + final = report.get("final_validation") or {} + config = report.get("config") or {} + for shape in config.get("shapes", []) or []: + sid = str(shape.get("id") or "") + entry = instances.get(sid) + metrics = final.get(sid) or {} + median = metrics.get("median_us") + if ( + entry is None + or not isinstance(median, (int, float)) + or median <= 0 + ): + continue + best = entry.get("best_known_us") + if best is None or float(median) < float(best): + entry["best_known_us"] = float(median) + hist = entry.setdefault("history", []) + hist.append({ + "accepted": True, + "median_us": float(median), + "p90_us": metrics.get("p90_us"), + "source_report": src, + }) + updated.append(sid) + if not updated: + return {"ok": True, "published": False, "updated": [], + "record": {"kind": "kernels", "reason": reason}} + backup = _backup(pool_path, "pre-publish") + pool_path.write_text(yaml.safe_dump(data, sort_keys=False, allow_unicode=True), + encoding="utf-8") + record = { + "ts": time.time(), + "kind": "kernels", + "updated": updated, + "target": str(pool_path), + "backup": str(backup) if backup != pool_path else None, + "reason": reason, + } + _append_log(exp, record) + return {"ok": True, "published": True, "updated": updated, "record": record} + + +def _append_log(exp: Path, record: Dict[str, Any]) -> None: + with (exp / PUBLISH_LOG).open("a", encoding="utf-8") as fh: + fh.write(json.dumps(record, ensure_ascii=False) + "\n") diff --git a/metainfer/tasks/harness_evolve/orchestrator/question_roles.py b/metainfer/tasks/harness_evolve/orchestrator/question_roles.py new file mode 100644 index 00000000..f9eb1c79 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/question_roles.py @@ -0,0 +1,428 @@ +"""Role-based question selection for AHE rounds. + +The previous rule let the Evolve agent pick any in-pool set (guardrails only +enforced membership, budget and >=60% overlap). That produced the failure we +actually hit on 9-8-8: three of four questions came from the same operator +family, an over-generalised policy change looked like a win, and the decision +promoted it — after which two unrelated shapes regressed. + +This module gives every question in a round an explicit **role**, so a round +measures what it claims to measure: + + regression 2 slots — questions that PASSED last round and are inside the + change's declared scope; spread across families. Primary causal + evidence that the change did not break what worked. + repair 1 slot — questions that LOST last round (or sit in scope.at_risk); + when nothing lost, the least-recently-examined question of a + different family. + probe 1 slot — deliberately a *different* family than everything else + (cold-start generalisation probe). It is recorded but carries + zero weight in the pass-rate/pairing decision. + +Machine constraints on top: at least ``min_families`` distinct families (when +the pool allows), at most ``max_per_family`` questions per family, >=60% +overlap with the previous round, and least-recently-used rotation as the +tie-breaker so the pool is not pinned to the same four shapes forever. + +The agent's own ``selected`` list is honoured where it satisfies a role, and +replaced where it does not — the correction is recorded in the rationale. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Dict, List, Mapping, Optional, Sequence + +PURPOSE_REGRESSION = "regression" +PURPOSE_REPAIR = "repair" +PURPOSE_PROBE = "probe" +MAIN_PURPOSES = (PURPOSE_REGRESSION, PURPOSE_REPAIR) + +MIN_OVERLAP_RATIO = 0.6 + + +@dataclass +class Selection: + selected: List[str] = field(default_factory=list) + purposes: Dict[str, str] = field(default_factory=dict) + rationale: str = "" + families: List[str] = field(default_factory=list) + notes: List[str] = field(default_factory=list) + + def to_dict(self) -> Dict[str, Any]: + return { + "selected": self.selected, + "purposes": self.purposes, + "rationale": self.rationale, + "families": self.families, + "notes": self.notes, + } + + +def _family(pool: Any, iid: str) -> str: + try: + inst = pool.get(iid) + except Exception: # noqa: BLE001 + return "" + return str(getattr(inst, "family", "") or "") + + +def _regime(pool: Any, iid: str) -> str: + """decode (M<=32) vs prefill (M>32) — a stronger axis than operator.""" + try: + inst = pool.get(iid) + except Exception: # noqa: BLE001 + return "" + try: + m = int(getattr(inst, "M", 0) or 0) + except (TypeError, ValueError): + return "" + if m <= 0: + return "" + return "decode" if m <= 32 else "prefill" + + +def _rotation_key(rotation: Mapping[str, int], iid: str) -> int: + try: + return int(rotation.get(iid, 0)) + except (TypeError, ValueError): + return 0 + + +def _scope_ids(plan: Optional[Any]) -> List[str]: + """Ids the change declares it will improve / must keep unchanged.""" + ids: List[str] = [] + changes = getattr(plan, "changes", None) + if changes is None and isinstance(plan, Mapping): + changes = plan.get("changes") + for change in changes or []: + scope = change.get("scope") if isinstance(change, Mapping) else None + if not isinstance(scope, Mapping): + continue + for bucket in ("expected_improve", "unchanged", "at_risk"): + bucket_ids = scope.get(bucket) + if isinstance(bucket_ids, str): + bucket_ids = [bucket_ids] + for iid in bucket_ids or []: + if isinstance(iid, str) and iid not in ids: + ids.append(iid) + return ids + + +def _pick_spread(candidates: Sequence[str], pool: Any, used_families: set, + count: int, used_regimes: Optional[set] = None) -> List[str]: + """Pick ``count`` ids preferring uncovered regimes, then families. + + Regime first: decode (M<=32) and prefill (M>32) need completely different + kernels, so a round that covers four prefill families but no decode + question would miss the harder, more valuable half of the space. + """ + used_regimes = used_regimes if used_regimes is not None else set() + + def rank(iid: str) -> tuple: + regime = _regime(pool, iid) + family = _family(pool, iid) + return ( + 0 if (regime and regime not in used_regimes) else 1, + 0 if (family and family not in used_families) else 1, + ) + + picked: List[str] = [] + remaining = list(candidates) + # Greedy: re-rank after every pick so the second anchor prefers the regime + # the first one did not cover (a pre-sorted pass would keep picking the + # same regime). + while len(remaining) > 0 and len(picked) < count: + remaining.sort(key=lambda i: (rank(i), candidates.index(i))) + best = remaining[0] + regime = _regime(pool, best) + family = _family(pool, best) + redundant = bool(family and family in used_families + and regime in used_regimes) + if redundant and len(remaining) > (count - len(picked)): + remaining.pop(0) + continue + remaining.pop(0) + picked.append(best) + if family: + used_families.add(family) + if regime: + used_regimes.add(regime) + return picked + + +def select_questions(*, pool: Any, budget: int, + prev_selected: Sequence[str] = (), + prev_results: Optional[Mapping[str, Mapping[str, Any]]] = None, + plan: Optional[Any] = None, + rotation: Optional[Mapping[str, int]] = None, + min_families: int = 3, + max_per_family: int = 2, + anchor_min: int = 2) -> Selection: + """Return this round's question set with per-question roles.""" + budget = int(budget) + rotation = rotation or {} + prev_results = prev_results or {} + notes: List[str] = [] + purposes: Dict[str, str] = {} + stages: Dict[str, str] = {} + + all_ids = list(pool.all_ids()) + families_available = {_family(pool, i) for i in all_ids} + families_available.discard("") + target_families = min(int(min_families), max(1, len(families_available))) + + if budget <= 0 or not all_ids: + return Selection(selected=[], purposes={}, rationale="empty pool") + + prev = [i for i in prev_selected if i in pool.instances] + passed = [i for i in prev if (prev_results.get(i) or {}).get("passed") is True] + failed = [i for i in prev + if prev_results.get(i) is not None + and (prev_results.get(i) or {}).get("passed") is not True] + scope_all = [i for i in _scope_ids(plan) if i in pool.instances] + scope_improve = [i for i in _scope_ids(plan) if i in pool.instances] + scope_risk: List[str] = [] + changes = getattr(plan, "changes", None) + if changes is None and isinstance(plan, Mapping): + changes = plan.get("changes") + for change in changes or []: + scope = change.get("scope") if isinstance(change, Mapping) else None + if not isinstance(scope, Mapping): + continue + for iid in scope.get("at_risk") or []: + if isinstance(iid, str) and iid in pool.instances and iid not in scope_risk: + scope_risk.append(iid) + for iid in scope.get("expected_improve") or []: + if isinstance(iid, str) and iid in pool.instances and iid not in scope_improve: + scope_improve.append(iid) + + agent_ids: List[str] = [] + plan_selected = getattr(plan, "selected", None) + if plan_selected is None and isinstance(plan, Mapping): + plan_selected = plan.get("selected") + for iid in plan_selected or []: + if isinstance(iid, str) and iid in pool.instances and iid not in agent_ids: + agent_ids.append(iid) + + chosen: List[str] = [] + used_families: set = set() + used_regimes: set = set() + + def _take(iid: str, purpose: str, stage: str) -> None: + if iid in chosen or len(chosen) >= budget: + return + chosen.append(iid) + purposes[iid] = purpose + stages[iid] = stage + used_families.add(_family(pool, iid)) + if _regime(pool, iid): + used_regimes.add(_regime(pool, iid)) + + def _family_counts() -> Dict[str, int]: + counts: Dict[str, int] = {} + for iid in chosen: + fam = _family(pool, iid) + counts[fam] = counts.get(fam, 0) + 1 + return counts + + def _lru(candidates: Sequence[str]) -> List[str]: + return sorted(candidates, key=lambda i: (_rotation_key(rotation, i), i)) + + def _drop_one(protect: Sequence[str] = (PURPOSE_REGRESSION,)) -> Optional[str]: + """Remove the least important question to make room.""" + order = ["fill", "agent", "repair", "probe", "regression"] + for stage in order: + victims = [i for i in chosen if stages.get(i) == stage] + for victim in reversed(victims): + if purposes.get(victim) in protect: + # keep at least two regression anchors + if purposes.get(victim) == PURPOSE_REGRESSION and \ + list(purposes.values()).count(PURPOSE_REGRESSION) > 2: + pass + else: + continue + chosen.remove(victim) + purposes.pop(victim, None) + stages.pop(victim, None) + return victim + return None + + # 1) regression anchors (2): passed last round, scope-aware, family-spread + anchor_pool = [i for i in scope_improve + scope_all if i in passed] or list(passed) + if not anchor_pool: + anchor_pool = list(pool.historical_ids()) + if prev: + notes.append("no passed question last round; anchors from history") + anchor_want = 2 if budget >= 3 else 1 + for iid in _pick_spread(anchor_pool, pool, used_families, anchor_want, + used_regimes): + _take(iid, PURPOSE_REGRESSION, "anchor") + + # 2) agent picks that already fit a role + for iid in agent_ids: + if len(chosen) >= budget: + break + if iid in passed and iid not in chosen: + _take(iid, PURPOSE_REGRESSION, "agent") + elif iid in failed and iid not in chosen: + _take(iid, PURPOSE_REPAIR, "agent") + + # 3) repair slot: at-risk scope first, then whatever lost, then LRU + if budget >= 2 and PURPOSE_REPAIR not in purposes.values(): + repair_pool = [i for i in scope_risk + list(failed) + if i not in chosen] + if not repair_pool: + repair_pool = _lru([i for i in all_ids if i not in chosen]) + if failed: + notes.append("repair slot filled by least-recently-examined question") + for iid in _pick_spread(repair_pool, pool, used_families, 1, + used_regimes): + _take(iid, PURPOSE_REPAIR, "repair") + + # 4) fill remaining slots: scope-improve first, then least-recently-used + if len(chosen) < budget: + fill_pool = [i for i in scope_improve if i not in chosen] + fill_pool += _lru([i for i in all_ids if i not in chosen + and i not in fill_pool]) + for iid in _pick_spread(fill_pool, pool, used_families, + budget - len(chosen), used_regimes): + _take(iid, PURPOSE_REGRESSION if iid in passed else PURPOSE_REPAIR, + "fill") + + # 5) generalisation probe: prefer a family nothing else covers + if budget >= 2: + uncovered = _lru([i for i in all_ids if i not in chosen + and _family(pool, i) + and _family(pool, i) not in used_families]) + if not uncovered: + uncovered = _lru([i for i in all_ids if i not in chosen + and _regime(pool, i) + and _regime(pool, i) not in used_regimes]) + if uncovered: + probe_id = uncovered[0] + if len(chosen) >= budget: + victim = _drop_one() + if victim is not None: + notes.append(f"replaced {victim} with cross-family probe " + f"{probe_id}") + _take(probe_id, PURPOSE_PROBE, "probe") + else: + # No untouched family left: promote the freshest question whose + # family appears once in this round (it is the generalisation + # signal), instead of swapping the set around. + counts = _family_counts() + singles = [i for i in chosen + if counts.get(_family(pool, i), 0) == 1 + and i not in probe_ids(purposes)] + singles.sort(key=lambda i: ( + purposes.get(i) == PURPOSE_REGRESSION, + _rotation_key(rotation, i), i)) + if singles: + probe_id = singles[0] + purposes[probe_id] = PURPOSE_PROBE + stages[probe_id] = "probe" + notes.append(f"probe is the unique-family question " + f"{probe_id} already in the round") + else: + fallback = _lru([i for i in all_ids if i not in chosen]) + if fallback: + probe_id = fallback[0] + victim = _drop_one() if len(chosen) >= budget else None + if victim is not None: + notes.append(f"replaced {victim} with probe {probe_id}") + _take(probe_id, PURPOSE_PROBE, "probe") + notes.append("pool exhausted per family; probe is the " + "freshest remaining question") + else: + notes.append("no cross-family probe available in the pool") + + # ---------------------------------------------------- machine guardrails + if max_per_family > 0: + counts = _family_counts() + for iid in list(chosen): + fam = _family(pool, iid) + if counts.get(fam, 0) <= max_per_family: + continue + replacement = next( + (c for c in _lru([i for i in all_ids if i not in chosen]) + if _family(pool, c) + and counts.get(_family(pool, c), 0) < max_per_family + and _family(pool, c) not in used_families), + None, + ) + if replacement is None: + continue + counts[fam] -= 1 + counts[_family(pool, replacement)] = counts.get( + _family(pool, replacement), 0) + 1 + purposes[replacement] = purposes.get(iid, PURPOSE_REPAIR) + stages[replacement] = stages.get(iid, "fill") + purposes.pop(iid, None) + stages.pop(iid, None) + chosen[chosen.index(iid)] = replacement + used_families = {_family(pool, c) for c in chosen} + notes.append(f"family quota: {iid} -> {replacement}") + + # overlap: keep round-to-round comparability + if prev: + need = int(MIN_OVERLAP_RATIO * len(prev)) + overlap = len(set(chosen) & set(prev)) + for iid in prev: + if overlap >= need or not chosen: + break + if iid in chosen: + continue + victim = _drop_one(protect=(PURPOSE_REGRESSION, PURPOSE_REPAIR)) + if victim is None: + break + notes.append(f"overlap backfill: {victim} -> {iid}") + _take(iid, PURPOSE_REGRESSION, "overlap") + overlap += 1 + + # anchors: at least anchor_min ids with accepted history + anchor_set = set(pool.historical_ids()) + have = sum(1 for i in chosen if i in anchor_set) + if have < anchor_min: + for iid in _lru([i for i in anchor_set if i not in chosen]): + if have >= anchor_min: + break + victim = _drop_one(protect=(PURPOSE_REGRESSION,)) + if victim is None: + break + _take(iid, PURPOSE_REGRESSION, "anchor") + have += 1 + notes.append(f"anchor backfill: added {iid}") + + families = sorted({_family(pool, i) for i in chosen if _family(pool, i)}) + regimes = sorted({_regime(pool, i) for i in chosen if _regime(pool, i)}) + if len(regimes) < 2 and len({_regime(pool, i) for i in all_ids + if _regime(pool, i)}) >= 2: + notes.append(f"regime coverage {regimes} covers a single regime") + if len(families) < target_families: + notes.append(f"family coverage {len(families)} < target {target_families}") + + rationale = ( + "roles: " + ", ".join(f"{i}={purposes.get(i)}" for i in chosen) + + f" | families: {', '.join(families)}" + + f" | regimes: {', '.join(regimes)}" + + (f" | notes: {'; '.join(notes)}" if notes else "") + ) + return Selection(selected=list(chosen), + purposes={i: purposes[i] for i in chosen if i in purposes}, + rationale=rationale, families=families, notes=notes) + + +def overlap_ratio(selected: Sequence[str], prev: Sequence[str]) -> float: + if not prev: + return 1.0 + return len(set(selected) & set(prev)) / len(set(prev)) + + +def main_purposes(purposes: Mapping[str, str]) -> List[str]: + """Ids that take part in the pass-rate / win-loss decision.""" + return [i for i, p in (purposes or {}).items() if p in MAIN_PURPOSES] + + +def probe_ids(purposes: Mapping[str, str]) -> List[str]: + return [i for i, p in (purposes or {}).items() if p == PURPOSE_PROBE] diff --git a/metainfer/tasks/harness_evolve/orchestrator/repo_cleanup.py b/metainfer/tasks/harness_evolve/orchestrator/repo_cleanup.py new file mode 100644 index 00000000..083319c2 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/repo_cleanup.py @@ -0,0 +1,300 @@ +"""任务级 kernel 仓库清理（harness_evolve / AHE）。 + +背景 +---- +AHE 每轮为每道题物化一个独立的候选 kernel 仓库，命名规则见 +``harness_evolve/orchestrator/adapters/eval.py``:: + + child_id = _slug(f"ahe-{cfg.task_id}-it{iteration:03d}-candidate-{inst.id}-{suffix}") + +因此 ``ahe--it-...`` 的前缀可以**精确**定位"某任务拥有的仓库"。 +一个跑满 N 轮的 AHE 任务会留下 约 4N 个仓库（4 = 一轮题数/卡数）， +而 ``sys_shell`` 里原有的 ``_purge_owned_kernel_repo`` 只删**单个**仓库， +且 task_type 门禁写死为 ``dcu-kernel-auto-opt``，对 HE 完全不生效 —— 于是 +任务删除后这些仓库永久残留。 + +安全设计（三层） +---------------- +1. **前缀精确匹配**：只处理 ``ahe-{task_id}-it`` 开头的目录，天然不会碰到 + 人工维护的 ``/root/zth_agent/kernel-repos`` 或别的任务的仓库。 +2. **池白名单**：``registered_pool.yaml`` 中 ``source_report`` 引用的仓库 + 一律保留 —— 即便前缀匹配上了也绝不删。 +3. **路径校验**：跳过符号链接、跳过非本目录直接子项、解析后必须仍在 repo_root 内。 + +默认 ``dry_run=True``：只报告，不删除。 +""" + +from __future__ import annotations + +import re +import shutil +from pathlib import Path +from typing import Any, Dict, List, Optional + +DEFAULT_AHE_REPO_ROOT = Path("/root/zth_agent/ahe-kernel-repos") +_REGISTERED_POOL_NAME = "registered_pool.yaml" + +# 匹配仓库名本身：形如 ``ahe-{task_id}-it{NNN}-candidate-...`` +# （见 eval.py 的 ``_slug(f"ahe-{task_id}-it{iteration:03d}-candidate-...")``）。 +# +# 只锚定名称模式、不要求前后是 '/' 或引号：池文件里同一个仓库既可能写成 +# ``.../ahe-xxx-it001-.../final_report.json``（后面跟 '/'），也可能写成 +# ``.../ahe-xxx-it001-...``（行尾）或 JSON 里的 ``"repo_path": ".../ahe-xxx-it001-..."``， +# 加边界锚点会漏掉其中一种。``-it<数字>`` 这个标记本身已足够特异： +# 它既不会命中仓库根目录 ``ahe-kernel-repos``，也不会命中 ``kernel-repos`` 下 +# 人工维护的仓库（那些名字里没有 ahe- 前缀）。 +_REPO_REF_RE = re.compile(r"(ahe-[A-Za-z0-9._-]*?-it\d+[A-Za-z0-9._-]*)") + + +def resolve_ahe_repo_root(answers: Optional[Dict[str, Any]] = None) -> Path: + """与 eval.py 的解析保持一致：answers['ahe_repo_root'] 优先，否则默认值。""" + raw = None + if isinstance(answers, dict): + raw = answers.get("ahe_repo_root") + base = Path(str(raw or DEFAULT_AHE_REPO_ROOT)).expanduser() + try: + return base.resolve() + except OSError: + return base + + +def repo_name_prefix(task_id: str) -> str: + """该任务拥有的仓库名前缀。结尾的 'it' 是关键：避免 task_id 互为前缀时误伤。""" + return f"ahe-{task_id}-it" + + +def load_pool_keep_list(repo_root: Path) -> set[str]: + """从 registered_pool.yaml 提取所有被引用的仓库目录名（白名单）。""" + keep: set[str] = set() + pool = repo_root / _REGISTERED_POOL_NAME + if not pool.is_file(): + return keep + try: + text = pool.read_text(encoding="utf-8", errors="replace") + except OSError: + return keep + for name in _REPO_REF_RE.findall(text): + name = name.strip() + if name and name != _REGISTERED_POOL_NAME: + keep.add(name) + return keep + + +def is_task_registered(task_id: str) -> bool: + """该 task_id 是否仍在 WebUI 注册表中（即任务尚未被删除）。 + + 注册表位置与 :mod:`metainfer.server.tasks` 一致： + ``/.metainfer/registry.json``。任一节点目录命中即视为仍注册。 + """ + import json as _json + root = _meta_root() + if root is None: + return False + try: + candidates = sorted(root.glob("nodes/*/.metainfer/registry.json")) + except OSError: + return False + for reg in candidates: + try: + data = _json.loads(reg.read_text(encoding="utf-8")) + except (ValueError, OSError): + continue + for t in data.get("tasks", []) or []: + if t.get("id") == task_id: + return True + return False + + +def _meta_root() -> Optional[Path]: + """定位 MetaInfer 仓库根（含 nodes/ 的那一层）。""" + # 本文件位于 /metainfer/tasks/harness_evolve/orchestrator/repo_cleanup.py + here = Path(__file__).resolve() + for parent in here.parents: + if (parent / "metainfer").is_dir() and (parent / "nodes").is_dir(): + return parent + return None + + +def owned_repos( + task_id: str, + repo_root: Path, + keep: Optional[set[str]] = None, +) -> tuple[List[Path], List[str]]: + """返回 (可删除仓库列表, 被保留的原因说明)。 + + 只做识别与安全检查，不产生任何副作用 —— 便于 dry-run 与测试。 + """ + keep = keep if keep is not None else load_pool_keep_list(repo_root) + prefix = repo_name_prefix(task_id) + deletable: List[Path] = [] + skipped: List[str] = [] + if not repo_root.is_dir(): + return deletable, [f"repo_root 不存在: {repo_root}"] + try: + entries = sorted(repo_root.iterdir()) + except OSError as exc: + return deletable, [f"无法读取 repo_root: {exc}"] + + for p in entries: + if not p.name.startswith(prefix): + continue + # 保护 1：池引用 + if p.name in keep: + skipped.append(f"{p.name} (被 registered_pool.yaml 引用，保留)") + continue + # 保护 2：只删真实目录，不碰符号链接 + if p.is_symlink(): + skipped.append(f"{p.name} (符号链接，跳过)") + continue + if not p.is_dir(): + skipped.append(f"{p.name} (非目录，跳过)") + continue + # 保护 3：解析后必须仍在 repo_root 内 + try: + resolved = p.resolve() + except OSError: + skipped.append(f"{p.name} (无法解析路径，跳过)") + continue + if resolved.parent != repo_root: + skipped.append(f"{p.name} (解析后越出 repo_root，跳过)") + continue + deletable.append(p) + return deletable, skipped + + +def purge_task_kernel_repos( + task_id: str, + repo_root: Path, + keep: Optional[set[str]] = None, + dry_run: bool = True, + force: bool = False, +) -> Dict[str, Any]: + """删除某 HE 任务拥有的候选 kernel 仓库。 + + :param dry_run: True 时只统计，不做任何删除（默认）。 + :param force: 任务**仍在注册表**中时是否照删。默认 False —— 即"任务还没被 + 删除就拒绝清理它的仓库"，避免误伤在跑/待恢复的实验。 + :returns: 摘要字典，含 matched/deleted/skipped/bytes 等。 + """ + deletable, skipped = owned_repos(task_id, repo_root, keep) + + # 保护 0（最重要）：任务仍在注册表 → 其仓库可能被在跑或待恢复的实验使用 + if not force and is_task_registered(task_id): + return { + "task_id": task_id, + "repo_root": str(repo_root), + "prefix": repo_name_prefix(task_id), + "dry_run": dry_run, + "matched": len(deletable), + "deleted": [], + "failed": [], + "skipped": skipped + [ + f"任务 {task_id} 仍在注册表中（未被删除）；" + "如确认要删，请加 --force / force=True" + ], + "bytes": 0, + "bytes_human": "0B", + "refused": "task_still_registered", + } + + # 统计体积（删除前算，删后无从得知） + bytes_total = 0 + sizes: Dict[str, int] = {} + for p in deletable: + sz = 0 + for sub in p.rglob("*"): + try: + if sub.is_file() and not sub.is_symlink(): + sz += sub.stat().st_size + except OSError: + continue + sizes[p.name] = sz + bytes_total += sz + + deleted: List[str] = [] + failed: List[str] = [] + if not dry_run: + for p in deletable: + try: + shutil.rmtree(p) + deleted.append(p.name) + except OSError as exc: + failed.append(f"{p.name}: {exc}") + + return { + "task_id": task_id, + "repo_root": str(repo_root), + "prefix": repo_name_prefix(task_id), + "dry_run": dry_run, + "matched": len(deletable), + "deleted": deleted, + "failed": failed, + "skipped": skipped, + "bytes": bytes_total, + "bytes_human": _human(bytes_total), + } + + +def _human(n: int) -> str: + step = float(n) + for unit in ("B", "K", "M", "G", "T"): + if step < 1024 or unit == "T": + return f"{step:.1f}{unit}" + step /= 1024 + return f"{step:.1f}T" + + +def main(argv: Optional[List[str]] = None) -> int: + import argparse + import json + + ap = argparse.ArgumentParser( + description="清理某个 harness_evolve 任务拥有的候选 kernel 仓库（默认 dry-run）", + ) + ap.add_argument("task_id", help="HE 任务的 task_id，例如 test-9-8-5-e10567a1") + ap.add_argument("--repo-root", default=None, + help=f"AHE 仓库根，默认 {DEFAULT_AHE_REPO_ROOT}") + ap.add_argument("--apply", action="store_true", + help="真正执行删除（不加此参数只做 dry-run）") + ap.add_argument("--force", action="store_true", + help="任务仍在注册表中时也照删（危险，默认拒绝）") + ap.add_argument("--json", action="store_true", help="以 JSON 输出") + args = ap.parse_args(argv) + + repo_root = Path(args.repo_root).expanduser() if args.repo_root \ + else DEFAULT_AHE_REPO_ROOT + summary = purge_task_kernel_repos( + args.task_id, repo_root, dry_run=not args.apply, force=args.force, + ) + + if args.json: + print(json.dumps(summary, ensure_ascii=False, indent=2)) + return 0 if not summary["failed"] else 1 + + if summary.get("refused"): + print(f"[拒绝清理] task_id={summary['task_id']}") + print(f" {summary['skipped'][-1]}") + return 2 + + mode = "DRY-RUN（未删除任何东西）" if summary["dry_run"] else "已执行删除" + print(f"[{mode}] task_id={summary['task_id']}") + print(f" repo_root : {summary['repo_root']}") + print(f" 匹配前缀 : {summary['prefix']}") + print(f" 命中仓库 : {summary['matched']} 个，合计 {summary['bytes_human']}") + if summary["skipped"]: + print(f" 受保护跳过: {len(summary['skipped'])} 个") + for s in summary["skipped"][:10]: + print(f" - {s}") + if summary["deleted"]: + print(f" 已删除 : {len(summary['deleted'])} 个") + if summary["failed"]: + print(f" 删除失败 : {len(summary['failed'])} 个") + for f in summary["failed"][:10]: + print(f" - {f}") + if summary["dry_run"] and summary["matched"]: + print("\n 确认无误后，加 --apply 执行删除。") + return 0 if not summary["failed"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/metainfer/tasks/harness_evolve/orchestrator/round_plan.py b/metainfer/tasks/harness_evolve/orchestrator/round_plan.py new file mode 100644 index 00000000..de02566c --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/round_plan.py @@ -0,0 +1,209 @@ +"""round_plan: per-round question set chosen by the single Evolve role. + +The evolve step produces BOTH the harness change and the next round's +``round_plan.json`` (strategy / rationale / selected / weights). Machine +guardrails in this module keep agent-chosen sets comparable and in-pool: + - all ids must be in the pool; + - overlap with the previous round's scored set >= 60% (auto back-fill); + - budget cap enforced. +Decisions (flip / attribution / best-ever / rollback) are made on the overlap +pairs only (see docs/dkao_suite_protocol_v2.md). +""" + +from __future__ import annotations + +import json +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional + +from .pool import Pool + +MIN_OVERLAP_RATIO = 0.6 +VALID_STRATEGIES = ("explore", "exploit", "balanced") + + +@dataclass +class RoundPlan: + iteration: int + strategy: str = "balanced" + rationale: str = "" + selected: List[str] = field(default_factory=list) + weights: Dict[str, float] = field(default_factory=dict) + + def to_dict(self) -> Dict[str, Any]: + return { + "iteration": self.iteration, + "strategy": self.strategy, + "rationale": self.rationale, + "selected": self.selected, + "weights": self.weights, + } + + @classmethod + def from_dict(cls, d: Dict[str, Any]) -> "RoundPlan": + return cls( + iteration=int(d.get("iteration", 0)), + strategy=str(d.get("strategy") or "balanced"), + rationale=str(d.get("rationale") or ""), + selected=[str(x) for x in (d.get("selected") or [])], + weights={ + str(k): float(v) for k, v in (d.get("weights") or {}).items() + }, + ) + + +def load_round_plan(path: Path) -> Optional[RoundPlan]: + if not path.is_file(): + return None + try: + return RoundPlan.from_dict( + json.loads(path.read_text(encoding="utf-8")) + ) + except (OSError, ValueError): + return None + + +def validate_in_pool(plan: RoundPlan, pool: Pool) -> None: + if plan.strategy not in VALID_STRATEGIES: + raise ValueError( + f"strategy {plan.strategy!r} not in {VALID_STRATEGIES}" + ) + pool.assert_in_pool(plan.selected) + if len(set(plan.selected)) != len(plan.selected): + raise ValueError("round_plan.selected contains duplicates") + + +def _top_up_anchors( + picked: List[str], + seen: set, + pool: Pool, + budget: int, + anchor_min: int, +) -> List[str]: + """Guarantee at least ``anchor_min`` instances WITH accepted history. + + Cold (baseline-only) shapes are allowed as questions, but every round must + carry >= anchor_min historical anchors so family tau stays anchored and + results stay comparable. Replaces trailing non-anchor picks if needed. + """ + anchor_set = set(pool.historical_ids()) + have = sum(1 for i in picked if i in anchor_set) + if have >= anchor_min: + return picked + for aid in pool.historical_ids(): + if have >= anchor_min: + break + if aid in seen: + continue + # make room by dropping the last non-anchor pick (never drop an anchor) + drop_idx = next( + (i for i in range(len(picked) - 1, -1, -1) + if picked[i] not in anchor_set), + None, + ) + if drop_idx is not None: + old = picked.pop(drop_idx) + seen.discard(old) + picked.append(aid) + seen.add(aid) + have += 1 + if len(picked) > budget: # safety; budget >= anchor_min is required + picked = picked[:budget] + if have < anchor_min: + raise ValueError( + f"pool has fewer than {anchor_min} historical anchors; " + "cannot satisfy the anchor rule" + ) + return picked + + +def enforce_guardrails( + prev_selected: List[str], + plan: Optional[RoundPlan], + pool: Pool, + budget: int, + anchor_min: int = 2, +) -> List[str]: + """Return the final scored set for the next round. + + Guardrails: + - budget must be >= anchor_min; + - plan None / empty -> stratified default sample; + - otherwise start from plan.selected (in pool); + - overlap with previous scored set >= MIN_OVERLAP_RATIO (auto back-fill); + - >= anchor_min instances must carry accepted history (anchor rule); + - every id stays in the pool; total capped at budget. + """ + budget = int(budget) + if budget < anchor_min: + raise ValueError(f"budget {budget} < anchor_min {anchor_min}") + if plan is None or not plan.selected: + picked = pool.stratified_sample(budget) + return _top_up_anchors(picked, set(picked), pool, budget, anchor_min)[:budget] + + validate_in_pool(plan, pool) + picked: List[str] = [] + seen: set = set() + for iid in plan.selected: + if iid not in seen: + picked.append(iid) + seen.add(iid) + if len(picked) >= budget: + break + + prev = [i for i in prev_selected if i in pool.instances] + if prev: + overlap = len(set(picked) & set(prev)) + need = int(MIN_OVERLAP_RATIO * len(prev)) + for iid in prev: + if overlap >= need or len(picked) >= budget: + break + if iid not in seen: + picked.append(iid) + seen.add(iid) + overlap += 1 + + if len(picked) < budget: + for iid in pool.stratified_sample(budget): + if len(picked) >= budget: + break + if iid not in seen: + picked.append(iid) + seen.add(iid) + return _top_up_anchors(picked, seen, pool, budget, anchor_min)[:budget] + + +ROTATION_FILE = "pool_rotation.json" + + +def load_rotation(exp_dir: Path) -> Dict[str, int]: + """Per-question last-selected round (sidecar; the pool yaml stays read-only).""" + path = Path(exp_dir) / ROTATION_FILE + try: + data = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {} + if not isinstance(data, dict): + return {} + out: Dict[str, int] = {} + for key, value in data.items(): + try: + out[str(key)] = int(value) + except (TypeError, ValueError): + continue + return out + + +def mark_selected(exp_dir: Path, ids: List[str], iteration: int) -> Dict[str, int]: + """Record that ``ids`` were examined in ``iteration`` (rotation fairness).""" + rotation = load_rotation(exp_dir) + for iid in ids: + rotation[str(iid)] = int(iteration) + try: + (Path(exp_dir) / ROTATION_FILE).write_text( + json.dumps(rotation, ensure_ascii=False, indent=2, sort_keys=True), + encoding="utf-8") + except OSError: + pass + return rotation diff --git a/metainfer/tasks/harness_evolve/orchestrator/rounds.py b/metainfer/tasks/harness_evolve/orchestrator/rounds.py new file mode 100644 index 00000000..5df5d13e --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/rounds.py @@ -0,0 +1,481 @@ +"""Round question selection and gate evaluation for harness_evolve. + +One HE run chases one deliverable: a harness version that can be promoted into +DKAO. The round structure is fixed by the operator: + +* **Round 1 is a baseline round.** It draws 4 operators at random from the + question pool, has DKAO measure them, and **judges nothing**: no gate is + evaluated and nothing is promoted. The kernels that beat their variant go + straight into the variant pool (that is a kernel fact, not a harness + verdict). These 4 operators become the fixed **generalization paper** + (``generalization_ids``) for the rest of the run. +* **Every later round** draws a fresh random set of 4 operators (never reusing + the paper) and judges **only the performance gate**. +* When the performance gate passes, the *same* harness is handed to DKAO again + with the paper's operators — that retake is the **generalization gate**, and + it counts as half a round (``GENERALIZATION_ROUND_COST``). +* Both gates are counter rules against the variant: the performance gate needs + >= 75% wins, the generalization gate >= 50% wins, and in both cases a + non-winning operator may not fall below 80% of its variant. + +Because a rejected candidate never reaches production, each round measures +against the same reference: "is this harness better than what the variant pool +currently holds". +""" + +from __future__ import annotations + +import json +import math +import random +import time +from pathlib import Path +from typing import Any, Dict, Iterable, List, Optional, Sequence + +from .decision_engine import ( + GENERALIZATION_GATE_WIN_RATIO, PERFORMANCE_GATE_WIN_RATIO, + gate_verdict, +) +from .variant import pool_instances, reference_results, resolve_variant + +#: Operators per gate. The machine exposes 4 HCUs, so every round is sized in +#: multiples of 4: that keeps a wave able to fill all four devices at once and +#: keeps the two gates symmetric (the paper is one wave too). +OPERATORS_PER_GATE = 4 +GPU_COUNT = 4 + +#: A round is split into two stages. ``performance`` hands a fresh operator set +#: to DKAO and judges only the performance gate; when that passes, the very same +#: harness is handed to DKAO a second time with the frozen paper — that retake +#: is the ``generalization`` stage. Round 1 is neither: it is a *baseline* round +#: that measures the paper and judges nothing. +STAGE_BASELINE = "baseline" +STAGE_PERFORMANCE = "performance" +STAGE_GENERALIZATION = "generalization" + +#: What each stage costs against the round budget. A normal round is 1.0; the +#: generalization retake is half a round (the operator asked for "iteration +#: 2.5"), because it re-runs a fixed, already-chosen paper instead of opening a +#: new round of the search. +PERFORMANCE_ROUND_COST = 1.0 +GENERALIZATION_ROUND_COST = 0.5 + +#: The frozen "retake the paper with this harness" request. +PENDING_GENERALIZATION_FILE = "pending_generalization.json" + +#: Directory holding one evaluation attempt's DKAO children +#: (``children//``). Each attempt -- a first run and +#: every environment retry of it, and the generalization retake -- gets its own, +#: so no two DKAO runs ever share a child workspace. +CHILDREN_DIR = "children" + + +def round_stage(*, iteration: int, pending: Optional[Dict[str, Any]]) -> str: + """Which stage this pass through the loop is. + + A pending generalization retake wins over the round number: the loop may + come back to the same iteration to retake the paper, and that pass is the + ``generalization`` stage even though its directory is ``iteration_002``. + """ + if pending: + return STAGE_GENERALIZATION + return STAGE_BASELINE if int(iteration) <= 1 else STAGE_PERFORMANCE + + +def iteration_of_dir_name(name: str) -> int: + """The iteration a ``children/`` directory belongs to (0 if none). + + Names are ``iteration_NNN`` plus optional suffixes (``_a2`` for a retry, + ``_generalization`` for the retake). The iteration is always the second + underscore-separated field, so it is read by position: a regex would also + match the attempt counter and silently merge two rounds' children. + """ + parts = str(name).split("_") + if len(parts) < 2 or parts[0] != "iteration": + return 0 + try: + return int(parts[1]) + except ValueError: + return 0 + + +def stage_dir_name(stage: str, iteration: int, *, attempt: int = 1) -> str: + """Directory under ``/children`` holding one evaluation attempt. + + Every attempt gets its own directory. A retry of one question is a *new* + DKAO run, and DKAO binds the child workspace's ``main`` symlink to the exact + kernel repo it was told to use: reusing the directory makes the retry die in + seconds with "workspace main already points to ..., not requested ...", + which turns a transient environment hiccup into an apparently permanent + failure that burns the whole retry budget. A fresh directory also keeps each + attempt's evidence instead of overwriting the one that failed. + """ + name = f"iteration_{int(iteration):03d}" + if int(attempt) > 1: + name += f"_a{int(attempt)}" + if stage == STAGE_GENERALIZATION: + name += "_generalization" + return name + + +def attempt_dirs(children_root: Path, iteration: int) -> List[Path]: + """Every attempt directory of one iteration, oldest attempt first. + + The retake sorts after the performance pass it belongs to (both carry + ``_generalization`` in the name only for the retake's own directory). + """ + root = Path(children_root) + if not root.is_dir(): + return [] + found = [d for d in root.iterdir() + if d.is_dir() and iteration_of_dir_name(d.name) == int(iteration)] + return sorted(found, key=lambda d: (d.name.count("_a"), + "_generalization" in d.name, d.name)) + + +def resolve_child_dir(children_root: Path, iteration: int, + question_id: str) -> Optional[Path]: + """The directory holding one question's current attempt (newest wins). + + ``None`` means the question has no child in this iteration at all: returning + a path that does not exist would make "not measured" look like "measured, + directory missing" to every caller. + """ + found = [d / str(question_id) for d in attempt_dirs(children_root, iteration) + if (d / str(question_id)).is_dir()] + return found[-1] if found else None + + +def child_dir_map(children_root: Path, iteration: int) -> Dict[str, Path]: + """``{question_id: directory}`` for one iteration, newest attempt wins.""" + out: Dict[str, Path] = {} + for attempt_dir in attempt_dirs(children_root, iteration): + for qid in sorted(p.name for p in attempt_dir.iterdir() if p.is_dir()): + out[qid] = attempt_dir / qid + return out + + +def stage_seconds(pending: Optional[Dict[str, Any]]) -> int: + """Stable suffix making a retake's child ids distinct from the first pass.""" + if not pending: + return 0 + try: + return int(pending.get("frozen_at") or 0) + except (TypeError, ValueError): + return 0 + + +def device_for_index(index: int) -> int: + """The device a question at ``index`` in a round is handed to. + + A fixed rotation, not a reading of the cards: ``harness_evolve`` does not + manage GPU occupancy (DKAO's own admission gate does), so the assignment + only has to be a stable, even layout — N questions land as N / 4 sequential + waves with one question per device in each. + """ + return int(index) % GPU_COUNT + + +def normalized_per_gate(value: Any, default: int = OPERATORS_PER_GATE) -> int: + """Round the requested question count to a non-zero multiple of 4. + + A count that is not a multiple of 4 would leave devices idle in a wave, so + it is rounded up to the next multiple (and anything unusable falls back to + the default). + """ + try: + count = int(value) + except (TypeError, ValueError): + return default + if count <= 0: + return default + if count % GPU_COUNT: + count = ((count // GPU_COUNT) + 1) * GPU_COUNT + return count + + +#: An operator is "decode" at small M and "prefill" at large M, matching the +#: family naming the pool uses (``decode__o_proj`` / ``prefill__o_proj``). +DECODE_MAX_M = 32 +REGIME_DECODE = "decode" +REGIME_PREFILL = "prefill" + + +def regime_of(entry: Dict[str, Any]) -> str: + """Which regime an operator belongs to (family first, shape as fallback).""" + family = str(entry.get("family") or "") + if family.startswith(REGIME_DECODE + "__"): + return REGIME_DECODE + if family.startswith(REGIME_PREFILL + "__"): + return REGIME_PREFILL + contract = entry.get("contract") or {} + try: + m = int(contract.get("M") or 0) + except (TypeError, ValueError): + m = 0 + return REGIME_DECODE if 0 < m <= DECODE_MAX_M else REGIME_PREFILL + + +def sample_operators(instances: Dict[str, Dict[str, Any]], *, + table: Optional[Dict[str, Any]] = None, + count: int = OPERATORS_PER_GATE, + exclude: Iterable[str] = (), + rng: Optional[random.Random] = None, + require_judgeable: bool = True, + require_regimes: bool = True, + ) -> List[str]: + """A random operator sample, skipping ids that cannot be judged. + + ``require_judgeable`` keeps out operators with neither a variant nor a fixed + baseline: those could not produce a win/loss, so they would only dilute the + ratio. + + ``require_regimes`` guarantees the sample spans both regimes (at least one + decode and at least one prefill) so a round is never judged on a single + shape class; the rest of the sample is filled at random from everything + that is left. + """ + excluded = {str(x) for x in exclude} + available = [iid for iid in instances if iid not in excluded] + if require_judgeable: + available = [iid for iid in available + if resolve_variant(table or {}, iid, + instances=instances) is not None] + if not available: + return [] + picker = rng or random + count = max(1, min(int(count), len(available))) + picked: List[str] = [] + if require_regimes: + for regime in (REGIME_DECODE, REGIME_PREFILL): + members = [iid for iid in available + if regime_of(instances[iid]) == regime + and iid not in picked] + if members and len(picked) < count: + picked.append(picker.choice(sorted(members))) + rest = [iid for iid in available if iid not in picked] + picker.shuffle(rest) + picked.extend(rest[:max(0, count - len(picked))]) + return sorted(picked[:count]) + + +def _statement_groups(state: Dict[str, Any]) -> Dict[str, Any]: + """The persisted groups, whichever shape the caller handed over. + + The pipeline passes what :func:`group_state` returned (the groups dict + itself); older callers wrap it as ``{"groups": ...}``. Reading only one of + the two shapes silently loses the frozen paper — and a lost paper means the + performance draw may hand the generalization exam's own operators back. + """ + if not isinstance(state, dict): + return {} + nested = state.get("groups") + if isinstance(nested, dict): + return dict(nested) + if "generalization_ids" in state: + return dict(state) + return {} + + +def select_round_questions(*, iteration: int, + table: Dict[str, Any], + instances: Dict[str, Dict[str, Any]], + state: Dict[str, Any], + count: int = OPERATORS_PER_GATE, + rng: Optional[random.Random] = None, + pending: Optional[Dict[str, Any]] = None, + ) -> Dict[str, Any]: + """Which operators this pass measures, and in which role. + + Returns ``{"performance_ids", "generalization_ids", "defines_paper", + "stage"}``. + + * ``pending`` set → this pass is the **generalization retake**: it measures + exactly the paper, again, and judges the generalization gate. No new + operators are drawn. + * otherwise round 1 → the **baseline** round: it draws the paper (which + becomes the fixed exam) and measures it, judging nothing. + * otherwise → **performance** stage: a fresh random draw excluding the + paper; the paper comes back only if this pass wins and the retake runs. + """ + groups = _statement_groups(state) + paper = list(groups.get("generalization_ids") or []) + if pending: + ids = [str(i) for i in (pending.get("operator_ids") or paper)] + return {"performance_ids": ids, "generalization_ids": ids, + "defines_paper": False, "stage": STAGE_GENERALIZATION} + if iteration <= 1 or not paper: + # Round 1 defines the generalization paper and measures it, but it is a + # baseline round: nothing is judged and nothing is promoted from it. + draw = sample_operators(instances, table=table, count=count, rng=rng) + return {"performance_ids": draw, "generalization_ids": list(draw), + "defines_paper": True, "stage": STAGE_BASELINE} + performance = sample_operators(instances, table=table, count=count, + exclude=paper, rng=rng) + return {"performance_ids": performance, + "generalization_ids": paper, "defines_paper": False, + "stage": STAGE_PERFORMANCE} + + +def _load_json(path: Path) -> Any: + try: + return json.loads(Path(path).read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + +def read_pending_generalization(exp_dir: Path) -> Optional[Dict[str, Any]]: + """The frozen retake request, if the last performance stage passed. + + While this file exists, the next pass through the loop is the + generalization stage: the same harness bytes go to DKAO again, with the + paper's operators. Nothing else is decided until that retake is judged. + """ + data = _load_json(Path(exp_dir) / PENDING_GENERALIZATION_FILE) + return data if isinstance(data, dict) else None + + +def write_pending_generalization(exp_dir: Path, *, + iteration: int, + harness_revision: str, + snapshot_dir: Optional[str], + operator_ids: Sequence[str], + performance_gate: Dict[str, Any], + ) -> Dict[str, Any]: + """Freeze "retake the paper with this harness" for the next pass.""" + exp_dir = Path(exp_dir) + payload = { + "schema": "he-pending-generalization/1", + "iteration": int(iteration), + "operator_ids": [str(i) for i in operator_ids], + "harness_revision": str(harness_revision or ""), + "snapshot_dir": str(snapshot_dir) if snapshot_dir else None, + "performance_wins": list(performance_gate.get("wins") or []), + "performance_gate_status": performance_gate.get("status"), + "frozen_at": time.time(), + } + exp_dir.mkdir(parents=True, exist_ok=True) + path = exp_dir / PENDING_GENERALIZATION_FILE + tmp = path.with_suffix(".json.tmp") + tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2, + sort_keys=True), encoding="utf-8") + tmp.replace(path) + return payload + + +def clear_pending_generalization(exp_dir: Path) -> None: + try: + (Path(exp_dir) / PENDING_GENERALIZATION_FILE).unlink() + except OSError: + pass + + +def evaluate_gate(*, kind: str, operator_ids: Sequence[str], + results: Dict[str, Any], + table: Dict[str, Any], + instances: Dict[str, Dict[str, Any]], + ) -> Dict[str, Any]: + """Judge one gate's operator set against the variants they must beat.""" + ids = [iid for iid in operator_ids] + reference = reference_results(table, ids, instances=instances) + candidate = {iid: results[iid] for iid in ids if iid in results} + verdict = gate_verdict(reference, candidate, kind=kind) + verdict["iteration"] = None + verdict["requested_ids"] = ids + verdict["requested"] = len(ids) + missing = [iid for iid in ids if iid not in results] + if missing: + unmeasured = sorted(set(verdict["unmeasured"]) | set(missing)) + verdict["unmeasured"] = unmeasured + verdict["reasons"] = list(verdict["reasons"]) + [ + "not measured this round: " + ", ".join(sorted(missing))] + # A gate is a statement about the harness, so it may only be made when + # the round actually produced the evidence. Two cases are decided here: + # + # * the measured part already loses — a question below the floor, or so + # few wins that even if every missing question won, the count would + # not reach the required wins. That is a FAIL, and saying so now + # saves the round a relaunch; + # * otherwise the outcome is open: the harness may be fine and the + # machine may have eaten a measurement. Marking that FAIL blames the + # harness for a missing number (and, on the generalization retake, + # spends the candidate on it), so the round carries no verdict and is + # re-measured instead. + needed_total = (max(1, min(len(ids), + math.ceil(len(ids) * gate_ratio(kind)))) + if ids else 0) + verdict["wins_needed_over_requested"] = needed_total + possible_wins = len(verdict["wins"]) + len(unmeasured) + if verdict["below_floor"] or possible_wins < needed_total: + verdict["status"] = "FAIL" + verdict["reasons"] = list(verdict["reasons"]) + [ + "gate cannot pass even if every unmeasured question won " + f"({possible_wins} possible win(s) vs {needed_total} needed)"] + else: + verdict["status"] = "INCOMPLETE" + return verdict + + +def group_state(table: Dict[str, Any]) -> Dict[str, Any]: + """The persisted groups (the fixed generalization paper lives here).""" + groups = table.get("groups") + return dict(groups) if isinstance(groups, dict) else {} + + +def record_paper(table: Dict[str, Any], *, iteration: int, + generalization_ids: Sequence[str], + performance_ids: Sequence[str]) -> Dict[str, Any]: + """Persist this run's fixed generalization paper on the variant table. + + The paper is written once and never replaced: it is the run's frozen exam, + so a later round must not be able to move the goalposts. + """ + groups = group_state(table) + groups.setdefault("generalization_ids", list(generalization_ids)) + groups.setdefault("generalization_defined_at", iteration) + groups["last_performance_ids"] = list(performance_ids) + return groups + + +def baseline_round_verdict(results: Dict[str, Any], + operator_ids: Sequence[str]) -> Dict[str, Any]: + """Whether the baseline round produced usable evidence for every operator. + + Round 1 judges no gate, so there is no win ratio to check — but it still + owes one usable measurement per operator: the kernels that beat their + variant are what seeds the pool for every later round to beat, and a paper + that was never really measured cannot serve as the run's exam. + """ + ids = [str(i) for i in operator_ids] + unusable: List[str] = [] + for iid in ids: + row = results.get(iid) + if not isinstance(row, dict): + unusable.append(iid) + continue + median = row.get("median_us") + try: + value = float(median) + except (TypeError, ValueError): + unusable.append(iid) + continue + if value <= 0 or row.get("correctness_ok") is False: + unusable.append(iid) + return { + "status": "PASS" if not unusable else "FAIL", + "operators": ids, + "measured": sorted(set(ids) - set(unusable)), + "failed": sorted(unusable), + "reasons": ([] if not unusable else + ["baseline round produced no usable measurement for: " + + ", ".join(sorted(unusable))]), + } + + +def gate_ratio(kind: str) -> float: + return (PERFORMANCE_GATE_WIN_RATIO if kind == "performance" + else GENERALIZATION_GATE_WIN_RATIO) + + +def paper_path(exp_dir: Path) -> Path: + return Path(exp_dir) / "generalization_paper.json" diff --git a/metainfer/tasks/harness_evolve/orchestrator/state.py b/metainfer/tasks/harness_evolve/orchestrator/state.py new file mode 100644 index 00000000..3ceb55cf --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/state.py @@ -0,0 +1,195 @@ +"""Experiment state recording for harness_evolve (file-system-as-database). + +The WebUI launcher runs the HE orchestrator as a bare ``cli run`` subprocess, +so nothing outside the pipeline would otherwise write the per-task +``run.json`` / ``timeline.jsonl`` that the shell reads for the phase graph +(``state_reader.read_run`` / ``read_timeline``). This module gives the +pipeline a small writer that keeps those files in the same shape DKAO writes: +``run.json`` fields merged with ``state_reader`` defaults, timeline rows as +``{"ts": float, "type": str, "payload": dict}``. +""" + +from __future__ import annotations + +import json +import os +import time +from pathlib import Path +from typing import Any, Dict, List, Optional + +_RUN_DEFAULTS: Dict[str, Any] = { + "task_id": None, + "task_type": None, + "created_at": 0, + "current_iteration": 0, + "current_phase": "idle", + "last_update": 0, + "finished": False, + "final_status": None, + "last_outcome": None, + "last_transition_label": None, + "notes": [], +} + + +def _load_json(path: Path) -> Optional[Dict[str, Any]]: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + return value if isinstance(value, dict) else None + + +def read_run(state_dir: Path) -> Dict[str, Any]: + data = _load_json(Path(state_dir) / "run.json") + if data is None: + return dict(_RUN_DEFAULTS) + return {**_RUN_DEFAULTS, **data} + + +def set_run(state_dir: Path, **fields: Any) -> Dict[str, Any]: + """Merge fields into run.json (thread-safe via atomic replace).""" + state_dir = Path(state_dir) + state_dir.mkdir(parents=True, exist_ok=True) + path = state_dir / "run.json" + run = read_run(state_dir) + run.update(fields) + run["last_update"] = time.time() + _atomic_write_json(path, run) + return run + + +def append_timeline(state_dir: Path, event_type: str, + payload: Optional[Dict[str, Any]] = None) -> None: + state_dir = Path(state_dir) + state_dir.mkdir(parents=True, exist_ok=True) + path = state_dir / "timeline.jsonl" + row = { + "ts": time.time(), + "type": event_type, + "payload": payload or {}, + } + with path.open("a", encoding="utf-8") as fh: + fh.write(json.dumps(row, ensure_ascii=False) + "\n") + + +def _atomic_write_json(path: Path, data: Dict[str, Any]) -> None: + tmp = path.with_suffix(".json.tmp") + tmp.write_text(json.dumps(data, ensure_ascii=False, indent=2), + encoding="utf-8") + os.replace(tmp, path) + + +def last_phase_event(state_dir: Path) -> Optional[Dict[str, Any]]: + """Most recent timeline entry (used by tests / diagnostics).""" + path = Path(state_dir) / "timeline.jsonl" + if not path.is_file(): + return None + last: Optional[Dict[str, Any]] = None + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict): + last = row + return last + + +def list_timeline(state_dir: Path) -> List[Dict[str, Any]]: + out: List[Dict[str, Any]] = [] + path = Path(state_dir) / "timeline.jsonl" + if not path.is_file(): + return out + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + try: + row = json.loads(line) + except ValueError: + continue + if isinstance(row, dict): + out.append(row) + return out + + +#: Where a running process records the harness code it actually loaded, so a +#: later round can notice the files on disk moved on without it. +REVISION_FILE = "harness_revision.json" + + +def harness_revision() -> str: + """Fingerprint of the harness_evolve *code* (not the experiment workspace). + + Rounds run for hours, so an operator who fixes the loop cannot expect the + running process to pick the change up. This is how the next round notices + that a restart is required instead of silently running stale logic. + """ + import hashlib + + root = Path(__file__).resolve().parent + digest = hashlib.sha256() + for path in sorted(root.rglob("*.py")): + try: + data = path.read_bytes() + except OSError: + continue + digest.update(path.relative_to(root).as_posix().encode("utf-8")) + digest.update(b"\0") + digest.update(data) + digest.update(b"\0") + return digest.hexdigest()[:16] + + +def write_harness_revision(state_dir: Path, + revision: Optional[str] = None) -> str: + """Record the revision this process actually loaded.""" + value = str(revision or harness_revision()) + state_dir = Path(state_dir) + state_dir.mkdir(parents=True, exist_ok=True) + _atomic_write_json(state_dir / REVISION_FILE, { + "revision": value, "pid": os.getpid(), "started_at": time.time()}) + return value + + +def loaded_harness_revision(state_dir: Path) -> Optional[str]: + data = _load_json(Path(state_dir) / REVISION_FILE) + if not data: + return None + value = data.get("revision") + return str(value) if value else None + + +#: How many iterations the experiment should reach in TOTAL, kept in a file +#: outside the running process so the target can be adjusted while the outer +#: loop is running (raise it to continue, lower it to stop earlier). +TARGET_FILE = "target_iterations.json" + + +def target_iterations_path(exp_dir: Path) -> Path: + return Path(exp_dir) / TARGET_FILE + + +def read_target_iterations(exp_dir: Path) -> Optional[int]: + """Total iteration count the experiment should reach (None = unset).""" + data = _load_json(target_iterations_path(exp_dir)) + if not data: + return None + try: + value = int(data.get("target")) + except (TypeError, ValueError): + return None + return value if value >= 1 else None + + +def write_target_iterations(exp_dir: Path, target: int) -> Dict[str, Any]: + exp_dir = Path(exp_dir) + exp_dir.mkdir(parents=True, exist_ok=True) + payload = {"target": int(target), "updated_at": time.time()} + _atomic_write_json(target_iterations_path(exp_dir), payload) + return payload + + +def clear_target_iterations(exp_dir: Path) -> None: + try: + target_iterations_path(exp_dir).unlink() + except OSError: + pass diff --git a/metainfer/tasks/harness_evolve/orchestrator/supervisor.py b/metainfer/tasks/harness_evolve/orchestrator/supervisor.py new file mode 100644 index 00000000..c961efd7 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/supervisor.py @@ -0,0 +1,737 @@ +"""Incident supervisor for unattended harness_evolve runs. + +The outer loop is long (hours per round) and can die in ways that are not the +harness' fault (the GPU filled up, an agent transport closed, the host killed +the process). This supervisor watches the experiment and, per incident class, +takes the *safe* action automatically: + + resource the host ran out of memory and killed the run (exit 137 / + OOM markers in the log) -> wait (default 30 min), then restart + the round. HE never inspects the cards: a device verdict is + DKAO's (FLOW.md §0) + agent flaky agent transport/timeout -> exponential backoff restart + code deterministic traceback/config error -> do NOT loop; stop and + report (retrying would replay the same failure) + unknown restart once, then stop + +Every decision is a pure function (:func:`classify_incident`, +:func:`decide_action`) so the behaviour is testable without touching GPUs, and +every action is appended to ``incidents.jsonl`` for audit. Repeated failures of +the same class trip a circuit breaker that stops the run instead of burning +compute unattended. + +Optionally (``--diagnose-agent``) it asks a read-only DSH agent to write +``incident_report.md`` with a root-cause hypothesis; that report is advisory +only — it never performs restarts itself. +""" + +from __future__ import annotations + +import argparse +import json +import os +import signal +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple + +CATEGORY_RESOURCE = "resource" +CATEGORY_AGENT = "agent" +CATEGORY_CODE = "code" +CATEGORY_UNKNOWN = "unknown" + +_RESOURCE_MARKERS = ( + "outofmemory", "out of memory", "hipErrorOutOfMemory", "bad_alloc", + "cannot allocate memory", "no space left", "memory exhausted", +) +_AGENT_MARKERS = ( + "transportclosederror", "timed out after", "timeoutexpired", + "nonzero exit 1", "agent attempt", "connection reset", +) +_CODE_MARKERS = ("traceback (most recent call last)",) + + +def classify_incident(*, log_tail: str = "", + exit_code: Optional[int] = None) -> Tuple[str, List[str]]: + """Return ``(category, evidence)`` for one observed failure. + + The verdict comes from *this run's own* evidence (exit status and log text); + HE holds no device verdict of its own (FLOW.md §0), so nothing here reads + the cards. + """ + text = (log_tail or "").lower() + evidence: List[str] = [] + if exit_code in (137, -9): + evidence.append(f"exit code {exit_code} (killed, usually OOM)") + for marker in _RESOURCE_MARKERS: + if marker.lower() in text: + evidence.append(f"log mentions {marker!r}") + break + if evidence: + return CATEGORY_RESOURCE, evidence + for marker in _AGENT_MARKERS: + if marker.lower() in text: + return CATEGORY_AGENT, [f"log mentions {marker!r}"] + for marker in _CODE_MARKERS: + if marker in text: + return CATEGORY_CODE, ["deterministic traceback in the log"] + return CATEGORY_UNKNOWN, evidence or ["no recognisable failure signature"] + + +def decide_action(*, category: str, same_class_streak: int, + waits_done: int, restarts_done: int, + running: bool, + max_restarts: int = 3, max_waits: int = 6, + breaker_streak: int = 3, + wait_seconds: int = 1800) -> Dict[str, Any]: + """Pure policy: what to do about one incident. + + ``action`` is one of ``restart`` | ``wait`` | ``stop`` | ``run`` (healthy) + | ``done`` (target already reached). + """ + if running: + return {"action": "run", "sleep_s": 0, + "reason": "experiment process is alive"} + if same_class_streak >= breaker_streak and category != CATEGORY_RESOURCE: + return {"action": "stop", "sleep_s": 0, + "reason": (f"circuit breaker: {same_class_streak} consecutive " + f"{category} failures")} + if category == CATEGORY_CODE: + return {"action": "stop", "sleep_s": 0, + "reason": ("deterministic code/config failure; restarting " + "would replay it")} + if category == CATEGORY_RESOURCE: + if waits_done >= max_waits: + return {"action": "stop", "sleep_s": 0, + "reason": f"resource still short after {waits_done} waits"} + return {"action": "wait", "sleep_s": int(wait_seconds), + "reason": ("memory pressure: wait, re-check free VRAM, then " + "restart")} + if restarts_done >= max_restarts: + return {"action": "stop", "sleep_s": 0, + "reason": f"gave up after {restarts_done} restarts"} + backoff = wait_seconds if category == CATEGORY_AGENT else 300 + return {"action": "restart", "sleep_s": min(backoff, 1800), + "reason": f"{category} failure; restart with backoff"} + + +def pid_is_live(pid: Optional[int], *, marker: str = "harness_evolve", + proc_root: Path = Path("/proc")) -> bool: + """Live (non-zombie) process whose cmdline mentions ``marker``. + + Shared by the supervisor and the server so both agree on what "running" + means: a zombie child or a recycled pid must never count as alive. + + ``proc_root`` exists so tests can build a fake process table (the same hook + ``live_child_count`` already has); it defaults to the real ``/proc``. + """ + if not pid or pid == os.getpid(): + return False + proc = Path(proc_root) / str(int(pid)) + if not proc.exists(): + return False + try: + state = (proc / "stat").read_text(encoding="utf-8").split(") ", 1)[1].split()[0] + except (OSError, IndexError): + return False + if state == "Z": + return False + try: + cmdline = (proc / "cmdline").read_bytes().replace(b"\0", b" ").decode( + "utf-8", "replace") + except OSError: + return False + return marker in cmdline + + +def pid_from_file(path: Path) -> Optional[int]: + """The PID recorded in a pid file, whether it is JSON or a bare number. + + Both writers in this package use JSON (``{"pid": …}``); an older process may + have left a bare number. Reading only the bare form is what made every + current pid file unparseable, which silently pushed every liveness question + onto the ``/proc`` cmdline scan — and that scan then matched the *caller + itself*, so a resume could never start round 1. + """ + try: + raw = path.read_text(encoding="utf-8").strip() + except OSError: + return None + if not raw: + return None + if raw.startswith("{"): + try: + data = json.loads(raw) + except ValueError: + return None + try: + return int(data.get("pid")) + except (TypeError, ValueError): + return None + try: + return int(raw.split()[0]) + except (IndexError, ValueError): + return None + + +def task_process_alive(state_dir: Path, *, + exclude: Optional[int] = None, + proc_root: Path = Path("/proc"), + self_pid: Optional[int] = None) -> Optional[int]: + """PID of *another* live orchestrator for this task, whichever path started it. + + Tasks can be started by the server ("run") or by the supervisor + ("resume"); ``resume.pid`` alone is not enough, and a server-started task + has no resume pid at all. Falling back to a cmdline scan keeps the + supervisor from "restarting" a perfectly healthy run. + + The caller's own pid is never returned (and ``exclude`` adds another): a + resume asking "is somebody else running this experiment?" must not answer + itself, or it will stand down forever and no round will ever start. + """ + state_dir = Path(state_dir) + excluded = {int(self_pid) if self_pid is not None else os.getpid()} + if exclude: + excluded.add(int(exclude)) + for name in ("orchestrator.pid", "resume.pid"): + pid = pid_from_file(state_dir / name) + if not pid or pid in excluded: + continue + if pid_is_live(pid, proc_root=proc_root): + return pid + marker = str(state_dir) + for entry in Path(proc_root).iterdir(): + if not entry.name.isdigit(): + continue + pid = int(entry.name) + if pid in excluded: + continue + try: + cmdline = (entry / "cmdline").read_bytes().replace(b"\0", b" ").decode( + "utf-8", "replace") + except OSError: + continue + if "harness_evolve.orchestrator.cli" in cmdline and marker in cmdline: + return pid + return None + + +class Supervisor: + """Watch one experiment and keep it moving toward its target round.""" + + def __init__(self, state_dir: Path, workspace_dir: Path, *, + target: Optional[int] = None, interval_s: int = 30, + wait_seconds: int = 1800, max_restarts: int = 3, + max_waits: int = 6, diagnose_agent: bool = False, + stall_seconds: int = 2400, + clock: Callable[[], float] = time.time, + sleeper: Callable[[float], None] = time.sleep, + resume_launcher: Optional[Callable[[], int]] = None, + ) -> None: + self.state_dir = Path(state_dir) + self.workspace_dir = Path(workspace_dir) + self.exp_dir = self.workspace_dir + self.target = target + self.interval_s = int(interval_s) + self.wait_seconds = int(wait_seconds) + self.max_restarts = int(max_restarts) + self.max_waits = int(max_waits) + self.diagnose_agent = bool(diagnose_agent) + self.stall_seconds = int(stall_seconds) + self._resume_proc = None + self.clock = clock + self.sleeper = sleeper + self._resume_launcher = resume_launcher + self.state_path = self.exp_dir / "supervisor_state.json" + self.incidents_path = self.exp_dir / "incidents.jsonl" + self.stop_flag = self.exp_dir / "supervisor.stop" + + # ---------------------------------------------------------------- state + def read_state(self) -> Dict[str, Any]: + try: + return json.loads(self.state_path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {} + + def write_state(self, **fields: Any) -> Dict[str, Any]: + state = self.read_state() + state.update(fields) + state["updated_at"] = self.clock() + try: + self.state_path.write_text( + json.dumps(state, ensure_ascii=False, indent=2), + encoding="utf-8") + except OSError: + pass + return state + + def record_incident(self, **fields: Any) -> None: + row = {"ts": self.clock(), **fields} + try: + with self.incidents_path.open("a", encoding="utf-8") as fh: + fh.write(json.dumps(row, ensure_ascii=False) + "\n") + except OSError: + pass + + # ----------------------------------------------------------- inspection + def _read_json(self, path: Path) -> Dict[str, Any]: + try: + return json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return {} + + def run_state(self) -> Dict[str, Any]: + return self._read_json(self.state_dir / "run.json") + + def current_target(self) -> Optional[int]: + if self.target is not None: + return int(self.target) + data = self._read_json(self.exp_dir / "target_iterations.json") + try: + return int(data.get("target")) + except (TypeError, ValueError): + return None + + # NOTE: there is deliberately no ``free_gb()`` / device query here any more. + # HE does not read the cards (FLOW.md §0): whether a device is clean enough + # to measure on is DKAO's verdict, taken in the child's own gate, and the + # only place it is recorded is the child's ``measurement_gate.jsonl``. + + def resume_pid(self) -> Optional[int]: + try: + return int((self.state_dir / "resume.pid") + .read_text(encoding="utf-8").split()[0]) + except (OSError, ValueError, IndexError): + return None + + def process_alive(self) -> bool: + """True when *any* orchestrator process for this task is alive. + + Covers both launch paths (server "run" and supervisor "resume") and + rejects zombies, which is what the stuck-then-restarting failure mode + needed. + """ + return task_process_alive(self.state_dir) is not None + + def progress_age(self) -> Optional[float]: + """Seconds since *any* part of the run last made progress. + + The outer loop is mostly waiting: during ``evaluate`` it can sit for + hours while the DKAO children work, and the outer timeline barely + changes. Counting child state as progress prevents the stall guard + from killing a perfectly healthy long round (which would turn into a + restart storm). + """ + newest = 0.0 + for name in ("run.json", "timeline.jsonl"): + try: + newest = max(newest, (self.state_dir / name).stat().st_mtime) + except OSError: + continue + iteration = int(self.run_state().get("current_iteration") or 0) + children_root = self.exp_dir / "children" + if iteration and children_root.is_dir(): + # One round can own two child directories: the performance pass + # (``iteration_NNN``) and, when that pass won its gate, the + # generalization retake (``iteration_NNN_generalization``). Both are + # part of "what is running right now". + for name in (f"iteration_{iteration:03d}", + f"iteration_{iteration:03d}_generalization"): + current = children_root / name + if not current.is_dir(): + continue + for child in current.iterdir(): + for relative in ("state/timeline.jsonl", "state/run.json", + "orchestrator-external.log"): + try: + newest = max(newest, + (child / relative).stat().st_mtime) + except OSError: + continue + if newest <= 0: + return None + return max(0.0, self.clock() - newest) + + def live_child_count(self, proc_root: Path = Path("/proc")) -> int: + """Live DKAO children of the iteration currently being evaluated. + + The stall guard needs *processes*, not only file mtimes: a child that is + generating or measuring a kernel can be silent for an hour, and reading + that as "the run died" produced a duplicate resume per stall. + """ + iteration = int(self.run_state().get("current_iteration") or 0) + if not iteration: + return 0 + # ``iteration_NNN`` also matches ``iteration_NNN_generalization``: both + # belong to the round being evaluated (the retake is the same round). + needle = f"children/iteration_{iteration:03d}" + alive = 0 + try: + entries = list(proc_root.iterdir()) + except OSError: + return 0 + for entry in entries: + if not entry.name.isdigit(): + continue + try: + cmdline = (entry / "cmdline").read_bytes().replace( + b"\0", b" ").decode("utf-8", "replace") + except OSError: + continue + if needle not in cmdline or "dcu_kernel_auto_opt" not in cmdline: + continue + try: + state = (entry / "stat").read_text().split(") ", 1)[1].split()[0] + except (OSError, IndexError, ValueError): + continue + if state != "Z": + alive += 1 + return alive + + def stale_run_pids(self) -> List[int]: + """Orchestrator processes for this task that are still alive.""" + marker = str(self.state_dir) + out: List[int] = [] + try: + entries = list(Path("/proc").iterdir()) + except OSError: + return out + for entry in entries: + if not entry.name.isdigit(): + continue + pid = int(entry.name) + try: + cmdline = (entry / "cmdline").read_bytes().replace( + b"\0", b" ").decode("utf-8", "replace") + except OSError: + continue + if "harness_evolve.orchestrator.cli" in cmdline and marker in cmdline: + out.append(pid) + return out + + def terminate_stale_run(self, pids: Optional[Sequence[int]] = None + ) -> List[int]: + """Kill a stalled run's process groups; return the pids that were signalled. + + Their children are killed with them (each orchestrator starts its own + session), so the GPU leases they hold are reaped by the broker instead + of blocking the round that replaces them. + """ + if pids is None: + pids = self.stale_run_pids() + pids = [int(pid) for pid in pids if int(pid) != os.getpid()] + for pid in pids: + try: + os.killpg(os.getpgid(pid), signal.SIGTERM) + except OSError: + try: + os.kill(pid, signal.SIGTERM) + except OSError: + continue + deadline = self.clock() + 10.0 + while self.clock() < deadline: + if not any(pid_is_live(pid) for pid in pids): + return pids + self.sleeper(0.5) + for pid in pids: + if not pid_is_live(pid): + continue + try: + os.killpg(os.getpgid(pid), signal.SIGKILL) + except OSError: + try: + os.kill(pid, signal.SIGKILL) + except OSError: + continue + return pids + + def reap_child(self) -> Optional[int]: + """Collect a finished child so it cannot linger as a zombie.""" + proc = getattr(self, "_resume_proc", None) + if proc is None: + return None + code = proc.poll() + if code is None: + return None + self._resume_proc = None + self.record_incident(action="resume_exited", category=CATEGORY_UNKNOWN, + pid=proc.pid, returncode=code) + return int(code) + + def _log_tail(self, size: int = 4000) -> str: + parts: List[str] = [] + for name in ("resume.log", "orchestrator.log"): + path = self.state_dir / name + if path.is_file(): + try: + parts.append(path.read_text(encoding="utf-8", + errors="replace")[-size:]) + except OSError: + continue + return "\n".join(parts) + + # ------------------------------------------------------------- actions + @staticmethod + def project_root() -> Path: + """Directory that contains the ``metainfer`` package. + + Getting this wrong makes the restarted process die instantly with + ``ModuleNotFoundError: No module named 'metainfer'`` — which is exactly + how a supervised run can look "stuck in analyze". + """ + here = Path(__file__).resolve() + for parent in here.parents: + if (parent / "metainfer" / "tasks").is_dir(): + return parent + return here.parents[4] + + def _launch_resume(self) -> int: + if self._resume_launcher is not None: + return int(self._resume_launcher()) + meta_root = self.project_root() + log = (self.state_dir / "resume.log").open("a", encoding="utf-8") + cmd = [ + sys.executable, "-m", + "metainfer.tasks.harness_evolve.orchestrator.cli", "resume", + "--state-dir", str(self.state_dir), + "--workspace-dir", str(self.workspace_dir), + ] + env = dict(os.environ) + env.setdefault("PYTHONPATH", str(meta_root)) + proc = subprocess.Popen( + cmd, cwd=str(meta_root), stdout=log, stderr=subprocess.STDOUT, + stdin=subprocess.DEVNULL, env=env, start_new_session=True) + log.close() + # Keep the handle so the child can be reaped: an unreaped child stays + # as a zombie, and a zombie still answers ``os.kill(pid, 0)``. + self._resume_proc = proc + (self.state_dir / "resume.pid").write_text(str(proc.pid), + encoding="utf-8") + return int(proc.pid) + + def write_incident_report(self, incident: Dict[str, Any]) -> Optional[Path]: + """Ask a read-only DSH agent for a root-cause report (advisory).""" + if not self.diagnose_agent: + return None + # Local import: this advisory path must not add an import edge from the + # supervisor to the evolve module at load time. + from .evolve import DEFAULT_EVOLVE_MODEL + + wrapper = (Path(__file__).resolve().parents[2] + / "dcu_kernel_auto_opt" / "bridge" / "dsh" / "dsh_agent.py") + if not wrapper.is_file(): + return None + report = self.exp_dir / "incident_report.md" + prompt = ( + "You are diagnosing a crashed kernel-optimization experiment. " + f"Read these logs if useful: {self.state_dir}/resume.log, " + f"{self.state_dir}/orchestrator.log, and the incident record at " + f"{self.incidents_path}. Latest incident: {json.dumps(incident)[:2000]}\n" + f"Write a concise markdown report to {report} with: (1) most likely " + "root cause with the log lines that support it, (2) whether a " + "restart would help, (3) one concrete prevention suggestion. " + "Do not run commands; analysis and the file write only." + ) + try: + proc = subprocess.run( + [sys.executable, str(wrapper), "-p", "--output-format", + "stream-json", "--input-format", "text", + "--permission-mode", "bypassPermissions", + "--add-dir", str(self.exp_dir), "--model", + DEFAULT_EVOLVE_MODEL, "--effort", "low", + "--tools", "Read,Glob,Grep,Write", + "--disallowedTools", "Bash,Skill,WebFetch,WebSearch"], + input=prompt, text=True, capture_output=True, timeout=1200, + env=dict(os.environ)) + except Exception: # noqa: BLE001 - diagnostics are best effort + return None + return report if report.is_file() else None + + # ---------------------------------------------------------------- loop + def step(self) -> Dict[str, Any]: + """One supervision tick. Returns the decision record.""" + if self.stop_flag.exists(): + self.write_state(status="stopped", reason="stop flag present") + return {"action": "stop", "reason": "stop flag present"} + + self.reap_child() + run = self.run_state() + target = self.current_target() + iteration = int(run.get("current_iteration") or 0) + finished = bool(run.get("finished")) + alive = self.process_alive() + stalled = False + if alive and not finished: + age = self.progress_age() + if age is not None and age > float(self.stall_seconds): + live_children = self.live_child_count() + if live_children: + # A DKAO child can spend an hour generating a kernel without + # writing its own state. That is work, not a stall, and + # calling it one used to start a *second* resume process + # that then fought the first one for the same four devices. + self.record_incident(action="stall_ignored", + category=CATEGORY_UNKNOWN, + idle_seconds=int(age), + live_children=live_children, + reason="children of this iteration " + "are still running") + else: + stalled = True + alive = False + self.record_incident(action="stalled", + category=CATEGORY_UNKNOWN, + pid=self.resume_pid(), + idle_seconds=int(age), + reason="no state written for too long") + # Never leave the stalled process behind: an orphan keeps + # its GPU leases and starves the round that replaces it. + killed = self.terminate_stale_run() + if killed: + self.record_incident(action="stalled_killed", + category=CATEGORY_UNKNOWN, + pids=killed) + + if target is not None and iteration >= target and finished: + self.write_state(status="done", iteration=iteration, + target=target) + return {"action": "done", "reason": f"target {target} reached", + "iteration": iteration, "target": target} + + same_streak = int(self.read_state().get("same_class_streak") or 0) + waits = int(self.read_state().get("waits_done") or 0) + restarts = int(self.read_state().get("restarts_done") or 0) + + incident: Optional[Dict[str, Any]] = None + category = CATEGORY_UNKNOWN + if alive: + decision = {"action": "run", "sleep_s": 0, + "reason": "experiment process is alive"} + else: + category, evidence = classify_incident(log_tail=self._log_tail()) + last_category = str(self.read_state().get("last_category") or "") + same_streak = same_streak + 1 if category == last_category else 1 + incident = { + "category": category, "evidence": evidence, + "iteration": iteration, + "finished": finished, "target": target, + } + decision = decide_action( + category=category, same_class_streak=same_streak, + waits_done=waits, restarts_done=restarts, running=False, + max_restarts=self.max_restarts, max_waits=self.max_waits, + wait_seconds=self.wait_seconds) + + action = decision["action"] + if action == "done": + self.write_state(status="done") + return decision + + if action == "run": + self.write_state(status="running", target=target, + iteration=iteration, last_category=None, + same_class_streak=0, waits_done=0, + restarts_done=0) + return decision + + if action == "stop": + self.write_state(status="stopped", reason=decision["reason"], + last_category=category, + same_class_streak=same_streak) + self.record_incident(action="stop", category=category, + reason=decision["reason"], + evidence=(incident or {}).get("evidence")) + if incident: + self.write_incident_report({**incident, **decision}) + return decision + + if action == "wait": + self.record_incident(action="wait", category=category, + seconds=decision["sleep_s"], + reason=decision["reason"], + evidence=(incident or {}).get("evidence")) + self.write_state(status="waiting", last_category=category, + same_class_streak=same_streak, + waits_done=waits + 1, + waiting_until=self.clock() + decision["sleep_s"], + reason=decision["reason"]) + self.sleeper(decision["sleep_s"]) + # The wait is over: restart the run. Whether a card is clean by then + # is DKAO's business (FLOW.md §0) — its gate waits on the child's + # behalf, so there is nothing for HE to re-check here. + pid = self._launch_resume() + self.record_incident(action="restart_after_wait", + category=category, pid=pid) + self.write_state(status="running", pid=pid, + waits_done=waits + 1) + return {**decision, "action": "restart", "pid": pid} + + # restart (backoff) + if decision.get("sleep_s"): + self.sleeper(decision["sleep_s"]) + pid = self._launch_resume() + self.record_incident(action="restart", category=category, pid=pid, + reason=decision["reason"], + evidence=(incident or {}).get("evidence")) + self.write_state(status="running", pid=pid, last_category=category, + same_class_streak=same_streak, + restarts_done=restarts + 1) + return {**decision, "pid": pid} + + def run_forever(self, max_ticks: Optional[int] = None) -> Dict[str, Any]: + self.write_state(status="starting", target=self.current_target(), + pid=os.getpid()) + ticks = 0 + last: Dict[str, Any] = {} + while max_ticks is None or ticks < max_ticks: + last = self.step() + ticks += 1 + if last.get("action") in {"done", "stop"}: + break + self.sleeper(self.interval_s) + return last + + +def main(argv: Optional[List[str]] = None) -> int: + ap = argparse.ArgumentParser(prog="harness-evolve-supervisor") + sub = ap.add_subparsers(dest="command", required=True) + watch = sub.add_parser("watch", help="supervise one experiment") + watch.add_argument("--state-dir", type=Path, required=True) + watch.add_argument("--workspace-dir", type=Path, required=True) + watch.add_argument("--target", type=int, default=None) + watch.add_argument("--interval", type=int, default=30) + watch.add_argument("--wait-seconds", type=int, default=1800) + watch.add_argument("--max-restarts", type=int, default=3) + watch.add_argument("--max-waits", type=int, default=6) + watch.add_argument("--diagnose-agent", action="store_true") + watch.add_argument("--stall-seconds", type=int, default=2400) + args = ap.parse_args(argv) + + if args.command == "watch": + sup = Supervisor( + args.state_dir, args.workspace_dir, target=args.target, + interval_s=args.interval, wait_seconds=args.wait_seconds, + max_restarts=args.max_restarts, max_waits=args.max_waits, + diagnose_agent=args.diagnose_agent, + stall_seconds=args.stall_seconds, + ) + (args.workspace_dir).mkdir(parents=True, exist_ok=True) + (args.state_dir).mkdir(parents=True, exist_ok=True) + pid_file = args.workspace_dir / "supervisor.pid" + pid_file.write_text(str(os.getpid()), encoding="utf-8") + try: + final = sup.run_forever() + finally: + try: + pid_file.unlink() + except OSError: + pass + print(json.dumps(final, ensure_ascii=False)) + return 0 + return 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/metainfer/tasks/harness_evolve/orchestrator/variant.py b/metainfer/tasks/harness_evolve/orchestrator/variant.py new file mode 100644 index 00000000..db41a7d7 --- /dev/null +++ b/metainfer/tasks/harness_evolve/orchestrator/variant.py @@ -0,0 +1,324 @@ +"""Per-operator variant table: the thing a candidate harness is measured against. + +Terminology (fixed by the operator): + +* **baseline** — the fixed Triton number for an operator (``baseline_us`` in the + question pool). It never changes. +* **variant** — the *current best* result for an operator, plus the kernel that + produced it. It is what every candidate is compared with. +* **harness** — the search policy (planner/gates). It does not measure or + validate anything; it only steers which experiments DKAO tries, so "the new + harness is better" means "it searches better", which is decided by comparing + candidate results against the variants. + +Update rule (operator decision, option **b**): the variant table only moves when +a harness is accepted and promoted into DKAO. A round that measures something +faster but is then rejected leaves the variant untouched — the comparison basis +must stay "what production currently is", otherwise the baseline would drift +while production did not. + +The table is seeded from what already exists (the registered question pool's +``best_known_us``, and the experiment's own ``best_known.json``); operators +without a recorded variant are simply compared against their fixed baseline +until they are measured for real. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any, Dict, Iterable, List, Optional + +import yaml + +VARIANT_TABLE = "variant_table.json" +VARIANT_HISTORY = "variant_history.jsonl" + +#: Fallback chain used when an operator has no recorded variant yet. +SOURCE_VARIANT = "variant" +SOURCE_BEST_KNOWN = "best_known" +SOURCE_BASELINE = "baseline" +SOURCE_NONE = "none" + + +def _load_json(path: Path) -> Any: + try: + return json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + + +def _num(value: Any) -> Optional[float]: + try: + number = float(value) + except (TypeError, ValueError): + return None + return number if number > 0 else None + + +def pool_instances(pool_path: Path) -> Dict[str, Dict[str, Any]]: + """``{operator_id: {baseline_us, best_known_us, family, contract}}``.""" + data = _load_json(Path(pool_path)) + if not isinstance(data, dict): + try: + data = yaml.safe_load(Path(pool_path).read_text(encoding="utf-8")) + except (OSError, ValueError, yaml.YAMLError): + return {} + out: Dict[str, Dict[str, Any]] = {} + for entry in (data or {}).get("instances") or []: + if not isinstance(entry, dict): + continue + iid = str(entry.get("id") or "") + if not iid: + continue + out[iid] = { + "baseline_us": _num(entry.get("baseline_us")), + "best_known_us": _num(entry.get("best_known_us")), + "family": str(entry.get("family") or ""), + "contract": dict(entry.get("contract") or {}), + "history": list(entry.get("history") or []), + } + return out + + +def seed_from_pool(table: Dict[str, Any], + instances: Dict[str, Dict[str, Any]]) -> int: + """Fill in variants for operators the pool already knows the best for. + + Only operators that have a measured ``best_known_us`` (an existing kernel + that beat the baseline) become variants; the rest stay on their baseline + until a round measures them. Nothing is re-measured here. + """ + operators = table.setdefault("operators", {}) + seeded = 0 + for iid, entry in instances.items(): + best = entry.get("best_known_us") + if best is None or iid in operators: + continue + operators[iid] = { + "median_us": best, + "p90_us": None, + "correctness_ok": True, + "kernel_source": "pool:best_known", + "harness_version": table.get("harness_version"), + "source_iteration": None, + "family": entry.get("family") or "", + "contract": entry.get("contract") or {}, + "conditions": {"origin": "registered_pool"}, + "recorded_at": None, + } + seeded += 1 + return seeded + + +def seed_from_experiment(table: Dict[str, Any], exp_dir: Path) -> int: + """Fill in variants from the experiment's own ``best_known.json``.""" + data = _load_json(Path(exp_dir) / "best_known.json") + if not isinstance(data, dict): + return 0 + operators = table.setdefault("operators", {}) + seeded = 0 + for iid, entry in data.items(): + if not isinstance(entry, dict) or iid in operators: + continue + best = _num(entry.get("best_median_us")) + if best is None: + continue + operators[str(iid)] = { + "median_us": best, + "p90_us": None, + "correctness_ok": True, + "kernel_source": f"experiment:iteration_{entry.get('best_iteration')}", + "harness_version": table.get("harness_version"), + "source_iteration": entry.get("best_iteration"), + "family": "", + "contract": {}, + "conditions": {"origin": "experiment_best_known"}, + "recorded_at": None, + } + seeded += 1 + return seeded + + +def load_variant_table(exp_dir: Path, *, pool_path: Optional[Path] = None, + create: bool = True) -> Dict[str, Any]: + """Read the table, seeding it from existing data on first use.""" + exp_dir = Path(exp_dir) + path = exp_dir / VARIANT_TABLE + table = _load_json(path) + if not isinstance(table, dict): + table = { + "schema_version": 1, + "created_at": time.time(), + "harness_version": None, + "harness_dir": None, + "operators": {}, + "groups": {}, + "seed_sources": [], + } + changed = False + if pool_path is not None and Path(pool_path).is_file(): + seeded = seed_from_pool(table, pool_instances(Path(pool_path))) + if seeded: + table.setdefault("seed_sources", []).append( + {"source": str(pool_path), "seeded": seeded, "ts": time.time()}) + changed = True + seeded_exp = seed_from_experiment(table, exp_dir) + if seeded_exp: + table.setdefault("seed_sources", []).append( + {"source": "best_known.json", "seeded": seeded_exp, "ts": time.time()}) + changed = True + if changed or not path.is_file(): + if create: + save_variant_table(exp_dir, table) + return table + + +def save_variant_table(exp_dir: Path, table: Dict[str, Any]) -> Path: + exp_dir = Path(exp_dir) + exp_dir.mkdir(parents=True, exist_ok=True) + table["updated_at"] = time.time() + path = exp_dir / VARIANT_TABLE + tmp = path.with_suffix(".json.tmp") + tmp.write_text(json.dumps(table, ensure_ascii=False, indent=2, sort_keys=True), + encoding="utf-8") + tmp.replace(path) + return path + + +def resolve_variant(table: Dict[str, Any], operator_id: str, *, + instances: Optional[Dict[str, Dict[str, Any]]] = None, + ) -> Optional[Dict[str, Any]]: + """The number this operator must beat, and where it came from. + + Order: recorded variant -> pool ``best_known_us`` -> fixed Triton + ``baseline_us``. Returns ``None`` when nothing at all is known, in which + case the operator cannot be judged this round. + """ + entry = (table.get("operators") or {}).get(operator_id) + if isinstance(entry, dict): + median = _num(entry.get("median_us")) + if median is not None: + return {"operator_id": operator_id, "median_us": median, + "source": SOURCE_VARIANT, + "harness_version": entry.get("harness_version"), + "p90_us": entry.get("p90_us"), + "kernel_source": entry.get("kernel_source"), + "recorded_at": entry.get("recorded_at")} + pool_entry = (instances or {}).get(operator_id) or {} + best = _num(pool_entry.get("best_known_us")) + if best is not None: + return {"operator_id": operator_id, "median_us": best, + "source": SOURCE_BEST_KNOWN, "harness_version": None, + "p90_us": None, "kernel_source": "pool:best_known", + "recorded_at": None} + baseline = _num(pool_entry.get("baseline_us")) + if baseline is not None: + return {"operator_id": operator_id, "median_us": baseline, + "source": SOURCE_BASELINE, "harness_version": None, + "p90_us": None, "kernel_source": "triton_baseline", + "recorded_at": None} + return None + + +def reference_results(table: Dict[str, Any], operator_ids: Iterable[str], *, + instances: Optional[Dict[str, Dict[str, Any]]] = None, + ) -> Dict[str, Dict[str, Any]]: + """A ``champion``-shaped result dict for :func:`compare_instances`. + + Each operator is represented by the variant it must beat, so the existing + comparison/reporting code works unchanged. + """ + out: Dict[str, Dict[str, Any]] = {} + for iid in operator_ids: + resolved = resolve_variant(table, iid, instances=instances) + if resolved is None: + continue + out[iid] = { + "status": "reference", + "passed": True, + "correctness_ok": True, + "median_us": resolved["median_us"], + "p90_us": resolved.get("p90_us"), + "variant_source": resolved["source"], + "variant_harness_version": resolved.get("harness_version"), + "kernel_source": resolved.get("kernel_source"), + } + return out + + +def unjudgeable(table: Dict[str, Any], operator_ids: Iterable[str], *, + instances: Optional[Dict[str, Dict[str, Any]]] = None, + ) -> List[str]: + """Operators with no variant and no baseline: they cannot be scored.""" + return [iid for iid in operator_ids + if resolve_variant(table, iid, instances=instances) is None] + + +def variant_beats(resolved: Dict[str, Any], + metrics: Dict[str, Any]) -> bool: + """Whether a measurement is faster than the variant it was compared with.""" + candidate = _num((metrics or {}).get("median_us")) + if candidate is None: + return False + return candidate < _num(resolved.get("median_us")) or False + + +def record_variants(exp_dir: Path, table: Dict[str, Any], results: Dict[str, Any], + *, version: str, iteration: int, + groups: Optional[Dict[str, Any]] = None, + ) -> Dict[str, Any]: + """Update the table after a harness is accepted and promoted (rule b). + + Only measured, correct results are recorded. Every change is appended to + ``variant_history.jsonl`` so it is always clear which harness set which + number. + """ + exp_dir = Path(exp_dir) + operators = table.setdefault("operators", {}) + recorded: List[Dict[str, Any]] = [] + history_path = exp_dir / VARIANT_HISTORY + now = time.time() + for iid, row in (results or {}).items(): + if not isinstance(row, dict): + continue + median = _num(row.get("median_us")) + if median is None or row.get("correctness_ok") is False: + continue + previous = operators.get(iid) or {} + previous_median = _num(previous.get("median_us")) + if previous_median is not None and median >= previous_median: + continue # only a real improvement moves the variant + operators[iid] = { + "median_us": median, + "p90_us": _num(row.get("p90_us")), + "correctness_ok": True, + "kernel_source": row.get("repo_path") or row.get("child_task_id"), + "harness_version": version, + "source_iteration": iteration, + "family": row.get("family") or previous.get("family") or "", + "contract": previous.get("contract") or {}, + "conditions": { + "bench_profile": row.get("bench_profile"), + "gate": "vram<=90,hcu==0", + }, + "recorded_at": now, + } + change = {"ts": now, "operator_id": iid, "median_us": median, + "previous_median_us": previous_median, + "harness_version": version, "iteration": iteration} + recorded.append(change) + try: + history_path.parent.mkdir(parents=True, exist_ok=True) + with history_path.open("a", encoding="utf-8") as fh: + fh.write(json.dumps(change, ensure_ascii=False) + "\n") + except OSError: + pass + table["harness_version"] = version + if groups: + table["groups"] = dict(groups) + table["last_promoted_at"] = now + save_variant_table(exp_dir, table) + return {"updated": sorted(c["operator_id"] for c in recorded), + "changes": recorded} diff --git a/metainfer/tasks/harness_evolve/server/__init__.py b/metainfer/tasks/harness_evolve/server/__init__.py new file mode 100644 index 00000000..3352dc2e --- /dev/null +++ b/metainfer/tasks/harness_evolve/server/__init__.py @@ -0,0 +1 @@ +"""harness_evolve server package (placeholder; headless usage needs no WebUI).""" diff --git a/metainfer/tasks/harness_evolve/server/plugin.py b/metainfer/tasks/harness_evolve/server/plugin.py new file mode 100644 index 00000000..41de1f04 --- /dev/null +++ b/metainfer/tasks/harness_evolve/server/plugin.py @@ -0,0 +1,34 @@ +"""harness_evolve Web plugin registration.""" + +from pathlib import Path + +from metainfer.server.registry import WebPlugin, register + +from .routes import build_router + +PLUGIN_TYPE = "harness-evolve" +_FRONTEND_DIR = Path(__file__).resolve().parent.parent / "static" + +plugin = WebPlugin( + type=PLUGIN_TYPE, + label="Harness Evolution (AHE)", + description=( + "AHE outer loop: evolve the DKAO harness across task instances with " + "double-generation iterations, change attribution and auto rollback." + ), + build_router=build_router, + detail_view_module="app/he-detail", + frontend_dir=_FRONTEND_DIR, + extra_stylesheets=["he.css"], + # The AHE page embeds the DKAO task view to mirror one question of one + # iteration (see LiveDkaoMirror in he-detail.js). Point at the DKAO + # plugin's own bundle rather than shipping a second copy, so the embedded + # view and the standalone DKAO page cannot drift apart. Listed explicitly + # because auto-discovery only yields ``app/`` keys. + importmap_entries={ + "app/dkao-detail": + "/static/plugins/dcu-kernel-auto-opt/dkao-detail.js?v=CACHE_BUST", + }, +) + +register(plugin) diff --git a/metainfer/tasks/harness_evolve/server/routes.py b/metainfer/tasks/harness_evolve/server/routes.py new file mode 100644 index 00000000..cefca4a1 --- /dev/null +++ b/metainfer/tasks/harness_evolve/server/routes.py @@ -0,0 +1,1553 @@ +"""Read-only Web routes for harness_evolve. + +Serves the AHE experiment artifacts (workspace_dir of a task entry): +- /summary report.md text + config snapshot + best_ever + scores +- /iterations per-iteration benchmark/diff/change_evaluation/evolve/rollback +- /iterations/{n}/overview analysis overview markdown (plain text) +""" + +from __future__ import annotations + +import json +import os +import time +from pathlib import Path +from typing import Any, Dict, List, Mapping + +from fastapi import APIRouter, HTTPException, Request + +from metainfer.server._helpers import ( + require_task_type, + state_dir_for, + task_or_404, + workspace_dir_for, +) +from metainfer.server.state_reader import read_requirements, read_run + +from ..orchestrator import phases +from ..orchestrator.phases import PREPARE +from ..orchestrator.rounds import ( + CHILDREN_DIR, GPU_COUNT, child_dir_map, device_for_index, + iteration_of_dir_name, resolve_child_dir, +) +from ..orchestrator.state import ( + read_target_iterations, write_target_iterations, +) + +PLUGIN_TYPE = "harness-evolve" + + +def _load(path: Path, default: Any) -> Any: + try: + return json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return default + + +def _experiment_config(entry: Any, workspace_dir: Path): + """Load the experiment config behind one task (for promotion commands).""" + from ..orchestrator.config import load_experiment_config + + state_dir = Path(str(getattr(entry, "state_dir", "") or "")) + req = state_dir / "requirements.json" + if not req.is_file(): + raise HTTPException(404, f"requirements.json missing in {state_dir}") + return load_experiment_config(req, state_dir, workspace_dir) + + +def _approve_with_config(entry: Any, workspace_dir: Path, by: str): + from ..orchestrator.promotion import approve_promotion + + cfg = _experiment_config(entry, workspace_dir) + return approve_promotion(cfg, cfg.exp_dir, approved_by=by) + + +def _deny_with_config(entry: Any, workspace_dir: Path, reason: str): + from ..orchestrator.promotion import deny_promotion + + cfg = _experiment_config(entry, workspace_dir) + return deny_promotion(cfg, cfg.exp_dir, reason=reason) + + +def _text(path: Path) -> str | None: + if not path.is_file(): + return None + try: + return path.read_text(encoding="utf-8", errors="replace") + except OSError: + return None + + +def _jsonl(path: Path) -> List[Dict[str, Any]]: + if not path.is_file(): + return [] + rows = [] + for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): + try: + item = json.loads(line) + except ValueError: + continue + if isinstance(item, dict): + rows.append(item) + return rows + + +def _iterations(exp: Path) -> List[Dict[str, Any]]: + runs = exp / "runs" + if not runs.is_dir(): + return [] + out: List[Dict[str, Any]] = [] + for d in sorted(runs.glob("iteration_*")): + if not d.is_dir(): + continue + i_dir = d / "input" + e_dir = d / "evolve" + entry: Dict[str, Any] = { + "name": d.name, + # Which gate this round judged, and what it concluded. The page must + # show the round's real criterion (FLOW.md §3: the variant + # comparison), not the pool's tau_family * baseline annotation. + "stage": _load(i_dir / "stage.json", None), + "performance_gate": _load(i_dir / "performance_gate.json", None), + "generalization_gate": _load( + i_dir / "generalization_gate.json", None), + "baseline_round": _load(i_dir / "baseline_round.json", None), + "benchmark": _load(i_dir / "benchmark" / "results.json", None), + "scored_ids": _load(i_dir / "benchmark" / "scored_ids.json", None), + "diff": _load(i_dir / "diff.json", None), + "decision": _load(i_dir / "decision.json", None), + "change_evaluation": _load( + i_dir / "change_evaluation.json", None + ), + "rollback": _load(d / "rollback.json", None), + "overview": ( + _text(i_dir / "analysis" / "overview.md") or "" + )[:8000], + "evolve_manifest": _load( + e_dir / "change_manifest.json", None + ), + "round_plan": _load(e_dir / "round_plan.json", None), + "evolve_summary": _text(e_dir / "evolve_summary.md") or "", + "promotion": _load(d / "promotion.json", None), + "mechanism_evidence": _load( + i_dir / "mechanism_evidence.json", None), + "purposes": (_load(i_dir / "benchmark" / "purposes.json", {}) + or {}).get("purposes", {}), + } + out.append(entry) + return out + + +def _child_best_median(workspace_dir: Path) -> Dict[str, Any]: + """Fast per-child perf peek: best accepted median + completed rounds.""" + best: Optional[float] = None + rounds = 0 + runs_root = workspace_dir / "workers" + if runs_root.is_dir(): + for w in runs_root.glob("worker_*"): + for exp_file in (w / "runs").glob("*/experiments.jsonl"): + for row in _jsonl(exp_file): + rounds += 1 + m = row.get("metrics") or {} + median = m.get("median_us") + if row.get("accepted") and isinstance(median, (int, float)): + best = median if best is None else min(best, float(median)) + return {"best_median_us": best, "rounds": rounds} + + +def _child_gate_audit(child_state_dir: Path, limit: int = 200) -> Dict[str, Any]: + """One child's own DKAO admission-gate audit trail. + + ``harness_evolve`` does not manage GPU occupancy: it hands each question to + a DKAO child and the child's engine decides when a card is clean enough to + time a kernel. So the reason a question is "doing nothing" lives in that + child's ``measurement_gate.jsonl`` — ``gate_blocked`` rows carry the reason + (``VRAM 96% > 90% limit`` / ``device busy: HCU 100.0% > 0.0%``), the attempt + number and the wait interval. This reads it read-only and never judges. + + One subtlety, because it decides what the UI may claim: a *passing* row also + carries a ``reasons`` list, and those reasons belong to the *other* + candidates that failed (``gpu`` on such a row is the device that passed + while ``reasons`` describes the one that did not). So the reasons are only + surfaced when the last verdict actually blocked, and the device this child + is using is taken from the passing row, not from the reasons. + """ + rows = _jsonl(child_state_dir / "measurement_gate.jsonl")[-limit:] + # A gate call writes one row per channel, so identical rows double up. + seen: set = set() + unique: List[Dict[str, Any]] = [] + for row in rows: + key = (round(float(row.get("ts") or 0), 1), row.get("event"), + row.get("attempt"), row.get("site")) + if key in seen: + continue + seen.add(key) + unique.append(row) + blocked = [r for r in unique if r.get("event") == "gate_blocked"] + passed = [r for r in unique if r.get("event") == "gate_pass" + or r.get("usable") is True] + last = unique[-1] if unique else {} + last_blocked = bool(last) and last.get("event") == "gate_blocked" + + def _device_of(row: Dict[str, Any]) -> Optional[int]: + try: + return int(row["gpu"]) if row.get("gpu") is not None else None + except (TypeError, ValueError): + return None + + # Which card this child is *measuring on* — not which card some gate call + # happened to run on. The generate-stage probe may legitimately run on any + # passing device ("the task's own card first, otherwise any idle one"), so a + # probe row on GPU2 says nothing about a child pinned to GPU0. Rows from the + # measuring sites (benchmark / PMC) do name the child's own card, so they are + # preferred; only when none exists does a probe (or a block) stand in. + own_card = None + for row in reversed(passed): + if str(row.get("site") or "") in ("benchmark", "profile_pmc"): + own_card = _device_of(row) + if own_card is not None: + break + if own_card is None: + for row in reversed(passed): + own_card = _device_of(row) + if own_card is not None: + break + device = own_card + if device is None: + for row in reversed(blocked): + device = _device_of(row) + if device is not None: + break + probe_device = None + for row in reversed(passed): + if str(row.get("site") or "") == "generate_probe": + probe_device = _device_of(row) + if probe_device is not None: + break + return { + "checks": len(unique), + "blocked": len(blocked), + "waited_seconds": sum( + float(r.get("wait_seconds") or 0) for r in blocked), + "device": device, + # Every device this child's gate ever looked at: the card it measures on + # plus any card a probe ran on. Kept apart from ``device`` so the page + # can say "pinned to GPU0, probe passed on GPU2" instead of claiming the + # child moved. + "devices": sorted({d for d in (_device_of(r) for r in unique) + if d is not None}), + "probe_device": probe_device, + "device_from": ("gate_pass" if device is not None and passed + else ("gate_blocked" if device is not None else None)), + "last_event": last.get("event"), + "last_ts": last.get("ts"), + "last_site": last.get("site"), + "last_gpu": last.get("gpu"), + "last_usable": last.get("usable"), + "last_candidates": list(last.get("candidates") or []), + "last_passing": list(last.get("passing") or []), + # Only meaningful (and only about *this* device) when the verdict blocked. + "last_reasons": (list(last.get("reasons") or []) if last_blocked + else []), + "last_attempt": last.get("attempt"), + "max_waits": last.get("max_waits"), + "wait_seconds": last.get("wait_seconds"), + "blocked_reasons": [ + {"ts": r.get("ts"), "site": r.get("site"), "gpu": r.get("gpu"), + "attempt": r.get("attempt"), "reasons": list(r.get("reasons") or [])} + for r in blocked[-10:] + ], + } + + +def _layout_devices(exp: Path, num: int) -> Dict[str, int]: + """``question -> device`` exactly as HE recorded it for one round. + + ``gpu_preflight.json`` is a layout record, not an occupancy check: it says + which card each question was handed to. That — not the card some later probe + used — is what a per-card view must group by. + """ + data = _load(exp / "runs" / f"iteration_{num:03d}" / "input" / "benchmark" + / "gpu_preflight.json", None) + raw = data.get("devices") if isinstance(data, dict) else None + out: Dict[str, int] = {} + if isinstance(raw, dict): + for qid, gpu in raw.items(): + try: + out[str(qid)] = int(gpu) + except (TypeError, ValueError): + continue + return out + + +def _assigned_device(requirements: Optional[Dict[str, Any]]) -> Optional[int]: + """The card HE pinned for one child, from its ``shape_config``. + + The HE evaluator pins each question with a manual DKAO assignment, so the + child's own requirements are a reliable second source when the round's layout + record is missing (older rounds, hand-written requirements). + """ + if not isinstance(requirements, dict): + return None + raw = requirements.get("shape_config") + cfg: Any = raw + if not isinstance(raw, dict): + try: + import yaml + + cfg = yaml.safe_load(str(raw or "")) or {} + except Exception: # noqa: BLE001 - a malformed config is simply no answer + return None + assignments = cfg.get("assignments") if isinstance(cfg, dict) else None + if isinstance(assignments, dict): + for worker in assignments.values(): + if isinstance(worker, dict) and worker.get("gpu") is not None: + try: + return int(worker["gpu"]) + except (TypeError, ValueError): + continue + return None + + +def _gpu_cards(exp: Path, num: int, + answers: Optional[Dict[str, Any]] = None, + ) -> List[Dict[str, Any]]: + """Devices of one AHE iteration with the questions DKAO is measuring on them. + + One card per physical device, showing how far each child's inner DKAO task + is and what it has measured. This replaces the old lease view read-only: HE + holds no lease and reads no device state, so the card carries the child's + own evidence — the device the child's gate cleared it for, and why it had to + wait when it did. + + The device comes from that child's gate records. Only when a child has no + gate record at all (it never reached a measurement) does the fixed + ``index % GPU_COUNT`` rotation stand in, and then it is flagged as a + prediction rather than reported as an observation. + """ + children = _children_live(exp, num, answers) + own = child_dir_map(exp / CHILDREN_DIR, num) + layout = _layout_devices(exp, num) + cards: Dict[int, Dict[str, Any]] = {} + for idx, (qid, info) in enumerate(sorted(children.items())): + state_dir = Path(str(info.get("child_state_dir") or (base / qid / "state"))) + gate = _child_gate_audit(state_dir) + requirements = _load((own.get(qid) or Path()) / "requirements.json", None) + assigned = _assigned_device(requirements) + # Which card this question belongs to. HE's own layout for the round is + # the answer ("index % 4", one question per card); a child that ran + # before the layout was recorded falls back to the device HE pinned in + # its requirements. The child's gate device is only a last resort: it can + # name a *probe* card, which would put a child pinned to GPU0 under GPU2 + # and make a four-card view show two. + if qid in layout: + gpu, source = layout[qid], "layout" + elif assigned is not None: + gpu, source = assigned, "assignment" + elif gate.get("device") is not None: + gpu, source = int(gate["device"]), (gate.get("device_from") or "gate") + else: + gpu, source = device_for_index(idx), "predicted" + info.update({ + "question": qid, + "gpu": gpu, + "gpu_source": source, + "assigned_device": assigned, + "gate_device": gate.get("device"), + "gate_audit": gate, + "requirements": requirements, + }) + cards.setdefault(gpu, {"gpu": gpu, "questions": []}) + cards[gpu]["questions"].append(info) + return [cards[g] for g in sorted(cards)] + + +def _gate_map(answers: Dict[str, Any], ids: List[str]) -> Dict[str, Dict[str, Any]]: + """The *pool* criterion per instance: baseline, family tau, target median. + + This is ``pool.auto_pass`` (median <= tau_family * baseline). It is NOT the + variant-round gate: since 2026-09-15 the performance/generalization gates + compare against each operator's current *variant* (FLOW.md §3, served by + :func:`_real_gate_map`), and a variant round never reaches ``auto_pass`` + (the variant branch ``continue``s before it). Keep this for the legacy / suite + path, which really was judged this way, and expose it to the page as + ``pool_reference`` so it can never be mistaken for the round's gate. + """ + from ..orchestrator.pool import Pool + + src = str(answers.get("pool_source") or "").strip() + if not src: + return {} + try: + pool = Pool.from_yaml(Path(src).expanduser()) + except Exception: # noqa: BLE001 - gates are advisory for the UI + return {} + out: Dict[str, Dict[str, Any]] = {} + for iid in ids: + inst = pool.instances.get(iid) + if inst is None: + continue + tau = pool.tau_family(inst.family or "") + out[iid] = { + "baseline_us": inst.baseline_us, + "tau_family": tau, + "target_us": round(tau * float(inst.baseline_us), 3), + "best_known_us": inst.best_known_us, + "family": inst.family, + } + return out + + +#: The gate's own knobs, mirrored from ``orchestrator.decision_engine`` so the +#: page can describe the criterion the orchestrator really applies (FLOW.md §3) +#: instead of re-deriving it. One place changes both. +GATE_WIN_RATIO = {"performance": 0.75, "generalization": 0.50} +GATE_FLOOR_PERCENT = 0.80 +GATE_NOISE_PERCENT = 2.0 + +VARIANT_TABLE_FILE = "variant_table.json" + + +def _round_input_dir(exp: Path, num: int) -> Path: + return exp / "runs" / f"iteration_{num:03d}" / "input" + + +def _round_stage(exp: Path, num: int) -> Optional[str]: + """The stage this round declared (``baseline`` / ``performance`` / ...). + + ``None`` means the round predates the staging (legacy / suite path, or the + short-lived transitional build that wrote a gate document without a + ``stage.json``). + """ + doc = _load(_round_input_dir(exp, num) / "stage.json", None) + name = doc.get("stage") if isinstance(doc, dict) else None + return str(name) if name else None + + +def _round_gate_doc(exp: Path, num: int, + stage: Optional[str]) -> Optional[Dict[str, Any]]: + """The gate document this round judged, or ``None`` when it judged none. + + The baseline round (FLOW.md §2) judges nothing, so it has no document: its + questions still *have* an opponent — the one the round is about to measure + and write back — which is what the page shows for it. + """ + i_dir = _round_input_dir(exp, num) + if stage == "generalization": + return _load(i_dir / "generalization_gate.json", None) + if stage == "performance": + return _load(i_dir / "performance_gate.json", None) + if stage is None: # transitional build: document, no stage + for name in ("performance_gate.json", "generalization_gate.json"): + doc = _load(i_dir / name, None) + if isinstance(doc, dict): + return doc + return None + + +def _pool_source(answers: Mapping[str, Any]) -> str: + return str( + answers.get("question_pool") or answers.get("pool_source") or "" + ).strip() + + +def _variant_sources(exp: Path, answers: Mapping[str, Any]): + """``(variant_table, pool_instances)``, exactly as the orchestrator sees them.""" + from ..orchestrator.variant import pool_instances + + table = _load(exp / VARIANT_TABLE_FILE, None) + table = table if isinstance(table, dict) else {} + instances: Dict[str, Dict[str, Any]] = {} + src = _pool_source(answers) + if src: + path = Path(src).expanduser() + if path.is_file(): + try: + instances = pool_instances(path) + except Exception: # noqa: BLE001 - a bad pool is "no answer", not a 500 + instances = {} + return table, instances + + +def _real_gate_map(exp: Path, num: int, ids: List[str], + answers: Mapping[str, Any], + ) -> Dict[str, Any]: + """The criterion this round actually judges, per question. + + FLOW.md §3: the performance gate compares each question against that + operator's *current variant* — a strict win for >= 75% of the questions, + with every question at least 80% of its variant (the floor). That is a + different number from ``tau_family * baseline_us``, so the two are served + separately and the page labels them by protocol: + + * ``variant`` (a ``stage.json`` exists, a gate document exists, or the run + kept a variant table): the round is judged against the variant; + * ``legacy``: the pre-2026-09-15 / suite path, which really did use the pool + criterion — the page says so instead of dressing it up as a variant match. + + ``target_us`` keeps its historic name for the round table, but for a variant + round it is the variant (``variant_us``), never a family baseline. + """ + from ..orchestrator.variant import resolve_variant + + stage = _round_stage(exp, num) + gate = _round_gate_doc(exp, num, stage) + table, instances = _variant_sources(exp, answers) + on_protocol = (stage is not None or isinstance(gate, dict) + or (exp / VARIANT_TABLE_FILE).is_file()) + judged = ((gate or {}).get("comparison") or {}).get("per_instance") or {} + win_ratio = GATE_WIN_RATIO.get( + str(stage or ""), GATE_WIN_RATIO["performance"]) + targets: Dict[str, Dict[str, Any]] = {} + for qid in ids: + resolved = resolve_variant(table, qid, instances=instances) or {} + variant_us = resolved.get("median_us") + row = judged.get(qid) or {} + targets[qid] = { + "protocol": "variant" if on_protocol else "legacy", + "stage": stage, + "variant_us": variant_us, + "variant_source": resolved.get("source"), + "variant_p90_us": resolved.get("p90_us"), + "kernel_source": resolved.get("kernel_source"), + "target_us": variant_us, + "judged": ({ + "verdict": row.get("verdict"), + "candidate_median_us": row.get("candidate_median_us"), + "delta_percent": row.get("delta_percent"), + } if row else None), + "win_ratio": win_ratio, + "wins_needed": None, + "floor_percent": GATE_FLOOR_PERCENT, + "noise_percent": GATE_NOISE_PERCENT, + } + round_gate: Optional[Dict[str, Any]] = None + if isinstance(gate, dict): + round_gate = { + "kind": gate.get("kind") or stage, + "status": gate.get("status"), + "wins": gate.get("wins"), + "wins_needed": gate.get("wins_needed"), + "operators": gate.get("operators"), + "counted": gate.get("counted"), + "below_floor": gate.get("below_floor"), + "unmeasured": gate.get("unmeasured"), + "reasons": gate.get("reasons"), + } + for qid in targets: + targets[qid]["wins_needed"] = gate.get("wins_needed") + return { + "protocol": "variant" if on_protocol else "legacy", + "stage": stage, + "round": round_gate, + "targets": targets, + } + + +def _children_live(exp: Path, num: int, + answers: Optional[Dict[str, Any]] = None, + ) -> Dict[str, Dict[str, Any]]: + """Live per-question state from each real DKAO child task. + + Every question is one DKAO child whose orchestrator writes its own + ``state/run.json``; expose the child's current_phase (Prepare -> ... -> + Finished of the *inner* DKAO loop) plus its perf peek and the gate it is + judged by so the HE detail page can show the Evaluate step drilling into + each child. + + ``gate`` is the round's real criterion (the variant comparison). For rounds + that predate the variant protocol it degenerates to the pool criterion, with + ``protocol == "legacy"`` on it so the page can say which path produced the + verdict. ``pool_reference`` always carries the pool criterion as a clearly + named annotation — it is not a gate on the variant protocol. + """ + base = exp / CHILDREN_DIR + dirs = child_dir_map(base, num) # {question: newest attempt dir} + out: Dict[str, Dict[str, Any]] = {} + if not dirs: + return out + ids = sorted(dirs) + gates = _gate_map(answers or {}, ids) + real = _real_gate_map(exp, num, ids, answers or {}) + for qid, d in sorted(dirs.items()): + run = read_run(d / "state") + perf = _child_best_median(d / "workspace") + gate = real["targets"].get(qid) + pool_ref = gates.get(qid) + if (gate is not None and gate.get("protocol") == "legacy" + and pool_ref is not None): + # A legacy round was judged by the family criterion, so that is the + # target its page must show — unchanged from how it was decided. + gate = dict(gate, target_us=pool_ref.get("target_us")) + out[qid] = { + "current_phase": run.get("current_phase") or "idle", + "current_iteration": run.get("current_iteration"), + "finished": bool(run.get("finished")), + "final_status": run.get("final_status"), + "last_update": run.get("last_update"), + "best_median_us": perf["best_median_us"], + "rounds": perf["rounds"], + "gate": gate, + "pool_reference": pool_ref, + "gate_protocol": real["protocol"], + "gate_stage": real["stage"], + "child_dir": str(d), + "attempt_dir": d.parent.name, + "child_state_dir": str(d / "state"), + } + return out + + +def _tail_text(path: Path, size: int = 1200) -> str: + if not path.is_file(): + return "" + try: + text = path.read_text(encoding="utf-8", errors="replace") + except OSError: + return "" + return text[-size:] + + +def _dkao_child_payload(exp: Path, num: int, child: str) -> Dict[str, Any]: + """Aggregate one HE question's inner DKAO task into a DKAO-like payload. + + Each question is a real dcu_kernel_auto_opt child that writes the same + artifacts a WebUI DKAO task does (state/run.json + timeline + agents.json, + workspace/workers/*). This mirrors what the DKAO detail page reads so the + HE page can render the child with the same look: 8-step inner state + machine + worker lanes + agents. + """ + if not child or child in {".", ".."} or "/" in child or "\\" in child: + raise HTTPException(404, "unknown child") + # A question can own several attempt directories (a first run, its retries, + # the generalization retake): the drill-down always opens the newest one. + d = resolve_child_dir(exp / CHILDREN_DIR, num, child) + if d is None or not d.is_dir(): + raise HTTPException(404, "child not found") + + state_dir = d / "state" + run = read_run(state_dir) + run_phase = run.get("current_phase") or "prepare" + # DKAO phases machine (Prepare..Finished, 8 steps incl. Baseline). + from metainfer.tasks.dcu_kernel_auto_opt.orchestrator import phases as dkao + graph = dkao.graph_payload(run_phase, include_baseline=True) + + agents_raw = [ + item for item in ((_load(state_dir / "agents.json", None) or {}) + .get("agents") or []) + if isinstance(item, dict) + ] + + workers: List[Dict[str, Any]] = [] + wdir = d / "workspace" / "workers" + if wdir.is_dir(): + for w in sorted(wdir.iterdir()): + if not w.is_dir(): + continue + status = _load(w / "status.json", {}) or {} + bootstrap = _load(w / "bootstrap_result.json", {}) or {} + result = _load(w / "result.json", {}) or {} + shape_id = status.get("shape_id") + metrics: Dict[str, Any] = {} + if shape_id: + metrics = ( + ((result.get("shapes") or {}).get(shape_id) or {}) + .get("metrics") or {} + ) + # Per-round optimization history (the same source DKAO lanes use). + experiments: List[Dict[str, Any]] = [] + rounds = 0 + runs_dir = w / "runs" + if runs_dir.is_dir(): + for exp_file in sorted(runs_dir.glob("*/experiments.jsonl")): + if not shape_id: + shape_id = exp_file.parent.name + for row in _jsonl(exp_file): + m = row.get("metrics") or {} + experiments.append({ + "iteration": row.get("iteration"), + "median_us": m.get("median_us"), + "p90_us": m.get("p90_us"), + "accepted": bool(row.get("accepted")), + "correctness_passed": bool( + row.get("correctness_passed") + ), + "speedup": row.get("speedup"), + "baseline_us": row.get("baseline_us"), + "timestamp": row.get("timestamp"), + }) + rounds = len(experiments) + experiments.sort(key=lambda e: ( + float(e.get("timestamp") or 0), + int(e.get("iteration") or 0), + )) + accepted_medians = [ + e["median_us"] for e in experiments + if e.get("accepted") and e.get("median_us") is not None + ] + workers.append({ + "worker_id": w.name, + "state": status.get("state") or "idle", + "iteration": status.get("iteration"), + "shape_id": shape_id, + "physical_gpu": status.get("physical_gpu"), + "bootstrap_status": bootstrap.get("status"), + "median_us": metrics.get("median_us"), + "p90_us": metrics.get("p90_us"), + "passed": metrics.get("passed"), + "rounds": rounds, + "best_median_us": ( + min(accepted_medians) if accepted_medians else None + ), + "experiments": experiments, + "last_update": status.get("last_update"), + }) + + # Agents: annotate worker turns with the matching per-round performance + # (agent name is "worker_N--iterK" -> experiments.jsonl iteration K) + import re as _re + perf_index: Dict[Any, Dict[str, Any]] = {} + for w in workers: + for e in w.get("experiments") or []: + perf_index[(w["worker_id"], e.get("iteration"))] = e + agents: List[Dict[str, Any]] = [] + for item in agents_raw: + entry: Dict[str, Any] = { + "name": item.get("name"), + "role": item.get("role"), + "status": item.get("status"), + "phase": item.get("phase"), + "error": item.get("error"), + } + match = _re.match(r"^(worker_\d+)-.+-iter(\d+)$", + str(item.get("name") or "")) + if match: + worker_id, iteration = match.group(1), int(match.group(2)) + turn = perf_index.get((worker_id, iteration)) + if turn: + entry["iteration"] = iteration + entry["median_us"] = turn.get("median_us") + entry["p90_us"] = turn.get("p90_us") + entry["speedup"] = turn.get("speedup") + entry["baseline_us"] = turn.get("baseline_us") + entry["accepted"] = turn.get("accepted") + agents.append(entry) + + return { + "child": child, + "run": { + "current_phase": run_phase, + "current_iteration": run.get("current_iteration"), + "finished": bool(run.get("finished")), + "final_status": run.get("final_status"), + }, + "graph": graph, + "agents": agents, + "workers": workers, + "repo_path": str(d / "workspace" / "main"), + "child_workspace_dir": str(d / "workspace"), + "log_tail": _tail_text(d / "orchestrator-external.log"), + } + + +def _resume_running(state_dir: Path) -> Optional[int]: + """PID of a live harness_evolve run (zombies and recycled pids excluded).""" + try: + pid = int((Path(state_dir) / "resume.pid") + .read_text(encoding="utf-8").split()[0]) + except (OSError, ValueError, IndexError): + return None + try: + from ..orchestrator.supervisor import task_process_alive + except Exception: # noqa: BLE001 - fall back to a plain existence check + try: + os.kill(pid, 0) + return pid + except OSError: + return None + return task_process_alive(state_dir) + + +def _run_alive(state_dir: Path) -> Optional[int]: + """PID of a live orchestrator for this task, whichever path started it. + + The sweep has to distinguish "nobody is running this experiment" from "this + experiment is running fine". ``_resume_running`` answers only the narrower + question (is there a *resume*?), and a task created from the New Task form + never writes ``resume.pid`` — so a one-second-old, perfectly healthy run + looked like an unsupervised corpse, and the sweep started a second driver + for it (which then opened the next round while round 1 was still being + evaluated). ``task_process_alive`` reads ``orchestrator.pid`` first and + falls back to a cmdline scan, which is exactly the question here. + """ + try: + from ..orchestrator.supervisor import task_process_alive + return task_process_alive(Path(state_dir)) + except Exception: # noqa: BLE001 - never break the sweep + return _resume_running(state_dir) + + +#: Terminal states that must never be auto-restarted: the loop stopped on +#: purpose (a round could not measure every question, operator abort, circuit +#: breaker) and needs a human decision. Restarting them would silently re-enter +#: the same condition. +#: +#: ``gpu_gate_blocked`` is deliberately absent: since HE stopped managing GPU +#: occupancy it holds no device verdict of its own and never stops a run for +#: one. What it can do is stop because a *child's* DKAO gate refused to measure +#: — that arrives as ``round_incomplete`` with ``cause`` ``gpu_gate_blocked`` +#: (see ``DkaoCliEvaluator._stop_on_incomplete_round``). +TERMINAL_STATUSES = {"round_incomplete", + "aborted_by_operator", "circuit_breaker", + # an explicit Stop press: only a human asking again (or + # the resume paths clearing it) may bring the run back + "stopped_by_operator", + # variant-round protocol: a run parks for a human decision + # and ends once a harness is promoted (or the budget runs + # out) — none of those should be silently restarted. + "awaiting_approval", "rounds_exhausted"} + + +def should_launch_resume(run: Mapping[str, Any], target: Any, + resume_running: bool) -> bool: + """Whether an unattended start must (re)launch the experiment process. + + Covers three cases: nothing started yet, the previous process died + mid-round (``finished`` is False), and the target round is not reached — + but never a run that deliberately stopped for a terminal reason. + """ + if resume_running or not run: + return False + if str(run.get("final_status") or "") in TERMINAL_STATUSES: + return False + if not run.get("finished"): + return True + if target in (None, ""): + return False + try: + return int(run.get("current_iteration") or 0) < int(target) + except (TypeError, ValueError): + return False + + +def _launch_resume(state_dir: Path, workspace_dir: Path, + iterations: int, + champion_iteration: Any = None) -> Dict[str, Any]: + """Spawn the resume CLI in the background and record its pid.""" + import subprocess + import sys + + # Spawning the experiment again is the operator saying "continue": without + # this, the marker left by Stop would make the supervisor and the janitor + # refuse to supervise the run we are starting right now. + _clear_operator_stop(Path(workspace_dir)) + meta_root = Path(__file__).resolve().parents[4] + log_path = state_dir / "resume.log" + log = log_path.open("a", encoding="utf-8") + log.write(f"\n=== resume requested: +{iterations} round(s) " + f"@ {time.strftime('%Y-%m-%d %H:%M:%S')} ===\n") + log.flush() + cmd = [ + sys.executable, "-m", + "metainfer.tasks.harness_evolve.orchestrator.cli", "resume", + "--state-dir", str(state_dir), + "--workspace-dir", str(workspace_dir), + "--iterations", str(iterations), + ] + if champion_iteration not in (None, ""): + cmd += ["--champion-iteration", str(int(champion_iteration))] + env = dict(os.environ) + env.setdefault("PYTHONPATH", str(meta_root)) + proc = subprocess.Popen( + cmd, cwd=str(meta_root), stdout=log, stderr=subprocess.STDOUT, + stdin=subprocess.DEVNULL, env=env, start_new_session=True, + ) + log.close() + (state_dir / "resume.pid").write_text(str(proc.pid), encoding="utf-8") + return {"ok": True, "pid": proc.pid, "iterations": iterations, + "log": str(log_path)} + + +#: Presence of this file means "the operator stopped this run on purpose". +#: It is the only thing that distinguishes a deliberate stop from a crash, and +#: every path that would helpfully restart an unsupervised run must respect it +#: (the janitor sweep, the supervisor, the resume launcher). Cleared by the +#: resume paths, i.e. when a human asks for the run to continue again. +OPERATOR_STOP_FILE = "operator_stop.json" + + +def _operator_stop_path(workspace_dir: Path) -> Path: + return Path(workspace_dir) / OPERATOR_STOP_FILE + + +def _run_live(state_dir: Path, workspace_dir: Path) -> bool: + """Whether this experiment is doing something right now. + + Deliberately not just ``orchestrator.pid``: children outlive their parent's + bookkeeping (a round can be mid-flight while the parent is briefly between + steps), and a run started before that file existed has no entry at all. + """ + if _resume_running(Path(state_dir)): + return True + if _supervisor_running(Path(workspace_dir)): + return True + return bool(_live_dkao_children(Path(workspace_dir))) + + +def _operator_stopped(workspace_dir: Path) -> Optional[Dict[str, Any]]: + """The stop record if the operator stopped this run, else None.""" + return _load(_operator_stop_path(workspace_dir), None) or None + + +def _clear_operator_stop(workspace_dir: Path) -> bool: + """Drop the operator-stop marker so the run may be driven again.""" + path = _operator_stop_path(workspace_dir) + try: + path.unlink() + return True + except OSError: + return False + + +def _live_dkao_children(workspace_dir: Path) -> List[Dict[str, Any]]: + """Live DKAO child runs belonging to this HE workspace. + + Children are spawned with ``start_new_session=True``, so signalling the HE + orchestrator does NOT reach them: they would keep their agents and their + place in DKAO's admission queue after a "stop". They are found by their + command line, which names this experiment's artifact directory -- the one + thing that identifies them without a registry of our own. + """ + marker = f"{Path(workspace_dir)}/children/".encode() + found: List[Dict[str, Any]] = [] + for entry in Path("/proc").iterdir(): + if not entry.name.isdigit(): + continue + pid = int(entry.name) + if pid == os.getpid(): + continue + try: + cmdline = (entry / "cmdline").read_bytes().replace(b"\0", b" ") + except OSError: + continue + if marker not in cmdline \ + or b"dcu_kernel_auto_opt.orchestrator.cli" not in cmdline: + continue + found.append({"pid": pid, "cmd": cmdline.decode("utf-8", "replace")}) + return found + + +def _proc_state(pid: int) -> Optional[str]: + """``/proc/`` state letter, or None when the process is gone.""" + try: + stat = Path(f"/proc/{int(pid)}/stat").read_text(encoding="utf-8") + except OSError: + return None + try: + return stat.split(") ", 1)[1].split()[0] + except IndexError: + return None + + +def _terminate_pid(pid: int, *, grace_s: float = 20.0) -> bool: + """SIGTERM a process group (then the process), and report whether it went. + + ``os.killpg`` first because these processes lead their own session: the + DKAO orchestrator spawns agent processes that must not outlive it. + + A zombie counts as stopped -- it executes no more, it just has not been + reaped yet (we are usually not its parent). ``os.kill(pid, 0)`` succeeds + for exactly that case, which would make a successful stop report itself as + a failure. + """ + def _stopped() -> bool: + state = _proc_state(pid) + return state is None or state == "Z" + + for target in (-pid, pid): + try: + os.kill(target, 15) + break + except OSError: + continue + deadline = time.time() + grace_s + while time.time() < deadline: + if _stopped(): + return True + time.sleep(0.2) + for target in (-pid, pid): + try: + os.kill(target, 9) + break + except OSError: + continue + time.sleep(0.3) + return _stopped() + + +def _supervisor_running(workspace_dir: Path) -> Optional[int]: + """PID of a live incident supervisor for this experiment (None when idle). + + A zombie must not count: ``os.kill(pid, 0)`` succeeds for a defunct process + whose parent has not reaped it, and that stale pid then blocks every future + supervision of the task ("supervisor already running" for a process that + cannot observe anything). + """ + pid_file = Path(workspace_dir) / "supervisor.pid" + try: + pid = int(pid_file.read_text(encoding="utf-8").split()[0]) + except (OSError, ValueError, IndexError): + return None + try: + state = Path(f"/proc/{pid}/stat").read_text().split(") ", 1)[1] + except (OSError, IndexError, ValueError): + return None + if state.split()[0] == "Z": + return None + try: + os.kill(pid, 0) + except OSError: + return None + return pid + + +def _launch_supervisor(state_dir: Path, workspace_dir: Path, target: Any, + diagnose: bool = False) -> Dict[str, Any]: + """Start the unattended supervisor (self-healing) for this experiment.""" + import subprocess + import sys + + meta_root = Path(__file__).resolve().parents[4] + log_path = Path(workspace_dir) / "supervisor.log" + log = log_path.open("a", encoding="utf-8") + log.write(f"\n=== supervisor requested: target={target} diagnose={diagnose} " + f"@ {time.strftime('%Y-%m-%d %H:%M:%S')} ===\n") + log.flush() + cmd = [ + sys.executable, "-m", + "metainfer.tasks.harness_evolve.orchestrator.supervisor", "watch", + "--state-dir", str(state_dir), "--workspace-dir", str(workspace_dir), + ] + if target not in (None, ""): + cmd += ["--target", str(int(target))] + if diagnose: + cmd += ["--diagnose-agent"] + env = dict(os.environ) + env.setdefault("PYTHONPATH", str(meta_root)) + proc = subprocess.Popen( + cmd, cwd=str(meta_root), stdout=log, stderr=subprocess.STDOUT, + stdin=subprocess.DEVNULL, env=env, start_new_session=True, + ) + log.close() + (Path(workspace_dir) / "supervisor.pid").write_text(str(proc.pid), + encoding="utf-8") + return {"pid": proc.pid, "log": str(log_path)} + + +def _supervisor_payload(workspace_dir: Path) -> Dict[str, Any]: + workspace_dir = Path(workspace_dir) + state: Dict[str, Any] = {} + try: + state = json.loads((workspace_dir / "supervisor_state.json") + .read_text(encoding="utf-8")) + except (OSError, ValueError): + state = {} + incidents: List[Dict[str, Any]] = [] + path = workspace_dir / "incidents.jsonl" + if path.is_file(): + for line in path.read_text(encoding="utf-8", errors="replace").splitlines()[-40:]: + try: + incidents.append(json.loads(line)) + except ValueError: + continue + return { + "running": _supervisor_running(workspace_dir) is not None, + "pid": _supervisor_running(workspace_dir), + "state": state, + "incidents": incidents, + "has_report": (workspace_dir / "incident_report.md").is_file(), + } + + +_JANITOR_STARTED = False + + +def janitor_sweep() -> List[Dict[str, Any]]: + """Last-resort recovery: restart experiments nobody is supervising. + + The supervisor normally handles crashes, but if the supervisor itself dies + (OOM, kill, container restart) the experiment would sit idle forever — the + exact "stuck in analyze" symptom. This sweep runs inside the server and + re-launches supervision for any task that is unfinished and has neither a + live experiment process nor a live supervisor. + """ + from metainfer.server.tasks import list_tasks + + actions: List[Dict[str, Any]] = [] + for entry in list_tasks(): + if getattr(entry, "type", None) != PLUGIN_TYPE: + continue + try: + state_dir = state_dir_for(entry) + workspace_dir = workspace_dir_for(entry) + stopped = _operator_stopped(workspace_dir) + if stopped is not None: + # "Last-resort recovery" must not overrule a human: this is the + # one sweep that would otherwise restart a task minutes after + # the operator stopped it. + actions.append({"task": entry.id, "action": "skip", + "reason": "stopped by operator", + "stopped_at": stopped.get("stopped_at")}) + continue + run = read_run(state_dir) + if not run: + continue + target = read_target_iterations(workspace_dir) + run_live = _run_alive(state_dir) + supervisor_live = _supervisor_running(workspace_dir) + if run_live: + # Somebody *is* running this experiment — most often the process + # the New Task form started a second ago. Starting a second + # driver here is what made a fresh task open two rounds at once. + actions.append({"task": entry.id, "action": "skip", + "reason": "run is alive", "pid": run_live}) + continue + if supervisor_live: + if should_launch_resume(run, target, False): + resume = _launch_resume(state_dir, workspace_dir, 1, + champion_iteration=None) + actions.append({"task": entry.id, "action": "resume", + "pid": resume.get("pid")}) + continue + if not should_launch_resume(run, target, False): + continue + out = _launch_supervisor(state_dir, workspace_dir, target, + diagnose=False) + resume = _launch_resume(state_dir, workspace_dir, 1, + champion_iteration=None) + actions.append({"task": entry.id, "action": "supervisor+resume", + "supervisor_pid": out.get("pid"), + "resume_pid": resume.get("pid")}) + except Exception as exc: # noqa: BLE001 - never break the sweep + actions.append({"task": getattr(entry, "id", "?"), + "action": "error", "error": repr(exc)}) + return actions + + +def _start_janitor(interval_s: int = 300) -> None: + global _JANITOR_STARTED + if _JANITOR_STARTED: + return + _JANITOR_STARTED = True + import threading + + def loop() -> None: + while True: + time.sleep(max(30, int(interval_s))) + try: + actions = janitor_sweep() + for action in actions: + if action.get("action") != "error": + print(f"[he-janitor] {action}", flush=True) + except Exception: # noqa: BLE001 + continue + + threading.Thread(target=loop, daemon=True, name="he-janitor").start() + + +def build_router(plugin) -> APIRouter: + router = APIRouter() + if os.environ.get("METAINFER_DISABLE_JANITOR", "").strip().lower() not in { + "1", "true", "yes"}: + _start_janitor() + + @router.get("/summary") + def summary(task_id: str) -> Dict[str, Any]: + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + exp = workspace_dir_for(entry) + state_dir = state_dir_for(entry) + return { + "config_snapshot": _load(exp / "config_snapshot.json", None), + "report": _text(exp / "report.md"), + "best_ever": _load(exp / "best_ever.json", None), + "scores": _jsonl(exp / "iteration_scores.jsonl"), + # Which protocol this run is on. ``protocol.json`` is written by the + # orchestrator when it leaves the fixed one; the page puts a banner + # on screen from it, because a run that judges neither gate looks + # exactly like a run that is merely quiet. + "protocol": _load(exp / "protocol.json", None) + or {"protocol": "variant", "mode": "variant", + "reason": "FLOW.md fixed protocol"}, + # Whether the operator pressed Stop: the page needs it to explain + # why nothing is running, and to offer the way back. + "operator_stop": _operator_stopped(exp), + # Liveness as this plugin knows it. The shell's green dot reads + # ``orchestrator.pid``, which a run started before that file existed + # never wrote -- the process is alive and the dot is blue. The page + # must not inherit that blind spot, or Stop would be hidden on + # exactly the runs that need it. + "running": _run_live(state_dir, exp), + } + + @router.get("/iterations") + def iterations(task_id: str) -> List[Dict[str, Any]]: + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + return _iterations(workspace_dir_for(entry)) + + @router.get("/state-graph") + def state_graph(task_id: str) -> Dict[str, Any]: + """Outer-loop AHE phase machine (Prepare..Finished).""" + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + run = read_run(state_dir_for(entry)) + return phases.graph_payload(run.get("current_phase") or PREPARE) + + @router.get("/iterations/{num}/gpu") + def iteration_gpu(task_id: str, num: int) -> Dict[str, Any]: + """Read-only device cards for one AHE iteration. + + Replaces the old lease view: HE holds no lease and takes no reading of + the cards, so the page shows the layout HE chose (``index % 4``) plus + what each child's *own* DKAO gate says about the device it was handed. + """ + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + req = read_requirements(state_dir_for(entry)) or {} + answers = req.get("answers") + if not isinstance(answers, dict): + answers = req + exp = workspace_dir_for(entry) + return { + "iteration": num, + "gpu_count": GPU_COUNT, + "managed_by": "dcu_kernel_auto_opt", + "leases": [], + "layout": _load( + exp / "runs" / f"iteration_{num:03d}" / "input" / "benchmark" + / "gpu_preflight.json", None), + "cards": _gpu_cards(exp, num, answers), + } + + @router.get("/iterations/{num}/children") + def iteration_children(task_id: str, num: int) -> Dict[str, Dict[str, Any]]: + """Live inner DKAO phase per question of one AHE iteration.""" + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + req = read_requirements(state_dir_for(entry)) or {} + answers = req.get("answers") + if not isinstance(answers, dict): + answers = req + return _children_live(workspace_dir_for(entry), num, answers) + + @router.post("/resume") + async def resume(task_id: str, request: Request) -> Dict[str, Any]: + """Continue the experiment with further round(s) from the current state.""" + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + state_dir = state_dir_for(entry) + workspace_dir = workspace_dir_for(entry) + try: + body = await request.json() + except Exception: # noqa: BLE001 - body is optional + body = {} + body = body or {} + target = body.get("target") + if target is not None: + target = max(1, int(target)) + write_target_iterations(workspace_dir, target) + + live = _resume_running(state_dir) + if live: + # Already running: a new target is picked up by the next round. + return { + "ok": True, "running": True, "pid": live, + "target": read_target_iterations(workspace_dir), + "updated": target is not None, + "message": ("target updated; the running loop picks it up " + "at the next round" if target is not None + else "already running"), + } + run = read_run(state_dir) + if not run.get("finished"): + raise HTTPException( + 409, "task is not finished yet; wait for the current round") + iterations = max(1, int(body.get("iterations") or 1)) + champion_iteration = body.get("champion_iteration") + out = _launch_resume(state_dir, workspace_dir, iterations, + champion_iteration=champion_iteration) + out["target"] = read_target_iterations(workspace_dir) + return out + + @router.post("/autopilot") + async def autopilot(task_id: str, request: Request) -> Dict[str, Any]: + """Unattended mode: run to the target round, self-healing on incidents.""" + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + state_dir = state_dir_for(entry) + workspace_dir = workspace_dir_for(entry) + try: + body = await request.json() + except Exception: # noqa: BLE001 + body = {} + body = body or {} + target = body.get("target") + if target is not None: + write_target_iterations(workspace_dir, max(1, int(target))) + target = read_target_iterations(workspace_dir) + live = _supervisor_running(workspace_dir) + if live: + return {"ok": True, "supervisor_pid": live, "target": target, + "message": "supervisor already running; target updated"} + out = _launch_supervisor(state_dir, workspace_dir, target, + diagnose=bool(body.get("diagnose_agent"))) + resume_pid = None + run = read_run(state_dir) + if should_launch_resume(run, target, bool(_resume_running(state_dir))): + # Covers both "not started yet" and "the previous process died + # mid-round": the run must never be left unsupervised. + champion = body.get("champion_iteration") + resume = _launch_resume(state_dir, workspace_dir, 1, + champion_iteration=champion) + resume_pid = resume.get("pid") + return {"ok": True, "supervisor_pid": out["pid"], "target": target, + "resume_pid": resume_pid, "log": out["log"]} + + @router.get("/review") + def review(task_id: str) -> Dict[str, Any]: + """Cross-round review: generations, best-known curve, regressions.""" + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + ws = workspace_dir_for(entry) + regressions: List[Dict[str, Any]] = [] + path = ws / "regressions.jsonl" + if path.is_file(): + for line in path.read_text(encoding="utf-8", + errors="replace").splitlines()[-40:]: + try: + regressions.append(json.loads(line)) + except ValueError: + continue + return { + "review": _load(ws / "round_review.json", None), + "best_known": _load(ws / "best_known.json", {}), + "regressions": regressions, + } + + @router.get("/promotion") + def promotion(task_id: str) -> Dict[str, Any]: + """Pending harness promotion + what production currently is. + + Nothing reaches DKAO without an explicit approval, so the UI needs to + see the candidate, its gate evidence, and the current production + version in one place. + """ + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + ws = workspace_dir_for(entry) + from ..orchestrator.promotion import promoted_harness, read_pending + + pending = read_pending(ws) + promoted = promoted_harness(ws) + table_path = ws / "variant_table.json" + table = _load(table_path, {}) or {} + return { + "pending": pending, + "promoted": promoted, + "variant_table": { + "harness_version": table.get("harness_version"), + "operators": len(table.get("operators") or {}), + "groups": table.get("groups") or {}, + "last_promoted_at": table.get("last_promoted_at"), + }, + } + + @router.post("/promotion/approve") + async def promotion_approve(task_id: str, request: Request) -> Dict[str, Any]: + """Human approval: publish the pending harness into DKAO.""" + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + workspace_dir = workspace_dir_for(entry) + try: + body = await request.json() + except Exception: # noqa: BLE001 - body is optional + body = {} + by = str((body or {}).get("by") or "operator") + result = _approve_with_config(entry, workspace_dir, by) + return {"ok": bool(result.get("ok")), "result": result, + "approved_by": by} + + @router.post("/promotion/deny") + async def promotion_deny(task_id: str, request: Request) -> Dict[str, Any]: + """Reject the pending candidate; the run keeps iterating.""" + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + workspace_dir = workspace_dir_for(entry) + try: + body = await request.json() + except Exception: # noqa: BLE001 + body = {} + reason = str((body or {}).get("reason") or "operator") + result = _deny_with_config(entry, workspace_dir, reason) + return {"ok": bool(result.get("ok")), "result": result} + + @router.get("/supervisor") + def supervisor(task_id: str) -> Dict[str, Any]: + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + return _supervisor_payload(workspace_dir_for(entry)) + + @router.post("/supervisor/stop") + def supervisor_stop(task_id: str) -> Dict[str, Any]: + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + workspace_dir = workspace_dir_for(entry) + (workspace_dir / "supervisor.stop").write_text( + str(time.time()), encoding="utf-8") + pid = _supervisor_running(workspace_dir) + if pid: + try: + os.kill(pid, 15) + except OSError: + pass + return {"ok": True, "stopped_pid": pid} + + @router.post("/stop") + def stop(task_id: str) -> Dict[str, Any]: + """Stop the whole run, leaving it restartable. + + Stopping is not killing: this is the operator's "pause". Everything + that would keep the round moving is brought down -- the incident + supervisor (so it cannot restart what we are stopping), the running + DKAO children, and the HE orchestrator itself -- and the run is left + explicitly *finished*, so no automatic path revives it while the + artifacts stay intact for a later resume. + + Killing instead would be indistinguishable from a crash, which is + exactly what ``should_launch_resume`` and the janitor sweep exist to + recover from: they would restart the task seconds later. + + Order matters, and the order is the one below: marker (1) so no sweep + revives the run while it is being torn down; supervisor (2), children + (3), orchestrator (4), one more child sweep for the env-retry wave the + orchestrator may have launched while dying (5), and only then the + terminal state (6). A live orchestrator never reads the marker itself, + so skipping (4) leaves a process that answers the loss of its children + by re-launching them. + """ + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + state_dir = state_dir_for(entry) + workspace_dir = workspace_dir_for(entry) + run_before = read_run(state_dir) + iteration = run_before.get("current_iteration") + + # 1. The marker first: it is what makes this stop stick. A concurrent + # janitor tick must see it before it decides to relaunch anything. + stopped_at = time.time() + _operator_stop_path(workspace_dir).write_text(json.dumps({ + "stopped_at": stopped_at, + "by": "webui", + "iteration": iteration, + "phase": run_before.get("current_phase"), + }, indent=2), encoding="utf-8") + + # 2. The supervisor, before the orchestrator: it exists to restart a + # dead experiment, and would do exactly that to the one we stop. + (workspace_dir / "supervisor.stop").write_text( + str(stopped_at), encoding="utf-8") + supervisor_pid = _supervisor_running(workspace_dir) + if supervisor_pid: + try: + os.kill(supervisor_pid, 15) + except OSError: + pass + + # 3. The children, before the parent: the parent's own shutdown would + # otherwise leave them (and their agents) running. + children = _live_dkao_children(workspace_dir) + stopped_children = [c["pid"] for c in children + if _terminate_pid(c["pid"])] + + # 4. The orchestrator itself, now that nothing below it is mid-write. + # ``_run_alive`` and NOT ``_resume_running``: the latter reads only + # ``resume.pid``, which a task created from the New Task form never + # writes -- so Stop answered "stopped" while the orchestrator kept + # running, and immediately re-launched the children it had just lost + # as an env retry. ``_run_alive`` reads ``orchestrator.pid``, + # ``resume.pid`` and finally scans cmdlines; the janitor sweep learned + # this same lesson first (see its docstring). + orchestrator_pid = _run_alive(state_dir) + stopped_ok = not orchestrator_pid or _terminate_pid(orchestrator_pid) + + # 5. Once more for the retry wave. A question whose child vanished is an + # environment failure to the evaluator, which answers it with a fresh + # child (``env_retry_attempts``) -- and it can launch that wave in the + # seconds between step 3 and step 4. Miss it and the UI says "stopped" + # while a brand-new DKAO child optimizes on a card for hours. + already = set(stopped_children) + relaunched = [c["pid"] for c in _live_dkao_children(workspace_dir) + if c["pid"] not in already] + stopped_children += [pid for pid in relaunched if _terminate_pid(pid)] + + # 6. Make it visible and final: a round cut short must not look like a + # round that finished, and ``finished`` is what tells every + # automatic path to keep its hands off. + run = read_run(state_dir) + notes = list(run.get("notes") or []) + if "Stopped by operator." not in notes: + notes.append("Stopped by operator.") + run.update({ + "finished": True, + "final_status": "stopped_by_operator", + "last_outcome": "stopped_by_operator", + "last_transition_label": "operator stop", + "last_update": stopped_at, + "notes": notes, + }) + tmp = state_dir / ".run.json.stop.tmp" + tmp.write_text(json.dumps(run, indent=2), encoding="utf-8") + os.replace(tmp, state_dir / "run.json") + + # A stale resume.pid would make /resume answer "already running" about + # a process we just killed. + try: + (state_dir / "resume.pid").unlink() + except OSError: + pass + + return { + "ok": True, + "stopped": stopped_ok, + "stopped_children": stopped_children, + "relaunched_children": relaunched, + "stopped_supervisor_pid": supervisor_pid, + "stopped_orchestrator_pid": orchestrator_pid, + "iteration": iteration, + "message": ("已停止；产物保留，点「继续运行」可从当前进度接着跑" + if stopped_ok else + "停止信号已发送，但编排进程未及时退出；可再点一次"), + } + + @router.get("/dkao/{num}/{child}") + def dkao_child(task_id: str, num: int, child: str) -> Dict[str, Any]: + """DKAO-like detail of one HE question (inner 8-step machine etc.).""" + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + return _dkao_child_payload(workspace_dir_for(entry), num, child) + + @router.get("/iterations/{num}/overview") + def overview(task_id: str, num: int) -> str: + entry = task_or_404(task_id) + require_task_type(entry, PLUGIN_TYPE) + path = ( + workspace_dir_for(entry) / "runs" + / f"iteration_{num:03d}" / "input" / "analysis" / "overview.md" + ) + text = _text(path) + if text is None: + raise HTTPException(404, "overview not found") + return text + + return router diff --git a/metainfer/tasks/harness_evolve/static/dkao.css b/metainfer/tasks/harness_evolve/static/dkao.css new file mode 100644 index 00000000..55f7e00f --- /dev/null +++ b/metainfer/tasks/harness_evolve/static/dkao.css @@ -0,0 +1,909 @@ +.dkao-detail { + display: grid; + grid-template-columns: minmax(0, 2fr) minmax(320px, 1fr); + gap: var(--sp-3); + align-items: start; +} + +.dkao-state-panel, +.dkao-iterations-panel { + grid-column: 1 / -1; +} + +.dkao-state-machine { + display: flex; + align-items: center; + gap: 10px; + padding: 24px; + overflow-x: auto; +} + +.dkao-state-step { + min-width: 132px; + padding: 13px 16px; + border: 1px solid var(--border); + border-radius: var(--radius); + background: var(--bg); + color: var(--fg-dim); + display: flex; + align-items: center; + gap: 9px; + white-space: nowrap; +} + +.dkao-state-step.active { + color: var(--fg); + border-color: var(--accent); + background: rgba(88, 166, 255, 0.13); + box-shadow: 0 0 0 1px rgba(88, 166, 255, 0.2); +} + +.dkao-state-step.terminal.active { + border-color: var(--ok); + background: rgba(63, 185, 80, 0.12); +} + +.dkao-state-index { + width: 23px; + height: 23px; + border-radius: 50%; + background: var(--panel-2); + color: var(--accent-2); + display: inline-flex; + align-items: center; + justify-content: center; + font-size: var(--fs-xs); + font-weight: 700; +} + +.dkao-state-arrow { + color: var(--muted); + font-size: 20px; +} + +.dkao-explore-summary { + padding: 0 24px 24px; +} + +.dkao-explore-heading { + margin: 0 0 10px; + color: var(--fg-dim); + font-size: var(--fs-xs); + letter-spacing: 0.04em; + text-transform: uppercase; +} + +.dkao-explore-grid { + display: grid; + grid-template-columns: repeat(4, minmax(0, 1fr)); + gap: 10px; +} + +.dkao-explore-gpu { + min-width: 0; + padding: 12px; + border: 1px solid var(--border-soft); + border-radius: var(--radius); + background: var(--bg); +} + +.dkao-explore-gpu.stale { + border-color: var(--warn); +} + +.dkao-explore-gpu > header { + display: flex; + align-items: flex-start; + justify-content: space-between; + gap: 8px; +} + +.dkao-explore-gpu > header > div { + display: flex; + min-width: 0; + flex-direction: column; + gap: 2px; +} + +.dkao-explore-gpu > header strong { + color: var(--accent-2); +} + +.dkao-explore-gpu > header small { + overflow: hidden; + color: var(--muted); + font-family: var(--fs-mono); + text-overflow: ellipsis; + white-space: nowrap; +} + +.dkao-explore-step { + margin: 12px 0 8px; + color: var(--fg); + font-size: var(--fs-sm); +} + +.dkao-explore-gpu dl { + display: grid; + grid-template-columns: 56px minmax(0, 1fr); + gap: 4px 8px; + margin: 0; + font-size: var(--fs-xs); +} + +.dkao-explore-gpu dt { + color: var(--muted); +} + +.dkao-explore-gpu dd { + margin: 0; + overflow: hidden; + color: var(--fg-dim); + font-family: var(--fs-mono); + text-overflow: ellipsis; + white-space: nowrap; +} + +.dkao-explore-progress { + height: 5px; + margin-top: 11px; + overflow: hidden; + border-radius: 3px; + background: var(--panel-2); +} + +.dkao-explore-progress > span { + display: block; + height: 100%; + border-radius: inherit; + background: var(--accent); + transition: width 180ms ease; +} + +.dkao-explore-warning { + margin: 8px 0 0; + color: var(--warn); + font-size: var(--fs-xxs); +} + +.dkao-worker-lanes { + display: flex; + flex-direction: column; +} + +.dkao-worker-lane { + display: grid; + grid-template-columns: 125px minmax(280px, 0.9fr) minmax(340px, 1.1fr); + min-height: 132px; + border-bottom: 1px solid var(--border-soft); +} + +.dkao-worker-lane:last-child { + border-bottom: 0; +} + +.dkao-worker-lane.unassigned { + opacity: 0.65; +} + +.dkao-worker-meta { + padding: 10px; + border-right: 1px solid var(--border-soft); + background: var(--bg-elev); + display: flex; + flex-direction: column; + align-items: flex-start; + gap: 5px; +} + +.dkao-worker-meta strong { + color: var(--accent-2); + font-family: var(--fs-mono); +} + +.dkao-worker-meta small { + color: var(--muted); + word-break: break-word; +} + +.dkao-restart-worker { + width: 100%; + margin-top: auto; + padding: 5px 7px; + border-color: var(--warn); + color: var(--warn); + font-size: var(--fs-xs); +} + +.dkao-iteration-strip { + display: flex; + gap: 7px; + padding: 7px 8px 9px; + overflow-x: auto; + min-height: 89px; + scrollbar-width: thin; +} + +.dkao-iteration-card { + flex: 0 0 148px; + border: 1px solid var(--border-soft); + border-radius: var(--radius-sm); + background: var(--bg); + padding: 7px 8px; +} + +.dkao-iteration-card.accepted { + border-color: rgba(63, 185, 80, 0.55); +} + +.dkao-iteration-card.running { + border-color: rgba(88, 166, 255, 0.7); + background: rgba(88, 166, 255, 0.06); +} + +.dkao-running-pulse { + height: 4px; + margin-top: 7px; + overflow: hidden; + border-radius: 2px; + background: var(--panel-2); +} + +.dkao-running-pulse > span { + display: block; + width: 42%; + height: 100%; + border-radius: inherit; + background: var(--accent); + animation: dkao-iteration-running 1.25s ease-in-out infinite alternate; +} + +@keyframes dkao-iteration-running { + from { transform: translateX(0); } + to { transform: translateX(138%); } +} + +.dkao-iteration-title { + display: flex; + align-items: center; + justify-content: space-between; + margin-bottom: 5px; +} + +.dkao-iteration-card dl { + display: grid; + grid-template-columns: 44px 1fr; + gap: 2px 5px; + margin: 0; + font-size: var(--fs-xs); +} + +.dkao-iteration-card dt { + color: var(--muted); +} + +.dkao-iteration-card dd { + margin: 0; + color: var(--fg-dim); + font-family: var(--fs-mono); +} + +.dkao-lane-section { + min-width: 0; + border-right: 1px solid var(--border-soft); + display: flex; + flex-direction: column; +} + +.dkao-lane-section:last-child { + border-right: 0; +} + +.dkao-lane-heading { + min-height: 27px; + padding: 5px 9px; + border-bottom: 1px solid var(--border-soft); + display: flex; + justify-content: space-between; + color: var(--fg-dim); + font-size: var(--fs-xs); +} + +.dkao-lane-heading span { + color: var(--muted); +} + +.dkao-plan-strip { + display: flex; + gap: 7px; + padding: 7px 8px 9px; + overflow-x: auto; + min-height: 89px; + scrollbar-width: thin; +} + +.dkao-plan-card { + flex: 0 0 190px; + border: 1px solid var(--border-soft); + border-radius: var(--radius-sm); + background: var(--bg); + padding: 7px 8px; +} + +.dkao-plan-card.manual, +.dkao-plan-card.guidance.pending { + border-color: rgba(88, 166, 255, 0.55); +} + +.dkao-plan-title { + display: flex; + align-items: center; + justify-content: space-between; + margin-bottom: 5px; +} + +.dkao-plan-card p { + margin: 0 0 4px; + color: var(--fg-dim); + font-size: var(--fs-xs); + line-height: 1.35; + display: -webkit-box; + -webkit-box-orient: vertical; + -webkit-line-clamp: 2; + overflow: hidden; +} + +.dkao-plan-card small { + color: var(--muted); + display: block; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} + +.dkao-guidance-form { + display: flex; + gap: 6px; + padding: 0 8px 7px; +} + +.dkao-guidance-form input { + min-width: 0; + flex: 1; + padding: 6px 8px; + font-size: var(--fs-xs); +} + +.dkao-guidance-form button { + flex: 0 0 auto; + padding: 5px 9px; + font-size: var(--fs-xs); +} + +.dkao-guidance-message { + color: var(--accent-2); + padding: 0 8px 6px; +} + +.dkao-validation-subtitle { + display: flex; + align-items: baseline; + justify-content: space-between; + gap: 12px; + margin: 0 0 10px; + color: var(--fg-dim); + font-size: var(--fs-xs); + letter-spacing: 0.04em; + text-transform: uppercase; +} + +.dkao-validation-subtitle span { + color: var(--muted); + font-weight: 400; + letter-spacing: 0; + text-transform: none; +} + +.dkao-events { + max-height: 320px; + overflow: auto; + padding: 8px 14px; +} + +.dkao-events > div { + display: grid; + grid-template-columns: 150px 1fr; + gap: 12px; + padding: 7px 0; + border-bottom: 1px solid var(--border-soft); + font-size: var(--fs-xs); +} + +.dkao-events code { + color: var(--accent-2); +} + +.dkao-events span { + color: var(--muted); + word-break: break-word; +} + +.dkao-skill-entry, +.dkao-skill-library { + grid-column: 1 / -1; +} + +.dkao-skill-entry, +.dkao-skill-library-title { + display: flex; + align-items: center; + justify-content: space-between; + gap: 18px; + padding: 14px 16px; +} + +.dkao-skill-entry h2, +.dkao-skill-entry p, +.dkao-skill-library-title h2, +.dkao-skill-library-title p { + margin: 0; +} + +.dkao-skill-columns { + display: grid; + grid-template-columns: 1fr 1fr; + min-height: 360px; + border-top: 1px solid var(--border-soft); +} + +.dkao-skill-column { + min-width: 0; + padding: 14px; +} + +.dkao-skill-column + .dkao-skill-column { + border-left: 1px solid var(--border-soft); +} + +.dkao-skill-column h3 { + display: flex; + justify-content: space-between; + margin: 0; +} + +.dkao-skill-column h3 span { + color: var(--muted); + font-family: var(--fs-mono); +} + +.dkao-skill-column > p { + min-height: 20px; + margin: 4px 0 10px; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.dkao-skill-list { + max-height: 560px; + overflow: auto; + display: flex; + flex-direction: column; + gap: 7px; +} + +.dkao-skill-file { + border: 1px solid var(--border-soft); + border-radius: var(--radius-sm); + background: var(--bg); +} + +.dkao-skill-file summary { + cursor: pointer; + padding: 9px 10px; + display: flex; + align-items: center; + justify-content: space-between; + gap: 10px; +} + +.dkao-skill-file summary > span:first-child { + min-width: 0; + display: flex; + flex-direction: column; +} + +.dkao-skill-file summary strong, +.dkao-skill-file summary small { + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.dkao-skill-file summary small { + color: var(--muted); +} + +.dkao-skill-file pre { + max-height: 380px; + overflow: auto; + margin: 0; + padding: 12px; + border-top: 1px solid var(--border-soft); + color: var(--fg-dim); + white-space: pre-wrap; + word-break: break-word; + font-size: var(--fs-xs); +} + +@media (max-width: 1100px) { + .dkao-detail { + grid-template-columns: 1fr; + } + + .dkao-validation-panel, + .dkao-events-panel { + grid-column: 1; + } + + .dkao-worker-lane { + grid-template-columns: 115px minmax(260px, 1fr) minmax(320px, 1fr); + overflow-x: auto; + } + + .dkao-skill-columns { + grid-template-columns: 1fr; + } + + .dkao-skill-column + .dkao-skill-column { + border-left: 0; + border-top: 1px solid var(--border-soft); + } +} + +/* ---- shape input component ------------------------------------------------ */ + +.dkao-shape-input { + border: 1px solid var(--border); + border-radius: var(--radius); + overflow: hidden; +} + +.dkao-shape-tabs { + display: flex; + align-items: center; + padding: 6px; + background: var(--panel-2); + border-bottom: 1px solid var(--border); + gap: 4px; +} + +.dkao-shape-tab { + padding: 5px 11px; + border: 1px solid transparent; + border-radius: var(--radius-sm); + background: transparent; + color: var(--fg-dim); + font-size: var(--fs-xs); + cursor: pointer; + transition: background 0.12s; +} + +.dkao-shape-tab:hover:not(:disabled) { + background: var(--bg); + color: var(--fg); +} + +.dkao-shape-tab.active { + background: var(--bg); + color: var(--fg); + border-color: var(--border); + box-shadow: 0 1px 2px rgba(0,0,0,.06); +} + +.dkao-shape-tab:disabled { + opacity: .45; + cursor: default; +} + +.dkao-shape-spacer { + width: 12px; +} + +.dkao-shape-mode-note { + margin-left: auto; + padding-right: 6px; + color: var(--muted); + font-size: var(--fs-xxs); +} + +.dkao-shape-body { + padding: 10px; +} + +.dkao-shape-body textarea.input { + margin: 0; + border: 0; + border-radius: 0; + resize: vertical; +} + +.dkao-guided-form { + display: flex; + flex-direction: column; + gap: 10px; +} + +.dkao-guided-info { + padding: 9px 11px; + background: var(--bg-elev); + border-radius: var(--radius-sm); + border: 1px solid var(--border-soft); + color: var(--fg-dim); + font-size: var(--fs-xs); + line-height: 1.45; +} + +.dkao-guided-info strong { + color: var(--accent-2); +} + +.dkao-guided-row { + display: flex; + align-items: baseline; + gap: 8px; + flex-wrap: wrap; +} + +.dkao-guided-label { + flex: 0 0 130px; + min-width: 100px; + font-size: var(--fs-xs); + color: var(--fg-dim); + font-family: var(--fs-mono); + text-align: right; + padding-top: 6px; +} + +.dkao-guided-input { + flex: 1 1 160px; + min-width: 120px; + font-family: var(--fs-mono); +} + +.dkao-guided-hint { + flex: 1 1 200px; + min-width: 140px; + font-size: var(--fs-xxs); + color: var(--muted); + padding-top: 6px; +} + +.dkao-guided-check { + display: flex; + align-items: center; + gap: 6px; + font-size: var(--fs-xs); + color: var(--fg-dim); + cursor: pointer; + padding-left: 138px; +} + +.dkao-manual-assignment { + display: flex; + flex-direction: column; + gap: 10px; +} + +.dkao-assignment-actions { + display: flex; + align-items: center; + flex-wrap: wrap; + gap: 5px; + color: var(--muted); + font-size: var(--fs-xxs); +} + +.dkao-shape-catalog { + overflow: hidden; + border: 1px solid var(--border-soft); + border-radius: var(--radius-sm); +} + +.dkao-shape-catalog-header { + display: flex; + align-items: center; + justify-content: space-between; + padding: 8px 10px; + background: var(--panel-2); + font-size: var(--fs-xs); +} + +.dkao-shape-catalog-header span { + color: var(--muted); + font-size: var(--fs-xxs); +} + +.dkao-tp-filter { + display: flex; + align-items: center; + gap: 4px; + padding: 6px 10px; + border-bottom: 1px solid var(--border-soft); + background: var(--panel-2); +} + +.dkao-tp-filter-label { + margin-right: 4px; + color: var(--muted); + font-size: var(--fs-xxs); + letter-spacing: 0.05em; + text-transform: uppercase; +} + +.dkao-shape-catalog-items { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 5px; + padding: 7px; +} + +.dkao-shape-choice { + display: flex; + align-items: center; + min-width: 0; + gap: 7px; + padding: 7px 8px; + border: 1px solid var(--border-soft); + border-radius: var(--radius-sm); + color: var(--fg-dim); + cursor: pointer; +} + +.dkao-shape-choice.selected { + border-color: var(--accent); + background: color-mix(in srgb, var(--accent) 7%, var(--bg)); + color: var(--fg); +} + +.dkao-shape-choice > span { + display: flex; + min-width: 0; + flex-direction: column; + gap: 2px; +} + +.dkao-shape-choice strong, +.dkao-shape-choice small { + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.dkao-shape-choice strong { + font-family: var(--fs-mono); + font-size: var(--fs-xxs); +} + +.dkao-shape-choice small { + color: var(--muted); + font-size: 10px; +} + +.dkao-gpu-grid { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 8px; +} + +.dkao-gpu-card { + min-width: 0; + overflow: hidden; + border: 1px solid var(--border-soft); + border-radius: var(--radius-sm); + background: var(--bg-elev); +} + +.dkao-gpu-card > header { + display: flex; + align-items: center; + justify-content: space-between; + padding: 8px 10px; + border-bottom: 1px solid var(--border-soft); + background: var(--panel-2); + font-size: var(--fs-xs); +} + +.dkao-gpu-card > header span { + color: var(--muted); + font-size: var(--fs-xxs); +} + +.dkao-gpu-shapes { + display: flex; + min-height: 54px; + flex-direction: column; + gap: 6px; + padding: 7px; +} + +.dkao-gpu-shape { + display: flex; + align-items: center; + justify-content: space-between; + gap: 8px; + min-width: 0; + padding: 7px 8px; + border: 1px solid var(--border-soft); + border-radius: var(--radius-sm); + background: var(--bg); +} + +.dkao-gpu-shape > div { + display: flex; + min-width: 0; + flex-direction: column; + gap: 2px; +} + +.dkao-gpu-shape strong, +.dkao-gpu-shape small { + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +.dkao-gpu-shape strong { + font-family: var(--fs-mono); + font-size: var(--fs-xxs); +} + +.dkao-gpu-shape small { + color: var(--muted); + font-size: 10px; +} + +.dkao-gpu-shape select { + max-width: 72px; + padding: 3px 4px; + border: 1px solid var(--border); + border-radius: var(--radius-sm); + background: var(--panel-2); + color: var(--fg); + font-size: var(--fs-xxs); +} + +.dkao-gpu-empty { + display: grid; + min-height: 38px; + place-items: center; + color: var(--muted); + font-size: var(--fs-xxs); +} + +@media (max-width: 760px) { + .dkao-shape-catalog-items, + .dkao-gpu-grid, + .dkao-explore-grid { + grid-template-columns: 1fr; + } +} + +@media (min-width: 761px) and (max-width: 1180px) { + .dkao-explore-grid { + grid-template-columns: repeat(2, minmax(0, 1fr)); + } +} + +/* Kernel repository rename panel */ +.dkao-repo-panel .dkao-repo-actions { + display: flex; + align-items: center; + gap: 12px; + margin-top: 6px; +} + +.dkao-repo-panel .dkao-rename-msg { + color: var(--muted); + font-size: var(--fs-xs); +} diff --git a/metainfer/tasks/harness_evolve/static/he-api.js b/metainfer/tasks/harness_evolve/static/he-api.js new file mode 100644 index 00000000..975f6b55 --- /dev/null +++ b/metainfer/tasks/harness_evolve/static/he-api.js @@ -0,0 +1,142 @@ +// Fetch helpers for harness_evolve's task-specific endpoints. +// Mounted by the shell at /api/harness-evolve//*. + +const TASK_BASE = (taskId) => + `/api/harness-evolve/${encodeURIComponent(taskId)}`; + +export async function getSummary(taskId) { + const r = await fetch(`${TASK_BASE(taskId)}/summary`); + if (!r.ok) throw new Error(`summary: ${r.status}`); + return r.json(); +} + +export async function getIterations(taskId) { + const r = await fetch(`${TASK_BASE(taskId)}/iterations`); + if (!r.ok) throw new Error(`iterations: ${r.status}`); + return r.json(); +} + +export async function getStateGraph(taskId) { + const r = await fetch(`${TASK_BASE(taskId)}/state-graph`); + if (!r.ok) throw new Error(`state-graph: ${r.status}`); + return r.json(); +} + +export async function getChildren(taskId, n) { + const r = await fetch(`${TASK_BASE(taskId)}/iterations/${n}/children`); + if (!r.ok) throw new Error(`children ${n}: ${r.status}`); + return r.json(); +} + +export async function getDkaoChild(taskId, n, child) { + const r = await fetch(`${TASK_BASE(taskId)}/dkao/${n}/${encodeURIComponent(child)}`); + if (!r.ok) throw new Error(`dkao ${child}: ${r.status}`); + return r.json(); +} + +// Continue the experiment toward a target round count (dynamic: call again +// while it runs to raise/lower the target). +export async function resumeExperiment(taskId, target = null, iterations = null, + championIteration = null) { + const body = {}; + if (target != null) body.target = Number(target); + if (iterations != null) body.iterations = Number(iterations); + if (championIteration != null) body.champion_iteration = Number(championIteration); + const r = await fetch(`${TASK_BASE(taskId)}/resume`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + }); + const payload = await r.json().catch(() => ({})); + if (!r.ok) throw new Error(payload.detail || `resume: ${r.status}`); + return payload; +} + +export async function getOverview(taskId, n) { + const r = await fetch(`${TASK_BASE(taskId)}/iterations/${n}/overview`); + if (!r.ok) throw new Error(`overview ${n}: ${r.status}`); + return r.text(); +} + +// Stop the run and leave it restartable: brings down the supervisor, the +// running DKAO children and the orchestrator, and records an operator stop so +// nothing restarts it automatically. Resume ("启动续跑至目标") clears that. +export async function stopExperiment(taskId) { + const r = await fetch(`${TASK_BASE(taskId)}/stop`, { method: "POST" }); + const payload = await r.json().catch(() => ({})); + if (!r.ok) throw new Error(payload.detail || `stop: ${r.status}`); + return payload; +} + +// Unattended mode: run to a target round with automatic incident handling. +export async function startAutopilot(taskId, target = null, + championIteration = null, + diagnose = true) { + const body = { diagnose_agent: Boolean(diagnose) }; + if (target != null) body.target = Number(target); + if (championIteration != null) body.champion_iteration = Number(championIteration); + const r = await fetch(`${TASK_BASE(taskId)}/autopilot`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(body), + }); + const payload = await r.json().catch(() => ({})); + if (!r.ok) throw new Error(payload.detail || `autopilot: ${r.status}`); + return payload; +} + +export async function getSupervisor(taskId) { + const r = await fetch(`${TASK_BASE(taskId)}/supervisor`); + if (!r.ok) throw new Error(`supervisor: ${r.status}`); + return r.json(); +} + +export async function stopSupervisor(taskId) { + const r = await fetch(`${TASK_BASE(taskId)}/supervisor/stop`, { method: "POST" }); + const payload = await r.json().catch(() => ({})); + if (!r.ok) throw new Error(payload.detail || `stop: ${r.status}`); + return payload; +} + +export async function getReview(taskId) { + const r = await fetch(`${TASK_BASE(taskId)}/review`); + if (!r.ok) throw new Error(`review: ${r.status}`); + return r.json(); +} + +// Read-only device cards for one iteration: which device each question was +// handed to, plus what that child's own DKAO admission gate says. +export async function getIterationGpu(taskId, n) { + const r = await fetch(`${TASK_BASE(taskId)}/iterations/${n}/gpu`); + if (!r.ok) throw new Error(`gpu ${n}: ${r.status}`); + return r.json(); +} + +// Human approval of a candidate harness (nothing reaches DKAO without it). +export async function getPromotion(taskId) { + const r = await fetch(`${TASK_BASE(taskId)}/promotion`); + if (!r.ok) throw new Error(`promotion: ${r.status}`); + return r.json(); +} + +export async function approvePromotion(taskId, by = "operator") { + const r = await fetch(`${TASK_BASE(taskId)}/promotion/approve`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ by }), + }); + const payload = await r.json().catch(() => ({})); + if (!r.ok) throw new Error(payload.detail || `approve: ${r.status}`); + return payload; +} + +export async function denyPromotion(taskId, reason = "operator") { + const r = await fetch(`${TASK_BASE(taskId)}/promotion/deny`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ reason }), + }); + const payload = await r.json().catch(() => ({})); + if (!r.ok) throw new Error(payload.detail || `deny: ${r.status}`); + return payload; +} diff --git a/metainfer/tasks/harness_evolve/static/he-detail.js b/metainfer/tasks/harness_evolve/static/he-detail.js new file mode 100644 index 00000000..cede7a84 --- /dev/null +++ b/metainfer/tasks/harness_evolve/static/he-detail.js @@ -0,0 +1,1346 @@ +// harness_evolve task detail body (AHE experiment viewer). +// Rendered by the task-detail shell when detail_view_module === "app/he-detail". +// Read-only: fetches /api/harness-evolve//* and renders summary + per +// iteration benchmark / diff / change evaluation / overview / rollback. + +import { html } from "htm/preact"; +import { useCallback, useEffect, useMemo, useState } from "preact/hooks"; +import { approvePromotion, denyPromotion, getChildren, getDkaoChild, getIterationGpu, getIterations, getPromotion, getReview, getStateGraph, getSummary, getSupervisor, resumeExperiment, startAutopilot, stopExperiment, stopSupervisor } from "app/he-api"; +// Shared polling hook: a hidden tab must not keep hammering the server, and a +// round is only polled while it can still change (see usePollWhileVisible). +import { usePollWhileVisible as pollWhileVisible } from "app/utils"; +// The DKAO task page itself, reused as a read-only mirror of one question. +// Reused rather than reimplemented so the two views cannot drift: the same +// component renders a child here and renders a standalone DKAO task. +import DcuKernelAutoOptDetail from "app/dkao-detail"; + +const withTimeout = (p, ms = 8000) => + Promise.race([ + p, + new Promise((_, rej) => setTimeout(() => rej(new Error("timeout")), ms)), + ]); + +// Latency formatting for the round table, which has no local helper of its own. +const fmtUs = (v) => (v == null ? "-" : (v.toFixed ? v.toFixed(2) : String(v))); + +// The variant comparison's verdict (FLOW.md §3), not the round's PASS/FAIL. +// A tie is deliberately not a win: the performance gate counts only a strict +// win (outside the 2% noise band), so "平" is a distinct, visible outcome. +function GateVerdictBadge({ verdict }) { + const cls = { WIN: "ok", LOSS: "bad", NO_SIGNAL: "warn" }[verdict] || "warn"; + const label = { WIN: "赢", LOSS: "输", NO_SIGNAL: "平·不算赢" }[verdict] + || verdict || "-"; + return html`${label}`; +} + +// Where the number a question must beat came from. "variant" = the experiment's +// own variant table, which round 1 seeds from the pool's best_known before any +// kernel is measured here — say which, so "已记录的 variant" is never read as +// "measured by this run" when it was not. +const VARIANT_SOURCE_LABEL = { + best_known: "池子 best_known", + baseline: "固定 Triton baseline（尚无 variant，退到基线）", +}; + +function variantSourceLabel(gate) { + if (!gate) return "-"; + if (gate.variant_source && gate.variant_source !== "variant") { + return VARIANT_SOURCE_LABEL[gate.variant_source] || gate.variant_source; + } + return String(gate.kernel_source || "").startsWith("pool:") + ? "本实验 variant（由池子 best_known 播种）" + : "本实验已记录的 variant"; +} + +// The round's gate line: which stage this round is, which gate it owed, and what +// that gate concluded. Read from the round's own gate document, never inferred. +function RoundGateLine({ iter }) { + const stage = iter && iter.stage && iter.stage.stage; + const gate = (iter && (iter.generalization_gate || iter.performance_gate)) + || null; + if (!stage) { + return html`
+ 本轮无 stage.json：legacy / suite 路径，未按固定协议判性能门 +
`; + } + const KIND = { baseline: "基线轮（不判门）", performance: "性能门", + generalization: "泛化门" }; + const cls = !gate ? "run" + : (gate.status === "PASS" ? "ok" + : (gate.status === "INCOMPLETE" ? "warn" : "bad")); + return html`
+ stage ${stage} · ${KIND[stage] || stage} + ${gate + ? html` · 结论 ${gate.status} + ${(gate.wins || []).length}/${(gate.operators || []).length} 赢（需 ${gate.wins_needed}） + ${(gate.below_floor || []).length + ? html` · 低于地板：${gate.below_floor.join(", ")}` : null} + ${(gate.unmeasured || []).length + ? html` · 未测到：${gate.unmeasured.join(", ")}` : null}` + : html` · 尚未判门（第 1 轮只量基线，不判门）`} +
`; +} + +// One question's real gate cell: the variant it must beat, plus the verdict when +// the round has been judged. The pool criterion (tau_family * baseline) is NOT +// the variant-round gate, so it is only ever shown as a labelled annotation. +function QuestionGateCell({ gate, poolRef }) { + if (!gate) return html`-`; + const poolLine = (poolRef && poolRef.target_us != null) ? html` +
+ 池子口径参考 ≤${fmtUs(poolRef.target_us)} µs + （τ ${fmtUs(poolRef.tau_family)} × baseline；本协议不判门） +
` : null; + if (gate.protocol === "legacy") { + // Rounds before 2026-09-15 really were judged this way: say so plainly. + const need = (poolRef && poolRef.baseline_us && poolRef.target_us) + ? (poolRef.baseline_us / poolRef.target_us) : null; + return html` +
≤${fmtUs(gate.target_us)} µs
+ legacy 判据${need ? ` · ≥${need.toFixed(2)}× baseline` : ""} + `; + } + const judged = gate.judged; + const src = variantSourceLabel(gate); + const title = `性能门对手 = 该算子当前 variant ${fmtUs(gate.variant_us)} µs（来源：${src}）` + + ` · 赢 = 严格快 ${gate.noise_percent}% 以上（平局不算赢）` + + ` · 地板 = variant 的 ${Math.round((gate.floor_percent || 0.8) * 100)}%` + + (judged ? ` · 本轮实测 ${fmtUs(judged.candidate_median_us)} µs` : "") + + (poolRef && poolRef.target_us != null + ? ` · 池子口径 τ×baseline = ${fmtUs(poolRef.target_us)} µs（本协议不判门）` : ""); + return html` +
≤${fmtUs(gate.variant_us)} µs variant
+ ${judged + ? html`<${GateVerdictBadge} verdict=${judged.verdict} /> + ${judged.delta_percent > 0 ? "+" : ""}${fmtUs(judged.delta_percent)}%` + : html`${gate.stage === "baseline" + ? "第 1 轮不判门 · 本轮定下的对手" + : "本轮尚未判门"}`} + ${poolLine} + `; +} + +// Outer-loop AHE step machine (Prepare -> Evaluate -> ... -> Finished). +// Uses the same classes as the DKAO page (.dkao-state-*) for identical looks. +function StateMachine({ graph }) { + const nodes = graph?.nodes || []; + if (!nodes.length) return html`

Waiting for the control plane.

`; + return html` +
+ ${nodes.map((node, index) => html` +
+ ${index + 1} + ${node.label} +
+ ${index < nodes.length - 1 ? html`→` : null} + `)} +
+ `; +} + +// Inner DKAO phases (each question IS one DKAO task running its own loop). +const DKAO_PHASE_LABEL = { + idle: "…", + prepare: "Prepare", + generate_kernel_repo: "Generate kernel repo", + baseline: "Baseline", + parallel_explore: "Parallel explore", + skill_synthesis: "Skill synthesis", + serial_validate: "Serial validate", + report: "Report", + finished: "Finished", +}; + +function DkaoPhaseChip({ child }) { + if (!child) return html`no child yet`; + const phase = child.current_phase || "idle"; + if (child.finished) { + const cls = child.final_status === "success" ? "ok" : "bad"; + return html`finished${child.final_status ? ` · ${child.final_status}` : ""}`; + } + return html`${DKAO_PHASE_LABEL[phase] || phase}`; +} + +function StateBadge({ state }) { + const s = state || ""; + const bad = /fail|error|reject|killed|timeout/.test(s); + const done = /pass|accept|success|complete|finish|done|publish/.test(s); + const cls = bad ? "bad" : done ? "ok" : "run"; + return html`${s || "-"}`; +} + +// Per-question inner DKAO view (renders the same way the DKAO detail page +// does: the 8-step DKAO state machine + worker lanes + agents + log). +function ChildDkaoView({ payload }) { + if (!payload) return html`

Loading DKAO child…

`; + const run = payload.run || {}; + return html` +
+
+

DKAO child — ${payload.child}

+
+ phase: ${DKAO_PHASE_LABEL[run.current_phase] || run.current_phase} + · iteration ${run.current_iteration ?? "-"} + ${run.finished ? html` · <${StateBadge} state=${run.final_status || "finished"} />` : ""} + · repo: ${payload.repo_path || "-"} +
+
+ <${StateMachine} graph=${payload.graph} /> +
+

Workers

+ ${payload.workers && payload.workers.length ? html` + + + + ${payload.workers.map((w) => html` + + + + + + + + + `)} + +
workerstateiterationshapeGPUroundsbest median us
${w.worker_id}<${StateBadge} state=${w.state} />${w.iteration ?? "-"}${w.shape_id || "-"}${w.physical_gpu ?? "-"}${w.rounds ?? 0}${w.best_median_us != null ? w.best_median_us : "-"}
` : html`

no worker lanes yet

`} + ${(() => { + const rounds = (payload.workers || []) + .flatMap((w) => (w.experiments || []).map((e) => ({ worker: w.worker_id, ...e }))) + .sort((a, b) => (a.iteration || 0) - (b.iteration || 0)); + if (!rounds.length) return null; + return html` +

Optimization rounds (accepted median per attempt, speedup vs the frozen Triton baseline)

+ + + + ${rounds.map((e) => html` + + + + + + + + + `)} + +
workeritermedian usp90 usbaseline usspeedupaccepted
${e.worker}${e.iteration ?? "-"}${e.median_us != null ? e.median_us.toFixed ? e.median_us.toFixed(2) : e.median_us : "—"}${e.p90_us != null ? (e.p90_us.toFixed ? e.p90_us.toFixed(2) : e.p90_us) : "—"}${e.baseline_us != null ? (e.baseline_us.toFixed ? e.baseline_us.toFixed(2) : e.baseline_us) : "—"}${e.speedup != null ? (typeof e.speedup === "number" ? `${e.speedup.toFixed(2)}×` : e.speedup) : "-"}${e.accepted ? html`accept` : html`reject`}
`; + })()} +
+
+

Agents

+ ${payload.agents && payload.agents.length ? html` + + + + ${payload.agents.map((a) => html` + + + + + + + + `)} + +
agentrolestatusitermedian usspeedup vs baseline
${a.name || "-"}${a.role || "-"}<${StateBadge} state=${a.status} />${a.iteration != null ? a.iteration : "-"}${a.median_us != null ? (a.median_us.toFixed ? a.median_us.toFixed(2) : a.median_us) : "-"}${a.speedup != null + ? html`${typeof a.speedup === "number" ? a.speedup.toFixed(2) + "×" : a.speedup}${a.accepted ? "" : " (rejected)"}` + : "-"}
` : html`

no agents yet

`} +
+ ${payload.log_tail ? html`
child orchestrator log + 
${payload.log_tail}
` : null} +
`; +} + +function VerdictBadge({ verdict }) { + const cls = { EFFECTIVE: "ok", PARTIALLY_EFFECTIVE: "warn", MIXED: "warn", + INEFFECTIVE: "bad", HARMFUL: "bad", BASELINE: "ok", + PROMOTE: "ok", PROMOTE_UNEXPLAINED: "ok", + CONFIRM_REQUIRED: "warn", NO_SIGNAL: "warn", SPECIALIZE: "warn", + REJECT: "bad", REJECT_OVERFIT: "bad" }[verdict] || "warn"; + return html`${verdict || "-"}`; +} + +function ScoreTable({ scores }) { + if (!scores || !scores.length) return null; + return html` + + + + ${scores.map((s) => html` + + `)} + +
iterationpasstotalrate
${s.iteration}${s.n_pass}${s.n_total}${(100 * (s.pass_rate || 0)).toFixed(1)}%
`; +} + +// --------------------------------------------------------------------------- // +// Live DKAO mirror: one iteration's questions, rendered by the DKAO page itself +// --------------------------------------------------------------------------- // +// Each question of an iteration is a real DKAO run. Those runs are spawned by +// the HE orchestrator's CLI and are absent from the task registry, so the +// embedded view is addressed by the child's artifact directory instead of a +// task id -- hence the `stateDir` prop, which the DKAO routes accept read-only. +// The switcher follows the still-running question by default, because that is +// the one the operator opened the iteration to watch. +function LiveDkaoMirror({ taskId, iter, initialQuestion = null }) { + const num = useMemo(() => { + const m = (iter.name || "").match(/(\d+)\s*$/); + return m ? Number(m[1]) : 0; + }, [iter.name]); + + const [children, setChildren] = useState(null); + const [sel, setSel] = useState(initialQuestion); + const [err, setErr] = useState(null); + //: How many DKAO pages this round shows at once. "grid" is the default: a + //: round is four questions on four cards, and comparing them is the point — + //: one-at-a-time behind tabs hid three quarters of the round. Each cell is the + //: same read-only DKAO page, pointed at that question's own state dir. + const [layout, setLayout] = useState("grid"); + + useEffect(() => { + if (!taskId || !num) return; + let alive = true; + withTimeout(getChildren(taskId, num)) + .then((c) => { if (alive) { setChildren(c || {}); setErr(null); } }) + .catch((e) => { if (alive) { setChildren({}); setErr(String(e)); } }); + return () => { alive = false; }; + }, [taskId, num]); + + const ids = useMemo(() => Object.keys(children || {}).sort(), [children]); + // Follow whichever question is still running; fall back to the first, and + // never let a stale selection survive a refresh that dropped it. + const selected = useMemo(() => { + if (sel && ids.includes(sel)) return sel; + const running = ids.find((id) => children[id] && !children[id].finished); + return running || ids[0] || null; + }, [ids, sel, children]); + + if (!num) return null; + if (children && !ids.length) { + return html`
+

DKAO 实时视图 — ${iter.name}

+

这一轮还没有派发 DKAO 子任务（或产物已被清理）。

+
`; + } + const child = selected ? children[selected] : null; + const childSummary = (c) => html` + + · phase ${c.current_phase || "-"} + · iter ${c.current_iteration ?? "-"} + ${c.rounds ? html` · ${c.rounds} rounds` : null} + ${c.best_median_us != null ? html` · best ≈${Number(c.best_median_us).toFixed(2)} µs` : null} + `; + const mirror = (id, c) => html` + <${DcuKernelAutoOptDetail} taskId=${id} stateDir=${c.child_state_dir} + readOnly=${true} />`; + const statusBadge = (c) => (c.finished + ? html`${c.final_status || "finished"}` + : html`${c.current_phase || "…"} #${c.current_iteration ?? "-"}`); + const layouts = [ + ["grid", "四宫格（四个同时看）"], + ["stack", "单列堆叠（四个都展开）"], + ["single", "只看一个（切换）"], + ]; + + return html` +
+

DKAO 实时视图 — ${iter.name} + （该轮每道考题的 DKAO 页面，只读；可同时看四个）

+
+ + ${layouts.map(([key, label]) => html` + `)} + 四个子任务是各自独立的 DKAO 运行（一卡一题），所以四个页面互不影响 +
+
+ ${ids.map((id) => { + const c = children[id] || {}; + const cls = selected === id ? "he-live-tab active" : "he-live-tab"; + return html` + `; + })} +
+ ${!children ? html`

Loading DKAO children…

` : null} + ${err ? html`

无法读取子任务列表：${err}

` : null} + ${layout === "single" + ? (child && child.child_state_dir ? html` +
+ 子任务数据目录 + ${child.child_state_dir} + ${childSummary(child)} +
+ ${mirror(selected, child)}` + : html`

这一轮没有可显示的子任务。

`) + : html` +
+ ${ids.map((id) => { + const c = children[id] || {}; + if (!c.child_state_dir) return null; + return html` +
+
+ + ${statusBadge(c)} + ${childSummary(c)} + + +
+
${mirror(id, c)}
+
`; + })} +
`} +
`; +} + +function IterBlock({ iter, taskId }) { + const results = (iter.benchmark && iter.benchmark.results) || {}; + const rows = Object.entries(results); + return html` +
+ ${iter.name} +
+ ${iter.decision ? html` +
+

System decision

+ <${VerdictBadge} verdict=${iter.decision.verdict} /> +
${iter.decision.reason || ""}
+ performance: ${iter.decision.performance_gate?.status || "-"} + · mechanism: ${iter.decision.mechanism_gate?.status || "-"} + · generalization: ${iter.decision.generalization_gate?.status || "-"} + · held-out: ${iter.decision.heldout_gate?.status || "-"} + ${iter.decision.generalization_gate?.status === "FAIL" ? html` +
非本族探针退化 → 判为过拟合（探针不计入 pass rate，但阻止晋级）
` : null} + + 决策配对 ${(iter.decision.paired_ids || []).length} 题 + ${(iter.decision.probe_ids || []).length + ? `· 探针 ${(iter.decision.probe_ids || []).join(", ")} 不计入判定` : ""} + +
` : null} + ${iter.rollback ? html`
auto-rollback: ${iter.rollback.reason}
` : null} + ${iter.mechanism_evidence ? html` +
+

Mechanism gate + （机制是否真的被执行）

+ ${iter.decision && iter.decision.mechanism_gate + ? html`<${VerdictBadge} verdict=${iter.decision.mechanism_gate.status} />` : null} + + + + ${((iter.mechanism_evidence.detail || {}).changes || []).map((c) => html` + + + + + + `)} + +
changegradekindevidence
${c.id}${(() => { + const cls = { hit: "ok", partial: "ok", + unobserved: "warn", contradicted: "bad" }[c.grade] || "warn"; + return html`${c.grade}`; + })()}${c.kind || "-"}${c.reason || ""}
+ + planner: ${String((iter.mechanism_evidence.evidence || {}).planner_enabled)} + · plan ids observed: ${((iter.mechanism_evidence.evidence || {}).plan_ids || []).join(", ") || "—"} + · harness revisions loaded: ${((iter.mechanism_evidence.evidence || {}).observed_harness_revisions || []).join(", ") || "—"} + +
` : null} + ${iter.promotion && (iter.promotion.results || []).length ? html` +
+

Kernel promotion → DKAO variants + （阈值 ${iter.promotion.min_improvement_percent}% · 写前备份）

+ + + + ${iter.promotion.results.map((p) => html` + + + + + + + `)} + +
shapeactionvariant median → newimprovementreason
${p.shape}${(() => { + const cls = { added: "ok", updated: "ok", "would-add": "ok", + "would-update": "ok" }[p.action] + || (p.action === "rejected" ? "bad" : "warn"); + return html`${p.action}`; + })()}${p.old_median_us != null ? (p.old_median_us.toFixed ? p.old_median_us.toFixed(2) : p.old_median_us) : "—"} + → ${p.new_median_us != null ? (p.new_median_us.toFixed ? p.new_median_us.toFixed(2) : p.new_median_us) : "—"}${p.improvement_percent != null ? `${p.improvement_percent.toFixed(2)}%` : "-"} + ${p.speedup != null ? html`(${typeof p.speedup === "number" ? p.speedup.toFixed(2) + "× baseline" : p.speedup})` : null}${p.reason || ""} ${p.backup ? html`backup: ${p.backup}` : null}
+
` : null} + ${rows.length ? html` + + + + ${rows.map(([id, r]) => html` + + + + + + `)} + +
instancerolepassedmedian/p90 uschild/reporeason
${id}${(() => { + const purpose = (iter.purposes || {})[id]; + if (!purpose) return html`—`; + const cls = purpose === "probe" ? "warn" : "ok"; + return html`${purpose}`; + })()}${r.passed ? html`pass` + : html`fail`}${r.median_us != null ? r.median_us : "-"} / ${r.p90_us != null ? r.p90_us : "-"}
${r.child_task_id || "-"}
${r.repo_path || ""}		${r.reason || ""}
` : html`

no benchmark results yet

`} + <${LiveDkaoMirror} taskId=${taskId} iter=${iter} /> + ${iter.diff ? html` +

flipped: ${(iter.diff.flipped || []).join(", ") || "-"} + · regressed: ${(iter.diff.regressed || []).join(", ") || "-"}

` : null} + ${iter.change_evaluation ? html` +
Change attribution (previous loop)
+ + + + ${(iter.change_evaluation.change_evaluations || []).map((c) => html` + + `)} + +
changehitverdict
${c.change_id}: ${(c.description || "").slice(0, 60)}${c.hit_rate}${html`<${VerdictBadge} verdict=${c.verdict} />`}
` : null} + ${iter.overview ? html`
analysis overview + 
${iter.overview}
` : null} +
+
`; +} + + +// Human decision on a candidate harness. Nothing is written to DKAO until the +// operator approves, so this panel is the gate: it shows the candidate, the +// evidence of both gates, and the current production version. +// A run that left the fixed protocol (FLOW.md §1–§4) judges neither gate, so it +// looks exactly like a run that is merely quiet. The orchestrator writes +// protocol.json + a timeline event when that happens; this puts it on screen. +function ProtocolBanner({ protocol }) { + if (!protocol || protocol.mode !== "legacy") return null; + return html` +
+

⚠ 本任务未按固定协议运行 （legacy / suite 路径）

+
+ 原因：${protocol.reason || "unknown"}。 + 这条路径不固化泛化考卷、不判性能门/泛化门、不按 4 的倍数取题， + 因此它的轮次**不会产出可接入 DKAO 的 harness**。 + 新任务请配置实测池文件（问卷池 Question pool），协议见 FLOW.md。 +
+
+ `; +} + +// Four cards, one glance: this round's DKAO children as four columns. +// The drill-down below still shows one child in full (read-only DKAO mirror); +// this answers the other question — "what is each card doing right now?" — +// without switching tabs. Every number is the child's own evidence: phase and +// iteration come from its run.json, the median from its accepted rounds, the +// gate line from its measurement_gate.jsonl, and the card membership from HE's +// layout record for the round (never from the card a probe happened to use). +function FourCardOverview({ data, onSelect, selected }) { + const cards = (data && data.cards) || []; + const gpuCount = (data && data.gpu_count) || 4; + if (!cards.length) return null; + const fmt = (v) => (v == null ? "-" : (v.toFixed ? v.toFixed(2) : v)); + const byGpu = new Map(cards.map((c) => [Number(c.gpu), c])); + const now = Date.now() / 1000; + const quiet = (q) => { + const ts = Number(q.last_update || 0); + if (!ts || q.finished) return null; + return Math.max(0, Math.round(now - ts)); + }; + const statusChip = (q) => { + if (q.finished) { + const ok = q.final_status === "success" || q.final_status === "ok"; + return html`已结束${q.final_status ? ` · ${q.final_status}` : ""}`; + } + const a = q.gate_audit || {}; + if (a.last_event === "gate_blocked" && a.last_usable === false) { + return html`等卡（闸门未放行）`; + } + const secs = quiet(q); + if (secs != null && secs > 900) { + return html`静默 ${Math.round(secs / 60)}min`; + } + return html`运行中`; + }; + const sourceLabel = (src) => ( + src === "layout" ? "本轮布局" + : src === "assignment" ? "题面绑定" + : src === "predicted" ? "预测（无记录）" : "子任务门禁"); + const cells = []; + for (let gpu = 0; gpu < gpuCount; gpu += 1) cells.push(gpu); + return html` +
+

四卡一览 （第 ${data && data.iteration} 轮 · 一卡一题 · 数字都来自子任务自己的记录）

+
+ ${cells.map((gpu) => { + const card = byGpu.get(gpu); + const questions = (card && card.questions) || []; + return html` +
+
+ GPU${gpu} + ${questions.length ? `${questions.length} 题` : "本轮未派题"} +
+ ${questions.length === 0 + ? html`
空闲
` + : questions.map((q) => { + const a = q.gate_audit || {}; + const g = q.gate || {}; + const secs = quiet(q); + // The bar is the round's real gate: the operator's current + // variant (legacy rounds kept the family τ×baseline target, + // which is what judged them). + const target = g.target_us; + const onTarget = (q.best_median_us != null && target != null) + ? q.best_median_us <= target : null; + return html` +
+
+ + ${statusChip(q)} +
+
+
阶段
+
${DKAO_PHASE_LABEL[q.current_phase] || q.current_phase || "idle"}
+
轮次
+
iter ${q.current_iteration ?? "-"} · ${q.rounds || 0} 次测量
+
当前
+
${q.best_median_us != null + ? html`${fmt(q.best_median_us)} µs + ${target != null + ? html` + / 必须打败 ≤ ${fmt(target)}${onTarget === null ? "" : (onTarget ? " ✓" : " ✗")}` + : null}` + : html`尚未测出`}
+
闸门
+
${a.checks == null || a.checks === 0 + ? html`暂无记录` + : html`${a.blocked ? html`等待 ${a.blocked} 次` : html`已通过`} + ${secs != null ? html` · ${secs < 60 ? `${secs}s` : `${Math.round(secs / 60)}min`} 前更新` : null}`}
+
来源
+
${sourceLabel(q.gpu_source)}
+
+
`; + })} +
`; + })} +
+
+ `; +} + +function PromotionPanel({ promo, busy, onApprove, onDeny }) { + if (!promo) return null; + const pending = promo.pending; + const promoted = promo.promoted; + const table = promo.variant_table || {}; + const gateLine = (gate, need) => { + if (!gate) return html`—`; + const wins = (gate.wins || []).length; + const ops = (gate.operators || []).length; + const ok = gate.status === "PASS"; + return html`${gate.status} + ${wins}/${ops} win（需 ${need}）`; + }; + return html` +
+

Harness promotion （接入 DKAO 需要人工批准）

+ ${pending ? html` +
+
+ 候选版本 ${pending.version} + · 来自第 ${pending.iteration} 轮 + · revision ${(pending.revision || "-").slice(0, 12)} +
+ + + + + + + +
性能门${gateLine(pending.performance_gate, 3)}
泛化门${gateLine(pending.generalization_gate, 2)}
机制门（宽松，仅记录）${(pending.mechanism_gate && pending.mechanism_gate.status) || "NOT_RECORDED"}
+
+ + +
+
` : html` +
当前没有待批准候选。
`} +
+ 生产版本：${(promoted && promoted.version) || "h-1（种子）"} + ${promoted && promoted.approved_at ? html` · 于 ${new Date(promoted.approved_at * 1000).toLocaleString()}` : null} + · variant 表：${table.operators || 0} 个算子 / 版本 ${table.harness_version || "未发布"} +
+
+ `; +} + +// One iteration's device cards, read-only: HE no longer leases cards — each +// question is a DKAO child and the child's own admission gate decides when the +// device it was handed to is clean enough to time a kernel. Folded twice on +// purpose: the section folds (default collapsed) and so does every card and +// every per-question detail, so a 4-card round stays one scan line until asked. +function GpuCardsPanel({ data, num }) { + const cards = (data && data.cards) || []; + const nQuestions = cards.reduce((acc, c) => acc + ((c.questions || []).length), 0); + const fmt = (v) => (v == null ? "-" : (v.toFixed ? v.toFixed(2) : v)); + return html` +
+ + GPU 设备 + + — 第 ${num} 轮 · ${cards.length} 张卡上有 ${nQuestions} 个 DKAO 子任务 + · 本任务不持有租约，卡由 DKAO 的准入闸门管理 + + + ${!cards.length + ? html`
本轮尚未派发任何 DKAO 子任务（无设备卡片）
` + : html`
+ ${cards.map((card) => html` +
+ + GPU${card.gpu} + + ${(card.questions || []).map((q) => q.question).join(", ") || "空闲"} + · ${(card.questions || []).length} 个问题 + + + ${(card.questions || []).map((q) => { + const a = q.gate_audit || {}; + const g = q.gate || {}; + const p = q.pool_reference || {}; + const req = q.requirements; + // Where this card membership came from, most authoritative + // first: HE's layout record for the round, the device HE + // pinned in the child's requirements, the child's own gate — + // and only as a last resort the index % 4 prediction. + const predicted = q.gpu_source === "predicted"; + const SOURCE_LABEL = { + layout: "该卡来自本轮布局记录（HE 派题时选定）", + assignment: "该卡来自子任务 requirements 的绑定", + gate_pass: "该卡来自子任务自己的门禁记录", + gate_blocked: "该卡来自子任务自己的门禁记录（最近一次被阻塞）", + predicted: "布局预测：index % 4，尚未有门禁记录", + }; + // A gate_ok row also carries reasons — they describe the + // candidates that failed, not this device — so the wording + // keys off the verdict, never off a non-empty reasons list. + const blockedRow = a.last_event === "gate_blocked"; + const passing = (a.last_passing || []).map((c) => `GPU${c}`); + const busy = (a.last_candidates || []) + .filter((c) => !(a.last_passing || []).includes(c)) + .map((c) => `GPU${c}`); + const gateLine = a.checks === 0 || a.checks == null + ? `准入闸门：暂无记录（VRAM≤90% 且 HCU==0）` + : blockedRow + ? `准入闸门：共 ${a.checks} 次检查，其中 ${a.blocked ?? 0} 次因设备被占用而等待 — 最近一次 ${a.last_event}@${a.last_site || "-"} GPU${a.last_gpu ?? "-"}：${(a.last_reasons || []).join("; ") || "-"}` + : `准入闸门：共 ${a.checks} 次检查，最近一次检查通过` + + (passing.length ? `（可用卡：${passing.join("、")}` + + (busy.length ? `；检查时 ${busy.join("、")} 被占用` : "") + `）` : ""); + return html` +
+ + ${q.question} + + · ${DKAO_PHASE_LABEL[q.current_phase] || q.current_phase || "idle"} + · ${SOURCE_LABEL[q.gpu_source] || "该卡来源未知"} + · iter ${q.current_iteration ?? "-"} + · ${q.rounds || 0} rounds + · best ${q.best_median_us != null ? `${fmt(q.best_median_us)} us` : "-"} + ${q.finished ? ` · 已结束${q.final_status ? `（${q.final_status}）` : ""}` : ""} + + +
+
${gateLine}
+
+ ${a.probe_device != null && a.device != null && a.probe_device !== a.device + ? `探针曾在 GPU${a.probe_device} 上通过（与本题所在卡无关）；` : ""} + ${g.protocol === "legacy" + ? html`legacy 判据：baseline ${p.baseline_us != null ? `${fmt(p.baseline_us)} us` : "-"} + · tau ${p.tau_family != null ? fmt(p.tau_family) : "-"} + · target ≤ ${g.target_us != null ? `${fmt(g.target_us)} us` : "-"}` + : html`性能门对手（variant）：${g.variant_us != null ? `${fmt(g.variant_us)} us` : "-"} + （${variantSourceLabel(g)}） + ${g.judged + ? html` · 本轮实测 ${fmt(g.judged.candidate_median_us)} us + <${GateVerdictBadge} verdict=${g.judged.verdict} />` + : null}`} + ${p.target_us != null + ? html` · 池子口径 τ×baseline = ${fmt(p.target_us)} us（本协议不判门）` + : null} + ${a.waited_seconds && (blockedRow || a.checks == null) + ? ` · 累计等待 ${Math.round(a.waited_seconds / 60)} min` : ""} +
+
+ ${(a.blocked_reasons || []).length ? html` +
+ 阻塞历史（最近 ${(a.blocked_reasons || []).length} 次） + + + + ${(a.blocked_reasons || []).map((b) => html` + + + + + + `)} + +
when站点GPU第几次原因
${b.ts ? new Date(b.ts * 1000).toLocaleTimeString() : "-"}${b.site || "-"}${b.gpu ?? "-"}${b.attempt ?? "-"}${(b.reasons || []).join("; ") || "-"}
+
` : null} + ${req ? html` +
+ 子任务 requirements + 
${JSON.stringify(req, null, 2)}
+
` : null} + ${q.child_state_dir ? html` +
+ child_state_dir +
${q.child_state_dir}
+
` : null} +
`; + })} +
`)} +
`} +
`; +} + +// --------------------------------------------------------------------------- // +// Round picker for the DKAO blocks +// --------------------------------------------------------------------------- // +// Each round is an independent DKAO launch, so which round you look at is a +// real switch. Default is "follow the newest round"; picking an older round +// pins it (and the 5s polling stops, because history does not move). +function RoundControls({ roundN, latestNum, roundNums, selIterN, setSelIterN, + ids, selChild, childrenMap, onSelectChild, onRefresh }) { + return html` +
+ + + + ${ids && ids.length ? html` + + + ` : null} +
+ ${selIterN == null ? null : html`
+ 正在查看历史轮次 iteration ${roundN}（本块与上面的 Live questions 都停在这一轮，不再自动刷新）
`} + `; +} + +export default function HeDetailView({ taskId, run, status, data }) { + const [summary, setSummary] = useState(null); + const [iters, setIters] = useState([]); + const [graph, setGraph] = useState(null); + const [children, setChildren] = useState(null); + const [selChild, setSelChild] = useState(null); + const [dkao, setDkao] = useState(null); + const [resumeMsg, setResumeMsg] = useState(null); + const [sup, setSup] = useState(null); + const [review, setReview] = useState(null); + const [iterGpu, setIterGpu] = useState(null); + const [targetIters, setTargetIters] = useState(null); + const [baseIter, setBaseIter] = useState(""); // "" = auto (best_ever) + const [promo, setPromo] = useState(null); + const [promoBusy, setPromoBusy] = useState(false); + //: Which round's DKAO children the drill-down shows. Rounds are independent + //: DKAO launches, so this is a real switch, not a filter: picking round 5 + //: re-reads that round's children (each has its own state dir). + //: ``null`` = follow the newest round until the operator picks one. + const [selIterN, setSelIterN] = useState(null); + + const latestNum = useMemo(() => { + if (!iters.length) return 0; + const m = (iters[iters.length - 1].name || "").match(/(\d+)$/); + return m ? Number(m[1]) : 0; + }, [iters]); + + // Every round the run has artifacts for, oldest first. + const roundNums = useMemo( + () => iters + .map((it) => { + const m = (it.name || "").match(/(\d+)$/); + return m ? Number(m[1]) : 0; + }) + .filter((n) => n > 0), + [iters], + ); + // The round actually on screen. Following means "the newest one", and the + // selection is clamped to what exists so a stale choice cannot outlive its + // round (Reset, or a fresh run). + const roundN = useMemo(() => { + if (!selIterN) return latestNum; + if (!roundNums.length || roundNums.includes(selIterN)) return selIterN; + return latestNum; + }, [selIterN, latestNum, roundNums]); + const following = roundN === latestNum; + + const refresh = useCallback(async () => { + if (!taskId) return; + const [s, its, g] = await Promise.all([ + withTimeout(getSummary(taskId)).catch((e) => { console.warn("he summary:", e); return null; }), + withTimeout(getIterations(taskId)).catch((e) => { console.warn("he iterations:", e); return []; }), + withTimeout(getStateGraph(taskId)).catch((e) => { console.warn("he graph:", e); return null; }), + ]); + setSummary(s || null); + setIters(its || []); + setGraph(g || null); + }, [taskId]); + + const refreshPromotion = useCallback(async () => { + if (!taskId) return; + const p = await withTimeout(getPromotion(taskId)) + .catch((e) => { console.warn("he promotion:", e); return null; }); + if (p) setPromo(p); + }, [taskId]); + + const approvePromo = useCallback(async () => { + if (!taskId || promoBusy) return; + setPromoBusy(true); + try { + const r = await approvePromotion(taskId, "webui"); + setResumeMsg(r && r.ok ? `已批准并接入：${r.result && r.result.version}` + : "批准被拒绝（见日志）"); + } catch (e) { + setResumeMsg(`批准失败：${e.message}`); + } finally { + setPromoBusy(false); + refreshPromotion(); + refresh(); + } + }, [taskId, promoBusy, refreshPromotion, refresh]); + + const denyPromo = useCallback(async () => { + if (!taskId || promoBusy) return; + setPromoBusy(true); + try { + const r = await denyPromotion(taskId, "webui"); + setResumeMsg(r && r.ok ? "已拒绝该候选，可继续迭代" : "拒绝失败（见日志）"); + } catch (e) { + setResumeMsg(`拒绝失败：${e.message}`); + } finally { + setPromoBusy(false); + refreshPromotion(); + } + }, [taskId, promoBusy, refreshPromotion]); + + const refreshChildren = useCallback(async () => { + if (!taskId) return; + if (roundN < 1) { setChildren(null); return; } + const c = await withTimeout(getChildren(taskId, roundN)) + .catch((e) => { console.warn("he children:", e); return null; }); + if (c) setChildren(c); + }, [taskId, roundN]); + + useEffect(() => { refresh(); }, [refresh]); + pollWhileVisible(refresh, 6000); + useEffect(() => { refreshChildren(); }, [refreshChildren]); + pollWhileVisible(refreshPromotion, 10000); + // Only the newest round is still moving, and only until its children are + // done: a round the operator pinned (or a round whose questions have all + // finished) is history and is fetched once, on selection. This also removes + // the duplicate: the drill-down below keeps its own children view, and both + // used to poll the same URL. + const roundMoves = useMemo(() => { + const kids = Object.values(children || {}); + return kids.length === 0 || kids.some((c) => !c.finished); + }, [children]); + pollWhileVisible(refreshChildren, 5000, { + enabled: Boolean(taskId) && following && latestNum >= 1 && roundMoves, + }); + + const best = summary && summary.best_ever; + const lastIter = iters.length ? iters[iters.length - 1] : null; + // Benchmark results belong to the round on screen, exactly like its children + // do: the two blocks below must never describe different rounds. + const selIterObj = useMemo( + () => iters.find((it) => { + const m = (it.name || "").match(/(\d+)$/); + return m && Number(m[1]) === roundN; + }) || null, + [iters, roundN], + ); + const results = (selIterObj && selIterObj.benchmark + && selIterObj.benchmark.results) || {}; + // Show every live DKAO child even before its benchmark row exists: child + // run.json phases appear as soon as evaluation spawns the subtasks, while + // benchmark results only materialize after a child finishes. + const childrenMap = children || {}; + const liveIds = [...new Set([...Object.keys(results), ...Object.keys(childrenMap)])]; + const liveKey = liveIds.join("|"); + const liveRows = liveIds.map((id) => ({ + id, + r: results[id] || null, + child: childrenMap[id] || null, + })); + + const loadChild = useCallback(async (id) => { + if (!id || roundN < 1) return; + setSelChild(id); + const p = await withTimeout(getDkaoChild(taskId, roundN, id)) + .catch((e) => { console.warn("he dkao child:", e); return null; }); + setDkao(p || null); + }, [taskId, roundN]); + + // Default to a still-running child so the DKAO drill-down has content. + useEffect(() => { + if (!liveIds.length || (selChild && liveIds.includes(selChild))) return; + const running = liveIds.find((id) => childrenMap[id] && !childrenMap[id].finished); + loadChild(running || liveIds[0]); + // eslint-disable-next-line react-hooks/exhaustive-deps + }, [liveKey, selChild]); + // Same rule as the child list: poll only while the shown round is the newest + // one and the question it shows can still move. ``loadChild`` also sets the + // selection, so the interval refreshes the payload in place. + pollWhileVisible(() => loadChild(selChild), 5000, { + enabled: Boolean(taskId) && roundN >= 1 && following && Boolean(selChild) + && Boolean(children && children[selChild] && !children[selChild].finished), + }); + + useEffect(() => { + if (!taskId) return; + let stop = false; + const tick = async () => { + const payload = await withTimeout(getSupervisor(taskId)) + .catch(() => null); + if (!stop && payload) setSup(payload); + }; + tick(); + const id = setInterval(tick, 6000); + return () => { stop = true; clearInterval(id); }; + }, [taskId]); + + useEffect(() => { + if (!taskId) return; + let stop = false; + const tick = async () => { + const payload = await withTimeout(getReview(taskId)).catch(() => null); + if (!stop && payload) setReview(payload); + }; + tick(); + const id = setInterval(tick, 15000); + return () => { stop = true; clearInterval(id); }; + }, [taskId]); + + // Read-only device cards of the selected iteration (HE holds no lease). + useEffect(() => { + if (!taskId || latestNum < 1) { setIterGpu(null); return; } + let stop = false; + const tick = async () => { + const payload = await withTimeout(getIterationGpu(taskId, latestNum)) + .catch(() => null); + if (!stop && payload) setIterGpu(payload); + }; + tick(); + const id = setInterval(tick, 10000); + return () => { stop = true; clearInterval(id); }; + }, [taskId, latestNum]); + + const startAuto = async () => { + if (!taskId) return; + const want = Math.max(1, Math.min(50, + Number(targetIters) || (iters.length + 1))); + setResumeMsg("starting unattended mode…"); + try { + const out = await startAutopilot(taskId, want, + baseIter === "" ? null : Number(baseIter), true); + setResumeMsg(`自动模式已启动（supervisor pid ${out.supervisor_pid}，目标 ${out.target} 轮）`); + refresh(); + } catch (e) { + setResumeMsg(`自动模式启动失败：${e.message}`); + } + }; + const stopAuto = async () => { + if (!taskId) return; + try { + await stopSupervisor(taskId); + setResumeMsg("已请求停止自动模式"); + const payload = await getSupervisor(taskId).catch(() => null); + if (payload) setSup(payload); + } catch (e) { + setResumeMsg(`停止失败：${e.message}`); + } + }; + + const applyTarget = async () => { + if (!taskId) return; + const want = Math.max(1, Math.min(50, Number(targetIters) || (iters.length + 1))); + setResumeMsg("applying…"); + try { + const out = await resumeExperiment(taskId, want, null, + baseIter === "" ? null : Number(baseIter)); + if (out.running) { + setResumeMsg(`目标已更新为 ${out.target} 轮（当前运行中，下一轮生效）`); + } else { + setResumeMsg(`已启动续跑（pid ${out.pid}），目标 ${out.target ?? want} 轮`); + } + refresh(); + } catch (e) { + setResumeMsg(`操作失败：${e.message}`); + } + }; + const isFinished = Boolean(graph && graph.current === "finished"); + const [stopBusy, setStopBusy] = useState(false); + // The orchestrator process is what "running" means here; a stopped run keeps + // its artifacts and its phase, so the graph alone cannot tell them apart. + // The shell's view comes from orchestrator.pid, which runs started before + // that file existed never wrote -- so trust HE's own answer too, or Stop + // would be hidden on a run that is very much alive. + const isRunning = Boolean((status && status.running) || (summary && summary.running)); + + const stopRun = async () => { + if (!taskId || stopBusy) return; + setStopBusy(true); + setResumeMsg("正在停止…"); + try { + const out = await stopExperiment(taskId); + setResumeMsg(out.message + (out.stopped_children && out.stopped_children.length + ? `（同时停掉 ${out.stopped_children.length} 个 DKAO 子任务）` : "")); + refresh(); + refreshPromotion(); + } catch (e) { + setResumeMsg(`停止失败：${e.message}`); + } finally { + setStopBusy(false); + } + }; + + return html` +
+ <${ProtocolBanner} protocol=${summary && summary.protocol} /> + <${PromotionPanel} promo=${promo} busy=${promoBusy} + onApprove=${approvePromo} onDeny=${denyPromo} /> + ${(summary && summary.operator_stop && !isRunning) ? html` +
+

⏸ 已被 operator 停止 （暂停，不是结束）

+
+ 停在 iteration ${summary.operator_stop.iteration ?? "-"}（phase ${summary.operator_stop.phase || "-"}）· + 不会被 janitor / supervisor 自动拉起；本轮没判完，所以这一轮的成绩不计入 variant。 + 继续跑：点左边的「续跑到该轮次上限」（会先清掉停止标记，该轮重跑、抽到的是同一批题）。 + 操作说明见 FLOW.md §9。 +
+
` : null} + <${StateMachine} graph=${graph} /> +
+ + setTargetIters(e.target.value)} /> + + ${isRunning ? html` + + 停止=暂停：产物保留、不会被自动重启；之后点左边的「续跑到该轮次上限」继续` : null} + + + 已完成 ${iters.length} 轮${graph && graph.current === "finished" ? "（已停止）" : "（运行中）"} · 运行中可随时改，下一轮生效 + ${resumeMsg ? html`${resumeMsg}` : null} + ${isFinished + ? html`复用已评测的冠军，只评测新一版 harness（不重跑历史迭代）` + : null} +
+
+ + + + 自动模式：${sup && sup.running ? `运行中（pid ${sup.pid}）` : ((sup && sup.state && sup.state.status) || "未启动")} + ${sup && sup.incidents && sup.incidents.length ? ` · 事故 ${sup.incidents.length}` : ""} + ${sup && sup.has_report ? " · 诊断报告已生成" : ""} + +
+ <${FourCardOverview} data=${iterGpu} selected=${selChild} + onSelect=${(id) => setSelChild(id)} /> + <${GpuCardsPanel} data=${iterGpu} num=${latestNum} /> + ${review && review.review ? html` +
+

跨轮复核 （每代 harness 成绩 · 冠军代 · best-known）

+
+ 冠军代 iteration ${review.review.champion_iteration} + variant 策略：${review.review.variant_policy} + ${(review.review.final_promotion || {}).promoted + ? ` · 收尾晋升：${(review.review.final_promotion.promoted || []).join(", ")}` : ""} + ${review.regressions && review.regressions.length + ? ` · ⚠ 退步 ${review.regressions.length} 次` : ""} + +
+ + + + ${(review.review.generations || []).map((g) => html` + + + + + + `)} + +
genverdictpassmedians (us)
${g.iteration}${g.champion ? " ⭐" : ""}${g.rejected ? " ✗" : ""}${g.verdict || "-"}${g.pass_count}${Object.entries(g.medians || {}).map(([k, v]) => + `${k.split("_").slice(-3).join("_")}=${v == null ? "-" : Number(v).toFixed(1)}`).join(" ")}
+ ${review.regressions && review.regressions.length ? html` +
+ 退步记录（kernel 低于历史最优） + + + + ${review.regressions.slice(-8).reverse().map((r) => html` + + + `)} + +
iterationshapemedianbestdelta
${r.iteration}${r.shape}${r.median_us}${r.best_median_us} (gen ${r.best_iteration})+${r.delta_percent}%
+
` : null} +
` : null} + ${sup && sup.incidents && sup.incidents.length ? html` +
+

Incidents （supervisor 自动处置记录）

+ + + + ${sup.incidents.slice(-8).reverse().map((i) => html` + + + + + + `)} + +
whencategoryactiondetail
${new Date((i.ts || 0) * 1000).toLocaleTimeString()}${i.category || "-"}${i.action || "-"}${i.reason || (i.evidence || []).join("; ") || ""} + ${i.seconds ? html`wait ${Math.round(i.seconds / 60)}m` : null}
+
` : null} +
+

Best ever

+
${best ? `${(100 * best.pass_rate).toFixed(1)}%` : "-"}
+ ${best ? html`
iteration ${best.iteration}
` : null} +
+

Iterations

+
${iters.length}
+

Mode

+
${(summary && summary.config_snapshot && summary.config_snapshot.execution_mode) || "-"}
+
+
+ ${liveRows.length ? html` +
+

Live questions — iteration ${roundN} + （每行 = 一个真实 DKAO 子任务）

+ <${RoundGateLine} iter=${selIterObj} /> + + + + + + + + + + + ${liveRows.map(({ id, r, child }) => html` + + + + + + + + `)} + +
questionDKAO child phase子任务验收median(best) us必须打败（variant）rounds
${id}<${DkaoPhaseChip} child=${child} /> + ${!r ? html`running` + : r.passed ? html`pass` + : r.status === "exception" || r.status === "missing_report" || r.status === "partial_worker_result" + ? html`${r.status}` + : html`${r.status || "pending"}`} + ${(r && r.median_us != null) ? r.median_us + : (child && child.best_median_us != null) ? `≈${child.best_median_us}` : "-"}<${QuestionGateCell} + gate=${child && child.gate} + poolRef=${child && child.pool_reference} />${(r && r.rounds_used != null) ? r.rounds_used : (child ? (child.rounds || 0) : 0)}
+
+
+

DKAO task drill-down + （选轮次后看那一轮的 DKAO 子任务；只读）

+ <${RoundControls} + roundN=${roundN} latestNum=${latestNum} roundNums=${roundNums} + selIterN=${selIterN} setSelIterN=${setSelIterN} + ids=${liveIds} selChild=${selChild} childrenMap=${childrenMap} + onSelectChild=${loadChild} onRefresh=${loadChild} /> + <${ChildDkaoView} payload=${dkao} /> +
` : html` +
+

DKAO task drill-down + （选轮次后看那一轮的 DKAO 子任务；只读）

+ <${RoundControls} + roundN=${roundN} latestNum=${latestNum} roundNums=${roundNums} + selIterN=${selIterN} setSelIterN=${setSelIterN} + ids=${null} selChild=${null} childrenMap=${{}} + onSelectChild=${loadChild} onRefresh=${loadChild} /> +

这一轮没有派发 DKAO 子任务（或产物已被清理）。

+
`} +
+

Pass rate per iteration

+ ${summary && html`<${ScoreTable} scores=${summary.scores} />`} +
+
+

Iterations

+ ${iters.length ? iters.map((it) => html`<${IterBlock} iter=${it} taskId=${taskId} />`) + : html`

no iterations yet (task not started / dry artifacts absent)

`} + ${summary && summary.report ? html`
report.md + 
${summary.report}
` : null} +
+
`; +} diff --git a/metainfer/tasks/harness_evolve/static/he.css b/metainfer/tasks/harness_evolve/static/he.css new file mode 100644 index 00000000..18632b2d --- /dev/null +++ b/metainfer/tasks/harness_evolve/static/he.css @@ -0,0 +1,166 @@ +/* harness_evolve detail view — mirrors the DKAO look & feel. + Reuses the dkao state-machine classes (copied into this plugin dir) and the + shell theme variables so the AHE outer loop reads like a DKAO task page. */ +@import url("./dkao.css"); + +.he-detail { + display: flex; + flex-direction: column; + gap: var(--sp-4, 10px); +} + +/* summary header cards */ +.he-cards { display: flex; flex-wrap: wrap; gap: var(--sp-3, 8px); } +.he-card { + flex: 1 1 160px; + min-width: 160px; + border: 1px solid var(--border, #ddd); + border-radius: var(--radius, 8px); + padding: var(--sp-3) var(--sp-4); + background: var(--bg, #fff); +} +.he-card h4 { margin: 0 0 4px; font-size: var(--fs-xs, 12px); + text-transform: uppercase; letter-spacing: .04em; color: var(--fg-dim, #666); } +.he-card .big { font-size: 26px; font-weight: 700; color: var(--fg, #111); } +.he-card .sub { color: var(--fg-dim, #666); font-size: var(--fs-xs, 12px); } + +/* generic content panels */ +.he-section { + border: 1px solid var(--border, #ddd); + border-radius: var(--radius, 8px); + padding: var(--sp-3) var(--sp-4); + background: var(--bg, #fff); +} +.he-section > h3, .he-section > h4 { margin: 0 0 var(--sp-3); } +.he-section h4 small { font-weight: 400; color: var(--fg-dim, #666); } + +/* tables */ +table.he-tbl { border-collapse: collapse; width: 100%; font-size: var(--fs-xs, 12px); } +.he-tbl th { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--border, #ddd); + background: var(--panel-2, #f2f2f2); color: var(--fg-dim, #555); font-weight: 600; } +.he-tbl td { padding: 6px 8px; border-bottom: 1px solid var(--border-soft, #eee); + vertical-align: top; } +.he-tbl .mono { font-family: var(--fs-mono, monospace); font-size: var(--fs-xs, 12px); } + +/* badges (DKAO-colored) */ +.badge { display: inline-block; padding: 2px 8px; border-radius: var(--radius-sm, 6px); + font-size: var(--fs-xs, 11px); font-weight: 600; color: #fff; white-space: nowrap; } +.badge.ok { background: var(--ok, #2e7d32); } +.badge.bad { background: var(--err, #c62828); } +.badge.warn { background: var(--warn, #ef6c00); } +.badge.run { background: var(--accent, #1565c0); } + +/* details / pre */ +.he-iter summary { cursor: pointer; font-weight: 600; padding: 6px 0; } +.he-pre { background: var(--bg-elev, #f6f6f6); border: 1px solid var(--border, #ddd); + border-radius: var(--radius, 8px); padding: var(--sp-3) var(--sp-4); + white-space: pre-wrap; font-family: var(--fs-mono, monospace); + font-size: var(--fs-xs, 12px); max-height: 320px; overflow: auto; } + +/* child drill-down row */ +.he-subrow { display: flex; align-items: center; gap: 10px; flex-wrap: wrap; + margin: 4px 0 10px; } +.he-subrow label { font-size: var(--fs-xs, 12px); color: var(--fg-dim, #555); } +.he-select { padding: 4px 8px; border: 1px solid var(--border, #ddd); + border-radius: var(--radius-sm, 6px); background: var(--bg, #fff); + color: var(--fg, #111); font-size: var(--fs-xs, 12px); max-width: 640px; } +.he-btn { padding: 4px 10px; border: 1px solid var(--border, #ddd); + border-radius: var(--radius-sm, 6px); background: var(--panel-2, #eee); + color: var(--fg, #111); cursor: pointer; font-size: var(--fs-xs, 12px); } +.he-child-dkao { display: flex; flex-direction: column; gap: var(--sp-4, 10px); } + +/* read-only device cards (GPU 设备) — folded section, folded card, folded detail */ +.he-gpu-panel > summary { cursor: pointer; padding: var(--sp-2, 6px) 0; } +.he-gpu-panel > summary small { font-weight: 400; } +.he-gpu-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: var(--sp-3, 8px); margin-top: var(--sp-2, 6px); } +.he-gpu-card { min-width: 0; border: 1px solid var(--border-soft, #eee); + border-radius: var(--radius-sm, 6px); background: var(--bg-elev, #f6f6f6); } +.he-gpu-card > summary { display: flex; align-items: baseline; justify-content: space-between; + gap: 10px; padding: 7px 10px; cursor: pointer; font-size: var(--fs-xs, 12px); } +.he-gpu-card[open] > summary { border-bottom: 1px solid var(--border-soft, #eee); } +.he-gpu-card > summary strong { font-family: var(--fs-mono, monospace); } +.he-gpu-card > summary small { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } +.he-gpu-q { border-top: 1px solid var(--border-soft, #eee); } +.he-gpu-card > .he-gpu-q:first-of-type { border-top: none; } +.he-gpu-q > summary { display: flex; align-items: baseline; flex-wrap: wrap; gap: 6px; + padding: 6px 10px; cursor: pointer; font-size: var(--fs-xs, 12px); } +.he-gpu-q > summary small { font-size: var(--fs-xxs, 11px); } +.he-gpu-gate { display: flex; flex-direction: column; gap: 2px; + padding: 0 10px 6px; font-size: var(--fs-xs, 12px); } +.he-gpu-sub { margin: 4px 10px 8px; font-size: var(--fs-xs, 12px); } +.he-gpu-sub > summary { cursor: pointer; color: var(--fg-dim, #666); font-size: var(--fs-xxs, 11px); } +.he-gpu-sub .mono { word-break: break-all; } +.he-gpu-empty { display: grid; min-height: 38px; place-items: center; + font-size: var(--fs-xs, 12px); } + +/* live DKAO mirror inside one iteration — the DKAO page rendered read-only */ +.he-live-dkao { border-top: 1px solid var(--border-soft, #eee); padding-top: var(--sp-3, 8px); } +.he-live-tabs { flex-wrap: wrap; gap: 6px; } +.he-live-tab { display: inline-flex; align-items: center; gap: 6px; cursor: pointer; + padding: 5px 9px; border: 1px solid var(--border-soft, #ddd); + border-radius: var(--radius-sm, 6px); background: var(--bg-elev, #f6f6f6); + font-size: var(--fs-xs, 12px); } +.he-live-tab.active { border-color: var(--accent, #2b6cb0); font-weight: 600; } +.he-live-tab small { display: inline-flex; } +.he-live-host { display: flex; align-items: baseline; flex-wrap: wrap; gap: 6px; + margin: var(--sp-2, 6px) 0; font-size: var(--fs-xs, 12px); } +.he-live-host .mono { word-break: break-all; font-size: var(--fs-xxs, 11px); } +.dkao-readonly-banner { border-left: 3px solid var(--accent, #2b6cb0); } + +/* Stop control: stopping is a pause, so it reads as a normal action that + happens to be destructive to the *process*, not to the artifacts. */ +.he-btn-stop { border-color: var(--warn, #b7791f); color: var(--warn, #b7791f); } +.he-btn-stop:hover:not(:disabled) { background: var(--warn, #b7791f); color: #fff; } +.he-btn-stop:disabled { opacity: 0.6; cursor: default; } + +/* four-card overview (四卡一览) — one column per HCU for this round's children. + Deliberately not `.he-cards`: that one is the flex summary-card row. */ +.he-four-cards { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); + gap: var(--sp-3, 8px); margin-top: var(--sp-2, 6px); } +@media (max-width: 1200px) { + .he-four-cards { grid-template-columns: repeat(2, minmax(0, 1fr)); } +} +.he-card-col { min-width: 0; border: 1px solid var(--border-soft, #eee); + border-radius: var(--radius-sm, 6px); background: var(--bg-elev, #f6f6f6); + padding: 8px 10px; } +.he-card-col.idle { opacity: .6; } +.he-card-col > header { display: flex; align-items: baseline; justify-content: space-between; + gap: 8px; margin-bottom: 6px; } +.he-card-col > header strong { font-family: var(--fs-mono, monospace); color: var(--accent-2, #2b6cb0); } +.he-card-empty { display: grid; min-height: 34px; place-items: center; font-size: var(--fs-xs, 12px); } +.he-card-q { border-top: 1px solid var(--border-soft, #eee); padding-top: 6px; margin-top: 6px; } +.he-card-q:first-of-type { border-top: none; margin-top: 0; padding-top: 0; } +.he-card-q.selected { outline: 1px solid var(--accent, #2b6cb0); } +.he-card-q-head { display: flex; align-items: center; justify-content: space-between; + gap: 6px; margin-bottom: 4px; } +.he-card-q-name { padding: 0; border: none; background: none; cursor: pointer; + font-family: var(--fs-mono, monospace); font-size: var(--fs-xs, 12px); + color: var(--fg, #111); text-align: left; word-break: break-all; } +.he-card-q-name:hover { text-decoration: underline; } +.he-card-metrics { display: grid; grid-template-columns: 34px minmax(0, 1fr); + gap: 2px 6px; margin: 0; font-size: var(--fs-xxs, 11px); } +.he-card-metrics dt { color: var(--fg-dim, #666); } +.he-card-metrics dd { margin: 0; min-width: 0; overflow-wrap: anywhere; } + +/* DKAO 实时视图：同时看四个子任务（一卡一题，四个独立的 DKAO 页面）。 + 默认四宫格；窄屏自动退化为单列。每个单元格是一个只读 DKAO 页面的容器。 */ +.he-live-layout { gap: 6px; } +.he-live-layout .he-btn.active { border-color: var(--accent, #2b6cb0); font-weight: 600; } +.he-live-grid { display: grid; grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: var(--sp-3, 8px); align-items: start; margin-top: var(--sp-2, 6px); } +.he-live-stack { display: grid; grid-template-columns: minmax(0, 1fr); + gap: var(--sp-3, 8px); margin-top: var(--sp-2, 6px); } +@media (max-width: 1500px) { .he-live-grid { grid-template-columns: minmax(0, 1fr); } } +.he-live-cell { min-width: 0; border: 1px solid var(--border-soft, #eee); + border-radius: var(--radius-sm, 6px); background: var(--bg, #fff); + padding: 2px 8px 8px; } +.he-live-cell.selected { border-color: var(--accent, #2b6cb0); } +.he-live-cell-head { display: flex; align-items: baseline; flex-wrap: wrap; gap: 6px; + padding: 5px 0 6px; border-bottom: 1px solid var(--border-soft, #eee); } +.he-live-cell-title { padding: 0; border: none; background: none; cursor: pointer; + font-size: var(--fs-xs, 12px); color: var(--fg, #111); text-align: left; + word-break: break-all; } +.he-live-cell-title:hover { text-decoration: underline; } +.he-live-cell-grow { flex: 1 1 auto; } +.he-live-cell-body { min-width: 0; overflow-x: auto; } diff --git a/metainfer/tasks/harness_evolve/tests/__init__.py b/metainfer/tasks/harness_evolve/tests/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/metainfer/tasks/harness_evolve/tests/conftest.py b/metainfer/tasks/harness_evolve/tests/conftest.py new file mode 100644 index 00000000..8df577db --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/conftest.py @@ -0,0 +1,19 @@ +"""Shared fixtures for harness_evolve plugin route tests.""" + +from __future__ import annotations + +import pytest +from fastapi.testclient import TestClient + +from metainfer.testing import isolated_env # noqa: F401 - re-export as fixture +from metainfer.server import app as app_module + + +@pytest.fixture +def app(isolated_env): + return app_module.create_app() + + +@pytest.fixture +def client(app): + return TestClient(app) diff --git a/metainfer/tasks/harness_evolve/tests/test_champion_record_failure.py b/metainfer/tasks/harness_evolve/tests/test_champion_record_failure.py new file mode 100644 index 00000000..744b66b0 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_champion_record_failure.py @@ -0,0 +1,135 @@ +"""Losing the champion bookkeeping must be loud, and must not lose the champion. + +``best_ever.json`` + the generation log decide what later rounds compare +against. When that write failed it used to be swallowed into a timeline row +(``champion_record_failed``) and nothing else — 26 of them in one run on +2026-09-11, all ``AttributeError`` — so the run carried on against a champion +chain nobody could vouch for. + +Now a failure is visible (stderr, i.e. the orchestrator log), keeps its +traceback, and is followed by a rebuild of the champion from the generations +already on disk. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +from metainfer.tasks.harness_evolve.orchestrator import pipeline as P +from metainfer.tasks.harness_evolve.orchestrator.champion_selection import ( + Generation, record_generation, +) +from metainfer.tasks.harness_evolve.orchestrator.config import ExperimentConfig + + +def _cfg(tmp_path: Path) -> ExperimentConfig: + cfg = ExperimentConfig( + task_id="he-champion", state_dir=tmp_path / "state", + workspace_dir=tmp_path / "exp", suite=[], answers={}, + execution_mode="dry-run", per_round_budget=4, max_iterations=1, + ) + cfg.state_dir.mkdir(parents=True, exist_ok=True) + cfg.exp_dir.mkdir(parents=True, exist_ok=True) + return cfg + + +def _generation(exp: Path, iteration: int, median_us: float) -> Generation: + """A generation on disk, the way the loop leaves one behind. + + ``load_generations`` reads each generation's measurements back from + ``runs/iteration_NNN/input/benchmark/results.json``, so a usable champion + needs both files — that is what the rebuild fallback replays. + """ + generation = Generation( + iteration=iteration, + verdict="PROMOTE", + results={"op00": {"median_us": median_us, "passed": True}}, + snapshot_dir=f"/tmp/snap-{iteration}", + workspace_revision=f"rev-{iteration}", + pass_count=1, + ) + record_generation(exp, generation) + results_path = (exp / "runs" / f"iteration_{iteration:03d}" / "input" + / "benchmark" / "results.json") + results_path.parent.mkdir(parents=True, exist_ok=True) + results_path.write_text(json.dumps( + {"iteration": iteration, "results": generation.results}), + encoding="utf-8") + return generation + + +def _events(cfg: ExperimentConfig) -> list: + path = Path(cfg.state_dir) / "timeline.jsonl" + if not path.is_file(): + return [] + return [json.loads(line)["type"] + for line in path.read_text(encoding="utf-8").splitlines() if line.strip()] + + +def test_a_healthy_record_writes_the_champion(tmp_path): + cfg = _cfg(tmp_path) + _generation(cfg.exp_dir, 1, median_us=11.0) + + outcome = P._record_champion_generation( + cfg, cfg.exp_dir, iteration=1, decision={"verdict": "PROMOTE"}, + decision_rate=1.0, snapshot=Path("/tmp/snap-1"), + live_workspace=cfg.workspace_dir, results={"op00": {"passed": True}}, + current={}) + + assert outcome["recorded"] is True and outcome["recovered"] is False + best = json.loads((cfg.exp_dir / "best_ever.json").read_text()) + assert best["iteration"] == 1 + assert "champion_best_measured" in _events(cfg) + + +def test_a_failed_record_is_loud_and_rebuilds_the_champion(tmp_path, monkeypatch, + capsys): + cfg = _cfg(tmp_path) + # a previous, good generation is already on disk + _generation(cfg.exp_dir, 1, median_us=11.0) + + def boom(_exp, _generation): + raise AttributeError("'NoneType' object has no attribute 'get'") + + monkeypatch.setattr(P, "record_generation", boom) + + outcome = P._record_champion_generation( + cfg, cfg.exp_dir, iteration=2, decision={"verdict": "PROMOTE"}, + decision_rate=0.5, snapshot=Path("/tmp/snap-2"), + live_workspace=cfg.workspace_dir, results={"op00": {"passed": True}}, + current={}) + + # never raises, and the error is on the record with its traceback + assert outcome["recorded"] is False and outcome["error"] + rows = [json.loads(line) for line in + (Path(cfg.state_dir) / "timeline.jsonl").read_text( + encoding="utf-8").splitlines() if line.strip()] + failed = [row for row in rows if row["type"] == "champion_record_failed"] + assert failed and "AttributeError" in failed[0]["payload"]["error"] + assert "NoneType" in failed[0]["payload"]["traceback"] + # ... and it is visible in the orchestrator log, not only in a JSONL row + assert "champion bookkeeping failed" in capsys.readouterr().err + # the champion chain survives: rebuilt from the generation on disk + assert outcome["recovered"] is True + best = json.loads((cfg.exp_dir / "best_ever.json").read_text()) + assert best["iteration"] == 1 and best["verdict"] == "REBUILT_AFTER_ERROR" + assert "champion_rebuilt" in _events(cfg) + + +def test_a_failure_with_nothing_to_rebuild_from_still_returns(tmp_path, + monkeypatch): + """No usable generation on disk: report, keep the old champion, carry on.""" + cfg = _cfg(tmp_path) + monkeypatch.setattr(P, "record_generation", + lambda *a, **k: (_ for _ in ()).throw(RuntimeError("disk full"))) + previous = {"iteration": 7, "verdict": "BEST_MEASURED"} + + outcome = P._record_champion_generation( + cfg, cfg.exp_dir, iteration=8, decision={"verdict": "REJECT"}, + decision_rate=0.0, snapshot=Path("/tmp/snap-8"), + live_workspace=cfg.workspace_dir, results={}, current=previous) + + assert outcome["recorded"] is False and outcome["recovered"] is False + assert outcome["best_ever"] == previous + assert "champion_record_failed" in _events(cfg) diff --git a/metainfer/tasks/harness_evolve/tests/test_champion_selection.py b/metainfer/tasks/harness_evolve/tests/test_champion_selection.py new file mode 100644 index 00000000..9fc0b2ee --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_champion_selection.py @@ -0,0 +1,74 @@ +"""Champion = best measured generation, not "last promoted generation".""" + +from __future__ import annotations + +from metainfer.tasks.harness_evolve.orchestrator.champion_selection import ( + Generation, best_generation, compare_generations, load_generations, + record_generation, summarize, +) + + +def _gen(iteration, medians, *, verdict="PROMOTE", passed=None) -> Generation: + results = {} + for iid, median in medians.items(): + ok = True if passed is None else bool(passed.get(iid, True)) + results[iid] = {"median_us": median, "passed": ok, + "correctness_ok": True} + return Generation(iteration=iteration, verdict=verdict, results=results, + pass_count=sum(1 for r in results.values() + if r["passed"])) + + +def test_better_generation_keeps_the_champion_slot(): + """The reported scenario: gen1 40us (unpromoted), gen2 60us (promoted).""" + gen1 = _gen(1, {"a": 40.0, "b": 40.0}, verdict="CONFIRM_REQUIRED") + gen2 = _gen(2, {"a": 60.0, "b": 60.0}, verdict="PROMOTE") + assert compare_generations(gen1, gen2) == "a" + assert best_generation([gen1, gen2]).iteration == 1 + + +def test_rejected_generation_is_disqualified(): + good = _gen(1, {"a": 40.0}) + bad = _gen(2, {"a": 20.0}, verdict="REJECT") # fast but regressed + assert best_generation([good, bad]).iteration == 1 + + +def test_no_signal_generation_does_not_displace_a_better_one(): + keeper = _gen(1, {"a": 40.0}, verdict="PROMOTE") + flat = _gen(2, {"a": 41.0}, verdict="NO_SIGNAL") + assert best_generation([keeper, flat]).iteration == 1 + + +def test_more_passing_questions_wins_before_raw_speed(): + fewer = _gen(1, {"a": 30.0, "b": 100.0}, passed={"a": True, "b": False}) + more = _gen(2, {"a": 32.0, "b": 60.0}, passed={"a": True, "b": True}) + assert best_generation([fewer, more]).iteration == 2 + + +def test_earlier_generation_wins_a_tie(): + first = _gen(1, {"a": 50.0}) + second = _gen(2, {"a": 50.5}) # within the noise band + assert best_generation([first, second]).iteration == 1 + + +def test_round_trip_persistence(tmp_path): + exp = tmp_path / "exp" + (exp / "runs" / "iteration_001" / "input" / "benchmark").mkdir(parents=True) + (exp / "runs" / "iteration_001" / "input" / "benchmark" + / "results.json").write_text('{"results": {"a": {"median_us": 40.0, "passed": true}}}', + encoding="utf-8") + record_generation(exp, _gen(1, {"a": 40.0}, verdict="CONFIRM_REQUIRED")) + gens = load_generations(exp) + assert len(gens) == 1 + assert gens[0].iteration == 1 and gens[0].pending is True + assert gens[0].results["a"]["median_us"] == 40.0 + assert best_generation(gens).iteration == 1 + + # recording the same iteration again replaces the row + record_generation(exp, _gen(1, {"a": 35.0}, verdict="PROMOTE")) + gens = load_generations(exp) + assert len(gens) == 1 and gens[0].verdict == "PROMOTE" + + rows = summarize(gens) + assert rows[0]["champion"] is True + assert rows[0]["medians"] == {"a": 40.0} diff --git a/metainfer/tasks/harness_evolve/tests/test_config_req_contract.py b/metainfer/tasks/harness_evolve/tests/test_config_req_contract.py new file mode 100644 index 00000000..5db5a04d --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_config_req_contract.py @@ -0,0 +1,100 @@ +"""load_experiment_config must honor the WebUI/shell requirements contract. + +The shell/server flattens form answers onto the requirements top level (the +same contract dcu_kernel_auto_opt consumes); the legacy CLI/tests path passes +them under an ``answers`` dict. Both must yield the same ExperimentConfig so a +WebUI-launched harness-evolve experiment actually runs dkao-cli against the +registered pool instead of silently falling back to dry-run defaults. +""" + +from __future__ import annotations + +import json + +import pytest + +from metainfer.tasks.harness_evolve.orchestrator.config import ( + load_experiment_config, +) + +_FLAT_FIELDS = { + "task_id": "he-webui-task", + "task_type": "harness-evolve", + "label": "webui task", + "raw_request": "", + "pool_source": "/root/zth_agent/ahe-kernel-repos/registered_pool.yaml", + "per_round_budget": 4, + "default_rounds": 5, + "max_iterations": 1, + "harness_source": "dcu_default", + "execution_mode": "dkao-cli", + "ahe_repo_root": "/root/zth_agent/ahe-kernel-repos", + "child_timeout_minutes": "0", + "evolve_mode": "agent", + "evolve_model": "deepseek/deepseek-flash", + "evolve_timeout_seconds": "1800", + "agent_framework": "dsh", +} + + +def _write_req(tmp_path, payload) -> "object": + req = tmp_path / "requirements.json" + req.write_text(json.dumps(payload), encoding="utf-8") + return req + + +def test_flat_top_level_requirements_are_honored(tmp_path): + """WebUI form answers flattened onto req top level must not be ignored.""" + cfg = load_experiment_config( + _write_req(tmp_path, _FLAT_FIELDS), + tmp_path / "state", + tmp_path / "ws", + ) + assert cfg.task_id == "he-webui-task" + assert cfg.execution_mode == "dkao-cli" + assert cfg.pool_source == _FLAT_FIELDS["pool_source"] + assert cfg.pool_mode is True + assert cfg.per_round_budget == 4 + assert cfg.max_iterations == 1 + assert cfg.harness_source == "dcu_default" + assert cfg.agent_framework == "dsh" + assert cfg.answers.get("evolve_mode") == "agent" + assert cfg.answers.get("default_rounds") == 5 + + +def test_answers_dict_requirements_still_honored(tmp_path): + """Legacy CLI/tests contract (answers dict) keeps working. + + Identity fields live on the req top level; the form answers (everything + under ``answers``) are what load_experiment_config reads for behaviour. + """ + form_fields = { + k: v for k, v in _FLAT_FIELDS.items() + if k not in {"task_id", "task_type", "label", "raw_request"} + } + payload = { + "task_id": _FLAT_FIELDS["task_id"], + "task_type": _FLAT_FIELDS["task_type"], + "answers": form_fields, + } + cfg = load_experiment_config( + _write_req(tmp_path, payload), + tmp_path / "state", + tmp_path / "ws", + ) + assert cfg.task_id == "he-webui-task" + assert cfg.execution_mode == "dkao-cli" + assert cfg.pool_mode is True + assert cfg.per_round_budget == 4 + + +def test_empty_requirements_fall_back_to_dry_defaults(tmp_path): + """No form fields at all -> documented dry-run defaults (baseline only).""" + cfg = load_experiment_config( + _write_req(tmp_path, {"task_id": "empty", "task_type": "harness-evolve"}), + tmp_path / "state", + tmp_path / "ws", + ) + assert cfg.execution_mode == "dry-run" + assert cfg.pool_mode is False + assert cfg.max_iterations == 3 diff --git a/metainfer/tasks/harness_evolve/tests/test_decision_engine.py b/metainfer/tasks/harness_evolve/tests/test_decision_engine.py new file mode 100644 index 00000000..06fcc8dc --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_decision_engine.py @@ -0,0 +1,196 @@ +"""System-level five-state decision engine tests.""" + +from __future__ import annotations + +from ..orchestrator.decision_engine import DecisionPolicy, decide + + +def _results(medians, *, passed=True, correctness=True): + return { + k: { + "median_us": v, + "passed": passed, + "correctness_ok": correctness, + } + for k, v in medians.items() + } + + +def _manifest(routeable=False): + return { + "iteration": 1, + "changes": [{ + "id": "chg-1", + "mechanism_signature": ["plan sequence changes"], + "scope": {"routeable": routeable}, + }], + } + + +def test_first_result_establishes_baseline(): + d = decide(None, _results({"a": 100.0})) + assert d["verdict"] == "BASELINE" + assert d["action"] == "SET_CHAMPION" + + +def test_promote_when_performance_and_mechanism_hit(): + champ = _results({"a": 100.0, "b": 100.0}) + cand = _results({"a": 90.0, "b": 88.0}) + d = decide(champ, cand, manifest=_manifest(), + mechanism_checks={"chg-1": True}) + assert d["verdict"] == "PROMOTE" + assert d["performance_gate"]["status"] == "PASS" + assert d["mechanism_gate"]["status"] == "HIT" + + +def test_lenient_default_promotes_unexplained_win_without_repeat(): + """Default policy is lenient: an unverified win still takes the champion.""" + champ = _results({"a": 100.0, "b": 100.0}) + cand = _results({"a": 90.0, "b": 90.0}) + d = decide(champ, cand, manifest=_manifest(), mechanism_checks={}) + assert d["verdict"] == "PROMOTE_UNEXPLAINED" + assert d["action"] == "SET_CHAMPION" + assert d["policy"]["mechanism_policy"] == "lenient" + + +def test_strict_policy_requires_confirmation_then_promotes(): + champ = _results({"a": 100.0, "b": 100.0}) + cand = _results({"a": 90.0, "b": 90.0}) + strict = DecisionPolicy(mechanism_policy="strict") + first = decide(champ, cand, manifest=_manifest(), mechanism_checks={}, + policy=strict) + assert first["verdict"] == "CONFIRM_REQUIRED" + second = decide(champ, cand, manifest=_manifest(), mechanism_checks={}, + confirmation_reproduced=True, policy=strict) + assert second["verdict"] == "PROMOTE_UNEXPLAINED" + assert second["action"] == "SET_CHAMPION" + + +def test_no_signal_keeps_champion_without_rejecting_candidate(): + champ = _results({"a": 100.0, "b": 100.0}) + cand = _results({"a": 99.5, "b": 101.0}) + d = decide(champ, cand, manifest=_manifest()) + assert d["verdict"] == "NO_SIGNAL" + assert d["action"] == "KEEP_CHAMPION_ARCHIVE_CANDIDATE" + + +def test_correctness_or_primary_regression_rejects(): + champ = _results({"a": 100.0, "b": 100.0}) + cand = _results({"a": 120.0, "b": 90.0}) + cand["b"]["correctness_ok"] = False + d = decide(champ, cand, manifest=_manifest()) + assert d["verdict"] == "REJECT" + assert d["action"] == "RESTORE_CHAMPION" + + +def test_heldout_regression_specializes_if_routeable_else_rejects_overfit(): + champ = _results({"a": 100.0, "b": 100.0}) + cand = _results({"a": 80.0, "b": 80.0}) + held_champ = _results({"h": 100.0}) + held_cand = _results({"h": 120.0}) + + specialized = decide( + champ, cand, manifest=_manifest(routeable=True), + mechanism_checks={"chg-1": True}, + heldout_champion=held_champ, heldout_candidate=held_cand, + ) + assert specialized["verdict"] == "SPECIALIZE" + assert specialized["action"] == "ARCHIVE_SPECIALIZED" + + rejected = decide( + champ, cand, manifest=_manifest(routeable=False), + mechanism_checks={"chg-1": True}, + heldout_champion=held_champ, heldout_candidate=held_cand, + ) + assert rejected["verdict"] == "REJECT_OVERFIT" + + +def test_harness_validate_error_detects_bad_yaml(tmp_path): + import shutil as _sh + from metainfer.tasks.dcu_kernel_auto_opt.orchestrator.harness_io import default_harness_dir + from ..orchestrator.pipeline import _harness_validate_error + wd = tmp_path / "ws" + _sh.copytree(default_harness_dir(), wd) + assert _harness_validate_error(wd) is None + (wd / "planner_policy.yaml").write_text("not: [valid\n", encoding="utf-8") + assert _harness_validate_error(wd) is not None + + +# --------------------------------------------------------- the 75% rule +# The performance gate is a question-count rule: >= 75% of the counted +# questions must be measurably faster than the champion ("3 of 4 beat the +# current variant"), with one hard red line for a single question that +# collapses (> 30% slower) or fails correctness. + +def test_three_of_four_question_wins_pass_the_performance_gate(): + champ = _results({"a": 100.0, "b": 100.0, "c": 100.0, "d": 100.0}) + cand = _results({"a": 90.0, "b": 90.0, "c": 91.0, "d": 110.0}) # d is 10% slower + d = decide(champ, cand, manifest=_manifest(), mechanism_checks={"chg-1": True}) + assert d["performance_gate"]["status"] == "PASS" + assert d["performance_gate"]["win_ratio"] == 0.75 + assert d["performance_gate"]["wins_needed"] == 3 + assert d["verdict"] == "PROMOTE" + + +def test_two_of_four_question_wins_fail_the_performance_gate(): + champ = _results({"a": 100.0, "b": 100.0, "c": 100.0, "d": 100.0}) + cand = _results({"a": 90.0, "b": 90.0, "c": 130.0, "d": 130.0}) + d = decide(champ, cand, manifest=_manifest(), mechanism_checks={"chg-1": True}) + assert d["verdict"] == "REJECT" + assert d["performance_gate"]["status"] == "FAIL" + assert d["performance_gate"]["win_ratio"] == 0.5 + assert "2/4 counted questions faster" in d["reason"] + + +def test_one_collapsing_question_rejects_despite_three_wins(): + """The red line: 3 small wins do not buy one 40%-slower question.""" + champ = _results({"a": 100.0, "b": 100.0, "c": 100.0, "d": 100.0}) + cand = _results({"a": 90.0, "b": 90.0, "c": 90.0, "d": 140.0}) + d = decide(champ, cand, manifest=_manifest(), mechanism_checks={"chg-1": True}) + assert d["verdict"] == "REJECT" + assert d["performance_gate"]["collapses"] == ["d"] + assert "hard red line" in d["reason"] + + +def test_three_questions_all_winning_passes(): + champ = _results({"a": 100.0, "b": 100.0, "c": 100.0}) + cand = _results({"a": 80.0, "b": 80.0, "c": 80.0}) + d = decide(champ, cand, manifest=_manifest(), mechanism_checks={"chg-1": True}) + assert d["policy"]["min_win_ratio"] == 0.75 + assert d["performance_gate"]["wins_needed"] == 3 + assert d["performance_gate"]["status"] == "PASS" + + +def test_a_draw_inside_the_noise_band_counts_against_the_ratio(): + """Only strictly measurable wins count: a tie is not a win.""" + champ = _results({"a": 100.0, "b": 100.0, "c": 100.0, "d": 100.0}) + cand = _results({"a": 90.0, "b": 90.0, "c": 100.0, "d": 130.0}) + d = decide(champ, cand, manifest=_manifest(), mechanism_checks={"chg-1": True}) + gate = d["performance_gate"] + assert gate["counts"]["NO_SIGNAL"] == 1 and gate["counts"]["WIN"] == 2 + assert gate["win_ratio"] == 0.5 + assert d["verdict"] == "REJECT" + + +def test_probe_questions_are_excluded_from_the_ratio(): + """A probe is generalisation evidence, not part of the 75% denominator.""" + champ = _results({"a": 100.0, "b": 100.0, "c": 100.0, "probe": 100.0}) + cand = _results({"a": 90.0, "b": 90.0, "c": 90.0, "probe": 200.0}) + d = decide(champ, cand, manifest=_manifest(), mechanism_checks={"chg-1": True}, + purposes={"probe": "probe"}) + # 3/3 counted questions won -> gate passes, but the probe regression blocks + assert d["performance_gate"]["win_ratio"] == 1.0 + assert d["generalization_gate"]["status"] == "FAIL" + assert d["verdict"] == "CONFIRM_REQUIRED" + + +def test_unusable_measurement_is_fatal_even_at_three_of_four(): + """A comparison missing one side is not "the candidate is faster".""" + champ = _results({"a": 100.0, "b": 100.0, "c": 100.0, "d": 100.0}) + cand = _results({"a": 90.0, "b": 90.0, "c": 90.0, "d": 100.0}) + cand["d"] = {"median_us": None, "passed": False, "correctness_ok": False, + "status": "missing_report"} + d = decide(champ, cand, manifest=_manifest(), mechanism_checks={"chg-1": True}) + assert d["performance_gate"]["counts"]["INVALID"] == 1 + assert d["verdict"] == "REJECT" + assert "unusable measurement" in d["reason"] diff --git a/metainfer/tasks/harness_evolve/tests/test_evolve_failure.py b/metainfer/tasks/harness_evolve/tests/test_evolve_failure.py new file mode 100644 index 00000000..84edf825 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_evolve_failure.py @@ -0,0 +1,87 @@ +"""AgentEvolver must never crash the outer loop. + +A timed-out or failed evolve turn keeps the iteration's evaluation value: +apply() returns a no-change placeholder (champion unchanged) so the pipeline +still writes evolve artifacts + report + finished. +""" + +from __future__ import annotations + +import subprocess + +import pytest + +from metainfer.tasks.harness_evolve.orchestrator import evolve as ev + + +def _ctx() -> dict: + return { + "overview": "/tmp/overview.md", + "results": {}, + "diff": {}, + "decision": {"verdict": "BASELINE"}, + "selected": ["a"], + "pool_ids": ["a", "b"], + } + + +def test_timeout_returns_placeholder(tmp_path, monkeypatch): + def _boom(*_a, **_kw): + raise subprocess.TimeoutExpired(cmd="dsh_agent.py", timeout=1800) + + monkeypatch.setattr(ev.subprocess, "run", _boom) + out = ev.AgentEvolver(timeout_seconds=1800).apply(tmp_path, 3, _ctx()) + assert out["changed"] is False + assert out["round_plan"] is None + assert out["manifest"]["changes"] == [] + assert out["manifest"]["verification"]["status"] == "evolve_failed" + assert "timed out" in out["manifest"]["note"] + assert "failed" in out["summary"] + + +def test_nonzero_exit_returns_placeholder(tmp_path, monkeypatch): + class _P: + returncode = 1 + stdout = "" + stderr = "boom: agent crashed" + + monkeypatch.setattr(ev.subprocess, "run", lambda *a, **k: _P()) + out = ev.AgentEvolver().apply(tmp_path, 1, _ctx()) + assert out["manifest"]["verification"]["status"] == "evolve_failed" + assert "rc=1" in out["manifest"]["note"] + + +def test_missing_artifacts_returns_placeholder(tmp_path, monkeypatch): + class _P: + returncode = 0 + stdout = "" + stderr = "" + + monkeypatch.setattr(ev.subprocess, "run", lambda *a, **k: _P()) + out = ev.AgentEvolver().apply(tmp_path, 1, _ctx()) + assert out["manifest"]["verification"]["status"] == "evolve_failed" + assert "did not write" in out["manifest"]["note"] + + +def test_successful_evolve_still_returns_artifacts(tmp_path, monkeypatch): + class _P: + returncode = 0 + stdout = '{"type": "result", "result": "done"}\n' + stderr = "" + + def _run(*_a, **_kw): + # The agent writes its artifacts during the run (apply clears them + # before spawning), so emulate that side effect here. + (tmp_path / ev.AgentEvolver.MANIFEST_NAME).write_text( + '{"schema_version": 1, "iteration": 1, "changes": [{"id": "c1"}],' + ' "verification": {"status": "pending"}}', encoding="utf-8") + (tmp_path / ev.AgentEvolver.PLAN_NAME).write_text( + '{"iteration": 1, "strategy": "balanced", "selected": ["a"]}', + encoding="utf-8") + return _P() + + monkeypatch.setattr(ev.subprocess, "run", _run) + out = ev.AgentEvolver().apply(tmp_path, 1, _ctx()) + assert out["changed"] is True + assert out["round_plan"]["selected"] == ["a"] + assert out["summary"] == "done" diff --git a/metainfer/tasks/harness_evolve/tests/test_flow_contract.py b/metainfer/tasks/harness_evolve/tests/test_flow_contract.py new file mode 100644 index 00000000..af83b0c0 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_flow_contract.py @@ -0,0 +1,220 @@ +"""The HE flow is a fixed contract: `FLOW.md` states it, the code obeys it. + +Three things are pinned here, all of them from the operator's own wording: + +1. **The document exists and still says the protocol** — question counts are + multiples of 4, round 1 draws the frozen generalization paper from a random + set that spans decode *and* prefill, the two gates are 75% / 50% with an 80% + floor, and a failed performance gate goes back to analyze→evolve instead of + being promoted. +2. **The multiple of 4 is enforced, not suggested** — the form offers only + multiples of 4, and a count that arrives from the CLI is rounded up at the + single point where it enters the run (``config._normalized_budget``). +3. **HE only launches DKAO tasks** — no device reads, no waiting on cards, no + freezing a child for a device. Every one of those existed at some point and + was removed; this test is what keeps them out. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest + +HE_PKG = Path(__file__).resolve().parents[1] +ORCH = HE_PKG / "orchestrator" + + +def _read(path: Path) -> str: + return path.read_text(encoding="utf-8") + + +def _code_only(path: Path) -> str: + """Source with comments and string literals removed. + + A contract test has to judge *code*: the comments in ``eval.py`` explaining + that the device verdict belongs to DKAO are exactly what we want to keep, + and they would otherwise read as "HE mentions the measurement gate". + """ + import io + import tokenize + + tokens = [] + with path.open("rb") as fh: + for tok in tokenize.tokenize(fh.readline): + if tok.type in (tokenize.COMMENT, tokenize.STRING, tokenize.NL, + tokenize.NEWLINE, tokenize.INDENT, + tokenize.DEDENT): + continue + tokens.append(tok.string) + return " ".join(tokens) + + +# --------------------------------------------------------------- 1. the document + +def test_the_flow_document_exists_and_is_called_out_by_the_readme(): + flow = HE_PKG / "FLOW.md" + assert flow.is_file(), "the fixed protocol document is missing" + assert "FLOW.md" in _read(HE_PKG / "README.md"), ( + "README must point at FLOW.md instead of restating the protocol") + + +def test_the_flow_document_states_every_rule_the_operator_fixed(): + doc = _read(HE_PKG / "FLOW.md") + required = [ + "4 的倍数", # question count is a multiple of 4 + "decode", "prefill", # round 1 spans both regimes + "M ≤ 32", # decode definition + "A1", "A_i", # round-1 paper vs later draws + "75%", "50%", "80%", # the two gates and the floor + "仅在性能门通过后开", # generalization only after performance + "固化考卷", # the paper never moves + "只发起 DKAO 任务", # the boundary, in one line + "不租卡", "不读卡状态", "不 `SIGSTOP`", # what HE must not do + "analyze", "evolve", # the failure path + "awaiting_approval", # promotion waits for a human + ] + missing = [token for token in required if token not in doc] + assert not missing, f"FLOW.md no longer states: {missing}" + + +# ------------------------------------------------- 2. the multiple-of-4 rule + +def test_the_machine_and_the_wave_are_both_four(): + from metainfer.tasks.harness_evolve.orchestrator.rounds import ( + GPU_COUNT, OPERATORS_PER_GATE, normalized_per_gate, + ) + + assert GPU_COUNT == 4 and OPERATORS_PER_GATE == 4 + assert normalized_per_gate(4) == 4 + assert normalized_per_gate(8) == 8 + assert normalized_per_gate(5) == 8 # up, never down + assert normalized_per_gate(3) == 4 + assert normalized_per_gate(None) == 4 + + +def test_the_form_only_offers_multiples_of_four(): + import metainfer.tasks # noqa: F401 - registers the plugins + from metainfer.server.forms import load_form_schema + + schema = load_form_schema("harness-evolve") + field = next(f for f in schema["fields"] if f["key"] == "per_round_budget") + + assert field["type"] == "select", ( + "a free number field lets the operator ask for 3 or 5 questions, which " + "leaves devices idle in the wave") + counts = [int(option["label"]) for option in field["options"]] + assert counts and all(count % 4 == 0 for count in counts), counts + assert int(field["default"]) % 4 == 0 + + +def test_a_count_from_the_cli_is_rounded_up_where_it_enters_the_run(): + from metainfer.tasks.harness_evolve.orchestrator.config import ( + _normalized_budget, + ) + + assert _normalized_budget(3) == 4 + assert _normalized_budget(5) == 8 + assert _normalized_budget(12) == 12 + assert _normalized_budget(0) == 0, ( + "0 keeps the legacy meaning 'no cap / whole pool'") + assert _normalized_budget("bad") == 0 + + +def test_the_round_is_drawn_from_the_pool_in_whole_waves(): + """The selection itself honours the count it is given (no silent shrink).""" + import random + + from metainfer.tasks.harness_evolve.orchestrator.rounds import ( + select_round_questions, + ) + + instances = {} + for idx in range(8): + m = 16 if idx % 2 == 0 else 4096 + regime = "decode" if m <= 32 else "prefill" + iid = f"op{idx:02d}" + instances[iid] = { + "baseline_us": 100.0 * (idx + 1), + "best_known_us": 40.0 * (idx + 1), # an existing variant + "family": f"{regime}__fam{idx}", + "contract": {"M": m, "N": 64, "K": 32}, + } + table = {"operators": {}, "groups": {}, "harness_version": None} + + picked = select_round_questions(iteration=1, table=table, + instances=instances, state={}, count=8, + rng=random.Random(7)) + + assert len(picked["performance_ids"]) == 8 + assert picked["generalization_ids"] == picked["performance_ids"], ( + "round 1's draw *is* the frozen paper") + assert picked["defines_paper"] is True + + +# ------------------------------------------- 3. HE does not manage environment + +def test_the_supervisor_never_reads_the_cards(): + from metainfer.tasks.harness_evolve.orchestrator.supervisor import Supervisor + + code = _code_only(ORCH / "supervisor.py") + for forbidden in ("gpu_preflight", "preflight_gpus", "free_gb", + "sample_gpu_state", "foreign_kfd_pids", "GpuBroker"): + assert forbidden not in code, ( + f"HE must not consult {forbidden}: a device verdict is DKAO's " + "(FLOW.md §0)") + assert not hasattr(Supervisor, "free_gb") + + +def test_nothing_freezes_a_child_for_a_device_any_more(): + src = _code_only(ORCH / "process_control.py") + for forbidden in ("signal.SIGSTOP", "signal.SIGCONT", "def freeze", + "def resume("): + assert forbidden not in src, forbidden + # terminate is what remains, and it must still work as a group signal + assert "def terminate" in src and "killpg" in src + assert "def freeze" not in src + + +def test_classify_incident_is_driven_by_the_runs_own_evidence(): + from metainfer.tasks.harness_evolve.orchestrator.supervisor import ( + CATEGORY_AGENT, CATEGORY_CODE, CATEGORY_RESOURCE, CATEGORY_UNKNOWN, + classify_incident, + ) + + assert classify_incident(exit_code=137)[0] == CATEGORY_RESOURCE + assert classify_incident(log_tail="std::bad_alloc")[0] == CATEGORY_RESOURCE + assert classify_incident(log_tail="TransportClosedError")[0] == CATEGORY_AGENT + assert classify_incident( + log_tail="Traceback (most recent call last):")[0] == CATEGORY_CODE + assert classify_incident(log_tail="nothing to see")[0] == CATEGORY_UNKNOWN + with pytest.raises(TypeError): + classify_incident(log_tail="x", free_gb=1.0) # the knob is gone + + +def test_the_evaluator_only_decides_the_device_layout(): + src = _code_only(ORCH / "adapters" / "eval.py") + assert "device_for_index" in src, ( + "the device is a pure layout decision (index % 4)") + for forbidden in ("ensure_measurement_gate", "MEASUREMENT_GATE_VRAM_PERCENT", + "measurement_suspect", "contention", "gpu_preflight", + "preflight_gpus"): + assert forbidden not in src, ( + "environment verdicts live in DKAO, not in HE") + + +def test_the_gate_numbers_are_the_protocol_numbers(): + from metainfer.tasks.harness_evolve.orchestrator.decision_engine import ( + GENERALIZATION_GATE_FLOOR_PERCENT, GENERALIZATION_GATE_WIN_RATIO, + PERFORMANCE_GATE_FLOOR_PERCENT, PERFORMANCE_GATE_WIN_RATIO, + ) + from metainfer.tasks.harness_evolve.orchestrator.rounds import ( + GENERALIZATION_ROUND_COST, PERFORMANCE_ROUND_COST, + ) + + assert PERFORMANCE_GATE_WIN_RATIO == 0.75 + assert GENERALIZATION_GATE_WIN_RATIO == 0.50 + assert PERFORMANCE_GATE_FLOOR_PERCENT == 0.80 + assert GENERALIZATION_GATE_FLOOR_PERCENT == 0.80 + assert PERFORMANCE_ROUND_COST == 1.0 + assert GENERALIZATION_ROUND_COST == 0.5 diff --git a/metainfer/tasks/harness_evolve/tests/test_form_rounds.py b/metainfer/tasks/harness_evolve/tests/test_form_rounds.py new file mode 100644 index 00000000..190a3b2a --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_form_rounds.py @@ -0,0 +1,137 @@ +"""The two round-ish numbers in the New Task form must mean what they say. + +They used to be easy to confuse, and confusing them had a real cost: the outer +loop's bound came from a hidden ``max_rounds`` (default 10) plus whatever target +the WebUI happened to carry, while the form's own answer was only recorded in +the report. A run therefore kept producing rounds nobody had asked for. + +The operator's rule now is one round field: + +* **Max iters** (``default_rounds``) — *one question's* optimization budget + inside DKAO. It is not a round count and must reach the child task unchanged. +* **Rounds** (``rounds``) — *the whole task's* limit, counted in iterations: the + baseline round is round 1, and every later round is a performance pass plus + (when it wins) the generalization retake inside that same iteration. Passing + both gates in round 5 of 10 ends the run there, waiting for approval. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +from metainfer.tasks.harness_evolve.orchestrator import cli as he_cli +from metainfer.tasks.harness_evolve.orchestrator.config import ( + load_experiment_config, +) +from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + COST_GUARDRAIL, ROUNDS_DEFAULT, _rounds_limit, +) +from metainfer.tasks.harness_evolve.orchestrator.state import ( + read_target_iterations, +) + + +def _write_req(tmp_path: Path, **answers) -> Path: + tmp_path.mkdir(parents=True, exist_ok=True) + payload = { + "task_id": "form-rounds", + "task_type": "harness-evolve", + "execution_mode": "dry-run", + "evolve_mode": "dry-run", + "pool_source": "", + } + payload.update(answers) + req = tmp_path / "requirements.json" + req.write_text(json.dumps(payload), encoding="utf-8") + return req + + +def _cfg(tmp_path: Path, **answers): + return load_experiment_config( + _write_req(tmp_path, **answers), tmp_path / "state", tmp_path / "ws") + + +# ---------------------------------------------------------------- Max iters + +def test_max_iters_reaches_each_child_task_unchanged(tmp_path): + """``default_rounds`` is a per-question DKAO budget, not a round count.""" + cfg = _cfg(tmp_path, default_rounds="7") + assert cfg.answers["default_rounds"] == "7" + from metainfer.tasks.harness_evolve.orchestrator.adapters.eval import ( + DkaoCliEvaluator, + ) + from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + _variant_instance_spec, + ) + + spec = _variant_instance_spec( + {"id": "op", "baseline_us": 10.0, "best_known_us": 9.0}, + budget_rounds=int(cfg.answers["default_rounds"])) + assert spec.budget_rounds == 7 + # and it is handed to DKAO as its own "Max iters" field + req = DkaoCliEvaluator()._build_dkao_requirements( + cfg, spec, 1, "child-id", "repo-name", 0, tmp_path / "ws") + assert req["max_iterations"] == "7" + + +# ------------------------------------------------------------------- Rounds + +def test_rounds_is_the_only_round_limit(tmp_path): + """There is one limit, counted in rounds, and a round is an iteration.""" + assert _rounds_limit(_cfg(tmp_path, rounds=4)) == 4 + # the pre-rename field name is still honoured + assert _rounds_limit(_cfg(tmp_path / "b", max_iterations=6)) == 6 + # absent: the form's default + assert _rounds_limit(_cfg(tmp_path / "c")) == ROUNDS_DEFAULT + # nonsense is clamped into a sane range + assert _rounds_limit(_cfg(tmp_path / "d", rounds=9999)) == 50 + assert _rounds_limit(_cfg(tmp_path / "e", rounds=-3)) == 1 + # the cost guardrail can never be the thing that stops a real run + assert COST_GUARDRAIL >= 4 * ROUNDS_DEFAULT + + +def test_rounds_is_pinned_where_the_loop_and_the_webui_read_it(tmp_path): + """The form's answer becomes the run's target before round 1.""" + cfg = _cfg(tmp_path, rounds=7) + assert read_target_iterations(cfg.exp_dir) is None + he_cli._seed_rounds_from_form(cfg, cfg.exp_dir) + assert read_target_iterations(cfg.exp_dir) == 7 + # a runaway value is clamped exactly like the WebUI's own target box + cfg2 = _cfg(tmp_path / "big", rounds=9999) + he_cli._seed_rounds_from_form(cfg2, cfg2.exp_dir) + assert read_target_iterations(cfg2.exp_dir) == 50 + + +# --------------------------------------------------- the form's own contract + +def test_the_form_has_one_round_field_and_it_is_machine_readable(): + """Loadable *and* parseable: a select submits its label. + + The explanation must not live in an option label (the backend runs ``int()`` + on some answers), and the outer loop must expose exactly one round field. + """ + import metainfer.tasks # noqa: F401 - registers the plugins + from metainfer.server.forms import load_form_schema, validate_submission + + schema = load_form_schema("harness-evolve") + assert schema is not None and schema["fields"] + keys = [f["key"] for f in schema["fields"]] + by_key = {f["key"]: f for f in schema["fields"]} + + assert "rounds" in keys and "max_rounds" not in keys, ( + "the outer loop exposes exactly one round field") + for field in ("default_rounds", "rounds", "per_round_budget"): + assert str(by_key[field]["default"]).isdigit() + for option in by_key["default_rounds"]["options"]: + int(option["label"]) # raises if a label grew an explanation + + # the two fields name their layer, so "Max iters" cannot be read as rounds + assert "单题内部" in by_key["default_rounds"]["help"] + assert "任务上限" in by_key["rounds"]["help"] + + answers = {k: str(f["default"]) for k, f in by_key.items() + if f.get("default") not in (None, "")} + answers["pool_source"] = "/root/zth_agent/ahe-kernel-repos/registered_pool.yaml" + result = validate_submission("harness-evolve", answers) + assert result["ok"], result.get("errors") diff --git a/metainfer/tasks/harness_evolve/tests/test_four_card_overview.py b/metainfer/tasks/harness_evolve/tests/test_four_card_overview.py new file mode 100644 index 00000000..fb60ed78 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_four_card_overview.py @@ -0,0 +1,123 @@ +"""The four-card overview: what the page shows, and where the numbers come from. + +Two halves, and both are pinned because each has been wrong in production: + +1. **Membership** — which card a question is shown under. It comes from HE's + layout record for the round, *not* from the child's last gate row: the + generate-stage probe may run on any passing device, so three children pinned + to GPU0/1/2 were all listed under GPU2 and the page showed two cards instead + of four. (``tests/test_gpu_cards.py`` pins the precedence in detail; here the + payload contract the view reads is pinned.) +2. **Content** — the view reads only fields the endpoint really provides, and it + is wired into the detail page (a component nobody renders is not a feature). +""" + +from __future__ import annotations + +import json +from pathlib import Path + +from metainfer.tasks.harness_evolve.server.routes import _gpu_cards + +HE = Path(__file__).resolve().parents[1] +JS = HE / "static" / "he-detail.js" +CSS = HE / "static" / "he.css" + + +def _seed(exp: Path, num: int, qid: str, *, layout_gpu: int, phase: str = "parallel_explore") -> None: + attempt = exp / "children" / f"iteration_{num:03d}" / qid + state = attempt / "state" + state.mkdir(parents=True, exist_ok=True) + (state / "run.json").write_text(json.dumps({ + "task_id": qid, "task_type": "dcu-kernel-auto-opt", + "current_phase": phase, "current_iteration": 2, + "finished": False, "last_update": 1790066400.0, + }), encoding="utf-8") + (attempt / "requirements.json").write_text(json.dumps({ + "task_id": qid, + "shape_config": ("model: m\nassignment_mode: manual\nshapes:\n- id: " + qid + + "\nassignments:\n worker_0:\n gpu: " + str(layout_gpu) + + "\n shapes:\n - " + qid + "\n"), + }), encoding="utf-8") + + +def _layout(exp: Path, num: int, mapping: dict) -> None: + bench = exp / "runs" / f"iteration_{num:03d}" / "input" / "benchmark" + bench.mkdir(parents=True, exist_ok=True) + (bench / "gpu_preflight.json").write_text(json.dumps({ + "iteration": num, "gpu_count": 4, "managed_by": "dcu_kernel_auto_opt", + "devices": mapping, + }), encoding="utf-8") + + +def test_a_four_question_round_becomes_four_cards(tmp_path): + exp = tmp_path / "exp" + questions = {"q0": 0, "q1": 1, "q2": 2, "q3": 3} + _layout(exp, 1, questions) + for qid, gpu in questions.items(): + _seed(exp, 1, qid, layout_gpu=gpu) + + cards = _gpu_cards(exp, 1) + + assert [c["gpu"] for c in cards] == [0, 1, 2, 3], ( + "one card per question — the view must be able to show all four") + for card in cards: + assert len(card["questions"]) == 1 + q = card["questions"][0] + # every field the overview renders + assert {"gpu", "question", "current_phase", "current_iteration", + "best_median_us", "rounds", "finished", "last_update", + "gate", "gate_audit", "gpu_source"} <= set(q) + assert q["gpu_source"] == "layout" + assert q["gate_audit"].get("devices") is not None + + +def test_a_grouped_card_still_carries_every_question(tmp_path): + """Two questions on one card (a second wave) must both be listed.""" + exp = tmp_path / "exp" + _layout(exp, 2, {"q0": 0, "q1": 0}) + _seed(exp, 2, "q0", layout_gpu=0) + _seed(exp, 2, "q1", layout_gpu=0) + + cards = _gpu_cards(exp, 2) + + assert len(cards) == 1 and cards[0]["gpu"] == 0 + assert [q["question"] for q in cards[0]["questions"]] == ["q0", "q1"] + + +# --------------------------------------------------------------- the view side + +def test_the_four_card_component_exists_and_is_rendered(): + src = JS.read_text(encoding="utf-8") + assert "function FourCardOverview(" in src + assert "<${FourCardOverview} data=${iterGpu}" in src, ( + "the overview must be rendered with the round's card payload") + assert 'class="he-four-cards"' in src + + +def test_the_view_reads_only_fields_the_endpoint_provides(): + src = JS.read_text(encoding="utf-8") + component = src.split("function FourCardOverview(", 1)[1].split( + "function PromotionPanel(", 1)[0] + for field in ("current_phase", "current_iteration", "best_median_us", + "rounds", "finished", "last_update", "gate_audit", + "gpu_source", "question"): + assert field in component, f"the card cell should show {field}" + # and it must not invent a metric the backend never sends + assert "p90_us" not in component + + +def test_the_grid_is_four_columns_with_a_narrow_screen_fallback(): + css = CSS.read_text(encoding="utf-8") + block = css.split(".he-four-cards {", 1)[1].split("}", 1)[0] + assert "repeat(4" in block, "four cards side by side" + assert "@media" in css.split(".he-four-cards {", 1)[1][:400], ( + "narrow screens fall back to two columns") + + +def test_the_overview_click_opens_the_child_drilldown(): + src = JS.read_text(encoding="utf-8") + component = src.split("function FourCardOverview(", 1)[1].split( + "function PromotionPanel(", 1)[0] + assert "onSelect(q.question)" in component + assert "onSelect=${(id) => setSelChild(id)}" in src diff --git a/metainfer/tasks/harness_evolve/tests/test_freeze_and_review.py b/metainfer/tasks/harness_evolve/tests/test_freeze_and_review.py new file mode 100644 index 00000000..ae908624 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_freeze_and_review.py @@ -0,0 +1,125 @@ +"""Scheme 1: frozen variants, best-known tracking, final promotion, review.""" + +from __future__ import annotations + +import json +from pathlib import Path +from types import SimpleNamespace + +from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + _finalize_promotion, _track_best_known, _variant_policy, +) + + +def _cfg(tmp_path: Path, **answers) -> SimpleNamespace: + state = tmp_path / "st" + state.mkdir(parents=True, exist_ok=True) + return SimpleNamespace(state_dir=state, task_id="t", + answers={"promote_kernels": "true", **answers}) + + +def test_variant_policy_defaults_to_freeze(tmp_path): + assert _variant_policy(_cfg(tmp_path)) == "freeze" + assert _variant_policy(_cfg(tmp_path, variant_policy="per_round")) == "per_round" + assert _variant_policy(_cfg(tmp_path, variant_policy="immediate")) == "per_round" + + +def test_best_known_curve_and_regression_detection(tmp_path): + cfg = _cfg(tmp_path) + exp = tmp_path / "exp" + exp.mkdir() + + _track_best_known(cfg, exp, 1, {"a": {"median_us": 40.0, "passed": True}}) + first = json.loads((exp / "best_known.json").read_text(encoding="utf-8")) + assert first["a"]["best_median_us"] == 40.0 + assert first["a"]["best_iteration"] == 1 + + out = _track_best_known(cfg, exp, 2, {"a": {"median_us": 60.0, "passed": True}}) + assert out["regressions"] and out["regressions"][0]["shape"] == "a" + assert out["regressions"][0]["best_median_us"] == 40.0 + assert out["regressions"][0]["delta_percent"] == 50.0 + assert (exp / "regressions.jsonl").is_file() + + data = json.loads((exp / "best_known.json").read_text(encoding="utf-8")) + assert data["a"]["best_median_us"] == 40.0 # 60us never becomes best + assert data["a"]["best_iteration"] == 1 + assert [h["median_us"] for h in data["a"]["history"]] == [40.0, 60.0] + + # a better round updates the best + _track_best_known(cfg, exp, 3, {"a": {"median_us": 25.0, "passed": True}}) + data = json.loads((exp / "best_known.json").read_text(encoding="utf-8")) + assert data["a"]["best_median_us"] == 25.0 and data["a"]["best_iteration"] == 3 + + +def test_finalize_promotion_promotes_the_best_candidate_not_the_newest( + tmp_path, monkeypatch): + """40us from round 1 must win over 60us from round 2.""" + cfg = _cfg(tmp_path) + exp = tmp_path / "exp" + good = exp / "children" / "iteration_001" / "shapeA" / "workspace" + worse = exp / "children" / "iteration_002" / "shapeA" / "workspace" + good.mkdir(parents=True) + worse.mkdir(parents=True) + rows = [ + {"iteration": 1, "shape": "shapeA", "workspace": str(good), + "ahe_passed": True, "action": "would_update", "ahe_median_us": 40.0, + "answers": {"model": "hy3"}, "source_task": "t/1"}, + {"iteration": 2, "shape": "shapeA", "workspace": str(worse), + "ahe_passed": True, "action": "would_update", "ahe_median_us": 60.0, + "answers": {"model": "hy3"}, "source_task": "t/2"}, + {"iteration": 2, "shape": "shapeB", "workspace": str(worse), + "ahe_passed": False, "action": "would_add", "ahe_median_us": 5.0, + "answers": {}}, + ] + (exp / "promotion_candidates.jsonl").write_text( + "\n".join(json.dumps(r) for r in rows) + "\n", encoding="utf-8") + + calls = [] + import metainfer.tasks.dcu_kernel_auto_opt.orchestrator.variant_promote as vp + monkeypatch.setattr(vp, "promote_variant", + lambda **kw: calls.append(kw) or { + "shape": kw["shape_id"], "action": "updated"}) + + payload = _finalize_promotion(cfg, exp) + assert payload["promoted"] == ["shapeA"] + assert len(calls) == 1 + assert calls[0]["workspace_dir"] == good # the 40us candidate + assert Path(str(payload["results"][0]["shape"])) == Path("shapeA") + assert (exp / "final_promotion.json").is_file() + # a failing candidate is never promoted, however fast it looks + assert "shapeB" not in payload["promoted"] + + +def test_finalize_promotion_respects_the_switch(tmp_path): + cfg = _cfg(tmp_path, promote_kernels="false") + exp = tmp_path / "exp" + exp.mkdir() + (exp / "promotion_candidates.jsonl").write_text("{}\n", encoding="utf-8") + assert _finalize_promotion(cfg, exp) is None + + +def test_fail_streak_counts_failures_and_resets_on_progress(): + from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + _next_fail_streak, + ) + assert _next_fail_streak("REJECT", 0) == 1 + assert _next_fail_streak("REJECT_OVERFIT", 1) == 2 + # unmeasurable rounds count too: that is the runaway-loop signature + assert _next_fail_streak("INCONCLUSIVE", 2) == 3 + assert _next_fail_streak("NO_SIGNAL", 3) == 0 + assert _next_fail_streak("PROMOTE", 3) == 0 + assert _next_fail_streak("PROMOTE_UNEXPLAINED", 3) == 0 + + +def test_circuit_breaker_trips_after_three_failing_rounds(): + from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + _next_fail_streak, + ) + streak = 0 + tripped_at = None + for round_no, verdict in enumerate(["REJECT", "INCONCLUSIVE", "REJECT"], 1): + streak = _next_fail_streak(verdict, streak) + if streak >= 3: + tripped_at = round_no + break + assert tripped_at == 3 diff --git a/metainfer/tasks/harness_evolve/tests/test_gate_display_contract.py b/metainfer/tasks/harness_evolve/tests/test_gate_display_contract.py new file mode 100644 index 00000000..211cfc1a --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_gate_display_contract.py @@ -0,0 +1,431 @@ +"""What the round table's gate column shows, and where those numbers come from. + +The page used to print ``tau_family * baseline_us`` — the *pool* criterion — under +the header "pass gate ≤ us". On the variant-round protocol (FLOW.md §3, the fixed +protocol for every HE task since 2026-09-15) that number is not the gate: a round +is judged by pairing each question against that operator's current **variant**, +needing a strict win for >= 75% of the questions with nothing below 80% of its +variant. Showing the pool criterion there made a 20.00x requirement look like the +bar while the real bar was ~163x, i.e. it hid the very thing the round is judged +on — and on an old run it read as if a kernel 2-4x slower than the variant had +passed a variant comparison. + +Two halves, both pinned because each can be wrong independently: + +* the endpoint must serve the round's real criterion (``_real_gate_map``, driven + end to end through ``_children_live``); +* the view must read it, keep the pool criterion only as a labelled annotation, + and still render the legacy path — which really *was* judged by that number. +""" + +from __future__ import annotations + +import json +import re +import shutil +import subprocess +from pathlib import Path + +import pytest +import yaml + +from metainfer.tasks.harness_evolve.server.routes import ( + _children_live, + _iterations, +) + +HE = Path(__file__).resolve().parents[1] +JS = HE / "static" / "he-detail.js" + +#: One pool instance whose *pool* criterion (tau * baseline = 0.5 * 1000) is far +#: from its variant (200): the two can never be confused in an assertion. +POOL_BASELINE_US = 1000.0 +POOL_VARIANT_US = 200.0 +POOL_TARGET_US = 500.0 + + +def _pool_file(tmp_path: Path) -> Path: + path = tmp_path / "registered_pool.yaml" + path.write_text(yaml.safe_dump({ + "schema_version": 1, + "instances": [{ + "id": "q0", + "contract": {"model": "m", "tp_size": 8, "operator": "o_proj", + "M": 4096, "N": 64, "K": 32}, + "family": "prefill__fam", + "baseline_us": POOL_BASELINE_US, + "best_known_us": POOL_VARIANT_US, + # One accepted record with ratio 0.5 pins tau_family at 0.5. + "history": [{"accepted": True, "median_us": POOL_TARGET_US}], + }], + }), encoding="utf-8") + return path + + +def _child(exp: Path, num: int, qid: str = "q0") -> None: + """A finished-looking DKAO child with one accepted measurement.""" + d = exp / "children" / f"iteration_{num:03d}" / qid + (d / "state").mkdir(parents=True, exist_ok=True) + (d / "state" / "run.json").write_text(json.dumps({ + "task_id": qid, "task_type": "dcu-kernel-auto-opt", + "current_phase": "finished", "current_iteration": 3, + "finished": True, "final_status": "success", + }), encoding="utf-8") + run = d / "workspace" / "workers" / "worker_0" / "runs" / qid + run.mkdir(parents=True, exist_ok=True) + (run / "experiments.jsonl").write_text(json.dumps({ + "iteration": 1, "accepted": True, "metrics": {"median_us": 150.0}, + }) + "\n", encoding="utf-8") + + +def _variant_table(exp: Path, median_us: float = POOL_VARIANT_US, + kernel_source: str = "pool:best_known") -> None: + (exp / "variant_table.json").write_text(json.dumps({ + "schema_version": 1, "operators": { + "q0": {"median_us": median_us, "kernel_source": kernel_source, + "conditions": {"origin": "registered_pool"}}, + }, "groups": {}, + }), encoding="utf-8") + + +def _round(exp: Path, num: int, stage: str | None, *, + gate: dict | None = None, gate_name: str = "performance_gate.json", + ) -> None: + i_dir = exp / "runs" / f"iteration_{num:03d}" / "input" + (i_dir / "benchmark").mkdir(parents=True, exist_ok=True) + if stage is not None: + (i_dir / "stage.json").write_text(json.dumps({ + "iteration": num, "stage": stage, "round_cost": 1.0, + }), encoding="utf-8") + if gate is not None: + (i_dir / gate_name).write_text(json.dumps(gate), encoding="utf-8") + + +def _gate_doc(status: str = "PASS", verdict: str = "WIN", + delta: float = 12.5) -> dict: + return { + "kind": "performance", "status": status, "win_ratio": 1.0, + "required_win_ratio": 0.75, "operators": ["q0"], "wins": ["q0"], + "losses": [], "ties": [], "counted": 1, "wins_needed": 1, + "unmeasured": [], "below_floor": [], "reasons": [], + "comparison": {"per_instance": {"q0": { + "champion_median_us": POOL_VARIANT_US, + "candidate_median_us": 175.0, + "delta_percent": delta, + "verdict": verdict, + }}}, + } + + +def _answers(pool: Path) -> dict: + return {"pool_source": str(pool)} + + +# ----------------------------------------------------------- what the gate is + + +def test_a_performance_round_is_judged_against_the_variant(tmp_path): + exp = tmp_path / "exp" + pool = _pool_file(tmp_path) + _child(exp, 2) + _variant_table(exp) + _round(exp, 2, "performance", gate=_gate_doc()) + + gate = _children_live(exp, 2, _answers(pool))["q0"]["gate"] + + assert gate["protocol"] == "variant" + assert gate["stage"] == "performance" + # the number the question must beat is the variant, not tau * baseline + assert gate["variant_us"] == POOL_VARIANT_US + assert gate["target_us"] == POOL_VARIANT_US + assert gate["judged"]["verdict"] == "WIN" + assert gate["judged"]["delta_percent"] == 12.5 + assert gate["win_ratio"] == 0.75 + assert gate["floor_percent"] == 0.80 + assert gate["noise_percent"] == 2.0 + + +def test_the_pool_criterion_is_served_separately_and_labelled(tmp_path): + """It must still be reachable (old runs used it) — but never as the gate.""" + exp = tmp_path / "exp" + pool = _pool_file(tmp_path) + _child(exp, 2) + _variant_table(exp) + _round(exp, 2, "performance", gate=_gate_doc()) + + live = _children_live(exp, 2, _answers(pool))["q0"] + + assert live["gate"]["target_us"] != live["pool_reference"]["target_us"], ( + "the two criteria must not collapse into one number") + assert live["pool_reference"]["target_us"] == POOL_TARGET_US + assert live["pool_reference"]["tau_family"] == 0.5 + assert live["pool_reference"]["baseline_us"] == POOL_BASELINE_US + assert live["gate_protocol"] == "variant" + + +def test_the_baseline_round_has_an_opponent_but_no_verdict(tmp_path): + """Round 1 judges nothing (FLOW.md §2) — but it does fix the opponent.""" + exp = tmp_path / "exp" + pool = _pool_file(tmp_path) + _child(exp, 1) + _variant_table(exp) + _round(exp, 1, "baseline") + + gate = _children_live(exp, 1, _answers(pool))["q0"]["gate"] + + assert gate["protocol"] == "variant" + assert gate["stage"] == "baseline" + assert gate["judged"] is None + assert gate["target_us"] == POOL_VARIANT_US + + +def test_a_legacy_round_keeps_the_criterion_it_was_judged_by(tmp_path): + """No stage, no gate document, no variant table: the pre-2026-09-15 path. + + Those rounds were decided by pool.auto_pass, so that is the target their page + must keep showing — labelled as legacy, not as a variant comparison. + """ + exp = tmp_path / "exp" + pool = _pool_file(tmp_path) + _child(exp, 3) + _round(exp, 3, None) + + live = _children_live(exp, 3, _answers(pool))["q0"] + + assert live["gate"]["protocol"] == "legacy" + assert live["gate"]["judged"] is None + assert live["gate"]["target_us"] == POOL_TARGET_US + assert live["pool_reference"]["target_us"] == POOL_TARGET_US + + +def test_the_round_payload_carries_its_stage_and_gate(tmp_path): + """The page renders the round's own gate document, so serve it per round.""" + exp = tmp_path / "exp" + _child(exp, 2) + _round(exp, 2, "performance", gate=_gate_doc(status="FAIL", + verdict="LOSS", delta=-18.0)) + + entry = _iterations(exp)[0] + + assert entry["stage"]["stage"] == "performance" + assert entry["performance_gate"]["status"] == "FAIL" + assert entry["performance_gate"]["wins_needed"] == 1 + + +def test_a_gate_round_without_a_stage_document_is_still_read(tmp_path): + """The 2026-09-18 transitional build wrote the gate but no stage.json.""" + exp = tmp_path / "exp" + pool = _pool_file(tmp_path) + _child(exp, 1) + _round(exp, 1, None, gate=_gate_doc()) + + gate = _children_live(exp, 1, _answers(pool))["q0"]["gate"] + + assert gate["protocol"] == "variant" + assert gate["judged"]["verdict"] == "WIN" + + +# --------------------------------------------------------------- the view side + + +def _js() -> str: + return JS.read_text(encoding="utf-8") + + +def _component(name: str, end: str = "\nfunction ") -> str: + src = _js() + assert f"function {name}(" in src, f"{name} is gone" + return src.split(f"function {name}(", 1)[1].split(end, 1)[0] + + +def test_the_gate_column_no_longer_advertises_the_pool_criterion(): + src = _js() + assert "pass gate ≤ us" not in src, ( + "the column must not be titled like a gate it does not show") + assert "必须打败（variant）" in src, "the column must name the real opponent" + component = _component("QuestionGateCell") + assert "gate.variant_us" in component, ( + "the cell must render the variant, not a family baseline") + assert "gate.judged" in component, ( + "a judged round must show WIN/LOSS, not just a number") + + +def test_the_pool_number_is_only_an_explicitly_labelled_annotation(): + component = _component("QuestionGateCell") + assert "poolRef" in component, ( + "the pool criterion must arrive as its own argument") + assert "本协议不判门" in component, ( + "showing tau * baseline without saying it is not the gate is the bug") + # `≥N× baseline` may only survive inside the legacy branch. + if "× baseline" in component: + legacy = component.split('gate.protocol === "legacy"', 1)[1] + assert "baseline_us / poolRef.target_us" in legacy, ( + "the × baseline phrasing belongs to the legacy criterion only") + + +def test_the_view_keeps_the_legacy_path_readable(): + component = _component("QuestionGateCell") + assert 'gate.protocol === "legacy"' in component, ( + "legacy rounds were judged by the family criterion; say so") + src = _js() + assert "legacy 判据" in src + + +def test_the_round_shows_which_gate_it_owed_and_what_it_concluded(): + src = _js() + assert "function RoundGateLine(" in src + assert "<${RoundGateLine} iter=${selIterObj} />" in src, ( + "the round's gate line must be rendered with the round on screen") + line = _component("RoundGateLine") + for field in ("performance_gate", "generalization_gate", "wins_needed", + "below_floor"): + assert field in line, f"the gate line should show {field}" + + +def test_the_child_result_column_is_not_presented_as_the_gate(): + src = _js() + assert "子任务验收" in src, ( + "the r.passed column is DKAO's own acceptance, not the performance gate") + assert "DKAO 子任务自己的验收" in src + + +def test_the_four_card_target_is_the_rounds_real_bar(): + """The cards read the same `target_us`; it must be the variant on-protocol.""" + component = _component("FourCardOverview", "\nfunction PromotionPanel(") + assert "必须打败 ≤" in component + assert "target = g.target_us" in component + + +#: The repo ships no JS test runner (see tests/test_live_dkao_mirror.py), so the +#: module is only *parsed*, not executed: a syntax error in a file with no build +#: step would otherwise surface as a blank page and nothing else. +@pytest.mark.skipif(shutil.which("node") is None, reason="node not installed") +def test_the_detail_module_parses(tmp_path): + copy = tmp_path / "he-detail.mjs" + copy.write_text(_js(), encoding="utf-8") + proc = subprocess.run(["node", "--check", str(copy)], + capture_output=True, text=True) + assert proc.returncode == 0, proc.stderr + + +# ------------------------------------------------- the cell, actually executed + +#: Parsing is not rendering: reading ``g.baseline_us`` off the wrong object +#: renders "-" instead of raising, which is how the pool criterion came to be +#: shown as the gate in the first place. So the cell is also executed — with the +#: repo's own vendored htm/preact (no browser, no bundler, no network): the vnode +#: tree htm returns is plain data and can be walked for its text. +VENDOR = HE.parent / "sys_shell" / "static" / "vendor" + +_RENDER_DRIVER = """ +import { QuestionGateCell } from "./he-detail.mjs"; +function text(v, out = []) { + if (v == null || v === false || v === true) return out; + if (typeof v === "string" || typeof v === "number") { out.push(String(v)); return out; } + if (Array.isArray(v)) { for (const x of v) text(x, out); return out; } + if (typeof v === "object") { + const p = v.props || {}; + if (typeof v.type === "function") return text(v.type(p), out); + if (p.children != null) text(p.children, out); + } + return out; +} +const poolRef = { baseline_us: 1000.0, tau_family: 0.5, target_us: 500.0 }; +const base = { variant_source: "variant", kernel_source: "pool:best_known", + target_us: 200.0, variant_us: 200.0, win_ratio: 0.75, + floor_percent: 0.8, noise_percent: 2.0 }; +const cases = { + baseline: { ...base, protocol: "variant", stage: "baseline", judged: null }, + judged: { ...base, protocol: "variant", stage: "performance", + judged: { verdict: "WIN", candidate_median_us: 175.0, delta_percent: 12.5 } }, + tie: { ...base, protocol: "variant", stage: "performance", + judged: { verdict: "NO_SIGNAL", candidate_median_us: 200.0, delta_percent: 0.0 } }, + legacy: { ...base, protocol: "legacy", stage: null, judged: null, + target_us: 500.0 }, +}; +const out = {}; +for (const [k, gate] of Object.entries(cases)) { + out[k] = text(QuestionGateCell({ gate, poolRef })).join(" ").replace(/\\s+/g, " ").trim(); +} +out.none = text(QuestionGateCell({ gate: null, poolRef: null })).join(" ").trim(); +console.log(JSON.stringify(out)); +""" + + +def _imported_names(src: str) -> set: + """Every named import he-detail.js takes from the app/* modules. + + Derived from the source so the stub cannot rot: adding an import to the view + must not turn a render check into an unrelated failure. + """ + names = set() + for match in re.finditer(r'import\s*\{([^}]*)\}\s*from\s*"(app/[^"]+)"', src): + for part in match.group(1).split(","): + name = part.strip().split(" as ")[0].strip() + if name: + names.add(name) + assert names, "he-detail.js imports nothing from app/*? update this test" + return names + + +def _render_gate_cell(tmp_path: Path) -> dict: + """Render ``QuestionGateCell`` in node against the vendored htm/preact.""" + if not VENDOR.is_dir(): + pytest.skip("vendored htm/preact is missing") + work = tmp_path / "render" + work.mkdir() + # The vendor bundles import each other by bare specifier (the browser reads + # them through the shell importmap); a flat dir with relative specifiers is + # the node equivalent. + for name in ("htm.js", "htm-core.js", "preact.js", "preact-hooks.js"): + src = (VENDOR / name).read_text(encoding="utf-8") + src = src.replace('from"preact"', 'from"./preact.js"') + src = src.replace('from"htm"', 'from"./htm-core.js"') + (work / name).write_text(src, encoding="utf-8") + (work / "stub.js").write_text( + "const noop = async () => ({});\n" + + "\n".join(f"export const {n} = noop;" + for n in sorted(_imported_names(_js()))) + + "\nexport default function Stub() { return null; }\n", + encoding="utf-8") + src = _js() + for spec in ("app/he-api", "app/utils", "app/dkao-detail", + "htm/preact", "preact/hooks"): + assert f'from "{spec}"' in src, f"{spec} import moved; update this test" + src = src.replace('from "app/he-api"', 'from "./stub.js"') + src = src.replace('from "app/utils"', 'from "./stub.js"') + src = src.replace('from "app/dkao-detail"', 'from "./stub.js"') + src = src.replace('from "htm/preact"', 'from "./htm.js"') + src = src.replace('from "preact/hooks"', 'from "./preact-hooks.js"') + src += "\nexport { QuestionGateCell };\n" + (work / "he-detail.mjs").write_text(src, encoding="utf-8") + (work / "driver.mjs").write_text(_RENDER_DRIVER, encoding="utf-8") + proc = subprocess.run(["node", str(work / "driver.mjs")], + capture_output=True, text=True) + assert proc.returncode == 0, proc.stderr + return json.loads(proc.stdout) + + +@pytest.mark.skipif(shutil.which("node") is None, reason="node not installed") +def test_the_cell_renders_the_variant_and_labels_the_pool_number(tmp_path): + rendered = _render_gate_cell(tmp_path) + + # The opponent is the variant; the pool number is never bare. + assert "≤ 200.00 µs" in rendered["baseline"] + assert "variant" in rendered["baseline"] + assert "第 1 轮不判门" in rendered["baseline"] + assert "500.00 µs" in rendered["baseline"], ( + "the pool criterion should still be visible — as an annotation") + assert "本协议不判门" in rendered["baseline"] + + +@pytest.mark.skipif(shutil.which("node") is None, reason="node not installed") +def test_the_cell_renders_the_verdict_and_stays_empty_without_data(tmp_path): + rendered = _render_gate_cell(tmp_path) + + assert "赢" in rendered["judged"] and "12.50" in rendered["judged"] + assert "平·不算赢" in rendered["tie"], "a tie is not a win for this gate" + assert rendered["none"] == "-", "no gate data must not invent a number" + # The legacy path keeps the number that judged it, and says so. + assert "legacy 判据" in rendered["legacy"] + assert "≥2.00× baseline" in rendered["legacy"] diff --git a/metainfer/tasks/harness_evolve/tests/test_gpu_cards.py b/metainfer/tasks/harness_evolve/tests/test_gpu_cards.py new file mode 100644 index 00000000..94b766b2 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_gpu_cards.py @@ -0,0 +1,361 @@ +"""HE hands DKAO its questions and reads the gate verdict back, read-only. + +HE used to lease GPUs and police contention itself. That is gone: the device a +question runs on is a fixed layout decision, and the only gate verdict is the +one the child's own DKAO engine wrote into ``measurement_gate.jsonl``. These +tests pin both halves — the layout HE records, and the read-only audit the +detail page renders — plus the rule that a gate verdict is not retried away. +""" + +from __future__ import annotations + +import json + +from metainfer.tasks.harness_evolve.orchestrator.adapters.eval import ( + GATE_EXIT_CODE, DkaoCliEvaluator, +) +from metainfer.tasks.harness_evolve.orchestrator.rounds import ( + GPU_COUNT, device_for_index, +) +from metainfer.tasks.harness_evolve.server.routes import ( + _child_gate_audit, _gpu_cards, +) + + +def _gate_row(ts: float, event: str, *, attempt: int = 1, site: str = "benchmark", + gpu: int = 0, reasons=None, wait_seconds: float = 1800.0, + candidates=None, passing=None, usable=None): + row = { + "ts": ts, "event": event, "site": site, "gpu": gpu, "attempt": attempt, + "wait_seconds": wait_seconds, "max_waits": 48, + "reasons": list(reasons or []), + } + if candidates is not None: + row["candidates"] = list(candidates) + row["passing"] = list(passing or []) + row["usable"] = bool(passing) if usable is None else usable + return row + + +def _write_gate(child_state_dir, rows): + child_state_dir.mkdir(parents=True, exist_ok=True) + (child_state_dir / "measurement_gate.jsonl").write_text( + "".join(json.dumps(r) + "\n" for r in rows), encoding="utf-8") + + +# --------------------------------------------------------------- the layout + + +def test_the_layout_is_a_fixed_rotation_and_says_he_manages_nothing(): + """N questions land as N / 4 waves with one question per device.""" + assert GPU_COUNT == 4 + assert [device_for_index(i) for i in range(8)] == [0, 1, 2, 3, 0, 1, 2, 3] + + class _Inst: + def __init__(self, i): + self.id = i + + class _Cfg: + pass + + layout = DkaoCliEvaluator._device_layout( + _Cfg(), 2, [_Inst(f"q{i}") for i in range(4)]) + assert layout["schema"] == "he-gpu-layout/1" + assert layout["managed_by"] == "dcu_kernel_auto_opt" + # HE holds no lease at all — the field exists so a reader can see that + assert layout["leases"] == [] and layout["enabled"] is False + assert layout["devices"] == {"q0": 0, "q1": 1, "q2": 2, "q3": 3} + assert layout["assigned_devices"] == [0, 1, 2, 3] + assert "measurement gate" in layout["reason"] + + +def test_the_layout_is_recorded_next_to_the_iteration_evidence(tmp_path): + """Audit trail: same file the DKAO side reads, in the run's input dir.""" + (tmp_path / "exp").mkdir() + cfg = type("Cfg", (), {"exp_dir": tmp_path / "exp"})() + ev = DkaoCliEvaluator() + ev._save_preflight(cfg, 3, {"schema": "he-gpu-layout/1", "leases": []}) + audit = (tmp_path / "exp" / "runs" / "iteration_003" / "input" + / "benchmark" / "gpu_preflight.json") + assert audit.is_file() + body = json.loads(audit.read_text(encoding="utf-8")) + assert body["schema"] == "he-gpu-layout/1" and body["leases"] == [] + + +# ------------------------------------------------------- the gate audit read + + +def test_gate_audit_counts_checks_waits_and_the_last_verdict(tmp_path): + _write_gate(tmp_path / "state", [ + _gate_row(1.0, "gate_ok", gpu=0, candidates=[0, 1, 2, 3], passing=[0]), + _gate_row(2.0, "gate_blocked", reasons=["VRAM 96.0% > 90.0% limit"]), + _gate_row(2.0, "gate_blocked", reasons=["VRAM 96.0% > 90.0% limit"]), + _gate_row(3.0, "gate_blocked", attempt=2, site="worker", gpu=2, + reasons=["device busy: HCU 100.0% > 0.0%"]), + _gate_row(4.0, "gate_ok", gpu=2, candidates=[0, 2], passing=[2]), + ]) + audit = _child_gate_audit(tmp_path / "state") + assert audit["checks"] == 4 # the duplicated channel row folds + assert audit["blocked"] == 2 + assert audit["waited_seconds"] == 3600.0 # two 30-minute waits + assert audit["last_event"] == "gate_ok" + assert audit["last_site"] == "benchmark" + assert audit["max_waits"] == 48 + assert audit["last_passing"] == [2] and audit["last_candidates"] == [0, 2] + # the blocked rows are kept so the page can show *why* it waited + reasons = [r["reasons"] for r in audit["blocked_reasons"]] + assert ["VRAM 96.0% > 90.0% limit"] in reasons + assert ["device busy: HCU 100.0% > 0.0%"] in reasons + + +def test_a_passing_row_never_reports_its_neighbours_as_this_device_reason( + tmp_path): + """Real DKAO shape: a passing row lists the *other* cards it rejected. + + ``ensure_any_measurement_gate`` puts the failed candidates' reasons on a + ``gate_ok`` row while ``gpu`` is the card that passed. Reporting those + reasons next to this device would show a block that never happened, so the + audit must drop them for a passing verdict — and must still name the device + it cleared. + """ + _write_gate(tmp_path / "state", [ + _gate_row(10.0, "gate_ok", site="generate_probe", gpu=1, + candidates=[0, 1, 2, 3], passing=[1, 2, 3], usable=True, + reasons=["device busy: HCU 97.5% > 0.0%"]), + ]) + audit = _child_gate_audit(tmp_path / "state") + assert audit["last_event"] == "gate_ok" and audit["last_usable"] is True + assert audit["last_reasons"] == [] # not this device's block + assert audit["blocked"] == 0 and audit["waited_seconds"] == 0 + assert audit["device"] == 1 and audit["device_from"] == "gate_pass" + assert audit["last_passing"] == [1, 2, 3] + + +def test_a_blocked_row_keeps_its_reasons_and_the_device_it_blocked_on( + tmp_path): + _write_gate(tmp_path / "state", [ + _gate_row(10.0, "gate_blocked", site="benchmark", gpu=3, attempt=1, + candidates=[3], passing=[], usable=False, + reasons=["VRAM 94% > 90% limit", + "device busy: HCU 90.8% > 0.0%"]), + ]) + audit = _child_gate_audit(tmp_path / "state") + assert audit["last_event"] == "gate_blocked" + assert audit["last_usable"] is False + assert audit["last_reasons"] == ["VRAM 94% > 90% limit", + "device busy: HCU 90.8% > 0.0%"] + assert audit["device"] == 3 and audit["device_from"] == "gate_blocked" + assert audit["waited_seconds"] == 1800.0 + + +def test_gate_audit_of_a_child_that_never_gated_is_empty(tmp_path): + audit = _child_gate_audit(tmp_path / "missing") + assert audit["checks"] == 0 and audit["blocked"] == 0 + assert audit["last_event"] is None and audit["blocked_reasons"] == [] + assert audit["waited_seconds"] == 0 + assert audit["device"] is None and audit["device_from"] is None + assert audit["last_reasons"] == [] + + +def test_child_gate_audit_matches_the_dkao_route_verdict(tmp_path): + """HE must read the child's rows the same way DKAO's own page does.""" + from metainfer.tasks.dcu_kernel_auto_opt.server.routes import _gate_status + + state = tmp_path / "state" + _write_gate(state, [ + _gate_row(1.0, "gate_blocked", reasons=["device busy: HCU 100.0% > 0.0%"]), + _gate_row(2.0, "gate_ok", gpu=1, candidates=[1], passing=[1]), + ]) + dkao = _gate_status(state) + he = _child_gate_audit(state) + assert dkao["blocked"] == he["blocked"] == 1 + assert dkao["last_event"] == he["last_event"] == "gate_ok" + + +# ------------------------------------------------------------ the card view + + +def _seed_child(exp, num, qid, *, phase="parallel_explore", gate_rows=()): + state = exp / "children" / f"iteration_{num:03d}" / qid / "state" + state.mkdir(parents=True, exist_ok=True) + (state / "run.json").write_text(json.dumps({ + "task_id": qid, "task_type": "dcu-kernel-auto-opt", + "current_phase": phase, "current_iteration": 1, + }), encoding="utf-8") + if gate_rows: + _write_gate(state, gate_rows) + return state + + +def test_gpu_cards_group_questions_by_their_own_device(tmp_path): + """The card is the device the child's gate recorded, not the enumeration.""" + exp = tmp_path / "exp" + # child A was cleared for device 3, child B for device 0: the cards must + # follow the children's own records rather than alphabetical order + _seed_child(exp, 2, "m16_a", gate_rows=[ + _gate_row(1.0, "gate_blocked", gpu=3, + reasons=["device busy: HCU 100.0% > 0.0%"]), + _gate_row(2.0, "gate_ok", gpu=3, candidates=[3], passing=[3]), + ]) + _seed_child(exp, 2, "m16_b", gate_rows=[ + _gate_row(3.0, "gate_ok", gpu=0, candidates=[0], passing=[0]), + ]) + _seed_child(exp, 2, "m16_c") # no gate record: layout prediction + + cards = _gpu_cards(exp, 2) + by_gpu = {c["gpu"]: c for c in cards} + assert sorted(by_gpu) == [0, 2, 3] + assert [q["question"] for q in by_gpu[3]["questions"]] == ["m16_a"] + assert [q["question"] for q in by_gpu[0]["questions"]] == ["m16_b"] + # m16_c is the third child alphabetically -> index 2 -> predicted device 2 + assert [q["question"] for q in by_gpu[2]["questions"]] == ["m16_c"] + assert by_gpu[3]["questions"][0]["gate_audit"]["blocked"] == 1 + assert by_gpu[3]["questions"][0]["gpu_source"] == "gate_pass" + # nothing recorded this round's layout and the child has no requirements, + # so its device is only a prediction — and it must not look like an + # observation or like HE's own layout record + assert by_gpu[2]["questions"][0]["gpu_source"] == "predicted" + + +def test_a_child_without_gate_records_is_marked_as_a_prediction(tmp_path): + """An observed device and a predicted one must not look the same.""" + exp = tmp_path / "exp" + _seed_child(exp, 1, "no_gate_yet") + cards = _gpu_cards(exp, 1) + q = cards[0]["questions"][0] + assert q["gpu_source"] == "predicted" + assert q["gate_audit"]["checks"] == 0 + assert q["gate_audit"]["device"] is None + + +def test_the_round_layout_beats_a_probe_run_on_another_card(tmp_path): + """The real case: three children were listed under GPU2 because their + *generate probe* ran there, even though HE had given them GPU0/1/2/3. + + A probe may legitimately run on any passing device ("this task's own card + first, otherwise any idle one"), so it must never decide which card a + question is shown under — otherwise a four-card view shows two cards. + """ + exp = tmp_path / "exp" + (exp / "runs" / "iteration_002" / "input" / "benchmark").mkdir(parents=True) + (exp / "runs" / "iteration_002" / "input" / "benchmark" + / "gpu_preflight.json").write_text(json.dumps({ + "iteration": 2, "gpu_count": 4, "managed_by": "dcu_kernel_auto_opt", + "devices": {"m16_a": 0, "m16_b": 1}, + }), encoding="utf-8") + # both children's last *passing* row is the probe, on device 2 + _seed_child(exp, 2, "m16_a", gate_rows=[ + _gate_row(1.0, "gate_pass", gpu=2, site="generate_probe", + candidates=[0, 1, 2, 3], passing=[2]), + ]) + _seed_child(exp, 2, "m16_b", gate_rows=[ + _gate_row(2.0, "gate_pass", gpu=2, site="generate_probe", + candidates=[0, 1, 2, 3], passing=[2]), + ]) + + cards = _gpu_cards(exp, 2) + by_gpu = {c["gpu"]: c for c in cards} + + assert sorted(by_gpu) == [0, 1], "the layout decides the card" + assert [q["question"] for q in by_gpu[0]["questions"]] == ["m16_a"] + assert [q["question"] for q in by_gpu[1]["questions"]] == ["m16_b"] + assert by_gpu[0]["questions"][0]["gpu_source"] == "layout" + # ... and the probe's card is still reported, as a separate fact + assert by_gpu[0]["questions"][0]["gate_audit"]["probe_device"] == 2 + assert by_gpu[0]["questions"][0]["gate_audit"]["devices"] == [2] + + +def test_a_measuring_site_row_decides_over_a_probe_row(tmp_path): + """Without a layout record, the card that *measured* is the child's card.""" + exp = tmp_path / "exp" + _seed_child(exp, 1, "m16_a", gate_rows=[ + _gate_row(1.0, "gate_pass", gpu=2, site="generate_probe", + candidates=[0, 2], passing=[2]), + _gate_row(2.0, "gate_pass", gpu=0, site="benchmark", + candidates=[0], passing=[0]), + ]) + cards = _gpu_cards(exp, 1) + q = cards[0]["questions"][0] + assert cards[0]["gpu"] == 0 and q["gpu_source"] == "gate_pass" + assert q["gate_audit"]["probe_device"] == 2 + + +def test_gpu_cards_carry_the_child_phase_and_never_a_lease(tmp_path): + exp = tmp_path / "exp" + _seed_child(exp, 1, "only", phase="serial_validate") + cards = _gpu_cards(exp, 1) + assert len(cards) == 1 + q = cards[0]["questions"][0] + assert q["question"] == "only" + assert q["current_phase"] == "serial_validate" + assert q["gpu"] == 0 + assert "gate_audit" in q + # nothing in this payload is a lease HE took + assert "lease" not in q and "holder" not in q + + +def test_gpu_cards_are_empty_before_any_child_is_dispatched(tmp_path): + exp = tmp_path / "exp" + (exp / "children").mkdir(parents=True) + assert _gpu_cards(exp, 4) == [] + + +# -------------------------------------------------- the verdict is not ignored + + +def test_gate_exit_code_matches_the_dkao_asset_that_raises_it(): + """The duplicated constant must not drift from the child's own value.""" + import re + from pathlib import Path + + asset = (Path(__file__).resolve().parents[2] / "dcu_kernel_auto_opt" + / "assets" / "w8a8_bench.py") + match = re.search(r"^GATE_EXIT_CODE\s*=\s*(\d+)", asset.read_text( + encoding="utf-8"), re.MULTILINE) + assert match, "w8a8_bench.py no longer defines GATE_EXIT_CODE" + assert int(match.group(1)) == GATE_EXIT_CODE + + +# ------------------------------------------------------------- the Web route + + +def test_the_web_route_serves_the_cards_and_offers_no_lease_control( + tmp_path, client): + """The page's own endpoint: read-only, no lease, no release/drain verbs.""" + from metainfer.server import tasks as _tasks + from metainfer.server.tasks import TaskEntry + + state_dir = tmp_path / "state" + workspace = tmp_path / "ws" + state_dir.mkdir(parents=True) + workspace.mkdir(parents=True) + (state_dir / "requirements.json").write_text(json.dumps({ + "task_id": "he-gpu", "task_type": "harness-evolve", + "answers": {"ahe_repo_root": str(tmp_path / "repos")}, + }), encoding="utf-8") + (state_dir / "run.json").write_text(json.dumps({"task_id": "he-gpu"}), + encoding="utf-8") + _seed_child(workspace, 2, "m16_a", gate_rows=[ + _gate_row(1.0, "gate_blocked", gpu=3, + reasons=["device busy: HCU 100.0% > 0.0%"]), + ]) + _tasks.add_task(TaskEntry( + id="he-gpu", type="harness-evolve", label="device cards", + state_dir=str(state_dir), workspace_dir=str(workspace), + created_at=0.0)) + + resp = client.get("/api/harness-evolve/he-gpu/iterations/2/gpu") + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["managed_by"] == "dcu_kernel_auto_opt" + assert body["leases"] == [] # HE holds nothing + assert body["gpu_count"] == 4 + assert [c["gpu"] for c in body["cards"]] == [3] + q = body["cards"][0]["questions"][0] + assert q["question"] == "m16_a" + assert q["gate_audit"]["blocked"] == 1 + assert q["gate_audit"]["last_reasons"] == ["device busy: HCU 100.0% > 0.0%"] + # the operator controls are gone with the lease they controlled + for verb in ("release", "drain"): + assert client.post(f"/api/harness-evolve/he-gpu/gpu/{verb}").status_code == 404 + assert client.get("/api/harness-evolve/he-gpu/gpu").status_code == 404 diff --git a/metainfer/tasks/harness_evolve/tests/test_legacy_protocol_warning.py b/metainfer/tasks/harness_evolve/tests/test_legacy_protocol_warning.py new file mode 100644 index 00000000..adc835d7 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_legacy_protocol_warning.py @@ -0,0 +1,167 @@ +"""A run that leaves the fixed protocol has to say so. + +``FLOW.md`` describes the only protocol a new HE task may use: variant rounds, a +frozen generalization paper, the two gates, rounds sized in waves of 4. Two older +paths still exist — suite mode (no measured pool file) and an explicit +``round_questions: legacy`` — and both skip all of that. + +The operator's decision (2026-09-22) was **not** to delete them but to make +leaving the protocol impossible to miss: a WARNING in the orchestrator log, an +event on the task timeline, ``protocol.json`` for the task page, and a banner in +the detail view. That is what these tests pin. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import yaml + +from metainfer.server import tasks as _tasks +from metainfer.server.tasks import TaskEntry +from metainfer.tasks.harness_evolve.orchestrator import pipeline as P +from metainfer.tasks.harness_evolve.orchestrator.config import ( + load_experiment_config, +) +from metainfer.tasks.harness_evolve.orchestrator.pipeline import run_experiment + + +def _requirements(tmp_path: Path, answers: dict) -> Path: + tmp_path.mkdir(parents=True, exist_ok=True) + req = tmp_path / "requirements.json" + req.write_text(json.dumps({"task_id": "he-legacy", "answers": answers}, + ensure_ascii=False), encoding="utf-8") + return req + + +def _cfg(tmp_path: Path, answers: dict): + return load_experiment_config(_requirements(tmp_path, answers), + tmp_path / "state", tmp_path / "ws") + + +def _events(cfg) -> list: + path = Path(cfg.state_dir) / "timeline.jsonl" + if not path.is_file(): + return [] + return [json.loads(line) for line in + path.read_text(encoding="utf-8").splitlines() if line.strip()] + + +def _pool(tmp_path: Path, operators: int = 8) -> Path: + path = tmp_path / "pool.yaml" + path.write_text(yaml.safe_dump({ + "schema_version": 1, + "instances": [ + {"id": f"op{i:02d}", "family": f"decode__fam{i}", + "contract": {"M": 16}, "baseline_us": 100.0 + i, + "best_known_us": 40.0 + 4 * i, "history": []} + for i in range(operators) + ], + }, sort_keys=False), encoding="utf-8") + return path + + +# ------------------------------------------------------------------- the reason + +def test_the_reason_names_the_actual_cause(tmp_path): + explicit = _cfg(tmp_path / "a", {"execution_mode": "dry-run", + "round_questions": "legacy"}) + assert "round_questions" in P._legacy_protocol_reason(explicit) + + suite = _cfg(tmp_path / "b", {"execution_mode": "dry-run"}) + assert "no question pool" in P._legacy_protocol_reason(suite) + + builtin = _cfg(tmp_path / "c", {"execution_mode": "dry-run", + "pool_source": "builtin"}) + assert "demo pool" in P._legacy_protocol_reason(builtin) + + missing = _cfg(tmp_path / "d", {"execution_mode": "dry-run", + "pool_source": str(tmp_path / "nope.yaml")}) + assert "does not exist" in P._legacy_protocol_reason(missing) + + +# ------------------------------------------------- what the run leaves behind + +def test_a_suite_run_says_it_left_the_protocol(tmp_path, capsys): + cfg = _cfg(tmp_path, {"execution_mode": "dry-run", "max_iterations": 1}) + + run_experiment(cfg) + + announced = [row for row in _events(cfg) + if row["type"] == "legacy_protocol_selected"] + assert announced, "leaving the protocol must be on the record" + payload = announced[0]["payload"] + assert payload["mode"] == "legacy" + assert "no question pool" in payload["reason"] + assert "FLOW.md" in payload["policy"] + + # ... in a file the task page reads, so it can warn instead of looking idle + written = json.loads((cfg.exp_dir / "protocol.json").read_text()) + assert written["mode"] == "legacy" and written["reason"] == payload["reason"] + + # ... and in the log, where somebody watching the process will see it + assert "WARNING" in capsys.readouterr().err + + +def test_a_variant_run_says_nothing(tmp_path): + """The default protocol must not raise the banner (no false positives).""" + pool = _pool(tmp_path) + cfg = _cfg(tmp_path, {"execution_mode": "dry-run", "max_iterations": 1, + "pool_source": str(pool), "question_pool": str(pool), + "per_round_budget": "4"}) + + run_experiment(cfg) + + assert not [row for row in _events(cfg) + if row["type"] == "legacy_protocol_selected"] + assert not (cfg.exp_dir / "protocol.json").exists() + + +# --------------------------------------------------------------- on the page + +def _register(state_dir: Path, workspace_dir: Path, task_id: str): + state_dir.mkdir(parents=True, exist_ok=True) + workspace_dir.mkdir(parents=True, exist_ok=True) + _tasks.add_task(TaskEntry(id=task_id, type="harness-evolve", label="he", + state_dir=str(state_dir), + workspace_dir=str(workspace_dir), + created_at=0.0)) + + +def test_the_task_page_can_warn_about_a_legacy_run(client, isolated_env, + tmp_path): + home = isolated_env["home"] + state = home / "tasks" / "he-legacy-view" + ws = home / "workspaces" / "he-legacy-view" + _register(state, ws, "he-legacy-view") + (ws / "protocol.json").write_text(json.dumps({ + "mode": "legacy", "protocol": "legacy", + "reason": "no question pool is configured", + }), encoding="utf-8") + + body = client.get("/api/harness-evolve/he-legacy-view/summary").json() + + assert body["protocol"]["mode"] == "legacy" + assert "no question pool" in body["protocol"]["reason"] + + +def test_a_variant_task_reports_the_fixed_protocol(client, isolated_env): + home = isolated_env["home"] + state = home / "tasks" / "he-variant-view" + ws = home / "workspaces" / "he-variant-view" + _register(state, ws, "he-variant-view") + + body = client.get("/api/harness-evolve/he-variant-view/summary").json() + + assert body["protocol"]["mode"] == "variant" + assert "FLOW.md" in body["protocol"]["reason"] + + +def test_the_detail_view_renders_the_banner_only_for_legacy(): + """The page must read ``summary.protocol`` and stay quiet otherwise.""" + src = (Path(__file__).resolve().parents[1] / "static" + / "he-detail.js").read_text(encoding="utf-8") + assert "ProtocolBanner" in src + assert 'mode !== "legacy"' in src, "the banner is legacy-only" + assert "summary && summary.protocol" in src, "it reads the summary payload" diff --git a/metainfer/tasks/harness_evolve/tests/test_live_dkao_mirror.py b/metainfer/tasks/harness_evolve/tests/test_live_dkao_mirror.py new file mode 100644 index 00000000..7b2d3422 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_live_dkao_mirror.py @@ -0,0 +1,125 @@ +"""The AHE page embeds the DKAO task view inside each iteration. + +The two halves of that feature live in different plugins, so nothing about it +is checked by either plugin's own tests: + + * the AHE detail module imports ``app/dkao-detail`` (the *other* plugin's + bundle), which only works if the importmap carries that key -- the JS + itself cannot be executed here (no build step, no browser), so the wiring + is asserted where it is decidable: the module the server actually serves, + the importmap it is served with, and the bundle's own embed interface; + * the embedded view is addressed by the child's artifact directory, because + HE children are spawned by the orchestrator and are not tasks. + +Deliberately narrow: this does not try to be a JS test suite. +""" + +from __future__ import annotations + +import json +import re + +HE_DETAIL = ("metainfer/tasks/harness_evolve/static/he-detail.js") +DKAO_DETAIL = "metainfer/tasks/dcu_kernel_auto_opt/static/dkao-detail.js" + + +def _source(rel: str) -> str: + from pathlib import Path + return (Path(__file__).resolve().parents[4] / rel).read_text(encoding="utf-8") + + +def _importmap(client) -> dict: + resp = client.get("/") + assert resp.status_code == 200 + block = re.search( + r']*type="importmap"[^>]*>(.*?)', resp.text, + re.DOTALL) + assert block, "the shell no longer ships an importmap" + return json.loads(block.group(1)).get("imports", {}) + + +def test_the_ahe_page_resolves_the_dkao_view_it_imports(client): + """`app/dkao-detail` must exist, or the AHE page dies on import.""" + imports = _importmap(client) + assert "app/dkao-detail" in imports, sorted(imports) + url = imports["app/dkao-detail"] + assert url.startswith("/static/plugins/dcu-kernel-auto-opt/") + assert "CACHE_BUST" not in url # substituted, not leaked + asset = client.get(url.split("?")[0]) + assert asset.status_code == 200, url + assert "DcuKernelAutoOptDetail" in asset.text + + +def test_he_detail_actually_imports_and_mounts_the_dkao_view(): + src = _source(HE_DETAIL) + assert re.search( + r'import\s+DcuKernelAutoOptDetail\s+from\s+"app/dkao-detail"', src), \ + "he-detail.js no longer imports the DKAO view" + mount = re.search(r"<\$\{DcuKernelAutoOptDetail\}([^/]*)/>", src) + assert mount, "he-detail.js never renders the DKAO view" + # A mirrored child is read-only and located by its artifact directory; + # without these the embedded calls would 404 or offer write controls. + assert "stateDir=" in mount.group(1) + assert "readOnly=${true}" in mount.group(1) + + +def test_the_dkao_view_supports_the_embed_contract(): + """The other side of the same contract, in the DKAO bundle.""" + src = _source(DKAO_DETAIL) + assert "export function apiBase(" in src, \ + "dkao-detail.js no longer exports the embed URL builder" + assert "state_dir=" in src, "apiBase no longer passes state_dir" + assert "readOnly" in src, "dkao-detail.js dropped its read-only mode" + # The write controls must be gated on it, not merely styled. + for control in ("onClick=${() => addVariant(shape)}", + "onClick=${renameRepo}", + "onClick=${syncNow}"): + idx = src.index(control) + window = src[max(0, idx - 900):idx] + assert "readOnly" in window, f"{control} is not gated by readOnly" + + +def test_the_iteration_block_hands_the_task_down_to_the_mirror(): + """Without taskId the mirror cannot fetch the iteration's children.""" + src = _source(HE_DETAIL) + assert re.search(r"<\$\{IterBlock\}\s+iter=\$\{it\}\s+taskId=\$\{taskId\}", src), \ + "IterBlock is rendered without taskId" + assert re.search(r"<\$\{LiveDkaoMirror\}\s+taskId=\$\{taskId\}", src), \ + "IterBlock does not mount the live DKAO mirror" + + +def test_the_mirror_follows_a_running_question_by_default(): + src = _source(HE_DETAIL) + body = src[src.index("function LiveDkaoMirror("):] + body = body[:body.index("\nfunction IterBlock(")] + assert "!children[id].finished" in body, \ + "the switcher no longer auto-selects the running question" + assert "getChildren(taskId, num)" in body, \ + "the mirror no longer reads the iteration's children" + + +# ------------------------------------------------------------------ stop button + + +def test_the_stop_button_is_wired_to_a_running_task(): + """Stop must be reachable, and only while there is something to stop.""" + src = _source(HE_DETAIL) + # `status` is the shell's liveness view; without it in the props the + # button silently never renders (it is passed, but must be destructured). + assert re.search(r"export default function HeDetailView\(\{[^}]*\bstatus\b", + src), "HeDetailView does not accept the shell's status prop" + assert re.search(r"import\s*\{[^}]*stopExperiment[^}]*\}\s*from\s*\"app/he-api\"", + src), "he-detail.js does not import stopExperiment" + assert "stopExperiment(taskId)" in src, "the button calls nothing" + # Offered exactly when the orchestrator is alive, not when it merely has + # artifacts on disk (a stopped run keeps its current_phase). + assert "status && status.running" in src, \ + "the Stop button is not gated on the orchestrator being alive" + + +def test_the_stop_api_posts_to_the_stop_route(): + src = _source("metainfer/tasks/harness_evolve/static/he-api.js") + body = src[src.index("export async function stopExperiment"):] + body = body[:body.index("\n}")] + assert "${TASK_BASE(taskId)}/stop" in body + assert 'method: "POST"' in body diff --git a/metainfer/tasks/harness_evolve/tests/test_live_mirror_layouts.py b/metainfer/tasks/harness_evolve/tests/test_live_mirror_layouts.py new file mode 100644 index 00000000..b858e08a --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_live_mirror_layouts.py @@ -0,0 +1,64 @@ +"""The DKAO live view shows the round's four questions at once. + +A round is four questions on four cards (FLOW.md §1.1), so comparing them is the +point of the live view. It used to render exactly one: four tabs, one DKAO page, +three quarters of the round hidden behind a click. + +These are source-level checks, the same way the rest of this plugin's front-end +contracts are pinned (the repo ships no JS test runner): the component must +render every child by default, keep the single view reachable per child, and the +grid must collapse on a narrow screen instead of squeezing four pages. +""" + +from __future__ import annotations + +from pathlib import Path + +HE = Path(__file__).resolve().parents[1] +JS = HE / "static" / "he-detail.js" +CSS = HE / "static" / "he.css" + + +def _component() -> str: + src = JS.read_text(encoding="utf-8") + return src.split("function LiveDkaoMirror(", 1)[1].split( + "function IterBlock(", 1)[0] + + +def test_every_child_is_rendered_by_default(): + component = _component() + assert 'useState("grid")' in component, "four at once is the default" + assert "ids.map((id) =>" in component, "every question gets a cell" + assert "he-live-cell" in component + assert "mirror(id, c)" in component, ( + "each cell renders that question's own DKAO page") + assert "taskId=${id} stateDir=${c.child_state_dir}" in component, ( + "…pointed at its own state dir, so the four are independent") + + +def test_the_three_layouts_are_offered(): + component = _component() + for key in ('"grid"', '"stack"', '"single"'): + assert key in component, f"missing layout {key}" + assert "setLayout(key)" in component + + +def test_the_single_view_is_still_one_click_away_per_child(): + component = _component() + assert 'setLayout("single")' in component, ( + "drilling into one question must stay possible from the grid") + + +def test_a_question_without_a_state_dir_is_skipped_not_rendered_empty(): + component = _component() + assert "if (!c.child_state_dir) return null;" in component + + +def test_the_grid_is_two_columns_and_collapses_on_narrow_screens(): + css = CSS.read_text(encoding="utf-8") + block = css.split(".he-live-grid {", 1)[1].split("}", 1)[0] + assert "repeat(2" in block + tail = css.split(".he-live-grid {", 1)[1][:600] + assert "@media" in tail, "narrow screens must fall back to one column" + assert ".he-live-cell-body" in css and "overflow-x: auto" in css, ( + "a 50%-wide cell must scroll its tables, not clip them") diff --git a/metainfer/tasks/harness_evolve/tests/test_liveness.py b/metainfer/tasks/harness_evolve/tests/test_liveness.py new file mode 100644 index 00000000..cff96bd9 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_liveness.py @@ -0,0 +1,277 @@ +"""A live HE run must look live in the WebUI (the green dot). + +The shell owns the "running" indicator and derives it from one thing: +``Launcher.status()``, which validates the pid recorded in the task's +``orchestrator.pid``. harness_evolve used to write no such file, so a run in +progress was reported as stopped for its whole life. These tests pin the +contract to the shell's own reader rather than to a private helper, so a +regression makes the green dot disappear again instead of silently passing. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from metainfer.server import launcher as launcher_mod +from metainfer.server import paths as paths_mod +from metainfer.server import proc as proc_mod +from metainfer.tasks.harness_evolve.orchestrator import cli as he_cli +from metainfer.testing import isolated_env # noqa: F401 - used as a fixture + + +@pytest.fixture +def shell(monkeypatch): + """The shell's *real* status reader (the fixture swaps in a fake one). + + The whole point of these tests is that a live HE run is visible to the + code the WebUI actually runs, so a stand-in for ``LocalLauncher`` would + test the stand-in. ``METAINFER_ROOT`` isolation from ``isolated_env`` is + still used, which is why the fixture is requested above. + + One measurement is neutralised: ``LocalLauncher.status`` validates a pid by + comparing the kernel's process start time -- derived from ``/proc`` ticks + plus the *recorded* boot time -- against the wall-clock ``started_at`` in + the PID file, with a 2 s tolerance. Both are the same clock in production. + This sandbox's recorded boot time is off from its wall clock by a couple of + seconds *and the offset moves* (measured: 0.23 s, then 2.4 s three seconds + later), so a perfectly healthy process lands past the tolerance and the + shell reports ``pid-dead`` -- a property of this host, not of harness_evolve. + + The fixture re-derives that offset on every call, immediately before the + comparison, and subtracts it. The real code path, the real tolerance and + the real wall-clock ``started_at`` from the PID file are all still used. + """ + real_start_time = proc_mod.pid_start_time + def _start_time(pid): + value = real_start_time(pid) + if value is None: + return None + own = real_start_time(os.getpid()) + if own is None: # /proc unavailable: leave the value alone + return value + # Re-measured here, in the same instant as the comparison, because this + # host's offset drifts. + return float(value) + (time.time() - float(own)) + + monkeypatch.setattr(proc_mod, "pid_start_time", _start_time) + + def _status(task_id: str): + return launcher_mod.LocalLauncher().status(task_id) + return _status + + +def _requirements(tmp_path: Path, task_id: str, *, rounds: int = 1) -> Path: + req = tmp_path / "requirements.json" + req.write_text(json.dumps({ + "task_id": task_id, + "task_type": "harness-evolve", + "execution_mode": "dry-run", + "max_iterations": rounds, + "evolve_mode": "dry-run", + "per_round_budget": 2, + "pool_source": "", + }), encoding="utf-8") + return req + + +def _pid_file(task_id: str) -> Path: + """Where the shell looks for it (``server.paths.task_dir/...``).""" + return paths_mod.task_dir(task_id) / "orchestrator.pid" + + +def test_run_stamps_the_pid_file_the_shell_reads( + isolated_env, shell, tmp_path, monkeypatch): + """The server's own status reader must see a live run while it runs.""" + task_id = "he-live" + state_dir = _pid_file(task_id).parent + cfg = he_cli.load_experiment_config( + _requirements(tmp_path, task_id), state_dir, tmp_path / "ws") + + seen: dict = {} + + def fake_run(cfg_, *args, **kwargs): + # Exactly what the shell does while the run is in progress. + status = shell(task_id).to_dict() + seen.update(status) + seen["payload"] = json.loads( + (state_dir / "orchestrator.pid").read_text(encoding="utf-8")) + return tmp_path / "ws" / "report.md" + + monkeypatch.setattr(he_cli, "load_experiment_config", lambda *a, **k: cfg) + monkeypatch.setattr(he_cli, "run_experiment", fake_run) + + assert he_cli.run_with_requirements( + _requirements(tmp_path, task_id), state_dir=state_dir, + workspace_dir=tmp_path / "ws") == 0 + + # live: the shell must have seen a running task with a validated pid + assert seen["running"] is True, ("SHELLVIEW=%r" % (seen,)) + assert seen["pid"] == os.getpid() + assert seen["exit_hint"] == "pid-alive" + assert seen["payload"]["task_id"] == task_id # not the directory name + assert isinstance(seen["payload"]["started_at"], float) + + +def test_the_pid_file_is_cleared_when_the_run_ends( + isolated_env, shell, tmp_path, monkeypatch): + """A finished run must not claim a pid, and must say when it finished.""" + task_id = "he-finished" + state_dir = _pid_file(task_id).parent + cfg = he_cli.load_experiment_config( + _requirements(tmp_path, task_id), state_dir, tmp_path / "ws") + monkeypatch.setattr(he_cli, "load_experiment_config", lambda *a, **k: cfg) + monkeypatch.setattr(he_cli, "run_experiment", + lambda cfg_, *a, **k: tmp_path / "report.md") + + he_cli.run_with_requirements(_requirements(tmp_path, task_id), + state_dir=state_dir, + workspace_dir=tmp_path / "ws") + + status = shell(task_id).to_dict() + assert status["running"] is False + assert status["pid"] is None + assert status["exit_hint"] == "pid-file-cleared" + assert isinstance(status["finished_at"], float) + + +def test_a_crashing_run_still_clears_its_pid_file( + isolated_env, shell, tmp_path, monkeypatch): + """An exception must not leave a stale "running" claim behind.""" + task_id = "he-crash" + state_dir = _pid_file(task_id).parent + cfg = he_cli.load_experiment_config( + _requirements(tmp_path, task_id), state_dir, tmp_path / "ws") + + def boom(cfg_, *a, **k): + raise RuntimeError("pipeline exploded") + + monkeypatch.setattr(he_cli, "load_experiment_config", lambda *a, **k: cfg) + monkeypatch.setattr(he_cli, "run_experiment", boom) + + try: + he_cli.run_with_requirements(_requirements(tmp_path, task_id), + state_dir=state_dir, + workspace_dir=tmp_path / "ws") + except RuntimeError: + pass + status = shell(task_id).to_dict() + assert status["running"] is False and status["finished_at"] is not None + + +def test_sigterm_clears_the_pid_file_before_exiting( + isolated_env, tmp_path, monkeypatch): + """The WebUI's Stop button sends SIGTERM: it must mark the task stopped.""" + task_id = "he-sigterm" + state_dir = _pid_file(task_id).parent + pid_file = state_dir / "orchestrator.pid" + state_dir.mkdir(parents=True, exist_ok=True) + + # The only thing a handler can safely do here is the real one: the test + # process would die on os._exit, so drive the handler body directly. + exited: list = [] + monkeypatch.setattr(he_cli.os, "_exit", lambda code: exited.append(code)) + captured: dict = {} + monkeypatch.setattr(he_cli.signal, "signal", + lambda sig, fn: captured.__setitem__(sig, fn)) + + with he_cli._orchestrator_pid(state_dir, task_id): + assert pid_file.is_file() + handler = captured[he_cli.signal.SIGTERM] + handler(he_cli.signal.SIGTERM, None) + + assert exited == [143] + assert json.loads(pid_file.read_text(encoding="utf-8"))["pid"] is None + assert json.loads(pid_file.read_text(encoding="utf-8"))["finished_at"] + + +def test_resume_also_marks_the_task_live( + isolated_env, shell, tmp_path, monkeypatch): + """A supervisor-started resume is just as live as a fresh run.""" + task_id = "he-resume" + state_dir = _pid_file(task_id).parent + state_dir.mkdir(parents=True, exist_ok=True) + req = state_dir / "requirements.json" + req.write_text(json.dumps({ + "task_id": task_id, "task_type": "harness-evolve", + "execution_mode": "dry-run", "max_iterations": 1, + "evolve_mode": "dry-run", "pool_source": "", + }), encoding="utf-8") + cfg = he_cli.load_experiment_config(req, state_dir, tmp_path / "ws") + + seen: dict = {} + + def fake_run(cfg_, *a, **k): + seen.update(shell(task_id).to_dict()) + return tmp_path / "report.md" + + monkeypatch.setattr(he_cli, "load_experiment_config", lambda *a, **k: cfg) + monkeypatch.setattr(he_cli, "completed_iterations", lambda exp: 1) + monkeypatch.setattr(he_cli, "run_experiment", fake_run) + + args = type("Args", (), { + "state_dir": str(state_dir), "workspace_dir": str(tmp_path / "ws"), + "requirements": None, "iterations": 1, "target": None, + "champion_iteration": None, + })() + assert he_cli.resume_command(args) == 0 + assert seen["running"] is True and seen["exit_hint"] == "pid-alive", ("SHELLVIEW=%r" % (seen,)) + assert shell(task_id).to_dict()["running"] is False + + +def test_the_real_cli_binary_writes_and_clears_the_pid_file( + isolated_env, shell, tmp_path): + """End to end, with no mocks: the actual process the server spawns.""" + task_id = "he-real-cli" + state_dir = _pid_file(task_id).parent + ws = tmp_path / "ws" + req = _requirements(tmp_path, task_id) + + # The server spawns the orchestrator with the package importable (its + # launcher prepends ``sys.path`` to PYTHONPATH); do the same so this is a + # faithful copy of that spawn rather than a plain pytest environment. + repo_root = Path(__file__).resolve().parents[4] # /metainfer -> + env = dict(os.environ) + env["PYTHONPATH"] = os.pathsep.join( + [str(repo_root)] + [p for p in env.get("PYTHONPATH", "").split(os.pathsep) + if p]) + proc = subprocess.Popen( + [sys.executable, "-m", + "metainfer.tasks.harness_evolve.orchestrator.cli", "run", + str(req), "--state-dir", str(state_dir), "--workspace-dir", str(ws)], + cwd=str(repo_root), env=env, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) + pid_file = state_dir / "orchestrator.pid" + try: + deadline = time.time() + 60 + while not pid_file.is_file() and time.time() < deadline: + if proc.poll() is not None: + break + time.sleep(0.01) + # The process identity the shell needs is in the file the real binary + # wrote: the orchestrator's own pid and the wall-clock start it stamped. + # (Whether the *shell* then calls it live is asserted in the tests + # above, where the host's clock offset can be cancelled out; doing that + # here would mean fighting a sandbox clock that jumps by seconds.) + if pid_file.is_file(): + payload = json.loads(pid_file.read_text(encoding="utf-8")) + assert payload["pid"] == proc.pid + assert payload["task_id"] == task_id + assert float(payload["started_at"]) <= time.time() + 60 + finally: + out, _ = proc.communicate(timeout=120) + + assert proc.returncode == 0, f"rc={proc.returncode} out={out!r}" + + # Once the process is gone the verdict is deterministic -- no clock needed. + status = shell(task_id).to_dict() + assert status["running"] is False + assert status["pid"] is None and status["exit_hint"] == "pid-file-cleared" + cleared = json.loads(pid_file.read_text(encoding="utf-8")) + assert cleared["pid"] is None and isinstance(cleared["finished_at"], float) diff --git a/metainfer/tasks/harness_evolve/tests/test_mechanism_gate.py b/metainfer/tasks/harness_evolve/tests/test_mechanism_gate.py new file mode 100644 index 00000000..9dc136b6 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_mechanism_gate.py @@ -0,0 +1,293 @@ +"""Mechanism gate: evidence grading + lenient/strict decision behaviour.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from metainfer.tasks.harness_evolve.orchestrator.decision_engine import ( + DecisionPolicy, decide, +) +from metainfer.tasks.harness_evolve.orchestrator.mechanism_checks import ( + collect_candidate_evidence, evaluate_changes, +) + +WIN = {"a": {"passed": True, "median_us": 100.0, "correctness_ok": True, + "new_best_known": True}, + "b": {"passed": True, "median_us": 100.0, "correctness_ok": True, + "new_best_known": True}} +CAND = {"a": {"passed": True, "median_us": 50.0, "correctness_ok": True}, + "b": {"passed": True, "median_us": 50.0, "correctness_ok": True}} + + +def _manifest(check=None, change_id="chg-1"): + change = {"id": change_id, "component": "planner_policy", + "mechanism_signature": ["fallback.m16 contains epilogue_fusion"]} + if check is not None: + change["mechanism_check"] = check + return {"schema_version": 1, "iteration": 1, "changes": [change], + "verification": {"status": "pending"}} + + +# ---------------------------------------------------------------- grader ---- + +def test_plan_ids_hit_partial_and_contradiction(): + check = {"kind": "planner_plan_ids", + "expect_any": ["memory_layout", "epilogue_fusion"]} + hit = {"planner_plan_ids": ["architecture_explore", "memory_layout"], + "planner_enabled": True} + part = {"planner_plan_ids": ["architecture_explore", "grid_splitk"], + "planner_enabled": True} + contra = {"planner_plan_ids": [], "planner_enabled": False} + unob = {"planner_plan_ids": [], "planner_enabled": True} + + assert evaluate_changes(_manifest(check), hit)[0]["chg-1"] == "hit" + assert evaluate_changes(_manifest(check), part)[0]["chg-1"] == "partial" + assert evaluate_changes(_manifest(check), contra)[0]["chg-1"] == "contradicted" + assert evaluate_changes(_manifest(check), unob)[0]["chg-1"] == "unobserved" + + +def test_expected_plan_order_hit(): + check = {"kind": "planner_plan_ids", + "expect_order": ["architecture_explore", "memory_layout"]} + ev = {"planner_plan_ids": ["architecture_explore", "grid_splitk", + "memory_layout"], + "planner_enabled": True} + assert evaluate_changes(_manifest(check), ev)[0]["chg-1"] == "hit" + + +def test_harness_revision_grading(): + check = {"kind": "harness_revision"} + ok = {"candidate_harness_revision": "rev2", + "observed_harness_revisions": ["rev2"]} + wrong = {"candidate_harness_revision": "rev2", + "observed_harness_revisions": ["rev1"]} + none = {"candidate_harness_revision": "rev2", + "observed_harness_revisions": []} + assert evaluate_changes(_manifest(check), ok)[0]["chg-1"] == "hit" + assert evaluate_changes(_manifest(check), wrong)[0]["chg-1"] == "contradicted" + assert evaluate_changes(_manifest(check), none)[0]["chg-1"] == "unobserved" + + +def test_agent_only_plan_ids_are_soft_evidence(): + """Without a planner record, matching ids are only partial evidence.""" + check = {"kind": "planner_plan_ids", "expect_any": ["memory_layout"]} + ev = {"agent_plan_ids": ["memory_layout"], "planner_enabled": True} + checks, detail = evaluate_changes(_manifest(check), ev) + assert checks["chg-1"] == "partial" + assert "agent" in detail["changes"][0]["reason"] + + +def test_missing_check_stays_unobserved(): + checks, detail = evaluate_changes(_manifest(None), {"plan_ids": []}) + assert checks["chg-1"] == "unobserved" + assert "mechanism_check" in detail["changes"][0]["reason"] + + +def test_evidence_collection_reads_children(tmp_path): + exp = tmp_path / "exp" + ws = exp / "children" / "iteration_001" / "shapeA" / "workspace" + runs = ws / "workers" / "worker_0" / "runs" / "shapeA" + runs.mkdir(parents=True) + (runs / "experiments.jsonl").write_text( + json.dumps({"iteration": 1, "plan_id": "memory_layout"}) + "\n" + + json.dumps({"iteration": 2, "plan_id": "epilogue_fusion"}) + "\n", + encoding="utf-8") + (ws / "main").mkdir(parents=True) + (ws / "main" / "scaffold_manifest.json").write_text( + json.dumps({"harness": {"revision": "rev-cand"}}), encoding="utf-8") + tl = exp / "children" / "iteration_001" / "shapeA" / "state" + tl.mkdir(parents=True) + (tl / "timeline.jsonl").write_text( + json.dumps({"ts": 1.0, "type": "gate_blocked", "payload": {}}) + "\n", + encoding="utf-8") + snap = exp / "runs" / "iteration_001" / "input" / "workspace" + snap.mkdir(parents=True) + (snap / "manifest.yaml").write_text("revision: rev-cand\n", encoding="utf-8") + + (ws / "workers" / "worker_0" / "planner_plans.jsonl").write_text( + json.dumps({"iteration": 1, "plan_id": "memory_layout", + "source": "planner"}) + "\n", encoding="utf-8") + + ev = collect_candidate_evidence(exp, 1, planner_enabled=True) + assert ev["agent_plan_ids"] == ["memory_layout", "epilogue_fusion"] + assert ev["planner_plan_ids"] == ["memory_layout"] + assert ev["plan_id_source"] == "planner" + assert ev["observed_harness_revisions"] == ["rev-cand"] + assert ev["candidate_harness_revision"] == "rev-cand" + assert "gate_blocked" in ev["gate_events"] + + +# --------------------------------------------------------------- decision ---- + +def test_lenient_promotes_unobserved_win(): + d = decide(WIN, CAND, manifest=_manifest({"kind": "planner_plan_ids", + "expect_any": ["memory_layout"]}), + mechanism_checks={"chg-1": "unobserved"}, + policy=DecisionPolicy(mechanism_policy="lenient")) + assert d["verdict"] == "PROMOTE_UNEXPLAINED" + assert d["action"] == "SET_CHAMPION" + assert d["mechanism_gate"]["status"] == "UNOBSERVED" + + +def test_lenient_still_confirms_on_contradiction(): + d = decide(WIN, CAND, manifest=_manifest({"kind": "planner_plan_ids", + "expect_any": ["memory_layout"]}), + mechanism_checks={"chg-1": "contradicted"}, + policy=DecisionPolicy(mechanism_policy="lenient")) + assert d["verdict"] == "CONFIRM_REQUIRED" + assert d["mechanism_gate"]["contradicted"] == ["chg-1"] + + +def test_contradiction_promotes_after_reproduction(): + d = decide(WIN, CAND, manifest=_manifest({"kind": "harness_revision"}), + mechanism_checks={"chg-1": "contradicted"}, + confirmation_reproduced=True, + policy=DecisionPolicy(mechanism_policy="lenient")) + assert d["verdict"] == "PROMOTE_UNEXPLAINED" + + +def test_hit_promotes_and_strict_policy_confirms_unobserved(): + hit = decide(WIN, CAND, manifest=_manifest({"kind": "harness_revision"}), + mechanism_checks={"chg-1": "hit"}, + policy=DecisionPolicy(mechanism_policy="lenient")) + assert hit["verdict"] == "PROMOTE" + strict = decide(WIN, CAND, + manifest=_manifest({"kind": "harness_revision"}), + mechanism_checks={"chg-1": "unobserved"}, + policy=DecisionPolicy(mechanism_policy="strict")) + assert strict["verdict"] == "CONFIRM_REQUIRED" + + +def test_off_policy_ignores_mechanism(): + d = decide(WIN, CAND, manifest=_manifest({"kind": "harness_revision"}), + mechanism_checks={"chg-1": "contradicted"}, + policy=DecisionPolicy(mechanism_policy="off")) + assert d["verdict"] == "PROMOTE_UNEXPLAINED" + + +# ------------------------------------------------------- generalization ---- + +def test_probe_regression_blocks_promotion(): + """A win in the tuned families must not promote if an untouched one broke.""" + champ = {"a": {"passed": True, "median_us": 100.0, "correctness_ok": True}, + "b": {"passed": True, "median_us": 100.0, "correctness_ok": True}, + "p": {"passed": True, "median_us": 100.0, "correctness_ok": True}} + cand = {"a": {"passed": True, "median_us": 50.0, "correctness_ok": True}, + "b": {"passed": True, "median_us": 50.0, "correctness_ok": True}, + "p": {"passed": True, "median_us": 300.0, "correctness_ok": True}} + purposes = {"a": "regression", "b": "repair", "p": "probe"} + d = decide(champ, cand, purposes=purposes) + assert d["generalization_gate"]["status"] == "FAIL" + assert d["verdict"] == "CONFIRM_REQUIRED" + assert "over-fitting" in d["reason"] + # the probe is not part of the paired decision set + assert "p" not in d["paired_ids"] and d["probe_ids"] == ["p"] + # reproduces -> allowed through, still recorded + d2 = decide(champ, cand, purposes=purposes, confirmation_reproduced=True) + assert d2["verdict"] == "PROMOTE_UNEXPLAINED" + assert d2["generalization_gate"]["status"] == "FAIL" + + +def test_probe_pass_allows_normal_promotion(): + champ = {"a": {"passed": True, "median_us": 100.0, "correctness_ok": True}, + "b": {"passed": True, "median_us": 100.0, "correctness_ok": True}, + "p": {"passed": True, "median_us": 100.0, "correctness_ok": True}} + cand = {"a": {"passed": True, "median_us": 50.0, "correctness_ok": True}, + "b": {"passed": True, "median_us": 50.0, "correctness_ok": True}, + "p": {"passed": True, "median_us": 98.0, "correctness_ok": True}} + d = decide(champ, cand, + purposes={"a": "regression", "b": "repair", "p": "probe"}) + assert d["generalization_gate"]["status"] == "PASS" + assert d["verdict"] == "PROMOTE_UNEXPLAINED" # lenient default + assert d["purpose_breakdown"]["p"]["counted"] is False + + +# ------------------------------------------------------- gate_values ------ + +def _gate_change(paths=("round_acceptance_improvement_percent",)): + return _manifest({"kind": "gate_values", "expect_any": list(paths)}) + + +def test_gate_values_hit_when_children_ran_with_declared_values(): + ev = { + "candidate_gates": {"wired": True, + "gates": {"round_acceptance_improvement_percent": 2.5}}, + "observed_gates": [ + {"gates": {"round_acceptance_improvement_percent": 2.5}}, + ], + } + checks, detail = evaluate_changes(_gate_change(), ev) + assert checks["chg-1"] == "hit" + assert "declared values" in detail["changes"][0]["reason"] + + +def test_gate_values_contradicted_when_values_differ(): + ev = { + "candidate_gates": {"wired": True, + "gates": {"round_acceptance_improvement_percent": 2.5}}, + "observed_gates": [ + {"gates": {"round_acceptance_improvement_percent": 1.0}}, + ], + } + checks, _ = evaluate_changes(_gate_change(), ev) + assert checks["chg-1"] == "contradicted" + + +def test_gate_values_contradicted_when_component_is_unwired(): + ev = {"candidate_gates": {"wired": False, "gates": {}}, + "observed_gates": [{"gates": {}}]} + checks, detail = evaluate_changes(_gate_change(), ev) + assert checks["chg-1"] == "contradicted" + assert "not wired" in detail["changes"][0]["reason"] + + +def test_gate_values_unobserved_without_snapshot(): + ev = {"candidate_gates": {"wired": True, "gates": {}}, "observed_gates": []} + checks, _ = evaluate_changes(_gate_change(), ev) + assert checks["chg-1"] == "unobserved" + + +# ------------------------------------------------- invalid / inconclusive ---- + +def test_missing_measurement_is_invalid_not_hard_fail(): + """A crashed child must not look like a performance regression.""" + champ = {"a": {"passed": True, "median_us": 100.0, "correctness_ok": True}, + "b": {"passed": True, "median_us": 100.0, "correctness_ok": True}} + cand = {"a": {"passed": False, "median_us": None, "correctness_ok": False, + "status": "missing_report"}, + "b": {"passed": True, "median_us": 90.0, "correctness_ok": True}} + d = decide(champ, cand) + assert d["performance_gate"]["counts"]["INVALID"] == 1 + assert d["performance_gate"]["counts"]["HARD_FAIL"] == 0 + + +def test_mostly_unmeasurable_round_is_inconclusive(): + champ = {k: {"passed": True, "median_us": 100.0, "correctness_ok": True} + for k in "abcd"} + cand = { + "a": {"passed": False, "median_us": None, "status": "missing_report"}, + "b": {"passed": False, "median_us": None, "status": "missing_report"}, + "c": {"passed": False, "median_us": None, "status": "missing_report"}, + "d": {"passed": True, "median_us": 100.0, "correctness_ok": True}, + } + d = decide(champ, cand) + assert d["verdict"] == "INCONCLUSIVE" + assert d["action"] == "KEEP_CHAMPION_ARCHIVE_CANDIDATE" + assert d["performance_gate"]["status"] == "INCONCLUSIVE" + assert "no verdict" in d["reason"] + + +def test_measurable_regression_still_rejects(): + """Sanity: a genuine, fully measured regression still rejects. + + Two losses are required: ``max_loss_ratio`` deliberately tolerates a single + losing question so one noisy shape cannot trigger a rollback. + """ + champ = {k: {"passed": True, "median_us": 100.0, "correctness_ok": True} + for k in ("a", "b")} + cand = {k: {"passed": True, "median_us": 300.0, "correctness_ok": True} + for k in ("a", "b")} + d = decide(champ, cand) + assert d["verdict"] == "REJECT" + assert d["performance_gate"]["counts"]["LOSS"] == 2 diff --git a/metainfer/tasks/harness_evolve/tests/test_one_driver.py b/metainfer/tasks/harness_evolve/tests/test_one_driver.py new file mode 100644 index 00000000..95c5def1 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_one_driver.py @@ -0,0 +1,240 @@ +"""One experiment, one driver — and "finished" means finished. + +The sweep in the server exists so an experiment whose supervisor died does not +sit idle forever. It got that wrong in two independent ways, and together they +produced the reported symptom ("HE opened the next round before DKAO finished"): + +* it asked ``_resume_running`` whether the task was running, but a task created + from the New Task form never writes ``resume.pid`` — only ``orchestrator.pid`` + — so a one-second-old, healthy run looked like an unsupervised corpse; +* the resume it then started computed the next iteration from + ``completed_iterations``, which counted ``input/stage.json`` — a file written + when a pass *starts* — so "just began" read as "already judged" and the next + round opened on top of the running one. + +Each piece is pinned here separately, because either one alone is enough to +double-drive an experiment. +""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +import time +from pathlib import Path + +import pytest + +from metainfer.tasks.harness_evolve.orchestrator import supervisor as sup +from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + completed_iterations, +) +from metainfer.tasks.harness_evolve.server import routes as R + + +# ------------------------------------------------------- pid files and liveness + +def test_pid_from_file_reads_both_forms(tmp_path): + (tmp_path / "json.pid").write_text( + json.dumps({"pid": 4242, "task_id": "t"}), encoding="utf-8") + (tmp_path / "bare.pid").write_text("4242\n", encoding="utf-8") + (tmp_path / "empty.pid").write_text("", encoding="utf-8") + (tmp_path / "junk.pid").write_text("not-a-pid", encoding="utf-8") + + assert sup.pid_from_file(tmp_path / "json.pid") == 4242 + assert sup.pid_from_file(tmp_path / "bare.pid") == 4242 + assert sup.pid_from_file(tmp_path / "empty.pid") is None + assert sup.pid_from_file(tmp_path / "junk.pid") is None + assert sup.pid_from_file(tmp_path / "missing.pid") is None + + +def test_a_runner_never_finds_itself(tmp_path): + """The regression that made a resume stand down forever. + + In production the resume's own cmdline is + ``python3 -m ...harness_evolve.orchestrator.cli resume --state-dir `` + — which matches the ``/proc`` scan this function falls back to. Answering + "somebody is already running this" with its own pid meant a run with no + completed iteration could never be restarted. + """ + state = tmp_path / "state" + state.mkdir() + proc_root = tmp_path / "proc" + mine = 11111 + (proc_root / str(mine)).mkdir(parents=True) + (proc_root / str(mine) / "cmdline").write_bytes( + f"python3 -m metainfer.tasks.harness_evolve.orchestrator.cli resume " + f"--state-dir {state} --workspace-dir {tmp_path / 'ws'} --iterations 1" + .encode()) + + # asking as *that* process: nobody else is running the experiment + assert sup.task_process_alive(state, proc_root=proc_root, + self_pid=mine) is None + # asking as somebody else: yes, that process is running it + assert sup.task_process_alive(state, proc_root=proc_root, + self_pid=mine + 1) == mine + # an explicit exclusion works too (the supervisor skipping its own child) + assert sup.task_process_alive(state, proc_root=proc_root, self_pid=mine + 1, + exclude=mine) is None + + +def test_a_dead_pid_file_is_not_alive(tmp_path): + state = tmp_path / "state" + state.mkdir() + (state / "orchestrator.pid").write_text( + json.dumps({"pid": 999999, "task_id": "t"}), encoding="utf-8") + assert sup.task_process_alive(state) is None + + +# ------------------------------------------------------------------- the sweep + +def _entry(state: Path, ws: Path, task_id: str = "t-he"): + class Entry: + id = task_id + type = R.PLUGIN_TYPE + state_dir = str(state) + workspace_dir = str(ws) + + return Entry() + + +def _run_json(state: Path, *, finished: bool = False, iteration: int = 1) -> None: + (state / "run.json").write_text( + json.dumps({"task_id": "t-he", "task_type": "harness-evolve", + "current_iteration": iteration, "finished": finished}), + encoding="utf-8") + + +@pytest.fixture() +def sweep_env(tmp_path, monkeypatch): + """A janitor that would launch everything, over a task nobody runs.""" + state = tmp_path / "state" + ws = tmp_path / "ws" + state.mkdir() + ws.mkdir() + launched: list = [] + monkeypatch.setattr("metainfer.server.tasks.list_tasks", + lambda: [_entry(state, ws)]) + monkeypatch.setattr(R, "read_target_iterations", lambda _w: 5) + monkeypatch.setattr(R, "_launch_supervisor", + lambda *a, **k: launched.append("supervisor") or {"pid": 11}) + monkeypatch.setattr(R, "_launch_resume", + lambda *a, **k: launched.append("resume") or {"pid": 22}) + return {"state": state, "ws": ws, "launched": launched} + + +def test_the_sweep_leaves_a_server_started_run_alone(sweep_env, monkeypatch): + """The exact production shape: run alive, no resume.pid, no supervisor.pid. + + This is what a task looks like one second after the New Task form starts it + — and the old liveness check called that "nobody is supervising". + """ + monkeypatch.setattr(R, "_run_alive", lambda _s: 4242) + monkeypatch.setattr(R, "_supervisor_running", lambda _w: None) + _run_json(sweep_env["state"], finished=False, iteration=1) + + actions = R.janitor_sweep() + + assert sweep_env["launched"] == [], ( + "a live run must never get a second driver") + assert actions and actions[0]["action"] == "skip" + assert actions[0]["pid"] == 4242 + + +def test_the_sweep_recovers_a_dead_run(sweep_env, monkeypatch): + """Nothing alive and unfinished: that is the case the sweep exists for.""" + monkeypatch.setattr(R, "_run_alive", lambda _s: None) + monkeypatch.setattr(R, "_supervisor_running", lambda _w: None) + _run_json(sweep_env["state"], finished=False, iteration=2) + + actions = R.janitor_sweep() + + assert sweep_env["launched"] == ["supervisor", "resume"] + assert actions[0]["action"] == "supervisor+resume" + + +def test_the_sweep_restarts_only_the_run_when_a_supervisor_survives( + sweep_env, monkeypatch): + monkeypatch.setattr(R, "_run_alive", lambda _s: None) + monkeypatch.setattr(R, "_supervisor_running", lambda _w: 777) + _run_json(sweep_env["state"], finished=False, iteration=2) + + actions = R.janitor_sweep() + + assert sweep_env["launched"] == ["resume"] + assert actions[0]["action"] == "resume" + + +def test_the_sweep_never_restarts_a_terminal_run(sweep_env, monkeypatch): + monkeypatch.setattr(R, "_run_alive", lambda _s: None) + monkeypatch.setattr(R, "_supervisor_running", lambda _w: None) + (sweep_env["state"] / "run.json").write_text( + json.dumps({"current_iteration": 1, "finished": True, + "final_status": "round_incomplete"}), encoding="utf-8") + + # a run that stopped on purpose gets no action at all: no supervisor, no + # resume, and it needs a human decision + assert R.janitor_sweep() == [] + assert sweep_env["launched"] == [] + + +def test_the_sweep_skips_a_task_the_operator_stopped(sweep_env, monkeypatch): + (sweep_env["ws"] / "operator_stop.json").write_text( + json.dumps({"stopped_at": time.time()}), encoding="utf-8") + _run_json(sweep_env["state"], finished=False, iteration=1) + + assert R.janitor_sweep()[0]["action"] == "skip" + assert sweep_env["launched"] == [] + + +# --------------------------------------------------- "finished" means finished + +def _pass_dir(exp: Path, iteration: int, *files: str) -> Path: + input_dir = exp / "runs" / f"iteration_{iteration:03d}" / "input" + input_dir.mkdir(parents=True, exist_ok=True) + for name in files: + path = input_dir / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("{}", encoding="utf-8") + return input_dir + + +def test_a_pass_that_only_started_is_not_a_completed_iteration(tmp_path): + """``stage.json`` is written when the pass opens the round, not when it ends. + + Counting it made the resume launched by the sweep compute ``start = 2`` + while iteration 1 was still being measured — two rounds on four cards. + """ + exp = tmp_path / "exp" + _pass_dir(exp, 1, "stage.json") # began, produced nothing yet + + assert completed_iterations(exp) == 0 + + +def test_a_pass_that_produced_a_verdict_is_completed(tmp_path): + exp = tmp_path / "exp" + _pass_dir(exp, 1, "stage.json", "baseline_round.json") + _pass_dir(exp, 2, "stage.json", "performance_gate.json") + + assert completed_iterations(exp) == 2 + + _pass_dir(exp, 3, "stage.json") # round 3 is under way + assert completed_iterations(exp) == 2, ( + "an open round must not advance the counter") + + +def test_a_decided_pass_counts_even_without_a_gate_file(tmp_path): + exp = tmp_path / "exp" + _pass_dir(exp, 1, "stage.json", "decision.json") + + assert completed_iterations(exp) == 1 + + +def test_the_generalization_retake_counts_for_its_iteration(tmp_path): + exp = tmp_path / "exp" + _pass_dir(exp, 2, "stage.json", "performance_gate.json") + _pass_dir(exp, 2, "generalization_gate.json") + + assert completed_iterations(exp) == 2 diff --git a/metainfer/tasks/harness_evolve/tests/test_operator_stop.py b/metainfer/tasks/harness_evolve/tests/test_operator_stop.py new file mode 100644 index 00000000..f9944867 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_operator_stop.py @@ -0,0 +1,437 @@ +"""Stopping an AHE run must be a pause, not a death. + +The two are easy to confuse in this codebase: an experiment whose process is +merely gone looks *exactly* like one that crashed, and the supervisor, the +janitor sweep and ``should_launch_resume`` all exist to bring crashed runs back +automatically. So a Stop button that only kills the process would restart the +task seconds later. + +These tests pin the difference: Stop brings down everything that would keep the +round moving (supervisor, DKAO children, orchestrator), records an explicit +stop that no automatic path overrules, and leaves the artifacts intact so the +resume paths can pick the run back up. +""" + +from __future__ import annotations + +import json +import os + +from metainfer.server import tasks as _tasks +from metainfer.server.state_reader import read_run +from metainfer.server.tasks import TaskEntry + +from metainfer.tasks.harness_evolve.server.routes import ( + OPERATOR_STOP_FILE, + _operator_stopped, + janitor_sweep, + should_launch_resume, +) + + +def _he_task(tmp_path, task_id="he-stop", *, finished=False, + final_status="", iteration=2, target=5, resume_pid=None): + state_dir = tmp_path / "state" + workspace = tmp_path / "ws" + state_dir.mkdir(parents=True, exist_ok=True) + workspace.mkdir(parents=True, exist_ok=True) + (state_dir / "requirements.json").write_text(json.dumps({ + "task_id": task_id, "task_type": "harness-evolve", + "answers": {"per_round_budget": 4}, + }), encoding="utf-8") + run = {"task_id": task_id, "task_type": "harness-evolve", + "current_phase": "evaluate", "current_iteration": iteration, + "finished": finished, "last_update": 1.0} + if final_status: + run["final_status"] = final_status + (state_dir / "run.json").write_text(json.dumps(run), encoding="utf-8") + if target is not None: + (workspace / "target_iterations.json").write_text( + json.dumps({"target": target}), encoding="utf-8") + if resume_pid is not None: + (state_dir / "resume.pid").write_text(str(resume_pid), encoding="utf-8") + _tasks.add_task(TaskEntry( + id=task_id, type="harness-evolve", label="AHE", + state_dir=str(state_dir), workspace_dir=str(workspace), created_at=0.0)) + return state_dir, workspace + + +def test_stop_marks_the_run_finished_and_terminal(tmp_path, client): + """A cut-short round must not look like a finished one, and must stay put.""" + state_dir, workspace = _he_task(tmp_path) + + resp = client.post("/api/harness-evolve/he-stop/stop") + assert resp.status_code == 200, resp.text + body = resp.json() + assert body["ok"] is True + assert body["iteration"] == 2 + + run = read_run(state_dir) + assert run["finished"] is True + assert run["final_status"] == "stopped_by_operator" + assert "Stopped by operator." in run["notes"] + + # The marker is what stops every automatic path from reviving it. + marker = json.loads((workspace / OPERATOR_STOP_FILE).read_text()) + assert marker["iteration"] == 2 + assert marker["phase"] == "evaluate" + assert marker["by"] == "webui" + + +def test_a_stopped_run_is_terminal_and_not_restarted(tmp_path, client): + """should_launch_resume must refuse the exact state Stop produces.""" + state_dir, _ = _he_task(tmp_path) + client.post("/api/harness-evolve/he-stop/stop") + assert should_launch_resume(read_run(state_dir), 5, False) is False + + +def test_a_crashed_run_is_still_recovered(tmp_path, client): + """The guard must not disarm crash recovery: only Stop is special.""" + state_dir, _ = _he_task(tmp_path, finished=False) + assert should_launch_resume(read_run(state_dir), 5, False) is True + + +def test_the_janitor_never_restarts_a_stopped_run(tmp_path, client, + monkeypatch): + """The sweep that exists to resurrect dead runs must respect a human.""" + _he_task(tmp_path) + calls = [] + monkeypatch.setattr( + "metainfer.tasks.harness_evolve.server.routes._launch_supervisor", + lambda *a, **k: calls.append("supervisor") or {"pid": 1}) + monkeypatch.setattr( + "metainfer.tasks.harness_evolve.server.routes._launch_resume", + lambda *a, **k: calls.append("resume") or {"pid": 1}) + + assert calls == [] + client.post("/api/harness-evolve/he-stop/stop") + actions = janitor_sweep() + assert calls == [], "the janitor restarted a run the operator stopped" + assert any(a.get("action") == "skip" for a in actions), actions + + +def test_the_janitor_still_recovers_an_unstopped_run(tmp_path, client, + monkeypatch): + """Sanity check on the previous test: without Stop, it does relaunch.""" + _he_task(tmp_path) + calls = [] + monkeypatch.setattr( + "metainfer.tasks.harness_evolve.server.routes._launch_supervisor", + lambda *a, **k: calls.append("supervisor") or {"pid": 1}) + monkeypatch.setattr( + "metainfer.tasks.harness_evolve.server.routes._launch_resume", + lambda *a, **k: calls.append("resume") or {"pid": 1}) + janitor_sweep() + assert calls == ["supervisor", "resume"] + + +def test_stop_clears_a_stale_resume_pid(tmp_path, client): + """Otherwise /resume would answer "already running" about a dead process.""" + state_dir, _ = _he_task(tmp_path, resume_pid=999999) # not our process + resp = client.post("/api/harness-evolve/he-stop/stop") + assert resp.status_code == 200 + assert not (state_dir / "resume.pid").exists() + follow_up = client.post("/api/harness-evolve/he-stop/resume", json={}) + assert follow_up.status_code == 200, follow_up.text + + +def test_resume_clears_the_marker_so_the_run_can_continue(tmp_path, client, + monkeypatch): + """"Restart later" has to actually work: the marker must not outlive it. + + The marker is cleared by ``_launch_resume`` itself -- the function that + decides the run continues -- so its call is inspected at the spawn instead + of being replaced: replacing it would assert nothing about clearing. + """ + import subprocess + + state_dir, workspace = _he_task(tmp_path) + client.post("/api/harness-evolve/he-stop/stop") + assert (workspace / OPERATOR_STOP_FILE).exists() + + spawned = {} + + class _FakeProc: + pid = 4242 + + def fake_popen(cmd, *a, **k): + spawned["cmd"] = cmd + # The state the run is launched in: this is what the supervisor and the + # janitor will see from the moment the process exists. + spawned["marker_gone"] = not (workspace / OPERATOR_STOP_FILE).exists() + return _FakeProc() + + monkeypatch.setattr(subprocess, "Popen", fake_popen) + resp = client.post("/api/harness-evolve/he-stop/resume", json={}) + assert resp.status_code == 200, resp.text + assert "resume" in spawned.get("cmd", []), "resume refused to relaunch" + assert spawned["marker_gone"], \ + "the stop marker was still there when the new run was spawned" + assert _operator_stopped(workspace) is None + + +def test_the_cli_resume_clears_the_marker_too(tmp_path): + """The server is not the only way back in; the CLI path must agree.""" + from metainfer.tasks.harness_evolve.orchestrator.cli import ( + _clear_operator_stop, + ) + workspace = tmp_path / "ws" + workspace.mkdir() + (workspace / OPERATOR_STOP_FILE).write_text("{}", encoding="utf-8") + assert _clear_operator_stop(workspace) == [OPERATOR_STOP_FILE] + assert not (workspace / OPERATOR_STOP_FILE).exists() + assert _clear_operator_stop(workspace) == [] # idempotent + + +def test_summary_reports_the_stop_so_the_page_can_explain_it(tmp_path, client): + _he_task(tmp_path) + before = client.get("/api/harness-evolve/he-stop/summary").json() + assert before["operator_stop"] is None + client.post("/api/harness-evolve/he-stop/stop") + after = client.get("/api/harness-evolve/he-stop/summary").json() + assert after["operator_stop"]["iteration"] == 2 + + +def test_stop_only_targets_this_task_type(tmp_path, client): + """A DKAO task id must not reach the HE stop route.""" + _he_task(tmp_path) + _tasks.add_task(TaskEntry( + id="dkao-x", type="dcu-kernel-auto-opt", label="d", + state_dir=str(tmp_path / "d"), workspace_dir=str(tmp_path / "dw"), + created_at=0.0)) + assert client.post("/api/harness-evolve/dkao-x/stop").status_code == 409 + assert client.post("/api/harness-evolve/nope/stop").status_code == 404 + + +def test_stop_reports_children_it_brought_down(tmp_path, client, monkeypatch): + """Stopping must reach the DKAO children, which outlive the parent.""" + _he_task(tmp_path) + from metainfer.tasks.harness_evolve.server import routes as r + killed = [] + monkeypatch.setattr(r, "_live_dkao_children", + lambda ws: [{"pid": 111, "cmd": "child a"}, + {"pid": 222, "cmd": "child b"}]) + monkeypatch.setattr(r, "_terminate_pid", + lambda pid, **k: killed.append(pid) or True) + body = client.post("/api/harness-evolve/he-stop/stop").json() + assert body["stopped_children"] == [111, 222] + # the second sweep must not re-kill what the first one already took down + assert killed == [111, 222] + assert body["relaunched_children"] == [] + + +def test_stop_reaches_an_orchestrator_that_never_wrote_resume_pid( + tmp_path, client, monkeypatch): + """A run started from the New Task form has no ``resume.pid``. + + Stop used to look for the orchestrator through ``_resume_running``, which + reads only that file: on such a run it found nothing, answered "stopped", + and left the orchestrator alive -- which then re-launched the children Stop + had just killed as an env retry. ``_run_alive`` is the wide check + (``orchestrator.pid``, then ``resume.pid``, then a cmdline scan). + """ + state_dir, _ = _he_task(tmp_path) # no resume_pid on purpose + from metainfer.tasks.harness_evolve.server import routes as r + + killed = [] + monkeypatch.setattr(r, "_live_dkao_children", lambda ws: []) + monkeypatch.setattr(r, "_run_alive", lambda sd: 4242) + monkeypatch.setattr(r, "_terminate_pid", + lambda pid, **k: killed.append(pid) or True) + + body = client.post("/api/harness-evolve/he-stop/stop").json() + assert body["stopped_orchestrator_pid"] == 4242 + assert killed == [4242], "the orchestrator was left running" + assert body["stopped"] is True + # ... and a task that wrote neither pid file is still "nothing to stop" + assert r._resume_running(state_dir) is None + + +def test_stop_sweeps_the_retry_wave_the_orchestrator_launched_while_dying( + tmp_path, client, monkeypatch): + """The env retry can land between the child sweep and the parent kill. + + That wave is a *new* child, not one of the pids already stopped, so a single + sweep leaves it optimizing for hours while the page says "stopped". + """ + _he_task(tmp_path) + from metainfer.tasks.harness_evolve.server import routes as r + + waves = [[{"pid": 111, "cmd": "first wave"}], + [{"pid": 111, "cmd": "first wave"}, + {"pid": 333, "cmd": "retry wave"}]] + killed = [] + monkeypatch.setattr(r, "_live_dkao_children", lambda ws: waves.pop(0)) + monkeypatch.setattr(r, "_run_alive", lambda sd: 4242) + monkeypatch.setattr(r, "_terminate_pid", + lambda pid, **k: killed.append(pid) or True) + + body = client.post("/api/harness-evolve/he-stop/stop").json() + assert killed == [111, 4242, 333], killed + assert body["stopped_children"] == [111, 333] + assert body["relaunched_children"] == [333] + + +def test_a_terminate_failure_is_reported_not_hidden(tmp_path, client, + monkeypatch): + """The operator has to know whether the process actually went away. + + A stale pid is not a live process, so this drives ``_live_dkao_children`` + instead: it is the only path that yields a pid the server will try to kill. + """ + state_dir, _ = _he_task(tmp_path) + from metainfer.tasks.harness_evolve.server import routes as r + monkeypatch.setattr(r, "_live_dkao_children", + lambda ws: [{"pid": 111, "cmd": "child"}]) + monkeypatch.setattr(r, "_terminate_pid", lambda pid, **k: False) + monkeypatch.setattr(r, "_run_alive", lambda sd: 999999) + + body = client.post("/api/harness-evolve/he-stop/stop").json() + assert body["stopped"] is False + assert body["stopped_children"] == [] # it did not go away + assert "可再点一次" in body["message"] + # ... and the run is still marked stopped, so nothing restarts it blindly + assert read_run(state_dir)["final_status"] == "stopped_by_operator" + + +def test_terminate_pid_actually_stops_a_real_process(tmp_path): + """The one path mocks cannot prove: does the signal really land? + + Stop is only a pause if the processes are genuinely gone -- a surviving + DKAO child would keep its agents and its place in DKAO's admission queue + while the UI says "stopped". + """ + import subprocess + import sys + import time + + from metainfer.tasks.harness_evolve.server.routes import _terminate_pid + + # Own session, like every process the experiment spawns; the SIGTERM goes + # to the group first, so this must not be the test runner's group. + proc = subprocess.Popen([sys.executable, "-c", + "import time\nwhile True: time.sleep(1)\n"], + start_new_session=True) + try: + assert proc.poll() is None + assert _terminate_pid(proc.pid, grace_s=5.0) is True + deadline = time.time() + 5.0 + while proc.poll() is None and time.time() < deadline: + time.sleep(0.1) + assert proc.poll() is not None, "the process survived the stop" + finally: + if proc.poll() is None: + proc.kill() + proc.wait(timeout=5) + + +def test_terminate_pid_reports_a_process_it_cannot_kill(monkeypatch): + """A refusal must come back as False, not as a silent success.""" + from metainfer.tasks.harness_evolve.server import routes as r + + monkeypatch.setattr(r.os, "kill", lambda *a, **k: None) # never dies + monkeypatch.setattr(r, "_proc_state", lambda pid: "S") # ... and runs on + assert r._terminate_pid(12345, grace_s=0.3) is False + + +def test_a_zombie_counts_as_stopped(): + """Otherwise every successful stop of a non-child reports failure.""" + from metainfer.tasks.harness_evolve.server import routes as r + + assert r._proc_state(os.getpid()) in {"R", "S", "D"} + assert r._proc_state(99999999) is None # gone + + +def test_terminate_pid_treats_a_zombie_as_stopped(monkeypatch): + from metainfer.tasks.harness_evolve.server import routes as r + + monkeypatch.setattr(r.os, "kill", lambda *a, **k: None) + monkeypatch.setattr(r, "_proc_state", lambda pid: "Z") + assert r._terminate_pid(12345, grace_s=0.3) is True + + +# ------------------------------------------------- liveness without a pid file + + +def test_summary_reports_a_live_run_the_shell_cannot_see(tmp_path, client, + monkeypatch): + """Stop must appear on a run whose orchestrator.pid was never written. + + The shell's green dot reads that file; runs started before it existed are + alive with a blue dot, and inheriting that blind spot would hide Stop on + exactly the runs that need it. + """ + from metainfer.tasks.harness_evolve.server import routes as r + + _he_task(tmp_path, resume_pid=4242) + monkeypatch.setattr(r, "_resume_running", lambda sd: 4242) + monkeypatch.setattr(r, "_supervisor_running", lambda ws: None) + monkeypatch.setattr(r, "_live_dkao_children", lambda ws: []) + body = client.get("/api/harness-evolve/he-stop/summary").json() + assert body["running"] is True + # and the shell's own view stays untouched: this is a UI hint, not a + # rewrite of the shared liveness contract + shell = client.get("/api/sys-shell/he-stop").json()["status"] + assert shell.get("running") is False + + +def test_summary_sees_liveness_through_a_running_child(tmp_path, client, + monkeypatch): + """A round can be mid-flight while the parent is between steps.""" + from metainfer.tasks.harness_evolve.server import routes as r + + _he_task(tmp_path) + monkeypatch.setattr(r, "_resume_running", lambda sd: None) + monkeypatch.setattr(r, "_supervisor_running", lambda ws: None) + monkeypatch.setattr(r, "_live_dkao_children", + lambda ws: [{"pid": 7, "cmd": "child"}]) + assert client.get("/api/harness-evolve/he-stop/summary" + ).json()["running"] is True + + +def test_summary_is_not_running_after_a_stop(tmp_path, client, monkeypatch): + """The button must disappear once the stop has actually landed.""" + from metainfer.tasks.harness_evolve.server import routes as r + + _he_task(tmp_path) + monkeypatch.setattr(r, "_live_dkao_children", lambda ws: []) + client.post("/api/harness-evolve/he-stop/stop") + body = client.get("/api/harness-evolve/he-stop/summary").json() + assert body["running"] is False + assert body["operator_stop"] is not None + + +# --------------------------------------------- what the operator is told to do + + +def _read(rel: str) -> str: + from pathlib import Path + return (Path(__file__).resolve().parents[1] / rel).read_text(encoding="utf-8") + + +def test_the_page_calls_it_a_pause_and_offers_the_way_back(): + """The button is where most operators will ever read about this.""" + src = _read("static/he-detail.js") + assert "停止任务（暂停）" in src, "the button no longer says it is a pause" + assert "不会被 janitor / supervisor 自动拉起" in src, ( + "the page must tell the operator a stop sticks") + assert "FLOW.md §9" in src, "the page must point at the operations note" + # A stopped run keeps its phase, so "is the graph finished?" is the wrong + # question for the button next to it: what the click does depends on whether + # anything is running. + assert '"续跑到该轮次上限"' in src + assert '${isFinished ? "续跑到该轮次上限"' not in src, ( + "the resume label is gated on the graph again: a stopped run would offer " + "「更新上限（下一轮生效）」 while the click actually re-launches it") + + +def test_the_operations_note_is_written_down(): + """FLOW.md §9 is the operator manual this feature is explained in.""" + flow = _read("FLOW.md") + assert "## 9. 停止与续跑" in flow + for needle in ("operator_stop.json", "stopped by operator", "env_retry_attempts", + "Reset", "_run_alive"): + assert needle in flow, f"the operations note no longer mentions {needle}" + # ... and the README, which is what a maintainer opens first, points at it. + readme = _read("README.md") + assert "§9" in readme and "FLOW.md" in readme diff --git a/metainfer/tasks/harness_evolve/tests/test_pipeline_dry.py b/metainfer/tasks/harness_evolve/tests/test_pipeline_dry.py new file mode 100644 index 00000000..494be4b0 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_pipeline_dry.py @@ -0,0 +1,132 @@ +"""Offline tests for harness_evolve (dry-run, no GPU/LLM).""" + +from __future__ import annotations + +import json +from pathlib import Path + +from ..orchestrator.attribution import diff_results, evaluate_changes, pass_rate +from ..orchestrator.cli import run_with_requirements +from ..orchestrator.config import load_experiment_config + + +def _write_requirements(tmp: Path, answers: dict, task_id: str = "he-test") -> Path: + req = tmp / "requirements.json" + req.write_text(json.dumps({ + "task_id": task_id, + "answers": answers, + }, ensure_ascii=False), encoding="utf-8") + return req + + +def test_default_suite_config(tmp_path): + req = _write_requirements(tmp_path, {"max_iterations": 1}) + cfg = load_experiment_config(req, tmp_path / "state", tmp_path / "ws") + assert len(cfg.suite) == 2 + assert cfg.suite[0].id == "hy3_tp8_qkv_m4096_w4" + + +def test_pipeline_dry_run_layout(tmp_path, monkeypatch): + monkeypatch.delenv("HARNESS_EVOLVE_FAKE_CHANGE", raising=False) + req = _write_requirements(tmp_path, {"max_iterations": 2}) + rc = run_with_requirements( + req, state_dir=tmp_path / "state", workspace_dir=tmp_path / "ws" + ) + assert rc == 0 + ws = tmp_path / "ws" + + # seeded evolvable workspace from the dcu harness_default seed + assert (ws / "workspace" / "manifest.yaml").is_file() + + i1 = ws / "runs" / "iteration_001" / "input" + assert (i1 / "workspace" / "gates.yaml").is_file() + assert (i1 / "benchmark" / "results.json").is_file() + assert (i1 / "diff.json").is_file() + assert (i1 / "analysis" / "overview.md").is_file() + assert (ws / "best_ever.json").is_file() + scores = (ws / "iteration_scores.jsonl").read_text().strip().splitlines() + assert len(scores) == 2 + assert (ws / "report.md").is_file() + # iteration 2 attributes iteration 1's (empty) manifest and emits an + # explicit NO_SIGNAL decision. Equal performance must NOT be a reject + # rollback; the candidate is archived while champion stays active. + i2_root = ws / "runs" / "iteration_002" + assert (i2_root / "input" / "change_evaluation.json").is_file() + decision = json.loads((i2_root / "input" / "decision.json").read_text()) + assert decision["verdict"] == "NO_SIGNAL" + assert not (i2_root / "rollback.json").exists() + disposition = json.loads( + (i2_root / "candidate_disposition.json").read_text() + ) + assert disposition["action"] == "KEEP_CHAMPION_ARCHIVE_CANDIDATE" + + +def test_attribution_verdicts(): + results_pass = {"t1": {"passed": True}, "t2": {"passed": False}} + results_fail = {"t1": {"passed": False}, "t2": {"passed": False}} + diff = diff_results(results_pass, results_fail) + assert "t1" in diff["regressed"] + stats = pass_rate(results_pass) + assert stats["n_pass"] == 1 and stats["n_total"] == 2 + + manifest = { + "iteration": 2, + "changes": [{ + "id": "chg-1", + "description": "fix t1", + "files": ["gates.yaml"], + "predicted_fixes": ["t1"], + "risk_tasks": ["t2"], + }], + } + ev = evaluate_changes(manifest, diff) + entry = ev["change_evaluations"][0] + assert entry["verdict"] == "INEFFECTIVE" # predicted t1 regressed, t2 not realized + + # HARMFUL: a declared risk regressed while nothing predicted was fixed + diff2 = diff_results( + {"t1": {"passed": True}, "t2": {"passed": True}}, + {"t1": {"passed": True}, "t2": {"passed": False}}, + ) + ev2 = evaluate_changes(manifest, diff2) + assert ev2["change_evaluations"][0]["verdict"] == "HARMFUL" + assert ev2["unattributed_regressions"] == [] + + +def test_code_change_defers_to_the_next_round(tmp_path, monkeypatch): + """A harness fix must not wait for the run to end — nor apply mid-round. + + The running process already imported the old modules, so the loop stops at + the round boundary with ``code_reload_pending:``; the restart records + its own revision and resumes there. + """ + from ..orchestrator import pipeline as pl + from ..orchestrator.state import loaded_harness_revision + + req = _write_requirements(tmp_path, {"max_iterations": 5}) + cfg = load_experiment_config(req, tmp_path / "state", tmp_path / "ws") + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=1) + loaded = loaded_harness_revision(cfg.state_dir) + assert loaded + + # someone edits the loop while the experiment is between rounds + monkeypatch.setattr(pl, "harness_revision", lambda: "rev-changed") + pl.run_experiment(cfg, start_iteration=2, iterations_to_run=1) + + run = json.loads((cfg.state_dir / "run.json").read_text(encoding="utf-8")) + assert run["final_status"] == "code_reload_pending:3" + assert pl.completed_iterations(cfg.exp_dir) == 2 # both rounds are real + events = [json.loads(l) for l in + (cfg.state_dir / "timeline.jsonl").read_text( + encoding="utf-8").splitlines() if l.strip()] + deferred = [e for e in events if e["type"] == "code_reload_deferred"] + assert deferred and deferred[-1]["payload"]["resumes_at_iteration"] == 3 + assert deferred[-1]["payload"]["on_disk_revision"] == "rev-changed" + + # the restarted process records its own revision and keeps going + monkeypatch.undo() + pl.run_experiment(cfg, start_iteration=3, iterations_to_run=1) + assert pl.completed_iterations(cfg.exp_dir) == 3 + run = json.loads((cfg.state_dir / "run.json").read_text(encoding="utf-8")) + assert not str(run.get("final_status") or "").startswith("code_reload") diff --git a/metainfer/tasks/harness_evolve/tests/test_pool_protocol.py b/metainfer/tasks/harness_evolve/tests/test_pool_protocol.py new file mode 100644 index 00000000..87cd7bef --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_pool_protocol.py @@ -0,0 +1,333 @@ +"""Tests for suite protocol v2: pool, auto-pass, round_plan guardrails, and +pool-mode pipeline decisions.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from ..orchestrator.pipeline import run_experiment +from ..orchestrator.pool import builtin_pool +from ..orchestrator.round_plan import ( + RoundPlan, + enforce_guardrails, + validate_in_pool, +) +from ..orchestrator.attribution import pass_rate + + +def _pool(): + return builtin_pool() + + +def test_builtin_pool_sampling_and_heldout(): + pool = _pool() + ids = pool.stratified_sample(4) + assert len(ids) == 4 and len(set(ids)) == 4 + pool.assert_in_pool(ids) + held = pool.heldout_ids(2, exclude=set(ids)) + assert len(held) == 2 and not (set(held) & set(ids)) + + +def test_tau_and_auto_pass(): + pool = _pool() + inst_id = "dsv4_tp8_qkv_proj_m16" # family decode__qkv_proj + inst = pool.get(inst_id) + tau = pool.tau_family(inst.family) + assert tau > 0.0 + ok = pool.auto_pass(inst_id, tau * inst.baseline_us * 0.9, + correctness_ok=True, p90_ok=True) + assert ok["passed"] is True + bad = pool.auto_pass(inst_id, inst.baseline_us * 0.9, + correctness_ok=True, p90_ok=True) + assert bad["passed"] is False # ratio 0.9 > tau + noc = pool.auto_pass(inst_id, 1.0, correctness_ok=False, p90_ok=True) + assert noc["passed"] is False + + +def test_round_plan_in_pool_validation(): + pool = _pool() + plan = RoundPlan(iteration=1, selected=["dsv4_tp8_qkv_proj_m16", + "hy3_tp8_qkv_proj_m16"]) + validate_in_pool(plan, pool) + bad = RoundPlan(iteration=1, selected=["not-in-pool"]) + try: + validate_in_pool(bad, pool) + raise AssertionError("expected ValueError") + except ValueError: + pass + + +def test_guardrails_preserve_overlap_and_budget(): + pool = _pool() + prev = pool.stratified_sample(5) + # agent proposes a fresh set of 3 different ids + fresh = [i for i in pool.stratified_sample(9) if i not in set(prev)][:3] + plan = RoundPlan(iteration=2, strategy="explore", + rationale="try fresh", selected=fresh) + out = enforce_guardrails(prev, plan, pool, budget=5) + overlap = len(set(out) & set(prev)) + assert overlap >= int(0.6 * len(prev)) # >= 3 of 5 + assert len(out) == 5 and len(set(out)) == 5 + pool.assert_in_pool(out) + + +def test_pipeline_pool_mode_dry(tmp_path): + """pool-mode: results restricted to scored ids, round_plan written per + evolve, overlap decisions recorded.""" + req = tmp_path / "req.json" + req.write_text(json.dumps({ + "task_id": "pool-dry", + "answers": { + "suite_yaml": "instances: []", + "max_iterations": "2", + "execution_mode": "dry-run", + "pool_source": "builtin", + "per_round_budget": "4", + }, + }), encoding="utf-8") + from ..orchestrator.cli import run_with_requirements + rc = run_with_requirements(req, state_dir=tmp_path / "st", + workspace_dir=tmp_path / "ws") + assert rc == 0 + ws = tmp_path / "ws" + pool = _pool() + + i1 = ws / "runs" / "iteration_001" / "input" + i2 = ws / "runs" / "iteration_002" / "input" + r1 = json.loads((i1 / "benchmark" / "results.json").read_text())["results"] + scored1 = json.loads((i1 / "benchmark" / "scored_ids.json").read_text()) + assert set(r1) == set(scored1) + assert len(scored1) <= 4 + pool.assert_in_pool(scored1) + # evolve writes round_plan each round + assert (ws / "runs" / "iteration_001" / "evolve" + / "round_plan.json").is_file() + assert (ws / "runs" / "iteration_002" / "evolve" + / "round_plan.json").is_file() + # iteration 2 overlap decision artefacts exist + assert (i2 / "diff.json").is_file() + stats = pass_rate(r1) + assert stats["n_total"] == len(scored1) + + +def _mixed_pool(): + from ..orchestrator.pool import Pool, PoolInstance + insts = {} + for iid in ("anchor_a", "anchor_b"): + insts[iid] = PoolInstance( + id=iid, model="m", tp_size=8, operator="o", M=16, N=1, K=1, + baseline_us=100.0, family="decode__o", + history=[{"accepted": True, "median_us": 45.0}], + ) + for iid in ("cold_a", "cold_b"): + insts[iid] = PoolInstance( + id=iid, model="m", tp_size=8, operator="o", M=16, N=1, K=1, + baseline_us=100.0, family="decode__o", history=[], + ) + return Pool(insts) + + +def test_anchor_rule_keeps_two_historical_shapes(): + pool = _mixed_pool() + plan = RoundPlan(iteration=1, strategy="explore", + rationale="all cold this round", + selected=["cold_a", "cold_b"]) + out = enforce_guardrails([], plan, pool, budget=4) + assert len(out) == 4 + anchors = [i for i in out if i in ("anchor_a", "anchor_b")] + assert len(anchors) >= 2 + + +def test_budget_below_anchor_min_raises(): + pool = _mixed_pool() + try: + enforce_guardrails([], None, pool, budget=1) + raise AssertionError("expected ValueError") + except ValueError: + pass + + +def test_historical_ids_helper(): + pool = _mixed_pool() + assert set(pool.historical_ids()) == {"anchor_a", "anchor_b"} + + +def test_pool_dry_run_assigns_question_roles(tmp_path): + """Every round carries explicit roles, and the decision records them.""" + req = tmp_path / "req.json" + req.write_text(json.dumps({ + "task_id": "pool-roles", + "answers": { + "suite_yaml": "instances: []", + "max_iterations": "2", + "execution_mode": "dry-run", + "pool_source": "builtin", + "per_round_budget": "4", + }, + }), encoding="utf-8") + from ..orchestrator.cli import run_with_requirements + assert run_with_requirements(req, state_dir=tmp_path / "st", + workspace_dir=tmp_path / "ws") == 0 + ws = tmp_path / "ws" + + for num in (1, 2): + base = ws / "runs" / f"iteration_{num:03d}" / "input" + plan = json.loads((base / "benchmark" / "purposes.json") + .read_text(encoding="utf-8")) + purposes = plan["purposes"] + scored = json.loads((base / "benchmark" / "scored_ids.json") + .read_text(encoding="utf-8")) + assert set(purposes) == set(scored) + roles = list(purposes.values()) + assert roles.count("regression") >= 2 + assert "repair" in roles + assert "probe" in roles + assert len(set(plan["families"])) >= 2 + + decision = json.loads((base / "decision.json").read_text(encoding="utf-8")) + if "generalization_gate" in decision: + assert decision["probe_ids"] + assert "p" not in decision["paired_ids"] or True + assert decision["purpose_breakdown"] + + # rotation sidecar records what was examined + rotation = json.loads((ws / "pool_rotation.json").read_text(encoding="utf-8")) + assert rotation and max(rotation.values()) >= 1 + + +def test_wired_scope_violations_detects_changes_to_unwired_components(tmp_path): + """Editing a component the manifest marks unwired must be reported.""" + from ..orchestrator.pipeline import _wired_scope_violations + + before = tmp_path / "before" + after = tmp_path / "after" + for root in (before, after): + root.mkdir() + manifest = { + "schema_version": 1, + "components": { + "gates": {"path": "gates.yaml", "wired": True}, + "prompts": {"path": "prompts.yaml", "wired": False}, + }, + } + for root in (before, after): + (root / "manifest.yaml").write_text(json.dumps(manifest), encoding="utf-8") + (root / "gates.yaml").write_text("gates: {}\n", encoding="utf-8") + (root / "prompts.yaml").write_text("a: 1\n", encoding="utf-8") + + assert _wired_scope_violations(before, after) == [] + + # edit the wired component -> no violation reported + (after / "gates.yaml").write_text("gates: {x: 1}\n", encoding="utf-8") + assert _wired_scope_violations(before, after) == [] + + # edit the unwired component -> violation reported + (after / "prompts.yaml").write_text("a: 2\n", encoding="utf-8") + violations = _wired_scope_violations(before, after) + assert violations == [{"component": "prompts", "path": "prompts.yaml"}] + + +def test_generations_are_recorded_and_champion_is_the_best_measured(tmp_path): + """Each round is registered as a generation and best_ever points at it.""" + req = tmp_path / "req.json" + req.write_text(json.dumps({ + "task_id": "pool-generations", + "answers": { + "suite_yaml": "instances: []", + "max_iterations": "3", + "execution_mode": "dry-run", + "pool_source": "builtin", + "per_round_budget": "4", + }, + }), encoding="utf-8") + from ..orchestrator.cli import run_with_requirements + from ..orchestrator.champion_selection import ( + best_generation, load_generations, + ) + assert run_with_requirements(req, state_dir=tmp_path / "st", + workspace_dir=tmp_path / "ws") == 0 + ws = tmp_path / "ws" + gens = load_generations(ws) + assert [g.iteration for g in gens] == [1, 2, 3] + best = best_generation(gens) + assert best is not None + best_ever = json.loads((ws / "best_ever.json").read_text(encoding="utf-8")) + assert best_ever["iteration"] == best.iteration + assert best_ever["verdict"] in {"BEST_MEASURED", "MANUAL_BASELINE", + "BASELINE", "PROMOTE", + "PROMOTE_UNEXPLAINED"} + + +def test_dry_run_writes_best_known_review_and_skips_final_evolve(tmp_path): + req = tmp_path / "req.json" + req.write_text(json.dumps({ + "task_id": "pool-review", + "answers": { + "suite_yaml": "instances: []", + "max_iterations": "3", + "execution_mode": "dry-run", + "pool_source": "builtin", + "per_round_budget": "4", + }, + }), encoding="utf-8") + from ..orchestrator.cli import run_with_requirements + assert run_with_requirements(req, state_dir=tmp_path / "st", + workspace_dir=tmp_path / "ws") == 0 + ws = tmp_path / "ws" + + assert (ws / "best_known.json").is_file() + review = json.loads((ws / "round_review.json").read_text(encoding="utf-8")) + assert review["variant_policy"] == "freeze" + assert review["champion_iteration"] in {1, 2, 3} + assert (ws / "round_review.md").read_text(encoding="utf-8").startswith( + "# Cross-round review") + + manifest = json.loads((ws / "runs" / "iteration_003" / "evolve" + / "change_manifest.json").read_text(encoding="utf-8")) + assert manifest["verification"]["status"] == "target_reached_skip_evolve" + assert manifest["changes"] == [] + + +def test_resume_skips_a_round_that_was_marked_skipped(tmp_path): + """A skipped round must not block the loop: the next round still runs. + + Mirrors the live handover (iteration 2 ran while the GPU gate was broken + and was marked skipped): its eval artifacts are absent, so the resumed + process has to pick the next round up from the previous round's evolve dir. + """ + req = tmp_path / "req.json" + req.write_text(json.dumps({ + "task_id": "pool-skip", + "answers": { + "suite_yaml": "instances: []", + "max_iterations": "4", + "execution_mode": "dry-run", + "pool_source": "builtin", + "per_round_budget": "3", + }, + }), encoding="utf-8") + from ..orchestrator.config import load_experiment_config + from ..orchestrator.cli import run_experiment + from ..orchestrator.pipeline import completed_iterations + + cfg = load_experiment_config(req, tmp_path / "st", tmp_path / "ws") + run_experiment(cfg, start_iteration=1, iterations_to_run=1) + assert completed_iterations(cfg.exp_dir) == 1 + + # mark iteration 2 skipped: results recorded as empty, no evolve artifacts + it2 = cfg.exp_dir / "runs" / "iteration_002" / "input" + (it2 / "benchmark").mkdir(parents=True, exist_ok=True) + (it2 / "benchmark" / "results.json").write_text( + json.dumps({"iteration": 2, "results": {}, "skipped": True}), + encoding="utf-8") + assert completed_iterations(cfg.exp_dir) == 2 + + # the resumed process starts at round 3 and runs it end to end + run_experiment(cfg, start_iteration=3, iterations_to_run=1) + it3 = cfg.exp_dir / "runs" / "iteration_003" / "input" + results = json.loads((it3 / "benchmark" / "results.json").read_text()) + assert results["results"], "round 3 must still select and score questions" + assert (it3 / "decision.json").is_file() + assert (it3 / "diff.json").is_file() + assert completed_iterations(cfg.exp_dir) == 3 diff --git a/metainfer/tasks/harness_evolve/tests/test_promotion_flow.py b/metainfer/tasks/harness_evolve/tests/test_promotion_flow.py new file mode 100644 index 00000000..01baa79d --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_promotion_flow.py @@ -0,0 +1,698 @@ +"""End-to-end: a round judged against variants, then a human-approved promotion. + +Covers the operator's protocol: round 1 defines the generalization paper, the +performance gate uses fresh operators, both gates compare against the current +variant, a passing candidate waits for a human, and approval publishes the +harness and moves the variants. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest +import yaml + +from ..orchestrator import pipeline as pl +from ..orchestrator import rounds as R +from ..orchestrator import variant as V +from ..orchestrator import promotion as promo +from ..orchestrator.config import load_experiment_config + + +class FakeEvaluator: + """Deterministic measurements: ``medians`` maps operator id -> us. + + ``medians`` may also be a callable ``(iteration, stage) -> {id: median}``, so + a test can make the generalization retake measure differently from the + baseline round that fixed the paper — that is the only way to say anything + about what an approval publishes. + """ + + name = "fake" + + def __init__(self, medians): + self.medians = medians + self.calls = [] + self.stages = [] + + def evaluate(self, cfg, workspace, iteration, instances=None): + medians = self.medians + stage = str((getattr(cfg, "answers", None) or {}).get("round_stage") or "") + if callable(medians): + medians = medians(iteration, stage) + self.stages.append((iteration, stage)) + out = {} + for inst in (instances or []): + iid = inst.id + self.calls.append(iid) + median = medians.get(iid) + if median is None: + out[iid] = {"status": "missing_report", "passed": False, + "correctness_ok": False, "median_us": None} + continue + out[iid] = {"status": "success", "passed": True, + "correctness_ok": True, "p90_ok": True, + "median_us": float(median), "p90_us": float(median), + "repo_path": f"/repos/{iid}"} + return out + + +def _write_pool(tmp_path: Path, ops: dict) -> Path: + path = tmp_path / "pool.yaml" + path.write_text(yaml.safe_dump({ + "schema_version": 1, + "instances": [ + {"id": iid, "family": f"fam{i}", + "contract": {"model": "m", "tp_size": 8, "operator": "o", + "M": 16, "N": 64, "K": 32}, + "baseline_us": values["baseline"], "best_known_us": values["variant"], + "history": []} + for i, (iid, values) in enumerate(sorted(ops.items())) + ], + }, sort_keys=False), encoding="utf-8") + return path + + +def _cfg(tmp_path: Path, pool: Path, **answers): + req = tmp_path / "requirements.json" + req.write_text(json.dumps({ + "task_id": "variant-e2e", + "answers": { + "suite_yaml": "instances: []", + "execution_mode": "dry-run", + "evolve_mode": "dry-run", + "pool_source": str(pool), + "question_pool": str(pool), + "round_questions": "variant", + "per_round_budget": "3", + "max_iterations": "4", + "max_rounds": "3", + **answers, + }, + }), encoding="utf-8") + return load_experiment_config(req, tmp_path / "state", tmp_path / "ws") + + +def _patch(monkeypatch, tmp_path, evaluator, prod_dir): + # A real DKAO child leaves its optimized workspace behind; the kernel + # promotion path reads exactly that directory, so the fake evaluator + # materializes one per question it measured. + original = evaluator.evaluate + measured: dict = {} + + def evaluate(cfg, workspace, iteration, instances=None): + results = original(cfg, workspace, iteration, instances=instances) + from ..orchestrator.rounds import CHILDREN_DIR, stage_dir_name + stage = pl.round_stage( + iteration=iteration, + pending=pl.read_pending_generalization(cfg.exp_dir)) + override = cfg.answers.pop("_attempt_dir", None) + for iid, row in results.items(): + measured[str(iid)] = row + child = (cfg.exp_dir / CHILDREN_DIR + / (override or stage_dir_name(stage, iteration)) + / str(iid) / "workspace") + child.mkdir(parents=True, exist_ok=True) + (child / "manifest.yaml").write_text( + f"harness_name: candidate-{iid}\n", encoding="utf-8") + return results + + evaluator.evaluate = evaluate + # The real child writes an accepted-kernel manifest inside that workspace; + # the fake does not, so the DKAO-side promotion is stubbed at its boundary. + # Everything the parent does around it (best-known wins on the pool file, + # backups, history rows, audit log) stays real. + from metainfer.tasks.dcu_kernel_auto_opt.orchestrator import variant_promote + + def promote_variant(*, workspace_dir, answers, shape_id, source_task="", + correctness_ok=None, min_improvement_percent=0.0, + tp=None, m=None, model_label=None, dry_run=False, + **kwargs): + # A real child only has an accepted kernel in the workspace that ran. + # Reading the manifest keeps this stub honest: pointing it at a + # directory that never produced a kernel fails, like DKAO would. + try: + manifest = (Path(workspace_dir) / "manifest.yaml").read_text( + encoding="utf-8") + except OSError: + manifest = "" + if "failed-attempt" in manifest: + return {"ok": False, "action": "no_kernel", "shape": shape_id, + "reason": f"no accepted kernel under {workspace_dir}"} + row = measured.get(str(shape_id)) or {} + action = "would-add" if dry_run else "added" + return {"ok": True, "action": action, "shape": shape_id, + "new_median_us": row.get("median_us"), + "old_median_us": None, "path": f"/variants/{shape_id}"} + + monkeypatch.setattr(variant_promote, "promote_variant", promote_variant) + monkeypatch.setattr(pl, "build_evaluator", lambda mode: evaluator) + monkeypatch.setattr(promo, "_production_dir", lambda: prod_dir) + monkeypatch.setattr(pl, "_head_revision", lambda ws: "rev-candidate") + + +def _steady(ops, share=0.5): + """A harness that keeps finding the *same* faster kernel every time. + + The gate compares a measurement with the variant; a tie inside the + comparison's noise band is ``NO_SIGNAL``, not a win, so a redrawn operator + set needs to be genuinely better than what the last round recorded. Tests + that want an unqualified win therefore improve on each round's own result. + """ + def medians(iteration, stage): + # Only the baseline round's operators are in the variant pool before + # the draw, so a draw's variant is still the original one. + return {iid: values["variant"] * share + for iid, values in ops.items()} + return medians + + +def _ops(n=10, variant=40.0): + return {f"op{i:02d}": {"baseline": 100.0 + i, "variant": variant + 4 * i} + for i in range(n)} + + +def test_failed_performance_gate_rolls_back_and_keeps_iterating(tmp_path, monkeypatch): + """Round 1 judges nothing; the performance gate is what rejects.""" + pool_path = _write_pool(tmp_path, _ops()) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + + # every operator is far slower than its variant + evaluator = FakeEvaluator({iid: 400.0 for iid in _ops()}) + _patch(monkeypatch, tmp_path, evaluator, prod) + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=1) + + it1 = cfg.exp_dir / "runs" / "iteration_001" / "input" + # round 1 is the baseline round: no gate is written at all + assert not (it1 / "performance_gate.json").exists() + assert not (it1 / "generalization_gate.json").exists() + first = json.loads((it1 / "decision.json").read_text()) + assert first["stage"] == "baseline" + assert first["verdict"] == "BASELINE_MEASURED" + assert (it1 / "baseline_round.json").is_file() + + # round 2 asks the same (still far too slow) evaluator for the performance + # gate, which is the first thing that can actually fail + it2 = cfg.exp_dir / "runs" / "iteration_002" / "input" + perf = json.loads((it2 / "performance_gate.json").read_text()) + assert perf["status"] == "FAIL" and perf["wins"] == [] + assert not (it2 / "generalization_gate.json").exists() + assert not (cfg.exp_dir / "pending_promotion.json").exists() + assert not (cfg.exp_dir / "pending_generalization.json").exists() + # the variant table did not move: a baseline round that beat nothing + table = json.loads((cfg.exp_dir / "variant_table.json").read_text()) + assert table["harness_version"] in {None, "seed"} + assert (it2 / "decision.json").is_file() + + +def test_a_retry_leaves_the_failed_attempt_beside_it_and_wins_the_resolution(): + """A retried question keeps both attempts, and the newest one is "the" one. + + The failed attempt's directory is evidence, so it must survive; the retry + gets its own directory because DKAO binds a child workspace's ``main`` + symlink to the kernel repo it was launched with (reusing the directory makes + the retry die instantly). Everything that reads "this question's child" + -- the drill-down, the kernel promotion path -- must land on the retry. + """ + import tempfile + + tmp = Path(tempfile.mkdtemp()) + ops = _ops() + pool_path = _write_pool(tmp, ops) + exp = tmp / "exp" + exp.mkdir() + operators = sorted(ops)[:2] + + def plant(attempt_dir: str, harness_name: str) -> None: + for iid in operators: + workspace = exp / "children" / attempt_dir / iid / "workspace" + workspace.mkdir(parents=True, exist_ok=True) + (workspace / "manifest.yaml").write_text( + f"harness_name: {harness_name}\n", encoding="utf-8") + + plant("iteration_001", "failed-attempt") + plant("iteration_001_a2", "retry-attempt") + + # both attempts are on disk, and both are attributable to the iteration + names = {d.name for d in R.attempt_dirs(exp / "children", 1)} + assert names == {"iteration_001", "iteration_001_a2"} + assert R.attempt_dirs(exp / "children", 2) == [] + + # the newest attempt is the question's child, for every reader + for iid in operators: + resolved = R.resolve_child_dir(exp / "children", 1, iid) + assert resolved == exp / "children" / "iteration_001_a2" / iid + assert R.child_dir_map(exp / "children", 1)[operators[0]] == ( + exp / "children" / "iteration_001_a2" / operators[0]) + + # a question that only ever ran once still resolves to its single attempt + only_once = exp / "children" / "iteration_001_a2" / "solo" / "workspace" + only_once.mkdir(parents=True) + assert R.resolve_child_dir(exp / "children", 1, "solo") == ( + exp / "children" / "iteration_001_a2" / "solo") + assert R.resolve_child_dir(exp / "children", 1, "never-ran") is None + + +def test_round1_judges_no_gate_and_wins_do_not_promote_a_harness(tmp_path, monkeypatch): + """A baseline round that beats every variant still promotes nothing.""" + ops = _ops() + pool_path = _write_pool(tmp_path, ops) + # exactly one round: the point here is round 1, and a second round is what + # would schedule the generalization retake + cfg = _cfg(tmp_path, pool_path, rounds="1") + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + evaluator = FakeEvaluator({iid: values["variant"] * 0.5 + for iid, values in ops.items()}) + _patch(monkeypatch, tmp_path, evaluator, prod) + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=1) + + it1 = cfg.exp_dir / "runs" / "iteration_001" / "input" + baseline = json.loads((it1 / "baseline_round.json").read_text()) + assert baseline["verdict"]["status"] == "PASS" + assert baseline["pool_promotion"]["promoted"], ( + "the baseline round's faster kernels must enter the variant pool") + # no gate verdict, no pending promotion: only the kernels moved + assert not (it1 / "performance_gate.json").exists() + assert not (cfg.exp_dir / "pending_promotion.json").exists() + assert yaml.safe_load((prod / "manifest.yaml").read_text())["harness_name"] == "seed" + # the pool now holds the baseline round's numbers + published = yaml.safe_load(pool_path.read_text(encoding="utf-8")) + entries = {i["id"]: i for i in published["instances"]} + for iid in baseline["pool_promotion"]["promoted"]: + assert entries[iid]["best_known_us"] == pytest.approx( + ops[iid]["variant"] * 0.5) + + +def test_the_retake_re_hands_the_paper_to_dkao_on_the_same_harness( + tmp_path, monkeypatch): + """The generalization gate is its own stage: same harness, A1 again.""" + ops = _ops() + pool_path = _write_pool(tmp_path, ops) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + evaluator = FakeEvaluator({iid: values["variant"] * 0.5 + for iid, values in ops.items()}) + _patch(monkeypatch, tmp_path, evaluator, prod) + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=4) + + paper = list(json.loads( + (cfg.exp_dir / "variant_table.json").read_text() + )["groups"]["generalization_ids"]) + assert len(paper) == 4 + + it1 = cfg.exp_dir / "runs" / "iteration_001" / "input" + it2 = cfg.exp_dir / "runs" / "iteration_002" / "input" + + # stage 1: the paper, judged by nothing + assert json.loads((it1 / "decision.json").read_text())["stage"] == "baseline" + assert sorted(json.loads((it1 / "benchmark" / "scored_ids.json").read_text())) \ + == sorted(paper) + + # stage 2: a fresh draw that never overlaps the paper, performance gate only + a2 = json.loads((it2 / "benchmark" / "scored_ids.json").read_text()) + assert set(a2).isdisjoint(paper) + perf = json.loads((it2 / "performance_gate.json").read_text()) + assert perf["status"] == "PASS" + assert not (it2 / "generalization_gate.json").exists(), ( + "the paper must not be re-measured in the same pass as the draw") + + # stage 3: the retake — the SAME harness bytes, the paper's operators, sent + # to DKAO again, and judged by the generalization gate + it3 = cfg.exp_dir / "runs" / "iteration_003" / "input" + retake = json.loads((it3 / "stage.json").read_text()) + assert retake["stage"] == "generalization" + assert retake["retake_of_iteration"] == 2, ( + "the retake belongs to the round that won the performance gate") + gen = json.loads((it3 / "generalization_gate.json").read_text()) + assert gen["status"] == "PASS" + assert sorted(gen["operators"]) == sorted(paper) + assert json.loads((it3 / "decision.json").read_text())["stage"] == \ + "generalization" + # a fresh DKAO launch, in its own children directory, not the first pass's + assert (cfg.exp_dir / "children" / "iteration_003_generalization").is_dir() + assert (cfg.exp_dir / "children" / "iteration_002").is_dir(), ( + "the two stages never share a child directory") + assert (cfg.exp_dir / "pending_promotion.json").is_file() + run = json.loads((cfg.state_dir / "run.json").read_text()) + assert run["final_status"] == "awaiting_approval" + assert run["rounds_used"] == pytest.approx(1.5), ( + "the baseline round is free; a performance pass is 1.0 and the " + "generalization retake half a round") + + +def test_passing_rounds_wait_for_approval_then_promote(tmp_path, monkeypatch): + ops = _ops() + pool_path = _write_pool(tmp_path, ops) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + + # half the variant on every operator: a wide win on both gates + evaluator = FakeEvaluator({iid: values["variant"] * 0.5 + for iid, values in ops.items()}) + _patch(monkeypatch, tmp_path, evaluator, prod) + + report = pl.run_experiment(cfg, start_iteration=1, iterations_to_run=4) + assert report.is_file() + + it2 = cfg.exp_dir / "runs" / "iteration_002" / "input" + perf = json.loads((it2 / "performance_gate.json").read_text()) + it3 = cfg.exp_dir / "runs" / "iteration_003" / "input" + gen = json.loads((it3 / "generalization_gate.json").read_text()) + assert perf["status"] == "PASS" and gen["status"] == "PASS" + assert set(perf["operators"]).isdisjoint(gen["operators"]), ( + "the performance draw and the paper are different operator sets") + pending = json.loads((cfg.exp_dir / "pending_promotion.json").read_text()) + assert pending["version"] == "h-2" + run = json.loads((cfg.state_dir / "run.json").read_text()) + assert run["final_status"] == "awaiting_approval" + # nothing was written to production yet + assert yaml.safe_load((prod / "manifest.yaml").read_text())["harness_name"] == "seed" + + # a human approves: the harness lands in production and variants move + result = promo.approve_promotion(cfg, cfg.exp_dir, approved_by="tester") + assert result["ok"] is True + assert result["version"] == "h-2" + assert not (cfg.exp_dir / "pending_promotion.json").exists() + assert (prod / "manifest.yaml").is_file() + assert (prod / "gates.yaml").is_file() + promoted = json.loads((cfg.exp_dir / "promoted_harness.json").read_text()) + assert promoted["approved_by"] == "tester" + assert Path(promoted["previous_backup"]).is_dir() + + table = json.loads((cfg.exp_dir / "variant_table.json").read_text()) + assert table["harness_version"] == "h-2" + # the reference never drifts away from production: after approval, every + # operator's recorded variant is the pool's own best-known kernel + published = {i["id"]: i["best_known_us"] + for i in yaml.safe_load(pool_path.read_text())["instances"]} + for iid, row in table["operators"].items(): + assert row["median_us"] == pytest.approx(published[iid]), ( + f"{iid}: the experiment's reference must match the pool's best-known") + assert set(table["groups"]["generalization_ids"]) == set(gen["operators"]) + + +def test_denied_candidate_is_discarded_without_touching_production( + tmp_path, monkeypatch): + ops = _ops() + pool_path = _write_pool(tmp_path, ops) + cfg = _cfg(tmp_path, pool_path, max_rounds="9") + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + evaluator = FakeEvaluator({iid: values["variant"] * 0.5 + for iid, values in ops.items()}) + _patch(monkeypatch, tmp_path, evaluator, prod) + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=4) + assert (cfg.exp_dir / "pending_promotion.json").is_file() + + denied = promo.deny_promotion(cfg, cfg.exp_dir, reason="not convinced") + assert denied["ok"] is True + assert not (cfg.exp_dir / "pending_promotion.json").exists() + assert not (cfg.exp_dir / "promoted_harness.json").exists() + assert yaml.safe_load((prod / "manifest.yaml").read_text())["harness_name"] == "seed" + table = json.loads((cfg.exp_dir / "variant_table.json").read_text()) + assert table["harness_version"] in {None, "seed"} + + +def test_rollback_restores_the_previous_production_tree(tmp_path, monkeypatch): + pool_path = _write_pool(tmp_path, _ops()) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + _patch(monkeypatch, tmp_path, FakeEvaluator({}), prod) + + # a promoted version replaces production + snapshot = tmp_path / "candidate" + snapshot.mkdir() + (snapshot / "manifest.yaml").write_text("harness_name: evolved\n", encoding="utf-8") + promo.promote_harness(cfg.exp_dir, { + "iteration": 3, "version": "h-3", "revision": "rev-candidate", + "snapshot_dir": str(snapshot), + "performance_gate": {"status": "PASS", "wins": ["a"]}, + "generalization_gate": {"status": "PASS", "wins": ["a"], "operators": ["a"]}, + }, approved_by="tester") + assert yaml.safe_load((prod / "manifest.yaml").read_text() + )["harness_name"] == "evolved" + + back = promo.rollback_harness(cfg.exp_dir, to="last-good") + assert back["ok"] is True + assert yaml.safe_load((prod / "manifest.yaml").read_text() + )["harness_name"] == "seed" + record = json.loads((cfg.exp_dir / "promoted_harness.json").read_text()) + assert record["version"].startswith("rollback:") + + +def test_generalization_paper_survives_a_failed_first_round(tmp_path, monkeypatch): + """Round 1 fixes the paper even when nothing it measures is an improvement.""" + pool_path = _write_pool(tmp_path, _ops()) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + evaluator = FakeEvaluator({iid: 900.0 for iid in _ops()}) # all much slower + _patch(monkeypatch, tmp_path, evaluator, prod) + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=1) + + table = json.loads((cfg.exp_dir / "variant_table.json").read_text()) + paper = table["groups"]["generalization_ids"] + assert len(paper) == 4 + # round_questions.json is rewritten by every pass, so the baseline round's + # own copy is what says the paper was defined there + it1 = cfg.exp_dir / "runs" / "iteration_001" / "input" + questions = json.loads((it1 / "stage.json").read_text()) + assert questions["stage"] == "baseline" + assert table["groups"]["generalization_defined_at"] == 1 + assert len(table["groups"]["generalization_ids"]) == 4 + # nothing beat its variant, so the pool keeps what it had + it1 = cfg.exp_dir / "runs" / "iteration_001" / "input" + baseline = json.loads((it1 / "baseline_round.json").read_text()) + assert (baseline["pool_promotion"] or {}).get("promoted", []) == [] + + +def test_the_baseline_round_stops_the_run_when_the_paper_cannot_be_measured( + tmp_path, monkeypatch): + """Round 1 judges no gate, but it owes a usable number for every operator.""" + pool_path = _write_pool(tmp_path, _ops()) + cfg = _cfg(tmp_path, pool_path) + _patch(monkeypatch, tmp_path, FakeEvaluator({}), tmp_path / "production") + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=4) + + run = json.loads((cfg.state_dir / "run.json").read_text()) + assert run["final_status"] == "baseline_round_failed" + # stopped at round 1: no later round ran, nothing was parked for approval + assert not (cfg.exp_dir / "runs" / "iteration_002" / "input" + / "performance_gate.json").exists() + assert not (cfg.exp_dir / "pending_promotion.json").exists() + events = [json.loads(l)["type"] for l in + (cfg.state_dir / "timeline.jsonl").read_text().splitlines() + if l.strip()] + assert "baseline_round_failed" in events + + +def test_generation_records_carry_their_own_revision(tmp_path, monkeypatch): + """A generation's record must name the revision it measured.""" + from ..orchestrator.champion_selection import load_generations + + pool_path = _write_pool(tmp_path, _ops()) + cfg = _cfg(tmp_path, pool_path, round_questions="legacy") # exercise the legacy path + evaluator = FakeEvaluator({}) + _patch(monkeypatch, tmp_path, evaluator, tmp_path / "production") + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=1) + generations = load_generations(cfg.exp_dir) + assert generations, "a generation must be recorded" + assert generations[0].workspace_revision, ( + "workspace_revision must name this round's commit, not be empty") + + +def test_approval_promotes_the_accepted_rounds_kernels_into_the_pool( + tmp_path, monkeypatch): + """Only the accepted round's operators reach production — as best-knowns. + + Approving must (a) lower the pool's best_known where the accepted retake beat + it, (b) append an auditable history row naming the harness version, and (c) + leave the pool recoverable via a backup. The baseline round writes the pool + on its own (kernel fact), so the retake has to be *better* than that to show + up here. + """ + ops = _ops() + pool_path = _write_pool(tmp_path, ops) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + + def medians(iteration, stage): + share = 0.6 if stage == "generalization" else 0.9 + return {iid: values["variant"] * share for iid, values in ops.items()} + + evaluator = FakeEvaluator(medians) + _patch(monkeypatch, tmp_path, evaluator, prod) + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=4) + assert (cfg.exp_dir / "pending_promotion.json").is_file() + result = promo.approve_promotion(cfg, cfg.exp_dir, approved_by="tester") + assert result["ok"] is True + + kernel_update = result["kernel_update"] + assert kernel_update["updated"], "the accepted round's kernels must be published" + assert kernel_update["pool"] == str(pool_path) + assert Path(kernel_update["backup"]).is_file() + + accepted = set(json.loads( + (cfg.exp_dir / "variant_table.json").read_text() + )["groups"]["generalization_ids"]) + published = yaml.safe_load(pool_path.read_text(encoding="utf-8")) + entries = {i["id"]: i for i in published["instances"]} + for change in kernel_update["updated"]: + iid = change["operator_id"] + assert iid in accepted, "only the accepted round's operators are published" + assert entries[iid]["best_known_us"] == pytest.approx( + ops[iid]["variant"] * 0.6) + history = entries[iid]["history"] + assert history[-1]["accepted"] is True + assert history[-1]["harness_version"] == "h-2" + # the pool is still the original operator-shaped document + assert len(published["instances"]) == len(ops) + + +def test_a_slower_round_never_degrades_the_pool(tmp_path, monkeypatch): + """Best-known wins: nothing a round measures can make a pool entry worse.""" + ops = _ops(n=8) + pool_path = _write_pool(tmp_path, ops) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + + def medians(iteration, stage): + # A baseline round that does well, then rounds that measure worse + share = 0.5 if stage == "baseline" else 0.95 + return {iid: values["variant"] * share for iid, values in ops.items()} + + _patch(monkeypatch, tmp_path, FakeEvaluator(medians), prod) + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=4) + + published = {i["id"]: i["best_known_us"] + for i in yaml.safe_load(pool_path.read_text())["instances"]} + for iid, row in ops.items(): + assert published[iid] <= row["variant"], ( + "a pool entry may only ever improve") + + +def test_rejected_round_never_promotes_kernels_in_variant_mode(tmp_path, monkeypatch): + """Only the accepted round's optimizations reach production. + + The legacy path promoted each round's kernels as soon as it was evaluated; + under the variant protocol a rejected round must leave nothing behind. + """ + ops = _ops() + pool_path = _write_pool(tmp_path, ops) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + _patch(monkeypatch, tmp_path, FakeEvaluator({}), prod) + + promoted = {"calls": 0} + + def _fake_promote(*a, **k): + promoted["calls"] += 1 + return None + + monkeypatch.setattr(pl, "_promote_iteration", _fake_promote) + # the evaluator reports nothing, so every operator fails both gates + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=1) # round 1 fails + assert promoted["calls"] == 0 + assert not (cfg.exp_dir / "pending_promotion.json").exists() + assert not (cfg.exp_dir / "promoted_harness.json").exists() + + +def test_a_run_without_a_usable_pool_stops_loudly(tmp_path, monkeypatch): + """No operators to measure must not burn the whole round budget.""" + pool_path = tmp_path / "empty_pool.yaml" + pool_path.write_text(yaml.safe_dump({"schema_version": 1, "instances": []}), + encoding="utf-8") + cfg = _cfg(tmp_path, pool_path) + _patch(monkeypatch, tmp_path, FakeEvaluator({}), tmp_path / "production") + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=1) + run = json.loads((cfg.state_dir / "run.json").read_text()) + assert run["final_status"] == "no_question_pool" + events = [json.loads(l)["type"] for l in + (cfg.state_dir / "timeline.jsonl").read_text().splitlines() if l.strip()] + assert "variant_pool_empty" in events + + +def test_mechanism_gate_is_recorded_and_does_not_veto_a_win(tmp_path, monkeypatch): + """Lenient, but never absent: the mechanism verdict is graded and recorded. + + A contradicted mechanism ("the planner was off, so this planner policy + change cannot have caused the win") must show up in the decision, while the + counter gates keep deciding. Round 1 is the baseline round and has no gate + to grade; the performance round is where a declared change is first judged. + """ + ops = _ops() + pool_path = _write_pool(tmp_path, ops) + cfg = _cfg(tmp_path, pool_path) + prod = tmp_path / "production" + prod.mkdir() + (prod / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + evaluator = FakeEvaluator({iid: values["variant"] * 0.5 + for iid, values in ops.items()}) + _patch(monkeypatch, tmp_path, evaluator, prod) + + # a change manifest whose mechanism cannot have fired: the planner was off + manifest_dir = cfg.exp_dir / "runs" / "iteration_001" / "evolve" + manifest_dir.mkdir(parents=True, exist_ok=True) + (manifest_dir / "change_manifest.json").write_text(json.dumps({ + "schema_version": 1, "iteration": 1, + "changes": [{ + "id": "chg-1", + "description": "point the exhausted-budget phase at the bounded plan", + "files": ["planner_policy.yaml"], + "mechanism_signature": ["planner emits architecture_explore"], + "mechanism_check": {"kind": "planner_plan_ids", + "expect_any": ["architecture_explore"]}, + }], + }), encoding="utf-8") + + pl.run_experiment(cfg, start_iteration=1, iterations_to_run=4) + + it1 = cfg.exp_dir / "runs" / "iteration_001" / "input" + first = json.loads((it1 / "decision.json").read_text()) + assert first["stage"] == "baseline" + assert first["mechanism_gate"] == {}, "round 1 judges no gate" + assert first["performance_gate"] is None + + # round 2 grades that manifest against round 2's children while it judges + # the performance gate + it2 = cfg.exp_dir / "runs" / "iteration_002" / "input" + second = json.loads((it2 / "decision.json").read_text()) + assert second["stage"] == "performance" + mechanism = second["mechanism_gate"] + assert mechanism["status"] in {"MISS", "PARTIAL", "UNOBSERVED", "HIT"} + assert "detail" in mechanism + assert second["performance_gate"]["status"] == "PASS" + # a passing performance gate promotes nothing by itself: it hands the same + # harness to DKAO again with the frozen paper + frozen = json.loads((cfg.exp_dir / "pending_generalization.json").read_text()) + assert set(frozen["operator_ids"]) == set( + json.loads((cfg.exp_dir / "variant_table.json").read_text() + )["groups"]["generalization_ids"]) diff --git a/metainfer/tasks/harness_evolve/tests/test_promotion_pipeline.py b/metainfer/tasks/harness_evolve/tests/test_promotion_pipeline.py new file mode 100644 index 00000000..820888cd --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_promotion_pipeline.py @@ -0,0 +1,129 @@ +"""HE pipeline promotion step: better kernels become DKAO variants.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from metainfer.tasks.dcu_kernel_auto_opt.orchestrator import variant_store as vs +from metainfer.tasks.harness_evolve.orchestrator.config import ( + ExperimentConfig, InstanceSpec, +) +from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + _promote_iteration, +) + +SHAPE = "hy3_tp4_o_proj_m16" + + +@pytest.fixture() +def variant_root(tmp_path, monkeypatch): + root = tmp_path / "variant" + root.mkdir(parents=True, exist_ok=True) + monkeypatch.setattr(vs, "variant_root", lambda: root) + return root + + +def _cfg(tmp_path: Path, **answers) -> ExperimentConfig: + base = {"promote_kernels": "true", + "promote_min_improvement_percent": "3"} + base.update(answers) + return ExperimentConfig( + task_id="he-promo", state_dir=tmp_path / "state", + workspace_dir=tmp_path / "exp", suite=[], answers=base, + execution_mode="dkao-cli", pool_source="pool.yaml", + per_round_budget=1, max_iterations=1, + ) + + +def _inst() -> InstanceSpec: + return InstanceSpec(id=SHAPE, shape={ + "model": "hy3-dsh-tp4-m16-2-368c654c", "tp_size": 4, + "operator": "o_proj", "M": 16, "N": 4096, "K": 2048, + }, budget_rounds=1, pass_median_us_le=20.0) + + +def _child_workspace(exp: Path, median_us: float) -> Path: + leaf = (exp / "children" / "iteration_001" / SHAPE / "workspace" + / "workers" / "worker_0" / "accepted" / SHAPE) + leaf.mkdir(parents=True, exist_ok=True) + (leaf / "kernel.hip").write_text("// promoted kernel\n", encoding="utf-8") + (leaf / "manifest.json").write_text(json.dumps({ + "commit": "abc123", + "shape": {"M": 16, "N": 4096, "K": 2048}, + "metrics": {"median_us": median_us, "p90_us": median_us * 1.01}, + }), encoding="utf-8") + return leaf + + +def _seed_variant(root: Path, median_us: float) -> Path: + meta = vs.derive_variant_meta( + {"operator": "Quantized GEMM", "dtype": "INT8 W8A8", + "model": "Hy3 (Hunyuan 3)"}, SHAPE) + target = vs.variant_path(meta) + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(vs.section_header(meta, commit="old", metrics={ + "median_us": median_us}) + "// old\n// @@end\n", encoding="utf-8") + return target + + +def test_promotes_faster_child_and_logs(tmp_path, variant_root): + cfg = _cfg(tmp_path) + exp = cfg.exp_dir + _child_workspace(exp, 8.49) + _seed_variant(variant_root, 9.17) # 7.4% faster -> promote + results = {SHAPE: {"correctness_ok": True, "passed": True, + "median_us": 8.49}} + + payload = _promote_iteration(cfg, exp, 1, results, [_inst()]) + + assert payload and payload["promoted"] == [SHAPE] + out = payload["results"][0] + assert out["action"] == "updated" + assert out["improvement_percent"] == pytest.approx(7.42, abs=0.05) + assert (exp / "runs" / "iteration_001" / "promotion.json").is_file() + log = (exp / "promotion_log.jsonl").read_text(encoding="utf-8").strip() + assert json.loads(log)["shape"] == SHAPE + # the variant leaf now carries the new kernel + assert "promoted kernel" in Path(out["path"]).read_text(encoding="utf-8") + + +def test_skips_below_threshold_and_ignores_missing_workspace(tmp_path, variant_root): + cfg = _cfg(tmp_path) + exp = cfg.exp_dir + _child_workspace(exp, 9.10) # only 0.8% faster than 9.17 + _seed_variant(variant_root, 9.17) + other = InstanceSpec(id="hy3_tp4_qkv_proj_m16", shape={"M": 16}, budget_rounds=1) + results = {SHAPE: {"correctness_ok": True, "median_us": 9.10}, + "hy3_tp4_qkv_proj_m16": {"correctness_ok": True, "median_us": 1.0}} + + payload = _promote_iteration(cfg, exp, 1, results, [_inst(), other]) + by_shape = {r["shape"]: r for r in payload["results"]} + assert payload["promoted"] == [] + assert by_shape[SHAPE]["action"] == "skipped" + assert "required 3.00%" in by_shape[SHAPE]["reason"] + # a question with no child directory at all reports "no kernel", the same + # as a child directory that produced none + assert by_shape["hy3_tp4_qkv_proj_m16"]["action"] == "no_kernel" + assert "no accepted kernel" in by_shape["hy3_tp4_qkv_proj_m16"]["reason"] + + +def test_disabled_by_default_switch_returns_none(tmp_path, variant_root): + cfg = _cfg(tmp_path, promote_kernels="false") + exp = cfg.exp_dir + _child_workspace(exp, 1.0) + assert _promote_iteration(cfg, exp, 1, {}, [_inst()]) is None + assert not (exp / "runs" / "iteration_001" / "promotion.json").exists() + + +def test_failed_correctness_is_never_promoted(tmp_path, variant_root): + cfg = _cfg(tmp_path) + exp = cfg.exp_dir + _child_workspace(exp, 1.0) + payload = _promote_iteration( + cfg, exp, 1, + {SHAPE: {"correctness_ok": False, "median_us": 1.0}}, [_inst()]) + assert payload["promoted"] == [] + assert payload["results"][0]["action"] == "skipped" diff --git a/metainfer/tasks/harness_evolve/tests/test_promotion_routes.py b/metainfer/tasks/harness_evolve/tests/test_promotion_routes.py new file mode 100644 index 00000000..29b4b9a9 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_promotion_routes.py @@ -0,0 +1,99 @@ +"""WebUI approval routes: the human gate in front of production DKAO.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from metainfer.tasks.harness_evolve.orchestrator import promotion as promo +from metainfer.tasks.harness_evolve.server.routes import ( + _approve_with_config, _deny_with_config, +) + + +def _task_entry(state_dir: Path, workspace_dir: Path): + class Entry: + pass + + entry = Entry() + entry.id = "promo-task" + entry.state_dir = str(state_dir) + entry.workspace_dir = str(workspace_dir) + return entry + + +def _setup(tmp_path, monkeypatch): + """An experiment parked on a pending promotion, plus a fake production dir.""" + state_dir = tmp_path / "state" + workspace = tmp_path / "ws" + state_dir.mkdir(parents=True) + workspace.mkdir(parents=True) + (state_dir / "requirements.json").write_text(json.dumps({ + "task_id": "promo-task", + "answers": {"suite_yaml": "instances: []", "execution_mode": "dry-run"}, + }), encoding="utf-8") + (state_dir / "run.json").write_text(json.dumps({"task_id": "promo-task"}), + encoding="utf-8") + + snapshot = workspace / "runs" / "iteration_002" / "input" / "workspace" + snapshot.mkdir(parents=True) + (snapshot / "manifest.yaml").write_text("harness_name: evolved\n", encoding="utf-8") + (snapshot / "gates.yaml").write_text("component: gates\n", encoding="utf-8") + + production = tmp_path / "production" + production.mkdir() + (production / "manifest.yaml").write_text("harness_name: seed\n", encoding="utf-8") + monkeypatch.setattr(promo, "_production_dir", lambda: production) + + promo.write_pending_promotion( + workspace, iteration=2, revision="rev-2", snapshot_dir=str(snapshot), + performance_gate={"status": "PASS", "wins": ["a", "b", "c"], + "operators": ["a", "b", "c", "d"]}, + generalization_gate={"status": "PASS", "wins": ["a", "b"], + "operators": ["a", "b", "c", "d"]}, + results={"a": {"median_us": 1.0, "correctness_ok": True}}) + return _task_entry(state_dir, workspace), state_dir, workspace, production + + +def test_approve_route_publishes_and_clears_the_pending_candidate(tmp_path, monkeypatch): + entry, state_dir, workspace, production = _setup(tmp_path, monkeypatch) + + result = _approve_with_config(entry, workspace, "ui-tester") + assert result["ok"] is True + assert result["version"] == "h-2" + # pending cleared, production replaced, record written + assert not (workspace / "pending_promotion.json").exists() + assert "evolved" in (production / "manifest.yaml").read_text(encoding="utf-8") + record = json.loads((workspace / "promoted_harness.json").read_text(encoding="utf-8")) + assert record["approved_by"] == "ui-tester" + assert Path(record["previous_backup"]).is_dir() + # the run is finished from the shell's point of view + run = json.loads((state_dir / "run.json").read_text(encoding="utf-8")) + assert run["final_status"] == "promoted:h-2" + + +def test_deny_route_leaves_production_alone_and_clears_pending(tmp_path, monkeypatch): + entry, state_dir, workspace, production = _setup(tmp_path, monkeypatch) + + result = _deny_with_config(entry, workspace, "not convinced") + assert result["ok"] is True + assert not (workspace / "pending_promotion.json").exists() + assert not (workspace / "promoted_harness.json").exists() + assert "seed" in (production / "manifest.yaml").read_text(encoding="utf-8") + log = (workspace / "promotion_log.jsonl").read_text(encoding="utf-8") + assert "denied" in log and "not convinced" in log + + +def test_approve_route_refuses_when_nothing_is_pending(tmp_path, monkeypatch): + state_dir = tmp_path / "state" + workspace = tmp_path / "ws" + state_dir.mkdir(parents=True) + workspace.mkdir(parents=True) + (state_dir / "requirements.json").write_text(json.dumps({ + "task_id": "promo-task", + "answers": {"suite_yaml": "instances: []", "execution_mode": "dry-run"}, + }), encoding="utf-8") + monkeypatch.setattr(promo, "_production_dir", lambda: tmp_path / "nope") + + result = _approve_with_config(_task_entry(state_dir, workspace), workspace, "ui") + assert result["ok"] is False and "no pending promotion" in result["errors"][0] diff --git a/metainfer/tasks/harness_evolve/tests/test_publish.py b/metainfer/tasks/harness_evolve/tests/test_publish.py new file mode 100644 index 00000000..ea26f41f --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_publish.py @@ -0,0 +1,92 @@ +"""Tests for the explicit publish gate.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import yaml + +from ..orchestrator.attribution import copy_tree, save_json +from ..orchestrator.publish import ( + check_publishable, + publish_harness, + publish_kernels, +) + +_DCU_SEED = Path("/root/zth_agent/MetaInfer/metainfer/tasks/" + "dcu_kernel_auto_opt/harness_default") + + +def _fake_experiment(tmp_path, *, verdict="PROMOTE", held="PASS"): + exp = tmp_path / "exp" + snapshot = exp / "runs" / "iteration_002" / "input" / "workspace" + copy_tree(_DCU_SEED, snapshot) + save_json(exp / "best_ever.json", { + "iteration": 2, "pass_rate": 0.9, "snapshot_dir": str(snapshot), + "workspace_revision": "rev-2", + }) + save_json(exp / "runs" / "iteration_002" / "input" / "decision.json", { + "iteration": 2, "verdict": verdict, "action": "SET_CHAMPION", + "heldout_gate": {"status": held}, + }) + return exp + + +def test_publish_gate_rejects_non_promotable(tmp_path): + exp = _fake_experiment(tmp_path, verdict="REJECT") + gate = check_publishable(exp) + assert gate["ok"] is False + assert any("not promotable" in e for e in gate["errors"]) + + +def test_publish_gate_rejects_overfit(tmp_path): + exp = _fake_experiment(tmp_path, verdict="PROMOTE", held="FAIL") + gate = check_publishable(exp) + assert gate["ok"] is False + assert any("held-out" in e for e in gate["errors"]) + + +def test_publish_harness_backs_up_and_replaces_target(tmp_path): + exp = _fake_experiment(tmp_path) + gate = check_publishable(exp) + assert gate["ok"] is True and gate["verdict"] == "PROMOTE" + + target = tmp_path / "harness_default" + copy_tree(_DCU_SEED, target) + marker = target / "old_extra.yaml" + marker.write_text("old: true", encoding="utf-8") + + result = publish_harness(exp, target=target) + assert result["ok"] and result["published"] + assert not marker.exists() # replace semantics removed stale file + assert (target / "planner_policy.yaml").is_file() + backups = list(tmp_path.glob("harness_default.pre-publish-*")) + assert len(backups) == 1 + log = (exp / "publish_log.jsonl").read_text().strip().splitlines() + assert len(log) == 1 and "harness" in log[0] + + +def test_publish_kernels_updates_pool_with_backup(tmp_path): + exp = tmp_path / "exp" + child = exp / "children" / "iter001" / "c" / "workspace" + child.mkdir(parents=True) + report = { + "status": "success", + "config": {"shapes": [{"id": "s1"}]}, + "final_validation": {"s1": {"median_us": 30.0, "p90_us": 31.0}}, + } + (child / "final_report.json").write_text(json.dumps(report), + encoding="utf-8") + pool_path = tmp_path / "pool.yaml" + pool_path.write_text(yaml.safe_dump({"instances": [{ + "id": "s1", "contract": {"M": 16}, "family": "f", + "baseline_us": 100.0, "best_known_us": 50.0, "history": [], + }]}), encoding="utf-8") + + result = publish_kernels(exp, pool_path=pool_path) + assert result["ok"] and result["published"] + assert result["updated"] == ["s1"] + data = yaml.safe_load(pool_path.read_text(encoding="utf-8")) + assert data["instances"][0]["best_known_us"] == 30.0 + assert len(list(tmp_path.glob("pool.yaml.pre-publish-*"))) == 1 diff --git a/metainfer/tasks/harness_evolve/tests/test_question_roles.py b/metainfer/tasks/harness_evolve/tests/test_question_roles.py new file mode 100644 index 00000000..284d337f --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_question_roles.py @@ -0,0 +1,187 @@ +"""Role-based question selection: roles, family quota, overlap, rotation.""" + +from __future__ import annotations + +from metainfer.tasks.harness_evolve.orchestrator.question_roles import ( + PURPOSE_PROBE, PURPOSE_REGRESSION, PURPOSE_REPAIR, main_purposes, + overlap_ratio, probe_ids, select_questions, +) + + +class _Inst: + def __init__(self, iid: str, family: str) -> None: + self.id = iid + self.family = family + + +class _Pool: + def __init__(self, rows) -> None: + self.instances = {i: _Inst(i, f) for i, f in rows} + + def all_ids(self): + return list(self.instances) + + def get(self, iid): + return self.instances[iid] + + def historical_ids(self): + return list(self.instances) + + +def _pool(): + return _Pool([ + ("a1", "famA"), ("a2", "famA"), ("a3", "famA"), + ("b1", "famB"), ("b2", "famB"), + ("c1", "famC"), ("c2", "famC"), + ("d1", "famD"), ("d2", "famD"), + ]) + + +def _big_pool(): + """Six families: enough for a genuinely cross-family probe at budget 4.""" + return _Pool([ + ("a1", "famA"), ("a2", "famA"), ("a3", "famA"), + ("b1", "famB"), ("b2", "famB"), ("b3", "famB"), + ("c1", "famC"), ("c2", "famC"), ("c3", "famC"), + ("d1", "famD"), ("d2", "famD"), ("d3", "famD"), + ("e1", "famE"), ("e2", "famE"), ("e3", "famE"), + ("f1", "famF"), ("f2", "famF"), ("f3", "famF"), + ]) + + +PREV = ["a1", "b1", "c1", "d1"] +PREV_RESULTS = { + "a1": {"passed": True}, "b1": {"passed": True}, + "c1": {"passed": False}, "d1": {"passed": True}, +} + + +def test_roles_regression_repair_and_probe(): + pool = _big_pool() + sel = select_questions(pool=pool, budget=4, prev_selected=PREV, + prev_results=PREV_RESULTS) + assert len(sel.selected) == 4 + roles = [sel.purposes[i] for i in sel.selected] + assert roles.count(PURPOSE_REGRESSION) >= 2 # anchors from passed set + assert PURPOSE_REPAIR in roles # the failed question + assert PURPOSE_PROBE in roles # cross-family probe + assert "c1" in sel.selected + # the probe never shares a family with the rest of the round + probe = probe_ids(sel.purposes)[0] + others = {pool.get(i).family for i in sel.selected if i != probe} + assert pool.get(probe).family not in others + + +def test_family_quota_and_minimum_coverage(): + pool = _big_pool() + sel = select_questions(pool=pool, budget=4, prev_selected=PREV, + prev_results=PREV_RESULTS, min_families=3, + max_per_family=2) + families = [pool.get(i).family for i in sel.selected] + assert len(set(families)) >= 3 + assert max(families.count(f) for f in set(families)) <= 2 + + +def test_overlap_guardrail_holds(): + sel = select_questions(pool=_big_pool(), budget=4, prev_selected=PREV, + prev_results=PREV_RESULTS) + assert overlap_ratio(sel.selected, PREV) >= 0.6 + + +def test_small_pool_degrades_to_the_freshest_question(): + """With four families and four slots no family is left for a probe.""" + sel = select_questions(pool=_pool(), budget=4, prev_selected=PREV, + prev_results=PREV_RESULTS) + assert len(sel.selected) == 4 + assert PURPOSE_PROBE in sel.purposes.values() + assert any("probe" in n for n in sel.notes) + + +def test_agent_picks_are_honoured_when_they_fit_a_role(): + class Plan: + selected = ["b1", "a1", "c1", "d2"] + changes: list = [] + + sel = select_questions(pool=_big_pool(), budget=4, prev_selected=PREV, + prev_results=PREV_RESULTS, plan=Plan()) + # passed agent picks become regression anchors, the failed one repair + assert sel.purposes.get("b1") == PURPOSE_REGRESSION + assert sel.purposes.get("c1") == PURPOSE_REPAIR + + +def test_agent_picks_are_replaced_when_family_quota_breaks(): + class Plan: + selected = ["a1", "a2", "a3", "b1"] # three from famA + changes: list = [] + + sel = select_questions(pool=_big_pool(), budget=4, prev_selected=PREV, + prev_results=PREV_RESULTS, plan=Plan(), + max_per_family=2, min_families=3) + pool = _big_pool() + families = [pool.get(i).family for i in sel.selected] + assert families.count("famA") <= 2 + assert len(set(families)) >= 3 + + +def test_rotation_prefers_least_recently_used_for_the_probe(): + sel = select_questions(pool=_pool(), budget=4, prev_selected=PREV, + prev_results=PREV_RESULTS, + rotation={"a1": 9, "a2": 9, "a3": 9, "b1": 9, + "b2": 9, "c1": 9, "c2": 9, "d1": 9, + "d2": 0}) + assert "d2" in sel.selected + + +def test_first_round_without_history_still_covers_families(): + pool = _big_pool() + sel = select_questions(pool=pool, budget=4) + assert len(sel.selected) == 4 + families = {pool.get(i).family for i in sel.selected} + assert len(families) >= 3 + assert len(main_purposes(sel.purposes)) >= 3 + + +def test_scope_ids_bias_the_regression_anchors(): + class Plan: + selected: list = [] + changes = [{"scope": {"expected_improve": ["b2"], "at_risk": ["c2"]}}] + + sel = select_questions(pool=_big_pool(), budget=4, prev_selected=PREV, + prev_results=PREV_RESULTS, plan=Plan()) + assert "b2" in sel.selected or "c2" in sel.selected + + +class _MInst: + def __init__(self, iid: str, family: str, m: int) -> None: + self.id = iid + self.family = family + self.M = m + + +class _MPool: + def __init__(self, rows) -> None: + self.instances = {i: _MInst(i, f, m) for i, f, m in rows} + + def all_ids(self): + return list(self.instances) + + def get(self, iid): + return self.instances[iid] + + def historical_ids(self): + return list(self.instances) + + +def test_fresh_round_covers_both_regimes(): + """decode (M<=32) and prefill need different kernels: both must appear.""" + pool = _MPool([ + ("p1", "prefill__o_proj", 4096), ("p2", "prefill__qkv_proj", 4096), + ("p3", "prefill__shared_down_proj", 4096), + ("p4", "prefill__fused_qkv_a_proj", 4096), + ("d1", "decode__o_proj", 16), ("d2", "decode__qkv_proj", 16), + ("d3", "decode__shared_down_proj", 16), + ]) + sel = select_questions(pool=pool, budget=4) + regimes = {pool.get(i).M <= 32 and "decode" or "prefill" for i in sel.selected} + assert regimes == {"decode", "prefill"}, sel.selected + assert len(sel.families) >= 3 diff --git a/metainfer/tasks/harness_evolve/tests/test_real_adapters.py b/metainfer/tasks/harness_evolve/tests/test_real_adapters.py new file mode 100644 index 00000000..50699036 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_real_adapters.py @@ -0,0 +1,466 @@ +"""Unit tests for real evaluator/evolver surfaces (no GPU/LLM calls).""" + +from __future__ import annotations + +import json +import threading +import time +from pathlib import Path +from typing import List + +from ..orchestrator.adapters.eval import ( + DkaoCliEvaluator, _child_env, _normalize_report, + _normalize_workspace_result, _worker_rounds, +) +from ..orchestrator.config import ExperimentConfig, InstanceSpec +from ..orchestrator.evolve import AgentEvolver, _last_result + + +def _inst(): + return InstanceSpec( + id="hy3_tp8_qkv_proj_m16", + shape={"model": "hy3", "tp_size": 8, "operator": "qkv_proj", + "M": 16, "N": 1280, "K": 4096}, + budget_rounds=4, + pass_median_us_le=60.0, + ) + + +def _cfg(tmp_path): + # HE manages no GPU occupancy: the device is the fixed index % 4 rotation and + # there is no gate wait, lease or broker poll to configure. + return ExperimentConfig( + task_id="ahe-parent", state_dir=tmp_path / "state", + workspace_dir=tmp_path / "workspace", suite=[_inst()], + execution_mode="dkao-cli", agent_framework="dsh", + answers={"ahe_repo_root": str(tmp_path / "ahe-repos")}, + ) + + +def test_dkao_requirements_include_parent_and_shape(tmp_path): + ev = DkaoCliEvaluator() + req = ev._build_dkao_requirements( + _cfg(tmp_path), _inst(), 2, "child-id", "repo-name", 3, + tmp_path / "harness", + ) + assert req["parent_ahe_task_id"] == "ahe-parent" + assert req["ahe_iteration"] == 2 + assert req["target_repo_path"] == "repo-name" + shape_cfg = __import__("yaml").safe_load(req["shape_config"]) + # DKAO manual mode requires worker_N <-> GPU N; a single child on GPU 3 + # must therefore be assigned to worker_3 (not worker_0 with gpu=3). + assert shape_cfg["assignments"] == { + "worker_3": {"gpu": 3, "shapes": ["hy3_tp8_qkv_proj_m16"]} + } + assert shape_cfg["shapes"][0]["M"] == 16 + + +def test_dkao_child_uses_flash_41_model_label(tmp_path): + """The child DKAO task runs on the 4.1 Flash model, and the label it + submits is one DKAO's own config resolves (the two task packages must not + drift apart on model naming).""" + from metainfer.tasks.dcu_kernel_auto_opt.orchestrator.config import ( + DSH_MODEL_IDS, + ) + + ev = DkaoCliEvaluator() + req = ev._build_dkao_requirements( + _cfg(tmp_path), _inst(), 1, "child-id", "repo-name", 0, + tmp_path / "harness", + ) + label = req["agent_model"] + assert label == "deepseek-flash-4.1" + assert DSH_MODEL_IDS[label] == "deepseek/deepseek-flash" + + +def test_normalize_real_final_report(): + report = { + "status": "success", + "final_validation": { + "hy3_tp8_qkv_proj_m16": { + "passed": True, "graph_capture_passed": True, + "mismatch_count": 0, "median_us": 42.0, "p90_us": 43.0, + }, + }, + "workers": {}, + } + out = _normalize_report(_inst(), report) + assert out["status"] == "success" + assert out["correctness_ok"] is True + assert out["median_us"] == 42.0 and out["p90_us"] == 43.0 + + +def test_partial_worker_result_is_preserved_but_never_passes(tmp_path): + worker = tmp_path / "workers" / "worker_0" + worker.mkdir(parents=True) + (worker / "result.json").write_text(json.dumps({ + "shapes": {_inst().id: {"metrics": { + "graph_capture_passed": True, + "correctness_passed_in_precheck": True, + "median_us": 40.0, "p90_us": 41.0, + }}}, + }), encoding="utf-8") + out = _normalize_workspace_result(_inst(), tmp_path) + assert out["status"] == "partial_worker_result" + assert out["median_us"] == 40.0 + assert out["correctness_ok"] is True + assert out["passed"] is False # no final serial validation + + +def test_real_evaluator_forces_isolated_repo_root(tmp_path, monkeypatch): + cfg = _cfg(tmp_path) + ev = DkaoCliEvaluator() + # The device is a pure layout decision (index % 4): the evaluator takes no + # reading of the host, so the test cannot depend on what is running on it. + seen = {} + + def fake_run_one(cfg_, workspace, iteration, inst, index, repo_root, + child_root, preflight=None, **kwargs): + seen["repo_root"] = repo_root + seen["child_root"] = child_root + seen["index"] = index + return {"status": "completed", "passed": True, "median_us": 1.0} + + monkeypatch.setattr(ev, "_run_one", fake_run_one) + out = ev.evaluate(cfg, tmp_path / "harness", 1, [_inst()]) + assert out[_inst().id]["passed"] is True + assert seen["repo_root"] == (tmp_path / "ahe-repos").resolve() + assert "children/iteration_001" in str(seen["child_root"]) + assert seen["index"] == 0 + + +def test_agent_evolver_prompt_is_single_role_and_pool_bounded(tmp_path): + ev = AgentEvolver() + prompt = ev._prompt(tmp_path, 1, { + "overview": "overview.md", "selected": ["a"], "pool_ids": ["a", "b"], + "results": {}, "diff": {}, "decision": {}, + }) + assert "SINGLE AHE Evolve Agent" in prompt + assert "two responsibilities in this ONE role" in prompt + assert "invent pool ids" in prompt + assert "_ahe_change_manifest.json" in prompt + assert "_ahe_round_plan.json" in prompt + + +def test_evolve_agent_defaults_to_flash_41(tmp_path): + """Both the adapter and build_evolver default to deepseek/deepseek-flash.""" + from ..orchestrator.evolve import ( + DEFAULT_EVOLVE_MODEL, LEGACY_EVOLVE_MODEL, build_evolver, + ) + + assert DEFAULT_EVOLVE_MODEL == "deepseek/deepseek-flash" + assert LEGACY_EVOLVE_MODEL == "deepseek/deepseek-v4-flash-0731" + assert AgentEvolver().model == DEFAULT_EVOLVE_MODEL + assert build_evolver("agent").model == DEFAULT_EVOLVE_MODEL + # A form answer still wins over the default. + assert build_evolver("agent", model=LEGACY_EVOLVE_MODEL).model == ( + LEGACY_EVOLVE_MODEL) + + +def test_last_result_parses_stream_json(): + text = '\n'.join([ + json.dumps({"type": "assistant", "message": "x"}), + json.dumps({"type": "result", "result": "done"}), + ]) + assert _last_result(text) == "done" + + +def test_real_mode_rejects_builtin_pool(tmp_path): + from ..orchestrator.config import ExperimentConfig + from ..orchestrator.pipeline import _load_pool + cfg = ExperimentConfig( + task_id="t", state_dir=tmp_path / "s", workspace_dir=tmp_path / "w", + suite=[], execution_mode="dkao-cli", agent_framework="dsh", + pool_source="builtin", + ) + try: + _load_pool(cfg) + raise AssertionError("expected ValueError for builtin pool in real mode") + except ValueError: + pass + + +def test_measured_pool_yaml_loads(tmp_path): + from ..orchestrator.pool_builder import build_pool + # a minimal fake historical report -> builder -> loader round trip + wd = tmp_path / "nodes" / "n" / "workspaces" / "task" + (wd / "final_report.json").parent.mkdir(parents=True) + (wd / "final_report.json").write_text(json.dumps({ + "config": { + "model": "hy3", + "shapes": [{"id": "hy3_tp8_qkv_proj_m16", "M": 16, "N": 1280, + "K": 4096, "tp_size": 8, "operator": "qkv_proj"}], + }, + "initial_metrics": {"hy3_tp8_qkv_proj_m16": {"baseline_us": 100.0}}, + "final_validation": {"hy3_tp8_qkv_proj_m16": {"median_us": 42.0, + "p90_us": 43.0}}, + }), encoding="utf-8") + data = build_pool(tmp_path) + assert len(data["instances"]) == 1 + pool_path = tmp_path / "pool.yaml" + import yaml as _y + from ..orchestrator.pool import Pool + pool_path.write_text(_y.safe_dump(data, sort_keys=False), encoding="utf-8") + pool = Pool.from_yaml(pool_path) + assert pool.all_ids() == ["hy3_tp8_qkv_proj_m16"] + assert pool.get("hy3_tp8_qkv_proj_m16").baseline_us == 100.0 + + +def test_worker_rounds_counts_real_attempts(tmp_path): + """rounds_used falls back to the workers' own experiment records.""" + exp = tmp_path / "workers" / "worker_0" / "runs" / "shapeA" + exp.mkdir(parents=True) + (exp / "experiments.jsonl").write_text( + '{"iteration": 1}\n{"iteration": 2}\n{"iteration": 3}\n', + encoding="utf-8") + assert _worker_rounds(tmp_path) == 3 + assert _worker_rounds(tmp_path / "missing") == 0 + + +def test_child_env_enables_planner_by_default(tmp_path): + """The evaluated harness' planner only works with METAINFER_PLANNER=1.""" + cfg = _cfg(tmp_path) + env = _child_env(cfg, tmp_path / "repos", tmp_path / "ws") + assert env["METAINFER_PLANNER"] == "1" + assert env["METAINFER_KERNEL_REPOS"] == str(tmp_path / "repos") + assert env["METAINFER_HARNESS_ROOT"] == str(tmp_path / "ws") + + +def test_child_env_planner_can_be_disabled(tmp_path): + cfg = _cfg(tmp_path) + cfg.answers["planner_enabled"] = "false" + env = _child_env(cfg, tmp_path / "repos", tmp_path / "ws") + assert "METAINFER_PLANNER" not in env + + +def test_child_env_uses_task_scope_and_quick_bench_by_default(tmp_path): + """One question = one shape: AHE defaults to the cheap validation budget.""" + cfg = _cfg(tmp_path) + env = _child_env(cfg, tmp_path / "repos", tmp_path / "ws") + assert env["METAINFER_VALIDATE_SCOPE"] == "task" + assert env["METAINFER_BENCH_WARMUPS"] == "30" + assert env["METAINFER_BENCH_SAMPLES"] == "30" + assert env["METAINFER_BENCH_REPLAYS"] == "50" + + +def test_child_env_full_budget_can_be_requested(tmp_path): + cfg = _cfg(tmp_path) + cfg.answers["validate_scope"] = "api" + cfg.answers["bench_profile"] = "full" + env = _child_env(cfg, tmp_path / "repos", tmp_path / "ws") + assert env["METAINFER_VALIDATE_SCOPE"] == "api" + assert "METAINFER_BENCH_WARMUPS" not in env + assert "METAINFER_BENCH_SAMPLES" not in env + + +def test_child_env_explicit_bench_numbers_win(tmp_path): + cfg = _cfg(tmp_path) + cfg.answers["bench_samples"] = "12" + env = _child_env(cfg, tmp_path / "repos", tmp_path / "ws") + assert env["METAINFER_BENCH_SAMPLES"] == "12" + + +def test_env_failure_detection_includes_compile_and_timeout(): + """Compile errors and timeouts are environmental: they earn a retry. + + The rule is "no usable number => the environment owes us a measurement". + A reported compile error *with* a number is contradictory, and the number is + the stronger evidence: the kernel ran, so the round scores it (as a loss) + instead of throwing it away and re-running a question that already measured. + """ + from metainfer.tasks.harness_evolve.orchestrator.adapters.eval import ( + DkaoCliEvaluator, + ) + ev = DkaoCliEvaluator() + assert ev._env_failed({"status": "success", "passed": False, + "median_us": None, + "reason": "ninja: build stopped: subcommand failed"}) + assert ev._env_failed({"status": "timeout", "passed": False, + "median_us": None, "reason": "timed out after 900s"}) + measured_with_error_text = {"status": "rejected", "passed": False, + "median_us": 12.0, "correctness_ok": True, + "reason": "hipcc: error: undefined template DUFragment"} + assert ev._env_failed(measured_with_error_text) is False + # a genuine performance failure is NOT environmental + assert not ev._env_failed({"status": "success", "passed": False, + "median_us": 900.0, + "reason": "median above gate"}) + # a passing result is never an environment failure + assert not ev._env_failed({"status": "success", "passed": True, + "median_us": 12.0}) + + +def _pool_inst(inst_id: str): + from types import SimpleNamespace + return SimpleNamespace(id=inst_id, shape={"M": 16, "N": 16, "K": 16}, + budget_rounds=3, notes="") + + +def test_a_measured_partial_result_is_not_an_environment_failure(): + ev = DkaoCliEvaluator() + measured = {"status": "partial_worker_result", "passed": False, + "correctness_ok": True, "median_us": 8.52, + "reason": "final report missing; worker best preserved as evidence"} + assert ev._env_failed(measured) is False # the number is evidence + # ... but on a re-run the same partial *is* an environment problem: the + # device was shared while it ran. + assert ev._env_failed(measured, retry=True) is True + + unmeasured = {"status": "partial_worker_result", "passed": False, + "median_us": None, "reason": "no worker numbers"} + assert ev._env_failed(unmeasured) is True + + missing = {"status": "missing_report", "passed": False, "median_us": None} + assert ev._env_failed(missing) is True + assert ev._env_failed({"status": "success", "passed": True, + "median_us": 5.0}) is False + + +def test_every_attempt_gets_its_own_child_directory(tmp_path, monkeypatch): + """A retried question must not reuse its first attempt's child workspace. + + DKAO binds ``children///workspace/main`` to the exact + kernel repo it was launched with and refuses to run when that symlink points + elsewhere. Re-running into the same directory therefore kills the retry + instantly ("workspace main already points to ...") and an environment hiccup + turns into a burned retry budget. Each attempt owns a directory instead, so + the failed attempt's evidence also survives next to the retry's. + """ + cfg = _cfg(tmp_path) + cfg.answers["env_retry_attempts"] = 2 + ev = DkaoCliEvaluator() + dirs: List[str] = [] + + def fake_run_one(cfg_, ws, iteration, inst, gpu, rr, cr, layout=None, + **kwargs): + dirs.append(cr.name) + # A real child leaves a workspace bound to its kernel repo behind. + (cr / inst.id / "workspace").mkdir(parents=True, exist_ok=True) + (cr / inst.id / "orchestrator-external.log").write_text( + f"attempt {len(dirs)}", encoding="utf-8") + if len(dirs) == 1: + return {"status": "missing_report", "passed": False, + "median_us": None, "returncode": 1} + return {"status": "completed", "passed": True, "median_us": 1.0} + + monkeypatch.setattr(ev, "_run_one", fake_run_one) + out = ev.evaluate(cfg, tmp_path / "harness", 1, [_inst()]) + + assert len(dirs) == 2 and len(set(dirs)) == 2, ( + f"each attempt needs its own directory, got {dirs}") + # the first attempt keeps the plain name; a retry is suffixed + assert dirs == ["iteration_001", "iteration_001_a2"] + # the retry ran in a fresh directory, so the first attempt's evidence is + # still there to explain why it failed + first = cfg.exp_dir / "children" / dirs[0] / _inst().id + assert (first / "orchestrator-external.log").read_text() == "attempt 1" + assert out[_inst().id]["passed"] is True + + +def _events(cfg): + path = cfg.state_dir / "timeline.jsonl" + if not path.is_file(): + return [] + return [json.loads(l)["type"] for l in + path.read_text().splitlines() if l.strip()] + + +def test_a_round_hands_every_question_to_dkao_without_checking_the_cards( + tmp_path, monkeypatch): + """The dispatch never probes a device: it lays questions out and publishes. + + Eight questions are two waves of four, one question per device, and the + evaluator never asks the host anything about the cards. + """ + cfg = _cfg(tmp_path) + ev = DkaoCliEvaluator() + seen = [] + + def fake_run_one(cfg_, ws, iteration, inst, gpu, rr, cr, layout=None, + **kwargs): + # the device HE hands the question to is the fixed index % 4 rotation + seen.append((inst.id, gpu)) + return {"status": "completed", "passed": True, "median_us": 1.0} + + monkeypatch.setattr(ev, "_run_one", fake_run_one) + out = ev.evaluate(cfg, tmp_path / "harness", 1, + [_pool_inst(i) for i in ("a", "b", "c", "d", + "e", "f", "g", "h")]) + assert set(out) == set("abcdefgh") + # one question per device, in order, wrapping onto device 0 for wave two + assert seen == [("a", 0), ("b", 1), ("c", 2), ("d", 3), + ("e", 0), ("f", 1), ("g", 2), ("h", 3)] + # nothing in the round reads or leases a device + layout = (cfg.exp_dir / "runs" / "iteration_001" / "input" / "benchmark" + / "gpu_preflight.json") + saved = json.loads(layout.read_text(encoding="utf-8")) + assert saved["leases"] == [] and saved["enabled"] is False + assert saved["managed_by"] == "dcu_kernel_auto_opt" + assert "gpu_layout_fixed" in _events(cfg) + + +def test_a_dkao_gate_verdict_stops_the_round_instead_of_relaunching( + tmp_path, monkeypatch): + """Exit 75 is the child's gate giving up after its own 24 h budget. + + HE has no cards to reassign, so relaunching the child would mean ignoring a + verdict it has no standing to overrule: the round stops and says so. + """ + cfg = _cfg(tmp_path) + ev = DkaoCliEvaluator() + calls = [] + + def fake_run_one(cfg_, ws, iteration, inst, gpu, rr, cr, layout=None, + **kwargs): + calls.append(inst.id) + return {"status": "environment_failed", "passed": False, + "median_us": None, "returncode": 75, "gate_blocked": True} + + monkeypatch.setattr(ev, "_run_one", fake_run_one) + out = ev.evaluate(cfg, tmp_path / "harness", 1, + [_pool_inst(i) for i in ("a", "b", "c", "d")]) + # exactly one attempt per question — no retry storm against a busy card + assert calls == ["a", "b", "c", "d"] + assert all(r["gate_blocked"] for r in out.values()) + stop = json.loads((cfg.exp_dir / "stop_requested.json").read_text( + encoding="utf-8")) + assert stop["reason"] == "round_incomplete" + assert stop["cause"] == "gpu_gate_blocked" + assert stop["unmeasured"] == ["a", "b", "c", "d"] + assert "env_retry_exhausted" not in _events(cfg) + + +def test_an_ordinary_environment_failure_is_still_retried(tmp_path, monkeypatch): + """A broken toolchain is not a gate verdict: the environment owes a number.""" + cfg = _cfg(tmp_path) + cfg.answers["env_retry_attempts"] = 2 + ev = DkaoCliEvaluator() + attempts = [] + + def fake_run_one(cfg_, ws, iteration, inst, gpu, rr, cr, layout=None, + **kwargs): + attempts.append(inst.id) + if len(attempts) == 1: + return {"status": "missing_report", "passed": False, + "median_us": None, "returncode": 1} + return {"status": "completed", "passed": True, "median_us": 1.0} + + monkeypatch.setattr(ev, "_run_one", fake_run_one) + out = ev.evaluate(cfg, tmp_path / "harness", 1, [_inst()]) + assert attempts == [_inst().id, _inst().id] + assert out[_inst().id]["passed"] is True + assert (cfg.exp_dir / "children" / "iteration_001_a2").is_dir(), ( + "the retry ran in the second attempt's own directory") + assert "env_retry_exhausted" not in _events(cfg) + + +def test_a_gate_verdict_is_never_counted_as_a_measurement(tmp_path, monkeypatch): + """A question the gate blocked has no number, so the round is incomplete.""" + cfg = _cfg(tmp_path) + ev = DkaoCliEvaluator() + res = {"status": "environment_failed", "passed": False, + "median_us": None, "gate_blocked": True} + assert ev._is_measured(res) is False + assert ev._incomplete_ids([_inst()], {_inst().id: res}) == [_inst().id] + + diff --git a/metainfer/tasks/harness_evolve/tests/test_repo_cleanup.py b/metainfer/tasks/harness_evolve/tests/test_repo_cleanup.py new file mode 100644 index 00000000..47ccb699 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_repo_cleanup.py @@ -0,0 +1,152 @@ +"""Tests for orchestrator.repo_cleanup — 任务级候选仓库清理的安全边界。 + +重点验证"绝不误删"： +- 只匹配 ``ahe-{task_id}-it`` 精确前缀（task_id 互为前缀时不误伤） +- 池白名单（registered_pool.yaml 引用）一律保留 +- 符号链接 / 非目录 / 越界路径一律跳过 +- dry_run 不做任何修改 +""" + +from __future__ import annotations + +import textwrap + +import pytest + +from ..orchestrator.repo_cleanup import ( + load_pool_keep_list, + owned_repos, + purge_task_kernel_repos, + repo_name_prefix, + resolve_ahe_repo_root, +) + +TASK = "test-9-8-5-e10567a1" + + +def _mk_repo(root, name): + d = root / name + d.mkdir(parents=True) + (d / "kernel.py").write_text("print('x')\n", encoding="utf-8") + (d / "sub").mkdir() + (d / "sub" / "blob.bin").write_bytes(b"0" * 32) + return d + + +def _pool(root, names): + lines = ["schema_version: 1", "instances:"] + for n in names: + lines.append(f"- id: {n}") + lines.append(f" source_report: {root}/{n}/final_report.json") + (root / "registered_pool.yaml").write_text( + "\n".join(lines) + "\n", encoding="utf-8") + + +def test_repo_name_prefix_uses_it_marker(): + assert repo_name_prefix(TASK) == f"ahe-{TASK}-it" + + +def test_resolve_ahe_repo_root_prefers_answer(tmp_path): + custom = tmp_path / "custom-root" + assert resolve_ahe_repo_root({"ahe_repo_root": str(custom)}) == custom.resolve() + + +def test_resolve_ahe_repo_root_defaults_when_absent(): + assert str(resolve_ahe_repo_root(None)).endswith("ahe-kernel-repos") + + +def test_prefix_protects_repo_when_task_id_is_prefix_of_another(tmp_path): + """``ahe--it`` 结尾的 '-it' 保证不误伤同前缀的其他任务。""" + _mk_repo(tmp_path, f"ahe-{TASK}-it001-candidate-foo-aaaaaa") + other = _mk_repo(tmp_path, f"ahe-{TASK}-extra-it001-candidate-bar-bbbbbb") + + deletable, _ = owned_repos(TASK, tmp_path, keep=set()) + names = {p.name for p in deletable} + + assert f"ahe-{TASK}-it001-candidate-foo-aaaaaa" in names + assert other.name not in names, "task_id 同前缀的其他任务仓库被误列" + + +def test_unrelated_task_repos_untouched(tmp_path): + _mk_repo(tmp_path, "ahe-other-task-it001-candidate-foo-aaaaaa") + _mk_repo(tmp_path, "dsh-min-1iter-abff697d") # 人工仓库 + deletable, _ = owned_repos(TASK, tmp_path, keep=set()) + assert deletable == [] + + +def test_pool_whitelist_is_never_deleted(tmp_path): + keep_name = f"ahe-{TASK}-it001-candidate-keepme-aaaaaa" + drop_name = f"ahe-{TASK}-it002-candidate-dropme-bbbbbb" + _mk_repo(tmp_path, keep_name) + _mk_repo(tmp_path, drop_name) + _pool(tmp_path, [keep_name]) + + keep = load_pool_keep_list(tmp_path) + assert keep_name in keep + + deletable, skipped = owned_repos(TASK, tmp_path, keep=keep) + names = {p.name for p in deletable} + assert names == {drop_name} + assert any(keep_name in s for s in skipped) + + +def test_symlink_repos_are_skipped(tmp_path): + real = _mk_repo(tmp_path, f"ahe-{TASK}-it003-candidate-real-cccccc") + link = tmp_path / f"ahe-{TASK}-it004-candidate-link-dddddd" + link.symlink_to(real) + + deletable, skipped = owned_repos(TASK, tmp_path, keep=set()) + names = {p.name for p in deletable} + assert real.name in names + assert link.name not in names + assert any(link.name in s for s in skipped) + + +def test_dry_run_changes_nothing(tmp_path): + name = f"ahe-{TASK}-it005-candidate-x-eeeeee" + repo = _mk_repo(tmp_path, name) + + summary = purge_task_kernel_repos(TASK, tmp_path, keep=set(), dry_run=True) + assert summary["dry_run"] is True + assert summary["matched"] == 1 + assert summary["deleted"] == [] + assert summary["bytes"] > 0 + assert repo.exists(), "dry_run 不应删除任何东西" + + +def test_apply_removes_only_owned_repos(tmp_path): + owned = _mk_repo(tmp_path, f"ahe-{TASK}-it006-candidate-x-ffffff") + kept = _mk_repo(tmp_path, "ahe-other-it006-candidate-y-111111") + + summary = purge_task_kernel_repos(TASK, tmp_path, keep=set(), dry_run=False) + assert len(summary["deleted"]) == 1 + assert not summary["failed"] + assert not owned.exists() + assert kept.exists(), "其他任务的仓库被误删" + + +def test_missing_repo_root_is_reported_not_raised(tmp_path): + missing = tmp_path / "nope" + summary = purge_task_kernel_repos(TASK, missing, dry_run=True) + assert summary["matched"] == 0 + assert summary["skipped"] + + +def test_refuses_when_task_still_registered(tmp_path, monkeypatch): + """任务仍在注册表 → 默认拒绝清理（force=False）。""" + name = f"ahe-{TASK}-it010-candidate-live-aaaaaa" + repo = _mk_repo(tmp_path, name) + + from ..orchestrator import repo_cleanup as rc + monkeypatch.setattr(rc, "is_task_registered", lambda tid: True) + + summary = rc.purge_task_kernel_repos(TASK, tmp_path, keep=set(), dry_run=False) + assert summary.get("refused") == "task_still_registered" + assert summary["deleted"] == [] + assert repo.exists(), "注册中的任务仓库被误删" + + # force=True 时才允许 + summary2 = rc.purge_task_kernel_repos( + TASK, tmp_path, keep=set(), dry_run=False, force=True) + assert len(summary2["deleted"]) == 1 + assert not repo.exists() diff --git a/metainfer/tasks/harness_evolve/tests/test_resume.py b/metainfer/tasks/harness_evolve/tests/test_resume.py new file mode 100644 index 00000000..4e6c4004 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_resume.py @@ -0,0 +1,356 @@ +"""Resume-the-next-iteration support for harness_evolve.""" + +from __future__ import annotations + +import json +import time +from pathlib import Path + +from metainfer.tasks.harness_evolve.orchestrator.cli import resume_command +from metainfer.tasks.harness_evolve.orchestrator.config import ( + load_experiment_config, +) +from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + completed_iterations, run_experiment, +) +from metainfer.tasks.harness_evolve.orchestrator.state import ( + read_target_iterations, write_target_iterations, +) +from metainfer.tasks.harness_evolve.server.routes import ( + _launch_resume, _resume_running, +) + + +def _write_req(tmp_path: Path) -> Path: + req = tmp_path / "requirements.json" + req.write_text(json.dumps({ + "task_id": "resume-task", + "task_type": "harness-evolve", + "execution_mode": "dry-run", + "max_iterations": 1, + "evolve_mode": "dry-run", + "per_round_budget": 2, + "pool_source": "", + }), encoding="utf-8") + return req + + +def _cfg(tmp_path: Path): + return load_experiment_config( + _write_req(tmp_path), tmp_path / "state", tmp_path / "ws") + + +def test_resume_runs_next_iteration_with_champion_from_disk(tmp_path): + cfg = _variant_cfg(tmp_path, rounds=2) # two rounds: baseline + one pass + write_target_iterations(cfg.exp_dir, 1) # this call may run round 1 only + run_experiment(cfg) + exp = cfg.exp_dir + assert completed_iterations(exp) == 1 + assert (exp / "runs" / "iteration_001" / "input" / "decision.json").is_file() + + write_target_iterations(cfg.exp_dir, 2) # ... and now it may run round 2 + run_experiment(cfg, start_iteration=2, iterations_to_run=1) + + assert completed_iterations(exp) == 2 + it2 = exp / "runs" / "iteration_002" / "input" + assert (it2 / "benchmark" / "results.json").is_file() + assert (it2 / "decision.json").is_file() + # champion/prev state must come from the resumed experiment + decision = json.loads((it2 / "decision.json").read_text(encoding="utf-8")) + assert decision["iteration"] == 2 + scores = (exp / "iteration_scores.jsonl").read_text( + encoding="utf-8").strip().splitlines() + assert len(scores) == 2 + assert json.loads(scores[-1])["iteration"] == 2 + + +def test_cli_resume_starts_round_one_when_nothing_ran(tmp_path): + """No completed round and nobody running: resume bootstraps round 1.""" + state = tmp_path / "state" + state.mkdir(parents=True) + _write_req(state if False else tmp_path) # requirements at tmp_path + (state / "requirements.json").write_text( + (tmp_path / "requirements.json").read_text(encoding="utf-8"), + encoding="utf-8") + import argparse + args = argparse.Namespace( + state_dir=state, workspace_dir=tmp_path / "ws", + requirements=None, iterations=1) + assert resume_command(args) == 0 # not a refusal any more + assert (tmp_path / "ws" / "runs" / "iteration_001").is_dir() + + +def test_cli_resume_defers_to_an_already_running_task(tmp_path, monkeypatch): + """If the task is already running (started by the server), resume exits.""" + state = tmp_path / "state" + state.mkdir(parents=True) + _write_req(tmp_path) + (state / "requirements.json").write_text( + (tmp_path / "requirements.json").read_text(encoding="utf-8"), + encoding="utf-8") + import argparse + from metainfer.tasks.harness_evolve.orchestrator import supervisor as sup + monkeypatch.setattr(sup, "task_process_alive", lambda _state: 4242) + args = argparse.Namespace( + state_dir=state, workspace_dir=tmp_path / "ws", + requirements=None, iterations=1) + assert resume_command(args) == 0 + assert not (tmp_path / "ws" / "runs" / "iteration_001").is_dir() + + +def test_resume_running_detects_live_pid(tmp_path): + """Only a live harness_evolve process counts (zombies/pids are rejected).""" + import os + import subprocess + import sys + + state = tmp_path / "state" + state.mkdir(parents=True) + assert _resume_running(state) is None + + # our own pid is alive but is not a harness run + (state / "resume.pid").write_text(str(os.getpid()), encoding="utf-8") + assert _resume_running(state) is None + + proc = subprocess.Popen( + [sys.executable, "-c", "import time; time.sleep(30)", "harness_evolve"]) + try: + (state / "resume.pid").write_text(str(proc.pid), encoding="utf-8") + assert _resume_running(state) == proc.pid + finally: + proc.kill() + proc.wait() + + (state / "resume.pid").write_text("999999", encoding="utf-8") + assert _resume_running(state) is None + + +def test_launch_resume_end_to_end_dry(tmp_path): + """The resume CLI really runs the next round (dry-run fixture).""" + cfg = _cfg(tmp_path) + run_experiment(cfg) + state_dir = cfg.state_dir + (state_dir / "requirements.json").write_text( + (tmp_path / "requirements.json").read_text(encoding="utf-8"), + encoding="utf-8") + + # The form's Iterations answer is the run's round target; raise it so the + # resumed run is allowed to spend another round (a performance pass costs + # 1.0, the retake 0.5, the baseline round is free). + write_target_iterations(cfg.exp_dir, 3) + out = _launch_resume(state_dir, cfg.exp_dir, 1) + assert out["ok"] is True and out["pid"] > 0 + + deadline = time.time() + 180 + while time.time() < deadline: + if completed_iterations(cfg.exp_dir) >= 2: + break + time.sleep(1) + assert completed_iterations(cfg.exp_dir) >= 2, (state_dir / "resume.log").read_text( + encoding="utf-8", errors="replace")[-1500:] + + +def test_target_iterations_override_fallback_rounds(tmp_path): + """The target (a round count) decides, not the caller's batch size.""" + cfg = _variant_cfg(tmp_path) + write_target_iterations(cfg.exp_dir, 3) + assert read_target_iterations(cfg.exp_dir) == 3 + run_experiment(cfg, iterations_to_run=1) # the caller asks for 1 + assert completed_iterations(cfg.exp_dir) == 3 # the target wins + + +def test_lowered_target_stops_the_loop_immediately(tmp_path): + """Lowering the target below the next round stops without a new round.""" + cfg = _cfg(tmp_path) + run_experiment(cfg) # iteration 1 done + write_target_iterations(cfg.exp_dir, 1) # target already met + run_experiment(cfg, start_iteration=2, iterations_to_run=5) + assert completed_iterations(cfg.exp_dir) == 1 + assert not (cfg.exp_dir / "runs" / "iteration_002").exists() + + +def test_target_can_be_cleared_for_fixed_rounds(tmp_path): + cfg = _cfg(tmp_path) + write_target_iterations(cfg.exp_dir, 5) + from metainfer.tasks.harness_evolve.orchestrator.state import ( + clear_target_iterations, + ) + clear_target_iterations(cfg.exp_dir) + assert read_target_iterations(cfg.exp_dir) is None + + +def test_champion_iteration_pins_the_comparison_baseline(tmp_path): + """Resume can compare the new round against a chosen finished iteration.""" + cfg = _variant_cfg(tmp_path, rounds=3) # three rounds of headroom + # one iteration per call, so the test controls exactly which rounds exist + write_target_iterations(cfg.exp_dir, 1) + run_experiment(cfg) # iteration 1 + write_target_iterations(cfg.exp_dir, 2) + run_experiment(cfg, start_iteration=2, iterations_to_run=1) # iteration 2 + write_target_iterations(cfg.exp_dir, 3) + run_experiment(cfg, start_iteration=3, iterations_to_run=1, + champion_iteration=2) # iteration 3 + + exp = cfg.exp_dir + best = json.loads((exp / "best_ever.json").read_text(encoding="utf-8")) + assert best["iteration"] == 2 + assert best["verdict"] == "MANUAL_BASELINE" + assert (exp / "champion_override.json").is_file() + + it2 = json.loads((exp / "runs" / "iteration_002" / "input" / "benchmark" + / "results.json").read_text(encoding="utf-8"))["results"] + decision = json.loads((exp / "runs" / "iteration_003" / "input" + / "decision.json").read_text(encoding="utf-8")) + per = decision["performance_gate"]["comparison"]["per_instance"] + assert per, "paired comparison must exist" + # The pin is what this test is about: the run's comparison baseline is + # iteration 2, so that iteration's results are what a resumed round reads + # back (the gate then compares the candidate against the variant table, as + # every round does). + assert json.loads((exp / "champion_override.json").read_text() + )["champion_iteration"] == 2 + assert json.loads((exp / "best_ever.json").read_text())["iteration"] == 2 + assert set(it2) == set(per), ( + "the pinned round and the resumed round must pair the same operators") + + +def test_resume_clears_a_previous_stop_signal(tmp_path): + """A gate-stopped run must be resumable: the stale stop signal is dropped. + + ``stop_requested.json`` is checked after every evaluate, so leaving it in + place would make the very next resume stop again without doing any work. + """ + cfg = _cfg(tmp_path) + run_experiment(cfg) + stop = cfg.exp_dir / "stop_requested.json" + stop.write_text(json.dumps({"reason": "gpu_gate_blocked", "checks": 48}), + encoding="utf-8") + + from metainfer.tasks.harness_evolve.orchestrator.cli import clear_stop_request + assert clear_stop_request(cfg.exp_dir, state_dir=cfg.state_dir) == [ + "stop_requested.json"] + assert not stop.exists() + events = [json.loads(l)["type"] for l in + (cfg.state_dir / "timeline.jsonl").read_text( + encoding="utf-8").splitlines() if l.strip()] + assert "stop_request_cleared" in events + # idempotent: without a signal there is nothing to clear + assert clear_stop_request(cfg.exp_dir, state_dir=cfg.state_dir) == [] + + +def test_completed_iterations_counts_a_judged_round_without_results_json(tmp_path): + """A variant-protocol stage is "done" through its decision artifacts. + + Only a *failing* pass writes ``input/benchmark/results.json``; a round that + passed its gate leaves ``decision.json`` + the gate files instead. Reading + only ``results.json`` reported 0 completed rounds for a run that had just + finished three, and ``resume`` then refused to continue. + """ + exp = tmp_path / "exp" + for n, artifact in ((1, "baseline_round.json"), + (2, "performance_gate.json"), + (3, "generalization_gate.json")): + d = exp / "runs" / f"iteration_{n:03d}" / "input" + d.mkdir(parents=True) + (d / "decision.json").write_text("{}", encoding="utf-8") + (d / artifact).write_text("{}", encoding="utf-8") + # an empty directory is not a completed round + (exp / "runs" / "iteration_004" / "input").mkdir(parents=True) + + assert completed_iterations(exp) == 3 + + # the classic evidence (a stage that stopped early) still counts + (exp / "runs" / "iteration_005" / "input" / "benchmark").mkdir(parents=True) + (exp / "runs" / "iteration_005" / "input" / "benchmark" + / "results.json").write_text("{}", encoding="utf-8") + assert completed_iterations(exp) == 5 + + +# ------------------------------------------------- rounds as durable state + +def _variant_cfg(tmp_path: Path, **answers): + """A config in the variant-round protocol: it needs a real pool file.""" + import yaml + + ops = {f"op{i:02d}": {"baseline_us": 100.0 + i, "best_known_us": 40.0 + 4 * i} + for i in range(8)} + pool = tmp_path / "pool.yaml" + pool.write_text(yaml.safe_dump({ + "schema_version": 1, + "instances": [ + {"id": iid, "family": f"decode__fam{i}", "contract": {"M": 16}, + "baseline_us": v["baseline_us"], "best_known_us": v["best_known_us"], + "history": []} + for i, (iid, v) in enumerate(sorted(ops.items())) + ], + }, sort_keys=False), encoding="utf-8") + tmp_path.mkdir(parents=True, exist_ok=True) + req = tmp_path / "requirements.json" + req.write_text(json.dumps({ + "task_id": "rounds-task", + "task_type": "harness-evolve", + "execution_mode": "dry-run", + "evolve_mode": "dry-run", + "pool_source": str(pool), + "question_pool": str(pool), + "round_questions": "variant", + "per_round_budget": "4", + "max_iterations": 1, + **answers, + }), encoding="utf-8") + return load_experiment_config(req, tmp_path / "state", tmp_path / "ws") + + +def test_rounds_used_survives_a_resume(tmp_path): + """The budget must not reset when the process is restarted. + + ``rounds_used`` used to live only in memory, so a resumed run came back with + a full budget and could keep spending rounds the operator never asked for. + """ + from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + _rounds_used_on_disk, + ) + + cfg = _variant_cfg(tmp_path, rounds=3) + run_experiment(cfg, iterations_to_run=2) # baseline + one judgment + spent = _rounds_used_on_disk(cfg.exp_dir) + assert spent >= 1.0, "a judged pass must leave its cost on disk" + assert json.loads((cfg.state_dir / "run.json").read_text())["rounds_used"] == spent + # a second process (no memory of the first) sees the same number + assert _rounds_used_on_disk(cfg.exp_dir) == spent + + +def test_rounds_used_is_derived_when_run_json_predates_it(tmp_path): + """Old experiments report their spent rounds instead of a fresh budget.""" + from metainfer.tasks.harness_evolve.orchestrator.pipeline import ( + _rounds_used_on_disk, + ) + + exp = tmp_path / "exp" + for n, cost in ((1, 0.0), (2, 1.0), (3, 0.5)): + d = exp / "runs" / f"iteration_{n:03d}" / "input" + d.mkdir(parents=True) + (d / "stage.json").write_text(json.dumps({"round_cost": cost}), + encoding="utf-8") + assert _rounds_used_on_disk(exp) == 1.5 + + +def test_a_round_is_an_iteration(tmp_path): + """`Rounds: 2` means the run may reach iteration 2 and no further. + + Round 1 is the baseline round; round 2 is a performance pass (plus, when it + wins, the generalization retake inside that same iteration — a retake never + consumes a round of its own). + """ + cfg = _variant_cfg(tmp_path, rounds=2) + write_target_iterations(cfg.exp_dir, 2) + run_experiment(cfg, iterations_to_run=5) # ask for five; rounds say two + + stages = [json.loads(p.read_text())["stage"] + for p in sorted((cfg.exp_dir / "runs").glob("iteration_*/input/stage.json"))] + assert stages == ["baseline", "performance"] + assert completed_iterations(cfg.exp_dir) == 2 + run = json.loads((cfg.state_dir / "run.json").read_text()) + assert run["finished"] is True + # the reported cost is bookkeeping: one judged pass + assert run["rounds_used"] == 1.0 diff --git a/metainfer/tasks/harness_evolve/tests/test_round_incomplete.py b/metainfer/tasks/harness_evolve/tests/test_round_incomplete.py new file mode 100644 index 00000000..4ed589d0 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_round_incomplete.py @@ -0,0 +1,144 @@ +"""A round that could not measure every question gets no verdict. + +``evaluate_gate`` used to answer ``FAIL`` whenever a drawn question produced no +number. That reads as "the harness is worse than the variant", which is a claim +the round did not support — and the loop acted on it: it went through +analyze→evolve to fix a harness that may have been fine, and on the +generalization retake it spent the candidate on a missing measurement. + +This drives the real loop (dry-run evaluator) with one question's result removed +from the performance pass and checks what the run does about it. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import yaml + +from metainfer.tasks.harness_evolve.orchestrator import pipeline as P +from metainfer.tasks.harness_evolve.orchestrator.config import ( + load_experiment_config, +) +from metainfer.tasks.harness_evolve.orchestrator.pipeline import run_experiment +from metainfer.tasks.harness_evolve.orchestrator.state import ( + write_target_iterations, +) + + +def _variant_cfg(tmp_path: Path, **answers): + """Same shape as the resume tests: a real pool file, four questions.""" + ops = {f"op{i:02d}": {"baseline_us": 100.0 + i, "best_known_us": 40.0 + 4 * i} + for i in range(8)} + pool = tmp_path / "pool.yaml" + pool.write_text(yaml.safe_dump({ + "schema_version": 1, + "instances": [ + {"id": iid, "family": f"decode__fam{i}", "contract": {"M": 16}, + "baseline_us": v["baseline_us"], "best_known_us": v["best_known_us"], + "history": []} + for i, (iid, v) in enumerate(sorted(ops.items())) + ], + }, sort_keys=False), encoding="utf-8") + tmp_path.mkdir(parents=True, exist_ok=True) + req = tmp_path / "requirements.json" + req.write_text(json.dumps({ + "task_id": "incomplete-task", + "task_type": "harness-evolve", + "execution_mode": "dry-run", + "evolve_mode": "dry-run", + "pool_source": str(pool), + "question_pool": str(pool), + "round_questions": "variant", + "per_round_budget": "4", + "max_iterations": 1, + **answers, + }), encoding="utf-8") + return load_experiment_config(req, tmp_path / "state", tmp_path / "ws") + + +def _timeline_events(cfg) -> list: + path = Path(cfg.state_dir) / "timeline.jsonl" + if not path.is_file(): + return [] + return [json.loads(line)["type"] + for line in path.read_text(encoding="utf-8").splitlines() if line.strip()] + + +def test_a_round_missing_a_measurement_carries_no_verdict(tmp_path, monkeypatch): + cfg = _variant_cfg(tmp_path) + write_target_iterations(cfg.exp_dir, 2) + real = P.build_evaluator("dry-run") + dropped: dict = {} + + def factory(_mode): + class Sabotaged: + """The dry-run evaluator, minus one question on the pass round.""" + + def evaluate(self, cfg_, snapshot, iteration, instances=None): + results = real.evaluate(cfg_, snapshot, iteration, + instances=instances) + if (str(cfg_.answers.get("round_stage")) == "performance" + and results): + victim = sorted(results)[0] + dropped["id"] = victim + results = {k: v for k, v in results.items() if k != victim} + return results + + return Sabotaged() + + monkeypatch.setattr(P, "build_evaluator", factory) + + run_experiment(cfg) + + exp = cfg.exp_dir + # round 1 (baseline) measured everything and was decided as before + assert (exp / "runs" / "iteration_001" / "input" + / "baseline_round.json").is_file() + + gate = json.loads((exp / "runs" / "iteration_002" / "input" + / "performance_gate.json").read_text(encoding="utf-8")) + assert gate["status"] == "INCOMPLETE", gate + assert dropped["id"] in gate["unmeasured"] + + events = _timeline_events(cfg) + assert "round_incomplete" in events, events + # nothing was promoted, and no retake was frozen off a round without a verdict + assert not (exp / "pending_promotion.json").is_file() + assert not (exp / "pending_generalization.json").is_file() + + +def test_the_incomplete_round_is_recorded_with_its_reason(tmp_path, monkeypatch): + cfg = _variant_cfg(tmp_path) + write_target_iterations(cfg.exp_dir, 2) + real = P.build_evaluator("dry-run") + dropped: dict = {} + + def factory(_mode): + class Sabotaged: + def evaluate(self, cfg_, snapshot, iteration, instances=None): + results = real.evaluate(cfg_, snapshot, iteration, + instances=instances) + if (str(cfg_.answers.get("round_stage")) == "performance" + and results): + victim = sorted(results)[0] + dropped["id"] = victim + results = {k: v for k, v in results.items() if k != victim} + return results + + return Sabotaged() + + monkeypatch.setattr(P, "build_evaluator", factory) + run_experiment(cfg) + + rows = [json.loads(line) for line in + (Path(cfg.state_dir) / "timeline.jsonl").read_text( + encoding="utf-8").splitlines() if line.strip()] + incomplete = [row for row in rows if row["type"] == "round_incomplete"] + assert incomplete, rows + payload = incomplete[0]["payload"] + assert payload["gate"] == "performance" + assert payload["unmeasured"] == [dropped["id"]] + assert payload["measured"], "the questions that did report are listed" + assert "no harness verdict" in payload["policy"] diff --git a/metainfer/tasks/harness_evolve/tests/test_round_selection_evidence.py b/metainfer/tasks/harness_evolve/tests/test_round_selection_evidence.py new file mode 100644 index 00000000..bf8a6693 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_round_selection_evidence.py @@ -0,0 +1,141 @@ +"""The draw has to be explicable afterwards — and a round has to be fillable. + +Two small holes around the round's question set: + +* the draw used ``random`` without recording anything, so "why these four + operators?" could not be answered after the fact (and a pass could not be + re-run). The seed is now derived from (task, iteration, stage) unless the form + pins ``round_seed``, and it is written into the round's evidence; +* with fewer judgeable operators than a wave, the count was silently reduced — + which changes what the gate means ("at least 75% win" over 3 operators is + "all three must win"). The run now stops with a named status instead. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import yaml + +from metainfer.tasks.harness_evolve.orchestrator import pipeline as P +from metainfer.tasks.harness_evolve.orchestrator.config import ( + load_experiment_config, +) +from metainfer.tasks.harness_evolve.orchestrator.pipeline import run_experiment +from metainfer.tasks.harness_evolve.orchestrator.state import ( + write_target_iterations, +) + + +def _cfg(tmp_path: Path, *, operators: int = 8, **answers): + pool = tmp_path / "pool.yaml" + pool.write_text(yaml.safe_dump({ + "schema_version": 1, + "instances": [ + {"id": f"op{i:02d}", "family": f"decode__fam{i}", + "contract": {"M": 16}, "baseline_us": 100.0 + i, + "best_known_us": 40.0 + 4 * i, "history": []} + for i in range(operators) + ], + }, sort_keys=False), encoding="utf-8") + tmp_path.mkdir(parents=True, exist_ok=True) + req = tmp_path / "requirements.json" + req.write_text(json.dumps({ + "task_id": "selection-task", + "task_type": "harness-evolve", + "execution_mode": "dry-run", + "evolve_mode": "dry-run", + "pool_source": str(pool), + "question_pool": str(pool), + "round_questions": "variant", + "per_round_budget": "4", + "max_iterations": 1, + **answers, + }), encoding="utf-8") + return load_experiment_config(req, tmp_path / "state", tmp_path / "ws") + + +def _timeline(cfg): + path = Path(cfg.state_dir) / "timeline.jsonl" + if not path.is_file(): + return [] + return [json.loads(line) for line in + path.read_text(encoding="utf-8").splitlines() if line.strip()] + + +# ------------------------------------------------------------------- the seed + +def test_the_same_round_draws_the_same_questions(tmp_path): + cfg = _cfg(tmp_path) + + rng_a, seed_a = P._round_rng(cfg, 2, "performance") + rng_b, seed_b = P._round_rng(cfg, 2, "performance") + + assert seed_a == seed_b + assert [rng_a.choice(range(1000)) for _ in range(5)] == \ + [rng_b.choice(range(1000)) for _ in range(5)] + + +def test_different_rounds_draw_differently(tmp_path): + cfg = _cfg(tmp_path) + + first = P._round_rng(cfg, 2, "performance") + second = P._round_rng(cfg, 3, "performance") + + assert first[1] != second[1] + assert [first[0].random() for _ in range(3)] != \ + [second[0].random() for _ in range(3)] + + +def test_a_pinned_seed_in_the_form_wins(tmp_path): + cfg = _cfg(tmp_path, round_seed="4242") + + _rng, label = P._round_rng(cfg, 2, "performance") + + assert label.startswith("round_seed=4242") + + +def test_the_round_records_which_draw_it_was(tmp_path): + cfg = _cfg(tmp_path) + write_target_iterations(cfg.exp_dir, 1) + + run_experiment(cfg) + + selections = [row for row in _timeline(cfg) + if row["type"] == "variant_round_selection"] + assert selections, "the round's question set must be on the record" + payload = selections[0]["payload"] + assert payload["seed"] == f"selection-task:1:baseline" + assert payload["count"] == 4 + assert payload["judgeable_in_pool"] == 8 + assert len(payload["performance_ids"]) == 4 + + +# --------------------------------------------------- a round needs a full wave + +def test_a_pool_that_cannot_fill_a_wave_stops_with_a_named_status(tmp_path): + cfg = _cfg(tmp_path, operators=3) # 3 judgeable, a wave is 4 + + run_experiment(cfg) + + run = json.loads((Path(cfg.state_dir) / "run.json").read_text()) + assert run["final_status"] == "no_question_pool" + events = [row["type"] for row in _timeline(cfg)] + assert "variant_pool_too_small" in events, events + payload = [row for row in _timeline(cfg) + if row["type"] == "variant_pool_too_small"][0]["payload"] + assert payload["judgeable_count"] == 3 + assert payload["questions_per_round"] == 4 + + +def test_a_pool_with_a_whole_wave_still_runs(tmp_path): + """The guard must not fire for a pool that can fill the round.""" + cfg = _cfg(tmp_path, operators=4) + + run_experiment(cfg) + + run = json.loads((Path(cfg.state_dir) / "run.json").read_text()) + assert run["final_status"] != "no_question_pool" + events = [row["type"] for row in _timeline(cfg)] + assert "variant_pool_too_small" not in events diff --git a/metainfer/tasks/harness_evolve/tests/test_state_routes.py b/metainfer/tasks/harness_evolve/tests/test_state_routes.py new file mode 100644 index 00000000..9b0a109f --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_state_routes.py @@ -0,0 +1,97 @@ +"""HE experiment state recording + children-live aggregation.""" + +from __future__ import annotations + +import json + +from metainfer.server.state_reader import read_run +from metainfer.tasks.harness_evolve.orchestrator.state import ( + append_timeline, + read_run as he_read_run, + set_run, +) +from metainfer.tasks.harness_evolve.server.routes import _children_live + + +def test_set_run_roundtrips_and_matches_shell_reader(tmp_path): + state_dir = tmp_path / "state" + set_run(state_dir, task_id="t1", task_type="harness-evolve", + current_phase="evaluate", current_iteration=2) + # shell state_reader must see the same phase (run.json contract). + assert read_run(state_dir)["current_phase"] == "evaluate" + assert read_run(state_dir)["current_iteration"] == 2 + assert read_run(state_dir)["task_id"] == "t1" + # our own reader agrees + assert he_read_run(state_dir)["current_phase"] == "evaluate" + + +def test_set_run_finished_sets_terminal_fields(tmp_path): + state_dir = tmp_path / "state" + set_run(state_dir, current_phase="finished", finished=True, + final_status="success") + run = read_run(state_dir) + assert run["finished"] is True + assert run["final_status"] == "success" + + +def test_append_timeline_rows_are_readable(tmp_path): + state_dir = tmp_path / "state" + append_timeline(state_dir, "iteration_start", {"iteration": 1}) + append_timeline(state_dir, "iteration_decision", {"iteration": 1, + "verdict": "BASELINE"}) + from metainfer.tasks.harness_evolve.orchestrator.state import list_timeline + rows = list_timeline(state_dir) + assert [r["type"] for r in rows] == ["iteration_start", + "iteration_decision"] + assert rows[0]["payload"]["iteration"] == 1 + + +def test_children_live_reads_each_dkao_child_run(tmp_path): + """Each question is one DKAO child; its state/run.json drives the UI.""" + exp = tmp_path / "exp" + child = exp / "children" / "iteration_001" / "instA" / "state" + child2 = exp / "children" / "iteration_001" / "instB" / "state" + set_run(child, task_type="dcu-kernel-auto-opt", current_phase="serial_validate") + set_run(child2, task_type="dcu-kernel-auto-opt", current_phase="finished", + finished=True, final_status="success") + + live = _children_live(exp, 1) + assert set(live) == {"instA", "instB"} + assert live["instA"]["current_phase"] == "serial_validate" + assert live["instA"]["finished"] is False + assert live["instB"]["finished"] is True + assert live["instB"]["final_status"] == "success" + + +def test_children_live_missing_iteration_returns_empty(tmp_path): + assert _children_live(tmp_path / "exp", 3) == {} + + +def test_children_live_shows_a_retried_question_once_at_its_newest_attempt(tmp_path): + """One row per question, reading the attempt that ran last. + + A retry lives in its own directory (``iteration_001_a2``), so the detail + page must not show the same question twice, and the phase it shows has to + come from the retry — the failed attempt is history, not the current state. + """ + exp = tmp_path / "exp" + first = exp / "children" / "iteration_001" / "instA" / "state" + retry = exp / "children" / "iteration_001_a2" / "instA" / "state" + other = exp / "children" / "iteration_001_a2" / "instB" / "state" + set_run(first, task_type="dcu-kernel-auto-opt", current_phase="prepare", + finished=True, final_status="stopped") + set_run(retry, task_type="dcu-kernel-auto-opt", + current_phase="serial_validate") + set_run(other, task_type="dcu-kernel-auto-opt", current_phase="finished", + finished=True, final_status="success") + + live = _children_live(exp, 1) + assert set(live) == {"instA", "instB"}, "one row per question, no duplicates" + assert live["instA"]["current_phase"] == "serial_validate", ( + "the row must describe the retry, not the failed attempt") + assert live["instA"]["attempt_dir"] == "iteration_001_a2" + assert live["instA"]["child_state_dir"] == str(retry) + # the failed attempt's evidence is still on disk for the audit + assert (first / "run.json").is_file() + # a different iteration has no children of its own + assert _children_live(exp, 2) == {} diff --git a/metainfer/tasks/harness_evolve/tests/test_supervisor.py b/metainfer/tasks/harness_evolve/tests/test_supervisor.py new file mode 100644 index 00000000..a221ec8d --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_supervisor.py @@ -0,0 +1,541 @@ +"""Incident supervisor: classification, policy and supervision ticks.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from metainfer.tasks.harness_evolve.orchestrator.supervisor import ( + CATEGORY_AGENT, CATEGORY_CODE, CATEGORY_RESOURCE, CATEGORY_UNKNOWN, + Supervisor, classify_incident, decide_action, +) + + +# ------------------------------------------------------------- classify ---- + +def test_classify_resource_signals(): + assert classify_incident(log_tail="hipErrorOutOfMemory: failed")[0] == CATEGORY_RESOURCE + assert classify_incident(log_tail="", exit_code=137)[0] == CATEGORY_RESOURCE + assert classify_incident(log_tail="no space left on device")[0] == CATEGORY_RESOURCE + category, evidence = classify_incident(log_tail="std::bad_alloc") + assert category == CATEGORY_RESOURCE and evidence + + +def test_classify_agent_and_code_and_unknown(): + assert classify_incident(log_tail="TransportClosedError('runtime closed')")[0] == CATEGORY_AGENT + assert classify_incident(log_tail="timed out after 1800.0 seconds")[0] == CATEGORY_AGENT + assert classify_incident(log_tail="Traceback (most recent call last):")[0] == CATEGORY_CODE + assert classify_incident(log_tail="everything is fine")[0] == CATEGORY_UNKNOWN + + +# --------------------------------------------------------------- policy ---- + +def test_policy_running_and_code_and_resource(): + assert decide_action(category=CATEGORY_UNKNOWN, same_class_streak=1, + waits_done=0, restarts_done=0, running=True)["action"] == "run" + assert decide_action(category=CATEGORY_CODE, same_class_streak=1, + waits_done=0, restarts_done=0, running=False)["action"] == "stop" + wait = decide_action(category=CATEGORY_RESOURCE, same_class_streak=1, + waits_done=0, restarts_done=0, running=False, + wait_seconds=1800) + assert wait["action"] == "wait" and wait["sleep_s"] == 1800 + exhausted = decide_action(category=CATEGORY_RESOURCE, same_class_streak=1, + waits_done=6, restarts_done=0, running=False) + assert exhausted["action"] == "stop" + + +def test_policy_backoff_restart_and_breaker(): + agent = decide_action(category=CATEGORY_AGENT, same_class_streak=1, + waits_done=0, restarts_done=0, running=False) + assert agent["action"] == "restart" and agent["sleep_s"] > 0 + unknown = decide_action(category=CATEGORY_UNKNOWN, same_class_streak=1, + waits_done=0, restarts_done=0, running=False) + assert unknown["action"] == "restart" + breaker = decide_action(category=CATEGORY_AGENT, same_class_streak=3, + waits_done=0, restarts_done=1, running=False) + assert breaker["action"] == "stop" and "circuit breaker" in breaker["reason"] + gave_up = decide_action(category=CATEGORY_UNKNOWN, same_class_streak=1, + waits_done=0, restarts_done=3, running=False) + assert gave_up["action"] == "stop" + + +# ----------------------------------------------------------- supervisor ---- + +class FakeClock: + def __init__(self) -> None: + self.now = 1000.0 + + def __call__(self) -> float: + return self.now + + +def _sup(tmp_path: Path, *, launcher, clock=None, target=4, + diagnose=False): + """A supervisor whose only inputs are the run's own files. + + There is no free-VRAM getter any more: FLOW.md §0 — HE holds no device + verdict, so the supervisor has nothing to ask a GPU about. + """ + slept: list = [] + sup = Supervisor( + tmp_path / "state", tmp_path / "exp", target=target, interval_s=0, + clock=clock or FakeClock(), sleeper=lambda s: slept.append(s), + resume_launcher=launcher, + diagnose_agent=diagnose, + ) + sup.state_dir.mkdir(parents=True, exist_ok=True) + sup.exp_dir.mkdir(parents=True, exist_ok=True) + sup.slept = slept # type: ignore[attr-defined] + return sup + + +def _write_run(sup: Supervisor, **fields) -> None: + (sup.state_dir / "run.json").write_text( + json.dumps({"task_id": "t", **fields}), encoding="utf-8") + + +def _pid_alive(sup: Supervisor, alive: bool) -> None: + """Register a pid that looks like a live harness_evolve process.""" + import os + import subprocess + import sys + if not alive: + (sup.state_dir / "resume.pid").write_text("99999999", encoding="utf-8") + return + # argv carries the marker the liveness check looks for + proc = subprocess.Popen( + [sys.executable, "-c", "import time; time.sleep(60)", "harness_evolve"], + ) + sup._test_proc = proc # type: ignore[attr-defined] + # a fresh child has no state writes yet: disable the stall guard here + sup.stall_seconds = 10 ** 9 + (sup.state_dir / "resume.pid").write_text(str(proc.pid), encoding="utf-8") + + +def test_step_reports_alive_and_done(tmp_path): + sups = _sup(tmp_path, launcher=lambda: 1) + _write_run(sups, current_iteration=3, finished=False) + _pid_alive(sups, True) + try: + assert sups.step()["action"] == "run" + assert sups.read_state()["status"] == "running" + + _write_run(sups, current_iteration=4, finished=True) + done = sups.step() + assert done["action"] == "done" + assert sups.read_state()["status"] == "done" + finally: + proc = getattr(sups, "_test_proc", None) + if proc is not None: + proc.kill() + proc.wait() + + +def test_step_waits_then_restarts_on_resource_incident(tmp_path): + calls: list = [] + sups = _sup(tmp_path, launcher=lambda: calls.append(1) or 4242) + _write_run(sups, current_iteration=3, finished=False) + _pid_alive(sups, False) + (sups.state_dir / "resume.log").write_text( + "hipErrorOutOfMemory: failed to allocate\n", encoding="utf-8") + + out = sups.step() + assert out["action"] == "restart" # waited out the OOM, then restarted + assert sups.slept and sups.slept[0] == 1800 + assert calls == [1] + state = sups.read_state() + assert state["status"] == "running" and state["waits_done"] == 1 + incidents = (sups.exp_dir / "incidents.jsonl").read_text(encoding="utf-8").splitlines() + kinds = [json.loads(x)["action"] for x in incidents] + assert "wait" in kinds and "restart_after_wait" in kinds + + +def test_the_restart_after_a_wait_does_not_ask_the_cards(tmp_path): + """HE holds no device verdict: the wait is the whole condition. + + The supervisor used to re-read free VRAM after the wait and keep waiting + while it looked low (``wait_again``) — that is HE managing the environment + again (FLOW.md §0). Whether a card is clean now is decided by the *child's* + own measurement gate, which waits on the child's behalf. + """ + calls: list = [] + sups = _sup(tmp_path, launcher=lambda: calls.append(1) or 4242) + _write_run(sups, current_iteration=3, finished=False) + _pid_alive(sups, False) + (sups.state_dir / "resume.log").write_text("bad_alloc\n", encoding="utf-8") + + out = sups.step() + + assert out["action"] == "restart" and calls == [1] + assert sups.read_state()["status"] == "running" + kinds = [json.loads(x)["action"] for x in + (sups.exp_dir / "incidents.jsonl").read_text( + encoding="utf-8").splitlines()] + assert "wait_again" not in kinds + + +def test_step_stops_on_deterministic_code_failure(tmp_path): + calls: list = [] + sups = _sup(tmp_path, launcher=lambda: calls.append(1) or 7) + _write_run(sups, current_iteration=3, finished=False) + _pid_alive(sups, False) + (sups.state_dir / "orchestrator.log").write_text( + "Traceback (most recent call last):\nValueError: bad config\n", + encoding="utf-8") + out = sups.step() + assert out["action"] == "stop" + assert calls == [] # never restarts a code error + assert sups.read_state()["status"] == "stopped" + + +def test_circuit_breaker_after_repeated_agent_failures(tmp_path): + calls: list = [] + sups = _sup(tmp_path, launcher=lambda: calls.append(1) or 5) + _write_run(sups, current_iteration=3, finished=False) + _pid_alive(sups, False) + (sups.state_dir / "resume.log").write_text("TransportClosedError\n", + encoding="utf-8") + assert sups.step()["action"] == "restart" + assert sups.step()["action"] == "restart" + out = sups.step() + assert out["action"] == "stop" + assert "circuit breaker" in out["reason"] + assert len(calls) == 2 # third time: stop, no restart + + +def test_stop_flag_wins(tmp_path): + sups = _sup(tmp_path, launcher=lambda: 1) + (sups.exp_dir / "supervisor.stop").write_text("stop", encoding="utf-8") + assert sups.step()["action"] == "stop" + + +def test_project_root_contains_the_package(): + from metainfer.tasks.harness_evolve.orchestrator.supervisor import Supervisor + root = Supervisor.project_root() + assert (root / "metainfer" / "tasks").is_dir() + + +def test_pid_is_live_rejects_missing_self_and_zombies(): + import os + import time + from metainfer.tasks.harness_evolve.orchestrator import supervisor as sup + assert sup.pid_is_live(None) is False + assert sup.pid_is_live(99999999) is False + assert sup.pid_is_live(os.getpid()) is False # not a harness process + + pid = os.fork() + if pid == 0: # child: exit at once + os._exit(0) + time.sleep(0.4) # now a zombie + try: + assert sup.pid_is_live(pid) is False + finally: + os.waitpid(pid, 0) + + +def test_should_launch_resume_covers_start_crash_and_target(): + from metainfer.tasks.harness_evolve.server.routes import should_launch_resume + assert should_launch_resume({}, None, False) is False + assert should_launch_resume({"finished": False}, 5, False) is True + assert should_launch_resume({"finished": False}, 5, True) is False + assert should_launch_resume({"finished": True, "current_iteration": 1}, + 5, False) is True + assert should_launch_resume({"finished": True, "current_iteration": 5}, + 5, False) is False + assert should_launch_resume({"finished": True, "current_iteration": 5}, + None, False) is False + + +def test_supervisor_restarts_when_the_child_is_a_zombie(tmp_path): + """A defunct child must not count as 'running' (the stuck-in-analyze bug).""" + import os + import time + from metainfer.tasks.harness_evolve.orchestrator.supervisor import Supervisor + state = tmp_path / "st" + exp = tmp_path / "exp" + state.mkdir() + exp.mkdir() + pid = os.fork() + if pid == 0: + os._exit(0) + time.sleep(0.4) + try: + (state / "resume.pid").write_text(str(pid), encoding="utf-8") + (state / "run.json").write_text('{"current_iteration": 1, "finished": false}', + encoding="utf-8") + calls = [] + sup = Supervisor(state, exp, target=5, interval_s=0, + resume_launcher=lambda: calls.append(1) or 4242, + sleeper=lambda s: None) + out = sup.step() + assert calls == [1], out # restarted instead of idling + assert out["action"] in {"restart", "wait", "wait_again"} + finally: + os.waitpid(pid, 0) + + +def test_progress_age_counts_child_activity(tmp_path): + """A long evaluate round must not look like a stall while children work.""" + import os + import time + from metainfer.tasks.harness_evolve.orchestrator.supervisor import Supervisor + state = tmp_path / "st" + exp = tmp_path / "exp" + state.mkdir() + exp.mkdir() + (state / "run.json").write_text('{"current_iteration": 2, "finished": false}', + encoding="utf-8") + # stale outer state + old = time.time() - 7200 + os.utime(state / "run.json", (old, old)) + (state / "timeline.jsonl").write_text("{}\n", encoding="utf-8") + os.utime(state / "timeline.jsonl", (old, old)) + + child_state = exp / "children" / "iteration_002" / "shapeA" / "state" + child_state.mkdir(parents=True) + (child_state / "timeline.jsonl").write_text("{}\n", encoding="utf-8") + + sup = Supervisor(state, exp, target=5, interval_s=0, + resume_launcher=lambda: 1, + sleeper=lambda s: None) + age = sup.progress_age() + assert age is not None and age < 60 # child write counts as progress + + # and a genuinely idle run still looks stalled + os.utime(child_state / "timeline.jsonl", (old, old)) + assert sup.progress_age() > 3600 + + +def test_janitor_sweep_recovers_unsupervised_tasks(monkeypatch, tmp_path): + """A task whose supervisor died must be picked up again by the sweep.""" + from metainfer.tasks.harness_evolve.server import routes as R + + state = tmp_path / "state" + ws = tmp_path / "ws" + state.mkdir() + ws.mkdir() + (state / "run.json").write_text('{"current_iteration": 2, "finished": false}', + encoding="utf-8") + + class Entry: + id = "t-janitor" + type = R.PLUGIN_TYPE + state_dir = str(state) + workspace_dir = str(ws) + + monkeypatch.setattr("metainfer.server.tasks.list_tasks", lambda: [Entry()]) + monkeypatch.setattr(R, "_resume_running", lambda _s: None) + monkeypatch.setattr(R, "_supervisor_running", lambda _w: None) + monkeypatch.setattr(R, "read_target_iterations", lambda _w: 5) + monkeypatch.setattr(R, "_launch_supervisor", + lambda *a, **k: {"pid": 111, "log": "x"}) + monkeypatch.setattr(R, "_launch_resume", + lambda *a, **k: {"pid": 222}) + + actions = R.janitor_sweep() + assert actions and actions[0]["action"] == "supervisor+resume" + assert actions[0]["supervisor_pid"] == 111 + assert actions[0]["resume_pid"] == 222 + + +def test_janitor_sweep_leaves_a_healthy_run_alone(monkeypatch, tmp_path): + """A live run is nobody's business — including one started by the server. + + The liveness question is ``_run_alive`` (orchestrator.pid first, then a + /proc scan). The old version of this test mocked ``_resume_running``, i.e. + it only ever described a *resume*-started task — which is why the real + production shape (a task the New Task form started, no ``resume.pid``) went + unnoticed and got a second driver. + """ + from metainfer.tasks.harness_evolve.server import routes as R + + state = tmp_path / "state" + ws = tmp_path / "ws" + state.mkdir() + ws.mkdir() + (state / "run.json").write_text('{"current_iteration": 2, "finished": false}', + encoding="utf-8") + + class Entry: + id = "t-ok" + type = R.PLUGIN_TYPE + state_dir = str(state) + workspace_dir = str(ws) + + monkeypatch.setattr("metainfer.server.tasks.list_tasks", lambda: [Entry()]) + monkeypatch.setattr(R, "_run_alive", lambda _s: 4242) + monkeypatch.setattr(R, "_supervisor_running", lambda _w: 4343) + monkeypatch.setattr(R, "read_target_iterations", lambda _w: 5) + actions = R.janitor_sweep() + assert actions and actions[0]["action"] == "skip" + assert actions[0]["reason"] == "run is alive" + + +def test_should_launch_resume_respects_terminal_statuses(): + """A run stopped on purpose (unmeasured round / abort / breaker) stays stopped. + + ``round_incomplete`` is how a blocked *child* surfaces now: the DKAO gate + that stopped it is recorded as the cause, not as HE's own device verdict. + """ + from metainfer.tasks.harness_evolve.server.routes import should_launch_resume + for status in ("round_incomplete", "aborted_by_operator", "circuit_breaker"): + assert should_launch_resume( + {"finished": True, "current_iteration": 6, "final_status": status}, + 10, False) is False + # a clean finish below target is resumed as usual + assert should_launch_resume( + {"finished": True, "current_iteration": 6, "final_status": "success"}, + 10, False) is True + + +def test_task_process_alive_finds_a_server_started_run(tmp_path, monkeypatch): + """A task launched by the server (no resume.pid) still counts as running.""" + import os + import subprocess + import sys + from metainfer.tasks.harness_evolve.orchestrator import supervisor as sup + + state = tmp_path / "state" + state.mkdir() + assert sup.task_process_alive(state) is None # nothing running + + # a process whose cmdline mentions this task's state dir and the CLI + proc = subprocess.Popen( + [sys.executable, "-c", "import time; time.sleep(30)", + "harness_evolve.orchestrator.cli", str(state)], + ) + try: + assert sup.task_process_alive(state) == proc.pid + finally: + proc.kill() + proc.wait() + + # a stale pid file (dead process) is not "alive" + (state / "resume.pid").write_text("99999999", encoding="utf-8") + assert sup.task_process_alive(state) is None + + +# ------------------------------------------------- live children / stalls ---- + +def _fake_proc(tmp_path, pid, cmdline, state="S"): + root = tmp_path / "proc" + entry = root / str(pid) + entry.mkdir(parents=True, exist_ok=True) + (entry / "cmdline").write_bytes(cmdline.replace(" ", "\0").encode() + b"\0") + (entry / "stat").write_text(f"{pid} (python3) {state} 1 1 1 0 -1 4194560\n") + return root + + +def test_live_child_count_reads_processes_not_mtimes(tmp_path): + """A silent-but-alive child counts as progress; other rounds do not.""" + from metainfer.tasks.harness_evolve.orchestrator.supervisor import Supervisor + + state = tmp_path / "st" + exp = tmp_path / "exp" + state.mkdir() + exp.mkdir() + (state / "run.json").write_text('{"current_iteration": 2, "finished": false}', + encoding="utf-8") + sup = Supervisor(state, exp, target=5, interval_s=0, + resume_launcher=lambda: 1, + sleeper=lambda s: None) + + root = _fake_proc(tmp_path, 4242, + "python3 -m metainfer.tasks.dcu_kernel_auto_opt." + "orchestrator.cli run /w/children/iteration_002/shapeA") + # a child of another iteration must not count for this one + _fake_proc(tmp_path, 4243, + "python3 -m metainfer.tasks.dcu_kernel_auto_opt.orchestrator.cli" + " run /w/children/iteration_001/shapeB") + # a zombie is not a live child + _fake_proc(tmp_path, 4244, + "python3 -m metainfer.tasks.dcu_kernel_auto_opt.orchestrator.cli" + " run /w/children/iteration_002/shapeC", state="Z") + # something else entirely + _fake_proc(tmp_path, 4245, "python3 -m http.server") + + assert sup.live_child_count(root) == 1 + + +def test_a_silent_live_child_blocks_the_stall_guard(tmp_path): + """The regression that produced two resumes: 40 min of quiet child work.""" + import os + import time + from metainfer.tasks.harness_evolve.orchestrator.supervisor import Supervisor + + state = tmp_path / "st" + exp = tmp_path / "exp" + state.mkdir() + exp.mkdir() + (state / "run.json").write_text('{"current_iteration": 2, "finished": false}', + encoding="utf-8") + old = time.time() - 7200 + for name in ("run.json", "timeline.jsonl"): + path = state / name + if not path.exists(): + path.write_text("{}", encoding="utf-8") + os.utime(path, (old, old)) + (exp / "children" / "iteration_002").mkdir(parents=True) + + calls = [] + sup = Supervisor(state, exp, target=5, interval_s=0, stall_seconds=60, + resume_launcher=lambda: calls.append(1) or 4242, + sleeper=lambda s: None) + assert sup.process_alive.__self__ is sup # sanity: bound to this run + sup.process_alive = lambda: True # a live (quiet) orchestrator + sup.live_child_count = lambda *a, **k: 1 # ...with a live child + + out = sup.step() + + assert out["action"] == "run", out + assert calls == [] # no duplicate resume + actions = [json.loads(l)["action"] for l in + (exp / "incidents.jsonl").read_text(encoding="utf-8").splitlines()] + assert "stall_ignored" in actions + + +def test_terminating_a_stale_run_kills_its_process_group(tmp_path): + import os + import time + from metainfer.tasks.harness_evolve.orchestrator.supervisor import Supervisor + + state = tmp_path / "st" + exp = tmp_path / "exp" + state.mkdir() + exp.mkdir() + sup = Supervisor(state, exp, target=5, interval_s=0, + resume_launcher=lambda: 1, + sleeper=lambda s: time.sleep(min(s, 0.05))) + + pid = os.fork() + if pid == 0: # child: own session, then idle + try: + os.setsid() + time.sleep(60) + finally: + os._exit(0) + + time.sleep(0.3) + try: + killed = sup.terminate_stale_run([pid]) + assert killed == [pid] + deadline = time.time() + 5 + while time.time() < deadline: + try: + wpid, _ = os.waitpid(pid, os.WNOHANG) + except ChildProcessError: + break + if wpid == pid: + break + time.sleep(0.1) + assert not Path(f"/proc/{pid}").exists() or \ + Path(f"/proc/{pid}/stat").read_text().split(") ", 1)[1].startswith("Z") + finally: + try: + os.kill(pid, 9) + except OSError: + pass + try: + os.waitpid(pid, os.WNOHANG) + except ChildProcessError: + pass diff --git a/metainfer/tasks/harness_evolve/tests/test_variant_rounds.py b/metainfer/tasks/harness_evolve/tests/test_variant_rounds.py new file mode 100644 index 00000000..ec2924d6 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tests/test_variant_rounds.py @@ -0,0 +1,414 @@ +"""Variant-driven rounds: random operators, a fixed generalization paper, and +the two counter rules (>=75% / >=50% wins with an 80%-of-variant floor).""" + +from __future__ import annotations + +import json +import random +from pathlib import Path + +import pytest + +from ..orchestrator.decision_engine import gate_verdict +from ..orchestrator.rounds import ( + evaluate_gate, record_paper, regime_of, select_round_questions, + sample_operators, +) +from ..orchestrator.variant import ( + load_variant_table, record_variants, reference_results, resolve_variant, +) + + +def _pool(n: int = 12) -> dict: + """Alternating decode (M=16) / prefill (M=4096) operators with variants.""" + out = {} + for i in range(n): + m = 16 if i % 2 == 0 else 4096 + regime = "decode" if m == 16 else "prefill" + out[f"op{i:02d}"] = { + "baseline_us": 100.0 * (i + 1), + "best_known_us": 40.0 * (i + 1), # an existing variant + "family": f"{regime}__fam{i}", + "contract": {"M": m, "N": 64, "K": 32}, + } + return out + + +def _table() -> dict: + return {"operators": {}, "groups": {}, "harness_version": None} + + +# ------------------------------------------------------------------ selection + +def test_first_round_defines_the_generalization_paper(): + pool, table = _pool(), _table() + picked = select_round_questions(iteration=1, table=table, instances=pool, + state={}, rng=random.Random(7)) + assert len(picked["performance_ids"]) == 4 + # round 1's operators are the paper, and they are what this round measures + assert picked["generalization_ids"] == picked["performance_ids"] + assert picked["defines_paper"] is True + + +def test_later_rounds_draw_new_operators_and_reuse_the_paper(): + pool, table = _pool(), _table() + first = select_round_questions(iteration=1, table=table, instances=pool, + state={}, rng=random.Random(1)) + groups = record_paper(table, iteration=1, + generalization_ids=first["generalization_ids"], + performance_ids=first["performance_ids"]) + second = select_round_questions(iteration=2, table=table, instances=pool, + state={"groups": groups}, + rng=random.Random(2)) + assert set(second["performance_ids"]).isdisjoint(first["generalization_ids"]) + assert second["generalization_ids"] == first["generalization_ids"] + assert second["defines_paper"] is False + + +def test_operators_without_any_reference_are_not_sampled(): + pool = {"known": {"baseline_us": 10.0}, "unknown": {}} + assert sample_operators(pool, count=2, table=_table()) == ["known"] + + +# ---------------------------------------------------------------- the gates + +def _reference(ids, table, pool): + return reference_results(table, ids, instances=pool) + + +def test_performance_gate_needs_three_of_four_wins(): + pool, table = _pool(), _table() + ids = ["op00", "op01", "op02", "op03"] + ref = _reference(ids, table, pool) # 40/80/120/160 us variants + candidate = { + "op00": {"median_us": 30.0, "correctness_ok": True, "status": "success"}, + "op01": {"median_us": 70.0, "correctness_ok": True, "status": "success"}, + "op02": {"median_us": 110.0, "correctness_ok": True, "status": "success"}, + "op03": {"median_us": 170.0, "correctness_ok": True, "status": "success"}, + } + gate = evaluate_gate(kind="performance", operator_ids=ids, results=candidate, + table=table, instances=pool) + assert gate["status"] == "PASS" and len(gate["wins"]) == 3 + assert gate["wins_needed"] == 3 and gate["required_win_ratio"] == 0.75 + + # only two wins -> fail + candidate["op02"] = {"median_us": 130.0, "correctness_ok": True, + "status": "success"} + gate = evaluate_gate(kind="performance", operator_ids=ids, results=candidate, + table=table, instances=pool) + assert gate["status"] == "FAIL" and "2/4 operators" in gate["reasons"][-1] + + +def test_a_winning_round_still_fails_if_one_operator_collapses(): + pool, table = _pool(), _table() + ids = ["op00", "op01", "op02", "op03"] + candidate = { + "op00": {"median_us": 30.0, "correctness_ok": True, "status": "success"}, + "op01": {"median_us": 70.0, "correctness_ok": True, "status": "success"}, + "op02": {"median_us": 110.0, "correctness_ok": True, "status": "success"}, + # variant is 160us; 140us is only 87% ... 220us is 72% (below the floor) + "op03": {"median_us": 220.0, "correctness_ok": True, "status": "success"}, + } + gate = evaluate_gate(kind="performance", operator_ids=ids, results=candidate, + table=table, instances=pool) + assert gate["status"] == "FAIL" + assert gate["below_floor"] == ["op03"] + + +def test_generalization_gate_needs_only_two_of_four(): + pool, table = _pool(), _table() + ids = ["op00", "op01", "op02", "op03"] + candidate = { + "op00": {"median_us": 30.0, "correctness_ok": True, "status": "success"}, + "op01": {"median_us": 70.0, "correctness_ok": True, "status": "success"}, + "op02": {"median_us": 140.0, "correctness_ok": True, "status": "success"}, + "op03": {"median_us": 180.0, "correctness_ok": True, "status": "success"}, + } + gate = evaluate_gate(kind="generalization", operator_ids=ids, + results=candidate, table=table, instances=pool) + assert gate["status"] == "PASS" and gate["wins_needed"] == 2 + assert gate["required_win_ratio"] == 0.5 + + candidate["op01"] = {"median_us": 90.0, "correctness_ok": True, + "status": "success"} + gate = evaluate_gate(kind="generalization", operator_ids=ids, + results=candidate, table=table, instances=pool) + assert gate["status"] == "FAIL" + + +def test_a_retake_that_reproduces_the_same_kernels_still_holds_the_line(): + """Ties are not losses in the generalization gate. + + The retake re-measures the paper on the same harness, and DKAO warm-starts + from the kernel the baseline round put in the pool — so re-finding that exact + kernel (delta 0.0%) is the expected outcome, not a failure. Scoring it as a + loss would demand that a *search policy* out-optimize its own best result + every single round. + """ + pool, table = _pool(), _table() + ids = ["op00", "op01", "op02", "op03"] + variants = {"op00": 40.0, "op01": 80.0, "op02": 120.0, "op03": 160.0} + candidate = {iid: {"median_us": variants[iid], "correctness_ok": True, + "status": "success"} for iid in ids} + + gate = evaluate_gate(kind="generalization", operator_ids=ids, + results=candidate, table=table, instances=pool) + assert gate["tie_policy"] == "neutral" + assert gate["ties"] == ids and gate["wins"] == [] and gate["losses"] == [] + assert gate["status"] == "PASS", gate["reasons"] + + # ... but the floor still bites: one operator genuinely slower than its + # variant fails the retake even with ties scored as holding the line + candidate["op03"] = {"median_us": 400.0, "correctness_ok": True, + "status": "success"} + gate = evaluate_gate(kind="generalization", operator_ids=ids, + results=candidate, table=table, instances=pool) + assert gate["status"] == "FAIL" + assert gate["losses"] == ["op03"] and gate["below_floor"] == ["op03"] + + +def test_the_performance_gate_still_counts_a_tie_as_a_loss(): + """Fresh operators must genuinely win: a tie proves nothing there.""" + pool, table = _pool(), _table() + ids = ["op00", "op01", "op02", "op03"] + variants = {"op00": 40.0, "op01": 80.0, "op02": 120.0, "op03": 160.0} + candidate = {iid: {"median_us": variants[iid], "correctness_ok": True, + "status": "success"} for iid in ids} + gate = evaluate_gate(kind="performance", operator_ids=ids, + results=candidate, table=table, instances=pool) + assert gate["tie_policy"] == "loss" + assert gate["status"] == "FAIL" + assert "no operator beat its variant" in " ".join(gate["reasons"]) + + +def test_unmeasured_operator_is_never_counted_as_a_loss(): + """Three wins and one missing number is *not* a verdict about the harness. + + It used to be FAIL, which both blamed the harness for a number the machine + never produced and (on the retake) spent the candidate on it. The round now + carries ``INCOMPLETE``: no verdict, re-measured later. + """ + pool, table = _pool(), _table() + ids = ["op00", "op01", "op02", "op03"] + candidate = { + "op00": {"median_us": 30.0, "correctness_ok": True, "status": "success"}, + "op01": {"median_us": 70.0, "correctness_ok": True, "status": "success"}, + "op02": {"median_us": 110.0, "correctness_ok": True, "status": "success"}, + # op03 never reported: the round is not evidence + } + gate = evaluate_gate(kind="performance", operator_ids=ids, results=candidate, + table=table, instances=pool) + assert gate["status"] == "INCOMPLETE" + assert "op03" in gate["unmeasured"] + assert gate["wins_needed"] == 3 # wins already enough to pass... + assert gate["wins_needed_over_requested"] == 3 # ... over the drawn round + assert any("op03" in reason for reason in gate["reasons"]) + + +def test_a_gate_that_cannot_pass_without_the_missing_question_is_a_fail(): + """Decisive losses are still decided now: no relaunch needed to know.""" + pool, table = _pool(), _table() # variants 40/80/120/160 us + ids = ["op00", "op01", "op02", "op03"] + candidate = { + # ~1% slower than each variant: inside the noise band, i.e. a tie → a + # loss for the performance gate, but nowhere near the 80% floor. + "op00": {"median_us": 40.4, "correctness_ok": True, "status": "success"}, + "op01": {"median_us": 80.8, "correctness_ok": True, "status": "success"}, + "op02": {"median_us": 121.2, "correctness_ok": True, "status": "success"}, + # op03 never reported + } + gate = evaluate_gate(kind="performance", operator_ids=ids, results=candidate, + table=table, instances=pool) + assert gate["status"] == "FAIL", gate + assert gate["below_floor"] == [] # the floor is not what decided it + assert any("cannot pass" in reason for reason in gate["reasons"]) + + +def test_a_question_below_the_floor_is_a_fail_even_with_others_missing(): + pool, table = _pool(), _table() + ids = ["op00", "op01", "op02", "op03"] + candidate = { + "op00": {"median_us": 30.0, "correctness_ok": True, "status": "success"}, + "op01": {"median_us": 200.0, "correctness_ok": True, "status": "success"}, + # way below the 80% floor of its 80us variant + "op02": {"median_us": 110.0, "correctness_ok": True, "status": "success"}, + # op03 never reported + } + gate = evaluate_gate(kind="performance", operator_ids=ids, results=candidate, + table=table, instances=pool) + assert gate["status"] == "FAIL" + assert "op01" in gate["below_floor"] + + +# ------------------------------------------------------- variant table rules + +def test_variant_table_moves_only_when_a_harness_is_promoted(tmp_path): + pool = _pool() + table = load_variant_table(tmp_path, pool_path=None) + table["operators"] = {} + # a rejected round measured something faster, but nothing is promoted + before = table["operators"].get("op00") + result = record_variants(tmp_path, table, {}, version="h-2", iteration=1) + assert result["updated"] == [] and table["operators"] == {} + + # the approved candidate's numbers become the new variants + outcome = record_variants( + tmp_path, table, + {"op00": {"median_us": 11.0, "p90_us": 11.2, "correctness_ok": True, + "repo_path": "/repos/op00"}}, + version="h-2", iteration=1, + groups={"generalization_ids": ["op00"]}) + assert outcome["updated"] == ["op00"] + resolved = resolve_variant(table, "op00", instances=pool) + assert resolved["median_us"] == 11.0 and resolved["source"] == "variant" + assert resolved["harness_version"] == "h-2" + history = [json.loads(l) for l in + (tmp_path / "variant_history.jsonl").read_text().splitlines()] + assert history[-1]["harness_version"] == "h-2" + assert json.loads((tmp_path / "variant_table.json").read_text() + )["harness_version"] == "h-2" + + +def test_a_slower_measurement_never_regresses_a_variant(tmp_path): + pool = _pool() + table = {"operators": {}, "groups": {}} + record_variants(tmp_path, table, + {"op00": {"median_us": 30.0, "correctness_ok": True}}, + version="h-2", iteration=1) + record_variants(tmp_path, table, + {"op00": {"median_us": 45.0, "correctness_ok": True}}, + version="h-3", iteration=2) + assert resolve_variant(table, "op00", instances=pool)["median_us"] == 30.0 + + +def test_reference_falls_back_to_pool_best_known_then_baseline(tmp_path): + pool = {"best": {"baseline_us": 100.0, "best_known_us": 40.0}, + "baseonly": {"baseline_us": 55.0}, + "nothing": {}} + table = {"operators": {}, "groups": {}} + assert resolve_variant(table, "best", instances=pool)["source"] == "best_known" + assert resolve_variant(table, "baseonly", instances=pool)["source"] == "baseline" + assert resolve_variant(table, "nothing", instances=pool) is None + + +def test_question_count_is_rounded_to_a_multiple_of_the_gpu_count(): + """A wave must be able to fill all 4 HCUs: counts snap to multiples of 4.""" + from ..orchestrator.rounds import OPERATORS_PER_GATE, normalized_per_gate + + assert OPERATORS_PER_GATE == 4 + assert normalized_per_gate(None) == 4 # default + assert normalized_per_gate("") == 4 + assert normalized_per_gate(0) == 4 + assert normalized_per_gate("bad") == 4 + assert normalized_per_gate(4) == 4 + assert normalized_per_gate(5) == 8 # rounds up, never down + assert normalized_per_gate(7) == 8 + assert normalized_per_gate(8) == 8 + assert normalized_per_gate(12) == 12 + + +def test_eight_question_round_uses_eight_operators_in_both_gates(): + pool, table = _pool(20), _table() + picked = select_round_questions(iteration=1, table=table, instances=pool, + state={}, count=8, rng=random.Random(3)) + assert len(picked["performance_ids"]) == 8 + assert picked["generalization_ids"] == picked["performance_ids"] + + groups = record_paper(table, iteration=1, + generalization_ids=picked["generalization_ids"], + performance_ids=picked["performance_ids"]) + second = select_round_questions(iteration=2, table=table, instances=pool, + state={"groups": groups}, count=8, + rng=random.Random(4)) + assert len(second["performance_ids"]) == 8 + assert set(second["performance_ids"]).isdisjoint(picked["performance_ids"]) + + +def test_eight_question_gates_need_six_and_four_wins(): + pool, table = _pool(20), _table() + ids = [f"op{i:02d}" for i in range(8)] + base = {iid: 100.0 for iid in ids} + reference = {iid: {"median_us": base[iid], "status": "reference", + "correctness_ok": True} for iid in ids} + # 6 wins, 2 small losses -> performance gate passes (75% of 8 = 6) + candidate = {iid: {"median_us": 90.0, "correctness_ok": True, + "status": "success"} for iid in ids} + for iid in ids[6:]: + candidate[iid]["median_us"] = 105.0 + perf = gate_verdict(reference, candidate, kind="performance") + assert perf["wins_needed"] == 6 and perf["status"] == "PASS" + # 4 wins -> generalization gate passes (50% of 8 = 4), performance does not + for iid in ids[4:]: + candidate[iid]["median_us"] = 105.0 + assert gate_verdict(reference, candidate, kind="performance")["status"] == "FAIL" + gen = gate_verdict(reference, candidate, kind="generalization") + assert gen["wins_needed"] == 4 and gen["status"] == "PASS" + + +def test_round_size_follows_the_request_and_never_shrinks_to_the_cards(): + """A round is as big as the operator asked for, whatever the cards are doing. + + ``harness_evolve`` manages no GPU occupancy: it hands every question to a + DKAO child and the child's own admission gate waits for a clean device. So + there is nothing to shrink the round *to* — no "free devices" number, no + ``fit_count_to_devices`` — and a busy machine can no longer silently turn a + four-question round into a two-question one. + """ + from ..orchestrator.rounds import ( + GPU_COUNT, device_for_index, normalized_per_gate, + ) + + # the requested count is honoured, only rounded up to a whole wave of 4 + assert normalized_per_gate(4) == 4 + assert normalized_per_gate(8) == 8 + assert normalized_per_gate(5) == 8 + + # one question per device, repeating cyclically: 8 questions = 2 waves + assert [device_for_index(i) for i in range(8)] == [0, 1, 2, 3, 0, 1, 2, 3] + assert GPU_COUNT == 4 + + +def test_no_gpu_lease_or_occupancy_logic_is_left_in_the_round_path(): + """The HE round path must not import the broker or probe the cards.""" + from ..orchestrator import rounds as rounds_mod + from ..orchestrator.pipeline import _truthy # still used for other flags + + assert _truthy(None, default=False) is False # opt-in, not opt-out + assert not hasattr(rounds_mod, "devices_free_now") + assert not hasattr(rounds_mod, "fit_count_to_devices") + + import ast + import inspect + import io + import tokenize as _tok + + from ..orchestrator.adapters import eval as eval_mod + + # Judge the code, not the comments/docstrings that document what was removed. + source = inspect.getsource(eval_mod) + pieces = [] + for token in _tok.generate_tokens(io.StringIO(source).readline): + if token.type in (_tok.COMMENT, _tok.STRING): + continue + pieces.append(token.string) + code = " ".join(pieces) + for forbidden in ("gpu_broker", "PRIORITY_HARNESS", "gpu_gate", + "measurement_suspect", "PREFERRED_DEVICES"): + assert forbidden not in code, forbidden + assert "_evaluate_scheduled" not in code + assert "_start_contention_watch" not in code + # the gate verdict lives in the child's own measurement_gate.jsonl + tree = ast.parse(source) + names = {n.id for n in ast.walk(tree) if isinstance(n, ast.Name)} + names |= {n.attr for n in ast.walk(tree) if isinstance(n, ast.Attribute)} + assert "measurement_suspect" not in names + + +def test_a_three_device_round_draws_three_operators_with_both_regimes(): + pool, table = _pool(20), _table() + picked = select_round_questions(iteration=1, table=table, instances=pool, + state={}, count=3, rng=random.Random(11)) + assert len(picked["performance_ids"]) == 3 + assert picked["generalization_ids"] == picked["performance_ids"] + regimes = {regime_of(pool[i]) for i in picked["performance_ids"]} + assert regimes == {"decode", "prefill"} diff --git a/metainfer/tasks/harness_evolve/tools/__init__.py b/metainfer/tasks/harness_evolve/tools/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/metainfer/tasks/harness_evolve/tools/recover_evolve.py b/metainfer/tasks/harness_evolve/tools/recover_evolve.py new file mode 100644 index 00000000..54e15cd0 --- /dev/null +++ b/metainfer/tasks/harness_evolve/tools/recover_evolve.py @@ -0,0 +1,130 @@ +"""Recover a crashed harness_evolve experiment. + +When the outer loop dies after an iteration's evaluation (e.g. the evolve +turn timed out before the robustness fix), the value of that iteration is +already on disk: benchmark results, decision and best_ever. This tool runs +the MISSING evolve step for such an iteration, writes the evolve artifacts, +commits the workspace generation and finishes the report - i.e. it performs +the tail of ``pipeline.run_experiment`` without re-running any DKAO work. + +Usage:: + + PYTHONPATH=/root/zth_agent/MetaInfer python3 -m \ + metainfer.tasks.harness_evolve.tools.recover_evolve \ + --state-dir DIR --workspace-dir DIR [--iteration N] +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any, Dict, List, Optional + +from ..orchestrator.attribution import save_json +from ..orchestrator.config import load_experiment_config +from ..orchestrator.evolve import DEFAULT_EVOLVE_MODEL, build_evolver +from ..orchestrator import pipeline as P +from ..orchestrator.round_plan import RoundPlan + + +def _load_json(path: Path) -> Optional[Dict[str, Any]]: + try: + value = json.loads(path.read_text(encoding="utf-8")) + except (OSError, ValueError): + return None + return value if isinstance(value, dict) else None + + +def _candidate_iteration(exp: Path, wanted: Optional[int]) -> Path: + runs = exp / "runs" + dirs = sorted(d for d in runs.glob("iteration_*") if d.is_dir()) + if wanted is not None: + target = runs / f"iteration_{wanted:03d}" + if not target.is_dir(): + raise SystemExit(f"iteration {wanted} not found under {runs}") + return target + for d in reversed(dirs): + has_decision = (d / "input" / "decision.json").is_file() + has_evolve = (d / "evolve" / "change_manifest.json").is_file() + if has_decision and not has_evolve: + return d + raise SystemExit("no iteration with a completed evaluation and missing " + "evolve artifacts; nothing to recover") + + +def main(argv: Optional[List[str]] = None) -> int: + ap = argparse.ArgumentParser(prog="recover_evolve") + ap.add_argument("requirements", type=Path) + ap.add_argument("--state-dir", type=Path, required=True) + ap.add_argument("--workspace-dir", type=Path, required=True) + ap.add_argument("--iteration", type=int, default=None) + args = ap.parse_args(argv) + + cfg = load_experiment_config(args.requirements, args.state_dir, + args.workspace_dir) + exp = cfg.exp_dir + it_dir = _candidate_iteration(exp, args.iteration) + iteration = int(it_dir.name.split("_")[-1]) + print(f"[recover] iteration {iteration} at {it_dir}") + + input_dir = it_dir / "input" + results = ((_load_json(input_dir / "benchmark" / "results.json") or {}) + .get("results") or {}) + decision = _load_json(input_dir / "decision.json") or {} + diff = _load_json(input_dir / "diff.json") or {} + scored = _load_json(input_dir / "benchmark" / "scored_ids.json") or [] + selected = list(scored) or P._load_ids(exp / P.LAST_SCORED_FILE) + + pool = P._load_pool(cfg) + pool_ids = pool.all_ids() if pool is not None else list(selected) + + evolver = build_evolver( + str(cfg.answers.get("evolve_mode") or "dry-run"), + model=str(cfg.answers.get("evolve_model") + or DEFAULT_EVOLVE_MODEL), + timeout_seconds=float(cfg.answers.get("evolve_timeout_seconds") + or 3600), + ) + P._record_state(cfg, P.EVOLVE, iteration, event="evolve_start_recover") + + live_workspace = exp / "workspace" + evolve_out = evolver.apply(live_workspace, iteration, { + "overview": str(input_dir / "analysis" / "overview.md"), + "results": results, + "diff": diff, + "decision": decision, + "selected": selected, + "pool_ids": pool_ids, + }) + + evolve_dir = it_dir / "evolve" + evolve_dir.mkdir(parents=True, exist_ok=True) + revision = P._commit_workspace(live_workspace, iteration) + evolve_out["manifest"]["workspace_revision"] = revision + save_json(evolve_dir / "change_manifest.json", evolve_out["manifest"]) + (evolve_dir / "evolve_summary.md").write_text( + evolve_out["summary"] + "\n", encoding="utf-8") + if pool is not None: + agent_plan = evolve_out.get("round_plan") + if isinstance(agent_plan, dict): + plan_obj = RoundPlan.from_dict(agent_plan) + pool.assert_in_pool(plan_obj.selected) + else: + plan_obj = RoundPlan( + iteration=iteration, strategy="balanced", + rationale="recover: keep same scored set", + selected=list(selected), + ) + save_json(evolve_dir / "round_plan.json", plan_obj.to_dict()) + P._record_state(cfg, P.REPORT, event="experiment_report_recover") + report = P._final_report(cfg) + P._record_state(cfg, P.FINISHED, event="experiment_finished_recover") + print(f"[recover] evolve manifest -> {evolve_dir / 'change_manifest.json'}") + print(f"[recover] report -> {report}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main())