Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions TASK_DETAILS.md
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,15 @@ We welcome new engineering problem ideas — even without complete verification
<td><code>EV2GymSmartCharging</code></td>
<td>Upstream-aligned EV smart charging scheduling</td>
</tr>
<tr>
<td rowspan="2"><b>ContinuousCasting</b></td>
<td><code>CuttingOptimization</code></td>
<td>Cut a continuously cast billet into pieces to minimize scrapped length and match a customer target (offline; the optimum is reachable)</td>
</tr>
<tr>
<td><code>CuttingOptimizationOnline</code></td>
<td>Online (closed-loop) cutting where 0.8 m scrap segments are revealed only within a reveal horizon — info asymmetry empirically keeps agents below the clairvoyant optimum (observed, not a proven lower bound)</td>
</tr>
<tr>
<td><b>AdditiveManufacturing</b></td>
<td><code>DiffSimThermalControl</code></td>
Expand Down
9 changes: 9 additions & 0 deletions TASK_DETAILS_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -346,6 +346,15 @@ Frontier-Eng 目前已覆盖以下领域的任务。每个任务均配有可运
<td><code>EV2GymSmartCharging</code></td>
<td>上游对齐的电动车智能充电调度</td>
</tr>
<tr>
<td rowspan="2"><b>ContinuousCasting</b></td>
<td><code>CuttingOptimization</code></td>
<td>把连续浇铸的钢坯切成成品,最小化报废并贴近客户目标值(离线;最优可达)</td>
</tr>
<tr>
<td><code>CuttingOptimizationOnline</code></td>
<td>在线(闭环)切割:0.8m 报废段只在揭示提前量内才告知 agent——信息不对称使 agent(经验上)低于全知最优(观察结果,非严格下界证明)</td>
</tr>
<tr>
<td><b>AdditiveManufacturing</b></td>
<td><code>DiffSimThermalControl</code></td>
Expand Down
2 changes: 2 additions & 0 deletions benchmarks/ContinuousCasting/CuttingOptimization/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
__pycache__/
*.pyc
149 changes: 149 additions & 0 deletions benchmarks/ContinuousCasting/CuttingOptimization/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
# CuttingOptimization: Continuous-Casting Cutting Optimization — offline/static (Frontier-Eng Benchmark)

An **original** Frontier-Engineering benchmark inspired by the CUMCM 2021 Problem D
(«连铸切割的在线优化»), formalized into a self-contained, deterministic optimization task.

A continuously cast steel billet is drawn at a fixed speed. Defects (crystalline-moulder
anomalies) create 0.8 m scrap segments inside the billet that must be cut out and scrapped.
A solver receives the billet length, the defect positions, and a customer target length with
an acceptance window, and must produce a **cutting plan** (a list of cut lengths that exactly
partition the whole billet) that minimizes the total scrapped length and then makes every
shipped piece as close to the target length as possible.

The full game rules and evaluation semantics are in [Task.md](./Task.md) (Chinese).

## Layout

```
benchmarks/ContinuousCasting/CuttingOptimization/
├── baseline/solver.py # Candidate solver (EVOLVE-BLOCK region is the only editable part)
├── verification/
│ ├── generator.py # Fixed-seed instance generator (defects + target window)
│ ├── simulator.py # Scoring simulator (validate + scrap / penalty metric)
│ ├── evaluate.py # Evaluation entry (subprocess + time budget + scoring)
│ ├── validator.py # Integrity checks (static + env stripping + determinism)
│ ├── ref_solver.py # Reference DP (1-D partition; documented "best" score)
│ ├── test_simulator.py # Unit tests: simulator correctness
│ ├── test_generator.py # Unit tests: deterministic / defect feasibility / headroom
│ ├── test_ref_solver.py # Unit tests: reference-solver validity + optimality
│ ├── test_validator.py # Unit tests: integrity checks / env stripping / determinism
│ ├── test_evaluator.py # Unit tests: end-to-end evaluation behavior
│ ├── data/instances/ # 8 fixed instances (seed-fixed, reproducible)
│ ├── docker/Dockerfile # Minimal stdlib-only python image
│ └── requirements.txt
├── frontier_eval/ # UnifiedTask metadata
├── Task.md # Task rules, interface, scoring, reference scores
└── README.md
```

## Requirements

- Python >= 3.10, standard library only (no third-party dependencies).
- Runtime is pure-Python; the reference DP and the baseline solver evaluate in well under a
second per instance.

## Run

```powershell
# Score a solver on the fixed 8-instance set (default 60s time budget per instance)
python verification/evaluate.py baseline/solver.py

# Add runtime-generated instances (anti-hardcoding)
python verification/evaluate.py baseline/solver.py --generate-seed <SEED>

# Tighter budget (challenge tier: 10s)
python verification/evaluate.py baseline/solver.py --time-budget 10
```

### Docker

The evaluator is pure stdlib, so a minimal `python` image suffices. Build it and use the
unified runtime's `isolation_mode=docker`:

```bash
# Build (from the repo root)
docker build -t cutting-opt-benchmark -f benchmarks/ContinuousCasting/CuttingOptimization/verification/docker/Dockerfile benchmarks/ContinuousCasting/CuttingOptimization

# Score the baseline under docker isolation (WSL/Linux; Windows hosts are limited by a
# framework path bug). docker mode must NOT set task.runtime.shell; set DOCKER_USER so the
# container can write the WSL /tmp sandbox.
FRONTIER_EVAL_UNIFIED_DOCKER_USER=1000:1000 \
.venvs/frontier-eval-driver-wsl/bin/python -m frontier_eval task=unified \
task.benchmark=ContinuousCasting/CuttingOptimization algorithm=openevolve algorithm.iterations=0 \
llm.timeout=600 task.runtime.isolation_mode=docker task.runtime.docker_image=cutting-opt-benchmark
```

**Verified (WSL, docker isolation)**: baseline `combined_score=72.49, valid=1.0, num_instances=8`
— identical to process mode. `docker` scoring works under WSL/Linux because `eval_command.txt`
injects `FRONTIER_EVAL_UNIFIED_SOURCE_BENCHMARK_DIR={benchmark_source}`, which the container
resolves to the mounted repo path (no framework env-forwarding needed).

## Tests

```powershell
# From the task directory (stdlib unittest, no dependencies)
python -m unittest discover -s verification -p "test_*.py"
```

35 tests across six modules (simulator / generator / ref_solver / validator / evaluator /
sandbox evaluator):
piece-level scrap & penalty rules, feasibility checks (sum, length window, defect isolation),
determinism of generation and reference solver, reference-solver optimality vs the baseline,
validator integrity (EVOLVE-BLOCK / forbidden references / absolute paths / per-instance
hardcoding / env stripping / determinism probe), evaluator behavior (scoring, runtime
generation, cheating-candidate rejection), and the `frontier_eval/evaluator.py` sandbox entry
(consistency with `verification/evaluate.py` + cheat rejection). `verification/multiseed_stat.py`
computes multi-run mean ± std.

## Integrity / threat model

- **Runtime-generated instances**: with `CUTTING_EVAL_GENERATE_SEED` set, the evaluator
generates fresh instances at evaluation time (temp dir, never in the repo/sandbox), so a
candidate cannot pre-position solutions for them.
- **Candidate env stripping**: candidate subprocesses get `FRONTIER_*` / `CUTTING_EVAL_*`
variables stripped (see `verification/validator.py`), closing the host-env side channel.
- **Static checks**: EVOLVE-BLOCK markers + fixed-region byte diff vs the initial baseline,
forbidden imports of evaluation / generation / reference modules, absolute paths,
per-instance hardcoding, plus a determinism probe (two runs must match). Any violation
scores 0.
- **Sandbox scope**: the 8 fixed instances and the evaluator / validator sources are visible
to the candidate during evolution (they are needed for scoring and `verification/simulator.py`
is intentionally usable as a white-box scorer). Anti-hardcoding therefore relies on
`CUTTING_EVAL_GENERATE_SEED` (fresh instances at evaluation time — set a seed, do not use a
fixed one); the name-keyed hardcoding check is best-effort. `verification/ref_solver.py` and
`verification/generator.py` are **not** copied into the sandbox and are additionally forbidden
by the validator.
- Honest note: in process mode the candidate has host filesystem access (framework-wide
limitation); this benchmark relies on the layered defenses above.

## Scoring

- Instances = 8 fixed (difficulties easy/medium/hard, S = 24..150 m, 0..6 defects, target
window ±0.5 m around the customer target) + runtime-generated when `CUTTING_EVAL_GENERATE_SEED`
is set.
- **Metric**: material utilization = `100 * (billet_length - (scrap + 1e-4*penalty)) / billet_length`,
averaged over instances (0..100, higher is better). `scrap` = total scrapped length (defect
pieces + sub-8.0 m pieces + over-window excess); `penalty = Σ|delivered − target|` over shipped
pieces; the `1e-4` weight is so small that scrap strictly dominates, respecting the lexicographic
objective of the original problem (the penalty only breaks ties among equal-scrap plans).
- Malformed output / out-of-range cuts / cuts that fail to isolate a defect / crash / timeout
⇒ 0 points for that instance.
- **Headroom guarantee**: the generator accepts only instances where the reference DP strictly
beats a naive equal-split baseline by ≥ 0.1 m of scrap, so every instance has real
optimization signal.
- Reference scores (verified on the fixed 8 instances, `verification/evaluate.py`):
- baseline (equal-split, no target-awareness): **72.5** utilization (mean scrap 25.2 m)
- reference DP (`verification/ref_solver.py`, 1-D partition): **88.7** utilization (mean scrap 10.6 m)
- per-instance reference utilization: 95.8 / 88.8 / 91.8 / 75.2 / 92.4 / 87.0 / 89.4 / 89.3
- agent (openevolve, 10 generations, best saved program): **88.7** utilization
(run `20260903_130517`) — **equals the reference DP exactly**.
- agent (ShinkaEvolve, 15 generations, via reasoning proxy): **88.4** utilization
(near the optimum, run `20260904_182017`).
- agent (AB-MCTS, 15 iterations): **72.5** utilization (= baseline; this run did not improve —
AB-MCTS is weaker here, and low-reasoning mutations mostly regressed/invalidated).
- honest design note: the offline optimum is **reachable** (a strong agent can derive the
1-D partition DP and hit ~88.7 = the ceiling). So the offline task's difficulty is *deriving*
the DP, not long-horizon search. The harder **online** companion
([CuttingOptimizationOnline](../CuttingOptimizationOnline/README.md)) is where info
asymmetry keeps agents below the clairvoyant ceiling — see that README for the 3×2 matrix.
- `verification/multiseed_stat.py` gathers multi-run mean ± std across framework run dirs.
100 changes: 100 additions & 0 deletions benchmarks/ContinuousCasting/CuttingOptimization/README_zh-CN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# CuttingOptimization:连铸切割优化(离线/静态版,Frontier-Eng 基准)

一个**原创**的 Frontier-Engineering 基准,灵感来自 2021 全国大学生数学建模竞赛 D 题
《连铸切割的在线优化》,并被形式化成**确定性、自包含**的切割优化任务。

一根连续浇铸的钢坯以固定速度被拉出;结晶器异常会在坯内产生 **0.8 m 的零废段**,必须被
切出并报废。求解器收到钢坯长度、零废段位置、以及一个"目标值 + 可接受窗口"的用户需求后,
要给出一个**切割方案**(一组恰好铺满整根钢坯的切段长度),使**报废总长度最小**,并让
**每块成品尽量贴近目标值**。

完整规则与评测语义见 [Task.md](./Task.md)。

## 目录结构

```
benchmarks/ContinuousCasting/CuttingOptimization/
├── baseline/solver.py # 候选求解器(仅 EVOLVE-BLOCK 区域可改)
├── verification/
│ ├── generator.py # 固定种子实例生成器(零废段 + 目标窗口)
│ ├── simulator.py # 计分模拟器(校验 + 报废 / 贴合度指标)
│ ├── evaluate.py # 评测入口(subprocess + 时间预算 + 打分)
│ ├── validator.py # 完整性校验(静态 + 环境剥离 + 确定性)
│ ├── ref_solver.py # 参考解(一维划分 DP;文档化的"最优"分)
│ ├── test_simulator.py # 单测:计分器正确性
│ ├── test_generator.py # 单测:确定性 / 零废段可行性 / headroom
│ ├── test_ref_solver.py # 单测:参考解合法性 + 最优性
│ ├── test_validator.py # 单测:完整性校验 / 环境剥离 / 确定性
│ ├── test_evaluator.py # 单测:端到端评测行为
│ ├── data/instances/ # 8 个固定实例(种子固定、可复现)
│ ├── docker/Dockerfile # 极简纯标准库 python 镜像
│ └── requirements.txt
├── frontier_eval/ # UnifiedTask 元数据
├── Task.md # 任务规则、接口、评分、参考分
└── README_zh-CN.md
```

## 运行

```powershell
# 在固定 8 实例上给求解器打分(默认每实例 60s)
python verification/evaluate.py baseline/solver.py

# 加运行时生成实例(防硬编码)
python verification/evaluate.py baseline/solver.py --generate-seed <SEED>

# 更紧的时间预算(挑战档 10s)
python verification/evaluate.py baseline/solver.py --time-budget 10
```

## Docker

提供最简 `python:3.11-slim` 镜像(`verification/docker/Dockerfile`)。构建后用 unified 的
`isolation_mode=docker`(WSL/Linux;Windows 宿主受框架路径 bug 限制)。**已在 WSL 下实测通过**:
基线 `combined_score=72.49, valid=1.0, num_instances=8`,与 process 模式一致。

## 测试

```powershell
python -m unittest discover -s verification -p "test_*.py"
```

共 35 个单测(simulator / generator / ref_solver / validator / evaluator / 沙箱 evaluator 六个模块):
单块报废与贴合度规则、合法校验(求和、长度窗口、零废段对齐)、生成与参考解的确定性、
参考解优于 baseline 的最优性、validator 完整性(EVOLVE-BLOCK / 禁引用 / 绝对路径 /
按实例名硬编码 / 环境剥离 / 确定性探针)、以及 evaluator 行为(打分、运行时生成、
作弊候选被拒)、`frontier_eval/evaluator.py` 沙箱入口一致性。`verification/multiseed_stat.py`
用于多轮均值±std。

## 完整性 / 威胁模型

- **运行时生成实例**:设置 `CUTTING_EVAL_GENERATE_SEED` 后,评测现场生成新实例(临时目录,
不进仓库/沙箱),候选无法预先记忆。
- **候选环境剥离**:候选子进程剥离 `FRONTIER_*` / `CUTTING_EVAL_*` 变量(见 validator.py)。
- **静态检查**:EVOLVE-BLOCK 标记 + 固定区字节比对、禁引用评测/生成/参考解模块、绝对路径、
按实例名硬编码、确定性探针(两次运行输出一致)。任何违规记 0 分。
- **沙箱范围**:8 个固定实例、evaluator/validator 源码对候选可见(打分需要,且
`verification/simulator.py` 有意作为白盒计分器);防硬编码依赖 `CUTTING_EVAL_GENERATE_SEED`。
`verification/ref_solver.py` 与 `verification/generator.py` **不**复制进沙箱,并被 validator 额外禁用。
- 说明:在进程模式下候选有主机文件系统访问(框架级限制),本基准依赖上述分层防御。

## 评分

- 实例 = 8 固定(easy/medium/hard,S=24..150 m,0..6 个零废段,目标窗口 ±0.5 m)+ 设置
`CUTTING_EVAL_GENERATE_SEED` 时的运行时生成实例。
- **指标**:材料利用率 `util = 100 * (S - (scrap + 1e-4*penalty)) / S`,多实例取平均(0~100,越高越好)。
`scrap` = 总报废长度(零废段小块 + <8.0 m 整损 + 超出窗口的余量);`penalty = Σ|交付 - 目标|`;
`1e-4` 权重极小,保证报废严格占主导(惩罚仅在报废相同时破平),符合赛题的字典序目标。
- 非法输出 / 越界切段 / 未对齐零废段 / 崩溃 / 超时 ⇒ 该实例 0 分。
- **headroom 保证**:生成器只接受"参考解严格优于朴素等分 ≥ 0.1 m 报废"的实例,
保证每个实例都有真实优化信号。
- 参考分(固定 8 实例实测):
- baseline(均匀等分):**72.5** 利用率(平均报废 25.2 m)
- ref_solver(一维划分 DP):**88.7** 利用率(平均报废 10.6 m)
- agent(openevolve,10 代,best 保存程序):**88.7** 利用率(run `20260903_130517`)——**恰好等于参考解 DP**。
- agent(ShinkaEvolve,15 代,经推理代理):**88.4** 利用率(run `20260904_182017`,接近最优)。
- agent(AB-MCTS,15 迭代):**72.5** 利用率(= baseline;本轮未提升——AB-MCTS 在此较弱,低推理下多数改进突变回归/无效)。
- 诚实设计说明:离线最优是**可达的**(强 agent 能推导出精确的一维划分 DP 并打到 88.7 = 天花板)。
所以离线任务的难度在"推导 DP",不在长程搜索。更难的**在线版**
(`CuttingOptimizationOnline`)才是信息不对称让 agent 无法达到全知天花板——见该 README 的 3×2 矩阵。
- `verification/multiseed_stat.py` 用于跨框架运行目录做多轮均值±std。
Loading
Loading