feat: add TelecomBackup benchmark — power-backup scheduling for telecom sites (PowerSystems) - #105
Conversation
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. 质量保证
5. 安全与隐私检查
|
…om sites (PowerSystems) Time-sequenced on/off scheduling of telecom backup power supplies to maximize outage backup time while keeping LTE coverage >= 80%. Includes 8 fixed instances plus runtime generation (TELECOM_EVAL_GENERATE_SEED), a multi-rest rotation reference heuristic, cross-size determinism probes, and dockerized verification. Co-Authored-By: deepseek-v4-flash <deepseek-ai@claude-code-best.win>
7975f28 to
843389d
Compare
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. Quality Assurance
5. 安全与隐私检查
|
y-ji24
left a comment
y-ji24
left a comment
There was a problem hiding this comment.
Review — TelecomBackup
Thank you for the contribution. Overall, I think TelecomBackup is a relatively mature benchmark implementation. The task definition is clear, the simulator / validator / evaluator / sandbox layers are reasonably well separated, and the 34 unit tests covering these components all pass.
I was also able to reproduce the baseline score of 176.25, which matches the documentation. The reference heuristic achieves 271.25 under direct simulation, and the generated-instance mechanism works as expected when --generate-seed 42 is enabled. The threat model is also discussed quite transparently in the README, which I appreciate.
That said, I found two issues that I think should be addressed before merging, since they directly affect the credibility and reproducibility of the evaluation.
1. Blocker — Runtime instance generation is not enforced in the default evaluation path
The README and Task.md describe runtime-generated instances as an important defense against hardcoding. However, this mechanism currently depends on the host environment variable TELECOM_EVAL_GENERATE_SEED.
The problem is that this variable is not enforced by the task configuration, while the default evaluation command in the README does not set it. As a result, the default evaluation path falls back to the 8 publicly available fixed instances.
I verified that this creates a practical hardcoding vulnerability.
I constructed a candidate by offline-optimizing the fixed instances and selecting the corresponding schedule based on the instance seed. The candidate does not contain explicit instance_N identifiers, and it passes the static checks and deterministic probes. It achieves:
valid = 1.0- score = 271.88
This is already slightly higher than the reference score of 271.25.
The issue here is not that the simulator itself is incorrect. Rather, under the default configuration, a candidate can memorize the public instances and obtain a near-reference score without actually solving the general problem.
I therefore recommend making runtime generation mandatory for the formal evaluation path. For example:
- Require
TELECOM_EVAL_GENERATE_SEEDfor official evaluation; - Fail explicitly if the required generation configuration is missing, rather than silently falling back to fixed instances;
- Keep the fixed-instance mode for local development / smoke testing, but clearly distinguish it from the official evaluation mode.
This would make the actual evaluation behavior consistent with the threat model described in the README.
2. Blocker — The documented reference-score reproduction command does not reproduce the reference score
The README / PR description currently provides a command along the lines of:
python verification/evaluate.py verification/ref_solver.py
as a way to reproduce the reference score.
However, running this command returns 0.0.
After checking the evaluation path, the reason appears to be that the reference solver does not contain the required EVOLVE-BLOCK marker and also contains the prohibited ref_solver token, so it is rejected during candidate preflight.
At the same time, result_log.txt correctly indicates that the reported reference score of 271.25 was obtained through direct simulation.
Therefore, the problem is mainly a mismatch between the documentation and the actual evaluation implementation:
The documented command does not reproduce the reference score reported by the benchmark.
I recommend resolving this in one of two ways:
- Provide a dedicated reference-scoring path that bypasses candidate completeness / anti-cheating checks; or
- Update the README to explicitly state that the reference score is obtained through direct simulation and provide the actual reproducible procedure.
The important point is that a maintainer or future contributor should be able to follow the documented instructions and obtain the documented reference score.
3. Non-blocking — Evaluation timeout should be aligned with the task's time budget
The framework-level evaluator currently has a default timeout of 300s, while the task defines a 60s per-instance budget.
Considering that the evaluation also performs deterministic probes and may run multiple instances, a compliant solver that legitimately consumes most of its per-instance budget could potentially exceed the framework-level timeout.
The README mentions adjusting FRONTIER_EVAL_EVALUATOR_TIMEOUT_S, but I would prefer to see the appropriate value explicitly included in the task configuration for the official evaluation path.
This would avoid relying on environment-specific manual configuration.
4. Non-blocking — The 300 / 60 / 10 second time-budget modes are not fully integrated into the framework
The three time-budget modes currently appear to be supported mainly through the local CLI, while the framework evaluation path still uses a hard-coded 60s budget in frontier_eval/evaluator.py, and run_eval.py does not expose the corresponding mode selection.
If these three modes are intended to be official benchmark settings, I recommend connecting them to the standard task configuration. Otherwise, it would be helpful to clarify in the documentation that they are development / local evaluation options rather than formal evaluation modes.
5. Minor — Repository integration and provenance
A few smaller issues would also be worth cleaning up:
- The top-level
TASK_DETAILS.mddoes not appear to have been updated, so the new benchmark is currently missing from the repository-level task index. - It would be useful to add a short provenance / design note explaining the origin of important parameters such as the 80% coverage requirement, -105 dBm threshold, and the power-consumption coefficients. Since this is an original benchmark, external references are not necessarily required, but documenting the rationale would improve long-term maintainability.
- There is a small numerical inconsistency in the documentation. My measurement gives an average improvement of approximately +57.5% for the fixed instances, with a range of +27.9% to +119.0%, while the README reports +54%. The +54% figure appears closer to the reference heuristic's improvement over the baseline. I recommend clarifying these two numbers and their respective definitions.
6. Documentation / collaboration — Please use English for future submissions
One additional request for future contributions: please make the PR description, task introduction, README-facing descriptions, and other repository-facing documentation in English.
This is not a blocker for the technical implementation itself, but I would appreciate keeping future PRs consistent with the language convention of the repository.
Overall
Overall, I think the core implementation is in good shape. The simulator, validator, tests, and task design are all reasonably solid, and I don't think the issues above require a major redesign.
My main concern is that the formal evaluation path currently does not fully enforce the assumptions described by the benchmark's threat model, and the documented reference-score reproduction procedure is not consistent with the actual implementation.
I would therefore recommend:
Request Changes
with priority on:
- Enforcing runtime-generated instances in the official evaluation path;
- Fixing the reference-score reproduction procedure;
- Aligning the evaluation timeout / time-budget configuration.
Once these are addressed, I think the benchmark will be in a much stronger position for long-term use in the repository.
2 participants
这个基准聚焦电信基础设施的停电备电调度：区域内基站由电池供电，停电时需要在"任意时刻 LTE 良好覆盖栅格比例 ≥ 80%"的约束下，为每个电源制定时序开关策略，最大化区域总备电时长。
任务内容
p_silent=0.05 / p_work_base=3.0 / p_work_coef=3.0），生成器保证"错峰调度 ≥ 全程开启×25%"的每实例可复现 headroom（实测 +28%~+119%，平均 +54%）verification/ref_solver.py（多路休息轮换，271.2 分钟，+54%）验证方法
Agent 实测（deepseek-v4-flash，3 轮/框架，全部由保存程序直跑验证）：
泛化证据（固定 8 + 生成 8，seed=42）：openevolve 410.9 / ShinkaEvolve 319.4 / AB-MCTS 280.3——agent 的解可泛化到未见过的生成实例。完整 run ID 见
baseline/result_log.txt。Docker 隔离：纯标准库评测，最小镜像即可；Linux/WSL 实测通过（
task.runtime.isolation_mode=docker）。评测完整性
evaluate.py/simulator.py/validator.py/data/instances），参考求解器与生成器不进沙箱且被静态检查禁止引用FRONTIER_*/TELECOM_EVAL_*环境变量剥离TELECOM_EVAL_GENERATE_SEED（评测时现场生成未见实例），威胁模型已在 README 如实说明simulator.py有意作为白盒计分器开放给候选使用（constraints 允许）备注
frontier_eval/algorithms/shinkaevolve）与 vendored ShinkaEvolve 的版本对齐属于框架环境问题，与基准无关