diff --git a/edgebench/tasks/apple_incremental_game/NOTES.md b/edgebench/tasks/apple_incremental_game/NOTES.md index b31adfcc..ef6c310c 100644 --- a/edgebench/tasks/apple_incremental_game/NOTES.md +++ b/edgebench/tasks/apple_incremental_game/NOTES.md @@ -2,7 +2,7 @@ AtCoder Heuristic Contest 058 "Apple Incremental Game", in ALE-Bench style. The agent writes `solution.py` (stdin -> stdout): 500 turns, each either strengthening one of the 4×10 machines or -doing nothing. The judge runs it on 50 fixed cases (8 at a time, 30 s wall clock each), scores each +doing nothing. The judge runs it on 50 fixed cases (`min(8, SFORGE_CPUS)` at a time, 30 s wall clock each), scores each output with `tools/bin/tester` (`round(1e5 · log2(S))`, S = apples after T turns), and sums the per-case scores of the `OK` cases (failed cases count 0). Raw score = that sum (maximize). The rescale is the v1 one (`piecewise_max`, baseline 45000000). @@ -13,7 +13,8 @@ rescale is the v1 one (`piecewise_max`, baseline 45000000). authors' repository (not public), identical by sha256 and mode to the published v1 images. The tools are Python scripts. - `environment/workspace/solution.py` (starter) and `judge/eval_apple_incremental_game.py`: - byte-identical to v1's inline scripts and to the published v1 images. `judge/grade.py` is new and + byte-identical to v1's inline scripts and to the published v1 images (the eval script until 1.1.0). + `judge/grade.py` is new and writes `result.json` with v1's scoring. - `instruction.md`: v1's prompt, verbatim. Base: `bases/python310`. @@ -47,9 +48,22 @@ timeouts depending on host load, on both images. Historical 3 uses unseeded rand extra pairs of runs, both images give overlapping ranges with means 0.05 M apart. The other time-limited searches differ by under 1 point on 0..100. +## Changes + +### 1.1.0 + +The judge ran 8 cases at once on the official 4 CPUs, against SCHEMA's rule that a parallel grader +sizes its pool with `SFORGE_CPUS`, so a deadline search got about half the CPU of 4 at once. The pool +is now `min(8, SFORGE_CPUS, cases)`; `SFORGE_CPUS` must be a positive integer, and 4 applies only when +it is absent. Deterministic solutions that finish on both judges keep their scores. Seven deadline-based +files gained 3.65-10.86 of 100 across fourteen pairs at true 4 CPUs, with no case lost on either judge. +There is no mapping from old scores, so regrade before comparing with v1 entries. `timeout_sec` goes +from 600 to 1800 for a smaller `--judge-cpu-limit`: one file took 1443 s at limit 1, and at 4 CPUs the +longest grade took about 391 s. Known issue 3 stays. Source, Deviations and Regression summary describe 1.0.0. + ## Known issues -Kept as in v1, so scores stay comparable. Fixing any of them changes scores and bumps `version`. +Kept as in v1. Fixing any of them changes scores and bumps `version`. 1. **The submission can forge its score through the tester's error message.** The eval script takes a score from any tester output line that looks like `Score = …` and ignores the tester's diff --git a/edgebench/tasks/apple_incremental_game/judge/Dockerfile b/edgebench/tasks/apple_incremental_game/judge/Dockerfile index 0e2f0968..009e7148 100644 --- a/edgebench/tasks/apple_incremental_game/judge/Dockerfile +++ b/edgebench/tasks/apple_incremental_game/judge/Dockerfile @@ -15,7 +15,7 @@ # 9 rm -rf .git && git init && git config user.* && add && commit -m init # -> kept, author/dates pinned to v1's commit # 10 heredoc > /tmp/eval_apple_incremental_game.py; chmod +x -> COPY judge/eval_apple_incremental_game.py /judge/ -# (byte-identical, moved out of /tmp) +# (v1's, except the pool size, moved out of /tmp) # v1 eval_cmd + parser score_sum -> judge/grade.py (task.toml [judge].cmd) # Base = bases/python310 (sforge build passes it as BASE_IMAGE). diff --git a/edgebench/tasks/apple_incremental_game/judge/eval_apple_incremental_game.py b/edgebench/tasks/apple_incremental_game/judge/eval_apple_incremental_game.py index ed36e1b4..0dacb41a 100755 --- a/edgebench/tasks/apple_incremental_game/judge/eval_apple_incremental_game.py +++ b/edgebench/tasks/apple_incremental_game/judge/eval_apple_incremental_game.py @@ -42,7 +42,7 @@ def run_case(args): os.chdir(WORK_DIR) case_files = sorted(f for f in glob.glob(os.path.join(CASES_DIR, "*")) if os.path.isfile(f)) results = {} - with ProcessPoolExecutor(max_workers=min(8, len(case_files))) as pool: + with ProcessPoolExecutor(max_workers=min(8, int(os.environ.get("SFORGE_CPUS", 4)), len(case_files))) as pool: futures = {pool.submit(run_case, (i, cf)): i for i, cf in enumerate(case_files)} for fut in as_completed(futures): cid, status, score = fut.result() diff --git a/edgebench/tasks/apple_incremental_game/judge/grade.py b/edgebench/tasks/apple_incremental_game/judge/grade.py index 5e58e2fb..e79d0c58 100755 --- a/edgebench/tasks/apple_incremental_game/judge/grade.py +++ b/edgebench/tasks/apple_incremental_game/judge/grade.py @@ -1,11 +1,11 @@ #!/usr/bin/env python3 """Grader for apple_incremental_game (Verified judge contract, see SCHEMA.md). -Reproduces v1 exactly: +Keeps v1's command, parser and score: v1 eval_cmd : cd /home/workspace/apple_incremental_game && python3 /tmp/eval_apple_incremental_game.py v1 parser : score_sum (sforge/harness/log_parsers/score_sum.py) v1 score : first `TOTAL_SCORE ` in the eval output (sforge/harness/grading.py:extract_score) -/judge/eval_apple_incremental_game.py is byte-identical to the v1 /tmp/eval_apple_incremental_game.py. +/judge/eval_apple_incremental_game.py is the v1 /tmp/eval_apple_incremental_game.py except for its pool size (1.1.0). This wrapper only runs it and translates its stdout into /opt/sforge/result.json. stdout = logs only. """ diff --git a/edgebench/tasks/apple_incremental_game/task.toml b/edgebench/tasks/apple_incremental_game/task.toml index 76d65fe1..a1070faa 100644 --- a/edgebench/tasks/apple_incremental_game/task.toml +++ b/edgebench/tasks/apple_incremental_game/task.toml @@ -4,7 +4,7 @@ name = "Apple Incremental Game" category = "Combinatorial Optimization" language = "Python" description = "Decide each turn whether to invest in machines or collect output in an incremental production game, balancing short-term gains against long-horizon compounding." -version = "1.0.0" +version = "1.1.0" status = "migrated" [environment] @@ -22,12 +22,12 @@ exclude = [] [judge] cmd = "python3 /judge/grade.py" -timeout_sec = 600 +timeout_sec = 1800 cpus = 4 memory = "8g" # Official leaderboard limits (v1 kept them only in the experiment YAMLs). -# eval_apple_incremental_game.py runs the 50 cases 8 at a time (ProcessPoolExecutor, -# max_workers=8), each `python3 solution.py` under a 30 s wall-clock timeout. Scores of +# eval_apple_incremental_game.py runs the 50 cases min(8, SFORGE_CPUS) at a time, +# each `python3 solution.py` under a 30 s wall-clock timeout. Scores of # time-limited (anytime) solutions depend on host CPU speed and load; see NOTES.md. [score]