Skip to content

wesnoth_tactical_ai: let top-level Lua files load - #56

Draft
findalexli wants to merge 1 commit into
ByteDance-Seed:mainfrom
findalexli:fix/wesnoth-lua-header
Draft

findalexli wants to merge 1 commit into
ByteDance-Seed:mainfrom
findalexli:fix/wesnoth-lua-header

Conversation

@findalexli

Copy link
Copy Markdown
Contributor

Task finding: #51.

What and why

Every top-level .lua file fails to load, because the sandbox header ends in if io then and Wesnoth's Lua raises on an undefined global (variable 'io' must be assigned before being used). The submitted Lua code is never reached and the grade doesn't mention it. The instruction tells solvers to write Lua CAs this way, and local testing, which has no header, loads them.

The fix is if rawget(_G, "io") then, plus the version bump and NOTES.

Old vs new

In one raw game each, the three Lua archives logged 150 to 454 lines of load errors and tracebacks on old and none on new. None of the 6 old judge_output.txt files for those archives mentions the error.

sforge eval, old then new on the same E2B worker (8 vCPU, 8 GiB), judge keys ace23e06dc3b and 567461e29768. eval_wesnoth.py is the only judge file that differs. The second starter and example runs are a second pair on a second worker. The real submission ran on its own worker with both images built there, alternating which judge went first. Scores are 0..100, n=2 per side unless stated.

Submission Old New
Qwen3.8 Flash Next, final archive (real, public; n=4 per side) 48, 67, 57, 58 81, 81, 76, 79
Stock CAs plus a Lua CA that stops every unit (test control) 41, 47 0, 0
Stock CAs plus a Lua CA that recruits 41, 41 47, 49
The instruction's example Lua CA 41, 43 43, 38
Stock AI, no Lua 39, 37 32, 52
Starter 0, 0 0, 0

The first row's submission has ten top-level .lua files. Its config loads ca_recruit.lua with location=~add-ons/my_ai/..., and that file requires my_ai_loader.lua and wes_ai_lib.lua. On old its load log shows the error and its Lua never runs; on new it loads cleanly and wins 44-47 of 60 games instead of 23-38. The remaining scoring rows are inconclusive at n=2 given Known issue 1.

Score effect

  • No top-level .lua: the header is never installed, so the same code runs and scores vary only as in Known issue 1.
  • A config that uses a top-level .lua: its code now runs, so its score can move either way. There is no mapping, because the old score never ran it.

We found one such submission in public rollouts, and can't say how many others there are. Your archived submissions with a top-level .lua are the ones to regrade. This change doesn't address Known issue 1.

Archives, raw outputs, job scripts and the results scripts: https://github.com/findalexli/EdgeBench/tree/da540564098c8bbc0ad5b9b4cfce0ce53045f8d6/wesnoth_tactical_ai. The Qwen3.8 archive isn't in that folder: it is export-full47/harness/wesnoth_tactical_ai/final_archive.tar.gz in qwen3.8-flash-next/qwen38fn-full47-part-02.zip of the public rollouts, sha256 7af57cb9a751b797d54c8484f0310c72407b4db2432288c73a8891a0b84ab6b1.

Checklist for task changes

  • version in task.toml is bumped.
  • NOTES.md: the change and its score effect are under Changes; Source notes the judge change. It isn't a listed Known issue.
  • Old vs new judge on the same submissions, each pair on one host, every row twice.
  • New blobs: none.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant