Skip to content

fix(evals): make the release-test audit helper run on Windows - #764

Open
CuSO41108 wants to merge 4 commits into
Tencent:mainfrom
CuSO41108:fix/windows-eval-helper
Open

CuSO41108 wants to merge 4 commits into
Tencent:mainfrom
CuSO41108:fix/windows-eval-helper

Conversation

@CuSO41108

@CuSO41108 CuSO41108 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #763

Problem

scripts/test_release_audit.py cannot run on Windows. All CI jobs use
ubuntu-latest, so none of this is caught there, but a Windows contributor
following the README gets Ran 23 tests — FAILED (failures=1, errors=21):

  1. Fixture bytes. fixture-manifest.json hashes every release-test fixture
    byte-for-byte. With the Git for Windows default core.autocrlf=true the
    checkout is CRLF, so every manifest check fails.
  2. WinError 193. The evidence contract allows --bsk to be a wrapper, and
    the tests use a shebang Python script. Windows cannot exec a file without an
    executable extension, so discover() and execute() abort.
  3. Locale decoding. serve.py writes server-events.jsonl as UTF-8 but read
    it back with the platform default encoding. A non-ASCII event aborted the
    request handler on non-UTF-8 locales (GBK on zh-CN Windows) and mojibaked
    silently elsewhere. The same applied to captured help text, stdout/stderr
    evidence and benchmark output.
  4. AF_UNIX assumption. The admin-home test asserts a 96-byte
    daemon.sock path. Windows uses named pipes, so that branch is unreachable.

Change

  • Pin fixture text files to eol=lf and media files to -text, following the
    existing crates/bsk-cli/tests/fixtures/legacy-skills/*.md precedent.
  • Add launcher(): on Windows, run Python wrappers through the current
    interpreter; .exe/.com/.cmd/.bat and every non-Windows platform are
    launched unchanged. Wire it into both discover() calls and execute().
  • Make all remaining helper text I/O explicitly UTF-8, including
    subprocess help capture.
  • Remap the POSIX-only admin-home assertion to Windows and add regressions for
    the launcher decision and the UTF-8 /events round trip.

No product code, no browser execution and no gate semantics change: gate
still requires every planned scenario to pass in the authoritative round.

Verification

Windows 11, Python 3.13.1, checkout with core.autocrlf=true:

before: Ran 23 tests — FAILED (failures=1, errors=21)
after:  Ran 25 tests — OK
  • All 26 fixtures match fixture-manifest.json after re-checkout, and the
    working tree carries no fixture content diff (git check-attr reports
    text: set / eol: lf for .html, text: unset for .wav).
  • test_launcher_runs_python_wrappers_through_the_interpreter passes an explicit
    platform argument, so the Windows branch is covered on the Linux CI runner
    without mutating the global os.name (which would also change how pathlib
    resolves paths).
  • POSIX behaviour is unchanged: launcher() returns the path untouched, and the
    helper already wrote/read UTF-8 on the CI image.

🤖 Generated with Claude Code

fixture-manifest.json hashes every release-test fixture byte-for-byte, so a
CRLF checkout (the Git for Windows default with core.autocrlf=true) makes
every manifest check fail. Follow the existing legacy-skills precedent:
pin text fixtures to LF and leave media fixtures untouched.
…odec

serve.py wrote server-events.jsonl as UTF-8 but read it back with the platform
default encoding, so a non-ASCII event aborted the request handler on any
non-UTF-8 locale (GBK on zh-CN Windows) and silently mojibaked elsewhere.
Pin every remaining text read/write in the helper, including CLI help
capture and stdout/stderr evidence, to UTF-8, and cover the /events round trip.
Copilot AI balanced review requested due to automatic review settings November 17, 2026 17:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

The evidence contract allows --bsk to point at a wrapper, but Windows cannot
exec a file without an executable extension even when it carries a shebang, so
discover() and execute() failed with WinError 193 and the audit tests could not
run there. Prefix Python wrappers with the current interpreter, leave native
executables and .cmd/.bat shims alone, and skip the AF_UNIX-only admin-home
assertion on platforms that use named pipes.
@CuSO41108
CuSO41108 force-pushed the fix/windows-eval-helper branch from f7720b2 to 0414dc7 Compare November 17, 2026 17:59

@NianJiuZst NianJiuZst left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clean, well-isolated fix and thorough root-cause writeup! A few Windows gaps worth closing before we land this:

  1. Non-CJK Windows locales (cp1252) still fail in test_release_audit.py:
    test_adversarial_server_routes_bfcache_cors_and_media_mimes writes "你好" without an explicit encoding:
    (root / "captions-zh.vtt").write_text("WEBVTT\n\n00:00.000 --> 00:01.000\n你好\n")
    That passes on zh-CN Windows (gbk), where "你好" is representable, but raises UnicodeEncodeError on en-US Windows (cp1252). Please add encoding="utf-8" there (and ideally across the remaining bare read_text/write_text calls in test_release_audit.py).

  2. Wire test_release_audit.py into the existing Windows CI job:
    .github/workflows/ci.yml already has a windows-daemon job on windows-latest (which runs lightweight script checks around line 165). Adding python evals/browser/skills/browserskill-release-test/scripts/test_release_audit.py there (~18s) will exercise this on a real windows-latest checkout and prevent future fixture/encoding regressions.

  3. Two small release_audit.py Windows follow-ups while we're here:

    • launcher(): On Windows, a --bsk wrapper.py script often omits a Unix #! shebang, and CreateProcessW still fails on .py/.pyw with WinError 193. Handling path.suffix.lower() in {".py", ".pyw"} via [sys.executable or "python", str(path)] covers both shebang-less .py wrappers and extensionless shebang scripts.
    • execute() timeout handling (p.send_signal(signal.SIGINT)): on Windows, Popen.send_signal(signal.SIGINT) raises ValueError: Unsupported signal: 2, aborting before meta.json is written. Using p.terminate() when os.name == "nt" avoids crashing the recorder on timeouts.
  4. (Nit) In test_events_endpoint_serves_utf8_payloads, please use with urllib.request.urlopen(...) and thread.join() in finally: to match the other server tests in the file and avoid unclosed socket warnings.

- Encode every helper text write explicitly; the captions fixture used the
  locale codec and failed on cp1252 Windows.
- Run the audit helper tests in the existing windows-daemon CI job so fixture
  and encoding regressions surface on a real windows-latest checkout.
- Treat shebang-less .py/.pyw --bsk wrappers as interpreter scripts, since
  CreateProcessW rejects them as well.
- Terminate instead of send_signal(SIGINT) on Windows timeouts, where the
  latter raises ValueError before meta.json is written.
- Match the other server tests: context-managed responses and thread.join().
@CuSO41108

Copy link
Copy Markdown
Contributor Author

Thanks for the clean, well-isolated fix and thorough root-cause writeup! A few Windows gaps worth closing before we land this:

  1. Non-CJK Windows locales (cp1252) still fail in test_release_audit.py:
    test_adversarial_server_routes_bfcache_cors_and_media_mimes writes "你好" without an explicit encoding:
    (root / "captions-zh.vtt").write_text("WEBVTT\n\n00:00.000 --> 00:01.000\n你好\n")
    That passes on zh-CN Windows (gbk), where "你好" is representable, but raises UnicodeEncodeError on en-US Windows (cp1252). Please add encoding="utf-8" there (and ideally across the remaining bare read_text/write_text calls in test_release_audit.py).

  2. Wire test_release_audit.py into the existing Windows CI job:
    .github/workflows/ci.yml already has a windows-daemon job on windows-latest (which runs lightweight script checks around line 165). Adding python evals/browser/skills/browserskill-release-test/scripts/test_release_audit.py there (~18s) will exercise this on a real windows-latest checkout and prevent future fixture/encoding regressions.

  3. Two small release_audit.py Windows follow-ups while we're here:

    • launcher(): On Windows, a --bsk wrapper.py script often omits a Unix #! shebang, and CreateProcessW still fails on .py/.pyw with WinError 193. Handling path.suffix.lower() in {".py", ".pyw"} via [sys.executable or "python", str(path)] covers both shebang-less .py wrappers and extensionless shebang scripts.
    • execute() timeout handling (p.send_signal(signal.SIGINT)): on Windows, Popen.send_signal(signal.SIGINT) raises ValueError: Unsupported signal: 2, aborting before meta.json is written. Using p.terminate() when os.name == "nt" avoids crashing the recorder on timeouts.
  4. (Nit) In test_events_endpoint_serves_utf8_payloads, please use with urllib.request.urlopen(...) and thread.join() in finally: to match the other server tests in the file and avoid unclosed socket warnings.

Thanks for the detailed review — all four points are addressed in f7579cb8.

  1. Encodings: every remaining read_text/write_text in
    test_release_audit.py now passes encoding="utf-8", including the
    captions-zh.vtt fixture. The gap was invisible on zh-CN Windows because
    gbk can represent 你好; cp1252 could not.
  2. CI: added Run release-test audit helper tests to the existing
    windows-daemon job, right after the skill metadata checks.
  3. Windows follow-ups: .py/.pyw now map to the interpreter regardless
    of shebang. execute() timeouts use p.terminate() on Windows and keep
    send_signal(SIGINT) elsewhere; a regression forces a timeout and asserts
    meta.json still lands with timed_out: true.
  4. Nit: the events test now uses context-managed responses and
    thread.join() in finally, matching the other server tests.

Verification: 26/26 on Windows 11 (zh-CN) and 26/26 on Linux (WSL, Python 3.12).

One CI note: the Browser geometry, screenshots, session routing and dynamic reading job is also red on main at c0309039 with the identical three
wheel.browser.test.ts zoom assertions, so it is not caused by this PR. The
other 11 checks are green. I don't have write access to re-run workflows on
this repository — could you re-run that job once main is green?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: release-test audit helper cannot run on Windows (fixture hashes, WinError 193, locale decoding)

3 participants