Flaky e2e eval on Windows: `ort_fusion`/`save_onnx` fails with WinError 32 on shared system TEMP

[KayMKM]

Summary

The e2e eval test TestEvalPerTask::test_zero_shot_image_classification
(openai/clip-vit-base-patch32, NPU/QNN) fails non-deterministically on Windows
with a temp-file locking error raised from the ort_fusion optimization pipe.
This is an environmental / flakiness issue tied to the shared system TEMP
directory, not a code regression — the same code passes as soon as a clean
TEMP directory is used.

Impact / Symptom

• Failing test: tests/e2e/test_eval_e2e.py::TestEvalPerTask::test_zero_shot_image_classification
• The CLIP vision model exports to ~335 MB, so save_onnx takes the
  external-data path, which is where the failure surfaces.
• Failure is intermittent: the nightly QNN e2e eval passed the previous run and
  failed the next with no relevant code change in between.

Observed failures

CI (Windows, QNN/NPU) — WinError 32:

[ERROR winml.modelkit.optim.optimizer] ✗ ort_fusion failed: Failed to apply fusion optimizations
  | Pipe: ort_fusion | Model info: {'model_type': 'clip'}
  | Caused by: Failed to write ONNX model to C:\Users\...\AppData\Local\Temp\tmppn29c2ry.onnx:
    [WinError 32] The process cannot access the file because it is being used by
    another process: 'C:\Users\...\AppData\Local\Temp'
Error: Evaluation failed: Failed to load model 'openai/clip-vit-base-patch32'.

Root cause

Two pre-existing fragilities amplify any transient TEMP locking into a hard failure:

1. save_onnx mutates process-global CWD.
   In src/winml/modelkit/onnx/persistence.py, the external-data branch does:

   os.chdir(path.parent)          # path.parent == system TEMP here
   onnx.save_model(model, path.name, save_as_external_data=True, ...)

   The WinError 32 is reported against path.parent (the TEMP directory),
   i.e. the os.chdir into TEMP failed. A process-global os.chdir is also
   unsafe under any concurrency.

2. ort_fusion calls save_onnx while the temp file handle is still open.
   In src/winml/modelkit/optim/pipes/fusion.py:

   with tempfile.NamedTemporaryFile(suffix=".onnx", delete=False) as f:
       input_path = f.name
       save_onnx(model, input_path)   # OS handle `f` still open during write

   Writing/opening the same path while the NamedTemporaryFile handle is open
   invites Windows sharing violations.

Why this is not a regression

• No PR merged between the last passing run and the failing run touches the
  failing path. optim/, eval/, onnx/persistence.py, and the e2e test file
  were all last modified weeks earlier; the in-window PRs are recipe JSON
  additions, a genai-only EP fix, and analyze/+pattern/ rule changes that are
  fully decoupled from eval/optim/export (verified: those packages do not
  import analyze/pattern).
• Re-running the exact failing test on main with a clean, dedicated TEMP passes
  in ~20s.

Reproduction / workaround

Point both the system temp env vars and pytest's basetemp at a clean directory:

$env:TEMP = "<repo>\temp\e2e-clip-rerun\sys"; $env:TMP = $env:TEMP
uv run pytest "tests/e2e/test_eval_e2e.py::TestEvalPerTask::test_zero_shot_image_classification" `
  -m e2e --basetemp="<repo>\temp\e2e-clip-rerun\pytest" -v
# => 1 passed in ~20s

Both must change: TEMP/TMP covers the CLI's internal tempfile
(save_onnx/ort_fusion), --basetemp covers pytest's own tmp_path.

Suggested fix

1. In ort_fusion, close the temp file handle before writing to it — create the
   name with mkstemp + os.close(fd) (or move save_onnx outside the
   with block) so no OS handle is held when onnx.save_model opens the path.
2. Remove the process-global os.chdir in save_onnx; write to an absolute
   path and pass an absolute/relative location to onnx.save_model instead of
   changing CWD.
3. (Optional) Have ort_fusion use a dedicated tempfile.mkdtemp() per run and
   clean it up, so a poisoned shared TEMP can't break unrelated code paths.
4. (CI) Run e2e jobs against a clean, job-scoped TEMP directory.

Environment

• OS: Windows (arm64)
• Python 3.11.14, pytest 9.1.1, onnxruntime QNN EP (NPU)
• Model: openai/clip-vit-base-patch32 (exported ~335 MB, external data)

[DingmaomaoBJTU]

Triage Summary

Field	Value
Priority	P2 — Medium (flaky, not blocking; clean TEMP workaround exists)
Component	optim (ort_fusion pipe), onnx/persistence (save_onnx)
Type	Bug — environmental flakiness / unsafe temp-file handling
Owner	@DingmaomaoBJTU (infra / quality)

Action items

1. Fix ort_fusion temp-file handle leak (close fd before write)
2. Remove process-global os.chdir in save_onnx; use absolute paths
3. CI: scope TEMP to a per-job directory

Low complexity fix; good candidate for a single PR addressing items 1+2.