tools/desktop-journeys/generative-workflows/g2a_csv_artifact_surface.py fails intermittently because the model does not call the artifact tool at all.
Measured over 15 consecutive real runs against OpenAI (2026-07-27): 13 passed, 2 failed — runs 6 and 13, both with the identical assertion:
File "g2_csv_lifecycle.py", line 578, in _dataset_artifact_from_run
assert artifacts, "agent did not create a dataset artifact"
AssertionError: agent did not create a dataset artifact
No run errors, no provider errors — the run completes, it just produces no dataset artifact. This is model non-determinism, not a product defect.
Why it matters: this journey is a verification harness for artifact work. A ~13% false-red rate means roughly one run in seven fails for reasons unrelated to the code under test, which erodes trust in the signal and encourages re-running until green — exactly the habit that hides real regressions.
Options:
- Retry once on this specific assertion before failing (cheapest; keeps the signal honest about repeated non-creation).
- Firm up
CREATE_PROMPT so artifact creation is unambiguous.
- Assert on a terminal run state first, so the failure distinguishes "model chose not to" from "artifact pipeline broken" — currently both surface identically.
Raw data: 15/15 runs also showed zero occurrences of the empty-call_id defect tracked in #402, so this flake is the journey's dominant failure mode today.
tools/desktop-journeys/generative-workflows/g2a_csv_artifact_surface.pyfails intermittently because the model does not call the artifact tool at all.Measured over 15 consecutive real runs against OpenAI (2026-07-27): 13 passed, 2 failed — runs 6 and 13, both with the identical assertion:
No run errors, no provider errors — the run completes, it just produces no dataset artifact. This is model non-determinism, not a product defect.
Why it matters: this journey is a verification harness for artifact work. A ~13% false-red rate means roughly one run in seven fails for reasons unrelated to the code under test, which erodes trust in the signal and encourages re-running until green — exactly the habit that hides real regressions.
Options:
CREATE_PROMPTso artifact creation is unambiguous.Raw data: 15/15 runs also showed zero occurrences of the empty-
call_iddefect tracked in #402, so this flake is the journey's dominant failure mode today.