Skip to content

feat(pytest): write failed suite artifacts - #2

Merged
vk-ai merged 1 commit into
vk-ai:mainfrom
mgalore:feat/suite-artifact
Sep 6, 2026
Merged

vk-ai merged 1 commit into
vk-ai:mainfrom
mgalore:feat/suite-artifact

Conversation

@mgalore

@mgalore mgalore commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add the suite_artifact pytest fixture for writing an indented, stable Suite.to_json() artifact when an eval row fails
  • avoid creating artifacts for fully passing suites
  • document the fixture and cover failure and pass behavior

Closes #1

Verification

  • PATH="$PWD/.venv/bin:$PATH" .venv/bin/python -m pytest -q — 8 passed
  • .venv/bin/python -m compileall -q src tests
  • git diff --check

Risk

This is opt-in and uses only the standard library; it does not add required dependencies or affect existing fixtures.

@vk-ai

vk-ai commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Thanks — LGTM for #1.

Opt-in suite_artifact fixture is clean (stdlib only, skips on full pass). Local pytest: 8 passed.

One note vs the issue: you shipped a fixture instead of an auto-write pytest hook — preferable (explicit I/O). Empty suite → no file, which is fine.

Happy to merge.

@vk-ai
vk-ai merged commit aa75221 into vk-ai:main Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Good first issue: pretty-print suite.to_json on pytest failure

2 participants