Skip to content

feat: add safe pytest and unittest migration - #8

Merged
polishdataengineer merged 5 commits into
mainfrom
agent/safe-test-migration
Jul 20, 2026
Merged

feat: add safe pytest and unittest migration#8
polishdataengineer merged 5 commits into
mainfrom
agent/safe-test-migration

Conversation

@polishdataengineer

@polishdataengineer polishdataengineer commented Jul 20, 2026

Copy link
Copy Markdown
Owner

Summary

  • add testenix migrate {auto,pytest,unittest} with dry-run, validation-only, and safe publication modes
  • convert the supported static pytest subset and wrap supported unittest suites without changing original files
  • validate source and generated suites in isolated shadow copies before an atomic create-only publish
  • compare exact per-test and aggregate outcomes, re-hash the source before publication, and emit a machine-readable migration report
  • document supported/blocked constructs, safety guarantees, recovery behavior, CLI examples, and LLM-ready references
  • add reproducible 3,000-test migration benchmarks and expose them in README and GitHub Pages
  • test migration from built wheel and sdist artifacts, including unittest execution from an unrelated working directory

Transaction and safety guarantees

  • originals are never overwritten, moved, or deleted
  • output and report paths must be new, disjoint from the source, and inside the project
  • validation failure leaves the original suite intact and publishes no output
  • publication is descriptor-anchored and atomic on supported POSIX filesystems
  • source hashes are checked again immediately before publication
  • generated unittest wrappers verify the complete source manifest, including helpers, on every module resolution
  • CRLF/LF differences are normalized only for the JSON manifest; source hashes remain byte-exact
  • unsupported or ambiguous constructs fail closed with an actionable report
  • this is filesystem transaction safety, not an OS security sandbox

Supported migration scope

Pytest support includes static module-level tests, one static parametrize, common static skip forms, plain markers, and a deliberately limited fixture subset. Unittest uses compatibility wrappers around real TestCase.run so lifecycle, assertions, mock, async cases, and per-test outcomes are preserved. Dynamic collection, plugins/hooks, class-based pytest suites, run-global session fixtures, and other constructs whose semantics cannot be guaranteed are blocked instead of guessed.

Reproducible benchmark snapshot

All rows use 3,000 tests across 64 modules, one warm-up, five alternating measured rounds, and four native workers:

Source runner / workload Source median Native median Result
sequential pytest / no-op 1.539 s 0.521 s Testenix 2.96x faster
sequential unittest outcome probe / no-op 0.161 s 1.192 s Testenix 7.40x slower
sequential unittest outcome probe / 1 ms work 4.066 s 2.577 s Testenix 1.58x faster

Migration is a one-time validation transaction and took 5.940–17.251 s in these synthetic cases. The raw JSON records include clean source commit 3a51a90, exact commands, versions, environment, every sample, integrity result, and publication status. These results are not presented as a universal advantage over pytest-xdist or real repositories.

Verification

  • 201 passed (full suite)
  • 115 passed (migration-focused suite)
  • Ruff check and format
  • mypy strict
  • Pyright
  • docs asset regeneration and freshness check
  • Sphinx dirhtml build with warnings as errors
  • source distribution and wheel build
  • strict Twine metadata check
  • wheel-contents check
  • clean wheel and sdist smoke migrations for pytest and unittest
  • CI matrix: CPython 3.11–3.14 on Ubuntu, macOS, and Windows

No PyPI release is performed by this PR.

@polishdataengineer
polishdataengineer marked this pull request as ready for review July 20, 2026 16:45
@polishdataengineer
polishdataengineer merged commit 19ab4ba into main Jul 20, 2026
15 checks passed
@polishdataengineer
polishdataengineer deleted the agent/safe-test-migration branch July 20, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant