Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,11 +56,18 @@ jobs:
- run: uv run --no-editable ruff format --check .
- run: uv run --no-editable mypy
- run: uv run --no-editable pyright --pythonversion 3.11 src/testenix
- run: uv run --no-editable testenix pytest -q
- run: uv build --no-sources
- run: uv run twine check --strict dist/*
- run: uv run check-wheel-contents dist/*.whl
- run: uv run --isolated --no-project --with ./dist/testenix-*.whl testenix --version
- run: uv run --isolated --no-project --with ./dist/testenix-*.whl testenix run examples/basic --no-history
- name: Verify pytest extra from wheel
shell: bash
run: |
wheel_path="$(find dist -maxdepth 1 -type f -name '*.whl' -print -quit)"
uv run --isolated --no-project --with "testenix[pytest] @ $wheel_path" testenix pytest --version
uv run --isolated --no-project --with "testenix[pytest] @ $wheel_path" --with "pytest==8.3.0" testenix pytest --version
- run: uv run --isolated --no-project --with ./dist/testenix-*.tar.gz testenix --version
- run: uv run --isolated --no-project --with ./dist/testenix-*.tar.gz testenix run examples/basic --no-history

Expand Down
7 changes: 7 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ jobs:
fi
- run: uv sync --locked --dev --no-editable
- run: uv run --no-editable pytest
- run: uv run --no-editable testenix pytest -q
- run: uv run --no-editable ruff check .
- run: uv run --no-editable ruff format --check .
- run: uv run --no-editable mypy
Expand All @@ -49,6 +50,12 @@ jobs:
- run: uv run check-wheel-contents dist/*.whl
- run: uv run --isolated --no-project --with ./dist/testenix-*.whl testenix --version
- run: uv run --isolated --no-project --with ./dist/testenix-*.whl testenix run examples/basic --no-history
- name: Verify pytest extra from wheel
shell: bash
run: |
wheel_path="$(find dist -maxdepth 1 -type f -name '*.whl' -print -quit)"
uv run --isolated --no-project --with "testenix[pytest] @ $wheel_path" testenix pytest --version
uv run --isolated --no-project --with "testenix[pytest] @ $wheel_path" --with "pytest==8.3.0" testenix pytest --version
- run: uv run --isolated --no-project --with ./dist/testenix-*.tar.gz testenix --version
- run: uv run --isolated --no-project --with ./dist/testenix-*.tar.gz testenix run examples/basic --no-history
- name: Store distributions
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,13 @@ project intends to use Semantic Versioning once its public API reaches stability

## [Unreleased]

### Added

- `testenix pytest [PYTEST_ARGS ...]` compatibility bridge for unchanged pytest suites, preserving
the real pytest collector, fixtures, parametrization, markers, plugins, output, and exit status.
- Optional `testenix[pytest]` installation extra and a documented native-versus-compatibility
capability matrix.

## [0.1.0] - 2026-07-20

### Added
Expand Down
67 changes: 55 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@ Testenix is an experimental native Python testing framework built around five gu
4. retries never erase previous failures;
5. every report is derived from one versioned, lossless result model.

The project is currently an alpha. Its runtime has no third-party dependencies and does not depend
on pytest.
The project is currently an alpha. Its native runtime has no third-party dependencies and does not
depend on pytest. An optional compatibility bridge can delegate unchanged pytest suites to the
pytest installation already present in a project.

## Installation

Expand All @@ -27,6 +28,15 @@ Install the published package with:
python -m pip install testenix
```

For an existing pytest project, install the optional convenience extra:

```bash
python -m pip install "testenix[pytest]"
```

If a supported pytest (`>=8.3,<10`) is already installed in the same environment, the base
`testenix` package is sufficient.

Testenix requires Python 3.11 or newer. The project is currently an alpha; pin the version before
using it in CI. Until the first PyPI release is visible, use the GitHub installation below.

Expand All @@ -42,9 +52,36 @@ You can also install the latest source directly from GitHub:

```bash
python -m pip install "testenix @ git+https://github.com/polishdataengineer/testenix.git@main"
# include pytest when the project environment does not already provide it
python -m pip install "testenix[pytest] @ git+https://github.com/polishdataengineer/testenix.git@main"
```

## Run an existing pytest suite

Yes. Testenix provides a transparent compatibility bridge for existing pytest projects:

```bash
testenix pytest -q tests
```

Everything after `testenix pytest` is forwarded unchanged to the same interpreter as
`python -m pytest`. This preserves pytest collection, `conftest.py`, fixtures, parametrization,
markers, assertion rewriting, plugins, configuration, output, node IDs, and exit codes.

```bash
testenix pytest tests/test_api.py::test_health -k smoke --maxfail=1
testenix pytest -n auto tests # requires pytest-xdist in the same environment
testenix pytest --junitxml=reports/pytest.xml tests
```

## Quick start
This command is a compatibility and migration bridge. It does not use the native Testenix
collector, scheduler, worker pool, retries, history, result model, or reporters. Its performance is
therefore pytest performance plus launcher and adapter overhead; the native benchmark results do
not apply to it. See the
[pytest compatibility guide](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/)
for the complete capability matrix and migration choices.

## Native quick start

Create `tests/test_multiplication.py` with this complete example:

Expand Down Expand Up @@ -138,10 +175,13 @@ test.

## Where Testenix is deliberately different

Testenix is not yet a drop-in replacement for pytest. Its v0.1 value is a smaller, coherent native
stack: async tests and async fixtures need no plugin, parallel execution and duration-aware
scheduling need no xdist, every retry remains visible, and a worker crash cannot silently erase
tests that completed before it. The runtime has no third-party dependencies.
The `testenix run` engine is not a native drop-in reimplementation of pytest. Its v0.1 value is a
smaller, coherent native stack: async tests and async fixtures need no plugin, parallel execution
and duration-aware scheduling need no xdist, every retry remains visible, and a worker crash cannot
silently erase tests that completed before it. The native runtime has no third-party dependencies.

For unchanged pytest semantics, use `testenix pytest`. That bridge intentionally delegates to
pytest instead of silently approximating fixtures, markers, plugins, or hook behavior.

The trade-off is ecosystem maturity. pytest currently has much broader plugin, IDE, assertion
rewriting, and migration support. Testenix should only claim to be better for the guarantees above
Expand All @@ -161,10 +201,11 @@ can copy its own text or the complete project reference for an LLM.

## Benchmarks

In the checked-in M4 Pro/CPython 3.11 synthetic baseline, Testenix completed 100,000 empty tests
across 16 modules in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30 seconds
for pytest-xdist. That is 3.15x the throughput of pytest for this specific workload, not a universal
performance promise. The result includes one warm-up and five measured, counterbalanced rounds.
In the checked-in M4 Pro/CPython 3.11 synthetic baseline, native `testenix run` completed 100,000
empty tests across 16 modules in a median 8.04 seconds, compared with 25.33 seconds for pytest and
21.30 seconds for pytest-xdist. That is 3.15x the throughput of pytest for this specific workload,
not a universal performance promise. The result includes one warm-up and five measured,
counterbalanced rounds. It does not describe `testenix pytest`, which executes through pytest.

See the [generated results and chart](https://polishdataengineer.github.io/testenix/benchmarks/results/),
[raw JSON](https://github.com/polishdataengineer/testenix/tree/main/benchmarks),
Expand Down Expand Up @@ -193,7 +234,9 @@ See the [generated results and chart](https://polishdataengineer.github.io/teste
- On Windows, a script that calls the programmatic `run()`/`run_async()` API must use the standard
`if __name__ == "__main__":` multiprocessing guard. The `testenix` CLI handles process startup
itself.
- Test impact analysis, result caching, remote workers, and the pytest migration adapter are not
- The pytest bridge does not translate delegated outcomes into Testenix `RunResult`, JSON, history,
retry, timeout, or scheduling semantics. Use pytest's own flags and installed plugins in that mode.
- Test impact analysis, result caching, remote workers, and deep pytest-result aggregation are not
part of version 0.1.

## Project status
Expand Down
21 changes: 19 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,22 @@ Testenix is a native Python testing framework. Its core does not depend on pytes
Compatibility adapters may translate foreign test frameworks into the same manifest and event
contracts, but they are not part of native execution.

The first pytest compatibility bridge deliberately does not translate pytest internals. On POSIX,
`testenix pytest` uses `os.execv` to replace itself with `sys.executable -m pytest`. On Windows it
calls pytest's public `console_main` entry point in the existing process because the platform's
`exec` family does not provide equivalent replacement semantics. Both paths keep pytest in the
foreground CLI process with the same working directory, environment, terminal, streams, signal
handling, and exit status. This preserves semantics without making the native core depend on
pytest:

```text
POSIX: testenix pytest ==exec==> Python -m pytest -> collector/plugins/executor -> output/status
Windows: testenix pytest =========> pytest.console_main -> collector/plugins/executor -> output/status
```

The bridge is a CLI infrastructure adapter, not a native collection adapter. It does not emit
Testenix events or construct a `RunResult` in version 0.1.

## Product contract

Testenix aims to be typed, async-native, parallel-first, deterministic, and lossless when reporting
Expand All @@ -25,7 +41,7 @@ Authoring API -> supervised collection -> inert manifest -> affinity scheduler -
- `events`, `aggregate`, and `scheduler` remain engine-independent.
- `runner` is the application service connecting the native engine with execution policy.
- reporters and storage consume completed domain results or versioned events.
- optional compatibility adapters depend inward; the core never imports them.
- optional compatibility adapters stay at the CLI boundary; the native core never imports pytest.

## Version 0.1 scope

Expand All @@ -37,7 +53,8 @@ Authoring API -> supervised collection -> inert manifest -> affinity scheduler -
- deterministic scheduling based on historical durations;
- append-only JSONL events and a pure reducer;
- console, JSON, and JUnit output plus local SQLite duration history;
- retries represented as immutable attempts and finalized as `FLAKY` when appropriate.
- retries represented as immutable attempts and finalized as `FLAKY` when appropriate;
- an optional platform-aware pytest handoff for unchanged legacy suites.

Remote workers, distributed storage, result caching, automatic quarantine, and a stable third-party
plugin SDK are deliberately outside version 0.1.
Expand Down
13 changes: 13 additions & 0 deletions docs/benchmarking.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,19 @@ files.
Correctness wins over speed: a run with a missing, duplicated, or incorrectly finalized result is
invalid and excluded from performance comparisons.

## Pytest compatibility bridge

Measurements of `testenix pytest` must be reported separately from native `testenix run`
measurements. The compatibility command hands the current interpreter to pytest through a POSIX
process overlay or pytest's in-process console entry point on Windows; it does not execute tests
through the Testenix engine.

A compatibility-overhead comparison must use the same interpreter, working directory,
environment, pytest configuration, plugins, and arguments for both `python -m pytest ...` and
`testenix pytest ...`. Any difference measures adapter overhead only and must not be presented as a
Testenix execution speedup. Native comparisons continue to use `testenix run` and must validate
that both runners execute the same tests and produce equivalent outcomes.

Run the reproducible local harness with:

```bash
Expand Down
3 changes: 2 additions & 1 deletion docs/benchmarks/results.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

These tables are generated from the raw JSON committed in `benchmarks/`. They are development
evidence for specific synthetic workloads, not a universal claim that Testenix is always faster
than pytest.
than pytest. `Testenix` in these results means the native `testenix run` engine. The
`testenix pytest` compatibility bridge delegates to pytest and is not represented here.

![Preliminary Testenix throughput ratios](../_static/benchmark-speedup.svg)

Expand Down
2 changes: 2 additions & 0 deletions docs/for-llms.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ context. Every documentation page also includes **Copy this page** and
| [`llms.txt`](https://polishdataengineer.github.io/testenix/llms.txt) | A model should first discover the project and choose relevant pages. |
| [`llms-full.txt`](https://polishdataengineer.github.io/testenix/llms-full.txt) | You want one self-contained reference with guides, API, architecture, and benchmark context. |
| [Python API reference](reference/api.md) | The task is specifically about authoring or embedding Testenix. |
| [Pytest compatibility](guides/pytest-compatibility.md) | A model must choose between delegation and native migration. |
| [Benchmark results](benchmarks/results.md) | A model needs to evaluate or repeat performance claims. |

`llms.txt` follows the emerging llms.txt proposal, but it should be treated as a convenience
Expand All @@ -29,6 +30,7 @@ workload-specific and preserve all documented limitations.
## What the full reference contains

- installation and first-run instructions;
- the pytest compatibility bridge, capability matrix, and migration boundary;
- native tests, cases, tags, skips, expected failures, and fixtures;
- parallelism, timeouts, retries, crash recovery, reports, and history;
- CLI, configuration, and generated Python API reference;
Expand Down
29 changes: 27 additions & 2 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ This guide takes a new project from installation to its first parallel Testenix

- CPython 3.11 or newer
- Linux, macOS, or Windows
- no runtime dependencies beyond the Python standard library
- no runtime dependencies beyond the Python standard library for native `testenix run`
- pytest in the same environment when using the optional compatibility bridge

## Install

Expand All @@ -24,13 +25,36 @@ $ uv add --dev testenix
$ uv run testenix --version
```

For an existing pytest project, install the compatibility extra:

```console
$ python -m pip install "testenix[pytest]"
# or
$ uv add --dev "testenix[pytest]"
```

Until the first PyPI release is visible, install directly from the protected `main` branch:

```console
$ python -m pip install "testenix @ git+https://github.com/polishdataengineer/testenix.git@main"
# include pytest when the project environment does not already provide it
$ python -m pip install "testenix[pytest] @ git+https://github.com/polishdataengineer/testenix.git@main"
```

## Choose an execution mode

Run an unchanged pytest suite through its real engine:

```console
$ testenix pytest -q tests
```

## Create a test
Use `testenix run` for native Testenix tests and the built-in scheduler, retries, history, and
lossless reports. The two commands have deliberately separate semantics. See
[pytest compatibility](guides/pytest-compatibility.md) for the capability matrix and migration
boundary.

## Create a native test

Testenix collects ordinary functions whose names begin with `test_`. Decorators are optional for
simple tests.
Expand Down Expand Up @@ -110,6 +134,7 @@ collection and execution process trees before control returns to the caller.
## Next steps

- [Write tests, cases, tags, skips, and expected failures](guides/writing-tests.md)
- [Run or migrate an existing pytest suite](guides/pytest-compatibility.md)
- [Build fixture graphs](guides/fixtures.md)
- [Understand process parallelism and timeouts](guides/parallelism.md)
- [Produce JSON and JUnit reports](guides/reports.md)
Expand Down
Loading
Loading