# Testenix complete documentation > Generated from the Testenix repository. Treat current documented behavior separately from roadmap items, and treat every benchmark as workload-specific. Canonical documentation: https://polishdataengineer.github.io/testenix/ Source repository: https://github.com/polishdataengineer/testenix --- # Document: Overview Canonical URL: https://polishdataengineer.github.io/testenix/ Source: docs/index.md --- description: Testenix is an async-native, parallel-first Python testing framework with lossless results. --- # Testenix
Python testing framework · Alpha
Fast tests. Clear results.

Testenix combines a dependency-free native runtime with a transparent bridge for running existing pytest suites unchanged.

Start testing Run pytest suites
Migration guide → Benchmarks →
```{toctree} :hidden: :caption: Start :maxdepth: 2 getting-started guides/pytest-compatibility guides/migration ``` ```{toctree} :hidden: :caption: Guides :maxdepth: 2 guides/writing-tests guides/fixtures guides/parallelism guides/reports ``` ```{toctree} :hidden: :caption: Reference :maxdepth: 2 reference/cli reference/configuration reference/api ``` ```{toctree} :hidden: :caption: Project :maxdepth: 2 benchmarks/results benchmarking performance-analysis architecture roadmap for-llms ``` ## Why Testenix
0dependencies in the native runtime
12Python and OS combinations in CI
3console, JSON, and JUnit reports
3.15×historical v0.1.0 result: 100k synthetic no-op tests, 4 workers, --no-history
Testenix is deliberately built around a few strong guarantees: - **Async is native.** Coroutine tests and async-generator fixtures use the same model as synchronous code and do not require a plugin. - **Parallelism is part of the runner.** Adaptive worker selection, module affinity, optional safety-checked sharding, process isolation, and duration-aware scheduling are designed together. - **Retries preserve evidence.** A failed attempt followed by a pass is `FLAKY`, never silently rewritten as a clean pass. - **Crashes cannot erase completed work.** Workers stream results as tests finish, and unfinished tests receive explicit terminal outcomes. - **Reports share one model.** Console, JSON, JUnit, history, and the library API are derived from the same versioned result contracts. ## A complete first test Already have a pytest suite? Keep its fixtures, parametrization, markers, classes, configuration, and plugins: ```console $ python -m pip install "testenix[pytest]" $ testenix pytest -q tests ``` [Read the compatibility contract](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/) before migrating individual modules to the native engine. To create a validated native copy without modifying the originals, use: ```console $ testenix migrate auto tests --dry-run $ testenix migrate auto tests --check $ testenix migrate auto tests --output tests_testenix ``` The migrator executes the source baseline and both serial and parallel native candidates in disposable project copies, compares their inventories and outcomes, and publishes only through an atomic no-overwrite rename. [Read the safe migration contract](https://polishdataengineer.github.io/testenix/guides/migration/). ```python from collections.abc import AsyncIterator from testenix import case, cases, fixture, test @fixture(scope="module") async def multiplier() -> AsyncIterator[int]: yield 2 @test("multiplication uses an async fixture", tags={"unit"}) @cases( case(id="positive", value=3, expected=6), case(id="zero", value=0, expected=0), ) async def multiplication(multiplier: int, value: int, expected: int) -> None: assert multiplier * value == expected ``` Run it locally: ```console $ python -m pip install testenix $ testenix run tests ``` `workers = "auto"` adapts to the schedulable work instead of launching one process per CPU. Use `testenix tune` (also available as `testenix benchmark`) for a measured project recommendation. Import-heavy native suites can explicitly generate a source-hashed collection manifest, and large independent modules can opt into conservative `--shard-modules` scheduling. See [parallel execution](https://polishdataengineer.github.io/testenix/guides/parallelism/) for both trust boundaries. To evaluate unreleased source changes, install the current `main` branch with `python -m pip install "testenix @ git+https://github.com/polishdataengineer/testenix.git@main"`. ## Performance evidence, with context The checked-in development baseline measured **Testenix 0.1.0**, not the current release. Native `testenix run` completed 100,000 generated no-op tests across 16 modules on one Apple M4 Pro and CPython 3.11 machine in a median 8.04 seconds, compared with 25.33 seconds for pytest and 21.30 seconds for pytest-xdist. It used four workers, `--no-history`, and pytest-xdist's default `load` scheduler. These measurements do not apply to Testenix 0.3.0, a real project, the default-history mode, or the delegated `testenix pytest` command.
This is historical synthetic evidence from one machine, not a promise that every project will be 3.15× faster. No clean Testenix 0.3.0 scaling matrix is checked in yet. The benchmark page publishes the raw samples, environment, variance, methodology, current matrix status, and limitations.
[Inspect the benchmark data](https://polishdataengineer.github.io/testenix/benchmarks/results/) or [reproduce the harness](https://polishdataengineer.github.io/testenix/benchmarking/). ## Project maturity Testenix is alpha software. The `testenix pytest` bridge preserves an existing suite by delegating to real pytest, while `testenix run` is a distinct native engine rather than a drop-in pytest reimplementation. Pytest still has a much broader plugin ecosystem, richer IDE integration, and mature assertion rewriting. Choose the bridge for compatibility and the native engine when its async model, built-in parallel execution, explicit failure semantics, or dependency-free core are more important. --- # Document: Getting started Canonical URL: https://polishdataengineer.github.io/testenix/getting-started/ Source: docs/getting-started.md # Getting started This guide takes a new project from installation to its first parallel Testenix run. ## Requirements - CPython 3.11 or newer - Linux, macOS, or Windows - no runtime dependencies beyond the Python standard library for native `testenix run` - pytest in the same environment when using the optional compatibility bridge ## Install Install the released package from PyPI: ```console $ python -m pip install testenix $ testenix --version ``` For a project managed by uv: ```console $ uv add --dev testenix $ uv run testenix --version ``` For an existing pytest project, install the compatibility extra: ```console $ python -m pip install "testenix[pytest]" # or $ uv add --dev "testenix[pytest]" ``` To evaluate unreleased development changes, install directly from the protected `main` branch: ```console $ python -m pip install "testenix @ git+https://github.com/polishdataengineer/testenix.git@main" # include pytest when the project environment does not already provide it $ python -m pip install "testenix[pytest] @ git+https://github.com/polishdataengineer/testenix.git@main" ``` ## Choose an execution mode Run an unchanged pytest suite through its real engine: ```console $ testenix pytest -q --tb=short tests ``` Use `testenix run` for native Testenix tests and the built-in scheduler, retries, history, and lossless reports. The two commands have deliberately separate semantics. See [pytest compatibility](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/) for the capability matrix and migration boundary. Create a validated native copy of supported pytest or unittest modules without changing the originals: ```console $ testenix migrate auto tests --dry-run $ testenix migrate auto tests --output tests_testenix $ testenix run tests_testenix ``` Migration requires a green source run and matching serial/parallel native runs before an atomic new-directory publish. Read [safe migration](https://polishdataengineer.github.io/testenix/guides/migration/) before using it on a suite with external side effects. ## Create a native test Testenix collects ordinary functions whose names begin with `test_`. Decorators are optional for simple tests. ```python # tests/test_math.py def test_addition() -> None: assert 2 + 2 == 4 ``` Run the suite: ```console $ testenix run tests ``` The default console output is a compact per-file report followed by complete failure details and a final summary. It is rendered deterministically after the run rather than updated as live progress. Use `-q` to omit the header and file table, `-v` for one row per test, or `-vv` for worker, attempt, and phase metadata. Collection errors and failure details remain visible with `-q`. ```console $ testenix run -q tests 4 tests, 3 passed, 1 skipped in 0.071s ``` The process exits with code `0` when every selected test has a non-gating terminal status. ## Add native metadata Use `@test` when a description, tags, or a hard timeout should be part of the test contract. ```python from testenix import test @test("addition remains correct", tags={"unit", "fast"}, timeout=2.0) def addition() -> None: assert 2 + 2 == 4 ``` Select tagged tests by repeating `--tag`. Repeated tags use AND semantics: ```console $ testenix run --tag unit --tag fast ``` ## Configure the project Put stable defaults in `pyproject.toml`: ```toml [tool.testenix] paths = ["tests"] workers = "auto" retries = 0 history = ".testenix/history.sqlite3" # shard_modules = true # manifest = ".testenix/collection.json" # timeout = 10 # json = "reports/testenix.json" # junit = "reports/junit.xml" ``` Command-line values override the project table: ```console $ testenix run --workers 4 --retries 1 --json reports/result.json ``` Use `--no-history` for a side-effect-free run. Duration history normally helps later runs schedule long tests earlier. `workers = "auto"` adapts to the selected suite instead of copying the logical CPU count. It is capped by the actual schedulable units and uses duration history when enough is available. Measure an explicit project setting with: ```console $ testenix tune tests --warmups 1 --repeats 5 $ testenix tune --write ``` `testenix benchmark` is an alias for `testenix tune`. Module affinity is the safe default. For a large module whose tests are known to be independent, `--shard-modules` opts eligible tests into finer scheduling after conservative static checks. Read [parallel execution](https://polishdataengineer.github.io/testenix/guides/parallelism/) before enabling it. To avoid repeating collection imports on later unchanged runs, create and trust a source-hashed manifest explicitly: ```console $ testenix manifest tests --output .testenix/collection.json $ testenix run tests --manifest .testenix/collection.json ``` Testenix verifies the complete file inventory and every SHA-256 before reuse. A stale manifest falls back to supervised collection rather than running a stale test selection. Parameter names remain visible in the artifact, but their values are redacted. ## Use it from Python The runner also exposes a typed API: ```python from testenix import Status, TestenixConfig, run def main() -> None: result = run("tests", TestenixConfig(workers=4, history_path=None)) failed = [item.test.id for item in result.tests if item.status is not Status.PASS] print(failed) if __name__ == "__main__": main() ``` Async applications can call `await testenix.run_async(...)`. Cancellation terminates active collection and execution process trees before control returns to the caller. Executable scripts must place the top-level call behind the same `if __name__ == "__main__":` guard on every platform. ## Next steps - [Write tests, cases, tags, skips, and expected failures](https://polishdataengineer.github.io/testenix/guides/writing-tests/) - [Run or migrate an existing pytest suite](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/) - [Convert pytest or unittest safely without replacing originals](https://polishdataengineer.github.io/testenix/guides/migration/) - [Build fixture graphs](https://polishdataengineer.github.io/testenix/guides/fixtures/) - [Understand process parallelism and timeouts](https://polishdataengineer.github.io/testenix/guides/parallelism/) - [Produce JSON and JUnit reports](https://polishdataengineer.github.io/testenix/guides/reports/) - [Review every configuration option](https://polishdataengineer.github.io/testenix/reference/configuration/) --- # Document: Pytest compatibility Canonical URL: https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/ Source: docs/guides/pytest-compatibility.md --- description: Run existing pytest suites unchanged through Testenix and understand the native migration boundary. --- # Pytest compatibility Testenix can run an existing pytest suite without rewriting it. Use the compatibility bridge when the suite depends on pytest fixtures, parametrization, markers, classes, plugins, hooks, or configuration: ```console $ python -m pip install "testenix[pytest]" $ testenix pytest -q --tb=short tests ``` For the supported static subset, `testenix migrate pytest tests` can instead create a validated native copy without modifying the source. [The safe migration guide](https://polishdataengineer.github.io/testenix/guides/migration/) defines its strict support and rollback contract. If a supported pytest (`>=8.3,<10`) is already installed in the same Python environment, installing the base `testenix` package is enough. The extra is a convenience for environments that do not already contain pytest. ## What the command does Everything after `testenix pytest` is forwarded unchanged to: ```console $ python -m pytest [PYTEST_ARGS ...] ``` Testenix hands its current CLI process to pytest from the same Python interpreter. On POSIX it uses a process overlay; on Windows it calls pytest's public console entry point in-process because the platform does not provide the same overlay semantics. Both paths preserve the foreground process, working directory, environment, terminal, standard streams, and pytest's signal handling. Pytest therefore remains responsible for collection, execution, configuration, plugin loading, output, descendants such as pytest-xdist workers, and exit status. This also explains the visual style: output from `uv run pytest tests` is rendered by pytest, and `testenix pytest` deliberately preserves that same renderer. It does not activate Testenix's compact native console. For shorter pytest output with concise tracebacks, use: ```console $ testenix pytest -q --tb=short tests ``` ```console $ testenix pytest tests/test_api.py::test_health -k smoke --maxfail=1 $ testenix pytest -m "unit and not slow" tests $ testenix pytest -n auto tests $ testenix pytest --junitxml=reports/pytest.xml tests ``` The `-n` example requires pytest-xdist in the same environment. Testenix does not translate its native `--workers` option into an xdist option. ## Capability matrix | Capability | `testenix pytest` | `migrate` then `run` | Direct `testenix run` | | --- | --- | --- | --- | | Plain module-level pytest-default `test*` functions | Yes, through pytest | Yes | `test_` or native `@test` | | Simple pytest classes | Yes | Fresh-instance wrappers | No direct class collection | | Complex class lifecycle/inheritance | Yes | Blocked | No | | Simple local fixture | Yes | Converted, including static autouse | Only native `@fixture` | | Session-scoped pytest fixture | Yes | Blocked: run-global vs worker-local | Native scope is worker-local | | `tmp_path` | Yes | Native built-in | Native built-in | | `monkeypatch` | Yes | `setattr`/`setenv` subset | Native reversible subset | | Other built-in/dynamic fixture | Yes | Blocked | No pytest fixture semantics | | Adjacent `conftest.py` fixture | Yes | Simple static subset | No automatic conftest discovery | | Static single `parametrize` | Yes | Converted to cases | Use `@case` or `@cases` | | Skip and plain custom marker | Yes | Converted | Use Testenix decorators/tags | | Pytest xfail | Yes | Blocked due semantic differences | Use native `@xfail` intentionally | | Pytest assertion rewriting | Yes | No | No | | Plugins, hooks, pytest config | Yes | Blocked/not translated | No | | Bare `@pytest.mark.asyncio` coroutine | According to plugin | Fresh function-scoped loop wrapper | Native async needs no plugin | | Configured async/anyio plugin semantics | According to plugins | Blocked | No plugin semantics | | Testenix worker scheduler/history | No | Yes after migration | Yes | | Testenix retries and lossless results | No | Yes after migration | Yes | | Published native speedups | No | Measure generated suite | Only documented workloads | | Source preservation | Runs source | Source stays untouched | Not applicable | Some simple pytest-authored functions also happen to run with `testenix run`, but that overlap is not a compatibility guarantee. In particular, the native collector does not interpret pytest markers. Running a pytest suite through the native command can turn a skip or xfail into an ordinary execution, collapse parametrized cases, miss class-based tests, or fail fixture setup. Use the explicit compatibility command until a suite has been intentionally migrated. ## Boundaries `testenix pytest` does not use `[tool.testenix]`, the native collector, worker pool, retries, timeouts, tags, history, event model, JSON reporter, or JUnit reporter. Pass the corresponding pytest or plugin options after the subcommand. For example, use pytest's `--junitxml`, not Testenix's native `--junit`. Native presentation options are not interpreted by the bridge either. Arguments such as `-q`, `-v`, `--color`, `--show-skips`, and `--durations` go straight to pytest and follow pytest's syntax and semantics. For example, pytest uses `-rs` for skipped reasons, `--durations=N` for its slowest tests list, and `--color=yes|no|auto`; Testenix does not translate the native spellings. Pytest and every required plugin must be installed beside the `testenix` executable in the same interpreter environment. For uv-managed projects, prefer: ```console $ uv add --dev "testenix[pytest]" $ uv run testenix pytest -q --tb=short tests ``` An isolated `uv tool install testenix` environment does not automatically see pytest plugins from the project environment. Install the required packages into the tool environment or invoke Testenix through the project environment instead. After the handoff, pytest owns signal handling and returns any status chosen by pytest or a plugin, unchanged. If pytest is missing, Testenix returns `2` with an installation hint before the handoff. Failure to hand the process to pytest returns `3`. ## Performance and benchmark interpretation Compatibility mode has pytest's execution performance plus launcher and adapter overhead, which has not yet been measured separately. It is not expected to be faster than invoking `python -m pytest` directly. The published Testenix benchmarks exercise the native `testenix run` engine and must not be used to describe `testenix pytest`. ## Migration path Use the bridge first to put Testenix in front of an unchanged suite without altering its semantics. Then let the conservative migrator identify and validate modules whose pytest dependencies have explicit equivalents: 1. keep the whole suite green with `testenix pytest`; 2. run `testenix migrate pytest tests --dry-run` and inspect every diagnostic; 3. use `--check --report-json ...` to execute the source, serial native, and parallel native gates; 4. publish to a new directory only after all three inventories/outcomes agree; 5. keep unsupported modules on `testenix pytest` and keep all originals during rollout; 6. benchmark the generated directory before making a native performance claim. The migrator replaces simple fixtures, parametrization, skip conditions, plain markers, bare pytest-asyncio coroutine markers with a fresh closed loop per test or case, and the supported built-in fixtures in the generated copy. It wraps simple pytest class methods with a fresh instance per test. It never performs edits in place. Manual rewrites are still necessary for blocked plugin, hook, complex class lifecycle, xfail, and dynamic behavior. The current bridge does not convert pytest outcomes into a Testenix `RunResult`. Deeper event and report aggregation is a future compatibility layer and requires explicit pytest hook integration. --- # Document: Safe migration Canonical URL: https://polishdataengineer.github.io/testenix/guides/migration/ Source: docs/guides/migration.md --- description: Safely convert supported pytest and unittest suites to native Testenix without replacing the originals. --- # Safe migration to native Testenix An existing suite can use Testenix in three ways: | Goal | Command | Execution engine | Original suite | | --- | --- | --- | --- | | Preserve all pytest behavior | `testenix pytest -q tests` | pytest | Runs directly | | Check whether native conversion is supported | `testenix migrate auto tests --dry-run` | no tests run | Read-only | | Create a validated native copy | `testenix migrate auto tests -o tests_testenix` | Testenix after conversion | Kept and used as the baseline | Migration is conservative. It stops on a construct whose behavior cannot be proven equivalent instead of producing a plausible-looking, incomplete suite. ## First migration For pytest, install the optional dependency because validation executes the real source suite: ```console $ python -m pip install "testenix[pytest]" $ testenix migrate pytest tests --dry-run $ testenix migrate pytest tests --check --workers 2 \ --report-json reports/migration-check.json $ testenix migrate pytest tests --output tests_testenix \ --report-json reports/migration.json $ testenix run tests_testenix ``` For a standard-library unittest suite, no extra package is needed: ```console $ testenix migrate unittest tests --output tests_testenix ``` Use `auto` when pytest-style modules and unittest modules live in the same selected directory. They must be separate modules; a file that mixes both test models is rejected. The destination defaults to `testenix_migrated`. It must be a new directory inside the project, with an existing real parent. There is deliberately no `--force` and no in-place mode. An integer `--workers` value must be at least 2 so the parallel gate cannot silently repeat the serial command. During `--check` or publication, a one-module candidate still has one schedulable affinity unit, which is disclosed as a `MIG006` warning; spread tests across independent modules to exercise multiple workers. The warning is not shown for `--dry-run` or an already-blocked migration because no parallel candidate is run in either case. An audit-report path must also be new, inside the project, and disjoint from every selected source and the output suite. Testenix never replaces an existing report. ## Transaction and rollback contract Testenix uses copy-and-validate rather than edit-and-undo: ```text preflight paths -> snapshot source names + SHA-256 -> convert in memory -> run original suite in disposable project copy -> run generated suite with one Testenix worker in a fresh copy -> run generated suite in parallel in another fresh copy -> compare inventory, totals, and every mapped test outcome -> recheck every source hash -> atomically publish a new directory without replacement ``` If parsing, conversion, the source baseline, either native run, parity comparison, a source hash, or publication fails before the atomic rename, the requested output directory remains absent. A failure to write the optional report after a successful rename instead leaves the validated output published, prints an explicit warning, and returns the successful migration status. Testenix never writes to, renames, deletes, or replaces a selected source file. There are therefore no old files to restore: the old suite was never changed. `--dry-run` stops after static analysis and a final source-hash check. `--check` performs the full three-run validation but does not publish the destination. The default performs the same checks and then uses an operating-system no-replace rename. If another process creates the destination during validation, publication fails closed. POSIX staging creation, artifact writes, publication, and cleanup are anchored to open directory descriptors. On a platform without safe descriptor-relative recursive deletion, a non-empty failed staging transaction is retained under `.testenix/migrations` for inspection instead of being removed with a path-based recursive walk. Validation runs with the shadow as both the current directory and `PWD`, so ordinary relative-path writes land in the disposable copy. A shadow is not an operating-system security sandbox: a test can still write through a hard-coded absolute path or another environment variable, and network, database, cloud, and subprocess side effects remain external. Selected Python sources are rehashed before every terminal result, so detected drift stops publication, but Testenix never tries to undo an independent user or test-process edit. Use test credentials and `--dry-run` first for suites with external effects. ## Pytest conversion contract The converter subset introduced in v0.2 supports the behavior below: - module-level pytest-default `test*` functions and normal Python `assert` statements; - simple `Test*` classes with a fresh zero-argument instance per test method, including ordinary helper methods; inheritance, class decorators, custom construction, and pytest class lifecycle hooks remain outside the safe subset; - bare `@pytest.mark.asyncio` on `async def` tests. The generated synchronous wrapper runs every test or parametrized case in a fresh, closed `asyncio.Runner`, matching pytest-asyncio's default function-scoped loop isolation; - one static `pytest.mark.parametrize` with static names, rows, IDs, and unmarked `pytest.param(..., id=...)` values; - local fixtures using function or module scope, including a statically boolean `autouse=True`; - simple fixtures from an adjacent `conftest.py` in the same directory; - native `tmp_path`, which supplies a fresh `pathlib.Path` and removes its temporary directory at test teardown; - native `monkeypatch.setattr` in object/attribute and dotted-import forms, plus `setenv` and idempotent `undo`; successful changes are restored in LIFO order during test teardown. `monkeypatch` may also flow through statically resolved module-local helpers when every use can be proven to stay inside this supported subset; - static `pytest.mark.skip` and `pytest.mark.skipif`; - plain argument-free custom markers, converted to Testenix tags; - pytest runtime helpers `approx`, `deprecated_call`, `fail`, `raises`, and `warns`. Generated modules using these helpers still require pytest at runtime. It blocks, with a file and line diagnostic: - complex pytest test classes, xfail, runtime skip/xfail/importorskip/exit, and xunit lifecycle hooks; - built-in fixtures other than `tmp_path` and `monkeypatch`, such as `capsys`, `caplog`, and `request`; monkeypatch operations outside the documented native subset, imported or dynamically rebound helpers, and values that escape static analysis are also unsupported; - dynamically configured autouse fixtures, parametrized fixtures, fixture overrides, and inherited ancestor-`conftest` fixtures; - session-scoped fixtures, because pytest creates one per run while Testenix session scope is currently worker-local; - stacked, dynamic, indirect, scoped, or per-case-marked parametrization; - `usefixtures`, module-level `pytestmark`, hook functions, plugin registration, and semantic plugin markers such as anyio, configured asyncio, timeout, order, repeat, or flaky; - unmarked async tests, async fixtures, custom `event_loop_policy`, non-function asyncio loop scopes, and enabled asyncio debug mode. Testenix checks the effective pytest configuration and relevant `PYTEST_ADDOPTS` overrides before accepting bare asyncio markers; - decorators and required parameters whose execution meaning cannot be established statically. Any converted pytest file whose name is not already `test_*.py` is renamed in the generated copy to a native-discoverable name. Explicit nonstandard files such as `specs.py` are supported when they contain static test inventory. Supporting Python modules and package `__init__.py` files under the selected source directories are copied to preserve common relative imports. Missing non-Python assets or path-sensitive `__file__` behavior will make candidate validation fail rather than publish a bad copy. Pytest plugins are intentionally not emulated. Keep using `testenix pytest` for modules that need them and migrate supported modules incrementally. ## Unittest conversion contract Rewriting `self.assertEqual`, lifecycle methods, cleanup stacks, and mock decorators would be fragile. Instead, Testenix generates one native function wrapper per test method. The wrapper: 1. resolves the original by the exact wrapper-to-project relative path, independent of `cwd`; 2. verifies a manifest containing every selected Python source and SHA-256; 3. loads the original class by exact path; 4. executes one method through the standard `TestCase.run(TestResult)` protocol; 5. exposes its pass, failure, skip, expected-failure, or unexpected-success outcome to Testenix. This preserves direct subclasses of `unittest.TestCase` and `unittest.IsolatedAsyncioTestCase`, including per-test `setUp`/`tearDown`, async lifecycle, `addCleanup`, `assert*`, context-manager assertions, and `unittest.mock.patch`. Static skips map to Testenix `SKIP`; `expectedFailure` maps to `XFAIL`; an unexpected success remains the gating `XPASS` outcome. The generated unittest suite deliberately depends on the unchanged originals. Do not delete or move them, and do not move or rename the published generated directory: wrappers encode the exact relative relationship between both trees. If either path or an original file changes, rerun migration; a changed source makes native collection fail with an explicit diagnostic. The converter blocks `subTest`, runtime `skipTest`/`SkipTest`, class/module lifecycle and cleanup, custom `run` or loader hooks, `load_tests`, mixins or indirect inheritance, `FunctionTestCase`, dynamic/DDT/parameterized method generation, metaclasses, and ambiguous decorators. ## What validation compares The source suite must be green. Testenix records and compares: - the exact collected test count against the converter's source-to-target mapping; - passed, failed, error, skipped, expected-failure, and unexpected-success totals; - the outcome of every source test against its exact generated target mapping; - native serial and native parallel outcome signatures; - every selected Python source path and SHA-256 immediately before publication. A pre-existing source failure is not treated as proof of equivalent conversion, even when the candidate happens to fail too. Fix the baseline or use `--dry-run` to inspect static support. The JSON audit report uses the versioned `testenix.migration-report` format. It includes source hashes, every source-to-target mapping, per-test outcomes, generated files, line diagnostics, timings and summaries for all validation runs, publication status, and an `originals_modified` flag. It is false for a successful transaction and true when a terminal source recheck detects drift; the flag reports observed state and does not claim that Testenix caused an independent edit. The console groups repeated diagnostics by code and shows the first location, so large suites do not produce hundreds of near-identical lines. `--report-json FILE` and `--report-json -` always retain every individual source- and line-addressed diagnostic. On a blocked transaction, the console calls any safe in-memory result a *statically convertible subset* rather than implying that those tests were published. ## Performance with thousands of migrated tests Migration unlocks the native scheduler, process supervision, async engine, retries, history, and Testenix reports. It does not guarantee that every converted suite is faster. By default Testenix keeps a normal module as one affinity unit so module-scoped fixtures are not duplicated. Consequently, 3,000 tests in one source module still form one schedulable unit; 3,000 tests spread across enough independent modules can use multiple workers. A project may later opt eligible native modules into `--shard-modules`, but only after validating the generated suite and the documented static-analysis trust boundary. Run `testenix tune` on the published native copy before fixing its CI worker count. The checked-in migration baseline used an Apple M4 Pro, CPython 3.11, 3,000 generated tests in 64 modules, four native workers, one warm-up, and five measured rounds: | Source runner | Test body | Source median | Native median | Native result | One-time migration | | --- | --- | ---: | ---: | ---: | ---: | | sequential pytest | no-op | 1.539 s | 0.521 s | 2.96x faster | 5.940 s | | sequential unittest outcome probe | no-op | 0.161 s | 1.192 s | 7.40x slower | 6.742 s | | sequential unittest outcome probe | 1 ms sleep | 4.066 s | 2.577 s | 1.58x faster | 17.251 s | The migration column is the complete copy, source-baseline, serial-candidate, parallel-candidate, integrity-check, and publication transaction. It is paid when regenerating a suite and is deliberately excluded from the recurring execution medians. The unittest adapter loads the unchanged source and translates `TestCase.run()` results for every native wrapper. That fixed cost dominates an empty method; once the generated methods contain 1 ms of synthetic work, parallel execution across 64 modules outweighs it in this scenario. These source measurements use sequential pytest and a sequential Testenix probe built on the stdlib unittest loader/result protocol. The probe serializes per-test outcomes for the parity gate, so its small audit overhead is included. They are not comparisons against pytest-xdist or a third-party parallel unittest runner. They are synthetic timing evidence, not a promise for a real project: imports, fixtures, I/O, test-duration distribution, module affinity, worker count, operating system, and CPU can all reverse the result. See the [generated benchmark page](https://polishdataengineer.github.io/testenix/benchmarks/results/) for raw samples and variance, or inspect the checked-in [pytest](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_pytest_3000.json), [unittest no-op](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_unittest_3000.json), and [unittest 1 ms](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_unittest_3000_delay_1ms.json) JSON files directly. Benchmark the generated directory only after parity validation: ```console $ python benchmarks/run_migration_benchmark.py \ --framework pytest --tests 3000 --modules 64 --workers 4 \ --warmups 1 --repeats 5 --output migration-pytest-3000.json $ python benchmarks/run_migration_benchmark.py \ --framework unittest --tests 3000 --modules 64 --workers 4 \ --warmups 1 --repeats 5 --output migration-unittest-3000.json $ python benchmarks/run_migration_benchmark.py \ --framework unittest --tests 3000 --modules 64 --workers 4 --delay-ms 1 \ --warmups 1 --repeats 5 --output migration-unittest-3000-delay-1ms.json ``` The harness checks the exact count and source hashes before accepting any timing sample. Publish the raw JSON, environment, module count, worker count, variance, and commands with a result. Do not apply the repository's synthetic native benchmark ratio to a real migrated project without measuring it. ## CI rollout A low-risk rollout keeps both suites for several releases and runs the exact newly published copy: ```yaml - name: Generate and validate a fresh native copy run: | testenix migrate auto tests --output tests_testenix_ci --workers 2 \ --report-json migration-report.json - name: Existing source suite run: testenix pytest -q tests - name: Run the exact validated copy run: testenix run tests_testenix_ci --no-history ``` The output and report paths must be absent at job start, which is natural on a fresh CI checkout. `--check` validates an ephemeral candidate; it does not prove that a separately committed `tests_testenix` directory has identical bytes. Regenerate and publish a fresh CI output as above, or add a project-specific content-manifest check for a committed copy. Regenerate whenever a pinned unittest source changes. ## Exit codes | Code | Migration meaning | | ---: | --- | | `0` | Analysis, check, or publication completed successfully. | | `1` | Source or candidate execution failed, timed out, or produced different outcomes. | | `2` | Unsafe/invalid path, existing output, source drift, or command usage error. | | `3` | Internal process or pre-publication report failure. A report-only failure after publication warns and preserves exit `0`. | | `4` | At least one construct is unsupported by the conservative converter. | | `130` | Interrupted by the user. | Diagnostics beginning with `PYT` describe pytest source, `UNIT` describes unittest source, and `MIG` describes cross-framework inventory or transaction safety. --- # Document: Writing tests Canonical URL: https://polishdataengineer.github.io/testenix/guides/writing-tests/ Source: docs/guides/writing-tests.md # Writing tests Testenix tests are ordinary typed functions. Discovery recognizes `test_*` names, while decorators add explicit metadata without wrapping or changing the callable. ## Plain functions ```python def test_total() -> None: assert sum([2, 3, 5]) == 10 ``` Both synchronous functions and coroutines are supported: ```python async def test_async_client() -> None: response = await fetch_record(42) assert response["id"] == 42 ``` ## Descriptions, tags, and timeouts ```python from testenix import test @test( "the cache expires stale entries", tags={"unit", "cache"}, timeout=1.5, ) async def cache_expiry() -> None: ... ``` The decorator attaches immutable metadata and preserves the original function signature, annotations, and traceback. A timeout is a hard process deadline. This is more expensive than shared-worker execution, but it can stop both a blocked coroutine and a blocking synchronous call. ## Parameter cases Use `case` for named examples: ```python from testenix import case, cases, test @test("discount rules") @cases( case("regular", amount=100, rate=0.0, expected=100), case("member", amount=100, rate=0.1, expected=90), ) def discount(amount: int, rate: float, expected: float) -> None: assert amount * (1 - rate) == expected ``` Use keyword dimensions to create a Cartesian product: ```python @cases( role=["admin", "editor"], active=[True, False], ) def test_permissions(role: str, active: bool) -> None: ... ``` Case values are rebuilt when the worker rediscovers the module, so they do not need to cross the process boundary through pickle. They must still be reproducible during module import. ## Skip and expected failure ```python import sys from testenix import skip, xfail @skip("Windows-only behavior", when=sys.platform != "win32") def test_windows_registry() -> None: ... @xfail("tracked as issue #42") def test_known_edge_case() -> None: assert current_behavior() == desired_behavior() ``` An expected failure becomes `XFAIL`. If the test unexpectedly passes, the result is `XPASS` and gates the run. ## Retries and flakiness Retries can be configured globally or on the command line: ```console $ testenix run --retries 2 ``` Every attempt remains in the result model. `FAIL -> PASS` is finalized as `FLAKY` and returns a gating exit code; Testenix never hides the initial failure. ## Tags Tags are normalized strings stored in the test specification: ```python @test(tags={"integration", "database"}) def database_round_trip() -> None: ... ``` Run tests containing every requested tag: ```console $ testenix run --tag integration --tag database ``` An explicit tag filter that selects no tests is treated as a usage error and exits with code `2`. --- # Document: Fixtures Canonical URL: https://polishdataengineer.github.io/testenix/guides/fixtures/ Source: docs/guides/fixtures.md # Fixtures Fixtures are typed dependency providers. A test requests a fixture by using its name as a function parameter. ## Return-value fixtures ```python from testenix import fixture @fixture def customer_id() -> int: return 42 def test_customer(customer_id: int) -> None: assert customer_id > 0 ``` ## Cleanup with generators Synchronous and asynchronous generator fixtures run cleanup after the consumer: ```python from collections.abc import AsyncIterator from testenix import fixture @fixture async def client() -> AsyncIterator[Client]: value = await Client.connect() try: yield value finally: await value.close() ``` Setup, call, and teardown are separate result phases. A teardown failure remains visible even when the test body passed. ## Fixture dependencies Fixtures can request other fixtures: ```python @fixture def database_url() -> str: return "sqlite:///:memory:" @fixture def repository(database_url: str) -> Repository: return Repository(database_url) def test_empty_repository(repository: Repository) -> None: assert repository.count() == 0 ``` The dependency graph is validated before execution. Missing fixtures and cycles become collection issues instead of hanging the run. ## Built-in fixtures Testenix 0.3 provides two dependency-free, test-scoped built-ins by name: ```python from pathlib import Path def test_isolated_file(tmp_path: Path, monkeypatch) -> None: target = tmp_path / "value.txt" target.write_text("ok", encoding="utf-8") monkeypatch.setenv("TESTENIX_EXAMPLE", "enabled") assert target.read_text(encoding="utf-8") == "ok" ``` `tmp_path` is a fresh `pathlib.Path` removed during teardown. `monkeypatch` supports reversible object/attribute and dotted-import `setattr`, `setenv`, and idempotent `undo`. Changes are restored in LIFO order even when the test fails. Other pytest monkeypatch operations and pytest built-ins such as `capsys`, `caplog`, and `request` are not native Testenix fixtures. ## Autouse fixtures Use `autouse=True` when setup and cleanup must apply to every test that can see a fixture: ```python @fixture(autouse=True) def isolated_environment(monkeypatch): monkeypatch.setenv("APP_ENV", "test") yield ``` Explicitly requesting the same fixture still resolves one cached value for the test. A local fixture definition overrides a visible imported definition with the same name before Testenix chooses which fixtures run automatically. ## Scopes ```python @fixture(scope="module") def module_resource() -> Resource: return Resource() @fixture(scope="session") def worker_resource() -> Resource: return Resource() ``` | Scope | Lifetime in Testenix 0.3 | | --- | --- | | `test` | One instance for one concrete test attempt. | | `module` | Shared by normal tests from the module inside one worker. | | `session` | Shared by normal tests assigned to one worker process. | Session scope is currently worker-local, not a single process-global instance. Timed tests run in dedicated processes and therefore receive isolated module/session fixtures. ## Custom fixture names ```python @fixture(name="api") def build_client() -> Client: return Client() def test_health(api: Client) -> None: assert api.health() == "ok" ``` The custom name changes dependency lookup without changing the Python function. --- # Document: Parallel execution Canonical URL: https://polishdataengineer.github.io/testenix/guides/parallelism/ Source: docs/guides/parallelism.md # Parallel execution Testenix treats local parallelism as part of the execution model rather than an optional plugin. ## Choose the worker count ```console $ testenix run --workers auto $ testenix run --workers 4 $ testenix run --workers 1 ``` `auto` is adaptive; it is not an alias for the logical CPU count. Testenix first counts the execution units it can actually schedule (module-affinity groups, isolated timeout tests, and any eligible opt-in module shards). It caps the choice by that count and the available CPUs. With reliable duration history, a startup-cost/makespan model chooses the smallest worker count within a narrow tolerance of the predicted best. Without enough history, cold-start auto uses a conservative cap so a short suite does not launch one process per CPU. An explicit number remains a hard project setting. Prefer one in tightly budgeted CI; for a project-specific measured value, use the tuner. ## Tune a project ```console $ testenix tune tests --warmups 1 --repeats 5 $ testenix benchmark tests --candidates 1,2,4,8 $ testenix tune --json reports/tuning.json --write ``` `benchmark` is an alias for `tune`. The command runs fresh CLI processes, uses a green one-worker inventory probe, disables history for all timing samples, measures native candidates in alternating order, and rejects a candidate whose test inventory or outcomes differ. The recommendation is the smallest worker count within a narrow tolerance of the best median, avoiding a larger setting for measurement noise. `--write` stores that integer in `[tool.testenix].workers`; it is an explicit file change, never an effect of `workers = "auto"`. The automatic sweep respects the process-visible CPU capacity and tests at most 1/2/4 workers; larger counts must be requested explicitly with `--candidates`. Each complete suite run has a 300-second deadline by default (`--run-timeout SECONDS`). Windows starts the command suspended and attaches a kill-on-close Job Object before resume. POSIX always signals the new root session and identity-checks observed detached descendants. A child that calls `setsid()` and exits between creation and the first process snapshot is outside an absolute kernel-containment guarantee; use a container for hostile suites. Pass `--shard-modules` to tune that explicit mode and `--manifest FILE` to include the verified single-import collection path in every native sample. When `--write` is present, Testenix refuses a transient sharding or manifest override that differs from `[tool.testenix]`, because writing only `workers` would persist a recommendation for a different execution profile. Configure the profile first or tune it without `--write`. It also fingerprints project Python/TOML sources, explicit suite files (including linked source directories), and the trusted manifest, then rejects any observed content or file-identity drift after a sample. The final configuration update rechecks the original bytes immediately before an atomic replacement. Use an immutable checkout for publishable tuning: it excludes a writer racing that last filesystem operation, while installed packages, non-source data, and other runtime dependencies remain external inputs. Use `--pytest-source PATH` to add an optional pytest timing for a corresponding source suite. The tuner's primary contract is worker selection for the native suite. Its optional pytest ratio is orientation for that exact invocation, not sufficient evidence for a public speed claim; public comparisons must also satisfy the [benchmarking contract](https://polishdataengineer.github.io/testenix/benchmarking/). ## Scheduling Normal tests from one module form an affinity unit and execute in the same persistent worker. This preserves module-fixture reuse and avoids splitting hidden module state between processes. When duration history exists, Testenix schedules longer units first. This longest-processing-time strategy is deterministic and reduces the chance that one slow shard becomes the tail of the run. ## Opt-in intra-module sharding One large module normally exposes one unit no matter how many tests it contains. If its tests are known to be independent, explicitly request finer units: ```console $ testenix run tests --shard-modules ``` or configure: ```toml [tool.testenix] shard_modules = true ``` This is deliberately off by default. Before splitting a module, Testenix statically fails closed when it detects module- or session-scoped fixtures, imported fixture providers, direct writes to module globals, nested mutable containers, mutable class state, or executable import-time lifecycle behavior. Function-scoped fixtures defined in the collected module, including autouse fixtures, may be recreated in separate workers and do not block sharding. Eager calls in module assignments, annotations, decorators, function defaults, and class bases or keywords are treated as import-time lifecycle behavior and keep the module intact. Static analysis cannot prove that arbitrary calls, imported libraries, environment state, or external services are free of shared effects. Enabling the option is therefore a project trust decision. Validate the suite both with and without sharding before adopting it in CI. Modules that fail the safety check retain normal module affinity while eligible modules can be split. ## Avoid the collection-side import Without a manifest, safe supervised collection imports selected modules once, then execution workers import their assigned modules again to materialize functions and case values. Projects for which imports are a meaningful part of wall time can generate an explicit trusted manifest: ```console $ testenix manifest tests --output .testenix/collection.json $ testenix run tests --manifest .testenix/collection.json ``` The manifest records collection roots, every selected test file plus statically discoverable project-local Python import dependencies and their SHA-256 hashes, test metadata, collection issues, and sharding decisions. Parameter names are retained but values are redacted so collection-time environment data is not persisted. Each run verifies the requested roots, the exact file set, and every source digest before reusing it. Malformed manifest JSON is rejected as invalid input. If a file was added or removed, a source changed, or current-source verification cannot establish an exact match, the manifest is stale and Testenix falls back to the normal isolated collection process. It does not execute a stale selection. This optimization removes the collection-side import on an unchanged run; execution workers still import the code they execute. It is not an implicit cache. If collection depends on environment variables, generated files outside the selected roots, network state, or other inputs not captured by source hashes, the producer must regenerate the manifest when those inputs change. Set `manifest = ".testenix/collection.json"` in `[tool.testenix]` only when that trust boundary is appropriate. ## Process isolation Workers are spawned processes. A worker streams each completed attempt to the coordinator before starting the next test. If the worker later crashes: - completed results remain authoritative; - unfinished tests receive explicit crash or infrastructure outcomes; - the crashed unit receives one framework recovery attempt; - `CRASH -> PASS` is still finalized as `FLAKY`. The coordinator never infers that a missing result passed. ## Hard timeouts Any test with an explicit or global timeout runs in its own supervised process: ```python from testenix import test @test(timeout=2) def test_external_tool() -> None: call_external_tool() ``` This boundary allows Testenix to terminate a blocking synchronous call as well as a stuck coroutine. Descendant processes are terminated with the timed-out worker. The trade-off is fixture reuse: a timed test cannot share module/session fixture instances with neighbouring tests. ## Cancellation The async library API is cancellation-aware: ```python import asyncio from testenix import TestenixConfig, run_async async def main() -> None: result = await run_async(("tests",), TestenixConfig(workers=4)) print(result.exit_code) if __name__ == "__main__": asyncio.run(main()) ``` Cancelling `run_async` terminates active collection and execution process trees before the coroutine returns control. ## Platform note On every supported platform, executable scripts that call `run()` or `run_async()` directly must use the standard multiprocessing guard because the supervised worker protocol uses the `spawn` start method: ```python if __name__ == "__main__": main() ``` The `testenix` command-line program handles process startup itself. --- # Document: Reports and history Canonical URL: https://polishdataengineer.github.io/testenix/guides/reports/ Source: docs/guides/reports.md # Reports and history Every output adapter consumes the same immutable run result. A status cannot be green in the console and red in JSON because both are derived from one model. ## Console The console report is always enabled. Its default mode is compact: Testenix prints an aggregate row for each source file, then complete failure details and the final summary. ```console $ testenix run tests Testenix | 5 tests | 2 files | 2 workers PASS tests/test_accounts.py 3 passed [9ms] FAIL tests/test_checkout.py 1 passed, 1 failed [12ms] Problems (1) FAIL tests/test_checkout.py::test_rejects_expired_card attempt 1, call: expected status 402, got 200 5 tests, 4 passed, 1 failed in 0.091s ``` Run IDs and timings naturally vary. The report is assembled in stable source order when execution finishes; it is not a live-progress display. Choose the amount of terminal detail without changing execution semantics: | Mode | Output | | --- | --- | | default | Run header, compact per-file table, collection/failure details, final summary. | | `-q`, `--quiet` | No header or file table; collection/failure details and final summary remain. | | `-v` | One stable result row per test, including duration. | | `-vv` | Per-test rows plus worker, attempt, and phase metadata, including captured output. | Skipped and expected-failure reasons are hidden unless they are requested explicitly: ```console $ testenix run --show-skips tests ``` List the slowest tests with `--durations N`; use `0` to list every test. The duration section is printed immediately before the final summary: ```console $ testenix run --durations 10 tests $ testenix run --durations 0 tests ``` ANSI styling defaults to `--color auto`, which considers the output terminal plus `NO_COLOR`, `FORCE_COLOR`, `CI`, and `TERM=dumb`. Force it with `--color always`, or produce plain output with either `--color never` or `--no-color`: ```console $ testenix run --no-color tests ``` These flags affect the native `testenix run` console only. The transparent `testenix pytest` bridge preserves pytest's renderer and argument meanings; a concise bridge command is `testenix pytest -q --tb=short tests`. ## JSON ```console $ testenix run --json reports/testenix.json ``` The JSON report preserves the complete hierarchy: ```text run └── test └── attempt ├── setup ├── call └── teardown ``` Consumers can distinguish failed assertions, setup errors, teardown errors, timeouts, crashes, expected failures, unexpected passes, flaky retries, and framework infrastructure errors. ## JUnit XML ```console $ testenix run --junit reports/junit.xml ``` JUnit XML is designed for CI systems that already understand the common test-report format. Testenix-specific identifiers and statuses are retained as properties where JUnit has no exact equivalent. ## Duration history By default, duration history is stored in `.testenix/history.sqlite3`. Later runs use the estimates to schedule longer work earlier. Choose a custom location: ```console $ testenix run --history .cache/testenix.sqlite3 ``` Disable all history writes: ```console $ testenix run --no-history ``` History changes scheduling estimates, not test semantics. A test is never skipped or treated as passing because of a historical record. ## Configure report paths ```toml [tool.testenix] json = "reports/testenix.json" junit = "reports/junit.xml" history = ".testenix/history.sqlite3" ``` CLI options override these paths for the current run. ## Exit codes | Code | Meaning | | ---: | --- | | `0` | No gating failure. | | `1` | Test failure, error, timeout, crash, unexpected pass, or flaky result. | | `2` | Collection, command, configuration, or empty explicit selection error. | | `3` | Internal runner, report, or history error. | | `130` | Interrupted by the user. | --- # Document: CLI reference Canonical URL: https://polishdataengineer.github.io/testenix/reference/cli/ Source: docs/reference/cli.md # Command-line reference ## Top-level command ```text testenix [-h] [--version] testenix [--config PYPROJECT] run [RUN_ARGS ...] testenix pytest [PYTEST_ARGS ...] testenix migrate FRAMEWORK PATH [PATH ...] [MIGRATION_ARGS ...] testenix [--config PYPROJECT] tune [PATH ...] [TUNING_ARGS ...] testenix [--config PYPROJECT] benchmark [PATH ...] [TUNING_ARGS ...] testenix manifest PATH [PATH ...] --output FILE ``` | Option | Description | | --- | --- | | `-h`, `--help` | Show command help. | | `--version` | Print the installed Testenix version. | | `--config PATH` | Load `[tool.testenix]` for native `run` and `tune`/`benchmark` commands. | ## `testenix run` ```text testenix run [PATH ...] [--config PYPROJECT] [-w auto|N] [--retries N] [--timeout SECONDS] [-t TAG ...] [--json FILE] [--junit FILE] [--history FILE | --no-history] [--shard-modules] [--manifest FILE] [-q | -v | -vv] [--color {auto,always,never} | --no-color] [--show-skips] [--durations N] ``` | Argument | Default | Description | | --- | --- | --- | | `PATH ...` | configured `paths`, otherwise `tests` | Files or directories to discover. | | `-w`, `--workers` | `auto` | Positive worker count, or adaptive selection from schedulable units, history, startup cost, and CPU capacity. | | `--retries` | `0` | Additional attempts after a gating outcome. | | `--timeout` | none | Global hard deadline for every selected test. | | `-t`, `--tag` | none | Required tag; repeat for AND selection. | | `--json` | none | Write the lossless JSON run result. | | `--junit` | none | Write a JUnit XML report. | | `--history` | `.testenix/history.sqlite3` | Override the duration-history database. | | `--no-history` | off | Disable reading and writing history. | | `--shard-modules` | off | Opt eligible modules into per-test execution units after conservative static safety checks. | | `--manifest FILE` | none | Reuse an explicitly generated, source-verified trusted collection manifest. | | `-q`, `--quiet` | off | Hide the run header and compact per-file table. Collection errors, failure details, and the final summary remain visible. | | `-v`, `--verbose` | off | Print one result row per test in the stable detailed format. Repeat for `-vv`. | | `-vv` | off | Add worker, attempt, and phase metadata, including captured output. | | `--color auto\|always\|never` | `auto` | Select automatic ANSI styling, force styling, or disable it. | | `--no-color` | off | Alias for `--color never`, useful in logs and snapshots. | | `--show-skips` | off | Include reasons for skipped and expected-failure tests. | | `--durations N` | none | List the `N` slowest tests before the summary; `0` lists every test. | CLI options override `[tool.testenix]` values for the current run. With no presentation flags, Testenix prints a compact row for each source file, complete collection/failure diagnostics, and a final summary. Console output is assembled in stable source order after execution completes; these modes do not promise live progress updates. Presentation flags change only terminal rendering, not selection, scheduling, result statuses, JSON, JUnit, or exit codes. `workers = auto` never means “start one process per logical CPU.” Testenix caps it by the number of units that can run independently. Reliable duration history feeds a makespan/startup-cost model; cold runs use a conservative cap. The final console and JSON results report the worker count actually used. An explicit integer remains useful for fixed CI resource limits and reproducible published benchmarks. `--shard-modules` is an explicit safety/performance trade-off. Modules with module/session fixtures, statically visible mutable global state, or import-time lifecycle hazards keep module affinity; eligible modules may be split. Static analysis cannot prove every dynamic side effect. `--manifest` accepts the versioned JSON produced by `testenix manifest`. Malformed input is a usage error. An otherwise valid manifest whose roots, complete Python-file inventory, or SHA-256 digests no longer match is treated as stale, and the run safely performs ordinary supervised collection. In `auto` color mode, Testenix requires a terminal, respects `NO_COLOR`, allows `FORCE_COLOR`, and disables styling for a truthy `CI` value or `TERM=dumb`. Explicit `always` or `never` takes precedence. ## `testenix pytest` ```text testenix pytest [PYTEST_ARGS ...] ``` This compatibility command hands the current CLI process to pytest from the same interpreter and forwards every argument without translation. It preserves pytest configuration, collection, fixtures, parametrization, markers, classes, hooks, plugins, output, and normal exit status. The compact native Testenix renderer is not involved: output is pytest's own output, just as with `uv run pytest tests`. For a concise bridge invocation, use `testenix pytest -q --tb=short tests`. ```console $ testenix pytest -q --tb=short tests $ testenix pytest tests/test_api.py::test_health -k smoke --maxfail=1 $ testenix pytest -n auto tests $ testenix pytest --junitxml=reports/pytest.xml tests ``` Pytest and its plugins must be installed in the same Python environment as Testenix. The optional `testenix[pytest]` extra installs a supported pytest (`>=8.3,<10`) when needed. Testenix does not consume a leading `--`: pytest receives it unchanged, including its normal end-of-options meaning. `[tool.testenix]` does not configure this command. Arguments after `pytest` are never interpreted as native Testenix options, even when their names overlap: `-q`, `-v`, `--color`, `--show-skips`, `--durations`, `--workers`, and every other token are forwarded unchanged. They can therefore be handled by pytest or a plugin, or rejected by pytest if unsupported. Pytest's equivalent spellings include `-rs`, `--durations=N`, and `--color=yes|no|auto`; Testenix does not translate native spellings. In particular, `testenix --config PATH pytest ...` is rejected; `pytest` must immediately follow `testenix`. See [pytest compatibility](https://polishdataengineer.github.io/testenix/guides/pytest-compatibility/) for the full boundary. ## `testenix migrate` ```text testenix migrate {auto,pytest,unittest} PATH [PATH ...] [-o OUTPUT] [-w auto|N] [--validation-timeout SECONDS] [--dry-run | --check] [--report-json FILE|-] ``` | Argument | Default | Description | | --- | --- | --- | | `FRAMEWORK` | required | `pytest`, `unittest`, or `auto` for separate modules of both kinds. | | `PATH ...` | required | Source files or directories. Sources are never modified. | | `-o`, `--output` | `testenix_migrated` | New directory to publish; it must not exist. | | `-w`, `--workers` | `auto` | Worker count for parallel candidate validation; an integer must be at least `2`. | | `--validation-timeout` | `300` | Independent deadline for each source/native subprocess. | | `--dry-run` | off | Static analysis and source-hash check only; run and publish nothing. | | `--check` | off | Full differential validation, with no published output. | | `--report-json` | none | New audit path inside the project and outside source/output suites, or `-` for clean JSON on standard output. | Without `--dry-run` or `--check`, migration requires a green source baseline, an equal native serial result, and an equal native parallel result, including every mapped test outcome. It then rechecks source hashes and atomically renames a complete staging directory to the new output without replacement. A failure before that rename leaves the output absent. A report-only failure after publication warns but leaves the validated output and successful exit status intact. See [safe migration](https://polishdataengineer.github.io/testenix/guides/migration/) for supported constructs, unittest's SHA-pinned wrapper model, rollback guarantees, and external-side-effect boundaries. Human-readable output distinguishes an analyzed candidate, a validated candidate, a generated candidate, a statically convertible subset, and a published conversion. Repeated diagnostics are grouped by severity and code, with the first source location shown. Use `--report-json FILE` or `--report-json -` when every individual line-addressed diagnostic is required. The JSON field `converted_tests` counts source-to-target mappings built in memory; only `status: published` and `published: true` mean that an output directory was created. `MIG006` warns that a candidate has only one schedulable module affinity unit despite a parallel worker setting. It is emitted only for `--check` or publication after static analysis succeeds, because dry-run and unsupported transactions never execute the parallel gate. ## `testenix tune` / `testenix benchmark` ```text testenix tune [PATH ...] [--config PYPROJECT] [--candidates N[,N...]] [--warmups N] [--repeats N] [--pytest-source PATH] ... [--json FILE|-] [--shard-modules] [--manifest FILE] [--write] ``` `testenix benchmark` is an exact alias for `testenix tune`. | Argument | Default | Description | | --- | --- | --- | | `PATH ...` | configured `paths`, otherwise `tests` | Native Testenix suite to tune. | | `--candidates N[,N...]` | resource-aware 1/2/4 sweep | Positive worker counts to measure. Counts above the number of execution units are deduplicated at that limit; values above four require this explicit option. | | `--warmups N` | `1` | Unrecorded warmups for each native candidate and optional pytest source. | | `--repeats N` | `5` | Recorded samples for each candidate. Candidate order alternates between rounds. | | `--run-timeout SECONDS` | `300` | Deadline for each complete native or pytest suite run; cleanup uses a Windows Job Object or POSIX root-session plus identity-tracked descendants. | | `--pytest-source PATH` | none | Also time a corresponding pytest source path; repeat for multiple paths. | | `--json FILE\|-` | none | Write the complete tuning report to a new file, or standard output with `-`. | | `--shard-modules` / `--no-shard-modules` | configured value | Tune with explicit safe intra-module sharding or module affinity. | | `--manifest FILE` | configured value | Use one source-verified collection manifest for every native sample. | | `--write` | off | Persist the measured recommendation as `[tool.testenix].workers`. | The command measures fresh CLI processes with Testenix history disabled. A one-worker probe establishes the inventory and outcomes, and every native candidate must match them. To avoid persisting noise, it recommends the smallest worker count within a narrow tolerance of the best median. `--write` is the only configuration-mutating mode; normal tuning and adaptive auto do not edit configuration. A workers-only write is rejected when a transient sharding or manifest override differs from the loaded project configuration, or when that configuration file changes during the measurement, because the recommendation would not describe the persisted execution profile. The writer compares the pre-tuning bytes again immediately before an atomic replacement. Project Python/TOML sources, linked source directories, explicit suite files, and a trusted manifest are fingerprinted by content and file identity after every sample; observed drift discards the complete result. Run publishable tuning from an immutable checkout to exclude a writer racing the final filesystem operation; installed packages, non-source data, and other runtime inputs are also outside that fingerprint. The optional pytest row is useful for local orientation only when it represents the corresponding source suite. A public pytest/Testenix claim still needs equivalent inventories and outcomes, counterbalanced commands, environment/version provenance, and the complete [benchmarking contract](https://polishdataengineer.github.io/testenix/benchmarking/). ## `testenix manifest` ```text testenix manifest PATH [PATH ...] --output FILE ``` The command performs supervised native collection and creates a new deterministic trusted manifest. It records collection roots, the complete selected Python-source inventory and SHA-256 fingerprints, collected tests and issues, and conservative module-sharding decisions. Test case parameter names are retained while their values are redacted, so environment-derived secrets are not copied into the trust artifact. `FILE` must not already exist; Testenix does not silently replace a previous trust artifact. Use it explicitly on later runs: ```console $ testenix manifest tests --output .testenix/collection.json $ testenix run tests --manifest .testenix/collection.json ``` This can avoid importing every selected module once for collection and once again for execution. Execution workers still import the modules they run. Source verification cannot cover dynamic collection inputs such as environment variables or external services; regenerate the manifest when those inputs change. ## Examples ```console $ testenix run $ testenix run tests/unit tests/integration --workers 4 $ testenix run --tag unit --tag fast $ testenix run --retries 1 --timeout 10 $ testenix run -v --show-skips --durations 10 $ testenix run --color never $ testenix run --json reports/run.json --junit reports/junit.xml $ testenix run tests --shard-modules $ testenix manifest tests --output .testenix/collection.json $ testenix run tests --manifest .testenix/collection.json $ testenix tune tests --candidates 1,2,4,8 --warmups 1 --repeats 5 $ testenix benchmark tests --json reports/tuning.json $ testenix tune --write $ testenix --config config/pyproject.toml run $ testenix pytest -q --tb=short tests $ testenix migrate pytest tests --dry-run $ testenix migrate auto tests --check --report-json reports/migration.json $ testenix migrate unittest tests --output tests_testenix ``` ## Exit codes The native `testenix run` command uses these codes: | Code | Meaning | | ---: | --- | | `0` | The run has no gating result. | | `1` | A test failed, errored, timed out, crashed, became flaky, or unexpectedly passed. | | `2` | Invalid CLI/configuration, collection error, or empty explicit tag selection. | | `3` | Internal runner, reporter, or history failure. | | `130` | User interruption. | The `testenix migrate` command uses these codes: | Code | Meaning | | ---: | --- | | `0` | Analysis, check, or publication succeeded; also retained after a report-only failure following publication. | | `1` | Source/candidate execution failed, timed out, or differed. | | `2` | Unsafe path, existing output, source drift, or invalid migration usage. | | `3` | Internal process or report failure before publication. | | `4` | An unsupported construct was found; nothing was published. | | `130` | User interruption. | `testenix pytest` hands the current CLI process to pytest, so pytest or plugin exit statuses are returned unchanged. Standard pytest statuses are `0` success, `1` test failure, `2` interruption, `3` internal error, `4` usage error, and `5` no tests collected. Before the handoff, Testenix returns `2` when pytest is missing and `3` when pytest cannot take over the process. --- # Document: Configuration reference Canonical URL: https://polishdataengineer.github.io/testenix/reference/configuration/ Source: docs/reference/configuration.md # Configuration reference Project defaults live in `[tool.testenix]` in `pyproject.toml`. This table configures only the native `testenix run` engine. The `testenix pytest` compatibility command delegates configuration to pytest and therefore uses `pytest.ini`, `pyproject.toml` `[tool.pytest.ini_options]`, or arguments passed after the subcommand. ```toml [tool.testenix] paths = ["tests"] workers = "auto" retries = 0 # timeout = 10.0 tags = [] # shard_modules = true # manifest = ".testenix/collection.json" # json = "reports/testenix.json" # junit = "reports/junit.xml" history = ".testenix/history.sqlite3" ``` Unknown options are rejected instead of being silently ignored. ## Options ### `paths` - Type: string or list of strings - Default: `["tests"]` Files and directories used when no positional path is passed to `testenix run`. ### `workers` - Type: positive integer or `"auto"` - Default: `"auto"` `auto` is adaptive rather than equal to Python's logical CPU count. After collection and selection, Testenix caps concurrency by the available CPUs and the number of independently schedulable units. Reliable duration history feeds a worker-startup/makespan estimate; without enough history, a conservative cold-start cap avoids oversubscribing short suites. The smallest worker count within a narrow tolerance of the predicted best is selected. An explicit number is recommended for strict CI resource limits and publishable benchmark runs. Use `testenix tune` (or its `testenix benchmark` alias) to measure native candidates for this project, and `testenix tune --write` to persist its recommendation explicitly. ### `retries` - Type: non-negative integer - Default: `0` Number of user-requested attempts after a gating outcome. Earlier failures remain part of the result, so a later pass is finalized as `FLAKY`. ### `timeout` - Type: positive finite number in seconds - Default: none Global hard timeout for selected tests. Timed tests run in isolated processes. ### `tags` - Type: string or list of strings - Default: empty Every configured tag must be present on a test for it to be selected. A comma-separated string and a TOML list are both accepted. ### `json` - Type: filesystem path or `null` - Default: none Location of the lossless JSON report. The programmatic field is `json_path`. ### `junit` - Type: filesystem path or `null` - Default: none Location of the JUnit XML report. The programmatic field is `junit_path`. ### `history` - Type: filesystem path, `false`, or `null` - Default: `.testenix/history.sqlite3` Duration-history database. Set it to `false` to disable history in TOML: ```toml [tool.testenix] history = false ``` The programmatic field is `history_path` and accepts a `pathlib.Path` or `None`. ### `shard_modules` - Type: boolean - Default: `false` Allow eligible modules to be divided into finer per-test execution units. The static analyzer keeps module affinity when it sees module/session fixtures, mutation of obvious module-global state, or import-time lifecycle hazards. Function-scoped fixtures, including autouse fixtures, do not block sharding. This option is explicitly opt-in because static analysis cannot prove the absence of all dynamic side effects. Validate the project with and without sharding before enabling it in CI. ### `manifest` - Type: filesystem path or `null` - Default: none Path to a trusted collection manifest generated explicitly with: ```console $ testenix manifest tests --output .testenix/collection.json ``` On each run Testenix verifies the requested collection roots, selected test files, statically discoverable project-local Python import dependencies, and SHA-256 source digests. An exact match bypasses the collection-side imports; a stale but well-formed manifest falls back to normal supervised collection. Malformed JSON is rejected. Execution workers still import the modules they run. Test parameter names remain available for diagnostics, but their values are stored only as `` to avoid persisting secrets obtained during collection. Regenerate the manifest when source files or dynamic collection inputs change. The programmatic field is `manifest_path` and accepts a `pathlib.Path` or `None`. ## Programmatic configuration ```python from pathlib import Path from testenix import TestenixConfig config = TestenixConfig( paths=("tests/unit",), workers=4, retries=1, timeout=5.0, tags=("unit",), shard_modules=True, manifest_path=Path(".testenix/collection.json"), json_path=Path("reports/run.json"), history_path=None, ) ``` `TestenixConfig` is immutable. Use `with_overrides` to derive a validated copy. --- # Document: Python API reference Canonical URL: https://polishdataengineer.github.io/testenix/reference/api/ Source: docs/reference/api.md # Python API reference The objects below are exported from `testenix` and form the public pre-1.0 API. Pre-1.0 releases may still make documented breaking changes. ## Authoring ```{eval-rst} .. autofunction:: testenix.test .. autofunction:: testenix.fixture .. autofunction:: testenix.case .. autofunction:: testenix.cases ``` ### `skip` ```python @skip(reason_or_function=None, /, *, reason=None, when=True) ``` Mark a test as skipped. It can be used as `@skip`, `@skip("reason")`, or with a conditional `when=...`. ### `xfail` ```python @xfail(reason_or_function=None, /, *, reason=None, when=True) ``` Mark a test as expected to fail. An unexpected pass becomes the gating `XPASS` status. ```{eval-rst} .. autoclass:: testenix.CaseDefinition :members: ``` ## Discovery and execution ```{eval-rst} .. autofunction:: testenix.discover .. autofunction:: testenix.run .. autofunction:: testenix.run_async .. autoclass:: testenix.TestenixConfig :members: ``` ## Result contracts ```{eval-rst} .. autoclass:: testenix.RunResult :members: .. autoclass:: testenix.TestResult :members: .. autoclass:: testenix.TestSpec :members: .. autoclass:: testenix.CollectionResult :members: .. autoclass:: testenix.Status :members: .. autoclass:: testenix.Scope :members: ``` ## Migration The programmatic API follows the same copy-only transaction as `testenix migrate`. `project_root` defaults to the current directory. Prefer `dry_run=True` before a validating or publishing call. ```python from pathlib import Path from testenix import MigrationOptions, migrate report = migrate( MigrationOptions( framework="auto", sources=(Path("tests"),), output=Path("tests_testenix"), check_only=True, ) ) assert report.originals_modified is False ``` ```{eval-rst} .. autofunction:: testenix.migrate .. autoclass:: testenix.MigrationOptions :members: .. autoclass:: testenix.MigrationReport :members: .. autoclass:: testenix.MigrationStatus :members: .. autoclass:: testenix.ValidationSummary :members: ``` ## Events ```{eval-rst} .. autoclass:: testenix.Event :members: .. autoclass:: testenix.EventSink :members: ``` --- # Document: Benchmark results Canonical URL: https://polishdataengineer.github.io/testenix/benchmarks/results/ Source: docs/benchmarks/results.md # Published benchmark results These tables are generated from the raw JSON committed in `benchmarks/`. They are development evidence for specific synthetic workloads, not a universal claim that Testenix is always faster than pytest. `Testenix` in these results means the native `testenix run` engine. The `testenix pytest` compatibility bridge delegates to pytest and is not represented here. ## Testenix 0.3.0 scaling matrix No current-version matrix is checked in yet. The historical results below must therefore not be described as Testenix 0.3.0 performance. The new provenance-gated harness covers 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/auto workers, and both default history and `--no-history`, plus explicit safe-module sharding. Its default design uses dimension sweeps; use `--full-cross-product` only when the much larger run is intentional. `auto` is passed literally to Testenix and remains adaptive; observed Testenix worker counts are stored per sample. pytest-xdist resolves its side of an `auto` row separately to the machine's logical CPU count. ```console $ uv run --no-editable python benchmarks/run_scaling_matrix.py \ --output benchmarks/scaling_matrix_0_3_0.json ``` The command refuses a dirty worktree or an installed Testenix version that differs from `pyproject.toml`. `--allow-dirty` is available only for unpublished smoke runs. A matrix becomes publishable here only after five measured rounds, one warm-up, clean commit provenance, and full axis coverage pass the documentation generator's validation. ## Real-project harness The 118-test project used during v0.2 migration validation was a semantic parity gate, not a publishable benchmark: its release-note timings were single observations without a committed multi-round record. Use the redaction-safe manifest harness for a real repository: ```console $ cp benchmarks/real_project_manifest.example.json /tmp/testenix-project-benchmark.json $ uv run --no-editable python benchmarks/run_project_benchmark.py \ --project /absolute/path/to/project \ --manifest /tmp/testenix-project-benchmark.json \ --output /tmp/testenix-project-result.json ``` The manifest stores argument arrays, never shell fragments. The result omits stdout, stderr, environment values, absolute project paths, and private source. It records only timings, aggregate output sizes, optional tree fingerprints, and redacted Git provenance. A migrated-suite comparison must point the manifest at a successful migration report to become publication-eligible. The harness verifies the report's exact per-test inventory and outcomes, complete source and generated Python-file inventories, current hashes, and binds canonical `python -m pytest` / `python -m testenix run` commands to the report's source/output roots. Publishable source roots are directories so support files such as `conftest.py` are covered. Without the report the result is diagnostic-only. Commands are retained for reproducibility. Publishable commands put options before `--` and exact suite targets after it, so an option value cannot impersonate a migration root. Keep secrets in the environment or list sensitive argument indexes in `redact_arguments`. ## Historical Testenix 0.1.0 synthetic baseline The checked-in `3.15×` figure is a Testenix 0.1.0 result for 100,000 generated no-op tests across 16 modules, four workers, disabled history (`--no-history`), and pytest-xdist's default `load` strategy. It is retained as transparent historical evidence; it is not a measurement of Testenix 0.3.0. ![Historical Testenix 0.1.0 throughput ratios](https://polishdataengineer.github.io/testenix/_static/benchmark-speedup.svg) ### Median wall-clock time Lower time is better. A speedup of `2.85×` means pytest's median wall time was 2.85 times the Testenix median for that exact scenario. | Scenario | Testenix | pytest | pytest-xdist | vs pytest | vs xdist | | --- | ---: | ---: | ---: | ---: | ---: | | 10,000 no-op tests / 16 modules | 0.869 s | 2.477 s | 2.106 s | 2.85× | 2.42× | | 10,000 uneven-duration tests / 16 modules | 1.345 s | 3.076 s | 2.138 s | 2.29× | 1.59× | | 100,000 no-op tests / 16 modules | 8.038 s | 25.333 s | 21.300 s | 3.15× | 2.65× |
The 100,000-test result meets the project's local five-run, one-warmup minimum. It remains a synthetic result from one machine, not a universal performance promise.
### Environment and controls - CPU: Apple M4 Pro (14 logical CPUs) - Machine: `arm64` - Platform: `macOS-26.5.1-arm64-arm-64bit` - Python: `3.11.14` - Testenix: `0.1.0` - Workers: four for Testenix and pytest-xdist - Testenix history: disabled with `--no-history` - pytest-xdist: version `3.8.0`, default `load` distribution - Measurement: complete subprocess wall-clock time, including discovery, execution, aggregation, and console rendering - Correctness gate: every command had to exit successfully and report the expected test count ### Raw samples and variance ### 10,000 no-op tests / 16 modules - Testenix range: 0.857 s–0.892 s - Testenix standard deviation: 0.013 s - Testenix raw samples: 0.869, 0.857, 0.863, 0.892, 0.871 seconds - pytest range: 2.447 s–2.522 s - pytest standard deviation: 0.027 s - pytest raw samples: 2.522, 2.477, 2.471, 2.447, 2.479 seconds - pytest-xdist range: 2.075 s–2.267 s - pytest-xdist standard deviation: 0.081 s - pytest-xdist raw samples: 2.170, 2.267, 2.077, 2.075, 2.106 seconds - Measured rounds: 5; warmups: 1 - Workers: 4 - Testenix history: disabled with `--no-history` - pytest-xdist strategy: default `load` - Recorded at: `2026-07-20T12:12:43.635798+00:00` - Commit: `8f24f8a7bd72fa876988a8ce96364be97e35c2b6` - Clean working tree at capture: yes - [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/baseline.json) ### 10,000 uneven-duration tests / 16 modules - Testenix range: 1.336 s–1.378 s - Testenix standard deviation: 0.016 s - Testenix raw samples: 1.336, 1.378, 1.342, 1.345, 1.356 seconds - pytest range: 3.043 s–3.085 s - pytest standard deviation: 0.016 s - pytest raw samples: 3.070, 3.085, 3.043, 3.076, 3.077 seconds - pytest-xdist range: 2.109 s–2.176 s - pytest-xdist standard deviation: 0.025 s - pytest-xdist raw samples: 2.146, 2.129, 2.109, 2.138, 2.176 seconds - Measured rounds: 5; warmups: 1 - Workers: 4 - Testenix history: disabled with `--no-history` - pytest-xdist strategy: default `load` - Recorded at: `2026-07-20T12:13:53.369799+00:00` - Commit: `18d9bba6cb5c8e39c2d5b211ee4384ae8f824524` - Clean working tree at capture: yes - [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/baseline_uneven.json) ### 100,000 no-op tests / 16 modules - Testenix range: 7.912 s–8.096 s - Testenix standard deviation: 0.075 s - Testenix raw samples: 7.912, 8.096, 8.086, 8.038, 7.997 seconds - pytest range: 25.188 s–27.005 s - pytest standard deviation: 0.772 s - pytest raw samples: 27.005, 25.333, 25.246, 25.188, 25.380 seconds - pytest-xdist range: 21.120 s–22.216 s - pytest-xdist standard deviation: 0.486 s - pytest-xdist raw samples: 21.239, 21.120, 22.216, 21.300, 21.949 seconds - Measured rounds: 5; warmups: 1 - Workers: 4 - Testenix history: disabled with `--no-history` - pytest-xdist strategy: default `load` - Recorded at: `2026-07-20T12:19:39.942492+00:00` - Commit: `24b877c2f98420e91dcd2c8bcbc9417c7cf1ac96` - Clean working tree at capture: yes - [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/baseline_100k.json) ## Migrated-suite measurements These separate measurements start with generated pytest or unittest sources, complete one safe copy-and-validate migration, and then compare recurring source-suite runs with recurring native Testenix runs. The migration transaction is a one-time cost shown separately; it is not included in either execution median. These records came from the pre-v0.2 source commit linked below; its distribution metadata still reported `0.1.0`. They are historical evidence, not measurements of the current release. | Source runner | Workload | Tests / modules | Source median | Native median | Native vs source | Migration transaction | | --- | --- | ---: | ---: | ---: | ---: | ---: | | pytest (sequential) | no-op | 3,000 / 64 | 1.539 s | 0.521 s | 2.96× faster | 5.940 s | | unittest outcome probe (sequential) | no-op | 3,000 / 64 | 0.161 s | 1.192 s | 7.40× slower | 6.742 s | | unittest outcome probe (sequential) | 1 ms body | 3,000 / 64 | 4.066 s | 2.577 s | 1.58× faster | 17.251 s | The native side used four workers. The source pytest and unittest outcome-probe baselines were sequential, so these rows do not compare Testenix with pytest-xdist or another parallel unittest runner. The unittest probe uses the standard-library loader and result semantics, then serializes per-test outcomes for parity checking; its timing therefore includes that small audit overhead. The no-op unittest wrappers are 7.40× slower than the probe because wrapper, loading, and result-adaptation costs dominate an empty body. With 1 ms of synthetic work per unittest method, parallel native execution is 1.58× faster in this 64-module layout. Module count and duration are therefore material, and none of these synthetic rows predicts a specific real project. ### Raw migration samples and variance ### pytest / no-op - Source command: `python -m pytest -q -p no:cacheprovider tests` - Native command: `python -m testenix run testenix_migrated --workers 4 --no-history` - Source median: 1.539 s - Source range: 1.519 s–1.573 s; standard deviation: 0.020 s - Source raw samples: 1.547, 1.535, 1.539, 1.573, 1.519 seconds - Native Testenix median: 0.521 s - Native Testenix range: 0.496 s–0.570 s; standard deviation: 0.030 s - Native Testenix raw samples: 0.532, 0.498, 0.570, 0.496, 0.521 seconds - Native workers: 4 - Measured rounds: 5; warmups: 1 - One-time copy, validation, and publication transaction: 5.940 s - Integrity gates: 3,000 converted tests, matching source/native outcomes, original SHA-256 values unchanged - Recorded at: `2026-07-20T16:38:49.510465+00:00` - Source commit: [`3a51a901d268b061e9a87168300b41f3a2714a84`](https://github.com/polishdataengineer/testenix/commit/3a51a901d268b061e9a87168300b41f3a2714a84); worktree clean - Lock SHA-256: `8ef0a9258aa5196bf2891f9da9f66c29bcf4e9bf297d178f3d4939cad36130cf` - Versions: pytest=9.1.1, python=3.11.14, testenix=0.1.0, unittest=stdlib-3.11.14 - Environment: cpu_count=14, cpu_model=Apple M4 Pro, machine=arm64, platform=macOS-26.5.1-arm64-arm-64bit, python_implementation=CPython, python_version=3.11.14 - [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_pytest_3000.json) ### unittest / no-op - Source command: `python -m testenix._unittest_probe --output /.benchmark-unittest.json tests` - Native command: `python -m testenix run testenix_migrated --workers 4 --no-history` - Source median: 0.161 s - Source range: 0.154 s–0.166 s; standard deviation: 0.004 s - Source raw samples: 0.161, 0.160, 0.161, 0.154, 0.166 seconds - Native Testenix median: 1.192 s - Native Testenix range: 1.151 s–1.264 s; standard deviation: 0.051 s - Native Testenix raw samples: 1.192, 1.171, 1.256, 1.151, 1.264 seconds - Native workers: 4 - Measured rounds: 5; warmups: 1 - One-time copy, validation, and publication transaction: 6.742 s - Integrity gates: 3,000 converted tests, matching source/native outcomes, original SHA-256 values unchanged - Recorded at: `2026-07-20T16:39:05.030979+00:00` - Source commit: [`3a51a901d268b061e9a87168300b41f3a2714a84`](https://github.com/polishdataengineer/testenix/commit/3a51a901d268b061e9a87168300b41f3a2714a84); worktree clean - Lock SHA-256: `8ef0a9258aa5196bf2891f9da9f66c29bcf4e9bf297d178f3d4939cad36130cf` - Versions: pytest=9.1.1, python=3.11.14, testenix=0.1.0, unittest=stdlib-3.11.14 - Environment: cpu_count=14, cpu_model=Apple M4 Pro, machine=arm64, platform=macOS-26.5.1-arm64-arm-64bit, python_implementation=CPython, python_version=3.11.14 - [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_unittest_3000.json) ### unittest / 1 ms body - Source command: `python -m testenix._unittest_probe --output /.benchmark-unittest.json tests` - Native command: `python -m testenix run testenix_migrated --workers 4 --no-history` - Source median: 4.066 s - Source range: 4.023 s–4.075 s; standard deviation: 0.021 s - Source raw samples: 4.075, 4.063, 4.066, 4.023, 4.067 seconds - Native Testenix median: 2.577 s - Native Testenix range: 2.454 s–2.644 s; standard deviation: 0.071 s - Native Testenix raw samples: 2.577, 2.601, 2.571, 2.454, 2.644 seconds - Native workers: 4 - Measured rounds: 5; warmups: 1 - One-time copy, validation, and publication transaction: 17.251 s - Integrity gates: 3,000 converted tests, matching source/native outcomes, original SHA-256 values unchanged - Recorded at: `2026-07-20T16:40:02.708652+00:00` - Source commit: [`3a51a901d268b061e9a87168300b41f3a2714a84`](https://github.com/polishdataengineer/testenix/commit/3a51a901d268b061e9a87168300b41f3a2714a84); worktree clean - Lock SHA-256: `8ef0a9258aa5196bf2891f9da9f66c29bcf4e9bf297d178f3d4939cad36130cf` - Versions: pytest=9.1.1, python=3.11.14, testenix=0.1.0, unittest=stdlib-3.11.14 - Environment: cpu_count=14, cpu_model=Apple M4 Pro, machine=arm64, platform=macOS-26.5.1-arm64-arm-64bit, python_implementation=CPython, python_version=3.11.14 - [Raw JSON](https://github.com/polishdataengineer/testenix/blob/main/benchmarks/migration_baseline_unittest_3000_delay_1ms.json) ## Interpretation The historical checked-in results show that Testenix 0.1.0 had low per-test overhead for the large generated suites above and was competitive with sequential pytest and pytest-xdist's default `load` strategy in those scenarios. They are not evidence for the current release. They do **not** yet answer how Testenix performs for import-heavy applications, complex fixture graphs, assertion failures, real repositories, or different operating systems. Pytest also has a far larger plugin and tooling ecosystem. Read the [full performance analysis](https://polishdataengineer.github.io/testenix/performance-analysis/) for profiling details, memory notes, implemented optimizations, and the Rust/PyO3 decision. ## Reproduce Run the same harness from a locked development environment: ```console $ uv sync --locked --dev --no-editable $ uv run python benchmarks/run_benchmark.py --tests 10000 --workers 4 --repeats 5 $ uv run python benchmarks/run_benchmark.py --tests 10000 --workers 4 --repeats 5 --uneven $ uv run python benchmarks/run_benchmark.py --tests 100000 --workers 4 --repeats 5 $ uv run python benchmarks/run_migration_benchmark.py --framework pytest --tests 3000 \ --modules 64 --workers 4 --warmups 1 --repeats 5 \ --output benchmarks/migration_baseline_pytest_3000.json $ uv run python benchmarks/run_migration_benchmark.py --framework unittest --tests 3000 \ --modules 64 --workers 4 --warmups 1 --repeats 5 \ --output benchmarks/migration_baseline_unittest_3000.json $ uv run python benchmarks/run_migration_benchmark.py --framework unittest --tests 3000 \ --modules 64 --workers 4 --delay-ms 1 --warmups 1 --repeats 5 \ --output benchmarks/migration_baseline_unittest_3000_delay_1ms.json ``` Review the [benchmarking contract](https://polishdataengineer.github.io/testenix/benchmarking/) before comparing or publishing new data. --- # Document: Benchmarking contract Canonical URL: https://polishdataengineer.github.io/testenix/benchmarking/ Source: docs/benchmarking.md # Benchmarking contract Testenix must be compared with both sequential pytest and pytest-xdist. Comparing only with sequential pytest would overstate the value of built-in parallelism. The benchmark suite will contain: - 100, 1,000, and 10,000 no-op tests to measure discovery and framework overhead; - tests with deliberately uneven durations to measure scheduler tail latency; - module- and session-scoped fixture suites to measure fixture reuse; - synchronous and asynchronous tests; - passing, failing, skipped, xfailed, flaky, timed-out, and crashing tests; - a worker-crash scenario that verifies every selected test reaches a terminal state; - cold and warm history runs. For every scenario record wall-clock duration, collection time, execution time, peak memory, worker utilization, number of process starts, result completeness, and output size. Performance claims require at least five runs per configuration and must publish the environment fingerprint. The checked-in headline files were recorded with Testenix 0.1.0 across 16 generated modules, four workers, and `--no-history`. They automate subprocess wall-clock samples and the environment fingerprint, validate the completed test count, rotate runner order between measured rounds, and record throughput, mean, standard deviation, provenance, and raw samples. Pytest plugin autoloading and its cache provider are disabled, pytest-xdist 3.8 is loaded explicitly with its default `load` distribution, and every tool runs from the generated suite directory so repository-level pytest configuration does not affect the comparison. Those historical records do not measure Testenix 0.3.0. Schema-version 2 harness output additionally records the requested and resolved worker counts, balanced/dominant/single-module test distributions, default-history versus `--no-history`, the pytest-xdist distribution strategy, and captured stdout/stderr byte counts. Collection/execution splits, peak memory, utilization, process counts, and the remaining telemetry above are still the acceptance contract for a later harness iteration, not data claimed by the historical baseline. Correctness wins over speed: a run with a missing, duplicated, or incorrectly finalized result is invalid and excluded from performance comparisons. ## Evidence levels 1. **Historical synthetic baseline.** The committed `3.15×` figure is Testenix 0.1.0 on 100,000 generated no-op tests, 16 modules, four workers, disabled history, and one M4 Pro. It remains useful provenance but must not be labelled as current-version performance. 2. **Current-version scaling matrix.** `run_scaling_matrix.py` requires the installed Testenix version to match `pyproject.toml` and refuses a dirty worktree by default. The dimension-sweep design covers 100, 500, 1,000, and 3,000 tests; 1, 2, 4, and `auto` workers; balanced/dominant/single-module layouts; and default history versus `--no-history`. The reference configuration changes one axis at a time. `--full-cross-product` is available when every combination is worth the substantially larger runtime; its artifact still projects one canonical balanced/no-history reference curve. `auto` remains an adaptive Testenix request; the harness records the worker count observed in each Testenix sample. For the xdist side of that row, `auto` resolves separately to Python's logical CPU count and is labelled as such. 3. **Real-project evidence.** `run_project_benchmark.py` executes argument arrays from a local JSON manifest without a shell. Its result excludes source, stdout/stderr, environment values, absolute project paths, and Git remotes. Publication requires a successful migration report: the harness verifies its exact per-test inventory and outcomes, complete current Python-file inventories and hashes under directory source roots, the complete generated Python inventory, and that canonical `python -m pytest` / `python -m testenix run` commands point at the report's source and output roots. It records only aggregate timings/output sizes, digests, redacted Git state, and optional content fingerprints. Repeated or conflicting worker/history/sharding flags fail closed, and an imported runtime must belong to the exact files owned by its installed distribution rather than merely sharing the same `site-packages` directory. Every synthetic and real-project command has a bounded deadline and bounded cleanup. Windows starts the command suspended, attaches a kill-on-close Job Object, then resumes it. POSIX always signals the new root session and identity-checks every observed detached descendant. Polling cannot provide an absolute kernel guarantee for a child that calls `setsid()` and exits before the first snapshot; run hostile benchmark inputs inside a container or other kernel containment boundary. On POSIX the harness also discovers workers that created their own sessions; on Windows it uses a kill-on-close Job Object with a bounded recursive fallback. A timed-out runner therefore cannot continue consuming CPU or contaminate later counterbalanced samples. The 118-test project mentioned in the v0.2.0 release was a differential semantic-validation gate. Its three timings were single observations, not a committed multi-round benchmark, and therefore do not establish a real-project speedup. ## Project-local tuning is not a published benchmark `testenix tune` and its `testenix benchmark` alias answer a narrow operational question: which native worker count is fastest for this selected project suite on this machine now? They use fresh CLI processes, disable history, validate a green one-worker inventory, alternate candidate order, require native candidate inventory/outcome parity, and recommend the smallest worker count within a narrow tolerance of the best measured median. This is appropriate for writing a local `[tool.testenix].workers` value. The optional `--pytest-source` measurement is orientation for a corresponding source suite. A tuning report alone is not sufficient for marketing because a migrated native path and its pytest source may differ in collection, wrappers, output, or configuration. Publishing a ratio still requires all of this contract: equivalent inventories and outcomes, full-process wall time, counterbalanced order, at least one warm-up and five measured rounds, command/version/environment provenance, output sizes, and an explanation of history, sharding, manifest, and plugin settings. For every published Testenix result, record both the requested and observed worker count. `auto` is adaptive and must never be relabelled as the logical CPU count. Also record whether `--shard-modules` was enabled and whether a trusted collection manifest was accepted or fell back to supervised collection; either choice changes the amount and shape of schedulable work. ## Pytest compatibility bridge Measurements of `testenix pytest` must be reported separately from native `testenix run` measurements. The compatibility command hands the current interpreter to pytest through a POSIX process overlay or pytest's in-process console entry point on Windows; it does not execute tests through the Testenix engine. A compatibility-overhead comparison must use the same interpreter, working directory, environment, pytest configuration, plugins, and arguments for both `python -m pytest ...` and `testenix pytest ...`. Any difference measures adapter overhead only and must not be presented as a Testenix execution speedup. Native comparisons continue to use `testenix run` and must validate that both runners execute the same tests and produce equivalent outcomes. Run one reproducible synthetic scenario with: ```bash uv run --no-editable python benchmarks/run_benchmark.py \ --tests 1000 --modules 16 --workers 4 --repeats 5 --warmups 1 \ --module-layout balanced --history-mode disabled --xdist-strategy load ``` Generate the current-version dimension sweeps from a clean checkout with: ```bash uv run --no-editable python benchmarks/run_scaling_matrix.py \ --output benchmarks/scaling_matrix_0_3_0.json ``` For an unpublished smoke test only, add `--quick --allow-dirty`. A publishable matrix must retain five rounds, one warm-up, clean provenance, and the complete requested coverage. Measure a real project without committing its code or private paths: ```bash cp benchmarks/real_project_manifest.example.json /tmp/testenix-project-benchmark.json # Edit expected counts, migration-report path, fingerprints, and runner commands. uv run --no-editable python benchmarks/run_project_benchmark.py \ --project /absolute/path/to/project \ --manifest /tmp/testenix-project-benchmark.json \ --output /tmp/testenix-project-result.json ``` Keep private manifests and results outside the repository until their labels, commit policy, and fingerprints have been reviewed for publication. Environment override **values** are never written to the result; only their key names are retained. Commands are recorded for reproducibility, so secrets should stay in the environment. If an unavoidable command argument is sensitive, list its zero-based index in that runner's `redact_arguments` field. Without `migration_report`, the harness can still produce a private diagnostic, but it always sets `publication_eligible` to `false`. A publishable run also requires the canonical module entrypoints, a clean project, at least one warm-up and five measured rounds, explicit Testenix workers and history mode, and installed-distribution identities for both pytest and Testenix. The pytest version is probed inside the same project environment used by the timed commands. In each runner command, place options before `--` and the exact benchmark suite roots after it. The harness uses that delimiter to distinguish positional targets from values of options such as `-k` or `--tag` and requires those targets to match the migration report exactly. Maintainers can run the same comparison from GitHub's **Benchmarks** workflow and download its raw JSON artifact. Shared GitHub runners are appropriate for reproducibility checks, not for silently replacing the approved marketing baseline: their timing variance is outside this project's control. The checked-in baseline files are historical development evidence, not universal or current-version performance claims. See `docs/performance-analysis.md` for the large-suite results, optimization profile, memory notes, and native-code decision. A clean Testenix 0.3.0 scaling matrix, publishable real-project suites, alternative pytest-xdist strategies, and cross-platform repetitions remain required before publishing broad comparative claims. An approved public baseline must be committed through a reviewed pull request. Do not remove slow but valid samples as outliers; invalid commands remain evidence and must be explained. The current checked-in 10,000- and 100,000-test files each contain five measured rounds, one warm-up, and clean commit provenance. They remain Testenix 0.1.0 single-machine synthetic evidence, so broader claims still require the current-version, real-project, and cross-platform scenarios above. --- # Document: Performance analysis Canonical URL: https://polishdataengineer.github.io/testenix/performance-analysis/ Source: docs/performance-analysis.md # Performance analysis ## Executive summary The checked-in headline numbers are a **historical Testenix 0.1.0 synthetic baseline**, not current Testenix 0.3.0 results. In the largest recorded comparison, 100,000 generated no-op tests were spread evenly across 16 modules and run with four workers and `--no-history`. Testenix completed the suite in a median 8.038 seconds, pytest in 25.333 seconds, and pytest-xdist 3.8's default `load` scheduler in 21.300 seconds. Every command had to report the expected test count or the harness rejected the sample. The baseline contains one warm-up and five counterbalanced measured rounds, and Testenix's samples ranged from 7.912 to 8.096 seconds. This is evidence for the tested workload and machine, not a universal claim about every Python project. Import-heavy suites, fixture-heavy suites, slow tests, failure output, default history, alternative pytest-xdist schedulers, different operating systems, and real repositories still need independent measurements. No clean Testenix 0.3.0 scaling matrix is checked in yet, so `3.15×` must not be presented as a 0.3.0 speedup. These results do not apply to `testenix pytest`. The compatibility command delegates to pytest and has pytest execution performance plus launcher and adapter overhead, which has not yet been measured separately. The pre-v0.2 safe-migration baseline is deliberately mixed rather than uniformly positive. For 3,000 no-op pytest tests across 64 modules, the generated native suite was 2.96x faster than sequential pytest. Empty unittest wrappers were 7.40x slower than the sequential stdlib-based outcome probe, while the same layout with 1 ms of work in each method was 1.58x faster under four native workers. These are synthetic, sequential-source comparisons; they do not establish an advantage over pytest-xdist, parallel unittest runners, or a real repository. ## Environment and method - Apple M4 Pro, 14 logical CPUs, 24 GiB RAM; - macOS 26.5.1 arm64; - CPython 3.11.14; - four workers unless stated otherwise; - generated test modules accepted by both Testenix and pytest; - full process wall-clock time, including discovery, execution, aggregation, and console render; - deterministic rotated execution order instead of always running Testenix last; - test-count validation from each runner's final output; - disabled pytest plugin autoloading and cache, with pytest-xdist loaded explicitly; - pytest-xdist's default `load` distribution rather than `loadfile`, `loadscope`, or `worksteal`; - generated-suite working directory, isolated from repository-level pytest configuration; - `--no-history` for the primary runner-overhead comparison. The harness records every sample, median, mean, standard deviation, environment fingerprint, and median throughput. All checked-in comparison files use one warm-up and five measured repetitions. They also record the clean source commit, lockfile hash, timestamp, CPU model, and installed Testenix, pytest, and pytest-xdist versions. ## Historical Testenix 0.1.0 results | Scenario | pytest | pytest-xdist | Native Testenix | Native Testenix advantage | |---|---:|---:|---:|---:| | 10,000 no-op tests, 16 modules | 2.477 s | 2.106 s | 0.869 s | 2.85x vs pytest | | 10,000 uneven tests, 16 modules | 3.076 s | 2.138 s | 1.345 s | 2.29x vs pytest | | 100,000 no-op tests, 16 modules | 25.333 s | 21.300 s | 8.038 s | 3.15x vs pytest | The 100,000-test median throughputs were 12,440 tests/s for Testenix 0.1.0, 3,947 tests/s for pytest, and 4,695 tests/s for pytest-xdist. The raw five-sample ranges and standard deviations are published in the generated benchmark page and the checked-in JSON files. In a separate exploratory 100,000-test profile, the coordinator's measured maximum resident set was approximately 513 MiB after the final manifest/event optimization. Sequential pytest measured approximately 520 MiB on the same generated suite. These older macOS `time` figures are not part of the current baseline JSON and are process maxima, not aggregate memory across every xdist/Testenix child process. The console renderer changed substantially after these captures, and the historical harness did not record output byte counts. Current schema-version 2 runs do record stdout/stderr sizes, but a clean 0.3.0 matrix is still pending. ### Migrated-suite measurements The migration harness generates a source suite, migrates and validates it without modifying any source SHA-256, then measures the original source runner and the published native copy in alternating rounds. All three checked-in scenarios contain 3,000 tests across 64 modules, use one warm-up and five measured rounds, and run native Testenix with four workers on the M4 Pro/CPython 3.11 environment described above. | Source runner | Test body | Source median | Native Testenix median | Native vs source | Migration transaction | |---|---|---:|---:|---:|---:| | sequential pytest | no-op | 1.539 s | 0.521 s | 2.96x faster | 5.940 s | | sequential unittest outcome probe | no-op | 0.161 s | 1.192 s | 7.40x slower | 6.742 s | | sequential unittest outcome probe | 1 ms sleep | 4.066 s | 2.577 s | 1.58x faster | 17.251 s | The transaction duration is reported separately from recurring execution. It includes generation, the source baseline, serial and parallel native candidates, parity checks, source rehashing, and atomic publication. The 1 ms unittest transaction is longer because validation executes the synthetic work repeatedly. Conversion is therefore an occasional safety cost, not a per-CI-run speedup input. The no-op unittest row exposes the adapter's fixed cost: each native test is a wrapper that loads the unchanged source and translates the result of `TestCase.run()`. The sequential source probe uses the stdlib loader and result semantics, then serializes per-test outcomes; it wins when the test body does essentially nothing. With 1 ms per method, four-worker execution across 64 modules amortizes that cost and overtakes the sequential source runner. By default, a suite concentrated in one module exposes only one affinity unit; opt-in safety-checked sharding may change that for an independent module. Too many workers can still add process and import overhead. Test duration, module distribution, imports, fixtures, I/O, failures, operating system, and competing parallel runners must all be measured on the target project. The generated [benchmark results](https://polishdataengineer.github.io/testenix/benchmarks/results/) publish every sample, range, standard deviation, command, environment field, and raw JSON link. These three synthetic records are useful for finding overhead boundaries; they are not evidence that converted suites are universally faster. ### The 118-test validation was not a benchmark The [v0.2.0 release notes](https://github.com/polishdataengineer/testenix/releases/tag/v0.2.0) reported one final validation observation from a real 118-test pytest project: 3.120 seconds for pytest, 2.870 seconds for native serial execution, and 2.423 seconds for native parallel execution. Those values correspond to roughly 1.09× and 1.29× for that observation, not 3.15×. They had no committed raw rounds, warm-up series, counterbalanced order, or publishable environment manifest, so their role was outcome parity (118/118 in all modes), not performance marketing. A small real suite can differ sharply from the 100,000-test no-op baseline. Interpreter spawn and application import costs are a much larger fraction of its wall time; fixtures, mocks, files, databases, and actual test bodies dominate framework overhead; and module affinity prevents one large module from being split between workers. `workers=auto` is adaptive in current Testenix and must be recorded as the worker count actually observed, not assumed to equal logical CPU count. For a demonstrably independent large module, opt-in `--shard-modules` can create finer units after static safety checks; it is not a safe default for arbitrary module state. Use `testenix tune` for a local worker recommendation, then use the real-project manifest harness and publish five counterbalanced rounds before drawing a project-specific conclusion. ### Worker-count sensitivity An earlier exploratory run of 10,000 no-op tests across 16 modules produced these Testenix medians: | Workers | Testenix median | |---:|---:| | 1 | 1.926 s | | 4 | 1.483 s | | 14 | 1.773 s | Four workers were best for this short-test workload. This does not imply a universal four-worker optimum: long CPU-bound tests can benefit from more processes. Adaptive worker selection should use measured history rather than a fixed cap tuned to one machine. ### History and event-log cost The original implementation coupled SQLite duration history to an event sink that opened, locked, wrote, and closed its JSONL file for every event. A 10,000-test default-history run took 10.97 seconds. Keeping the descriptor open, avoiding internal fanout serialization, and storing one self-contained attempt event instead of redundant post-hoc phase events reduced an equivalent run to 1.98–2.30 seconds. Replay files still contain every test specification, complete attempt phases, and final status. ## Profile and implemented optimizations The first 10,000-test profile performed approximately 25.9 million Python calls. It spent large fractions of coordinator time serializing the same 80,000 events more than once, reducing those events again, resolving paths per test, and waiting for duplicated IPC payloads. The optimized profile performed approximately 3.46 million calls. The main changes were: 1. Untimed synchronous tests reuse an executor thread instead of creating one OS thread per test. Timed tests retain a daemon-thread plus hard process deadline, so a stuck Python thread cannot block worker termination. 2. A successful worker streams each completed result for crash recovery and sends only a final ACK; it no longer sends the entire result tuple a second time. 3. Contract paths are computed once per module, source lines use `co_firstlineno`, and worker rediscovery builds an O(1) test-ID index instead of doing an O(m²) sequence of linear lookups. 4. The one-sink event path avoids JSON fanout. Duplicate event serialization is lazy and only runs when an event ID actually repeats. 5. Event IDs use the already unique `run_id:sequence` form instead of calling the OS random source tens of thousands of times. 6. Coordinator attempts are persisted as one complete replay event rather than a redundant started/three-phase/finished sequence emitted only after the attempt had already finished. 7. Unchanged selected specifications reuse discovered objects; selection events are omitted when every test is selected and no effective contract changed. 8. JSONL keeps one append descriptor for the run rather than performing open/close per event. 9. An explicit trusted collection manifest can remove the collection-side import from later unchanged runs. Roots, the complete file inventory, and every source SHA-256 are verified first; stale manifests fall back to supervised collection. Execution still imports assigned modules. Correctness was retained throughout: the framework's full resilience suite passes after every optimization, including worker crashes, timeout process-tree cleanup, collection crashes/hangs, async task leaks, fixture teardown attribution, cancellation, retries, and event replay. ## Rust decision Rust is not the next highest-value optimization. Python imports, Python test and fixture bodies, and CPython object interaction remain Python work. PyO3 can release the interpreter only while performing Rust-only work; it cannot make arbitrary Python callbacks execute in parallel under the GIL. The migration path does not change that decision. Parsing Python ASTs, hashing and copying files, and publishing a new directory happen only when regenerating a suite, while most validation time comes from executing the source and candidate Python tests. A Rust converter would not shorten a 1 ms Python test body or `unittest.TestCase.run()`. The empty-unittest result instead points to profiling wrapper loading and result adaptation, then optimizing or caching those Python-level boundaries without weakening SHA verification and fail-closed publication. Direction-finding microbenchmarks on this machine showed: | Candidate | Current signal | Rust/PyO3 result | Decision | |---|---:|---:|---| | LPT scheduling | ~2.1 ms for 1,000 synthetic units; real plan usually 16 units | bulk Rust loop ~17.5x faster | Absolute saving is below 2 ms; do not add native packaging for it | | Event build + reduce | ~32 ms per 1,000 tests in a light microbenchmark | likely reducible in bulk | Revisit only if it exceeds 10% of final wall time | | JSON event encoding | ~83 ms for ~8,000 detailed events | plausible 2–5x native gain | Relevant mainly to log-heavy mode; Python event compaction already recovered more | | Pipe transport | 36.7 ms for 1,000 individual messages vs 12.2 ms as one batch | native framing could help | Batch/checkpoint in Python first | | Sync invocation | 40–57 ms per 1,000 `to_thread` calls | Python body still crosses the GIL | Requires executor architecture, not Rust | | Python list/record conversion | FFI conversion dominated the operation | PyO3 was slower than Python built-ins | Avoid fine-grained FFI calls | If a later profile identifies a native-worthy data plane, the preferred design is an optional PyO3 `abi3` extension with one bulk call per batch/run and a pure-Python fallback. Candidate scope: framed IPC encoding/decoding or a bulk event reducer. Acceptance requires more than 10% end-to-end improvement on Linux, macOS, and Windows with identical terminal results. A Rust sidecar or embedded Python runtime is not justified: both retain Python interpreter startup/import costs while adding another protocol and a much larger release matrix. Relevant upstream constraints are documented in the [PyO3 parallelism guide](https://pyo3.rs/main/parallelism), [PyO3 performance guide](https://pyo3.rs/main/performance.html), and [Maturin mixed-project guide](https://www.maturin.rs/project_layout.html). ## Next measurement gates - publish the clean Testenix 0.3.0 dimension-sweep matrix for 100/500/1,000/3,000 tests, balanced/dominant/single-module layouts, 1/2/4/adaptive-auto workers, and both history modes; - compare pytest-xdist `load`, `loadfile`, `loadscope`, and `worksteal` where each strategy is valid; - collection, execution, IPC-byte, process-start, CPU, and aggregate-memory telemetry; - 1,000/10,000/100,000 tests across 1, 16, 1,000, and 10,000 modules; - synchronous and asynchronous fixtures, failures, captured output, timeouts, and retries; - cold and warm SQLite history runs; - real project suites and Linux/macOS/Windows CI runners; - real migrated pytest and unittest suites, including sequential and established parallel source runners, multiple module layouts, and a break-even curve by median test duration; - checkpoint-batched IPC followed by another profile; - measure trusted-manifest hit and stale-fallback paths on import-heavy real projects, and consider a persistent collect-and-execute worker only if the remaining execution import is still material. No universal “always faster than pytest” statement should be published until the current-version, real-project, and cross-platform gates pass. The supported claim today is historical and narrower: Testenix 0.1.0 was materially faster in the recorded native large passing-suite scenarios while retaining supervised isolation and complete results. Testenix 0.3.0 has no checked-in speedup claim yet. --- # Document: Architecture Canonical URL: https://polishdataengineer.github.io/testenix/architecture/ Source: docs/architecture.md # Testenix architecture Testenix is a native Python testing framework. Its core does not depend on pytest. Compatibility adapters may translate foreign test frameworks into the same manifest and event contracts, but they are not part of native execution. The first pytest compatibility bridge deliberately does not translate pytest internals. On POSIX, `testenix pytest` uses `os.execv` to replace itself with `sys.executable -m pytest`. On Windows it calls pytest's public `console_main` entry point in the existing process because the platform's `exec` family does not provide equivalent replacement semantics. Both paths keep pytest in the foreground CLI process with the same working directory, environment, terminal, streams, signal handling, and exit status. This preserves semantics without making the native core depend on pytest: ```text POSIX: testenix pytest ==exec==> Python -m pytest -> collector/plugins/executor -> output/status Windows: testenix pytest =========> pytest.console_main -> collector/plugins/executor -> output/status ``` The bridge is a CLI infrastructure adapter, not a native collection adapter. It does not emit Testenix events or construct a `RunResult` in version 0.3. The migration adapter is separate from that handoff. It statically converts a deliberately small pytest subset or generates SHA-pinned wrappers around the standard unittest protocol. Its application service owns a fail-closed transaction: ```text immutable source snapshot |--- original runner in project shadow --------------------> baseline summary |--- generated copy -> native 1-worker shadow ------------> serial summary |--- generated copy -> native parallel shadow ------------> parallel summary +--- per-mapping/count/outcome/hash parity -> no-replace rename -> new output ``` Converters consume serializable `SourceFile` values and return generated artifacts, mappings, and line-addressed diagnostics. They do not access the live filesystem. `migration_fs` is the only publication adapter: it rejects link traversal and overlapping paths, creates private same-filesystem staging, rehashes the source, and uses an operating-system atomic no-replace primitive. POSIX staging creation, writes, publication, and cleanup remain anchored to captured directory identities; platforms without safe recursive descriptor deletion retain a non-empty failed staging tree. A pre-rename failure has no destination to roll back because the source was never a write target. A post-rename durability or audit-report warning is reported as published rather than as a fictional rollback. Unittest wrappers deliberately keep original `TestCase.run()` as the semantic authority. They are native scheduler units, but resolve the original through an exact wrapper-relative path and load the class only after verifying the complete selected-Python-source SHA-256 manifest. This avoids approximating setup, teardown, cleanup, assertion, mocking, async, skip, and expected failure behavior. ## Product contract Testenix aims to be typed, async-native, parallel-first, deterministic, and lossless when reporting test outcomes. A retry never overwrites an earlier attempt, infrastructure failures are distinct from test failures, and setup/call/teardown are preserved as separate phases. ```text Authoring API -> supervised collection -------> inert manifest -> scheduler -> process workers ^ ^ | -> streamed attempts | | +------------> append-only events trusted manifest +-- roots/inventory/SHA-256 verify -> reducer exact match bypasses collection imports -> reports/history ``` ## Dependency rules - `contracts` contains serializable domain values and imports no infrastructure. - `api`, `discovery`, `fixtures`, and `executor` form the native engine. - `events`, `aggregate`, and `scheduler` remain engine-independent. - `runner` is the application service connecting the native engine with execution policy. - `tuning` models adaptive worker selection and runs explicit project-local candidate measurements. - `sharding` contains fail-closed static module decisions and the versioned trusted-manifest serialization/verification boundary. - reporters and storage consume completed domain results or versioned events. - optional compatibility adapters stay at the CLI boundary; the native core never imports pytest. - migration analyzers depend on serializable migration contracts, while shadow execution and atomic publication remain application/infrastructure concerns. ## Version 0.3 scope - explicit `@test` and `@fixture` authoring API, plus conventional `test_*` discovery; - sync functions, coroutines, generators, and async-generator fixture teardown; - explicit cases, tags, skip, expected failure, and per-test timeout metadata; - fixture scopes: test, module, session, with broader scopes currently bounded by a worker shard; - sequential and local process execution; - deterministic scheduling based on historical durations; - adaptive `workers = "auto"` capped by real execution units, history-informed predicted makespan, process-start cost, and CPU capacity; - explicit project-local worker tuning, conservative opt-in intra-module sharding, and an explicitly generated source-verified collection manifest; - append-only JSONL events and a pure reducer; - console, JSON, and JUnit output plus local SQLite duration history; - retries represented as immutable attempts and finalized as `FLAKY` when appropriate; - an optional platform-aware pytest handoff for unchanged legacy suites. - conservative pytest/unittest migration with static diagnostics, differential validation, source fingerprints, and create-only publication. - dependency-free `tmp_path` and reversible `monkeypatch` fixtures, plus native autouse resolution; - conservative migration of bare pytest-asyncio coroutine markers through isolated fresh-loop wrappers, plus simple pytest classes. Remote workers, distributed storage, result caching, automatic quarantine, and a stable third-party plugin SDK are deliberately outside version 0.3. ## Fixture scopes and process isolation By default, the scheduler treats every normal test module as one affinity unit and never splits that unit between parallel shared workers. Multiple modules assigned to one shard execute in one persistent process and fixture runtime. A test with an explicit timeout (including a global timeout applied at selection) is instead a single-test isolation unit with a hard process deadline. An explicit intra-module sharding policy can turn tests in an eligible module into finer units. The static analysis fails closed for module/session fixtures, writes or obvious mutations of module globals, and import-time lifecycle behavior. Function-scoped fixtures can be recreated per worker. Because arbitrary dynamic calls and external effects cannot be proven safe, passing this policy is a caller trust decision; ineligible modules keep normal affinity. Scope therefore has the following concrete meaning in version 0.3: | Scope | Lifetime | | --- | --- | | `test` | One instance for one concrete test attempt. | | `module` | One instance for all normal attempts from that module in a shared worker; a timed test has an isolated instance. | | `session` | One instance for all normal tests assigned to a shared worker process; one per isolated timed process. | Session fixtures are intentionally not run-global. Suites that require a true global singleton must keep that resource outside the fixture runtime until a coordinator-owned resource protocol is implemented. Setting one worker gives one shared session for normal tests, but timed tests remain isolated by design. ## Worker protocol and recovery Workers receive only primitive rediscovery locators (path, function/case identity, and effective timeout), so arbitrary case values never cross the spawn boundary. A worker streams each completed `AttemptResult` before starting the next test and later republishes an owner if session teardown changes its result. The final batch envelope is still sent for normal completion. The supervisor drains the pipe while the child runs, avoiding pipe-buffer deadlocks for large results. If a later test crashes the process, already streamed attempts remain authoritative and only unfinished tests receive infrastructure/crash outcomes. A lost final ACK after all provisional results is a teardown failure, never a green run. A process crash gets one automatic recovery attempt without consuming the user's retry budget, but `CRASH -> PASS` remains `FLAKY`; only a framework-owned `INFRA_ERROR -> PASS` recovery becomes plain `PASS`. Collection itself uses the same spawn-based supervisor and returns only a JSON-safe manifest. A top-level import crash or deadline becomes a `CollectionIssue`, so user import code cannot exit or indefinitely block the coordinator. Timed execution units send a ready handshake after rediscovery; the test deadline therefore does not include interpreter startup or module import. A caller may instead supply a `TrustedCollectionManifest` created by an earlier explicit collection. Before using it, Testenix enumerates the current roots and verifies the complete file set and every SHA-256 digest. An exact match bypasses collection imports; a stale manifest falls back to the supervised collector. Malformed serialized input is rejected at the adapter boundary. Manifest parameter values are redacted at creation and serialization; only their names remain. The manifest also carries prior sharding decisions so collection and scheduling agree. A module using a fixture provider from outside its fingerprinted source fails closed to module affinity. This removes one module import per unchanged run, not the execution-worker import needed to reconstruct Python objects. Inputs to dynamic collection beyond fingerprinted source bytes remain the producer's trust responsibility. Workers normally create a separate POSIX process session (or use recursive tree termination on Windows). During migration validation they remain in the validator's process group so an outer validation deadline can terminate native workers too. Timeout and cancellation terminate ordinary descendants started by a test as well as the direct worker; this is process supervision, not an OS sandbox against a test that deliberately detaches itself. --- # Document: Roadmap Canonical URL: https://polishdataengineer.github.io/testenix/roadmap/ Source: docs/roadmap.md # Roadmap ## 0.1 — native vertical slice - Native function tests and explicit test descriptions. - Typed fixture graph with test, module, and session scopes. - Sync, coroutine, generator, and async-generator execution. - Explicit cases, tags, skip, expected failure, and timeouts. - Local process workers with deterministic LPT scheduling. - Immutable attempts and lossless setup/call/teardown aggregation. - Console, JSON, JUnit XML, JSONL events, and SQLite duration history. - Optional `testenix pytest` compatibility bridge preserving the real pytest engine and plugins. - Safe native migration for a conservative pytest subset and direct unittest TestCase classes, with source fingerprints, disposable validation copies, serial/parallel parity, and atomic create-only publication. ## 0.2 — real-world pytest migration - Dependency-free native `tmp_path` and reversible `monkeypatch` fixtures for common `setattr`/`setenv` usage. - Static autouse fixtures with native setup and teardown ownership. - Bare pytest-asyncio coroutine markers translated to isolated fresh-loop wrappers that preserve the plugin's default function-scoped loop lifecycle. - Fresh-instance wrappers for simple pytest classes, while complex lifecycle and inheritance stay fail-closed. - Compact migration diagnostics on the console with complete line detail retained in JSON. - Differential validation against a 118-test real-world suite before publication. ## 0.3 — fast feedback - Adaptive worker selection based on real execution units, duration history, and process cost, plus an explicit project-local `tune`/`benchmark` command. - Conservative opt-in intra-module sharding and a source-verified trusted collection manifest that can bypass duplicate collection imports without trusting stale source metadata. - Dynamic micro-shards and work stealing. - `--last-failed`, watch mode, and failure fingerprints. - Test-impact analysis in shadow mode with an explanation for every selection decision. - Stable assertion-diff protocol and improved plain-assert diagnostics. ## 0.4 — adoption - Pytest hook adapter translating collection and outcomes into Testenix events and `RunResult`. - Expand migration beyond the v0.2 static subset only when new transformations have differential semantics tests on real projects. - Versioned reporter and selector plugin interfaces. - IDE protocol built on the versioned machine-readable collection manifest. ## Later - Remote workers with leases and heartbeats. - Resource-capacity scheduling and explicit test affinity. - Opt-in caching for tests that declare all external inputs. - Flakiness history, quarantine ownership, and expiration policy. Result caching and impact selection will not become build-gating defaults until their false-negative rate is measured continuously against full-suite runs. --- # Document: Changelog Canonical URL: https://github.com/polishdataengineer/testenix/blob/main/CHANGELOG.md Source: CHANGELOG.md # Changelog All notable changes will be documented in this file. The format follows Keep a Changelog and the project intends to use Semantic Versioning once its public API reaches stability. ## [Unreleased] ## [0.3.0] - 2026-07-21 ### Added - `testenix tune` and its `testenix benchmark` alias for fresh-process, counterbalanced, history-disabled native worker-candidate measurements, native inventory/outcome validation, JSON reports, bounded per-run process-tree deadlines, and explicit `--write` persistence of the measured recommendation with project-source fingerprinting and optimistic byte-drift protection immediately before an atomic configuration replacement. - Explicit `--shard-modules` / `shard_modules = true` support for splitting eligible modules into finer execution units. Conservative static checks retain module affinity for module/session fixtures, visible global mutation, and import-time lifecycle hazards, including eager calls in assignments, decorators, function defaults, and class construction expressions. - Versioned trusted collection manifests generated with `testenix manifest ... --output FILE` and consumed with `testenix run --manifest FILE` or `[tool.testenix].manifest`. Exact collection roots, selected test files, statically discoverable project-local import dependencies, and SHA-256 digests are verified before collection imports are bypassed; stale manifests fall back to supervised collection, and parameter values are redacted. - Synthetic scaling-matrix tooling for 100/500/1,000/3,000 tests and balanced, dominant, and single-module layouts, plus a redaction-safe real-project benchmark harness. ### Changed - `workers = "auto"` now selects adaptively from the actual execution-unit count, available CPUs, duration-history coverage, predicted process-start cost, and makespan instead of equalling the logical CPU count. Explicit integer worker settings remain unchanged. - Benchmark documentation labels the historical `3.15×` result with its Testenix 0.1.0 version, four-worker configuration, 100,000-test/16-module synthetic workload, and `--no-history` mode; it is not presented as a current-version or real-project claim. ### Fixed - Safe-module analysis now fails closed for imported fixture providers, nested mutable containers, mutable class state, and all import-time calls including nested `sys.path` mutations. - Benchmark and tuning timeouts use Windows Job Objects or POSIX root-session plus identity-tracked descendant cleanup instead of allowing observed workers to contaminate later measurements. ## [0.2.1] - 2026-07-21 ### Added - Native console controls for quiet output, one- or two-level verbosity, skipped/expected-failure reasons, slow-test duration lists, and explicit automatic/forced/disabled ANSI color handling. ### Changed - `testenix run` now defaults to a compact per-file report while retaining complete collection and failure diagnostics plus the final summary. Console rendering remains deterministic and is emitted after execution rather than presented as live progress. - Documentation now distinguishes native Testenix rendering from the unchanged pytest output produced by the transparent `testenix pytest` compatibility bridge. ### Fixed - Compact reports retain complete failing node IDs and wrap unusually varied per-file status summaries without dropping the file name or any status counts. - Generated LLM API snapshots are stable across all supported Python versions. ## [0.2.0] - 2026-07-20 ### Added - `testenix pytest [PYTEST_ARGS ...]` compatibility bridge for unchanged pytest suites, preserving the real pytest collector, fixtures, parametrization, markers, plugins, output, and exit status. - Optional `testenix[pytest]` installation extra and a documented native-versus-compatibility capability matrix. - `testenix migrate {auto,pytest,unittest}` with dry-run, full check, JSON audit reporting, source SHA-256 snapshots, disposable project shadows, serial/parallel differential validation, and atomic create-only publication to a new directory. - Conservative pytest transformations for static module functions, cases, simple fixtures, adjacent `conftest.py`, skips, and selection markers, with stable diagnostics for unsupported semantics. - Native unittest wrapper generation that retains the original `TestCase.run()` lifecycle and assertions while resolving sources independently of `cwd` and pinning every selected Python module through a SHA-256 manifest. - A reproducible migrated-suite benchmark harness for 3,000+ pytest or unittest tests, including count/hash gates, counterbalanced rounds, environment provenance, and raw JSON output. - Per-test source-to-target outcome parity, configured parallel validation with an explicit one-affinity-unit warning, bounded descendant cleanup, disjoint create-only audit reports, and descriptor-anchored POSIX staging creation, writes, publication, and cleanup. - Truthful post-commit durability/report warnings, package-aware unittest outcome mapping, Testenix validation-worker containment, and conservative blocking of pytest session fixtures and unittest class-cleanup hooks whose lifecycle cannot be preserved. - Native `tmp_path` and transactional `monkeypatch` fixtures. The initial monkeypatch contract covers the object/attribute and dotted-import forms of `setattr`, plus `setenv`, with automatic per-test rollback. Static module-local helper calls are accepted only when every propagated use can be proven safe; aliases, dynamic rebinding, unsupported methods, and escaped values remain blocked. - Safe conversion of bare `@pytest.mark.asyncio` coroutine tests, simple pytest classes through fresh-instance wrappers, and statically declared autouse fixtures. Async migration creates and closes an isolated `asyncio.Runner` per test or case, validates effective pytest-asyncio loop and debug configuration, and blocks custom event-loop policies or unmarked async semantics. - Fail-closed class conversion for lifecycle hooks, decorated or inherited classes, annotated class state, custom constructors, and method defaults that cannot be preserved by wrappers. ### Changed - Migration console output now distinguishes analyzed, validated, generated, and published candidates. Repeated diagnostics are grouped by code, while JSON audit reports retain every source- and line-addressed entry. - The one-affinity-unit `MIG006` warning is emitted only for statically supported check/publication candidates, not for dry-run or already-blocked migrations. ## [0.1.0] - 2026-07-20 ### Added - Native authoring API, fixture graph, cases, and sync/async execution. - Versioned event stream and lossless run/test/attempt/phase result model. - Deterministic local scheduling, process-worker supervision, and retries. - Console, JSON, JUnit XML, and SQLite history adapters. - `testenix` command-line interface and `pyproject.toml` configuration. - Supervised collection, worker-ready handshakes, process-tree cleanup, and cancellable async runs. - Module affinity, streamed partial-result recovery, and stable rediscovery locators for arbitrary case values. - Strict mypy/pyright validation and adversarial tests for crashes, hangs, retries, and teardown ownership. - Counterbalanced, count-validating 1k/10k/100k performance harness with configurable module count and scalable uneven workloads. - A searchable Sphinx/Furo documentation site for GitHub Pages, generated Python API reference, and one-click page/full-document copying for LLM context. - Deterministic `llms.txt` and `llms-full.txt` references plus generated benchmark tables and SVG. - Manual GitHub benchmark workflow with downloadable raw JSON results. ### Changed - Reused sync executor threads, removed duplicate final IPC payloads, indexed worker rediscovery, and eliminated per-test path/source inspection. - Compacted coordinator events, made event IDs deterministic, retained one JSONL descriptor per run, and removed unchanged manifest/selection copies. - Reduced the recorded 100k-test median to 8.038 seconds versus 25.333 seconds for pytest on the documented M4 Pro development baseline; comparative claims remain workload-specific. - Isolated benchmark runs from repository pytest configuration, disabled pytest cache/plugin autoloading, and added commit, lockfile, version, and dirty-state provenance to new results. - Hardened PyPI releases by requiring the release tag to reference a commit contained in `main`. --- # Document: Security policy Canonical URL: https://github.com/polishdataengineer/testenix/blob/main/SECURITY.md Source: SECURITY.md # Security policy ## Supported versions Testenix is currently pre-1.0. Security fixes are applied to the latest released version. ## Reporting a vulnerability Please do not open a public issue for a suspected vulnerability. Use GitHub's private vulnerability reporting for this repository instead. Include the affected version, a minimal reproduction, impact, and any suggested mitigation. We will acknowledge a complete report within seven days and coordinate disclosure after a fix is available. --- # Expanded public API snapshot This section is generated from `testenix.__all__` so an LLM receives concrete symbols, signatures, enum values, dataclass fields, and docstrings rather than unresolved documentation directives. ## `testenix.CaseDefinition` ```text CaseDefinition(parameters: 'Mapping[str, Any]', id: 'str | None' = None) -> None ``` Fields: - `parameters`: `Mapping[str, Any]` - `id`: `str | None` One concrete parameter mapping attached to a test function. ## `testenix.CollectionResult` ```text CollectionResult(items: 'tuple[CollectedTest, ...]', registry: 'FixtureRegistry', issues: 'tuple[CollectionIssue, ...]' = ()) -> None ``` Fields: - `items`: `tuple[CollectedTest, ...]` - `registry`: `FixtureRegistry` - `issues`: `tuple[CollectionIssue, ...]` Complete collection output, including non-fatal authoring issues. ## `testenix.CollectionManifestError` A trusted collection manifest is malformed or unsafe to resolve. ## `testenix.Event` ```text Event(event_id: 'str', run_id: 'str', event_type: 'EventType', timestamp: 'float', sequence: 'int', schema_version: 'int' = 1, test_id: 'str | None' = None, attempt: 'int | None' = None, worker_id: 'str | None' = None, payload: 'dict[str, Any]' = ) -> None ``` Fields: - `event_id`: `str` - `run_id`: `str` - `event_type`: `EventType` - `timestamp`: `float` - `sequence`: `int` - `schema_version`: `int` - `test_id`: `str | None` - `attempt`: `int | None` - `worker_id`: `str | None` - `payload`: `dict[str, Any]` Append-only fact emitted while a run is executing. ## `testenix.EventSink` ```text EventSink(*args, **kwargs) ``` Minimal sink contract used by runners, workers and adapters. ## `testenix.MigrationOptions` ```text MigrationOptions(framework: 'Framework', sources: 'tuple[Path, ...]', output: 'Path' = PosixPath('testenix_migrated'), workers: 'WorkerCount' = 'auto', validation_timeout: 'float' = 300.0, dry_run: 'bool' = False, check_only: 'bool' = False, project_root: 'Path | None' = None) -> None ``` Fields: - `framework`: `Framework` - `sources`: `tuple[Path, ...]` - `output`: `Path` - `workers`: `WorkerCount` - `validation_timeout`: `float` - `dry_run`: `bool` - `check_only`: `bool` - `project_root`: `Path | None` Validated input for :func:`migrate`. ## `testenix.MigrationReport` ```text MigrationReport(status: 'MigrationStatus', mode: 'str', framework: 'str', project_root: 'str', sources: 'tuple[str, ...]', output: 'str', source_hashes: 'Mapping[str, str]', generated_files: 'tuple[str, ...]', mappings: 'tuple[TestMapping, ...]', diagnostics: 'tuple[MigrationDiagnostic, ...]', baseline: 'ValidationSummary | None', native_serial: 'ValidationSummary | None', native_parallel: 'ValidationSummary | None', originals_modified: 'bool', published: 'bool', message: 'str', started_at: 'float', finished_at: 'float') -> None ``` Fields: - `status`: `MigrationStatus` - `mode`: `str` - `framework`: `str` - `project_root`: `str` - `sources`: `tuple[str, ...]` - `output`: `str` - `source_hashes`: `Mapping[str, str]` - `generated_files`: `tuple[str, ...]` - `mappings`: `tuple[TestMapping, ...]` - `diagnostics`: `tuple[MigrationDiagnostic, ...]` - `baseline`: `ValidationSummary | None` - `native_serial`: `ValidationSummary | None` - `native_parallel`: `ValidationSummary | None` - `originals_modified`: `bool` - `published`: `bool` - `message`: `str` - `started_at`: `float` - `finished_at`: `float` Complete audit record for a migration attempt. ## `testenix.MigrationStatus` ```text MigrationStatus(value) ``` Values: ANALYZED='analyzed', VALIDATED='validated', PUBLISHED='published', UNSUPPORTED='unsupported', VALIDATION_FAILED='validation_failed', SAFETY_ERROR='safety_error' Terminal state of one migration transaction. ## `testenix.TestenixConfig` ```text TestenixConfig(paths: 'tuple[str, ...]' = ('tests',), workers: "int | Literal['auto']" = 'auto', retries: 'int' = 0, timeout: 'float | None' = None, tags: 'tuple[str, ...]' = (), json_path: 'Path | None' = None, junit_path: 'Path | None' = None, history_path: 'Path | None' = PosixPath('.testenix/history.sqlite3'), shard_modules: 'bool' = False, manifest_path: 'Path | None' = None) -> None ``` Fields: - `paths`: `tuple[str, ...]` - `workers`: `int | Literal['auto']` - `retries`: `int` - `timeout`: `float | None` - `tags`: `tuple[str, ...]` - `json_path`: `Path | None` - `junit_path`: `Path | None` - `history_path`: `Path | None` - `shard_modules`: `bool` - `manifest_path`: `Path | None` Validated execution and reporting settings. Paths are kept relative to the process working directory. This mirrors normal command-line path handling and makes a config object safe to pass to a worker process without retaining hidden project state. ## `testenix.RunResult` ```text RunResult(run_id: 'str', tests: 'tuple[TestResult, ...]', collection_issues: 'tuple[CollectionIssue, ...]', started_at: 'float', finished_at: 'float', workers_used: 'int | None' = None, shardable_paths: 'tuple[str, ...]' = ()) -> None ``` Fields: - `run_id`: `str` - `tests`: `tuple[TestResult, ...]` - `collection_issues`: `tuple[CollectionIssue, ...]` - `started_at`: `float` - `finished_at`: `float` - `workers_used`: `int | None` - `shardable_paths`: `tuple[str, ...]` RunResult(run_id: 'str', tests: 'tuple[TestResult, ...]', collection_issues: 'tuple[CollectionIssue, ...]', started_at: 'float', finished_at: 'float', workers_used: 'int | None' = None, shardable_paths: 'tuple[str, ...]' = ()) ## `testenix.ShardingPolicy` ```text ShardingPolicy(intra_module: 'bool' = False) -> None ``` Fields: - `intra_module`: `bool` Core scheduling policy independent of CLI/configuration concerns. Intra-module sharding is deliberately opt-in. Callers which do not pass a policy, or pass the default instance, retain the original module-affinity behaviour exactly. ## `testenix.Scope` ```text Scope(value) ``` Values: TEST='test', MODULE='module', SESSION='session' Lifetime of a fixture instance. ## `testenix.Status` ```text Status(value) ``` Values: PASS='pass', FAIL='fail', ERROR_SETUP='error_setup', ERROR_TEARDOWN='error_teardown', SKIP='skip', XFAIL='xfail', XPASS='xpass', TIMEOUT='timeout', CRASH='crash', INFRA_ERROR='infra_error', CANCELLED='cancelled', NOT_RUN='not_run', FLAKY='flaky', CACHED_PASS='cached_pass' A lossless outcome vocabulary for phases, attempts, and tests. ## `testenix.TestResult` ```text TestResult(test: 'TestSpec', status: 'Status', attempts: 'tuple[AttemptResult, ...]', duration: 'float') -> None ``` Fields: - `test`: `TestSpec` - `status`: `Status` - `attempts`: `tuple[AttemptResult, ...]` - `duration`: `float` TestResult(test: 'TestSpec', status: 'Status', attempts: 'tuple[AttemptResult, ...]', duration: 'float') ## `testenix.TestSpec` ```text TestSpec(id: 'str', path: 'str', module_name: 'str', function_name: 'str', display_name: 'str', parameters: 'dict[str, Any]' = , case_id: 'str | None' = None, tags: 'frozenset[str]' = , skip_reason: 'str | None' = None, xfail_reason: 'str | None' = None, timeout: 'float | None' = None, source_line: 'int | None' = None) -> None ``` Fields: - `id`: `str` - `path`: `str` - `module_name`: `str` - `function_name`: `str` - `display_name`: `str` - `parameters`: `dict[str, Any]` - `case_id`: `str | None` - `tags`: `frozenset[str]` - `skip_reason`: `str | None` - `xfail_reason`: `str | None` - `timeout`: `float | None` - `source_line`: `int | None` Serializable description of one concrete test case. ## `testenix.TrustedCollectionManifest` ```text TrustedCollectionManifest(collection_roots: 'tuple[str, ...]', files: 'tuple[SourceFingerprint, ...]', tests: 'tuple[TestSpec, ...]', issues: 'tuple[CollectionIssue, ...]' = (), sharding: 'tuple[ModuleShardingDecision, ...]' = ()) -> None ``` Fields: - `collection_roots`: `tuple[str, ...]` - `files`: `tuple[SourceFingerprint, ...]` - `tests`: `tuple[TestSpec, ...]` - `issues`: `tuple[CollectionIssue, ...]` - `sharding`: `tuple[ModuleShardingDecision, ...]` Explicit portable collection result that may bypass collection imports. This is deliberately not an implicit cache. A caller creates or loads the manifest and opts into trusting it for a run. Parameter names are retained for diagnostics, but every parameter value is replaced with the explicit ```` sentinel. Testenix still compares the complete test-file inventory plus local import dependencies and every SHA-256 digest before using it. Dynamic collection influenced by anything other than source bytes (for example environment variables) remains the manifest producer's trust decision. ## `testenix.ValidationSummary` ```text ValidationSummary(runner: 'str', tests: 'int', passed: 'int', failed: 'int', errors: 'int', skipped: 'int', xfailed: 'int', xpassed: 'int', exit_code: 'int', duration: 'float', timed_out: 'bool' = False, detail: 'str | None' = None, outcomes: 'Mapping[str, str]' = ) -> None ``` Fields: - `runner`: `str` - `tests`: `int` - `passed`: `int` - `failed`: `int` - `errors`: `int` - `skipped`: `int` - `xfailed`: `int` - `xpassed`: `int` - `exit_code`: `int` - `duration`: `float` - `timed_out`: `bool` - `detail`: `str | None` - `outcomes`: `Mapping[str, str]` Framework-neutral outcome counts from one isolated validation run. ## `testenix.case` ```text case(values: 'Mapping[str, Any] | str | None' = None, /, *, id: 'str | None' = None, **parameters: 'Any') -> 'CaseDefinition' ``` Attach one concrete parameter case. Examples:: @case(user="alice", expected=True) @case("anonymous", user=None, expected=False) @case({"id": 42}, id="record-42") ## `testenix.cases` ```text cases(*definitions: 'CaseDefinition | Mapping[str, Any]', ids: 'Sequence[str] | None' = None, **dimensions: 'Iterable[Any]') -> 'Callable[[F], F]' ``` Attach several cases, either explicitly or as a Cartesian product. ``@cases({"x": 1}, {"x": 2})`` declares explicit mappings, while ``@cases(role=["admin", "editor"], active=[True, False])`` creates the Cartesian product of the supplied dimensions. ## `testenix.collect_trusted_manifest` ```text collect_trusted_manifest(paths: 'Sequence[str] | str', *, project_root: 'str | Path | None' = None) -> 'TrustedCollectionManifest' ``` Create an explicit collection manifest in a supervised worker process. ## `testenix.deserialize_trusted_collection_manifest` ```text deserialize_trusted_collection_manifest(data: 'str | bytes | bytearray | Mapping[str, Any]') -> 'TrustedCollectionManifest' ``` Decode and validate trusted collection manifest JSON or mapping data. ## `testenix.discover` ```text discover(paths: 'str | Path | Iterable[str | Path]' = '.') -> 'CollectionResult' ``` Discover native tests and fixtures below one or more paths. Directories are searched recursively for ``test_*.py``. An explicitly supplied Python file may have any name, which is useful for focused runs. Inside a module, ``test_*`` functions and functions decorated with ``@test`` (or parameterized with ``@case(s)``) are collected. ## `testenix.fixture` ```text fixture(function: 'Callable[..., Any] | None' = None, /, *, scope: 'Scope | str' = , name: 'str | None' = None, autouse: 'bool' = False) -> 'Callable[..., Any]' ``` Declare a dependency provider with test, module or session lifetime. ## `testenix.migrate` ```text migrate(options: 'MigrationOptions') -> 'MigrationReport' ``` Analyze, validate, and optionally publish a native Testenix copy. No branch of this function writes to a source path. A destination is only published by a no-overwrite rename after the source fingerprints have been checked for concurrent changes. ## `testenix.run` ```text run(paths: 'Sequence[str] | str | None' = None, config: 'TestenixConfig | None' = None, *, event_sink: 'EventSink | None' = None, sharding_policy: 'ShardingPolicy | None' = None, trusted_manifest: 'TrustedCollectionManifest | None' = None) -> 'RunResult' ``` Discover and execute a native Testenix suite. The coordinator is the only component that decides final outcomes. Workers return immutable attempts, which are emitted to a versioned event stream and reduced after execution. ## `testenix.run_async` ```text run_async(paths: 'Sequence[str] | str | None' = None, config: 'TestenixConfig | None' = None, *, event_sink: 'EventSink | None' = None, sharding_policy: 'ShardingPolicy | None' = None, trusted_manifest: 'TrustedCollectionManifest | None' = None) -> 'RunResult' ``` Cancellable embedding facade around the process-oriented coordinator. ## `testenix.serialize_trusted_collection_manifest` ```text serialize_trusted_collection_manifest(manifest: 'TrustedCollectionManifest') -> 'str' ``` Serialize a trusted collection manifest as deterministic JSON. ## `testenix.skip` ```text skip(reason_or_function: 'str | F | None' = None, /, *, reason: 'str | None' = None, when: 'bool' = True) -> 'F | Callable[[F], F]' ``` Mark a test as skipped, optionally only when a condition is true. ## `testenix.test` ```text test(function_or_description: 'Callable[..., Any] | str | None' = None, /, *, description: 'str | None' = None, tags: 'Iterable[str] | str' = (), timeout: 'float | None' = None) -> 'Callable[..., Any]' ``` Mark a function as a native Testenix test. Supported forms are ``@test``, ``@test()``, ``@test("description")`` and ``@test(description="description", tags={...}, timeout=...)``. ## `testenix.xfail` ```text xfail(reason_or_function: 'str | F | None' = None, /, *, reason: 'str | None' = None, when: 'bool' = True) -> 'F | Callable[[F], F]' ``` Mark a test as expected to fail, optionally only when a condition is true.