From bc86a0875b3a2119db9e4f72b61d58e0845b2664 Mon Sep 17 00:00:00 2001 From: Dov Alperin Date: Wed, 6 May 2026 19:04:13 -0400 Subject: [PATCH] antithesis: scaffold harness, multi-scenario layout, broadened testdrive corpus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Brings up the Antithesis Test Composer harness for Materialize. The branch covers research artifacts, the test-driver mzbuild image, scenario infrastructure, workload helpers (testdrive runner, corpus lists, the upsert-sources prototype), and a few small upstream fixes that were prerequisites. Layout ------ antithesis/ AGENTS.md directory map + scenarios table scratchbook/ SUT analysis, deployment topology, test-driver integration plan, scenario strategy, existing-assertions inventory configs// mzcompose.py source-of-truth composition docker-compose.yaml generated artifact (snouty consumes) bin/render-compose-yaml.py renders configs//docker-compose.yaml from mzcompose.py and layers on platform/hostname/container_name/ NO_COLOR Antithesis attributes test-driver/ mzbuild image: MZFROM testdrive + Python + Antithesis SDK + the workload tree at /opt/antithesis/test/v1/ + curated test/testdrive corpus at /opt/materialize/td/testdrive/ test/v1/ helper_bootstrap.py shared sys.path injector testdrive_{sql,kafka,load_generator,recovery}/ area templates; each picks a random .td from materialize.antithesis.testdrive_corpus upsert_sources/ randomized helper-driven upsert workload with expected-state model misc/python/materialize/antithesis/ sdk.py SDK wrapper with local fallbacks; the antithesis package coexists with our scaffolding directory because both are namespace packages testdrive_config.py shared TestdriveConfig dataclass td_runner.py generic .td runner: subprocess + tolerated-failure retry + reachable() on success and on tolerated failure testdrive_corpus.py curated lists of base-compatible .td files split into 4 area buckets (BASE_SQL=22, BASE_KAFKA=35, BASE_LOAD_GENERATOR=12, BASE_RECOVERY=8 — 72 unique files) upsert_sources.py prototype workload helper Why per-scenario configs ------------------------ `antithesis/scratchbook/scenario-strategy.md` documents the design with citations to the Antithesis docs (snouty docs CLI). In short: incompatible topologies (e.g. base SQL+Kafka vs MySQL CDC with multithreaded replicas) get separate config dirs and separate `snouty launch` invocations; incompatible workloads on the same topology coexist as multiple test templates inside one config (Antithesis selects exactly one template per execution history). The base scenario is the only one wired up here; `mysql_mt_replicas` for the SS-95 ticket is the planned next addition. Why no per-area eventually_* recovery checks -------------------------------------------- Earlier drafts of this branch had per-area `eventually_*` commands. They either reduced to tautologies (`SELECT count(*) >= 0` always passes if pgwire is up) or to a generic CREATE/INSERT/SELECT/DROP round-trip that was only weakly correlated with the chosen .td's actual semantics. When the singleton picks a random .td you can't write a useful recovery property without knowing what state was created. Real recovery properties belong either in SUT-side Rust assertions or in scenario-specific `eventually_*` commands tied to a specific workload — which is exactly the shape `upsert_sources/eventually_*` already has (it writes a sentinel and waits for it). The scratchbook entry warns against re-adding generic per-area recovery checks. Upstream fixes pulled in (could be split out later) --------------------------------------------------- * misc/python/materialize/cli/mzcompose.py — the shtab-Enum-choices workaround broke `--arch` and `--sanitizer` on Python 3.13. Argparse's post-conversion `member in choices` check failed because choices were member names while `type=Enum` returned member objects. Switched to `list(action.choices)` (Enum members) so argparse and shtab are both happy. Includes the regenerated bash/zsh shell completions. * misc/python/materialize/mzbuild.py — the Copy pre-image plugin was unused upstream and crashed when used: `Copy.inputs()` returned paths relative to the source dir, but the mzbuild fingerprinter expected paths relative to the repo root, so `os.lstat(rd.root / rel_path)` hit FileNotFoundError. Made `Copy.inputs()` repo-root-relative and `Copy.run()` strip the source prefix before computing the destination inside the build context. * misc/python/materialize/mzcompose/service.py — added `container_name` to `ServiceConfig` (a real Compose field). Antithesis requires it for log/fault attribution. * ci/test/lint-main/checks/check-mzcompose-files.sh — exclude `antithesis/configs/*/mzcompose.py` from the 'unused in any CI pipeline file' check; Antithesis runs are submitted via snouty, not Buildkite. * ci/builder/requirements.txt — added `antithesis==0.2.0` so the SDK resolves locally for type-checking and ad-hoc imports. Coexists with our `antithesis/` scaffolding directory because both are namespace packages and the merge picks up submodules from site-packages. Known limitations ----------------- * On Apple Silicon, `snouty validate antithesis/configs/base` fails end-to-end because the amd64 `materialized` image's `clusterd` child segfaults under Rosetta during lgalloc init (`unix_wait_status(11)` -> container `Exited (139)`). Run validate on Linux/x86 instead. Documented in antithesis/AGENTS.md. * On Apple Silicon, `bin/mzimage acquire --arch x86_64` currently fails to link with `ld.lld: error: undefined symbol: getauxval`: the `materializeinc/crosstools/x86_64-unknown-linux-gnu` homebrew formula ships glibc 2.12.1, but Rust 1.95's stdlib references getauxval which needs glibc 2.16+. Workaround: `bin/ci-builder run stable bin/mzimage acquire --arch x86_64 antithesis-test-driver` uses the Docker builder's current glibc. The homebrew formula needs an upstream update; CI is unaffected (Linux hosts route through ci-builder by default). Next steps ---------- * SS-95: `mysql_mt_replicas` scenario per scratchbook/test-driver-integration.md * Tier 2 scenarios: pg_cdc, mysql_cdc, sql_server_cdc, s3_copy * Tier 3 structural refactors: parallel-workload regression scenario (gate Database.create() Kafka/CSR/AWS/PG/MySQL/SQLServer/Iceberg CONNECTIONs on flags), zippy execution adapter * Per-scenario template gating via `ANTITHESIS_SCENARIO` env in the test-driver entrypoint, so a scenario only sees its compatible templates * SUT-side Rust assertions where they're justified (rare/dangerous internal states, branch outcomes) --- antithesis/AGENTS.md | 104 ++++++ antithesis/bin/render-compose-yaml.py | 201 ++++++++++++ antithesis/configs/base/docker-compose.yaml | 113 +++++++ antithesis/configs/base/mzcompose.py | 119 +++++++ antithesis/scratchbook/deployment-topology.md | 112 +++++++ antithesis/scratchbook/existing-assertions.md | 64 ++++ antithesis/scratchbook/scenario-strategy.md | 101 ++++++ antithesis/scratchbook/sut-analysis.md | 106 ++++++ .../scratchbook/test-driver-integration.md | 301 ++++++++++++++++++ antithesis/setup-complete.sh | 32 ++ antithesis/test-driver/.gitignore | 5 + antithesis/test-driver/Dockerfile | 91 ++++++ antithesis/test-driver/entrypoint.sh | 105 ++++++ antithesis/test-driver/mzbuild.yml | 41 +++ antithesis/test/v1/helper_bootstrap.py | 20 ++ .../singleton_driver_random_kafka | 32 ++ .../singleton_driver_random_load_generator | 33 ++ .../singleton_driver_random_recovery | 39 +++ .../testdrive_sql/singleton_driver_random_sql | 34 ++ .../anytime_upsert_source_health | 17 + .../eventually_upsert_source_catches_up | 17 + .../finally_upsert_expected_rows_visible | 19 ++ .../upsert_sources/first_create_upsert_source | 17 + .../parallel_driver_read_stale_safe | 17 + .../parallel_driver_write_upserts | 17 + ci/builder/requirements.txt | 6 + .../lint-main/checks/check-mzcompose-files.sh | 1 + misc/completions/bash/_mzcompose | 2 +- misc/completions/zsh/_mzcompose | 2 +- .../python/materialize/antithesis/__init__.py | 10 + misc/python/materialize/antithesis/sdk.py | 83 +++++ .../materialize/antithesis/td_runner.py | 109 +++++++ .../antithesis/testdrive_config.py | 95 ++++++ .../antithesis/testdrive_corpus.py | 212 ++++++++++++ .../materialize/antithesis/upsert_sources.py | 266 ++++++++++++++++ misc/python/materialize/cli/mzcompose.py | 13 +- misc/python/materialize/mzbuild.py | 19 +- misc/python/materialize/mzcompose/service.py | 7 + 38 files changed, 2574 insertions(+), 8 deletions(-) create mode 100644 antithesis/AGENTS.md create mode 100755 antithesis/bin/render-compose-yaml.py create mode 100644 antithesis/configs/base/docker-compose.yaml create mode 100644 antithesis/configs/base/mzcompose.py create mode 100644 antithesis/scratchbook/deployment-topology.md create mode 100644 antithesis/scratchbook/existing-assertions.md create mode 100644 antithesis/scratchbook/scenario-strategy.md create mode 100644 antithesis/scratchbook/sut-analysis.md create mode 100644 antithesis/scratchbook/test-driver-integration.md create mode 100755 antithesis/setup-complete.sh create mode 100644 antithesis/test-driver/.gitignore create mode 100644 antithesis/test-driver/Dockerfile create mode 100755 antithesis/test-driver/entrypoint.sh create mode 100644 antithesis/test-driver/mzbuild.yml create mode 100755 antithesis/test/v1/helper_bootstrap.py create mode 100755 antithesis/test/v1/testdrive_kafka/singleton_driver_random_kafka create mode 100755 antithesis/test/v1/testdrive_load_generator/singleton_driver_random_load_generator create mode 100755 antithesis/test/v1/testdrive_recovery/singleton_driver_random_recovery create mode 100755 antithesis/test/v1/testdrive_sql/singleton_driver_random_sql create mode 100755 antithesis/test/v1/upsert_sources/anytime_upsert_source_health create mode 100755 antithesis/test/v1/upsert_sources/eventually_upsert_source_catches_up create mode 100755 antithesis/test/v1/upsert_sources/finally_upsert_expected_rows_visible create mode 100755 antithesis/test/v1/upsert_sources/first_create_upsert_source create mode 100755 antithesis/test/v1/upsert_sources/parallel_driver_read_stale_safe create mode 100755 antithesis/test/v1/upsert_sources/parallel_driver_write_upserts create mode 100644 misc/python/materialize/antithesis/__init__.py create mode 100644 misc/python/materialize/antithesis/sdk.py create mode 100644 misc/python/materialize/antithesis/td_runner.py create mode 100644 misc/python/materialize/antithesis/testdrive_config.py create mode 100644 misc/python/materialize/antithesis/testdrive_corpus.py create mode 100644 misc/python/materialize/antithesis/upsert_sources.py diff --git a/antithesis/AGENTS.md b/antithesis/AGENTS.md new file mode 100644 index 0000000000000..62574c2604f73 --- /dev/null +++ b/antithesis/AGENTS.md @@ -0,0 +1,104 @@ +This directory contains files relevant to running tests in Antithesis. + +Use the `antithesis-setup` skill to scaffold and manage this directory. Use +the `antithesis-research` skill to analyze the system and build a property +catalog. Use the `antithesis-workload` skill to implement assertions and test +commands. Use the `antithesis-launch` skill to build, validate, and submit +Antithesis runs — do not run `snouty run` directly. + +## Scenarios + +Antithesis runs are organized as **N narrow scenarios**, each with its own +compose file under `configs//`. The split rule and rationale live in +`scratchbook/scenario-strategy.md`. Each scenario is launched separately via +`snouty launch -c antithesis/configs/ --source `. + +Available scenarios: + +| Scenario | Topology | Status | +| --- | --- | --- | +| `base` | materialized + redpanda + driver | active | + +When adding a scenario: + +1. Drop a new `configs//mzcompose.py` defining its `SERVICES`. +2. Run `bin/pyactivate antithesis/bin/render-compose-yaml.py --scenario ` + to produce `configs//docker-compose.yaml`. +3. Add an entry to the table above. + +## snouty validate + +Validate a scenario locally: + + snouty validate antithesis/configs/ + +The validator runs Docker Compose, watches for the `setup_complete` event, +and inspects discovered Test Composer commands. Run `bin/mzcompose --find + build` (or `bin/mzimage acquire --arch x86_64 antithesis-test-driver`) +first to make sure the local images are present. + +> **Apple Silicon caveat.** The compose configs pin every service to +> `platform: linux/amd64` because the Antithesis platform is amd64-only. On +> macOS / Apple Silicon, Docker runs amd64 containers under Rosetta, and the +> `clusterd` subprocess that `materialized` spawns segfaults during lgalloc +> initialization (`ExitStatus(unix_wait_status(11))` → container `Exited (139)`). +> This is a host-side emulation limitation, not a harness bug. Run +> `snouty validate` on Linux/amd64 (CI, an x86 dev box, or a remote runner) +> to exercise the full bring-up. + +## snouty launch + +Once validation passes, use the `antithesis-launch` skill to submit a run. + +## Directory layout + +**configs/** +One subdirectory per scenario. Each contains: +* `mzcompose.py` — source-of-truth composition for the scenario. +* `docker-compose.yaml` — generated artifact consumed by snouty / Antithesis. + +**bin/** +Helpers that operate on this directory: + +* `render-compose-yaml.py` — regenerates `configs//docker-compose.yaml` + from `configs//mzcompose.py` while layering in the + Antithesis-required attributes mzcompose does not emit + (`platform: linux/amd64`, matching `container_name`/`hostname`, + `NO_COLOR=1`, no host port bindings): + + bin/pyactivate antithesis/bin/render-compose-yaml.py --scenario + +**test/** +Shared Test Composer command tree. A test template is a directory under +`test/v1/