diff --git a/FEATURES.md b/FEATURES.md index 2a92c8e..523435c 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -29,16 +29,16 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | -| SCN-01 | Scenario file in YAML with a JSON Schema, versioned | M | 0.1 | done | -| SCN-02 | Validation with precise errors (line, column and source line for every error, the path in the file for errors of content, what was expected, how to fix it; reserved keys name their release) | M | 0.1 | done | -| SCN-03 | Units and durations parsed strictly (`40ms`, `10mbit`, `2%`) | M | 0.1 | done | -| SCN-04 | Scenarios built in code (Python and Rust builders), same model as the file | M | 0.1 | done (the Rust builder, and the Python builder, which gives the same canonical form as the file: a test compares them) | +| SCN-01 | Scenario file in YAML with a JSON Schema, versioned | M | 0.1 | done (tests: [`schema.rs`](crates/netscenario-model/tests/schema.rs), [`corpus.rs`](crates/netscenario-model/tests/corpus.rs)) | +| SCN-02 | Validation with precise errors (line, column and source line for every error, the path in the file for errors of content, what was expected, how to fix it; reserved keys name their release) | M | 0.1 | done (tests: [`corpus.rs`](crates/netscenario-model/tests/corpus.rs)) | +| SCN-03 | Units and durations parsed strictly (`40ms`, `10mbit`, `2%`) | M | 0.1 | done (tests: [`roundtrip.rs`](crates/netscenario-model/tests/roundtrip.rs)) | +| SCN-04 | Scenarios built in code (Python and Rust builders), same model as the file | M | 0.1 | done (the Rust builder, and the Python builder, which gives the same canonical form as the file: a test compares them; tests: [`test_scenario.py`](python/netscenario/tests/test_scenario.py), [`roundtrip.rs`](crates/netscenario-model/tests/roundtrip.rs)) | | SCN-05 | Parameters, includes and templates (reuse a link, vary a value) | S | 0.3 | done (`params:` with `--param name=value`, exact `{expressions}`, `templates:` with `use:`, files with `include:`; the expanded scenario is what is hashed, recorded and replayed, and an error names the file, the line and each place of use, D-093: `crates/netscenario-model/tests/expand.rs`, `python/netscenario/tests/test_params.py`, the fuzz target `expand`, the examples [parameters](scenarios/parameters.yaml), [templates](scenarios/templates.yaml), [include](scenarios/include.yaml)) | -| SCN-06 | Profile library: 3G, LTE, 5G, Wi-Fi, DSL, GEO and LEO satellite, datacenter, lossy radio. Every value has a source ([PROFILES.md](docs/PROFILES.md)) | S | 0.3 | done (version 1: 14 profiles of 3G, LTE, 5G, DSL, GEO, LEO and datacenter, mostly delays; Wi-Fi, cable, lossy radio and the rest of the keys wait for sources, D-095) | +| SCN-06 | Profile library: 3G, LTE, 5G, Wi-Fi, DSL, GEO and LEO satellite, datacenter, lossy radio. Every value has a source ([PROFILES.md](docs/PROFILES.md)) | S | 0.3 | done (version 1: 14 profiles of 3G, LTE, 5G, DSL, GEO, LEO and datacenter, mostly delays; Wi-Fi, cable, lossy radio and the rest of the keys wait for sources, D-095; tests: [`profiles.rs`](crates/netscenario-model/tests/profiles.rs), [`profiles.rs`](crates/netscenario-backend-kernel/tests/profiles.rs)) | | SCN-07 | Matrices and sweeps (grids, random sampling) | S | 0.5 | planned | | SCN-08 | Migration between schema versions (`netscenario migrate`) | S | 0.7 | planned | | SCN-09 | Convert to and from `tc netem` command lines | C | 0.7 | idea | -| SCN-10 | Hints for misspelled names and keys ("did you mean 'server'?"); node names have them since 0.1, keys, units and values follow | S | 0.2 | done | +| SCN-10 | Hints for misspelled names and keys ("did you mean 'server'?"); node names have them since 0.1, keys, units and values follow | S | 0.2 | done (tests: [`corpus.rs`](crates/netscenario-model/tests/corpus.rs)) | ## Topology (TOP) @@ -81,11 +81,11 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | -| LMD-01 | Static model | M | 0.1 | done (a link keeps its values for the whole run; changes over time are TML-01, 0.2) | -| LMD-02 | Scheduled steps | M | 0.2 | done (format with D-050; the Kernel backend applies the steps by the clock or at a barrier of the workload, D-060, D-062, D-064) | +| LMD-01 | Static model | M | 0.1 | done (a link keeps its values for the whole run; changes over time are TML-01, 0.2; tests: [`provision.rs`](crates/netscenario-backend-kernel/tests/provision.rs), [`fidelity.rs`](crates/netscenario-backend-kernel/tests/fidelity.rs)) | +| LMD-02 | Scheduled steps | M | 0.2 | done (format with D-050; the Kernel backend applies the steps by the clock or at a barrier of the workload, D-060, D-062, D-064; tests: [`timeline.rs`](crates/netscenario-backend-kernel/tests/timeline.rs), [`timeline.rs`](crates/netscenario-model/tests/timeline.rs)) | | LMD-03 | Trace-driven rate, delay and loss (including import of Mahimahi traces) | M | 0.3 | done (`model: { kind: trace }` with a CSV or a Mahimahi file for each direction, D-091; a rate that changes is a class of `htb` that is changed, so a trace is followed to within 0.4 % at steps of 100 ms; closer steps need `allow-limited`, D-092: `crates/netscenario-backend-kernel/tests/traces.rs`, `crates/netscenario-model/tests/link_traces.rs`) | | LMD-04 | Markov models with states and transitions | S | 0.3 | done (D-090; `tests/models.rs` of the model and of the Kernel backend: the share of time against the stationary distribution, the same seed gives the same states and lost packets) | -| LMD-05 | Periodic patterns (a satellite pass, a duty cycle) | S | 0.3 | done (D-090; the period is exact, and the run does what the same events written by hand do) | +| LMD-05 | Periodic patterns (a satellite pass, a duty cycle) | S | 0.3 | done (D-090; the period is exact, and the run does what the same events written by hand do; tests: [`models.rs`](crates/netscenario-backend-kernel/tests/models.rs), [`models.rs`](crates/netscenario-model/tests/models.rs)) | | LMD-06 | Geometry-based: distance to loss, delay and rate (free space, log-distance) | M | 0.6 | planned | | LMD-07 | Plug-in interface for your own model | S | 0.7 | planned | @@ -93,9 +93,9 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | ------ | ---------------------------------------------------------- | ---- | ------- | ------- | -| TML-01 | Events at times: change parameters | M | 0.2 | done (format with D-050; the queues and the whole-state change, D-059; every step is recorded in `events`, D-060, and done at a barrier of the workload so that the same seed gives the same lost packets, D-062, by a `tc` that lives for the run, D-063; `outage`, `down` and `up` are steps too, D-064) | +| TML-01 | Events at times: change parameters | M | 0.2 | done (format with D-050; the queues and the whole-state change, D-059; every step is recorded in `events`, D-060, and done at a barrier of the workload so that the same seed gives the same lost packets, D-062, by a `tc` that lives for the run, D-063; `outage`, `down` and `up` are steps too, D-064; tests: [`timeline.rs`](crates/netscenario-backend-kernel/tests/timeline.rs), [`changes.rs`](crates/netscenario-backend-kernel/tests/changes.rs)) | | TML-02 | Link down and up | M | 0.2 | done (format with D-050; the Kernel backend since D-064; `tests/timeline.rs`: the sender is refused for exactly the packets of the window, the statistics leave the refused packets out) | -| TML-03 | Repeating events | S | 0.3 | done (`every`, `until`, `count`; D-090) | +| TML-03 | Repeating events | S | 0.3 | done (`every`, `until`, `count`; D-090; tests: [`models.rs`](crates/netscenario-backend-kernel/tests/models.rs), [`models.rs`](crates/netscenario-model/tests/models.rs)) | | TML-04 | Events tied to a milestone of a workload (after N packets) | C | 0.7 | idea | | TML-05 | Change a link from test code while the run is going | S | 0.3 | done (D-096; `crates/netscenario-backend-kernel/tests/live.rs`: the change lands between two packets, a replay loses the same packets; `python/netscenario/tests/test_live.py`) | @@ -115,7 +115,7 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | -| WRK-01 | UDP constant bit rate | M | 0.1 | done (`netscenario run`, tested in real namespaces; [AGENT.md](docs/AGENT.md)) | +| WRK-01 | UDP constant bit rate | M | 0.1 | done (`netscenario run`, tested in real namespaces; [AGENT.md](docs/AGENT.md); tests: [`run.rs`](crates/netscenario-backend-kernel/tests/run.rs), [`agent.rs`](crates/netscenario-backend-kernel/tests/agent.rs)) | | WRK-02 | UDP Poisson, on/off and bursty | M | 0.2 | done (`udp-poisson`, `udp-onoff`, `udp-bursty`; a seeded schedule from a generator written out in the agent, pinned against an independent implementation; unit tests of the distributions with derived bounds, [patterns.rs](crates/netscenario-backend-kernel/tests/patterns.rs) in real namespaces; [AGENT.md](docs/AGENT.md), [SCENARIO-FORMAT.md](docs/SCENARIO-FORMAT.md), D-071) | | WRK-03 | Replay of a trace of packet sizes and times | S | 0.3 | done (`trace-replay` with a CSV file or a capture, D-085: `crates/netscenario-backend-kernel/tests/trace.rs`, `crates/netscenario-model/tests/trace.rs`; the packets are expanded into the scenario, D-083) | | WRK-04 | TCP bulk transfer | M | 0.3 | done (`tcp-bulk` with `bytes`, `connections` and `cc`, the goodput by bin, the completion of each transfer, the retransmissions and the round trip time of the kernel in the `tcp` entry of the result; [tcp.rs](crates/netscenario-backend-kernel/tests/tcp.rs), [RESULTS.md](docs/RESULTS.md), D-086) | @@ -124,21 +124,21 @@ A line moves to `in progress` when the first code of the feature exists, and to | WRK-07 | A container as a node | S | 0.5 | planned | | WRK-08 | Replay of a pcap | C | 0.8 | idea | | WRK-09 | Models of real traffic (video, voice, game) | S | 0.7 | planned | -| WRK-10 | Supervision: start order, readiness checks, orderly stop, captured logs, timeouts | M | 0.2 | done (order of the file, five ways of being ready with a limit, `TERM` then `KILL` to the process group, logs of the first megabyte with the dropped bytes counted, `on-exit`; D-072) | +| WRK-10 | Supervision: start order, readiness checks, orderly stop, captured logs, timeouts | M | 0.2 | done (order of the file, five ways of being ready with a limit, `TERM` then `KILL` to the process group, logs of the first megabyte with the dropped bytes counted, `on-exit`; D-072; tests: [`command.rs`](crates/netscenario-backend-kernel/tests/command.rs)) | | WRK-11 | UDP without the checksum, so that a corrupted payload arrives and is counted (`udp-checksum: false`) | S | 0.2 | done (D-056: raw socket, wire format 2 with a CRC-32, `corrupted` in the result; `tests/fidelity.rs` cases 19 to 21 and `tests/agent.rs`) | ## Measurement (PRB) | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | -| PRB-01 | Per-packet records (sequence, send time, receive time) | M | 0.1 | done (record file, JSON Lines export; [RESULTS.md](docs/RESULTS.md)) | +| PRB-01 | Per-packet records (sequence, send time, receive time) | M | 0.1 | done (record file, JSON Lines export; [RESULTS.md](docs/RESULTS.md); tests: [`run.rs`](crates/netscenario-backend-kernel/tests/run.rs)) | | PRB-02 | Loss and delivery ratio | M | 0.1 | done (`tests/run.rs`, canaries in `stats.rs`) | | PRB-03 | One-way delay with percentiles | M | 0.1 | done (one-way delay with nearest-rank percentiles, `stats.rs` and `tests/run.rs`); the round trip time is PRB-12 | | PRB-04 | Jitter (as in RFC 3550) and reordering metrics | S | 0.2 | done (`patterns.rs`: the extent and late time of RFC 4737 against all its worked examples, the RFC 3550 estimate against hand-worked series; modelled and measured in `tests/fidelity.rs`; [RESULTS.md](docs/RESULTS.md), D-058) | -| PRB-05 | Throughput and goodput over time | M | 0.1 | done (payload per second; for UDP throughput and goodput are the same number; the series by time bin of PRB-07 adds sent, lost and the delays for each bin) | -| PRB-06 | Burst statistics (length of loss runs) | S | 0.2 | done (`loss_runs` in the result: runs, longest, mean, histogram; D-058) | +| PRB-05 | Throughput and goodput over time | M | 0.1 | done (payload per second; for UDP throughput and goodput are the same number; the series by time bin of PRB-07 adds sent, lost and the delays for each bin; tests: [`run.rs`](crates/netscenario-backend-kernel/tests/run.rs)) | +| PRB-06 | Burst statistics (length of loss runs) | S | 0.2 | done (`loss_runs` in the result: runs, longest, mean, histogram; D-058; tests: [`fidelity.rs`](crates/netscenario-backend-kernel/tests/fidelity.rs), [`test_results.py`](python/netscenario/tests/test_results.py)) | | PRB-07 | Time to recover after an event | M | 0.2 | done (a series by time bin for each workload in `series-.jsonl`, sent and arrived side by side, and `recoveries` for each event of the timeline with explicit parameters; unit tests on series with known answers and canaries in `recovery.rs` and `series.rs`, a real run with a drop in the rate and an outage in [timeline.rs](crates/netscenario-backend-kernel/tests/timeline.rs); [RESULTS.md](docs/RESULTS.md), D-070); and for TCP workloads on the goodput of a bin, only while the workload is working: [tcp.rs](crates/netscenario-backend-kernel/tests/tcp.rs), D-088) | -| PRB-08 | Metrics from your program (JSON on stdout, exit code) | M | 0.2 | done (`{"metric", "value", "unit"}` lines with the time they were read, a summary and a file of readings, invalid lines counted and shown, the exit code and signal in the result; D-072) | +| PRB-08 | Metrics from your program (JSON on stdout, exit code) | M | 0.2 | done (`{"metric", "value", "unit"}` lines with the time they were read, a summary and a file of readings, invalid lines counted and shown, the exit code and signal in the result; D-072; tests: [`command.rs`](crates/netscenario-backend-kernel/tests/command.rs), [`test_echo.py`](examples/echo-server/test_echo.py)) | | PRB-09 | Packet capture per link | M | 0.2 | done (`measure.capture`: both interfaces of a link as `capture--.pcap`, nanosecond timestamps, checked against the counter of the interface; [capture.rs](crates/netscenario-backend-kernel/tests/capture.rs): read by `tcpdump`, lost packets absent and duplicates present, timestamps against the records, a capture cut short refused, a receive buffer that overflows reported; D-047, D-073) | | PRB-10 | Queue occupancy and drops from the kernel | S | 0.3 | done (`queues` in the result: the counters at the end and a sample each second, D-082; `crates/netscenario-backend-kernel/tests/queues.rs`) | | PRB-11 | CPU and memory of the program under test | C | 0.8 | idea | @@ -148,7 +148,7 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | -| ASR-01 | Threshold assertions on metrics | M | 0.1 | done (plain `assert` on the numbers of the result in test code: [PYTEST.md](docs/PYTEST.md), [examples/python-test/](examples/python-test/); the `expect` section is ASR-07) | +| ASR-01 | Threshold assertions on metrics | M | 0.1 | done (plain `assert` on the numbers of the result in test code: [PYTEST.md](docs/PYTEST.md), [examples/python-test/](examples/python-test/); the `expect` section is ASR-07; tests: [`test_plugin.py`](python/pytest-netscenario/tests/test_plugin.py)) | | ASR-02 | Repetitions with derived seeds | M | 0.2 | done (`run --repetitions`, `run(..., repetitions=n)`, the marker and `netscenario_runs`: [STATISTICS.md](docs/STATISTICS.md), D-074; tests in [`test_runset.py`](python/netscenario/tests/test_runset.py) and [`binary.rs`](crates/netscenario-cli/tests/binary.rs)) | | ASR-03 | Confidence intervals (bootstrap) | M | 0.2 | done (expanded percentile for the mean, percentile for the median, coverage measured: [STATISTICS.md](docs/STATISTICS.md), D-074; tests in [`bootstrap.rs`](crates/netscenario-core/src/bootstrap.rs), the stability of the claim in [ACCEPTANCE.md](docs/ACCEPTANCE.md)) | | ASR-04 | Baselines and regression detection (effect size, interval, "erratic" marking) | M | 0.7 | planned | @@ -162,7 +162,7 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | | BKD-01 | Kernel backend: namespaces, veth and `netem` as root | M | 0.1 | done (the CI runs the Kernel backend a second time as root, and as an ordinary user with the two ambient capabilities: `tests/rights.rs`, D-035) | -| BKD-02 | Kernel backend in an unprivileged user namespace | M | 0.1 | done (the CI runs the tests as an ordinary user on `ubuntu-22.04` and `ubuntu-24.04`) | +| BKD-02 | Kernel backend in an unprivileged user namespace | M | 0.1 | done (the CI runs the tests as an ordinary user on `ubuntu-22.04` and `ubuntu-24.04`; tests: [`doctor.rs`](crates/netscenario-backend-kernel/tests/doctor.rs), [`run.rs`](crates/netscenario-backend-kernel/tests/run.rs)) | | BKD-03 | Every backend declares what it reproduces faithfully, and refuses the rest with an error | M | 0.1 | done (`tests/refusal.rs`, `tests/coverage.rs`, `capability.rs`; D-023; a restriction that a scenario accepts with `allow-limited` is recorded in the result, D-092) | | BKD-04 | Cleanup guarantee: no leftovers after a crash (PID namespace, D-022), and `netscenario clean` as a safety net | M | 0.1 | done (the guarantee is tested at 18 steps in `tests/crash.rs`; `clean` in `tests/clean.rs`, D-024) | | BKD-05 | Netlink instead of calls to `ip` and `tc` | S | 0.4 | planned | @@ -180,11 +180,11 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | | ENV-01 | Seeds: one per scenario, derived for each component | M | 0.1 | done (derivation pinned, drawing and recording in the result, `tests/run.rs`; [RESULTS.md](docs/RESULTS.md)) | -| ENV-02 | Provenance in every result (versions, kernel, `tc`, CPU, backend, seeds, scenario hash) | M | 0.1 | done | +| ENV-02 | Provenance in every result (versions, kernel, `tc`, CPU, backend, seeds, scenario hash) | M | 0.1 | done (tests: [`run.rs`](crates/netscenario-backend-kernel/tests/run.rs), [`machine.rs`](crates/netscenario-core/src/machine.rs)) | | ENV-03 | Replay a run from its result file | S | 0.2 | done (`netscenario replay `: [CLI.md](docs/CLI.md), D-075; tests in [`replay.rs`](crates/netscenario-core/src/replay.rs) and [`binary.rs`](crates/netscenario-cli/tests/binary.rs)) | | ENV-04 | `netscenario doctor`: what this machine can do, and what it cannot | M | 0.1 | done for the Kernel backend ([DOCTOR.md](docs/DOCTOR.md), `tests/doctor.rs`); the probe of other backends comes with them | | ENV-05 | Report on timer and clock precision | S | 0.3 | done (`provenance.timers` of every result, `netscenario doctor --timers`, `ns.doctor(timers=True)`: [DOCTOR.md](docs/DOCTOR.md), D-097; tests in [`timers.rs`](crates/netscenario-core/src/timers.rs) and [`timers.rs`](crates/netscenario-backend-kernel/src/timers.rs)) | -| ENV-06 | Result format with a version | M | 0.1 | done (`format` and `version`, 0 until 1.0) | +| ENV-06 | Result format with a version | M | 0.1 | done (`format` and `version`, 0 until 1.0; tests: [`test_results.py`](python/netscenario/tests/test_results.py), [`run.rs`](crates/netscenario-backend-kernel/tests/run.rs)) | | ENV-07 | Reproducibility check: the run compares the packet counter of each queue with the packets of the workload, and marks a run in which something else entered the queue as not reproducible; `doctor` checks that `tc` knows `seed` | M | 0.1 | done (`doctor` checks that `tc` knows `seed`; a run counts the packets of the queue of each direction with loss and compares them with the packets the workloads sent: `check_queues` in `run.rs`, `tests/run.rs` with a canary that sends one extra packet, D-036; two workloads in one direction whose queue draws are said not to be reproducible, D-065, because the count cannot see the order of their packets) | ## Safety (SAF) @@ -200,25 +200,25 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | | CLI-01 | `run`, `validate`, `doctor`, `clean` | M | 0.1 | done ([CLI.md](docs/CLI.md), `tests/binary.rs`) | -| CLI-02 | `report`: HTML and Markdown from result files | M | 0.2 | done (`netscenario report --format md\|html\|junit`: [REPORTS.md](docs/REPORTS.md), D-076) | -| CLI-03 | `profiles` and `capabilities`: what exists, what a backend can do | S | 0.3 | done | +| CLI-02 | `report`: HTML and Markdown from result files | M | 0.2 | done (`netscenario report --format md\|html\|junit`: [REPORTS.md](docs/REPORTS.md), D-076; tests: [`binary.rs`](crates/netscenario-cli/tests/binary.rs), [`tests.rs`](crates/netscenario-core/src/report/tests.rs), [`test_reports.py`](python/netscenario/tests/test_reports.py)) | +| CLI-03 | `profiles` and `capabilities`: what exists, what a backend can do | S | 0.3 | done (tests: [`profiles_cmd.rs`](crates/netscenario-cli/src/profiles_cmd.rs)) | | CLI-07 | `expand` (the long form of a scenario that uses shortcuts, parameters or profiles) and `show` (the graph, the addresses, the effective routes: text and Graphviz) | S | 0.3 | in progress (`expand` and `show` of the topology, with the shortcuts, are in [CLI.md](docs/CLI.md); the addresses join `show` when the Kernel backend builds networks; parameters and profiles when they exist) | -| CLI-04 | `schema`: print the JSON Schema | S | 0.1 | done (`netscenario schema`) | +| CLI-04 | `schema`: print the JSON Schema | S | 0.1 | done (`netscenario schema`; tests: [`schema.rs`](crates/netscenario-model/tests/schema.rs), [`lib.rs`](crates/netscenario-cli/src/lib.rs)) | | CLI-05 | Shell completions and man pages | S | 0.8 | planned | -| CLI-06 | Machine-readable output and documented exit codes | M | 0.1 | done (exit codes in [CLI.md](docs/CLI.md), 130 and 143 for Ctrl-C and `kill` with the result of the part that ran, D-066; `--json` on `run`, `validate`, `doctor`, `clean`; the code 1 comes with assertions) | +| CLI-06 | Machine-readable output and documented exit codes | M | 0.1 | done (exit codes in [CLI.md](docs/CLI.md), 130 and 143 for Ctrl-C and `kill` with the result of the part that ran, D-066; `--json` on `run`, `validate`, `doctor`, `clean`; the code 1 comes with assertions; tests: [`binary.rs`](crates/netscenario-cli/tests/binary.rs)) | ## Python (PYT) | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | | PYT-01 | Python package with bindings and API (builder, run, results) | M | 0.1 | done ([PYTHON.md](docs/PYTHON.md); `python/netscenario/tests`, wheels: PYT-08); since 0.2 also repetitions and sets of runs, `report`, `replay` and `junit_properties` (D-077) | -| PYT-02 | `pytest` plugin: fixtures, markers, options | M | 0.1 | done (fixture `netscenario`, marker and `netscenario_result`, the guard, report section and JUnit properties; [PYTEST.md](docs/PYTEST.md), tested with `pytester` in real runs) | -| PYT-03 | Typed API (`py.typed`, checked with a strict type checker) | M | 0.1 | done (`pyright` strict over the package and its tests, part of `just check`) | +| PYT-02 | `pytest` plugin: fixtures, markers, options | M | 0.1 | done (fixture `netscenario`, marker and `netscenario_result`, the guard, report section and JUnit properties; [PYTEST.md](docs/PYTEST.md), tested with `pytester` in real runs; tests: [`test_plugin.py`](python/pytest-netscenario/tests/test_plugin.py)) | +| PYT-03 | Typed API (`py.typed`, checked with a strict type checker) | M | 0.1 | done (`pyright` strict over the package and its tests, part of `just check`; see: [`pyproject.toml`](pyproject.toml)) | | PYT-04 | `asyncio` friendly API | S | 0.3 | done (`run_async`, `start_async`, a cancelled task cancels the run and no thread is left; D-096; `python/netscenario/tests/test_live.py`) | | PYT-05 | Per-packet data as pandas and polars frames | S | 0.3 | done (`Workload.packets_table`, `Series.table`, `Result.steps_table`, `Table.to_pandas`/`to_polars` as the extras `netscenario[pandas]` and `[polars]`; `python/netscenario/tests/test_frames.py`, also in the minimum environment without them) | | PYT-06 | Notebook helpers for plots | C | 0.8 | idea | | PYT-07 | Safe with `pytest-xdist` (parallel runs) | M | 0.5 | planned | -| PYT-08 | Wheels for manylinux on x86_64 and aarch64 | M | 0.1 | done (built and installed into clean environments by every CI run, and published to PyPI: REL-01) | +| PYT-08 | Wheels for manylinux on x86_64 and aarch64 | M | 0.1 | done (built and installed into clean environments by every CI run, and published to PyPI: REL-01; see: [`wheels.yml`](.github/workflows/wheels.yml)) | ## Rust API (RST) @@ -231,10 +231,10 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | -| RPT-01 | Results as JSON, per-packet data as JSON Lines | M | 0.1 | done (`result.json`, `--packets`; [RESULTS.md](docs/RESULTS.md)) | +| RPT-01 | Results as JSON, per-packet data as JSON Lines | M | 0.1 | done (`result.json`, `--packets`; [RESULTS.md](docs/RESULTS.md); tests: [`run.rs`](crates/netscenario-backend-kernel/tests/run.rs), [`test_results.py`](python/netscenario/tests/test_results.py)) | | RPT-02 | JUnit XML | M | 0.2 | done ([REPORTS.md](docs/REPORTS.md#junit---format-junit-rpt-02); [`tests.rs`](crates/netscenario-core/src/report/tests.rs)) | -| RPT-03 | HTML report with time series and distributions | M | 0.2 | done (one file, inline SVG, no external reference: [REPORTS.md](docs/REPORTS.md#html---format-html-rpt-03), [`sample-report/`](docs/sample-report/)) | -| RPT-04 | Markdown for the summary of a CI run | S | 0.2 | done ([REPORTS.md](docs/REPORTS.md#in-a-ci-job)) | +| RPT-03 | HTML report with time series and distributions | M | 0.2 | done (one file, inline SVG, no external reference: [REPORTS.md](docs/REPORTS.md#html---format-html-rpt-03), [`sample-report/`](docs/sample-report/); tests: [`tests.rs`](crates/netscenario-core/src/report/tests.rs)) | +| RPT-04 | Markdown for the summary of a CI run | S | 0.2 | done ([REPORTS.md](docs/REPORTS.md#in-a-ci-job); tests: [`tests.rs`](crates/netscenario-core/src/report/tests.rs)) | | RPT-05 | Comparison of two runs or two backends | S | 0.7 | planned | | RPT-06 | OpenMetrics export | C | 0.8 | idea | | RPT-07 | Trends over many runs | C | 1.x | idea | @@ -252,14 +252,14 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | -| QA-01 | Unit tests and property tests | M | 0.1 | done for 0.1 (unit tests in every crate, property tests of the units in `netscenario-model`; it grows with every release) | -| QA-02 | Integration tests inside namespaces, on the CI | M | 0.1 | done (the CI creates real namespaces on `ubuntu-22.04` and `ubuntu-24.04`, as a user and as root) | -| QA-03 | Fuzzing of the parsers that read input that is not trusted: the scenario, the packets and records of the agent, the metrics of a program | S | 0.2 | done (eight targets and the weekly workflow `fuzz.yml`: the scenario, the values, a datagram, a file of records, a result file, the metrics of a program, the file of a set of runs and the reports made of a result file; [fuzz/](fuzz/), D-045, D-053; the replay reads the same two files and the scenario, which have targets; brought forward from 0.3, because release 0.2 adds parsers of such input; ClusterFuzzLite and a 24-hour run stay in 0.8) | +| QA-01 | Unit tests and property tests | M | 0.1 | done for 0.1 (unit tests in every crate, property tests of the units in `netscenario-model`; it grows with every release) (tests: [`roundtrip.rs`](crates/netscenario-model/tests/roundtrip.rs)) | +| QA-02 | Integration tests inside namespaces, on the CI | M | 0.1 | done (the CI creates real namespaces on `ubuntu-22.04` and `ubuntu-24.04`, as a user and as root; tests: [`provision.rs`](crates/netscenario-backend-kernel/tests/provision.rs), [`ci.yml`](.github/workflows/ci.yml)) | +| QA-03 | Fuzzing of the parsers that read input that is not trusted: the scenario, the packets and records of the agent, the metrics of a program | S | 0.2 | done (eight targets and the weekly workflow `fuzz.yml`: the scenario, the values, a datagram, a file of records, a result file, the metrics of a program, the file of a set of runs and the reports made of a result file; [fuzz/](fuzz/), D-045, D-053; the replay reads the same two files and the scenario, which have targets; brought forward from 0.3, because release 0.2 adds parsers of such input; ClusterFuzzLite and a 24-hour run stay in 0.8; tests: [`fuzz_targets/`](fuzz/fuzz_targets/), [`corpus.rs`](crates/netscenario-model/tests/corpus.rs)) | | QA-04 | Snapshot tests for reports and the schema | S | 0.2 | done (the Markdown, the JUnit and each drawing, and a sample report that a test makes again: [`tests.rs`](crates/netscenario-core/src/report/tests.rs); the schema: [`schema.rs`](crates/netscenario-model/tests/schema.rs)) | | QA-05 | Benchmarks of the engine (throughput, added delay) | S | 0.4 | planned | | QA-06 | Fidelity tests: measured against configured delay, loss and rate, with published tolerances | M | 0.1 | done ([FIDELITY.md](docs/FIDELITY.md), `tests/fidelity.rs`: delay, loss, rate, both jitters, jitter with a rate; three canaries; CI numbers per runner as artifacts) | -| QA-07 | Supported Rust version policy, checked on the CI, and checks for breaking changes of the API | M | 0.1 | done (the Rust version is checked on the CI with 1.89; `cargo-semver-checks` compares the public API of `netscenario-model` and `netscenario-core` with `main` in every pull request and with the last release in a release, D-037) | -| QA-08 | Supply chain: dependency audit and policy, OpenSSF Scorecard, actions pinned to commits | M | 0.1 | done (`cargo deny` in every pull request and weekly, OpenSSF Scorecard, CodeQL, pinned actions, Dependabot) | +| QA-07 | Supported Rust version policy, checked on the CI, and checks for breaking changes of the API | M | 0.1 | done (the Rust version is checked on the CI with 1.89; `cargo-semver-checks` compares the public API of `netscenario-model` and `netscenario-core` with `main` in every pull request and with the last release in a release, D-037; see: [`ci.yml`](.github/workflows/ci.yml)) | +| QA-08 | Supply chain: dependency audit and policy, OpenSSF Scorecard, actions pinned to commits | M | 0.1 | done (`cargo deny` in every pull request and weekly, OpenSSF Scorecard, CodeQL, pinned actions, Dependabot; see: [`deny.toml`](deny.toml), [`security.yml`](.github/workflows/security.yml)) | | QA-09 | Coverage figures (for information) | C | - | idea | | QA-10 | Tests on several kernels and distributions | S | 0.5 | planned | @@ -270,7 +270,7 @@ A line moves to `in progress` when the first code of the feature exists, and to | DOC-01 | README, architecture, decisions, scenario format, coverage | M | 0.1 | done for 0.1 ([README.md](README.md), [examples/](examples/); later releases add to it) | | DOC-02 | User guide with tutorials | M | 0.3 | done (mdBook in [docs/book/](docs/book/), three paths and a chapter for each shape; the scenarios, the Python, the commands and the links of every page are checked by [`test_book.py`](python/netscenario/tests/test_book.py), published to GitHub Pages by [`book.yml`](.github/workflows/book.yml)) | | DOC-03 | Cookbook: QUIC, WebRTC, games, IoT, satellite | S | 0.7 | planned | -| DOC-04 | API documentation for Rust and Python | M | 0.1 | done (`missing_docs` and ruff `D` are errors in the CI; a runnable example in `netscenario-model` and `netscenario-core`) | +| DOC-04 | API documentation for Rust and Python | M | 0.1 | done (`missing_docs` and ruff `D` are errors in the CI; a runnable example in `netscenario-model` and `netscenario-core`; see: [`pyproject.toml`](pyproject.toml)) | | DOC-05 | Troubleshooting (permissions, kernels, containers) | M | 0.1 | done ([docs/TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md)) | | DOC-06 | Chapter on fidelity and limits, generated from the coverage matrix | M | 0.4 | planned | | DOC-07 | Validation report: how well the emulation matches what it claims | C | 0.9 | idea | @@ -280,8 +280,8 @@ A line moves to `in progress` when the first code of the feature exists, and to | ID | Feature | Prio | Release | Status | | --- | --- | --- | --- | --- | | REL-01 | Release by tag to crates.io and PyPI with provenance and a staged approval | M | 0.1 | done (`release.yml`: the checks, the builds, trusted publishing to crates.io and PyPI with attestations after the approval of a maintainer; `0.1.0-rc.1` and `0.1.0-rc.2` went through it, [RELEASING.md](docs/RELEASING.md), D-034) | -| REL-02 | Automatic changelog from the history | M | 0.1 | done | -| REL-03 | Compatibility policy for the scenario format, the result format, the APIs | M | 0.1 | done | +| REL-02 | Automatic changelog from the history | M | 0.1 | done (see: [`cliff.toml`](cliff.toml)) | +| REL-03 | Compatibility policy for the scenario format, the result format, the APIs | M | 0.1 | done (see: [`VERSIONING.md`](docs/VERSIONING.md)) | | REL-04 | Review list for 1.0 | M | 0.9 | planned | ## Integration with other projects (INT) diff --git a/crates/netscenario-backend-kernel/tests/coverage.rs b/crates/netscenario-backend-kernel/tests/coverage.rs index 87f50a9..2838a34 100644 --- a/crates/netscenario-backend-kernel/tests/coverage.rs +++ b/crates/netscenario-backend-kernel/tests/coverage.rs @@ -118,3 +118,137 @@ fn the_reader_of_links_finds_a_missing_link_and_a_missing_file() { let docs = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../docs"); assert!(!docs.join("../crates/no-such-crate/tests/a.rs").is_file()); } + +/// One row of a table of the page: the section it is in, its label and its three cells (Kernel, Engine, Proxy). +struct Row { + section: String, + label: String, + cells: Vec, +} + +fn rows_of(markdown: &str) -> Vec { + let mut section = String::new(); + let mut rows = Vec::new(); + for line in markdown.lines() { + if let Some(title) = line.strip_prefix("## ") { + title.trim().clone_into(&mut section); + } + let Some(inner) = line.strip_prefix('|') else { + continue; + }; + let cells: Vec = inner + .trim_end_matches('|') + .split('|') + .map(|cell| cell.trim().to_owned()) + .collect(); + let separator = cells + .iter() + .all(|c| c.chars().all(|ch| matches!(ch, '-' | ':' | ' '))); + if cells.len() >= 4 && !separator && cells[1] != "Kernel" { + rows.push(Row { + section: section.clone(), + label: cells[0].clone(), + cells: cells[1..4].to_vec(), + }); + } + } + rows +} + +/// The release of a faithful mark (`✔ 0.2 (...)` is `Some(2)`), or `None` for any other cell. +fn faithful_release(cell: &str) -> Option { + let rest = cell.strip_prefix("✔ 0.")?; + rest.split(|c: char| !c.is_ascii_digit()) + .next()? + .parse() + .ok() +} + +/// Every link of a cell: `[text](target)` gives `target`. +fn links_in(cell: &str) -> Vec<&str> { + cell.split("](") + .skip(1) + .filter_map(|part| part.split(')').next()) + .collect() +} + +fn is_a_test(target: &str) -> bool { + let path = target.split('#').next().unwrap_or(target); + path.split('/').any(|part| part == "tests") + || path + .rsplit('/') + .next() + .is_some_and(|name| name.starts_with("test_")) +} + +/// A faithful mark of a release that is out is proven: its Kernel cell links to a test, for every row of the page and not only for +/// the rows that the code declares (`Feature`). A mark of a later release is a plan (the page says so) and has no link yet; the +/// release pull request raises the version of the crates, and from then on the marks of that release need theirs. Links in any +/// cell of any column must lead to a file. +#[test] +fn every_faithful_mark_of_a_released_version_links_to_a_test_and_no_link_leads_nowhere() { + let docs = Path::new(env!("CARGO_MANIFEST_DIR")).join("../../docs"); + let minor: u32 = env!("CARGO_PKG_VERSION") + .split('.') + .nth(1) + .unwrap() + .parse() + .unwrap(); + let rows = rows_of(COVERAGE); + assert!( + rows.len() > 50, + "the table was not read: {} rows", + rows.len() + ); + for row in &rows { + for cell in &row.cells { + for target in links_in(cell) { + let path = target.split('#').next().unwrap(); + if !path.starts_with("http") && !path.is_empty() { + assert!( + docs.join(path).exists(), + "'{}' links to {target}, which is not there", + row.label + ); + } + } + } + let kernel = &row.cells[0]; + if faithful_release(kernel).is_some_and(|release| release <= minor) { + let links = links_in(kernel); + assert!( + !links.is_empty(), + "'{}' is faithful in release 0.{minor} and has no link: {kernel}", + row.label + ); + // Where a run works, the proof is the CI; for a feature it is a test + if row.section != "Where it can run" { + assert!( + links.iter().any(|link| is_a_test(link)), + "'{}' links to no test: {kernel}", + row.label + ); + } + } + } +} + +/// Canary: the reader of the page and the rules find what they are for. +#[test] +fn the_rules_for_the_marks_find_a_plan_a_proof_and_a_missing_test() { + let page = "## Topology\n| Feature | Kernel | Engine | Proxy |\n| --- | --- | --- | --- |\n| A | ✔ 0.1 ([t](../crates/x/tests/a.rs)) | ✔ 0.4 | ✘ |\n| B | ✔ 0.6 | ✔ 0.6 | ✘ |\n## Where it can run\n| Place | Kernel | Engine | Proxy |\n| P | ✔ 0.1 ([ci](../.github/workflows/ci.yml)) | · | · |\n"; + let rows = rows_of(page); + assert_eq!(rows.len(), 3); + assert_eq!(rows[0].section, "Topology"); + assert_eq!(rows[2].section, "Where it can run"); + assert_eq!(faithful_release(&rows[0].cells[0]), Some(1)); + assert_eq!(faithful_release(&rows[1].cells[0]), Some(6)); + assert_eq!(faithful_release("~ 0.2"), None); + assert_eq!(faithful_release("✘"), None); + assert_eq!(links_in(&rows[0].cells[0]), ["../crates/x/tests/a.rs"]); + assert_eq!(links_in(&rows[1].cells[0]), Vec::<&str>::new()); + assert!(is_a_test("../crates/x/tests/a.rs#L3")); + assert!(is_a_test("../python/netscenario/tests/test_a.py")); + assert!(!is_a_test("decisions/D-083.md")); + assert!(!is_a_test("../.github/workflows/ci.yml")); +} diff --git a/crates/netscenario-core/src/backend.rs b/crates/netscenario-core/src/backend.rs index 667612e..424de41 100644 --- a/crates/netscenario-core/src/backend.rs +++ b/crates/netscenario-core/src/backend.rs @@ -672,7 +672,7 @@ pub trait Backend { /// [`Error::Backend`] if the link does not exist, or the backend has no interfaces to name (the default). fn interface_name(&self, _link: LinkHandle, _end: usize) -> Result { Err(Error::Backend { - backend: "this".to_owned(), + backend: self.name().to_owned(), action: "name an interface".to_owned(), message: "the backend cannot do it".to_owned(), }) @@ -706,7 +706,7 @@ pub trait Backend { _signal: ProgramSignal, ) -> Result<(), Error> { Err(Error::Backend { - backend: "this".to_owned(), + backend: self.name().to_owned(), action: "signal a program".to_owned(), message: "the backend cannot do it".to_owned(), }) @@ -726,7 +726,7 @@ pub trait Backend { _port: u16, ) -> Result { Err(Error::Backend { - backend: "this".to_owned(), + backend: self.name().to_owned(), action: "look at the ports of a node".to_owned(), message: "the backend cannot do it".to_owned(), }) @@ -748,3 +748,178 @@ pub trait Backend { /// [`Error::Backend`] if something could not be removed. The message says what is left. fn teardown(&mut self) -> Result<(), Error>; } + +#[cfg(test)] +mod tests { + use super::*; + + /// A backend that does only what every backend must, and leaves the rest to the defaults of the trait. What a user gets from + /// a backend that lacks a feature is what these defaults say, so each is checked: a default that says "yes" or says nothing + /// would let a scenario run on a backend that cannot reproduce it. + struct Bare; + + impl Backend for Bare { + fn name(&self) -> &'static str { + "bare" + } + fn capabilities(&self) -> Capabilities { + Capabilities::new("bare") + } + fn create_node(&mut self, _name: &Name) -> Result { + Ok(NodeHandle(0)) + } + fn create_link( + &mut self, + _name: &Name, + _between: [NodeHandle; 2], + ) -> Result { + Ok(LinkHandle(0)) + } + fn apply( + &mut self, + _link: LinkHandle, + _direction: Direction, + _impairments: &Impairments, + _seed: Option, + ) -> Result<(), Error> { + Ok(()) + } + fn start_program( + &mut self, + _node: NodeHandle, + _program: &Program, + ) -> Result { + Ok(ProgramHandle(0)) + } + fn address(&self, _link: LinkHandle, _end: usize) -> Result { + Ok(Ipv4Addr::LOCALHOST) + } + fn describe(&self) -> BackendInfo { + BackendInfo { + name: "bare".to_owned(), + details: BTreeMap::new(), + } + } + fn poll_program( + &mut self, + _program: ProgramHandle, + _wait: Duration, + ) -> Result { + Ok(ProgramState::default()) + } + fn send_line(&mut self, _program: ProgramHandle, _line: &str) -> Result<(), Error> { + Ok(()) + } + fn teardown(&mut self) -> Result<(), Error> { + Ok(()) + } + } + + const LINK: LinkHandle = LinkHandle(1); + const NODE: NodeHandle = NodeHandle(1); + + /// The three parts of a refusal, or a panic with what the call gave instead. + fn refusal(result: Result) -> (String, String, String) { + match result { + Err(Error::Backend { + backend, + action, + message, + }) => (backend, action, message), + other => panic!("a backend without the feature must refuse, and it gave {other:?}"), + } + } + + /// Every default that refuses says which backend refuses, what it was asked and why: the message is what the user reads + /// ("the bare backend failed to ..."). Canary: three of them said "this" for the backend, which no reader can place. + #[test] + fn a_backend_without_a_feature_refuses_and_names_itself_and_the_action() { + let mut bare = Bare; + let route = NodeRoute { + family: IpFamily::V4, + destination: None, + metric: 100, + next_hops: Vec::new(), + }; + let name = Name::new("a").unwrap(); + let refusals = [ + refusal(bare.change(LINK, Direction::Forward, &Impairments::default())), + refusal(bare.set_outage(LINK, Direction::Forward, true)), + refusal(bare.set_link(LINK, false)), + refusal(bare.set_ip_families(true, true)), + refusal(bare.set_node_address(NODE, IpAddr::V4(Ipv4Addr::LOCALHOST))), + refusal(bare.create_link_through_fabric(&name, [NODE, NODE])), + refusal(bare.set_link_mtu(LINK, 1400)), + refusal(bare.set_forwarding(NODE, true)), + refusal(bare.add_route(NODE, &route)), + refusal(bare.address_v6(LINK, 0)), + refusal(bare.interface_name(LINK, 0)), + refusal(bare.signal_program(ProgramHandle(0), ProgramSignal::Term)), + refusal(bare.port_bound(NODE, PortKind::Udp, 4433)), + ]; + let mut actions = std::collections::BTreeSet::new(); + for (backend, action, message) in refusals { + assert_eq!(backend, "bare", "{action}: {message}"); + assert!(!action.is_empty() && !message.is_empty()); + actions.insert(action); + } + assert_eq!( + actions.len(), + 13, + "each refusal says what it was asked: {actions:?}" + ); + // What the user reads + let (backend, action, message) = refusal(bare.set_link(LINK, false)); + let error = Error::Backend { + backend, + action, + message, + }; + assert_eq!( + error.to_string(), + "the bare backend failed to take a link down or bring it up: this backend does not take links down" + ); + } + + /// The defaults that have nothing to say are neutral: they accept, or answer that the backend cannot count, and a run then + /// leaves the comparison out instead of failing on it. + #[test] + fn a_backend_that_counts_nothing_says_so_and_accepts_the_steps_that_need_no_work() { + let mut bare = Bare; + let scenario = Scenario::from_yaml( + "version: 0\nname: bare\nduration: 1s\ntopology:\n nodes: [a, b]\n links:\n - { id: l, between: [a, b] }\nworkloads:\n - { id: w, kind: udp-cbr, from: a, to: b, rate: 1mbit, packet-size: 1000b }\n", + "n.yaml", + ) + .unwrap(); + assert!(bare.check_values(&scenario).is_ok()); + assert!(bare.plan_states(LINK, Direction::Forward, &[]).is_ok()); + assert!(bare.plan_outage(LINK, Direction::Reverse).is_ok()); + assert_eq!(bare.notes(), Vec::::new()); + assert_eq!(bare.queue_packets(LINK, Direction::Forward).unwrap(), None); + assert_eq!(bare.queue_backlog(LINK, Direction::Forward).unwrap(), None); + assert!( + bare.node_counts(LINK, Direction::Forward) + .unwrap() + .is_none() + ); + assert!( + bare.queue_stats(LINK, Direction::Forward) + .unwrap() + .is_none() + ); + assert!(bare.set_ip_families(true, false).is_ok()); + assert!(bare.finish_network().is_ok()); + } + + #[test] + fn a_signal_is_the_signal_of_the_model() { + assert_eq!( + ProgramSignal::from(netscenario_model::Signal::Int), + ProgramSignal::Int + ); + assert_eq!( + ProgramSignal::from(netscenario_model::Signal::Term), + ProgramSignal::Term + ); + } +} diff --git a/crates/netscenario-core/src/capability.rs b/crates/netscenario-core/src/capability.rs index be232af..23038b8 100644 --- a/crates/netscenario-core/src/capability.rs +++ b/crates/netscenario-core/src/capability.rs @@ -380,4 +380,86 @@ mod tests { assert_ne!(crate::provision::checked(&scenario), Vec::new()); assert_eq!(unknown_allowed_features(&self::scenario()), Vec::new()); } + + /// What the user reads when a scenario is refused: the line of the file, the source line with a mark under the value, the + /// feature, the backend, the reason and the way out. The mutation test found that nothing proved the text: a `render` that + /// returned an empty string passed every test of this crate. + #[test] + fn a_refusal_is_shown_at_its_line_with_the_feature_the_backend_the_reason_and_the_fix() { + let caps = all_exact().with( + Feature::FixedDelay, + Support::Unsupported { + reason: "this machine has no netem".to_owned(), + fix: "load the module sch_netem".to_owned(), + }, + ); + let error = check(&scenario(), &caps).unwrap_err(); + let text = error.render("t.yaml", FILE); + assert!(text.contains("t.yaml:"), "{text}"); + assert!( + text.contains("the fake backend cannot reproduce 'Fixed delay'"), + "{text}" + ); + assert!(text.contains("this machine has no netem"), "{text}"); + assert!(text.contains("load the module sch_netem"), "{text}"); + // The place: the line of the link with its delay, which is line 10 of the file + assert!( + text.contains("both: { delay: 20ms"), + "the source line is shown: {text}" + ); + // For a scenario read from a file with nothing expanded it is the same text + let same = error.render_for(&scenario(), "t.yaml", FILE); + assert_eq!(same, text); + // Canary: the text is made of the refusals, so no refusal gives no text + assert_eq!( + CapabilityError::new("fake", Vec::new()).render("t.yaml", FILE), + "" + ); + } + + /// A refusal of a value that came from a parameter is shown where the value was written (D-093), not at the place of the + /// expanded scenario that has no line of its own. + #[test] + fn a_refusal_of_a_value_from_a_parameter_is_shown_at_the_line_where_it_was_written() { + let file = "version: 0\nname: t\nduration: 10s\nparams:\n rtt: 40ms\ntopology:\n nodes: [a, b]\n links:\n - id: l\n between: [a, b]\n both: { delay: \"{rtt / 2}\" }\nworkloads:\n - { id: w, kind: udp-cbr, from: a, to: b, rate: 1mbit, packet-size: 1200b }\n"; + let scenario = Scenario::from_yaml(file, "p.yaml").expect("valid"); + let caps = all_exact().with( + Feature::FixedDelay, + Support::Unsupported { + reason: "no".to_owned(), + fix: "none".to_owned(), + }, + ); + let error = check(&scenario, &caps).unwrap_err(); + let text = error.render_for(&scenario, "p.yaml", file); + assert!(text.contains("p.yaml:11"), "the line of the value: {text}"); + assert!( + text.contains("{rtt / 2}"), + "the value as it was written: {text}" + ); + } + + /// The declaration can be read back: the name of the backend, and the features that were declared, in the order of the features + /// and without those that were not. + #[test] + fn the_declaration_gives_back_the_backend_and_the_features_that_were_declared() { + let caps = Capabilities::new("fake") + .with(Feature::SeededLoss, Support::Exact) + .with( + Feature::FixedDelay, + Support::Limited { + restriction: "steps".to_owned(), + }, + ); + assert_eq!(caps.backend(), "fake"); + let declared: Vec = caps.iter().map(|(feature, _)| feature).collect(); + assert_eq!(declared.len(), 2); + let mut ordered = declared.clone(); + ordered.sort(); + assert_eq!(declared, ordered, "in the order of Feature"); + assert!(declared.contains(&Feature::FixedDelay) && declared.contains(&Feature::SeededLoss)); + assert!(caps.iter().any(|(_, support)| *support == Support::Exact)); + assert_eq!(Capabilities::new("none").iter().count(), 0); + assert_eq!(Capabilities::new("x").backend(), "x"); + } } diff --git a/crates/netscenario-model/tests/command.rs b/crates/netscenario-model/tests/command.rs new file mode 100644 index 0000000..bcb35aa --- /dev/null +++ b/crates/netscenario-model/tests/command.rs @@ -0,0 +1,151 @@ +//! The faults in a program of your own (`kind: command`, WRK-06, D-072) that the files of `scenarios/invalid/` do not show: the limits +//! (the number of arguments, the size of a text, the number of variables), braces, and the waits of `ready` and `shutdown`. Each is refused +//! with the place of the fault; a scenario that is a little less wrong is accepted, so the bound is where the message says. + +// A test file: a failing unwrap is a failing test +#![allow(clippy::unwrap_used, clippy::expect_used)] + +use netscenario_model::{MAX_ARGUMENT_BYTES, MAX_ARGUMENTS, MAX_ENV_ENTRIES, Scenario}; + +/// A scenario with one program; `command` is the lines under the program (indented by 4 spaces). +fn with_program(command: &str) -> String { + format!( + "version: 0\nname: t\nduration: 5s\ntopology:\n nodes: [client, server]\n links:\n - id: access\n between: [client, server]\nworkloads:\n - id: app\n kind: command\n node: server\n{command}" + ) +} + +/// What the reader says about a scenario, or `None` if it accepts it. +fn refusal(yaml: &str) -> Option { + Scenario::from_yaml(yaml, "t.yaml") + .err() + .map(|error| error.to_string()) +} + +fn refused(command: &str) -> String { + refusal(&with_program(command)).unwrap_or_else(|| panic!("accepted:\n{command}")) +} + +fn accepted(command: &str) { + if let Some(message) = refusal(&with_program(command)) { + panic!("refused:\n{command}\n{message}"); + } +} + +#[test] +fn a_program_may_have_as_many_arguments_as_the_limit_and_not_one_more() { + let list = |count: usize| format!(" run: [{}]\n", vec!["\"a\""; count].join(", ")); + accepted(&list(MAX_ARGUMENTS)); + let message = refused(&list(MAX_ARGUMENTS + 1)); + assert!( + message.contains(&format!( + "`run` has {} entries, at most {MAX_ARGUMENTS} are accepted", + MAX_ARGUMENTS + 1 + )), + "{message}" + ); + assert!(message.contains("workloads[0].run"), "{message}"); +} + +#[test] +fn the_program_cannot_be_an_empty_text_but_an_argument_may() { + let message = refused(" run: [\"\"]\n"); + assert!( + message.contains("the program is an empty text"), + "{message}" + ); + assert!(message.contains("run[0]"), "{message}"); + accepted(" run: [\"./server\", \"\"]\n"); +} + +#[test] +fn a_text_may_have_as_many_bytes_as_the_limit_and_not_one_more() { + let argument = |bytes: usize| format!(" run: [\"./server\", \"{}\"]\n", "x".repeat(bytes)); + accepted(&argument(MAX_ARGUMENT_BYTES)); + let message = refused(&argument(MAX_ARGUMENT_BYTES + 1)); + assert!( + message.contains(&format!( + "this text has {} bytes, at most {MAX_ARGUMENT_BYTES} are accepted", + MAX_ARGUMENT_BYTES + 1 + )), + "{message}" + ); + assert!(message.contains("run[1]"), "{message}"); +} + +#[test] +fn a_program_may_have_as_many_variables_as_the_limit_and_not_one_more() { + let variables = |count: usize| { + let entries: Vec = (0..count).map(|i| format!("V{i}: \"1\"")).collect(); + format!( + " run: [\"./server\"]\n env: {{ {} }}\n", + entries.join(", ") + ) + }; + accepted(&variables(MAX_ENV_ENTRIES)); + let message = refused(&variables(MAX_ENV_ENTRIES + 1)); + assert!( + message.contains(&format!( + "`env` has {} variables, at most {MAX_ENV_ENTRIES} are accepted", + MAX_ENV_ENTRIES + 1 + )), + "{message}" + ); +} + +#[test] +fn a_brace_is_a_placeholder_or_is_written_twice() { + let message = refused(" run: [\"./server\", \"a}b\"]\n"); + assert!( + message.contains("a closing brace without an opening one"), + "{message}" + ); + assert!( + message.contains("`}}`"), + "the help says how to write a brace: {message}" + ); + let message = refused(" run: [\"./server\", \"a{b\"]\n"); + assert!( + message.contains("an opening brace without a closing one"), + "{message}" + ); + // Twice is a brace; a placeholder of a node that exists is an address + accepted(" run: [\"./server\", \"a}}b{{c\", \"{server.addr}\"]\n"); +} + +#[test] +fn a_way_to_be_ready_with_an_empty_text_or_a_wait_of_zero_is_refused() { + let empty = refused(" run: [\"./server\"]\n ready: { stdout-line: \"\", within: 5s }\n"); + assert!(empty.contains("this text is empty"), "{empty}"); + assert!(empty.contains("ready.stdout-line"), "{empty}"); + let file = refused(" run: [\"./server\"]\n ready: { file: \"\", within: 5s }\n"); + assert!(file.contains("ready.file"), "{file}"); + let zero_delay = refused(" run: [\"./server\"]\n ready: { delay: 0s }\n"); + assert!( + zero_delay.contains("a delay of zero is no wait"), + "{zero_delay}" + ); + let zero_within = + refused(" run: [\"./server\"]\n ready: { udp-port: 4433, within: 0s }\n"); + assert!( + zero_within.contains("`within` must be longer than zero"), + "{zero_within}" + ); + assert!(zero_within.contains("ready.within"), "{zero_within}"); + accepted(" run: [\"./server\"]\n ready: { udp-port: 4433, within: 1ms }\n"); + accepted(" run: [\"./server\"]\n ready: { delay: 1ms }\n"); +} + +#[test] +fn a_shutdown_that_waits_for_zero_seconds_is_refused() { + let message = refused(" run: [\"./server\"]\n shutdown: { signal: TERM, within: 0s }\n"); + assert!( + message.contains("`within` must be longer than zero"), + "{message}" + ); + assert!(message.contains("shutdown.within"), "{message}"); + assert!( + message.contains("`KILL`"), + "the help says what within is: {message}" + ); + accepted(" run: [\"./server\"]\n shutdown: { signal: TERM, within: 1ms }\n"); +} diff --git a/crates/netscenario-model/tests/models.rs b/crates/netscenario-model/tests/models.rs index 84e2297..bb515dd 100644 --- a/crates/netscenario-model/tests/models.rs +++ b/crates/netscenario-model/tests/models.rs @@ -542,3 +542,232 @@ fn the_examples_compile_to_valid_scenarios() { } assert!(found >= 3, "the examples with a model: {found}"); } + +// ---- the faults of a model that no file of scenarios/invalid/ shows ---- + +fn model(kind: &str, rest: &str) -> String { + format!(" model:\n kind: {kind}\n{rest}") +} + +/// Each fault is refused with the reason and the place; the same scenario with the fault mended is accepted, so the message is about +/// that fault and not about something else in the file. +#[test] +fn a_periodic_model_with_a_fault_is_refused_with_the_reason_and_the_place() { + let periodic = |rest: &str| model("periodic", rest); + let one = " states:\n - { for: 2s, set: { both: { rate: 1mbit } } }\n"; + let cases: Vec<(&str, String, &str, &str)> = vec![ + ( + "begins after the end", + periodic(&format!(" begin: 20s\n{one}")), + "which is not before the end of the measured phase", + "model.begin", + ), + ( + "state name twice", + periodic( + " states:\n - { id: a, for: 1s, set: { both: { rate: 1mbit } } }\n - { id: a, for: 1s, set: { both: { rate: 2mbit } } }\n", + ), + "the state name 'a' is used twice", + "states[1].id", + ), + ( + "state without for", + periodic(" states:\n - { set: { both: { rate: 1mbit } } }\n"), + "needs `for`: how long it lasts", + "states[0]", + ), + ( + "state of zero length", + periodic(" states:\n - { for: 0s, set: { both: { rate: 1mbit } } }\n"), + "a state must last longer than zero", + "states[0].for", + ), + ( + "outage and down", + periodic(" states:\n - { for: 1s, outage: true, down: true }\n"), + "a state is an `outage` or `down`, not both", + "states[0]", + ), + ( + "direction without outage", + periodic( + " states:\n - { for: 1s, direction: forward, set: { both: { rate: 1mbit } } }\n", + ), + "`direction` belongs to `outage`", + "states[0].direction", + ), + ( + "set without a key", + periodic(" states:\n - { for: 1s, set: {} }\n"), + "this state writes no key, so it changes nothing", + "states[0].set", + ), + ( + "set of a direction without a key", + periodic(" states:\n - { for: 1s, set: { both: {} } }\n"), + "this state writes no key, so it changes nothing", + "states[0].set.both", + ), + ]; + faults_are_refused(&cases); +} + +/// The same for the models that are chains. +#[test] +fn a_markov_model_with_a_fault_is_refused_with_the_reason_and_the_place() { + let markov = |rest: &str| model("markov", rest); + let cases: Vec<(&str, String, &str, &str)> = vec![ + ( + "markov state without id", + markov( + " start: a\n states:\n - { stay: 1s, to: { a: 100% } }\n", + ), + "needs an `id`: the chances refer to it", + "states[0]", + ), + ( + "markov state without stay", + markov(" start: a\n states:\n - { id: a, to: { a: 100% } }\n"), + "needs `stay`: how long it lasts, or the mean", + "states[0]", + ), + ( + "markov state of zero stay", + markov( + " start: a\n states:\n - { id: a, stay: 0s, to: { a: 100% } }\n", + ), + "a stay must be longer than zero", + "states[0].stay", + ), + ( + "markov state with for", + markov( + " start: a\n states:\n - { id: a, stay: 1s, for: 1s, to: { a: 100% } }\n", + ), + "belongs to", + "states[0].for", + ), + ( + "markov without start", + markov(" states:\n - { id: a, stay: 1s, to: { a: 100% } }\n"), + "a Markov model needs `start`: the state it begins in", + "model", + ), + ( + "markov state without chances", + markov(" start: a\n states:\n - { id: a, stay: 1s }\n"), + "needs `to`: the chances of the next state", + "states[0]", + ), + ]; + faults_are_refused(&cases); +} + +/// Each case is a name, the lines of the link, the reason that must be said and the place that must be named. +fn faults_are_refused(cases: &[(&str, String, &str, &str)]) { + for (name, link, expected, place) in cases { + let message = refused(&head("12s", link, "")); + assert!(message.contains(expected), "{name}: {message}"); + assert!( + message.contains(place), + "{name}: the place {place} is not in\n{message}" + ); + } +} + +#[test] +fn the_start_of_a_chain_that_is_a_misspelling_is_corrected_in_the_help() { + let link = model( + "markov", + " start: goood\n states:\n - { id: good, stay: 1s, to: { good: 100% } }\n", + ); + let message = refused(&head("12s", &link, "")); + assert!( + message.contains("the start state 'goood' does not exist"), + "{message}" + ); + assert!(message.contains("did you mean 'good'?"), "{message}"); +} + +#[test] +fn the_faults_of_a_trace_of_a_link_are_refused_with_the_key() { + let trace = |keys: &str| model("trace", &format!(" both: {{ {keys} }}\n")); + let cases = [ + ("no file", trace(""), "a trace needs a `file`", "model.both"), + ( + "a window of zero", + trace("file: walk.mahimahi, format: mahimahi, window: 0s"), + "the window must be longer than zero", + "model.both.window", + ), + ( + "a period of zero", + trace("file: drive.csv, period: 0s"), + "the period must be longer than zero", + "model.both.period", + ), + ( + "a window for a csv file", + trace("file: drive.csv, window: 50ms"), + "`window` belongs to a mahimahi trace, and this one is csv", + "model.both.window", + ), + ( + "a period for a mahimahi file", + trace("file: walk.mahimahi, format: mahimahi, period: 1s"), + "`period` belongs to a csv trace, and this one is mahimahi", + "model.both.period", + ), + ( + "a format that cannot be told", + trace("file: drive.dat"), + "cannot tell the format of 'drive.dat' from its name", + "model.both.file", + ), + ( + "a hash of the wrong length", + trace("file: drive.csv, sha256: abc"), + "a SHA-256 is 64 hexadecimal digits", + "model.both.sha256", + ), + ]; + for (name, link, expected, place) in &cases { + let message = refused(&head("12s", link, "")); + assert!(message.contains(expected), "{name}: {message}"); + assert!( + message.contains(place), + "{name}: the place {place} is not in\n{message}" + ); + } +} + +#[test] +fn an_event_that_repeats_zero_times_or_ends_before_it_begins_is_refused() { + let timeline = |event: &str| format!("timeline:\n - {event}\n"); + let message = refused(&head( + "12s", + "", + &timeline("{ at: 2s, link: l, outage: 100ms, every: 3s, count: 0 }"), + )); + assert!(message.contains("`count` must be at least 1"), "{message}"); + let message = refused(&head( + "12s", + "", + &timeline("{ at: 4s, link: l, outage: 100ms, every: 3s, until: 3s }"), + )); + assert!( + message.contains("must be after the first event"), + "{message}" + ); + // One more than zero, and an end after the start, are accepted + read(&head( + "12s", + "", + &timeline("{ at: 4s, link: l, outage: 100ms, every: 3s, until: 11s }"), + )); + read(&head( + "12s", + "", + &timeline("{ at: 4s, link: l, outage: 100ms, every: 3s, count: 1 }"), + )); +} diff --git a/docs/COVERAGE.md b/docs/COVERAGE.md index 2113969..e193769 100644 --- a/docs/COVERAGE.md +++ b/docs/COVERAGE.md @@ -36,7 +36,7 @@ Docker is not a column: it is a way to start programs and to reach the Kernel ba | Dumbbell (shared bottleneck) | ✔ 0.3 ([test](../crates/netscenario-backend-kernel/tests/network.rs), [fairness](../python/netscenario/tests/test_acceptance_tcp.py)) | ✔ 0.4 | ✘ | | Multipath, multi-homing | ✔ 0.3 ([test](../crates/netscenario-backend-kernel/tests/network.rs), [TCP over two paths](../python/netscenario/tests/test_acceptance_tcp_models.py)) | ✔ 0.4 | ✘ | | Arbitrary graph | ✔ 0.3 ([test](../crates/netscenario-backend-kernel/tests/network.rs)) | ✔ 0.4 | ✘ | -| Links up and down | ✔ 0.2 | ✔ 0.4 | ✔ 0.5 | +| Links up and down | ✔ 0.2 ([test](../crates/netscenario-backend-kernel/tests/timeline.rs)) | ✔ 0.4 | ✔ 0.5 | | Nodes joining and leaving | ✔ 0.6 | ✔ 0.6 | ✘ | | IPv4 | ✔ 0.1 ([test](../crates/netscenario-backend-kernel/tests/run.rs)) | ✔ 0.4 | ✔ 0.5 | | IPv6 | ✔ 0.3 ([test](../crates/netscenario-backend-kernel/tests/network.rs): a chain, both families, fragments, the seeded loss against IPv4) | ✔ 0.4 | ~ 0.5 | diff --git a/python/netscenario/tests/test_ledger.py b/python/netscenario/tests/test_ledger.py new file mode 100644 index 0000000..6e13980 --- /dev/null +++ b/python/netscenario/tests/test_ledger.py @@ -0,0 +1,202 @@ +"""The ledger (``FEATURES.md``): a row that says ``done`` names the test that proves it. + +``CLAUDE.md`` and ``CONTRIBUTING.md`` ask for it: a feature has a line before it is built, and a +check mark needs a link to the test that proves it. This test keeps the ledger honest. A row +without a proof, a link that leads nowhere, and a proof that is only a document for a feature that +runs, each fail here (and the tests below prove that they would). +""" + +from __future__ import annotations + +import os +import re +from dataclasses import dataclass +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[3] +LEDGER = ROOT / "FEATURES.md" + +#: The rows whose proof is a document, a configuration or a workflow and not a test: there is +#: no behaviour of the program to run. Each still needs a reference that exists. +PROOF_IS_A_FILE = frozenset( + { + "DOC-01", # the README, the architecture, the decisions + "DOC-04", # `missing_docs` and ruff `D` are errors: the configuration is the proof + "DOC-05", # the troubleshooting page + "CIX-04", # the permissions page + "REL-01", # the release workflow, which runs on a tag + "REL-02", # the configuration of the changelog + "REL-03", # the versioning policy + "PYT-03", # pyright in strict mode over the package + "PYT-08", # the wheels workflow builds and installs them + "QA-07", # the semver and Rust-version jobs of the CI + "QA-08", # `cargo deny`, Scorecard, pinned actions + } +) + +#: Where a reference may point to. A backtick word is a reference only if it looks like a path. +TOP = ("crates", "python", "docs", "examples", "scenarios", "fuzz", "spikes", ".github") +PATH_WORD = re.compile( + r"^(?:[\w./-]+\.(?:rs|py|yml|yaml|toml|md)|(?:" + "|".join(TOP) + r")/[\w./-]*)(?:#.*)?$" +) +LINK = re.compile(r"\]\(([^)\s]+)\)") +WORD = re.compile(r"`([^`\s]+)`") + + +@dataclass(frozen=True) +class Row: + """One row of the ledger.""" + + id: str + status: str + + +def rows(text: str) -> list[Row]: + """The rows of the tables of the ledger: an id like ``SCN-01`` in the first column.""" + found: list[Row] = [] + for line in text.splitlines(): + cells = re.split(r"(?= 7 and re.fullmatch(r"\s*[A-Z]+-\d+\s*", cells[1]): + found.append(Row(cells[1].strip(), cells[5].strip())) + return found + + +def repo_files() -> list[str]: + """The files of the repository that a reference may name, relative to the root.""" + files: list[str] = [] + for top in TOP: + for directory, names, filenames in os.walk(ROOT / top): + built = Path(directory) / "book" if Path(directory) == ROOT / "docs" / "book" else None + names[:] = [ + n + for n in names + if n not in {"target", ".venv", "__pycache__"} and Path(directory) / n != built + ] + files.extend(str((Path(directory) / name).relative_to(ROOT)) for name in filenames) + files.extend(p.name for p in ROOT.iterdir() if p.is_file()) + return files + + +def references(status: str) -> list[str]: + """What the status of a row points to: links to files, and backtick words that are paths.""" + links = [t for t in LINK.findall(status) if not t.startswith(("http", "#", "mailto"))] + words = [w for w in WORD.findall(status) if PATH_WORD.match(w)] + return [r.split("#")[0] for r in links + words if r.split("#")[0]] + + +def resolve(reference: str, files: list[str]) -> list[str]: + """The files a reference names: the file itself, the files in a directory, or the files whose + path ends with it (a word like ``run.rs`` stands for the file of that name).""" + path = os.path.normpath(reference) + if path in files: + return [path] + inside = [f for f in files if f.startswith(path.rstrip("/") + "/")] + if inside: + return inside + return [f for f in files if f.endswith("/" + path)] + + +def is_test(path: str) -> bool: + """Whether a file is a test: in a ``tests`` directory, named ``test_*``, a fuzz target, or a + source file with a ``#[cfg(test)]`` module.""" + parts = path.split("/") + name = parts[-1] + if "tests" in parts or "test" in parts or name.startswith("test_") or name == "tests.rs": + return True + if path.startswith("fuzz/fuzz_targets/"): + return True + if path.endswith(".rs") and "/src/" in path: + return "#[cfg(test)]" in (ROOT / path).read_text() + return False + + +def problems(text: str, files: list[str]) -> list[str]: + """What is wrong with a ledger: a reference to nothing, a ``done`` row with no proof, a + ``done`` row of a feature that runs whose proof is no test.""" + found: list[str] = [] + for row in rows(text): + resolved: list[str] = [] + for reference in references(row.status): + matches = resolve(reference, files) + if not matches: + found.append(f"{row.id}: {reference} is not in the repository") + resolved.extend(matches) + if not row.status.lower().startswith("done"): + continue + if not resolved: + found.append(f"{row.id}: done, and names no test or file that proves it") + elif row.id not in PROOF_IS_A_FILE and not any(is_test(p) for p in resolved): + found.append(f"{row.id}: done, and none of {sorted(set(resolved))[:3]} is a test") + return found + + +def test_every_done_row_of_the_ledger_names_a_test_that_exists() -> None: + text = LEDGER.read_text() + assert len(rows(text)) > 100, "the table was not read" + assert problems(text, repo_files()) == [] + + +def test_a_row_that_is_a_file_is_only_one_where_there_is_nothing_to_run() -> None: + # The exceptions must be rows of the ledger, and must not hide a feature that runs + ids = {row.id for row in rows(LEDGER.read_text())} + assert ids >= PROOF_IS_A_FILE + assert not {i for i in PROOF_IS_A_FILE if i.startswith(("WRK", "LNK", "PRB", "TML"))} + + +# ---- the check can fail (canaries) ---- + +FILES = [ + "crates/x/tests/run.rs", + "crates/x/src/units.rs", + "docs/COVERAGE.md", + "python/netscenario/tests/test_a.py", +] + + +def ledger(*lines: str) -> str: + return "\n".join(f"| {line} |" for line in lines) + + +def test_a_done_row_with_no_reference_is_a_problem() -> None: + text = ledger("WRK-99 | A stream | M | 0.1 | done | ", "WRK-98 | Another | M | 0.1 | planned") + assert problems(text, FILES) == ["WRK-99: done, and names no test or file that proves it"] + + +def test_a_reference_to_a_file_that_is_not_there_is_a_problem_in_any_row() -> None: + text = ledger( + "WRK-99 | A stream | M | 0.1 | done ([test](crates/x/tests/gone.rs))", + "WRK-98 | Another | M | 0.1 | planned (see `docs/GONE.md`)", + ) + found = problems(text, FILES) + assert any(p == "WRK-99: crates/x/tests/gone.rs is not in the repository" for p in found) + assert any(p == "WRK-98: docs/GONE.md is not in the repository" for p in found) + + +def test_a_document_is_no_proof_of_a_feature_that_runs() -> None: + text = ledger("WRK-99 | A stream | M | 0.1 | done ([the page](docs/COVERAGE.md))") + (problem,) = problems(text, FILES) + assert problem.startswith("WRK-99: done, and none of") + # ... but it is the proof of a row that is a document + assert ( + problems( + ledger("DOC-05 | Troubleshooting | M | 0.1 | done ([page](docs/COVERAGE.md))"), FILES + ) + == [] + ) + + +def test_a_test_in_a_tests_directory_a_test_file_or_a_source_with_tests_is_a_proof() -> None: + for status in ( + "done ([t](crates/x/tests/run.rs))", + "done (`run.rs` and `test_a.py`)", + "done ([t](python/netscenario/tests/test_a.py))", + ): + assert problems(ledger(f"WRK-99 | A | M | 0.1 | {status}"), FILES) == [], status + + +def test_words_in_backticks_are_references_only_if_they_look_like_paths() -> None: + status = ( + "done (`netscenario run`, `result.json` is made by a run, `10.100.n.0/30`, `tests/run.rs`)" + ) + # A file a run makes (`result.json`), an address and a command are not files of the repository + assert references(status) == ["tests/run.rs"]