Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ The development toolchain is the current stable Rust (`rust-toolchain.toml` inst
| `just fidelity` | Run the fidelity tests alone and rewrite the measured block of [docs/FIDELITY.md](docs/FIDELITY.md) with what your machine measures |
| `just test-quick`, `just fidelity-test [part of]` | The tests but the fidelity ones (a minute), and the fidelity tests without touching the page (four and a half minutes; `just fidelity-test 1 2` is the first half). The CI runs them as separate jobs |
| `just msrv` | Build with the lowest supported Rust |
| `just coverage`, `just mutants <crate> <file>` | How much of the code the tests run, and whether they would notice a change of it ([docs/TESTING.md](docs/TESTING.md)) |
| `just schema`, `just snapshots` | After a change to the scenario model: write the JSON Schema; accept changed error messages (read the diff before you commit it) |
| `just test-kernel` (and `just test`) | The tests of `netscenario-backend-kernel` create real network namespaces as an unprivileged user. On a machine that does not allow that (Ubuntu 24.04 until `sudo sysctl kernel.apparmor_restrict_unprivileged_userns=0`) they say `SKIPPED` and pass; on the CI (`CI` is set) they fail instead. The CI runs them a second time as root; `unshare --user --map-root-user just test-kernel` takes the same code path on your machine. Needs `ip`, `tc`, `unshare`, `nsenter` and `ping`. |
| `just spikes` | Run the experiments of [spikes/](spikes/) (they need `iproute2`, `python3` and user namespaces) |
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ A Rust core (scenario model, scheduler, backends, traffic generators, probes, st
- [docs/PRIOR-ART.md](docs/PRIOR-ART.md): what exists, and where netscenario differs.
- [docs/PROFILES.md](docs/PROFILES.md): the draft of the link profiles (3G to LEO satellite) with the source of every number.
- [docs/ACCEPTANCE.md](docs/ACCEPTANCE.md): how each criterion of a release is proven, and what the proof does not cover.
- [docs/TESTING.md](docs/TESTING.md): how the tests are organised, where each kind runs, how coverage and mutation tests are measured, and the rules for a new test.
- [docs/VERSIONING.md](docs/VERSIONING.md): what a version number promises, and what a release may break.
- [docs/RELEASING.md](docs/RELEASING.md): how a release is made, and the one-time setup of the registries.
- [docs/DECISIONS.md](docs/DECISIONS.md): the decisions made so far, with reasons (one file each in [docs/decisions/](docs/decisions/)).
Expand Down
32 changes: 32 additions & 0 deletions crates/netscenario-core/src/capability.rs
Original file line number Diff line number Diff line change
Expand Up @@ -462,4 +462,36 @@ mod tests {
assert_eq!(Capabilities::new("none").iter().count(), 0);
assert_eq!(Capabilities::new("x").backend(), "x");
}

/// Several refusals are shown one after the other with an empty line between them, and none before the first or after the last.
/// The mutation test found that a display of one refusal was all that was checked.
#[test]
fn several_refusals_are_separated_by_an_empty_line() {
let refusal = |feature, path: &str| Refusal {
feature,
path: path.to_owned(),
reason: "no".to_owned(),
fix: "none".to_owned(),
};
let error = CapabilityError::new(
"fake",
vec![
refusal(Feature::FixedDelay, "a"),
refusal(Feature::SeededLoss, "b"),
],
);
let text = error.to_string();
let blocks: Vec<&str> = text.trim_end().split("\n\n").collect();
assert_eq!(blocks.len(), 2, "{text:?}");
assert!(
blocks[0].contains("path: a") && !blocks[0].contains("path: b"),
"{text:?}"
);
assert!(
blocks[1].starts_with("error: ") && blocks[1].contains("path: b"),
"{text:?}"
);
assert!(text.starts_with("error: ") && text.ends_with('\n'));
assert!(!text.ends_with("\n\n"));
}
}
123 changes: 123 additions & 0 deletions crates/netscenario-core/src/expect.rs
Original file line number Diff line number Diff line change
Expand Up @@ -560,4 +560,127 @@ mod tests {
assert_eq!(show_value(MetricUnit::Number, 0.5), "0.5");
assert_eq!(show_value(MetricUnit::Ratio, f64::NAN), "n/a");
}

/// The median and the smallest of a set are judged with their own values, not with the mean or the largest. The mutation test
/// found both statistics unchecked: a set of eight losses with an outlier tells them apart.
#[test]
fn the_median_and_the_minimum_of_a_set_are_judged_with_their_own_values() {
let (metrics, confidence) = set_of(&[0.01, 0.01, 0.01, 0.01, 0.02, 0.02, 0.02, 0.09]);
let set = metrics
.iter()
.find(|set| set.name == "w.loss_ratio")
.unwrap();
assert!(set.min < set.median && set.median < set.mean && set.mean < set.max);
let scenario = scenario(
"expect:\n repetitions: 8\n checks:\n - { name: m, metric: w.loss_ratio, median: \"< 2%\", min: \">= 1%\", mean: \"> 2%\", max: \"<= 9%\" }\n",
);
let verdict = judge_set(&scenario, &metrics, &[], &confidence, 8).unwrap();
assert!(verdict.passed, "{verdict:?}");
let value_of = |statistic| {
verdict
.checks
.iter()
.find(|check| check.statistic == statistic)
.unwrap()
.actual
.unwrap()
};
assert!((value_of(Statistic::Median) - set.median).abs() < 1e-15);
assert!((value_of(Statistic::Min) - set.min).abs() < 1e-15);
assert!((value_of(Statistic::Mean) - set.mean).abs() < 1e-15);
assert!((value_of(Statistic::Max) - set.max).abs() < 1e-15);
}

/// Canary: the interval of a set that was made another way than the 95 % interval around the mean is not judged, whichever of the
/// two differs, and the check fails with the reason.
#[test]
fn an_interval_of_another_level_or_around_the_median_is_not_judged_as_the_95_interval() {
let (metrics, confidence) = set_of(&[0.01; 8]);
let scenario = scenario(
"expect:\n repetitions: 8\n checks:\n - { metric: w.loss_ratio, ci95-high: \"< 5%\", ci95-low: \">= 0%\" }\n",
);
assert!(
judge_set(&scenario, &metrics, &[], &confidence, 8)
.unwrap()
.passed
);
let other_level = Confidence {
level: 0.9,
..confidence.clone()
};
let other_statistic = Confidence {
statistic: IntervalAround::Median,
..confidence
};
for (name, other) in [("level", other_level), ("statistic", other_statistic)] {
let verdict = judge_set(&scenario, &metrics, &[], &other, 8).unwrap();
assert!(!verdict.passed, "{name}");
assert_eq!(verdict.failures().count(), 2, "{name}: both ends");
for check in verdict.failures() {
assert!(
check
.note
.as_deref()
.unwrap()
.contains("another interval than the 95 %"),
"{name}: {check:?}"
);
}
}
}

/// What a missing metric usually means is said for the metrics where that is known, and not for the others.
#[test]
fn a_missing_metric_says_what_it_usually_means() {
assert_eq!(
why_missing("w.delay.p99"),
"no packet or response arrived, so there is no delay to judge"
);
assert_eq!(
why_missing("f.rtt.p50"),
"the kernel gave no samples of the round trip time"
);
assert_eq!(why_missing("f.completion"), "no connection did its work");
assert_eq!(
why_missing("w.loss_ratio"),
"the workload did not report it"
);
// `completion` is a name only at the end: a delay of a workload with that name is a delay
assert_eq!(
why_missing("completion.delay.p99"),
"no packet or response arrived, so there is no delay to judge"
);
}

/// The edges of the units in which a value is shown: the value that is exactly one unit is shown in that unit.
#[test]
fn a_value_at_the_edge_of_a_unit_is_shown_in_that_unit() {
let shown = [
(MetricUnit::Seconds, 1.0, "1s"),
(MetricUnit::Seconds, 0.001, "1ms"),
(MetricUnit::Seconds, 0.000_001, "1us"),
(MetricUnit::Seconds, 0.999, "999ms"),
(MetricUnit::BitsPerSecond, 1e9, "1gbit"),
(MetricUnit::BitsPerSecond, 1e6, "1mbit"),
(MetricUnit::BitsPerSecond, 1e3, "1kbit"),
(MetricUnit::BitsPerSecond, 999.0, "999bit"),
(MetricUnit::BitsPerSecond, 999_000.0, "999kbit"),
];
for (unit, value, text) in shown {
assert_eq!(show_value(unit, value), text, "{value}");
}
}

/// Four significant digits, whatever the size of the number: a large one has no decimals, a small one has as many as it needs.
#[test]
fn a_value_is_shown_with_four_significant_digits() {
assert_eq!(show_value(MetricUnit::Number, 1234.5678), "1235");
assert_eq!(show_value(MetricUnit::Number, 123.456_78), "123.5");
assert_eq!(show_value(MetricUnit::Number, 12.3456), "12.35");
assert_eq!(show_value(MetricUnit::Number, 1.234_56), "1.235");
assert_eq!(show_value(MetricUnit::Number, 0.123_456), "0.1235");
assert_eq!(show_value(MetricUnit::Number, 0.012_345_6), "0.01235");
assert_eq!(show_value(MetricUnit::Number, 100.0), "100");
assert_eq!(show_value(MetricUnit::Number, -12.3456), "-12.35");
}
}
92 changes: 92 additions & 0 deletions crates/netscenario-core/src/recovery.rs
Original file line number Diff line number Diff line change
Expand Up @@ -861,4 +861,96 @@ mod tests {
(Some(10e6), Some(0))
);
}

/// The median of an even number of values is the mean of the two in the middle, of an odd number the one in the middle, and of
/// none there is none. The values are not in order.
#[test]
fn the_median_is_the_middle_value_or_the_mean_of_the_two_in_the_middle() {
assert_eq!(median(vec![4.0, 1.0, 3.0, 2.0]), Some(2.5));
assert_eq!(median(vec![1.0, 100.0]), Some(50.5));
assert_eq!(median(vec![9.0, 1.0, 5.0]), Some(5.0));
assert_eq!(median(vec![7.0]), Some(7.0));
assert_eq!(median(Vec::new()), None);
}

/// Two outages on the same link: the end of each belongs to its own start. With the same direction and another event the start
/// of the first would be taken for both.
#[test]
fn the_end_of_an_outage_belongs_to_the_start_of_the_same_event() {
let events = [
event("outage-start", 0, 2, Some("forward")),
event("outage-end", 0, 3, Some("forward")),
event("outage-start", 1, 5, Some("forward")),
event("outage-end", 1, 6, Some("forward")),
event("outage-start", 2, 8, Some("reverse")),
event("outage-end", 2, 9, Some("reverse")),
];
let at = |index: usize| disturbance_of(&events[index], &events) / SECOND;
assert_eq!((at(1), at(3), at(5)), (2, 5, 8));
// The same event in the other direction is another outage
let both = [
event("outage-start", 0, 2, Some("forward")),
event("outage-start", 0, 4, Some("reverse")),
event("outage-end", 0, 7, Some("reverse")),
];
assert_eq!(disturbance_of(&both[2], &both) / SECOND, 4);
// An end without a start is a disturbance of its own moment
let alone = [event("outage-end", 3, 6, Some("forward"))];
assert_eq!(disturbance_of(&alone[0], &alone) / SECOND, 6);
}

/// The level to reach is the share of the baseline, and not the share divided by it: a stream that delivered half of what it sent
/// before the disturbance has recovered when it delivers 45 % again, and does not have to reach 180 %.
#[test]
fn the_level_is_the_fraction_of_a_baseline_that_is_not_one() {
let mut delivered = vec![50; 10];
delivered.extend([0, 0, 0]);
delivered.extend([50; 7]);
let events = [
event("outage-start", 0, 10, Some("forward")),
event("outage-end", 0, 13, Some("forward")),
];
let found = recoveries(
&events,
&series(&delivered),
SECOND,
&RecoveryOptions::default(),
);
let r = &found[0];
assert_eq!(r.baseline, Some(0.5));
assert!((r.level.unwrap() - 0.45).abs() < 1e-12, "{:?}", r.level);
assert_eq!(time_of(&found), [Some(0)]);
// 40 % is not enough
let mut weak = vec![50; 10];
weak.extend([0, 0, 0]);
weak.extend([40; 7]);
let found = recoveries(&events, &series(&weak), SECOND, &RecoveryOptions::default());
assert_eq!(time_of(&found), [None]);
}

/// A window in which the stream sent nothing in one of the bins is not a recovery, even if the bins around it did well: the
/// link was down in that bin and the sender was refused.
#[test]
fn a_window_with_a_bin_in_which_nothing_was_sent_is_not_a_recovery() {
let mut bins = series(&[100; 20]);
// The link is down from 10 s to 13 s: nothing arrives in the bins of 10 s and 11 s, and in the bin of 12 s the sender is refused
for (index, slot) in bins.iter_mut().enumerate().take(12).skip(10) {
*slot = bin(index as u64, 0, 0);
}
bins[12].by_send.sent = 0;
bins[12].by_send.lost = 0;
bins[12].by_send.received = 0;
bins[12].by_send.delivered_ratio = None;
let events = [event("down", 0, 10, None), event("up", 1, 13, None)];
let options = RecoveryOptions {
window_bins: 3,
..RecoveryOptions::default()
};
let found = recoveries(&events, &bins, SECOND, &options);
assert_eq!(found.len(), 2, "{found:?}");
// From the `down` at 10 s: the window 12..15 would hold 200 packets sent and 200 delivered, but it has a bin of nothing sent,
// so the first window that counts is the one from 13 s
// From the `up` at 13 s: the baseline is taken before the link went down, and the windows from 13 s on are all good
assert_eq!(time_of(&found), [Some(3), Some(0)]);
}
}
26 changes: 26 additions & 0 deletions crates/netscenario-core/src/stats.rs
Original file line number Diff line number Diff line change
Expand Up @@ -537,6 +537,32 @@ mod tests {
}
}

/// The edges of the ranges and of the sequence numbers that a sender used: a run of refused numbers hides its first and its last
/// number, and the number `slots` itself was never sent. (The mutation test found the edges unchecked.)
#[test]
fn the_edges_of_a_refused_run_and_of_the_slots_are_exact() {
let seen: Vec<Record> = [2, 3, 4, 5, 9, 10, 11].into_iter().map(at).collect();
// Ten slots, 3 and 4 refused: 3 and 4 are inside the run, 5 just after it, 9 the last slot, 10 and 11 were never used
assert_eq!(name_damaged_refused(10, &[[3, 4]], &[], &seen), [2, 5, 9]);
// Two runs, a single number as a run, and a run at the very end
assert_eq!(
name_damaged_refused(10, &[[2, 2], [4, 5], [9, 9]], &[], &seen),
[3]
);
// Without a refusal every number below the slots is named, the edge included
assert_eq!(name_damaged_refused(10, &[], &[], &seen), [2, 3, 4, 5, 9]);
assert_eq!(name_damaged_refused(9, &[], &[], &seen), [2, 3, 4, 5]);
}

/// A rank beyond the last delay is the last delay (the defence of the function against a permille above 1000).
#[test]
fn the_nearest_rank_never_goes_past_the_last_delay() {
let sorted = [10, 20, 30, 40];
assert_eq!(nearest_rank(&sorted, 1000), 40);
assert_eq!(nearest_rank(&sorted, 5000), 40);
assert_eq!(nearest_rank(&[7], 999), 7);
}

#[test]
fn the_nearest_rank_is_a_measured_delay() {
// Four delays: the median is the second one (rank 2), not the mean of the middle two
Expand Down
7 changes: 6 additions & 1 deletion crates/netscenario-model/src/reserved.rs
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,12 @@ const RESERVED_VALUES: &[(&str, &str, &str)] = &[
];

fn planned(what: &str, release: &str) -> String {
format!("{what} is planned for release {release} (see FEATURES.md)")
// "release 0.5", or "a later release" when it is not decided which
if release.starts_with("a later") {
format!("{what} is planned for {release} (see FEATURES.md)")
} else {
format!("{what} is planned for release {release} (see FEATURES.md)")
}
}

/// Finds everything in the file that belongs to a later release.
Expand Down
9 changes: 9 additions & 0 deletions crates/netscenario-model/src/units.rs
Original file line number Diff line number Diff line change
Expand Up @@ -751,6 +751,15 @@ mod tests {
text.parse()
}

/// A name is its text, for the functions that take anything that is a `str`.
#[test]
fn a_name_is_seen_as_its_text() {
let name = Name::new("client-1").unwrap();
let text: &str = name.as_ref();
assert_eq!(text, "client-1");
assert_eq!(name.as_str(), text);
}

#[test]
fn durations_are_exact_and_print_in_the_largest_exact_unit() {
assert_eq!(duration("40ms").unwrap().as_nanos(), 40_000_000);
Expand Down
15 changes: 15 additions & 0 deletions crates/netscenario-model/tests/jitter.rs
Original file line number Diff line number Diff line change
Expand Up @@ -272,6 +272,21 @@ fn the_reach_of_a_table_made_from_samples_is_checked_when_it_is_read() {
);
}

/// The bound is exact down to the nanosecond: `pareto` reaches 5461 / 8192 of the amount below the mean, so with an amount of
/// 8192 ns a delay of 5461 ns is accepted and 5460 ns is refused (the mutation test found the edge unchecked).
#[test]
fn a_delay_exactly_as_long_as_the_reach_of_the_table_is_accepted() {
let with = |delay: &str| {
scenario(
&format!("{{ delay: {delay}, jitter: {{ amount: 8192ns, distribution: pareto }} }}"),
"",
)
};
assert!(with("5461ns").is_ok());
let refused = with("5460ns").unwrap_err();
assert!(refused.contains("must be at least 0.67 times"), "{refused}");
}

const STEP_BASE: &str =
"{ delay: 80ms, jitter: { amount: 20ms, distribution: table, table: { file: d.txt } } }";

Expand Down
Loading