Skip to content

fix experiment harness netem target, env precedence, and figure loaders - #163

Closed
fabracht wants to merge 1 commit into
mainfrom
bench-harness-fixes
Closed

fabracht wants to merge 1 commit into
mainfrom
bench-harness-fixes

Conversation

@fabracht

Copy link
Copy Markdown
Contributor

Four independent defects in the experiment harness and its analysis scripts, all found while preparing an Experiment 2 re-run.

run/ applied network impairment to the wrong host

run/common.sh applied netem via ssh_client, while parallel/common_parallel.sh applies it via ssh_broker. The head-of-line blocking experiments measure delivery latency, which travels the broker-to-client path, so impairing the client's egress degrades the publish path instead of the path under measurement. It also throttles ingest: a 60 s run at 500 msg/s recorded 23,590 messages under client-egress loss against 30,001 (the full offered load) under broker-egress loss. run/ now matches parallel/ and the documented methodology.

Environment variables were silently overridden by the config file

Both harnesses sourced their environment file (setup/config.env, parallel/group{N}.env) after the caller's environment was in place, so RUNS_PER_DATAPOINT=1 bash run/02c_hol_topic_scaling.sh silently ran the full 15-run sweep. Any key the env file defines behaved the same way. Both now snapshot keys already present in the environment, source the file for defaults, and restore the caller's values, so an explicit setting wins.

02c_hol_topic_scaling.sh had drifted from the data it produces

The script specified --duration 30 --warmup 5 --payload-size 512 --rate 5000, while the results it appends to record duration 60, payload 256, rate 500 — the same operating point as the 8-topic cells it is compared against. Restored, so a re-run extends the existing series instead of silently replacing it with a different condition.

Figure loaders crashed on truncated results

fig03_spike_iso_vs_topics.py and fig14_strategy_comparison.py checked that a result file existed and then called json.load unguarded, so a single zero-byte file (a run whose broker died, which the harness already reports via warn_if_empty) aborted figure generation. Both now skip unparseable files with a warning, matching how aggregate.py handles the same case.

Validated with bash -n and shellcheck -S error on the shell changes, ast.parse on the Python, and a behavioural check that an environment override is honoured while the config default still applies when none is given.

@fabracht

Copy link
Copy Markdown
Contributor Author

Superseded by #176, which includes all four fixes.

@fabracht fabracht closed this Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant