(draft) ci: run netsim perf and integration tests via patchbay instead of chucks#4046
Draft
(draft) ci: run netsim perf and integration tests via patchbay instead of chucks#4046
Conversation
Switch patchbay dep to git branch with TestGuard support. Each test now gets a guard from lab_with_relay that records "success" (via .ok()) or "failed" (on drop without .ok()) into state.json. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…re reported The guard was consumed by `.ok()` inside the helper before the caller's assertion ran. Now the guard is returned and only marked `.ok()` after the assertion in each test function, so a failing assertion correctly sets the status to "failed" in state.json. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Picks up fix for state.json not being written for fast tests (writer now blocks on completion in drop, creates state.json from scratch if it doesn't exist). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Port all network simulations from the chuck/netsim JSON format to patchbay TOML format. This migrates 57 sim files covering performance benchmarks (1G, 10G, latency variants), integration tests (public, relay, relay-only), and paused/experimental sims (NAT traversal, adverse conditions, interface switching). Includes: - iroh-defaults.toml: shared binaries, prepare, templates, groups - 22 topology files for various network configurations - 40 iroh-* performance sims (8 topologies x 5 variants) - 7 intg-* integration sims - 10 paused-* inactive sims (excluded from CI by default) - netsim-patchbay.yaml CI workflow Skipped: iroh_full and adverse dns sims (need iroh-dns-server binary). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Relay-only mode clears IP transports so direct_addresses is empty. The --remote-direct-address flag is unnecessary for relay-only transfers and caused the capture to block forever waiting for a value that would never resolve. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace 46 individual sim files with 7 matrix files that generate the same
set of simulations. Uses patchbay's new [matrix] support for Cartesian product
expansion, conditional steps, and ${i % N} index expressions.
- iroh-transfer-direct.toml: 4 topos × 4 conditions = 16 sims
- iroh-transfer-relay-only.toml: 4 topos = 4 sims
- iroh-transfer-multi-direct.toml: 4 topos × 4 conditions = 16 sims
- iroh-transfer-multi-relay-only.toml: 4 topos = 4 sims
- intg-direct.toml: 2 topos = 2 sims
- intg-multi-direct.toml: 2 topos = 2 sims
- intg-relay-only.toml: 2 topos = 2 sims
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move paused-* sims to integration/, remove "paused-" prefix from names - Move intg-* sims to integration/ - Move iroh-transfer-* sims to perf/ - Port missing DNS sims from old chuck: relay+dns (1to1, 1to3), relay-dns-relay-only, relay-dns-throttled, relay-dns-lossy - Add iroh-dns-server binary and dns-setup step group to defaults - Add relay-dns topologies (1to1, 1to3) - Remove ready_after in favor of capture-based readiness - Rename netsim-patchbay.yaml to patchbay-runner.yaml - Run integration and perf groups separately in CI - Remove sudo and explicit build step (patchbay prepare handles it) - Remove "skip paused" logic — all sims run - Add perf-summary.sh script for chuck-style throughput table Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Pass directories directly to patchbay instead of shell find/expand - Add Frando/patchbay-tomls branch trigger for CI testing Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Frando
changed the title
- Specify patchbay-runner package to disambiguate - Fix repo URL to n0-computer/patchbay Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh/pr/4046/docs/iroh/ Last updated: 2026-03-23T15:07:34Z |
Merge pairs of near-identical sim files into matrix sims:
- direct-lossy + direct-throttled → direct-adverse.toml
- nat-both-lossy + nat-both-throttled → nat-adverse.toml
- relay-dns-lossy + relay-dns-throttled → relay-dns-adverse.toml
- route-switch + route-switch-both → route-switch.toml
- 1to1-nat-{both,fetcher,provider} → 1to1-nat.toml
18 → 16 sim files, same test coverage.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The CI runner has Node 12 which can't build the devtools UI. Use --no-default-features to skip patchbay-server dependency. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Patchbay doesn't support ${matrix.*} interpolation in TOML inline
table values (condition = { latency_ms = ${matrix.latency} }) since
those aren't valid TOML — the parser fails before interpolation.
Split files that used matrix params in condition fields:
- direct-adverse → direct-throttled + direct-lossy
- nat-adverse → nat-both-throttled + nat-both-lossy
- relay-dns-adverse → relay-dns-throttled + relay-dns-lossy
- iroh-transfer-direct → 4 files per condition (baseline/20ms/200ms/10g)
- iroh-transfer-multi-direct → 4 files per condition
Files using matrix only in string values (topology, name, args) are
kept as-is since string interpolation is valid TOML.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Patchbay does not have matrix expansion support. All [matrix] sections
with ${matrix.*} interpolation were non-functional — they parsed as
literal strings and would fail at runtime trying to resolve topology
names like "${matrix.topo}-public".
Expand all 16 matrix files into 53 individual sim files:
- 6 integration matrix files → 13 individual files
- 10 perf matrix files → 40 individual files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace 62 individual sim TOML files with 16 matrix-based files that expand to the same set of simulations. Integration: 22 → 12 files, perf: 40 → 4 files. Uses [matrix] with params for topology and condition variations, and `when` for conditional steps. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Merge public/relay/relay-dns topology variants into single files that always include relay and dns devices. Sims that don't use them simply have idle nodes. 24 → 14 topology files. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The prepare step builds all binaries upfront, so the run steps can skip rebuilding with --no-build. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove nested ${provider-${i % 2}} syntax (now handled by patchbay
count expansion rewriting ${provider.capture} automatically)
- Remove asserts on fetcher.size from multi-provider sims (captures
not populated in duration mode)
- Remove assert from route-switch (capture unreliable after route change)
- Reduce 200ms condition transfer size from 1G to 100M to fit timeout
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
11 tasks
Frando
force-pushed
the
Frando/netsim
branch
3 times, most recently
from
62c5ae9 to
9f7f09c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This has the netsim/chuck integration and perf tests ported to patchbay. Untested and not yet reviewed.
Breaking Changes
Notes & open questions
Change checklist
quic-rpciroh-gossipiroh-blobsdumbpipesendme