Repository navigation
Fix 31 runtime/documentation divergences found by a bidirectional audit - #189
starswallow wants to merge 65 commits into
Conversation
Closes #184. `horus install horus-monitor` symlinks into `PluginRegistry::global_bin_dir()`, which is `paths::horus_dir()/bin` — the platform CONFIG dir, so `~/.config/horus/bin` on Linux. `which_monitor_binary` checked `~/.horus/bin` and nothing else, so the install succeeded and `horus monitor` then reported the plugin as not installed. The two halves sit about 800 lines apart in different files, and the comment on the path that was checked still called it "the plugin install location". The lookup now asks the registry for that directory rather than making a second hardcoded guess at it, and keeps `~/.horus/bin` behind it so an existing install goes on working. The candidate list is split out as `plugin_bin_dirs()` so the relationship can be asserted rather than assumed — the same shape as `run_python::python_cache_roots`, which exists because the package cache had this exact defect. Two tests: that the lookup searches the directory the installer writes, and that the legacy path is still searched. Verified non-vacuous: removing the registry lookup fails the first with exit 101. Three documentation pages carried this as a working recipe and now carry an `export PATH=...` workaround (softmata/horus-docs#16). Those lines should come out once this ships.
`restart_node` sent PauseNode immediately followed by ResumeNode and printed
"The scheduler will re-initialize this node on next tick". It did not. Both arms
only call `NodeControls::set_paused` on a shared atomic; the code that re-runs
`init()` is `reinit_pending_nodes`, which keys off `RegisteredNode::initialized`,
and outside registration nothing in production ever set that back to false — the
only two such assignments in the workspace are in `#[cfg(test)]` code.
Four places in the runtime asserted the behaviour anyway: the clap help
("re-initialize without killing scheduler"), `restart_node`'s doc comment, an
inline comment reading "Pause then resume triggers re-init in the scheduler",
and `reinit_pending_nodes`'s own comment, "set by control commands". No control
command set it. On a robot the operator restarts a wedged node, is told it
worked, and moves on.
Adds `ControlCommand::RestartNode { name }`, appended to the enum so a command
serialised by an older CLI still decodes. The scheduler collects restart
requests while draining the control topic — the loop holds a mutable borrow of
`control_topic` and clearing the flag needs `self.nodes` — then clears
`initialized` and lifts any operator pause, since a wedged node is often paused
before being restarted and would otherwise be skipped by the very check that
would have re-inited it. An unknown name logs rather than silently doing
nothing.
Tests drive the real control topic through `run_for`, not `tick_once`: the
control topic is created in `run_with_filter`, so under `tick_once` it is None
and no control command is polled at all. Worth knowing separately — operator
control reaches a scheduler only while it is in `run()`.
Verified non-vacuous: removing the RestartNode arm fails the test with exit 101
and `left: 1, right: 2`, which is precisely the pre-fix behaviour.
Closes #184 is separate; this addresses the restart defect found by the
bidirectional audit.
`consecutive_misses` is the number the `max_deadline_misses` ceiling and the whole degradation ladder compare against, and nothing reset it from the deadline check. The reset lived in `NodeTimingState::record_tick`, reachable only through `check_budget`, which gave three wrong answers: - RT executor: `tick_node` calls `check_tick_budget` unconditionally and the deadline check afterwards, so a node woken 5 ms late that executes in microseconds cleared the run its own miss had just started. The counter was pinned at 1 forever. Neither `warn_after: 3` nor the ceiling could fire for the failure mode `check_deadline_from_release` was written to catch -- the one `primitives.rs` calls dominant. - Main loop: the call sits inside the budget-violation branch, so a clean tick never reaches it and the count is a lifetime total. - A node with `.deadline()` and no `.tick_budget()` skips the budget block entirely, so nothing cleared it on either path. `node_builder` derives a deadline from a budget and never the reverse, so this is an ordinary shape, and for it the per-node consecutive ceiling was a lifetime total wearing a safety threshold's name -- exactly what the per-node ceiling was introduced to end. A budget overrun and a deadline miss are different failures, so a tick that stayed inside its budget is no evidence its deadline was met. The reset moves to `SafetyMonitor::record_deadline_met`, called wherever the deadline check comes back clean on both dispatch paths. Also: a miss now clears `recovery_counter`. `record_successful_tick` means "did not panic", and both callers reach it on a tick already recorded as a miss in the same pass, so a node at RateReduced that missed EVERY deadline still accumulated `recovery_ticks` of "success" and was restored to the rate it had just proved it could not hold. The field's own comment already called itself a consecutive run. Two existing tests asserted the old contract by calling `check_budget` with an in-budget duration to mean "recovered"; they now say `record_deadline_met`, which is the thing they were reaching for. Each fix was reverted in turn to confirm the tests catch it: the in-budget run reads 1 instead of 10, the no-budget scheduler node reads 10 instead of 0, the RT node stays at 5 after meeting its deadline, and the never-recovering node is declared recovered at tick 8.
`max_deadline_misses` is documented as the per-node consecutive ceiling that stops the robot, defaults to 100, and could not fire at that value -- or at any value above 20 -- in any shipped build. `DegradationAction::Kill` sets `is_stopped`, and a stopped node is skipped forever on both dispatch paths. So the ladder's terminal rung, hardcoded at 20 consecutive misses, permanently retired the node before its count could climb any higher: `consecutive_misses` froze at 20 and the ceiling was unreachable. Nothing warned. `max_deadline_misses(n)` only clamps with `n.max(1)`, and the sole setter for the ladder is `#[cfg(test)]`, so no user build could raise the rung either. Two of the runtime's own defaults contradicted each other, and the unit tests only ever exercised the ceiling at 3 and 5 -- both below 20 -- which is why the interaction went unnoticed. A hardcoded constant must not silently cap a documented safety threshold, so the rung moves: `kill_after` is raised to the ceiling when the ceiling is higher. The graduated stages below it (warn at 3, half rate at 5, isolate at 10) are untouched, and an isolated node keeps ticking -- every tick on the RT executor, on a probe interval on the main loop -- so it can still reach the ceiling. Where the two now coincide the e-stop wins, because both paths record the miss, which evaluates the ceiling, before evaluating the ladder. Note this changes what a chronically missing node does under the default configuration: it is no longer killed at 20 while the robot runs on, it e-stops the robot at 100. That is what the knob, its default, the builder doc and five documentation pages all say it does. Both tests assert the ORDER of the two terminal events rather than the counter -- at this level nothing removes the node, so the count climbs regardless and the e-stop would fire on its own; what killed the ceiling is the `Kill` that lands first. Reverting to the hardcoded rung fails them at 20 for both the configured ceiling of 25 and the default of 100.
There was a problem hiding this comment.
🔵 Needs a closer look
It changes safety-critical scheduler and timing enforcement behavior (including e-stop reachability), so it warrants final human review despite strong test coverage.
Pull request overview
Aligns horus runtime behavior with documented user-facing guarantees surfaced by the horus / horus-docs audit: plugin discovery for horus monitor, correctness of horus node restart, and safety/timing enforcement semantics around deadline misses and emergency-stop thresholds.
Changes:
- Fix
horus monitorplugin lookup by searching the installer’sPluginRegistry::global_bin_dir()(while keeping the legacy~/.horus/binfallback) and add an assertion test to prevent future drift. - Make
horus node restartactually re-runinit()by introducingControlCommand::RestartNodeand implementing scheduler-side handling to clearRegisteredNode::initialized. - Correct safety timing logic: consecutive deadline misses are reset only by a met deadline, recovery counters don’t advance on missed deadlines, and the degradation ladder no longer prevents
max_deadline_missesfrom being reachable.
File summaries
| File | Description |
|---|---|
| horus_manager/src/main.rs | Adds plugin_bin_dirs() and updates which_monitor_binary() to search the installer’s global bin dir first, plus tests to assert path agreement. |
| horus_manager/src/commands/node.rs | Updates restart_node to send RestartNode instead of PauseNode+ResumeNode. |
| horus_core/src/scheduling/control.rs | Extends ControlCommand with RestartNode and adds a human-readable description. |
| horus_core/src/scheduling/scheduler/mod.rs | Implements RestartNode handling; records record_deadline_met on clean deadline checks; updates max_deadline_misses docs. |
| horus_core/src/scheduling/safety_monitor.rs | Moves consecutive-miss reset semantics to record_deadline_met, fixes recovery counter behavior, and adjusts degradation policy so configured ceiling is reachable. |
| horus_core/src/scheduling/rt_executor.rs | Records record_deadline_met when RT ticks meet deadlines; adds RT-path regression test coverage for consecutive misses. |
| horus_core/src/scheduling/scheduler/tests.rs | Adds regression coverage for “deadline but no budget” consecutive-miss reset behavior in deterministic mode. |
| horus_core/tests/control_restart.rs | Adds integration tests ensuring RestartNode re-initializes the intended node without affecting others. |
Review details
Suppressed comments (1)
horus_core/tests/control_restart.rs:129
- Same flakiness risk here: a fixed sleep assumes the scheduler has performed the initial
init()within 400ms. Polling with a timeout makes the test resilient to transient slowness.
std::thread::sleep(std::time::Duration::from_millis(400));
assert_eq!(inits.load(Ordering::SeqCst), 1);
- Files reviewed: 8/8 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Benchmark gateBaseline: rolling window of 8 run(s) on Result: FAIL
|
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
40.0 ns | 29.0 ns | -27.5% | 65.0 ns | IMPROVED |
p95 |
50.0 ns | 50.0 ns | +0.0% | 100.0 ns | INCONCLUSIVE |
p99 |
69.0 ns | 60.0 ns | -13.0% | 169.0 ns | INCONCLUSIVE |
p99.9 |
80.0 ns | 80.0 ns | +0.0% | 330.0 ns | INCONCLUSIVE |
p99.99 |
— | 110.0 ns | — | — | N/A (samples) |
max |
15.92 µs | 13.40 µs | -15.8% | 31.85 µs | INCONCLUSIVE |
max_jitter |
15.91 µs | 13.39 µs | -15.8% | 31.83 µs | INCONCLUSIVE |
p99/median |
1.73x | 2.07x | +19.9% | 2.23x | INCONCLUSIVE |
p99.9/median |
2.00x | 2.76x | +37.9% | 3.00x | INCONCLUSIVE |
ns/msg |
170.3 ns | 165.8 ns | -2.7% | 220.3 ns | INCONCLUSIVE |
Adaptive_Imu@304B — 50000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
40.0 ns | 40.0 ns | +0.0% | 65.0 ns | INCONCLUSIVE |
p95 |
60.0 ns | 60.0 ns | +0.0% | 110.0 ns | INCONCLUSIVE |
p99 |
89.5 ns | 90.0 ns | +0.6% | 189.5 ns | INCONCLUSIVE |
p99.9 |
110.0 ns | 110.0 ns | +0.0% | 360.0 ns | INCONCLUSIVE |
p99.99 |
— | 140.0 ns | — | — | N/A (samples) |
max |
13.85 µs | 12.61 µs | -8.9% | 27.70 µs | INCONCLUSIVE |
max_jitter |
13.83 µs | 12.59 µs | -8.9% | 27.66 µs | INCONCLUSIVE |
p99/median |
2.24x | 2.25x | +0.6% | 2.80x | INCONCLUSIVE |
p99.9/median |
2.75x | 2.75x | +0.0% | 3.75x | INCONCLUSIVE |
ns/msg |
270.3 ns | 272.5 ns | +0.8% | 324.3 ns | INCONCLUSIVE |
Adaptive_JointCommand@928B — 50000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
80.0 ns | 80.0 ns | +0.0% | 105.0 ns | INCONCLUSIVE |
p95 |
100.0 ns | 100.0 ns | +0.0% | 150.0 ns | INCONCLUSIVE |
p99 |
130.0 ns | 130.0 ns | +0.0% | 230.0 ns | INCONCLUSIVE |
p99.9 |
160.0 ns | 160.0 ns | +0.0% | 410.0 ns | INCONCLUSIVE |
p99.99 |
— | 240.0 ns | — | — | N/A (samples) |
max |
15.53 µs | 13.14 µs | -15.4% | 31.07 µs | INCONCLUSIVE |
max_jitter |
15.48 µs | 13.10 µs | -15.4% | 30.97 µs | INCONCLUSIVE |
p99/median |
1.62x | 1.62x | +0.0% | 2.12x | INCONCLUSIVE |
p99.9/median |
2.00x | 2.00x | +0.0% | 3.00x | INCONCLUSIVE |
ns/msg |
335.0 ns | 332.0 ns | -0.9% | 402.0 ns | INCONCLUSIVE |
Adaptive_LaserScan@1480B — 10000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
89.0 ns | 89.0 ns | +0.0% | 114.0 ns | INCONCLUSIVE |
p95 |
110.0 ns | 110.0 ns | +0.0% | 160.0 ns | INCONCLUSIVE |
p99 |
150.0 ns | 150.0 ns | +0.0% | 250.0 ns | INCONCLUSIVE |
p99.9 |
170.0 ns | 170.0 ns | +0.0% | 420.0 ns | INCONCLUSIVE |
p99.99 |
— | 12.27 µs | — | — | N/A (samples) |
max |
13.15 µs | 15.97 µs | +21.4% | 26.30 µs | INCONCLUSIVE |
max_jitter |
13.09 µs | 15.92 µs | +21.6% | 26.18 µs | INCONCLUSIVE |
p99/median |
1.69x | 1.69x | +0.0% | 2.19x | INCONCLUSIVE |
p99.9/median |
1.91x | 1.91x | +0.0% | 2.91x | INCONCLUSIVE |
ns/msg |
689.0 ns | 688.3 ns | -0.1% | 826.8 ns | INCONCLUSIVE |
Topic_cross_process_oneway_paced_stream@1024B — 200000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
260.5 ns | 275.0 ns | +5.6% | 316.8 ns | INCONCLUSIVE |
p95 |
323.0 ns | 415.0 ns | +28.5% | 468.3 ns | INCONCLUSIVE |
p99 |
597.5 ns | 596.0 ns | -0.3% | 905.9 ns | INCONCLUSIVE |
p99.9 |
4.92 µs | 6.90 µs | +40.3% | 16.22 µs | INCONCLUSIVE |
p99.99 |
15.34 µs | 15.59 µs | +1.7% | 24.58 µs | INCONCLUSIVE |
max |
35.19 µs | 37.90 µs | +7.7% | 79.53 µs | INCONCLUSIVE |
max_jitter |
34.99 µs | 37.70 µs | +7.7% | 79.20 µs | INCONCLUSIVE |
p99/median |
2.33x | 2.16x | -7.3% | 3.91x | INCONCLUSIVE |
p99.9/median |
19.48x | 25.81x | +32.5% | 71.66x | INCONCLUSIVE |
ns/msg |
10.00 µs | 10.00 µs | +0.0% | 12.00 µs | INCONCLUSIVE |
Topic_cross_process_oneway_paced_stream@64B — 200000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
170.5 ns | 175.0 ns | +2.6% | 203.1 ns | INCONCLUSIVE |
p95 |
205.5 ns | 205.0 ns | -0.2% | 267.8 ns | INCONCLUSIVE |
p99 |
468.0 ns | 456.0 ns | -2.6% | 1.03 µs | INCONCLUSIVE |
p99.9 |
3.75 µs | 3.36 µs | -10.3% | 13.44 µs | INCONCLUSIVE |
p99.99 |
13.08 µs | 13.11 µs | +0.2% | 26.19 µs | INCONCLUSIVE |
max |
51.25 µs | 36.93 µs | -27.9% | 270.47 µs | INCONCLUSIVE |
max_jitter |
51.13 µs | 36.82 µs | -28.0% | 270.28 µs | INCONCLUSIVE |
p99/median |
2.70x | 2.66x | -1.4% | 5.43x | INCONCLUSIVE |
p99.9/median |
20.29x | 20.37x | +0.4% | 68.21x | INCONCLUSIVE |
ns/msg |
10.00 µs | 10.00 µs | +0.0% | 12.00 µs | INCONCLUSIVE |
Topic_cross_process_oneway_pingpong@1024B — 1000000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
278.0 ns | 310.0 ns | +11.5% | 305.8 ns | REGRESSED (blocking) |
p95 |
361.0 ns | 375.0 ns | +3.9% | 415.1 ns | INCONCLUSIVE |
p99 |
408.5 ns | 411.0 ns | +0.6% | 508.5 ns | INCONCLUSIVE |
p99.9 |
1.40 µs | 1.44 µs | +2.8% | 1.96 µs | INCONCLUSIVE |
p99.99 |
10.48 µs | 10.67 µs | +1.8% | 15.72 µs | INCONCLUSIVE |
max |
27.51 µs | 20.11 µs | -26.9% | 94.84 µs | INCONCLUSIVE |
max_jitter |
27.31 µs | 19.89 µs | -27.2% | 94.60 µs | INCONCLUSIVE |
p99/median |
1.45x | 1.33x | -8.2% | 1.95x | INCONCLUSIVE |
p99.9/median |
4.97x | 4.65x | -6.4% | 7.16x | INCONCLUSIVE |
ns/msg |
344.5 ns | 374.0 ns | +8.6% | 413.4 ns | INCONCLUSIVE |
Topic_cross_process_oneway_pingpong@64B — 1000000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
150.0 ns | 165.0 ns | +10.0% | 179.7 ns | INCONCLUSIVE |
p95 |
180.0 ns | 200.0 ns | +11.1% | 230.0 ns | INCONCLUSIVE |
p99 |
200.0 ns | 225.0 ns | +12.5% | 300.0 ns | INCONCLUSIVE |
p99.9 |
728.5 ns | 752.0 ns | +3.2% | 1.27 µs | INCONCLUSIVE |
p99.99 |
7.78 µs | 9.52 µs | +22.4% | 11.67 µs | INCONCLUSIVE |
max |
23.44 µs | 31.38 µs | +33.9% | 60.24 µs | INCONCLUSIVE |
max_jitter |
23.33 µs | 31.26 µs | +34.0% | 60.09 µs | INCONCLUSIVE |
p99/median |
1.34x | 1.36x | +1.8% | 1.84x | INCONCLUSIVE |
p99.9/median |
4.79x | 4.56x | -4.9% | 7.76x | INCONCLUSIVE |
ns/msg |
200.6 ns | 216.5 ns | +7.9% | 250.6 ns | INCONCLUSIVE |
Topic_cross_process_unpaced_queue_delay@64B — 200000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
147.5 ns | 155.0 ns | +5.1% | 197.9 ns | INCONCLUSIVE |
p95 |
205.0 ns | 205.0 ns | +0.0% | 267.3 ns | INCONCLUSIVE |
p99 |
230.5 ns | 245.0 ns | +6.3% | 330.5 ns | INCONCLUSIVE |
p99.9 |
3.18 µs | 2.55 µs | -19.8% | 13.64 µs | INCONCLUSIVE |
p99.99 |
9.26 µs | 8.87 µs | -4.2% | 13.89 µs | INCONCLUSIVE |
max |
38.40 µs | 37.75 µs | -1.7% | 80.93 µs | INCONCLUSIVE |
max_jitter |
38.30 µs | 37.65 µs | -1.7% | 80.76 µs | INCONCLUSIVE |
p99/median |
1.60x | 1.58x | -1.5% | 2.10x | INCONCLUSIVE |
p99.9/median |
23.14x | 16.45x | -28.9% | 108.60x | INCONCLUSIVE |
ns/msg |
112.7 ns | 118.9 ns | +5.5% | 162.7 ns | INCONCLUSIVE |
Blocking metrics: median, p99/median. Report-only: p95, p99, p99.9, p99.99, max, max_jitter, p99.9/median, ns/msg. Tail metrics past p99.9 are advisory on a shared runner — one host preemption owns max.
Raw benchmark output
----------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=13.9 ns window=0.430 s
measured window: minor_faults=0 major_faults=0 ctx_switches vol=84 invol=4
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
cross-core TSC offset theta_hat = +5.3 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7671 <-> pid 7681
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way, offset-free 1000000 225.3 300.6 400.7 1212.0 10218.7 20111.9 | 1000 98
round trip (RTT) 1000000 490.7 641.2 841.5 2735.2 20567.9 40253.7 | 999 100
RTT halved (one-way) 1000000 245.3 320.6 420.8 1367.6 10283.9 20126.9 | 999 100
leg A->B (apparent) 1000000 215.5 285.8 446.1 566.3 19330.5 39557.8 | 942 100
leg B->A (apparent) 1000000 195.1 315.3 465.8 545.9 9812.7 39947.9 | 956 100
responder turnaround 1000000 29.9 30.3 40.1 40.1 70.3 22691.4 | 283 91
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=40.1 ns window=0.732 s
measured window: minor_faults=0 major_faults=0 ctx_switches vol=143 invol=6
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
cross-core TSC offset theta_hat = -14.7 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7671 <-> pid 7683
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 114.9 165.2 445.7 3360.9 13108.5 40549.0 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=8.6 ns window=2.000 s
measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=12
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7671 <-> pid 7686
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 215.1 285.0 615.8 4983.8 15593.1 35930.7 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=20.4 ns window=2.000 s
measured backlog: 0.03 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=393 invol=10
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -14.7 ns (add to correct).
== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
!! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
!! time spent waiting in the ring. It is a backpressure/throughput figure.
!! Do not quote it as an IPC latency; that is the exact mistake the retired
!! `Topic_cross_process` metric shipped for months. Read `measured backlog`
!! below: ~1 message means the subscriber kept up and no queue formed.
json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7671 <-> pid 7688
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
publish->receive incl. queue 200000 114.9 155.0 245.3 2549.6 8870.9 42292.2 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=21.3 ns window=0.024 s
measured backlog: 1.30 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=5 invol=0
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).
SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
json metric key what it measures n median p99 p99.9 p99.99 max
------------------------------------------------------------------------------------------------------------------------------------------
Topic_cross_process_oneway_pingpong@64B ONE-WAY latency 1000000 160.3 225.3 721.3 9241.8 31377.5
Topic_cross_process_oneway_pingpong@1024B ONE-WAY latency 1000000 300.6 400.7 1212.0 10218.7 20111.9
Topic_cross_process_oneway_paced_stream@64B ONE-WAY latency 200000 165.2 445.7 3360.9 13108.5 40549.0
Topic_cross_process_oneway_paced_stream@1024B ONE-WAY latency 200000 285.0 615.8 4983.8 15593.1 35930.7
Topic_cross_process_unpaced_queue_delay@64B QUEUE DELAY (not latency) 200000 155.0 245.3 2549.6 8870.9 42292.2
------------------------------------------------------------------------------------------------------------------------------------------
Instrumentation is INCLUDED in every figure and never subtracted.
`Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
`RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
the row to compare against any competitor publishing a halved ping-pong.
`Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
however long the message waited in the ring, so it scales with the achieved backlog
(printed in its scenario block) and is NOT a latency.
A figure without its payload size, ring depth and topology is not comparable to anything.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! pingpong (1024 B) `one-way, offset-free`: p99.99 rests on 98 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.30 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-2.json
keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
will report it as missing and fail the gate until the window is refreshed —
that is intended, and is why the key was not silently reused.
Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
7 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║ HORUS Robotics Message Types Benchmark ║
╠══════════════════════════════════════════════════════════════════╣
║ Testing real-world robotics message latency (REP 2014) ║
╚══════════════════════════════════════════════════════════════════╝
Platform: AMD EPYC 7763 64-Core Processor (4 cores)
Iterations: 50000
Warmup: 5000
╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type │ Size (bytes) │ Typical Rate │ Use Case ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 1000+ Hz │ Velocity control commands ║
║ Imu │ 296 │ 500+ Hz │ IMU sensor fusion ║
║ LaserScan │ 1480 │ 10-40 Hz │ 2D lidar navigation ║
║ JointCommand │ 1032 │ 500+ Hz │ Manipulator control ║
╚═════════════════════════════════════════════════════════════════════════════╝
[Topic] Running benchmarks — timing tx.send() enqueue cost only
(cross-thread consumer drains in the background; no sample
waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
CmdVel 16 bytes │ median: 29ns │ p99: 60ns │ CV: 3.2160
Imu 304 bytes │ median: 40ns │ p99: 90ns │ CV: 1.7597
LaserScan 1480 bytes │ median: 89ns │ p99: 140ns │ CV: 2.1870
JointCommand 928 bytes │ median: 80ns │ p99: 130ns │ CV: 1.3849
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message │ Size │ Median (ns) │ p99 (ns) │ CV │ Misses │ Sustained rate ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 29 │ 60 │ 3.2160 │ 0 │ 6.03 M msg/s ║
║ Imu │ 304 │ 40 │ 90 │ 1.7597 │ 0 │ 3.68 M msg/s ║
║ LaserScan │ 1480 │ 89 │ 140 │ 2.1870 │ 0 │ 1.46 M msg/s ║
║ JointCommand │ 928 │ 80 │ 130 │ 1.3849 │ 0 │ 3.02 M msg/s ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.
╔══════════════════════════════════════════════════════════════════╗
║ REAL-TIME SUITABILITY ANALYSIS ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop ║
║ guarantees: a real loop also pays the receive side, the node's ║
║ compute, and the scheduler's jitter, none of which is measured ║
║ here. The headroom is printed so a pass by four orders of ║
║ magnitude cannot be read as a tight result. ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel (1kHz control): ~ PASS by 16667x — gate not discriminating (p99=60ns vs 1000000ns)
║ CmdVel (10kHz control): ~ PASS by 1667x — gate not discriminating (p99=60ns vs 100000ns)
║ Imu (500Hz fusion) : ~ PASS by 22222x — gate not discriminating (p99=90ns vs 2000000ns)
║ LaserScan (40Hz lidar) : ~ PASS by 178571x — gate not discriminating (p99=140ns vs 25000000ns)
║ JointCmd (500Hz control): ~ PASS by 15385x — gate not discriminating (p99=130ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝
Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-2.json
===== repetition 3/3 =====
HORUS cross-process benchmark — ONE-WAY LATENCY (headline) + unpaced queue delay
Two OS processes on two distinct physical cores over the real Topic SHM transport.
Every handle holds exactly one role, so the `role == Both` same-thread fast path that
produces the repo's headline send-cost figure is never taken. Round-trip rows are
labelled where they are a halved round trip. See the module header for the clock
decomposition and for why the `Topic_cross_process` JSON key was retired.
MACHINE — every nanosecond below is conditional on this block
------------------------------------------------------------------------------------------------------------------------------------------
cpu model : AMD EPYC 7763 64-Core Processor (2 physical / 4 logical cores)
kernel : 6.17.0-1022-azure (x86_64)
benchmark cores : cpu3 = process A (pinger / subscriber), cpu1 = process B (responder / publisher) [DistinctPhysical]
governor : cpu3=n/a cpu1=n/a driver=n/a
scaling range : n/a .. n/a (hw max n/a) turbo_disabled=unknown
clocksource : tsc tsc flags: [tsc, rdtscp, constant_tsc, tsc_reliable, nonstop_tsc]
TSC rate (calibrated) : 2445.515 MHz (0.4089 ns/cycle) — a WALL clock; NOT the core clock
clock instrumentation : 49 cycles = 20.0 ns for serialize()+rdtsc()+rdtscp(), INCLUDED in every sample and never subtracted
transparent_hugepage : [always] madvise never
sched_rt_runtime/period: 950000 / 1000000 us perf_event_paranoid=4
isolcpus=(none) nohz_full=(none) smt_active=1 mlockall(MCL_CURRENT)=true
------------------------------------------------------------------------------------------------------------------------------------------
running ping-pong (one-way latency), 64 B payload ...
running ping-pong (one-way latency), 1024 B payload ...
running paced stream (one-way latency), 64 B payload ...
running paced stream (one-way latency), 1024 B payload ...
running unpaced stream (QUEUE DELAY, not latency), 64 B payload ...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! READ THIS BEFORE QUOTING ANY NUMBER BELOW
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! pingpong (1024 B) `one-way, offset-free`: p99.99 rests on 99 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.29 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7722
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way, offset-free 1000000 125.1 165.2 225.3 751.6 9522.3 512222.7 | 1000 100
round trip (RTT) 1000000 290.3 360.7 490.7 1553.0 19455.6 1024546.0 | 997 100
RTT halved (one-way) 1000000 145.2 180.3 245.3 776.5 9727.8 512273.0 | 997 100
leg A->B (apparent) 1000000 125.5 165.6 255.6 1197.7 15604.1 1024230.7 | 990 100
leg B->A (apparent) 1000000 114.9 155.0 244.9 275.2 10103.4 56918.9 | 953 100
responder turnaround 1000000 29.9 30.3 40.1 40.1 170.5 23853.5 | 501 87
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=13.9 ns window=0.439 s
measured window: minor_faults=0 major_faults=0 ctx_switches vol=86 invol=3
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
cross-core TSC offset theta_hat = +5.3 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7724
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way, offset-free 1000000 220.4 310.4 410.5 1442.2 10674.6 18649.2 | 1000 99
round trip (RTT) 1000000 480.9 651.4 851.4 2965.4 21419.2 37328.3 | 999 100
RTT halved (one-way) 1000000 240.4 325.7 425.7 1482.7 10709.6 18664.2 | 999 100
leg A->B (apparent) 1000000 215.5 296.1 455.9 566.3 20793.2 36943.1 | 990 100
leg B->A (apparent) 1000000 184.8 315.3 455.5 535.7 10303.8 32624.6 | 962 100
responder turnaround 1000000 29.9 30.3 40.1 40.1 100.2 21479.3 | 317 100
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=32.7 ns window=0.748 s
measured window: minor_faults=0 major_faults=0 ctx_switches vol=147 invol=5
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
cross-core TSC offset theta_hat = -9.6 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7726
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 114.9 175.0 465.8 3741.5 14872.1 35399.5 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=8.6 ns window=2.000 s
measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=13
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7728
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 204.9 275.2 575.7 7097.9 17947.6 37904.1 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=20.9 ns window=2.000 s
measured backlog: 0.03 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=393 invol=11
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -9.6 ns (add to correct).
== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
!! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
!! time spent waiting in the ring. It is a backpressure/throughput figure.
!! Do not quote it as an IPC latency; that is the exact mistake the retired
!! `Topic_cross_process` metric shipped for months. Read `measured backlog`
!! below: ~1 message means the subscriber kept up and no queue formed.
json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7731
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
publish->receive incl. queue 200000 104.7 155.0 255.2 7959.5 9462.2 37754.0 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=21.3 ns window=0.024 s
measured backlog: 1.29 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=5 invol=1
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).
SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
json metric key what it measures n median p99 p99.9 p99.99 max
------------------------------------------------------------------------------------------------------------------------------------------
Topic_cross_process_oneway_pingpong@64B ONE-WAY latency 1000000 165.2 225.3 751.6 9522.3 512222.7
Topic_cross_process_oneway_pingpong@1024B ONE-WAY latency 1000000 310.4 410.5 1442.2 10674.6 18649.2
Topic_cross_process_oneway_paced_stream@64B ONE-WAY latency 200000 175.0 465.8 3741.5 14872.1 35399.5
Topic_cross_process_oneway_paced_stream@1024B ONE-WAY latency 200000 275.2 575.7 7097.9 17947.6 37904.1
Topic_cross_process_unpaced_queue_delay@64B QUEUE DELAY (not latency) 200000 155.0 255.2 7959.5 9462.2 37754.0
------------------------------------------------------------------------------------------------------------------------------------------
Instrumentation is INCLUDED in every figure and never subtracted.
`Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
`RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
the row to compare against any competitor publishing a halved ping-pong.
`Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
however long the message waited in the ring, so it scales with the achieved backlog
(printed in its scenario block) and is NOT a latency.
A figure without its payload size, ring depth and topology is not comparable to anything.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! pingpong (1024 B) `one-way, offset-free`: p99.99 rests on 99 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.29 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-3.json
keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
will report it as missing and fail the gate until the window is refreshed —
that is intended, and is why the key was not silently reused.
Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
7 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║ HORUS Robotics Message Types Benchmark ║
╠══════════════════════════════════════════════════════════════════╣
║ Testing real-world robotics message latency (REP 2014) ║
╚══════════════════════════════════════════════════════════════════╝
Platform: AMD EPYC 7763 64-Core Processor (4 cores)
Iterations: 50000
Warmup: 5000
╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type │ Size (bytes) │ Typical Rate │ Use Case ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 1000+ Hz │ Velocity control commands ║
║ Imu │ 296 │ 500+ Hz │ IMU sensor fusion ║
║ LaserScan │ 1480 │ 10-40 Hz │ 2D lidar navigation ║
║ JointCommand │ 1032 │ 500+ Hz │ Manipulator control ║
╚═════════════════════════════════════════════════════════════════════════════╝
[Topic] Running benchmarks — timing tx.send() enqueue cost only
(cross-thread consumer drains in the background; no sample
waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
CmdVel 16 bytes │ median: 29ns │ p99: 60ns │ CV: 1.9834
Imu 304 bytes │ median: 40ns │ p99: 90ns │ CV: 1.9723
LaserScan 1480 bytes │ median: 89ns │ p99: 160ns │ CV: 1.2659
JointCommand 928 bytes │ median: 80ns │ p99: 130ns │ CV: 1.7959
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message │ Size │ Median (ns) │ p99 (ns) │ CV │ Misses │ Sustained rate ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 29 │ 60 │ 1.9834 │ 0 │ 6.03 M msg/s ║
║ Imu │ 304 │ 40 │ 90 │ 1.9723 │ 0 │ 3.66 M msg/s ║
║ LaserScan │ 1480 │ 89 │ 160 │ 1.2659 │ 0 │ 1.44 M msg/s ║
║ JointCommand │ 928 │ 80 │ 130 │ 1.7959 │ 0 │ 2.99 M msg/s ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.
╔══════════════════════════════════════════════════════════════════╗
║ REAL-TIME SUITABILITY ANALYSIS ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop ║
║ guarantees: a real loop also pays the receive side, the node's ║
║ compute, and the scheduler's jitter, none of which is measured ║
║ here. The headroom is printed so a pass by four orders of ║
║ magnitude cannot be read as a tight result. ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel (1kHz control): ~ PASS by 16667x — gate not discriminating (p99=60ns vs 1000000ns)
║ CmdVel (10kHz control): ~ PASS by 1667x — gate not discriminating (p99=60ns vs 100000ns)
║ Imu (500Hz fusion) : ~ PASS by 22222x — gate not discriminating (p99=90ns vs 2000000ns)
║ LaserScan (40Hz lidar) : ~ PASS by 156250x — gate not discriminating (p99=160ns vs 25000000ns)
║ JointCmd (500Hz control): ~ PASS by 15385x — gate not discriminating (p99=130ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝
Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-3.json
Gated against a rolling window of recent main runs. Each metric's band is max(relative, center + k·sigma_observed, absolute floor), so it is derived from the noise the baseline itself measured — where a band is printed wider than the change you care about, this run did not detect a regression and could not have. Thresholds and their reasoning: RegressionPolicy in benchmarks/src/output.rs.
Commit: 5393b83
`safety.on_link_lost`, `import` and `deny_export` are safety controls. The 2026-07-30 audit found the whole `[network]` table parsed and then read by nothing, and `apply_network_config` was written to fix that -- then wired into ONE of the five paths that spawn a run child. That one is `execute_multiple_files`, reached only when `horus run` is handed more than one file. A bare `horus run`, a `horus.toml` run, a directory run and a `[workspace]` run all resolve to a single executable. On those the child environment came from `build_child_env()`, which sets PATH, LD_LIBRARY_PATH and PYTHONPATH and nothing else, so `NetConfig::default()` read an environment nobody had populated: `import` stayed `Auto`, `deny_export` stayed empty, and the documented comms-loss safe state never armed. The gap the two existing comments describe was still open on every invocation a real single-binary robot makes. `apply_network_config` becomes `network_env() -> Vec<(String, String)>`, which `build_child_env` now extends with -- so every single-executable path picks it up from the function they all already call, and no future path can quietly miss it. `execute_workspace` builds its `Command` by hand rather than through `build_child_env`, so it gets an explicit `apply_network_config` call. The operator still wins: `network_env` skips any key already present in the environment, so an explicit `HORUS_NET_*=...` beats the manifest on every path. That is asserted too. Reverting the `build_child_env` line gives `left: None` for HORUS_NET_ON_LINK_LOST -- the safety control simply absent. `horus launch` still performs no translation; that is a separate gap and the documentation says so.
`start_log_file_drain` was called by no shipped code path. A tree-wide grep for the symbol returned four hits: the definition, its re-export, and two lines of one integration test that invoked it by hand. So `HORUS_LOG_FILE=1 horus run` wrote no `.horus/logs/horus.log`, and `HORUS_LOG_DIR`, `HORUS_LOG_MAX_SIZE` and `HORUS_LOG_MAX_FILES` were dead with it -- while the test that proves the drain works kept it green. The env guard lives inside the function precisely so a call site can be unconditional, and the function's own doc-comment calls the variable an opt-in, which is only true if something always calls it. `Scheduler::new` already does this kind of env-driven wiring twice, for HORUS_NET_ENABLED and HORUS_RECORD_SESSION, so the drain goes beside them. A process that does not set the variable pays one env lookup and spawns nothing. Once per process, via a `OnceLock`: a program may build several schedulers, and each drain thread tracks its own index into the same global ring, so N of them would write every entry N times into one file. This also keeps the promise for C++ and Python, which the doc-comment's "or calling this function directly" escape hatch could not: the symbol has no FFI export anywhere in the tree. Both bindings construct a core `Scheduler` (horus_cpp/src/scheduler_ffi.rs:26, horus_py/src/scheduler.rs:882), so both now get the drain. The test does not call the drain. It sets the variable, builds a `Scheduler` the way any program does, and asserts a log file appears; it is alone in its binary because it writes process-global environment. Reverting the wiring leaves the file absent and it fails. An operator who sets a documented variable to get a post-incident log and finds an empty disk is the exact failure that variable exists to prevent.
There was a problem hiding this comment.
🟡 Changes recommended
It introduces a real behavioral edge case in start_log_file_drain_once (caching the disabled state) and adds tests that can be environment-dependent/flaky unless process-global env is isolated and restored.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 12/12 changed files
- Comments generated: 3
- Review effort level: Lite
`messages_total` is not a metric. `SubscriptionFreshness::refresh` watches it as the "new data exists" signal behind `.subscribe_with_timeout()`, and its answer drives `StalePolicy::SafeState` and `StalePolicy::Stop`. Only `RingTopic::send` bumped it. `Topic::try_send` and `Topic::send_blocking` go straight to the ring, so a topic published entirely through either read as 0 messages and 0 Hz while carrying every message -- and could safe-state or halt a subscriber node whose data was arriving normally. That is exactly the failure the counter was introduced to fix, reintroduced for backpressure-aware publishers, and these are the two APIs the documentation points users to on critical topics. The runtime states the invariant in both places that consume the counter: "every publisher increments `messages_total` in the topic's SHM header on every send, so watching that counter is a true 'new data exists' signal" (scheduling/types.rs) and "incremented on every `send()` regardless of backend" (header.rs). Neither is true. Bumped once per public publish call, not inside `RingTopic::try_send`, which `send_lossy_retry` calls up to ~70 times for one message. `try_send` counts before the call, as `send()` does: an attempt count, and a publisher hammering a full ring is alive. `send_blocking` does NOT count a `NoBackpressure` refusal -- that is returned before the send is even attempted, so nothing reached the ring, and counting it would report a healthy flow on a topic delivering nothing. A `Timeout` did engage the ring, so it counts like `send()`'s dropped message does. Reverting both bumps reads 0 for a topic that carried ten messages.
A topic holds at most 16 participants. `register_role` builds a precise
error for the one that does not fit -- it names the limit, the count, and
what to do about it -- and all three call sites threw it away:
`recv_uninitialized` and `read_latest` were `if ....is_err() { return
None }`, `send_uninitialized` the same with `Err(msg)`. Those are the only
`ensure_consumer`/`ensure_producer` call sites in the tree, so that
message reached nothing, ever.
Because the role stays `Unregistered`, every later `recv()` re-enters the
same cold path and fails identically: `None` forever, from a subscriber
indistinguishable from an idle one.
Nothing anywhere defends the silence, and the identical 16-slot wall one
layer down is handled the opposite way on purpose -- rate-limited
`tracing::warn!` under "COMM-H1: never let 'all 16 slots live' become a
SILENT no-comms endpoint (worse than the pre-fix loud panic)". The rate
limiter for it sits in the same file, unused by this path. Meanwhile
header.rs defends a separate design decision by claiming the residual cost
"surfaces as the existing 'No available participant slots' error. Loud,
and it says what to do" -- a guarantee the code did not provide.
All three sites now report through that same limiter, so a hot loop warns
once a minute per topic rather than once ever.
The limiter is also keyed by reason now, not just topic. A topic can hit
two different walls -- its FanoutShm endpoint bitmap and the participant
table -- with different causes and different fixes, and one bucket let
whichever fired first silence the other for a minute. That is the masking
the per-topic keying was introduced to prevent, one level down.
The test fills the table from 20 threads (the participant-count leak is
what lets one process do that), asserts it really is full before asserting
anything about diagnosis, and captures `tracing` events through a
hand-rolled `Subscriber` so it needs no new dependency. Reverting the fix
prints nothing at all: "Warnings seen: []".
There was a problem hiding this comment.
🟡 Changes recommended
There are a couple of concrete issues in the new logging/diagnostic paths (log warning formatting artifacts, and start_log_file_drain_once memoizing “disabled” forever if first called before HORUS_LOG_FILE is set) that should be corrected before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
horus_core/src/communication/topic/dispatch.rs:2640
- The warning message contains a large run of spaces ("receive nothing"), which will show up verbatim in logs and makes the message harder to read/search. This looks like an accidental formatting artifact from line wrapping.
if should_report_endpoint_exhaustion(topic.name(), Exhausted::ParticipantTable) {
tracing::warn!(
"topic '{}': this subscriber is NOT registered and will receive nothing — {}",
topic.name(),
e
);
horus_core/src/core/log_buffer.rs:660
OnceLock<Option<...>>memoizes the first call result. IfScheduler::new()runs once beforeHORUS_LOG_FILEis set (sostart_log_file_drain()returnsNone), later schedulers in the same process can never start the drain even when the variable is set. That contradicts the “if HORUS_LOG_FILE asks for it” contract and can make log-file output depend on initialization order in long-lived processes/tests.
pub fn start_log_file_drain_once() -> bool {
LOG_FILE_DRAIN.get_or_init(start_log_file_drain).is_some()
}
- Files reviewed: 15/15 changed files
- Comments generated: 1
- Review effort level: Lite
`send_blocking_pod` ended in `.is_ok()`, discarding which `SendBlockingError` came back, and both arms then reported `HorusTimeoutError`. So the one error horus_core writes specifically to be un-retryable arrived in Python as the one error callers retry. The core is explicit about the distinction, twice: "Refusing is deliberate: reporting success for a delivery nobody guaranteed is worse on an e-stop path than failing loudly. It does not clear on retry -- it lasts as long as the topology that caused it." The variant carries a four-line message naming the fix. The binding replaced it with the exception whose own doc-comment means the opposite -- "the buffer stayed full for the whole timeout" -- and its private helper asserted, falsely, that a `false` return meant the ring stayed full. This is easy to reach and permanent: `detect_optimal_backend` resolves any POD topic with a second subscriber to `PodShm`, so the moment a `horus topic echo` or a monitor attaches to a POD e-stop topic, every `send_blocking` starts raising forever -- as a "timeout", on the path the method is documented for. The variant is now threaded out of both arms and mapped to a new `horus.HorusBackpressureError` carrying the core's own remediation text. `HorusTimeoutError` keeps `Timeout` alone. Also exposes `Topic.provides_backpressure`, which the core's doc-comment instructs callers to assert on after the first send and which Python had no way to ask. It is answered through a new `horus_core::communication::topic::backend_provides_backpressure`, keyed by backend NAME because a name is all a binding can see -- and `provides_backpressure()` is now defined in terms of it too, so the two cannot disagree. A test walks every `BackendMode` through an exhaustive match, so a new backend stops compiling rather than silently defaulting to "no backpressure". Reverting the mapping fails the test at "must raise HorusBackpressureError".
There was a problem hiding this comment.
🟡 Changes recommended
It introduces at least one likely CI-breaking Rust unit test dependency on a live Python interpreter (and minor log-message formatting issues), and the PR description should be updated to reflect the expanded user-facing scope.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (3)
Previously missed (1) — in code that hasn't changed since the last review.
horus_core/src/scheduling/scheduler/mod.rs:629
- PR title/description says this PR fixes four runtime defects, but the diff also includes additional user-facing changes (e.g., wiring HORUS_LOG_FILE via Scheduler::new, new HorusBackpressureError + provides_backpressure in Python, and topic staleness/endpoint-exhaustion reporting). Please update the PR description to enumerate these extra fixes/behavior changes so reviewers and release notes don’t miss the expanded scope.
horus_core/src/communication/topic/dispatch.rs:1895
- The warning message contains a large run of extra spaces ("deliver nothing"), which makes logs harder to read and looks like an accidental formatting artifact. Collapse it to a single space.
"topic '{}': this publisher is NOT registered and will deliver nothing — {}",
horus_core/src/communication/topic/dispatch.rs:2637
- The warning message contains a large run of extra spaces ("receive nothing"), which makes logs harder to read and looks like an accidental formatting artifact. Collapse it to a single space.
"topic '{}': this subscriber is NOT registered and will receive nothing — {}",
- Files reviewed: 20/20 changed files
- Comments generated: 1
- Review effort level: Lite
`_setup_topics` documents a three-step type resolution whose middle step,
"auto-detect from SHM header", was unreachable code. It did:
probe = Topic(topic_name, 64)
type_name = getattr(probe, 'type_name', None)
Both halves were dead. `PyTopic` exposes no `type_name` — its getters are
`name`, `is_network_topic`, `endpoint`, `backend_type` — so the value was
always `None`. And the probe could not have worked even with that
attribute: `Topic(name, capacity)` builds a `Topic<GenericMessage>`, and
attaching 4364-byte GenericMessage to an existing typed ring is refused by
the size check in `RingTopic::new`, so it always raised into a bare `except
Exception`. Step 2 therefore never ran for any topic that has ever existed,
and the ~70-entry `_TYPE_NAME_MAP` built for it had exactly one reader,
inside the unreachable branch.
The size check is correct and stays: its comment records that a name-based
exemption once let a `Topic<GenericMessage>` join a `Topic<CmdVel>` ring and
"write megabytes past the end of the mapping". The question just cannot be
answered by attaching -- opening requires choosing a type, which is what the
caller is trying to find out.
So: `horus_core::communication::peek_topic_type_name(name)` reads the
header without attaching, exposed to Python as `_horus.peek_topic_type`.
`_detect_topic_type` calls it and maps through `_TYPE_NAME_MAP`, which now
has a reachable reader.
Second half: when the ring exists and holds a type this build cannot name,
`_setup_topics` raises with the topic, the ring's type and the concrete fix
(`subs=[Sub('scan', LaserScan)]`) instead of falling through to a generic
topic and letting the runtime's byte-count RuntimeError out -- which named
neither `subs=`, nor the type the ring holds, nor what to do.
Auto-detection was only ever scoped to subscriptions; the `pubs=` case is
unchanged, since a publisher creates the ring and there is nothing to read.
Tests both ends. The Rust test asserts the old probe's refusal still
happens (it must) and that the peek answers anyway; reverting the peek
gives `left: None`, which is what the old probe returned for every topic.
The Python test drives a real `Node` with a bare `subs=` string against a
typed ring; restoring the old probe fails it.
There was a problem hiding this comment.
🔵 Needs a closer look
A new unit test is environment-dependent (fails if HORUS_NET_* is set), and there are also formatting issues in newly added warning strings that should be corrected.
Review details
Suppressed comments (3)
Previously missed (1) — in code that hasn't changed since the last review.
horus_core/src/communication/topic/dispatch.rs:2640
- This warning string contains a large run of spaces before "nothing"; that looks accidental and will produce messy log output.
horus_manager/src/commands/run/mod.rs:1329
- This test asserts a precondition that certain HORUS_NET_* env vars are not set in the process environment; that makes it environment-dependent and can fail on CI or developer machines. Prefer temporarily clearing and restoring those variables within the test so it is hermetic.
assert!(
preset.is_empty(),
"precondition: {:?} set in this environment would override the manifest",
preset
);
horus_core/src/communication/topic/dispatch.rs:1898
- The warning string has a large run of spaces before "nothing", which will render oddly in logs and makes the message harder to read/search.
tracing::warn!(
"topic '{}': this publisher is NOT registered and will deliver nothing — {}",
topic.name(),
e
);
- Files reviewed: 23/23 changed files
- Comments generated: 0 new
- Review effort level: Lite
**`post_test` skipped teardown exactly when teardown matters.** It sat after an early `?` in the `Commands::Test` arm, so a failing `horus test` never ran it, while `post_run` and `post_build` go through `run_teardown_hooks` and run on both paths deliberately: "Skipping it when the run failed would skip it in exactly the case where the hardware was most likely left live." The hooks contract file calls `test` "the only phase with teardown" and then pinned only the symmetry of the KEY NAMES, so `post_test` kept success-only semantics while the other two were given real teardown semantics. A `post_test = ["release-bench"]` was skipped precisely when the tests aborted mid-run. The new guard reads the dispatch in main.rs rather than the table: every `post_` phase must go through `run_teardown_hooks` and none through the plain `run_hooks`. Reverting the arm fails it on both halves. **`horus doctor` recommended a variable that does nothing.** It printed "use --net or HORUS_NET=1 to enable" and reported networking from `HORUS_NET`, which nothing else in the tree reads. Following the advice changed nothing except that `doctor` then printed "horus_net: enabled", confirming a state that did not exist -- worse than inert. main.rs already recorded this exact confusion when it fixed `--net`; the diagnostic handing users the broken half of the same remedy was not fixed with it. `doctor` now asks the question the BUILD asks, through a shared `cargo_gen::net_capability_requested`, and names only routes that work. `HORUS_NET` is retired: no reader remains. **`horus install --plugin -t <workspace>` threw the name away.** The call site collapsed it to `let local = target.is_some()`, and what was left called `detect_or_select_workspace` -- which resolves the CURRENT DIRECTORY's workspace, or prints a menu and blocks on stdin. Both are things `horus install` documents it never does, and with no tty the read returns empty and the command dies with "Invalid selection". The package half of the same flag already resolved the name correctly; the bool was a vestige of a `--local` flag that no longer exists, still named in the doc-comment. The ablation shows how completely the name vanished: with the bug the install proceeds past the target entirely and fails with an unrelated "no package registry is configured".
There was a problem hiding this comment.
🔵 Needs a closer look
It changes safety-critical scheduling/degradation behavior and core runtime/IPC semantics across multiple crates, warranting final human review despite strong test coverage.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
horus_core/src/communication/topic/dispatch.rs:2640
- The warning message has a large run of literal spaces ("receive nothing") which will show up in logs and looks like an accidental formatting artifact. It should be a single space so the message is readable and stable for any log parsing.
horus_core/src/communication/topic/dispatch.rs:1896
- The warning message has a large run of literal spaces ("deliver nothing") which will show up in logs and looks like an accidental formatting artifact. It should be a single space so the message is readable and stable for any log parsing.
tracing::warn!(
"topic '{}': this publisher is NOT registered and will deliver nothing — {}",
topic.name(),
- Files reviewed: 27/27 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
The installer’s rust-version gate still fails closed on awk errors despite being documented to fail open, and the new shm_base_dir_for helper should sanitize its input to avoid path-escape footguns.
Review details
Suppressed comments (2)
Previously missed (1) — in code that hasn't changed since the last review.
install.sh:571
- The rust-version gate is documented here as needing to “fail OPEN”, but
if ! awk ...; thenwill still fail CLOSED on any awk error (e.g., awk missing/unusable, unexpected version format), aborting the install with “toolchain too old” without having established that. To actually fail open, only treat awk exit status 1 as “too old”; treat other non-zero statuses as “unknown” and skip the check.
horus_sys/src/shm/mod.rs:261
shm_base_dir_for(namespace)accepts an arbitrary string and feeds it directly intoPathBuf::join(format!("horus_{namespace}")). If a caller passes an absolute path or path separators,joincan escapeshm_parent_dir()(absolute path) or create nested paths, which is a footgun for any future use that forwards user input (e.g.,horus clean). Since this module already hassanitize_namespace, consider applying it here as well so the named-namespace helper is as safe asgenerate_namespace().
- Files reviewed: 72/72 changed files
- Comments generated: 0 new
- Review effort level: Lite
`every_env_var_the_code_reads_is_documented` caught HORUS_COMPUTE_DIAG_CHILD, which is correct — it is a HORUS_* variable read from src/. It is the child half of the test that a compute-only program still drains its RT diagnostics to stdout, read inside a `#[cfg(test)]` module and unreachable from a shipped binary, so it goes in TEST_ONLY beside HORUS_CPP_FFI_EMIT_CHILD rather than into the user-facing environment-variables page.
There was a problem hiding this comment.
🔵 Needs a closer look
A few correctness/safety issues remain in changed code paths (fish prefix export update behavior, unsanitized namespace-to-path mapping, and one remaining signal-exit-code propagation site), and they should be fixed before approval.
Review details
Suppressed comments (4)
install.sh:961
- In the fish rc path, this only checks whether any
set -gx HORUS_PREFIXline exists, and leaves a stale prefix untouched on reinstall/upgrade. That means a moved prefix can keep exporting the old tree in fish shells, unlike the POSIX branch which is value-aware and replaces stale values.
horus_sys/src/shm/mod.rs:261 shm_base_dir_for()currently interpolates the rawnamespaceinto the path without sanitizing. That can drift from the actual on-disk namespace (which is alwayssanitize_namespaced viashm_namespace()), and it also allows path separators /..components to escape the intended parent dir when used by cleanup code.
horus_manager/src/commands/run/mod.rs:335- This still collapses signal deaths into exit code 1 on Unix. Elsewhere in this PR you introduced
error_wrapper::shell_exit_code()specifically so 128+N exits (e.g. SIGKILL→137) and related diagnostics/hints stay reachable; this call site should use it as well.
horus_manager/src/commands/doctor.rs:1820 - This status line contains a large run of literal spaces before
enable = ..., which will render as an awkward gap inhorus doctoroutput. It looks unintentional and should be a single space.
- Files reviewed: 73/73 changed files
- Comments generated: 0 new
- Review effort level: Lite
…ostics
`RtExecutor::stop` ends with a final drain of the RT diagnostic ring, and says
why:
Without it a run shorter than `RT_DIAG_POLL` — or one whose last deadline
miss came in the final milliseconds, which is the interesting case — would
exit with its diagnostics still sitting in the ring.
Compute, Event and AsyncIo had no such flush, though all three call
`start_rt_diag_drain` and all three queue through `honor_safe_state_request`.
So for a compute-only, event-only or async-only program the guarantee simply
did not hold: anything queued inside the last 25 ms — "PANICKED in
enter_safe_state", a final safing failure — was still in the ring when the
process exited. The same class partition that has caught this code repeatedly:
fixed on the RT path, left on the other three.
It also makes an existing warning true. When the drain thread cannot be
spawned, `start_rt_diag_drain` tells the operator diagnostics "will only be
flushed at shutdown" — with no shutdown flush on three of the four executors,
they were not flushed anywhere.
The drain is now one named `flush_rt_diag()` called from all four `stop()`
paths.
Found via a CI failure in
`a_compute_only_program_still_prints_its_safing_diagnostics`, which slept 200 ms
and assumed the 25 ms background drainer would have been scheduled inside that
window. On a loaded runner it was not, and the test reported "no drain thread
was running" against code that was working. With the flush the line is emitted
synchronously inside `stop()`, so the sleep is gone and no timing window is
involved.
The flush itself is deliberately NOT asserted on. The background drainer's
first poll runs when its thread starts — the same few microseconds as the
`rt_diag` under test — so an ordering assertion passes whether or not `stop()`
flushes. I wrote that assertion, ablated the flush, watched it pass three times
out of three, and removed it: a guard that cannot fail for the reason it states
is worse than no guard. Isolating the flush needs a way to suppress that
thread, which does not exist today. The test keeps the claim it can support —
the line reaches stdout — and the reasoning for the flush lives at
`flush_rt_diag`.
horus_core --lib: 2595 passed, 0 failed. (An earlier run of the same suite
showed 3 unrelated scheduler failures at load average ~70; all three pass in
isolation and the suite is clean at load 6.)
There was a problem hiding this comment.
🔵 Needs a closer look
There are at least two user-facing correctness issues in changed code (the installer’s “fail open” Rust version gate and a Python RuntimeError’s suggested fix being invalid in its own failure mode).
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
horus_py/horus/init.py:764
- This RuntimeError is raised specifically when the ring holds a type this build cannot map to a Python class (
_detect_topic_typereturned None), but the suggested fix interpolates{ring_type}directly intoSub(..., {ring_type}), which is not valid Python unless a symbol of that name exists in scope (and in this failure mode it likely doesn’t). The error text should instead explain that bindings for the detected type are missing and show a syntactically valid placeholder for the required message class.
install.sh:567 - The comment above says this Rust version gate must “fail OPEN”, but as written it aborts the install whenever awk is missing or errors (any non-0 exit), not only when it has positively determined have < want. Under
set -e, moving the awk call out of a conditional would also make non-zero exits fatal before you can interpret them.
- Files reviewed: 73/73 changed files
- Comments generated: 0 new
- Review effort level: Lite
…heir file
From a workspace sweep for the same shape of defect the docs audit
found: code that contradicts its own comments.
replay_from() built a name -> priority map from the recorded execution
order and then looked it up by node_id — the key of node_recordings,
a generated hex string (`format!("{:x}{:x}", unix_nanos, node_count)`)
that can never equal a node name. Every lookup missed, `unwrap_or(0)`
gave every node priority 0, and add_replay's stable sort left the
replay in HashMap iteration order: neither the recorded order nor the
same order twice. The module's whole claim is "replay with tick-perfect
determinism". The name comes back out of `<name>@<id>.horus`, and a
node that is genuinely absent from the recorded order now says so
instead of silently landing at 0 with everything else.
The existing tests could not catch it: a replay whose ordering is
arbitrary still ticks every node the right number of times. The new one
asserts the priorities.
hlog!'s doc said "The default level is DEFAULT_LEVEL (`info`, i.e.
`debug` is off)". DEFAULT_LEVEL is LEVEL_DEBUG, and three other places
in that file say so, including the constant's own doc and the test
nothing_is_suppressed_by_default. That is the doc a node author reads
before deciding a `hlog!(debug, ...)` in a tick loop is free — it is
not: unconfigured, every one runs format!, a timestamp and a serialise
into the shared buffer. The guard reads the constant and the macro's
doc block and requires them to agree; it lives in its own test file so
it cannot match its own text, and it fails when either side moves.
launch.rs still carried this branch's "INERT ... nothing reads it" doc
and a --dry-run line reading "IGNORED, see the warning at launch", both
made false by main's #194 — the merge took main's behaviour and left
its own description behind. The field is honoured, and what it does
(a separate shared-memory tree, so that node cannot see its siblings)
is now what the doc and the plan both say. run_launch and
print_launch_plan go through one function so they cannot disagree again.
node_namespace_construction was a test that computed `global/local/node`
from its own inputs and asserted the result equalled what it had just
computed — it called nothing from the module and described an identity
no process is given. Replaced with one that exercises the shipped
function.
deny_export was never consulted for a system topic. allow_export handles `_horus.*` explicitly and its comment says `deny_export = ["_horus.logs"]` "now actually works", with a green test beside it asserting exactly that. Both were true of the guard function and false of the process: allow_export's one production caller filters self.readers, which is populated from the TopicRegistry, and nothing registers `_horus.logs` or `_horus.presence` there. They left the machine through send_system_topic, which never asked — every Error and Warning line this node emits and its full node inventory, multicast once per timer tick, to any host that announced an interest. An operator who set deny_export got no change and a passing test telling them otherwise. send_system_topic now asks. `_horus.estop` does not come through it, so this cannot silence a fleet e-stop. topic_name_from_c says "Every topic constructor starts here, which is what keeps the two invariants the last-error channel rests on". The two JsonWireMessage constructors did not: they called CStr::from_ptr(name) directly — undefined behaviour on null, the exact "segfault at best" case the guard closes — and never touched the error slot, so horus_topic_last_error after one of them answered with whatever the previous topic had failed for. json_wire is the transport under every C++ service and action call. The new test enumerates the constructors by hand rather than by grep, because a scan can only find the ones that already look like the others, and it arms a stale reason before each call so a constructor that neither clears nor sets its own is caught. While there: the four service and action constructors had the same null-CStr::from_ptr UB with no doc claiming otherwise. Adjacent rather than part of the same defect, but it is the same hazard on the same public ABI, so they now refuse a null name too. TransformFrame's wait_for_transform and tf_at_strict documented `Raises: ValueError`. Both route through to_py_err, which maps Timeout to HorusTimeoutError and Transform to HorusTransformError — built on PyException, so neither is a ValueError subclass. The `except ValueError` recovery path the docstring told you to write never caught the timeout, and it left the node's tick() on exactly the case the timeout parameter exists to handle. The test catches the documented exception and also reads the docstring, so the two cannot drift apart again.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Copilot reviewed 80 out of 80 changed files in this pull request and generated no new comments.
Suppressed comments (5)
horus_manager/src/commands/run/mod.rs:1
execute_workspacestill usesstatus.code().unwrap_or(1), which collapses signal deaths on Unix into1. This reintroduces the exact issueshell_exit_code()fixes elsewhere (unreachable 128+N/137/139 hinting and misleading reporting). Usecrate::error_wrapper::shell_exit_code(&status)here as well, to keep exit-code behavior consistent across run paths.
install.sh:1- The fish-shell branch only checks for the presence of
set -gx HORUS_PREFIXand does not replace a stale value when the prefix changes. That contradicts the POSIX branch directly below (which is value-aware) and can leave a prefix install pointing at a deleted tree on fish. Make the fish branch value-aware too (delete/replace existingset -gx HORUS_PREFIX ...if it differs).
horus_sys/src/shm/mod.rs:1 shm_base_dir_fornow accepts an arbitrary&strand interpolates it directly into a path component. Ifnamespacecan contain path separators or.., callers (including cleanup code usingremove_dir_all) could be tricked into targeting unintended paths. Consider validating/sanitizingnamespace(e.g., reject/,\\, NUL, and.., and/or enforce an allowlist like[A-Za-z0-9_-]+) before constructing thePathBuf.
horus_py/horus/init.py:1- The suggested fix
subs=[Sub(<topic>, <ring_type>)]is not actionable as written:ring_typeis a string like'CmdVel', not a Python symbol, and this branch is specifically the case where the build cannot map that ring type to a Python class. Suggest a concrete, runnable remediation instead (e.g., instruct to import the message class (fromhorusorhorus._horus) and declaresubs=[Sub('<topic>', <MsgClass>)], and/or state that the Python binding for<ring_type>is missing and must be installed/upgraded).
uninstall.sh:1 - Rewriting rc files via
> ${profile}.horus-newandmvcan change the file mode/permissions (and potentially ownership/extended attrs), since the new file is created with the current umask. For shell rc files this can be a privacy/security regression (e.g.,600→644). Preserve the original metadata (e.g., capture the original mode and reapply it after rewrite, or prefer an in-place edit approach that keeps permissions intact).
… for good
`rt_diag` claims its slot by bumping `RT_DIAG_HEAD` FIRST, then marks the slot
odd, writes the bytes, and only then stores the completing even sequence. A
drain landing anywhere in that window got `None` from `read_slot`, hit
`continue`, and then ran `*tail = head` — stepping the tail permanently past a
line whose producer had done nothing wrong. Nobody could emit it afterwards:
the slot is skipped now and the tail never goes back.
`read_slot` returned one `None` for two opposite situations. A LAPPED slot has
been recycled by a later producer and really is gone. An IN-FLIGHT slot is
microseconds from being complete. The drain treated both as "skip it". They are
now distinguished — the sequence is derived from the monotonic claim index, so
`seq < want` is exactly "claimed, not finished yet" — and an in-flight slot
stops the drain instead of being discarded, leaving the line for the next one.
Bounded on purpose. A producer that panics between the head bump and its
completing store leaves a slot odd forever, and an unbounded retry would then
wedge the ring for the life of the process and cost every LATER diagnostic too
— strictly worse than the bug being fixed. Stalling twice on the same index is
taken as a dead producer: that one line is given up and draining continues.
This is what was actually behind the CI failure in
`a_compute_only_program_still_prints_its_safing_diagnostics`. The window is
nanoseconds on an idle machine, which is why it reproduced on a contended
runner and nowhere else, and why the previous commit's shutdown flush did not
help: by the time `stop()` flushed, a drain had already stepped over the line
and there was nothing left to find. The flush is still right — it is what makes
the last diagnostics reach stdout without waiting on a 25 ms poll — but it was
not the cause.
Two tests, both driving the ring by hand so they do not depend on thread
timing:
- a drain must not advance the tail past an index whose producer has not
finished writing (asserts on the tail, not on what got emitted, because the
background drainer would race an "the next drain emits it" assertion)
- repeated drains must get past a slot whose producer never completes, so the
bound is real
They share the one process-global ring, and the second one's bounded skip steps
over the first one's in-flight slot, so they take a lock rather than racing each
other.
Ablation: skipping in-flight slots as before fails the first and leaves the
second green.
`read_slot`'s existing flood test asserted `.is_none()` for a recycled slot;
it now asserts `SlotRead::Lapped`, which is the thing it was always trying to
say.
There was a problem hiding this comment.
🔵 Needs a closer look
Two user-facing install/cleanup helpers have correctness issues (fish HORUS_PREFIX updates and SHM namespace sanitization consistency) that should be fixed before merging.
Review details
Suppressed comments (2)
install.sh:961
- In the fish rc path, the installer only checks for the presence of any "set -gx HORUS_PREFIX" line. If a user moved/changed their prefix, an existing line with the OLD value prevents writing the new value, so new shells keep exporting the stale prefix.
horus_sys/src/shm/mod.rs:261 - shm_base_dir_for() does not apply the same namespace sanitization/defaulting as shm_namespace()/shm_base_dir(). If a caller passes the raw HORUS_NAMESPACE value (e.g. containing '-' or other chars), this will point at a different on-disk directory than the one real processes use, so cleanup/sweeps can miss live namespaces (or create an unexpected new one).
- Files reviewed: 80/80 changed files
- Comments generated: 0 new
- Review effort level: Lite
…s shell rc
Not all 21 threads needed code: five describe install/uninstall regex problems
this branch already fixed (the `\?`-in-BRE portability bug and the unanchored
`HORUS_PREFIX` guards), and one describes a whitespace run in
`topic/dispatch.rs` that is no longer there. Those are answered on the threads.
Nine were live.
Severity first:
* `uninstall.sh` could replace a shell profile with an empty file. The rewrite
is `{ grep -vF ... > "$profile.horus-new"; } || true`, and the comment above
it correctly explains that `|| true` is needed because grep exits 1 when it
selects no lines — the rc whose only content was our PATH line. But grep also
exits 2 when it cannot READ the file, and the redirect creates the target
either way, so `[ -f ]` could not tell those apart and the `mv` clobbered a
profile it had failed to read. Now requires the profile to be readable and
refuses to turn a non-empty profile into an empty one unless it genuinely
held nothing else.
* `shm_base_dir_for` interpolated its namespace argument raw. It is public and
exists precisely for namespaces that are not this process's own, and callers
hand the result to `remove_dir_all` — so `../../../../etc` resolved to
`/dev/shm/horus_../../../../etc`, outside the SHM parent entirely. The module
already had `sanitize_namespace`, strict enough and used everywhere else;
this was the one path that skipped it. Ablated: reverting the one-line fix
turns the new test red with the escaping path in the message.
* `install.sh`'s fish arm tested only for the PRESENCE of `set -gx
HORUS_PREFIX`, so a line naming a DIFFERENT prefix counted as "already
configured" and a fish user who reinstalled elsewhere kept an export pointing
at the tree the installer had just replaced. The POSIX arm was made
value-aware for exactly this; fish was left behind.
* `start_log_file_drain_once` memoized the negative answer. `OnceLock::
get_or_init` caches whatever the first call produced, `None` included, so a
process that built a `Scheduler` before `HORUS_LOG_FILE` was set never wrote
a log file again for its whole life. A `Mutex` keeps the at-most-one-drain
guarantee without freezing "no".
* `status.code().unwrap_or(1)` reported every signal death — SIGSEGV, SIGKILL,
the operator's Ctrl-C — as "exited with code 1". `shell_exit_code` exists for
this and returns 128+N; this call site was missed when it was introduced.
* `bytes.len() as i32` wraps. A value longer than `i32::MAX` came back negative,
and negative is the documented "no such param" return, so an overlong string
was indistinguishable from a missing one. Saturates now.
* `horus doctor` printed fourteen consecutive spaces mid-sentence.
* `control_restart` waited a fixed 400 ms for the scheduler to start. That is a
bet that the thread got scheduled inside the window; on a loaded runner it
does not, and the test fails reporting 0 inits as though `RestartNode` were
broken. Now a bounded poll — same upper bound, no assumption.
* `messages_total` — comment only, and deliberately. The reviewer is right that
it advances even when `try_send` drops, and the comment defending that said a
ring nobody drains is "one the subscriber's own staleness sees". That is
circular: `SubscriptionFreshness::refresh` derives freshness from this counter
and nothing else, so a publisher hammering a full ring keeps a subscriber
"fresh" while it receives nothing — the watchdog cannot see the fault it
watches for. Bumping on success would close it and is the right shape, but it
moves the counter's meaning at seven call sites across two crates and changes
when `StalePolicy::SafeState`/`Stop` fire on a live robot. That belongs in its
own change with its own tests, not in a merge. The comment now records the
limitation instead of denying it.
Verified: `horus_core --lib` 2597 passed, `horus_sys --lib` 225, `control_restart`
and `log_file_drain_wiring` green, `cargo fmt --check` and `clippy --all-targets
-D warnings` across horus_sys/horus_core/horus_manager clean, and both shell
scripts parse under `bash -n` (they are `#!/bin/bash`; a `sh -n` failure at
line 115 predates this and is a dash-vs-bash artifact, not a syntax error).
`registered.is_paused` is the one-tick `Miss::Skip` flag and is what `reinit_pending_nodes` actually reads; the operator pause lives in `node_controls` and is not read there at all. The comment blurred them, so it justified lifting the operator pause with "would otherwise be skipped by the very check that would have re-inited it" — which is not true of that check. The code was already right to clear both. Lifting the operator pause is what lets the node TICK once re-inited, not what permits the re-init: a wedged node is usually paused before it is restarted, and without it the node comes back initialised and still frozen.
There was a problem hiding this comment.
🟡 Changes recommended
install.sh’s fish HORUS_PREFIX matcher uses \$\ as a literal $ instead of an end-of-line anchor, which can prevent replacing stale exports and reintroduce the prefix-move failure mode.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 80/80 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core scheduling/safety behavior and multiple public surfaces across Rust/Python/C++ plus installer scripts, which warrants final domain-owner validation beyond automated review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
horus_manager/src/cargo_gen.rs:919
- The generated Cargo.toml comment says driver deps are "from [hardware]", but this generator now feeds
write_driver_depsfrommanifest.hardware_entries(), which merges both[hardware]and legacy[drivers]. As written, projects still using[drivers]will get the right deps but a misleading comment in the generated file.
- Files reviewed: 80/80 changed files
- Comments generated: 0 new
- Review effort level: Lite
Seven conflicts, all resolved as unions rather than picks, plus one break that existed in neither branch. * `communication/mod.rs`, `topic/mod.rs` re-exports — both sides added names. * `topic/metrics.rs` — this branch removes `recv_failures` from the internal `MigrationMetrics` (the real transport never increments it; only `MockTopic` does), #203 adds `send_retry_overruns`. Kept both changes: the field is gone and the counter is there. * `topic/mod.rs` `metrics()` — same shape. This branch's hard `0` for recv-failures, with its reasoning, plus #203's overrun read. * `rt_executor.rs` — `flush_rt_diag()` is this branch's wrapper around the very `rt_diag_drain(print_line)` main open-codes, so the wrapper stays and #203's `hlog_rt::flush()` is added beside it. * `async_executor.rs`, `compute_executor.rs` — this branch starts the rt_diag drain so a safe-state panic on those executors is not queued into a ring nothing reads; #203 moved the spawn to `spawn_best_effort` so a non-real-time executor stops inheriting the tick thread's policy and reserved-core mask. Both are needed and neither replaces the other. * `scheduler/mod.rs` — this branch wires `set_blackbox_hook`, without which horus_net's `record_external_event` dropped every peer-lost and desync event; #203 creates the `BbRing` the executors publish into. They reach the same recorder by different routes on purpose: the hook is for events raised OFF the tick path, the ring for events raised ON it. The break neither branch had: #203 deleted `EstopTrigger::trigger` once every caller had moved to `trigger_fmt`, and this branch added a NEW `trigger` caller in `primitives.rs`. Both are correct in isolation and git merged them cleanly into code that does not compile — E0599 at primitives.rs:102. Fixed by using `trigger_fmt`/`format_args!`, which is what that site wanted anyway: it is the RT panic path, and `format!` allocates on it. Verified on the merged tree: `horus_core --lib` 2623 passed, `horus_sys --lib` 249, `cargo fmt --check` and `clippy --all-targets -D warnings` clean, and all of #203's RT guards still hold — rt_helper_thread_isolation 5, ready_dispatch_pool 2, rt_hlog_off_the_tick_path 4, blackbox_off_the_tick_path 2, recording_is_not_a_syscall_storm 1, rt_thread_status 13, control_restart 2.
`CLI contract (live docs)` clones horus-docs at a SHA pinned in this workflow,
deliberately — the job runs in a context with write access to this repo's
Actions cache, so building an arbitrary ref of another repository would be a
cross-repo escalation this repo's review never sees. Verifying different docs
means bumping the SHA here, in a reviewed PR.
horus-docs#17 is the documentation half of the same bidirectional audit this
branch is the code half of, and it merged as 9a1a0c1. Leaving the pin at
4a612b1 would validate this branch's runtime corrections against pages that
still describe what the runtime did before them — which is precisely the drift
the audit was run to find, reintroduced by the gate meant to catch it.
One commit, whose parent is the old pin. Verified against the pinned commit
itself rather than a local branch:
HORUS_DOCS_DIR=<horus-docs at 9a1a0c1> \
cargo test -p horus_manager --test docs_contract -- --include-ignored
14 passed, 0 failed.
There was a problem hiding this comment.
🟡 Changes recommended
The new uninstall rc-file rewrite condition uses grep -cvF ... || echo 1, which breaks the “rc contained only our PATH line” cleanup case by contaminating the count output, so prefix PATH cleanup can silently fail.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 81/81 changed files
- Comments generated: 2
- Review effort level: Lite
Six CI jobs went red on one test, and it was mine:
`an_rc_file_that_is_only_the_prefix_path_line_is_emptied_not_skipped`. Code
Coverage, Cross-Process IPC Tests, CLI Integration Tests and Parity
(macOS ARM64) all run `completion_install_contract`, so a single shell bug
failed four leaves and two aggregates.
The guard I added to stop an unreadable profile being replaced with an empty
file also required "the result is non-empty, unless the profile held nothing
but our lines", and tested that second half with
[ "$(grep -cvF "$prefix_path_line" "$profile" 2>/dev/null || echo 1)" = "0" ]
`grep -c` PRINTS "0" and EXITS 1 when the count is zero. So on the legitimate
case the `|| echo 1` also ran, the substitution became "0\n1", the comparison
failed, and the rewrite was skipped — leaving the stale PATH line in exactly
the file where removing it matters most. That test's doc comment already
records two earlier ways this line went wrong; this made a third.
`-r` was always the whole guard. An unreadable file is what `grep` exit 2 means
and what `[ -r ]` detects; given a readable profile, an empty result is
legitimate by construction and needs no second opinion. The extra condition
bought nothing and cost the case it was meant to protect.
Verified in all three directions, not just the one the test covers:
* profile is only our line -> emptied (0 bytes)
* profile has other content -> ours removed, rest kept
* profile is unreadable (000) -> untouched (36 bytes, contents intact)
and `completion_install_contract` is 26 passed, 0 failed.
Also: the generated Cargo header said "# Driver dependencies (from [hardware])"
for entries that `Manifest::hardware_entries()` merges out of BOTH `[hardware]`
and the legacy `[drivers]` table, so a user still on `[drivers]` would look for
a `[hardware]` table they do not have and conclude the generator invented the
dependency. It names both sources now.
There was a problem hiding this comment.
🟡 Changes recommended
Two user-facing developer-experience issues remain (a copy/pastable-but-invalid subs=[Sub(...)] RuntimeError suggestion, and missing stubs for the newly exported peek_topic_type API).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 81/81 changed files
- Comments generated: 2
- Review effort level: Lite
…nable
Two review findings, and I got the first one wrong the first time.
`peek_topic_type` is registered on the extension module (lib.rs:123) and
imported by `horus/__init__.py` as `_peek_topic_type`, but it was never declared
in `horus/_horus.pyi`. I previously resolved this thread as stale after
grepping `__init__.py` and finding the symbol there — the finding was about the
STUB, and those are different files. The stub is what a type checker reads, so
to mypy/pyright the function did not exist. Declared now, with a note that
`__init__.py` carries a None-returning fallback for builds without it.
The `RuntimeError` raised when a bare-string subscription collides with a typed
ring suggested `subs=[Sub('cmd_vel', CmdVel)]`. The shape is right — the second
argument is a class, not the string `peek_topic_type` returned — but the message
never said the class has to be imported, so following it literally raises
`NameError` for anyone who has not already got the type in scope. It now names
the import, and says what it means when this build has no such class: the ring
was written by a peer carrying a message type this install does not.
There was a problem hiding this comment.
🟡 Changes recommended
There are two concrete correctness issues in changed code (awk split delimiter in install.sh and a fragile relative-path source read in hooks_contract.rs) that should be fixed before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
horus_manager/tests/hooks_contract.rs:252
- This test reads "src/main.rs" relative to the current working directory, but
cargo test -p horus_managerdoes not guarantee the test binary runs with CWD set to the crate root (often it’s the repo root). That makes the test fail to find the file. Resolve the path via CARGO_MANIFEST_DIR like the other source-reading contract tests do.
- Files reviewed: 81/81 changed files
- Comments generated: 1
- Review effort level: Lite
|
Merged as part of the #216 merge stack, squashed onto The benchmark-gate regression that blocked this PR is fixed in the stack. The only per-message additions on the ping-pong path were the two Also folded in: the review follow-ups already on this branch, the |
A bidirectional audit compared the runtime against the documentation and asked, for each divergence, which side is wrong — instead of assuming the code is ground truth. 31 rulings came out of it. Of those, 22 were fix-horus: the documentation was right and the runtime was not.
A second sweep then went looking for the same shape of defect without a documentation prompt: code that contradicts its own comments, doc-comments, error strings or sibling implementations. Those fixes are in the section below.
The bar for a runtime fix was deliberately strict: the runtime had to contradict its own stated intent — a comment, doc-comment, error string, test name, or a sibling implementation in the same file. "I would have designed it differently" did not qualify.
Every fix was reverted in turn to confirm its test catches it. Ablation results are in each commit message.
The ones that matter most
max_deadline_missescould not fire. Documented as the per-node consecutive ceiling that stops the robot, default 100 — and unreachable at that value, or any value above 20, in every shipped build.DegradationAction::Killsetsis_stopped, so the ladder's terminal rung retired the node before its count could climb. Nothing warned: the ladder's only setter is#[cfg(test)], so no user build could raise the rung either.Consecutive deadline misses were counted by the budget check. On the RT executor the counter was pinned at 1 forever for a node woken late that executes in microseconds — the failure mode
primitives.rsitself calls dominant. On the main loop nothing cleared it at all, making a "consecutive" ceiling a lifetime total.messages_total— the staleness watchdog's "new data exists" signal — moved only onsend(). A topic published entirely throughtry_send/send_blockingread as 0 messages while carrying every one of them, and could safe-state a healthy subscriber. Those are the two APIs the docs point users to on critical topics.A panicking
shutdown()cancelled the emergency stop that called it.BudgetPolicy::EmergencyStopran the callback bare and before latching, so an unwind escaped past the latch, pastrunning.store(false), into the loop's outercatch_unwind.A critical RT node that could not reach a safe state did not halt the robot.
note_safing_failurestates the rule as a property of the system;is_critical_nodehad exactly one caller, on the main loop.Participant registrations were never returned. One transient subscriber on a fresh thread permanently converted a command topic from backpressured to overwrite-oldest, for the life of the segment — silently voiding
try_send's full-ringErrandsend_blocking's backpressure.[network]reached one of five spawn paths. On every invocation a real single-binary robot makes, the documented comms-loss safe state never armed.Behaviour changes worth reviewing
max_deadline_missesinstead of being killed at 20 while the robot runs on. That is what the knob, its default, the builder doc and five documentation pages all say it does.detect_optimal_backendnever narrows a running topic. Returning registrations made a downgrade possible for the first time, and a backwards migration mid-stream loses messages — measured: the surviving subscriber of two received 198 of 200. Narrowing waits for a migration drain that is real.HORUS_NETis retired;horus doctornow asks the same question the build asks.Where a fix was deliberately not attempted
One ruling needed a design decision rather than a repair, and inventing one would have been worse than saying so:
Failedbranch built on it deleted.The per-node
namespace:was also on this list. It is not any more: main's #194 landed first and made a node-levelnamespace:exportHORUS_NAMESPACEso the more specific value wins, with a guard requiring everyHORUS_*key the launcher exports to have a reader. This branch had warned that the field was inert; that premise is gone, so the warning went with it in the merge.The second sweep: code that contradicts itself
Same strict bar, no documentation prompt — the evidence is always in the file.
horus node pausedid nothing to three of the four node classes. Operator pause travels through the sharednode_controlsmap and only the RT executor read it. Pausing a Compute, Async I/O or Event node reported success and the node kept ticking. The control handler says the pause holds "for ALL node classes … both the executor threads andshould_tick_nodehonor it";should_tick_nodesays it is "honored the same way the executor threads do".The blackbox never recorded a single network event.
set_blackbox_hooksays "Called by the Scheduler at startup to wire the blackbox" and nothing called it, sorecord_external_event— the entry pointhorus_net's replicator uses for peer loss, replication failure and desync — looked up an unsetOnceLockand dropped the event. A distributed robot's flight recorder held a complete local timeline and nothing about the network..no_alloc()never warned that it was unenforced.warn_if_unenforcednames its own call site — "Intended call site: node registration, once per.no_alloc()node" — and had no caller outside its own tests. A binary withoutRtAwareAllocatorgot an inert flag and no banner.Safing diagnostics were queued with nothing to drain them.
primitives.rsopens by sayinghonor_safe_state_request"is reached only fromrt_executor" — its own doc-comment says "every executor must call this", and all four do. The drain thread was started byRtExecutor::start_poolalone, so a program with no RT nodes wrote every safing line into a ring nothing ever read.Scheduler introspection invented answers while the scheduler ran. Eleven
PyScheduleraccessors answered the running state with a fabricated value — zero nodes, tick 0,"PythonScheduler",falseforis_recording().get_node_statsin the same block already refused, and the module header states a re-entrant call "will findself.inner == Noneand receiveRuntimeError".--simdid nothing at all on the Python driver path.load_config_entrieshad no simulation handling, sohorus.drivers.load()constructed the real driver class underhorus run --sim— the failureasks_for_simulation's own comment calls "the exact failure--simexists to prevent".A C++ log line reached
horus logwith no time on it. Both C++ entry points build their entry withtimestamp: String::new()and the comment "filled by log_buffer if empty". Nothing filled it.Two exit-code hints could never fire, and the preferred config section did less than the legacy one.
exit_code_hintdocuments that "signal-based exits (128+N) are translated"; every caller reached it throughstatus.code().unwrap_or(1), which isNone→ 1 for a signal death, so the OOM hint was unreachable in production. Separately,[hardware]is documented as preferred and[drivers]as legacy, butcargo_genandpyproject_genreaddriversalone — a driver declared the preferred way generated no dependency at all.Verification
horus_core2572,horus_manager3503,horus_cpp151,horus_py70 Rust + 490 Python testscargo fmt --all --check,clippyclean across every crate,cargo check --workspace --all-targetsDocumentation side: softmata/horus-docs#17.