Skip to content

Fix 31 runtime/documentation divergences found by a bidirectional audit - #189

Closed
starswallow wants to merge 65 commits into
mainfrom
fix/docs-runtime-divergence
Closed

starswallow wants to merge 65 commits into
mainfrom
fix/docs-runtime-divergence

Conversation

@starswallow

@starswallow starswallow commented Sep 4, 2026 •

Copy link
Copy Markdown
Collaborator

A bidirectional audit compared the runtime against the documentation and asked, for each divergence, which side is wrong — instead of assuming the code is ground truth. 31 rulings came out of it. Of those, 22 were fix-horus: the documentation was right and the runtime was not.

A second sweep then went looking for the same shape of defect without a documentation prompt: code that contradicts its own comments, doc-comments, error strings or sibling implementations. Those fixes are in the section below.

The bar for a runtime fix was deliberately strict: the runtime had to contradict its own stated intent — a comment, doc-comment, error string, test name, or a sibling implementation in the same file. "I would have designed it differently" did not qualify.

Every fix was reverted in turn to confirm its test catches it. Ablation results are in each commit message.

The ones that matter most

max_deadline_misses could not fire. Documented as the per-node consecutive ceiling that stops the robot, default 100 — and unreachable at that value, or any value above 20, in every shipped build. DegradationAction::Kill sets is_stopped, so the ladder's terminal rung retired the node before its count could climb. Nothing warned: the ladder's only setter is #[cfg(test)], so no user build could raise the rung either.

Consecutive deadline misses were counted by the budget check. On the RT executor the counter was pinned at 1 forever for a node woken late that executes in microseconds — the failure mode primitives.rs itself calls dominant. On the main loop nothing cleared it at all, making a "consecutive" ceiling a lifetime total.

messages_total — the staleness watchdog's "new data exists" signal — moved only on send(). A topic published entirely through try_send/send_blocking read as 0 messages while carrying every one of them, and could safe-state a healthy subscriber. Those are the two APIs the docs point users to on critical topics.

A panicking shutdown() cancelled the emergency stop that called it. BudgetPolicy::EmergencyStop ran the callback bare and before latching, so an unwind escaped past the latch, past running.store(false), into the loop's outer catch_unwind.

A critical RT node that could not reach a safe state did not halt the robot. note_safing_failure states the rule as a property of the system; is_critical_node had exactly one caller, on the main loop.

Participant registrations were never returned. One transient subscriber on a fresh thread permanently converted a command topic from backpressured to overwrite-oldest, for the life of the segment — silently voiding try_send's full-ring Err and send_blocking's backpressure.

[network] reached one of five spawn paths. On every invocation a real single-binary robot makes, the documented comms-loss safe state never armed.

Behaviour changes worth reviewing

  • A chronically missing node now e-stops the robot at max_deadline_misses instead of being killed at 20 while the robot runs on. That is what the knob, its default, the builder doc and five documentation pages all say it does.
  • detect_optimal_backend never narrows a running topic. Returning registrations made a downgrade possible for the first time, and a backwards migration mid-stream loses messages — measured: the surviving subscriber of two received 198 of 200. Narrowing waits for a migration drain that is real.
  • HORUS_NET is retired; horus doctor now asks the same question the build asks.

Where a fix was deliberately not attempted

One ruling needed a design decision rather than a repair, and inventing one would have been worse than saying so:

  • The migration drain needs an in-flight counter on the send and recv hot paths of a real-time framework — a cost the existing comment argues against paying by accident. The guarantee was withdrawn rather than faked, and the unreachable retry/Failed branch built on it deleted.

The per-node namespace: was also on this list. It is not any more: main's #194 landed first and made a node-level namespace: export HORUS_NAMESPACE so the more specific value wins, with a guard requiring every HORUS_* key the launcher exports to have a reader. This branch had warned that the field was inert; that premise is gone, so the warning went with it in the merge.

The second sweep: code that contradicts itself

Same strict bar, no documentation prompt — the evidence is always in the file.

horus node pause did nothing to three of the four node classes. Operator pause travels through the shared node_controls map and only the RT executor read it. Pausing a Compute, Async I/O or Event node reported success and the node kept ticking. The control handler says the pause holds "for ALL node classes … both the executor threads and should_tick_node honor it"; should_tick_node says it is "honored the same way the executor threads do".

The blackbox never recorded a single network event. set_blackbox_hook says "Called by the Scheduler at startup to wire the blackbox" and nothing called it, so record_external_event — the entry point horus_net's replicator uses for peer loss, replication failure and desync — looked up an unset OnceLock and dropped the event. A distributed robot's flight recorder held a complete local timeline and nothing about the network.

.no_alloc() never warned that it was unenforced. warn_if_unenforced names its own call site — "Intended call site: node registration, once per .no_alloc() node" — and had no caller outside its own tests. A binary without RtAwareAllocator got an inert flag and no banner.

Safing diagnostics were queued with nothing to drain them. primitives.rs opens by saying honor_safe_state_request "is reached only from rt_executor" — its own doc-comment says "every executor must call this", and all four do. The drain thread was started by RtExecutor::start_pool alone, so a program with no RT nodes wrote every safing line into a ring nothing ever read.

Scheduler introspection invented answers while the scheduler ran. Eleven PyScheduler accessors answered the running state with a fabricated value — zero nodes, tick 0, "PythonScheduler", false for is_recording(). get_node_stats in the same block already refused, and the module header states a re-entrant call "will find self.inner == None and receive RuntimeError".

--sim did nothing at all on the Python driver path. load_config_entries had no simulation handling, so horus.drivers.load() constructed the real driver class under horus run --sim — the failure asks_for_simulation's own comment calls "the exact failure --sim exists to prevent".

A C++ log line reached horus log with no time on it. Both C++ entry points build their entry with timestamp: String::new() and the comment "filled by log_buffer if empty". Nothing filled it.

Two exit-code hints could never fire, and the preferred config section did less than the legacy one. exit_code_hint documents that "signal-based exits (128+N) are translated"; every caller reached it through status.code().unwrap_or(1), which is None → 1 for a signal death, so the OOM hint was unreachable in production. Separately, [hardware] is documented as preferred and [drivers] as legacy, but cargo_gen and pyproject_gen read drivers alone — a driver declared the preferred way generated no dependency at all.

Verification

  • horus_core 2572, horus_manager 3503, horus_cpp 151, horus_py 70 Rust + 490 Python tests
  • cargo fmt --all --check, clippy clean across every crate, cargo check --workspace --all-targets
  • The C++ and Python surfaces were rebuilt and exercised, not just compiled

Documentation side: softmata/horus-docs#17.

Closes #184.

`horus install horus-monitor` symlinks into `PluginRegistry::global_bin_dir()`,
which is `paths::horus_dir()/bin` — the platform CONFIG dir, so
`~/.config/horus/bin` on Linux. `which_monitor_binary` checked `~/.horus/bin`
and nothing else, so the install succeeded and `horus monitor` then reported the
plugin as not installed. The two halves sit about 800 lines apart in different
files, and the comment on the path that was checked still called it "the plugin
install location".

The lookup now asks the registry for that directory rather than making a second
hardcoded guess at it, and keeps `~/.horus/bin` behind it so an existing install
goes on working.

The candidate list is split out as `plugin_bin_dirs()` so the relationship can be
asserted rather than assumed — the same shape as `run_python::python_cache_roots`,
which exists because the package cache had this exact defect. Two tests: that the
lookup searches the directory the installer writes, and that the legacy path is
still searched. Verified non-vacuous: removing the registry lookup fails the
first with exit 101.

Three documentation pages carried this as a working recipe and now carry an
`export PATH=...` workaround (softmata/horus-docs#16). Those lines should come
out once this ships.
`restart_node` sent PauseNode immediately followed by ResumeNode and printed
"The scheduler will re-initialize this node on next tick". It did not. Both arms
only call `NodeControls::set_paused` on a shared atomic; the code that re-runs
`init()` is `reinit_pending_nodes`, which keys off `RegisteredNode::initialized`,
and outside registration nothing in production ever set that back to false — the
only two such assignments in the workspace are in `#[cfg(test)]` code.

Four places in the runtime asserted the behaviour anyway: the clap help
("re-initialize without killing scheduler"), `restart_node`'s doc comment, an
inline comment reading "Pause then resume triggers re-init in the scheduler",
and `reinit_pending_nodes`'s own comment, "set by control commands". No control
command set it. On a robot the operator restarts a wedged node, is told it
worked, and moves on.

Adds `ControlCommand::RestartNode { name }`, appended to the enum so a command
serialised by an older CLI still decodes. The scheduler collects restart
requests while draining the control topic — the loop holds a mutable borrow of
`control_topic` and clearing the flag needs `self.nodes` — then clears
`initialized` and lifts any operator pause, since a wedged node is often paused
before being restarted and would otherwise be skipped by the very check that
would have re-inited it. An unknown name logs rather than silently doing
nothing.

Tests drive the real control topic through `run_for`, not `tick_once`: the
control topic is created in `run_with_filter`, so under `tick_once` it is None
and no control command is polled at all. Worth knowing separately — operator
control reaches a scheduler only while it is in `run()`.

Verified non-vacuous: removing the RestartNode arm fails the test with exit 101
and `left: 1, right: 2`, which is precisely the pre-fix behaviour.

Closes #184 is separate; this addresses the restart defect found by the
bidirectional audit.
`consecutive_misses` is the number the `max_deadline_misses` ceiling and
the whole degradation ladder compare against, and nothing reset it from
the deadline check. The reset lived in `NodeTimingState::record_tick`,
reachable only through `check_budget`, which gave three wrong answers:

- RT executor: `tick_node` calls `check_tick_budget` unconditionally and
  the deadline check afterwards, so a node woken 5 ms late that executes
  in microseconds cleared the run its own miss had just started. The
  counter was pinned at 1 forever. Neither `warn_after: 3` nor the
  ceiling could fire for the failure mode `check_deadline_from_release`
  was written to catch -- the one `primitives.rs` calls dominant.
- Main loop: the call sits inside the budget-violation branch, so a
  clean tick never reaches it and the count is a lifetime total.
- A node with `.deadline()` and no `.tick_budget()` skips the budget
  block entirely, so nothing cleared it on either path. `node_builder`
  derives a deadline from a budget and never the reverse, so this is an
  ordinary shape, and for it the per-node consecutive ceiling was a
  lifetime total wearing a safety threshold's name -- exactly what the
  per-node ceiling was introduced to end.

A budget overrun and a deadline miss are different failures, so a tick
that stayed inside its budget is no evidence its deadline was met. The
reset moves to `SafetyMonitor::record_deadline_met`, called wherever the
deadline check comes back clean on both dispatch paths.

Also: a miss now clears `recovery_counter`. `record_successful_tick`
means "did not panic", and both callers reach it on a tick already
recorded as a miss in the same pass, so a node at RateReduced that
missed EVERY deadline still accumulated `recovery_ticks` of "success"
and was restored to the rate it had just proved it could not hold. The
field's own comment already called itself a consecutive run.

Two existing tests asserted the old contract by calling `check_budget`
with an in-budget duration to mean "recovered"; they now say
`record_deadline_met`, which is the thing they were reaching for.

Each fix was reverted in turn to confirm the tests catch it: the
in-budget run reads 1 instead of 10, the no-budget scheduler node reads
10 instead of 0, the RT node stays at 5 after meeting its deadline, and
the never-recovering node is declared recovered at tick 8.
`max_deadline_misses` is documented as the per-node consecutive ceiling
that stops the robot, defaults to 100, and could not fire at that value
-- or at any value above 20 -- in any shipped build.

`DegradationAction::Kill` sets `is_stopped`, and a stopped node is
skipped forever on both dispatch paths. So the ladder's terminal rung,
hardcoded at 20 consecutive misses, permanently retired the node before
its count could climb any higher: `consecutive_misses` froze at 20 and
the ceiling was unreachable. Nothing warned. `max_deadline_misses(n)`
only clamps with `n.max(1)`, and the sole setter for the ladder is
`#[cfg(test)]`, so no user build could raise the rung either.

Two of the runtime's own defaults contradicted each other, and the unit
tests only ever exercised the ceiling at 3 and 5 -- both below 20 --
which is why the interaction went unnoticed.

A hardcoded constant must not silently cap a documented safety
threshold, so the rung moves: `kill_after` is raised to the ceiling when
the ceiling is higher. The graduated stages below it (warn at 3, half
rate at 5, isolate at 10) are untouched, and an isolated node keeps
ticking -- every tick on the RT executor, on a probe interval on the
main loop -- so it can still reach the ceiling. Where the two now
coincide the e-stop wins, because both paths record the miss, which
evaluates the ceiling, before evaluating the ladder.

Note this changes what a chronically missing node does under the default
configuration: it is no longer killed at 20 while the robot runs on, it
e-stops the robot at 100. That is what the knob, its default, the
builder doc and five documentation pages all say it does.

Both tests assert the ORDER of the two terminal events rather than the
counter -- at this level nothing removes the node, so the count climbs
regardless and the e-stop would fire on its own; what killed the ceiling
is the `Kill` that lands first. Reverting to the hardcoded rung fails
them at 20 for both the configured ceiling of 25 and the default of 100.
Copilot AI lite review requested due to automatic review settings September 4, 2026 23:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes safety-critical scheduler and timing enforcement behavior (including e-stop reachability), so it warrants final human review despite strong test coverage.

Pull request overview

Aligns horus runtime behavior with documented user-facing guarantees surfaced by the horus / horus-docs audit: plugin discovery for horus monitor, correctness of horus node restart, and safety/timing enforcement semantics around deadline misses and emergency-stop thresholds.

Changes:

  • Fix horus monitor plugin lookup by searching the installer’s PluginRegistry::global_bin_dir() (while keeping the legacy ~/.horus/bin fallback) and add an assertion test to prevent future drift.
  • Make horus node restart actually re-run init() by introducing ControlCommand::RestartNode and implementing scheduler-side handling to clear RegisteredNode::initialized.
  • Correct safety timing logic: consecutive deadline misses are reset only by a met deadline, recovery counters don’t advance on missed deadlines, and the degradation ladder no longer prevents max_deadline_misses from being reachable.
File summaries
File Description
horus_manager/src/main.rs Adds plugin_bin_dirs() and updates which_monitor_binary() to search the installer’s global bin dir first, plus tests to assert path agreement.
horus_manager/src/commands/node.rs Updates restart_node to send RestartNode instead of PauseNode+ResumeNode.
horus_core/src/scheduling/control.rs Extends ControlCommand with RestartNode and adds a human-readable description.
horus_core/src/scheduling/scheduler/mod.rs Implements RestartNode handling; records record_deadline_met on clean deadline checks; updates max_deadline_misses docs.
horus_core/src/scheduling/safety_monitor.rs Moves consecutive-miss reset semantics to record_deadline_met, fixes recovery counter behavior, and adjusts degradation policy so configured ceiling is reachable.
horus_core/src/scheduling/rt_executor.rs Records record_deadline_met when RT ticks meet deadlines; adds RT-path regression test coverage for consecutive misses.
horus_core/src/scheduling/scheduler/tests.rs Adds regression coverage for “deadline but no budget” consecutive-miss reset behavior in deterministic mode.
horus_core/tests/control_restart.rs Adds integration tests ensuring RestartNode re-initializes the intended node without affecting others.
Review details

Suppressed comments (1)

horus_core/tests/control_restart.rs:129

  • Same flakiness risk here: a fixed sleep assumes the scheduler has performed the initial init() within 400ms. Polling with a timeout makes the test resilient to transient slowness.
    std::thread::sleep(std::time::Duration::from_millis(400));
    assert_eq!(inits.load(Ordering::SeqCst), 1);
  • Files reviewed: 8/8 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread horus_core/tests/control_restart.rs Resolved
@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Benchmark gate

Baseline: rolling window of 8 run(s) on AMD EPYC 7763 64-Core Processor. Current arm: median of 6 repetition(s).

Result: FAIL

  • Topic_cross_process_oneway_pingpong@1024B — median: 278.0 ns -> 310.0 ns (+11.5%), threshold 305.8 ns (relative threshold (window is tight enough)).
Adaptive_CmdVel@16B — 50000 samples
metric baseline current change band (<=) verdict
median 40.0 ns 29.0 ns -27.5% 65.0 ns IMPROVED
p95 50.0 ns 50.0 ns +0.0% 100.0 ns INCONCLUSIVE
p99 69.0 ns 60.0 ns -13.0% 169.0 ns INCONCLUSIVE
p99.9 80.0 ns 80.0 ns +0.0% 330.0 ns INCONCLUSIVE
p99.99 — 110.0 ns — — N/A (samples)
max 15.92 µs 13.40 µs -15.8% 31.85 µs INCONCLUSIVE
max_jitter 15.91 µs 13.39 µs -15.8% 31.83 µs INCONCLUSIVE
p99/median 1.73x 2.07x +19.9% 2.23x INCONCLUSIVE
p99.9/median 2.00x 2.76x +37.9% 3.00x INCONCLUSIVE
ns/msg 170.3 ns 165.8 ns -2.7% 220.3 ns INCONCLUSIVE
Adaptive_Imu@304B — 50000 samples
metric baseline current change band (<=) verdict
median 40.0 ns 40.0 ns +0.0% 65.0 ns INCONCLUSIVE
p95 60.0 ns 60.0 ns +0.0% 110.0 ns INCONCLUSIVE
p99 89.5 ns 90.0 ns +0.6% 189.5 ns INCONCLUSIVE
p99.9 110.0 ns 110.0 ns +0.0% 360.0 ns INCONCLUSIVE
p99.99 — 140.0 ns — — N/A (samples)
max 13.85 µs 12.61 µs -8.9% 27.70 µs INCONCLUSIVE
max_jitter 13.83 µs 12.59 µs -8.9% 27.66 µs INCONCLUSIVE
p99/median 2.24x 2.25x +0.6% 2.80x INCONCLUSIVE
p99.9/median 2.75x 2.75x +0.0% 3.75x INCONCLUSIVE
ns/msg 270.3 ns 272.5 ns +0.8% 324.3 ns INCONCLUSIVE
Adaptive_JointCommand@928B — 50000 samples
metric baseline current change band (<=) verdict
median 80.0 ns 80.0 ns +0.0% 105.0 ns INCONCLUSIVE
p95 100.0 ns 100.0 ns +0.0% 150.0 ns INCONCLUSIVE
p99 130.0 ns 130.0 ns +0.0% 230.0 ns INCONCLUSIVE
p99.9 160.0 ns 160.0 ns +0.0% 410.0 ns INCONCLUSIVE
p99.99 — 240.0 ns — — N/A (samples)
max 15.53 µs 13.14 µs -15.4% 31.07 µs INCONCLUSIVE
max_jitter 15.48 µs 13.10 µs -15.4% 30.97 µs INCONCLUSIVE
p99/median 1.62x 1.62x +0.0% 2.12x INCONCLUSIVE
p99.9/median 2.00x 2.00x +0.0% 3.00x INCONCLUSIVE
ns/msg 335.0 ns 332.0 ns -0.9% 402.0 ns INCONCLUSIVE
Adaptive_LaserScan@1480B — 10000 samples
metric baseline current change band (<=) verdict
median 89.0 ns 89.0 ns +0.0% 114.0 ns INCONCLUSIVE
p95 110.0 ns 110.0 ns +0.0% 160.0 ns INCONCLUSIVE
p99 150.0 ns 150.0 ns +0.0% 250.0 ns INCONCLUSIVE
p99.9 170.0 ns 170.0 ns +0.0% 420.0 ns INCONCLUSIVE
p99.99 — 12.27 µs — — N/A (samples)
max 13.15 µs 15.97 µs +21.4% 26.30 µs INCONCLUSIVE
max_jitter 13.09 µs 15.92 µs +21.6% 26.18 µs INCONCLUSIVE
p99/median 1.69x 1.69x +0.0% 2.19x INCONCLUSIVE
p99.9/median 1.91x 1.91x +0.0% 2.91x INCONCLUSIVE
ns/msg 689.0 ns 688.3 ns -0.1% 826.8 ns INCONCLUSIVE
Topic_cross_process_oneway_paced_stream@1024B — 200000 samples
metric baseline current change band (<=) verdict
median 260.5 ns 275.0 ns +5.6% 316.8 ns INCONCLUSIVE
p95 323.0 ns 415.0 ns +28.5% 468.3 ns INCONCLUSIVE
p99 597.5 ns 596.0 ns -0.3% 905.9 ns INCONCLUSIVE
p99.9 4.92 µs 6.90 µs +40.3% 16.22 µs INCONCLUSIVE
p99.99 15.34 µs 15.59 µs +1.7% 24.58 µs INCONCLUSIVE
max 35.19 µs 37.90 µs +7.7% 79.53 µs INCONCLUSIVE
max_jitter 34.99 µs 37.70 µs +7.7% 79.20 µs INCONCLUSIVE
p99/median 2.33x 2.16x -7.3% 3.91x INCONCLUSIVE
p99.9/median 19.48x 25.81x +32.5% 71.66x INCONCLUSIVE
ns/msg 10.00 µs 10.00 µs +0.0% 12.00 µs INCONCLUSIVE
Topic_cross_process_oneway_paced_stream@64B — 200000 samples
metric baseline current change band (<=) verdict
median 170.5 ns 175.0 ns +2.6% 203.1 ns INCONCLUSIVE
p95 205.5 ns 205.0 ns -0.2% 267.8 ns INCONCLUSIVE
p99 468.0 ns 456.0 ns -2.6% 1.03 µs INCONCLUSIVE
p99.9 3.75 µs 3.36 µs -10.3% 13.44 µs INCONCLUSIVE
p99.99 13.08 µs 13.11 µs +0.2% 26.19 µs INCONCLUSIVE
max 51.25 µs 36.93 µs -27.9% 270.47 µs INCONCLUSIVE
max_jitter 51.13 µs 36.82 µs -28.0% 270.28 µs INCONCLUSIVE
p99/median 2.70x 2.66x -1.4% 5.43x INCONCLUSIVE
p99.9/median 20.29x 20.37x +0.4% 68.21x INCONCLUSIVE
ns/msg 10.00 µs 10.00 µs +0.0% 12.00 µs INCONCLUSIVE
Topic_cross_process_oneway_pingpong@1024B — 1000000 samples
metric baseline current change band (<=) verdict
median 278.0 ns 310.0 ns +11.5% 305.8 ns REGRESSED (blocking)
p95 361.0 ns 375.0 ns +3.9% 415.1 ns INCONCLUSIVE
p99 408.5 ns 411.0 ns +0.6% 508.5 ns INCONCLUSIVE
p99.9 1.40 µs 1.44 µs +2.8% 1.96 µs INCONCLUSIVE
p99.99 10.48 µs 10.67 µs +1.8% 15.72 µs INCONCLUSIVE
max 27.51 µs 20.11 µs -26.9% 94.84 µs INCONCLUSIVE
max_jitter 27.31 µs 19.89 µs -27.2% 94.60 µs INCONCLUSIVE
p99/median 1.45x 1.33x -8.2% 1.95x INCONCLUSIVE
p99.9/median 4.97x 4.65x -6.4% 7.16x INCONCLUSIVE
ns/msg 344.5 ns 374.0 ns +8.6% 413.4 ns INCONCLUSIVE
Topic_cross_process_oneway_pingpong@64B — 1000000 samples
metric baseline current change band (<=) verdict
median 150.0 ns 165.0 ns +10.0% 179.7 ns INCONCLUSIVE
p95 180.0 ns 200.0 ns +11.1% 230.0 ns INCONCLUSIVE
p99 200.0 ns 225.0 ns +12.5% 300.0 ns INCONCLUSIVE
p99.9 728.5 ns 752.0 ns +3.2% 1.27 µs INCONCLUSIVE
p99.99 7.78 µs 9.52 µs +22.4% 11.67 µs INCONCLUSIVE
max 23.44 µs 31.38 µs +33.9% 60.24 µs INCONCLUSIVE
max_jitter 23.33 µs 31.26 µs +34.0% 60.09 µs INCONCLUSIVE
p99/median 1.34x 1.36x +1.8% 1.84x INCONCLUSIVE
p99.9/median 4.79x 4.56x -4.9% 7.76x INCONCLUSIVE
ns/msg 200.6 ns 216.5 ns +7.9% 250.6 ns INCONCLUSIVE
Topic_cross_process_unpaced_queue_delay@64B — 200000 samples
metric baseline current change band (<=) verdict
median 147.5 ns 155.0 ns +5.1% 197.9 ns INCONCLUSIVE
p95 205.0 ns 205.0 ns +0.0% 267.3 ns INCONCLUSIVE
p99 230.5 ns 245.0 ns +6.3% 330.5 ns INCONCLUSIVE
p99.9 3.18 µs 2.55 µs -19.8% 13.64 µs INCONCLUSIVE
p99.99 9.26 µs 8.87 µs -4.2% 13.89 µs INCONCLUSIVE
max 38.40 µs 37.75 µs -1.7% 80.93 µs INCONCLUSIVE
max_jitter 38.30 µs 37.65 µs -1.7% 80.76 µs INCONCLUSIVE
p99/median 1.60x 1.58x -1.5% 2.10x INCONCLUSIVE
p99.9/median 23.14x 16.45x -28.9% 108.60x INCONCLUSIVE
ns/msg 112.7 ns 118.9 ns +5.5% 162.7 ns INCONCLUSIVE

Blocking metrics: median, p99/median. Report-only: p95, p99, p99.9, p99.99, max, max_jitter, p99.9/median, ns/msg. Tail metrics past p99.9 are advisory on a shared runner — one host preemption owns max.

Raw benchmark output
----------
  samples=1000000  warmup=200000  transferred=2000000  lost=0  invalid(TSC-skew)=0  poll cost=13.9 ns  window=0.430 s
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=84 invol=4
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  cross-core TSC offset theta_hat = +5.3 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.

== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
   json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
   payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7671 <-> pid 7681
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way, offset-free             1000000     225.3     300.6     400.7    1212.0    10218.7     20111.9 |     1000        98
  round trip (RTT)                 1000000     490.7     641.2     841.5    2735.2    20567.9     40253.7 |      999       100
  RTT halved (one-way)             1000000     245.3     320.6     420.8    1367.6    10283.9     20126.9 |      999       100
  leg A->B (apparent)              1000000     215.5     285.8     446.1     566.3    19330.5     39557.8 |      942       100
  leg B->A (apparent)              1000000     195.1     315.3     465.8     545.9     9812.7     39947.9 |      956       100
  responder turnaround             1000000      29.9      30.3      40.1      40.1       70.3     22691.4 |      283        91
------------------------------------------------------------------------------------------------------------------------------------------
  samples=1000000  warmup=200000  transferred=2000000  lost=0  invalid(TSC-skew)=0  poll cost=40.1 ns  window=0.732 s
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=143 invol=6
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  cross-core TSC offset theta_hat = -14.7 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.

== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
   json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7671 <-> pid 7683
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way (assumes TSC coherence)    200000     114.9     165.2     445.7    3360.9    13108.5     40549.0 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=8.6 ns  window=2.000 s
  measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=12
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).

== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
   json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
   payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7671 <-> pid 7686
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way (assumes TSC coherence)    200000     215.1     285.0     615.8    4983.8    15593.1     35930.7 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=20.4 ns  window=2.000 s
  measured backlog: 0.03 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=393 invol=10
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -14.7 ns (add to correct).

== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
   !! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
   !! time spent waiting in the ring. It is a backpressure/throughput figure.
   !! Do not quote it as an IPC latency; that is the exact mistake the retired
   !! `Topic_cross_process` metric shipped for months. Read `measured backlog`
   !! below: ~1 message means the subscriber kept up and no queue formed.
   json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7671 <-> pid 7688
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  publish->receive incl. queue      200000     114.9     155.0     245.3    2549.6     8870.9     42292.2 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=21.3 ns  window=0.024 s
  measured backlog: 1.30 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=5 invol=0
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).

SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
  json metric key                                what it measures                  n    median       p99     p99.9     p99.99         max
------------------------------------------------------------------------------------------------------------------------------------------
  Topic_cross_process_oneway_pingpong@64B        ONE-WAY latency             1000000     160.3     225.3     721.3     9241.8     31377.5
  Topic_cross_process_oneway_pingpong@1024B      ONE-WAY latency             1000000     300.6     400.7    1212.0    10218.7     20111.9
  Topic_cross_process_oneway_paced_stream@64B    ONE-WAY latency              200000     165.2     445.7    3360.9    13108.5     40549.0
  Topic_cross_process_oneway_paced_stream@1024B  ONE-WAY latency              200000     285.0     615.8    4983.8    15593.1     35930.7
  Topic_cross_process_unpaced_queue_delay@64B    QUEUE DELAY (not latency)    200000     155.0     245.3    2549.6     8870.9     42292.2
------------------------------------------------------------------------------------------------------------------------------------------
  Instrumentation is INCLUDED in every figure and never subtracted.
  `Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
  nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
  `RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
  the row to compare against any competitor publishing a halved ping-pong.
  `Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
  however long the message waited in the ring, so it scales with the achieved backlog
  (printed in its scenario block) and is NOT a latency.
  A figure without its payload size, ring depth and topology is not comparable to anything.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!  CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!!  No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!!  pingpong (1024 B) `one-way, offset-free`: p99.99 rests on 98 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  unpaced_queue (64 B): measured backlog is only 1.30 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!!  unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-2.json
  keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
  NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
  will report it as missing and fail the gate until the window is refreshed —
  that is intended, and is why the key was not silently reused.
  Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
  7 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║        HORUS Robotics Message Types Benchmark                    ║
╠══════════════════════════════════════════════════════════════════╣
║  Testing real-world robotics message latency (REP 2014)          ║
╚══════════════════════════════════════════════════════════════════╝

Platform: AMD EPYC 7763 64-Core Processor (4 cores)
Iterations: 50000
Warmup: 5000

╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type    │ Size (bytes) │ Typical Rate │ Use Case                    ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel          │           16 │ 1000+ Hz     │ Velocity control commands   ║
║ Imu             │          296 │ 500+ Hz      │ IMU sensor fusion           ║
║ LaserScan       │         1480 │ 10-40 Hz     │ 2D lidar navigation         ║
║ JointCommand    │         1032 │ 500+ Hz      │ Manipulator control         ║
╚═════════════════════════════════════════════════════════════════════════════╝


[Topic] Running benchmarks — timing tx.send() enqueue cost only
        (cross-thread consumer drains in the background; no sample
         waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
  CmdVel              16 bytes │ median:      29ns │ p99:      60ns │ CV: 3.2160
  Imu                304 bytes │ median:      40ns │ p99:      90ns │ CV: 1.7597
  LaserScan         1480 bytes │ median:      89ns │ p99:     140ns │ CV: 2.1870
  JointCommand       928 bytes │ median:      80ns │ p99:     130ns │ CV: 1.3849

╔══════════════════════════════════════════════════════════════════════════════════════════╗
║           SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency           ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message      │   Size │ Median (ns) │ p99 (ns) │    CV   │ Misses │ Sustained rate       ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel       │     16 │          29 │       60 │  3.2160 │      0 │      6.03 M msg/s     ║
║ Imu          │    304 │          40 │       90 │  1.7597 │      0 │      3.68 M msg/s     ║
║ LaserScan    │   1480 │          89 │      140 │  2.1870 │      0 │      1.46 M msg/s     ║
║ JointCommand │    928 │          80 │      130 │  1.3849 │      0 │      3.02 M msg/s     ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.

╔══════════════════════════════════════════════════════════════════╗
║              REAL-TIME SUITABILITY ANALYSIS                      ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop       ║
║ guarantees: a real loop also pays the receive side, the node's   ║
║ compute, and the scheduler's jitter, none of which is measured   ║
║ here. The headroom is printed so a pass by four orders of        ║
║ magnitude cannot be read as a tight result.                      ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel    (1kHz  control): ~ PASS by 16667x — gate not discriminating (p99=60ns vs 1000000ns)
║ CmdVel    (10kHz control): ~ PASS by 1667x — gate not discriminating (p99=60ns vs 100000ns)
║ Imu       (500Hz fusion) : ~ PASS by 22222x — gate not discriminating (p99=90ns vs 2000000ns)
║ LaserScan (40Hz  lidar)  : ~ PASS by 178571x — gate not discriminating (p99=140ns vs 25000000ns)
║ JointCmd  (500Hz control): ~ PASS by 15385x — gate not discriminating (p99=130ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝

Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-2.json
===== repetition 3/3 =====

HORUS cross-process benchmark — ONE-WAY LATENCY (headline) + unpaced queue delay
Two OS processes on two distinct physical cores over the real Topic SHM transport.
Every handle holds exactly one role, so the `role == Both` same-thread fast path that
produces the repo's headline send-cost figure is never taken. Round-trip rows are
labelled where they are a halved round trip. See the module header for the clock
decomposition and for why the `Topic_cross_process` JSON key was retired.

MACHINE — every nanosecond below is conditional on this block
------------------------------------------------------------------------------------------------------------------------------------------
  cpu model              : AMD EPYC 7763 64-Core Processor (2 physical / 4 logical cores)
  kernel                 : 6.17.0-1022-azure (x86_64)
  benchmark cores        : cpu3 = process A (pinger / subscriber), cpu1 = process B (responder / publisher)  [DistinctPhysical]
  governor               : cpu3=n/a cpu1=n/a  driver=n/a
  scaling range          : n/a .. n/a  (hw max n/a)  turbo_disabled=unknown
  clocksource            : tsc   tsc flags: [tsc, rdtscp, constant_tsc, tsc_reliable, nonstop_tsc]
  TSC rate (calibrated)  : 2445.515 MHz (0.4089 ns/cycle) — a WALL clock; NOT the core clock
  clock instrumentation  : 49 cycles = 20.0 ns for serialize()+rdtsc()+rdtscp(), INCLUDED in every sample and never subtracted
  transparent_hugepage   : [always] madvise never
  sched_rt_runtime/period: 950000 / 1000000 us    perf_event_paranoid=4
  isolcpus=(none)  nohz_full=(none)  smt_active=1  mlockall(MCL_CURRENT)=true
------------------------------------------------------------------------------------------------------------------------------------------

running ping-pong (one-way latency), 64 B payload ...
running ping-pong (one-way latency), 1024 B payload ...
running paced stream (one-way latency), 64 B payload ...
running paced stream (one-way latency), 1024 B payload ...
running unpaced stream (QUEUE DELAY, not latency), 64 B payload ...

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! READ THIS BEFORE QUOTING ANY NUMBER BELOW
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!  CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!!  No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!!  pingpong (1024 B) `one-way, offset-free`: p99.99 rests on 99 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  unpaced_queue (64 B): measured backlog is only 1.29 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!!  unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
   json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7722
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way, offset-free             1000000     125.1     165.2     225.3     751.6     9522.3    512222.7 |     1000       100
  round trip (RTT)                 1000000     290.3     360.7     490.7    1553.0    19455.6   1024546.0 |      997       100
  RTT halved (one-way)             1000000     145.2     180.3     245.3     776.5     9727.8    512273.0 |      997       100
  leg A->B (apparent)              1000000     125.5     165.6     255.6    1197.7    15604.1   1024230.7 |      990       100
  leg B->A (apparent)              1000000     114.9     155.0     244.9     275.2    10103.4     56918.9 |      953       100
  responder turnaround             1000000      29.9      30.3      40.1      40.1      170.5     23853.5 |      501        87
------------------------------------------------------------------------------------------------------------------------------------------
  samples=1000000  warmup=200000  transferred=2000000  lost=0  invalid(TSC-skew)=0  poll cost=13.9 ns  window=0.439 s
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=86 invol=3
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  cross-core TSC offset theta_hat = +5.3 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.

== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
   json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
   payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7724
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way, offset-free             1000000     220.4     310.4     410.5    1442.2    10674.6     18649.2 |     1000        99
  round trip (RTT)                 1000000     480.9     651.4     851.4    2965.4    21419.2     37328.3 |      999       100
  RTT halved (one-way)             1000000     240.4     325.7     425.7    1482.7    10709.6     18664.2 |      999       100
  leg A->B (apparent)              1000000     215.5     296.1     455.9     566.3    20793.2     36943.1 |      990       100
  leg B->A (apparent)              1000000     184.8     315.3     455.5     535.7    10303.8     32624.6 |      962       100
  responder turnaround             1000000      29.9      30.3      40.1      40.1      100.2     21479.3 |      317       100
------------------------------------------------------------------------------------------------------------------------------------------
  samples=1000000  warmup=200000  transferred=2000000  lost=0  invalid(TSC-skew)=0  poll cost=32.7 ns  window=0.748 s
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=147 invol=5
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  cross-core TSC offset theta_hat = -9.6 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.

== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
   json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7726
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way (assumes TSC coherence)    200000     114.9     175.0     465.8    3741.5    14872.1     35399.5 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=8.6 ns  window=2.000 s
  measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=13
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).

== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
   json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
   payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7728
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  one-way (assumes TSC coherence)    200000     204.9     275.2     575.7    7097.9    17947.6     37904.1 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=20.9 ns  window=2.000 s
  measured backlog: 0.03 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=393 invol=11
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -9.6 ns (add to correct).

== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
   !! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
   !! time spent waiting in the ring. It is a backpressure/throughput figure.
   !! Do not quote it as an IPC latency; that is the exact mistake the retired
   !! `Topic_cross_process` metric shipped for months. Read `measured backlog`
   !! below: ~1 message means the subscriber kept up and no queue formed.
   json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
   payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7713 <-> pid 7731
------------------------------------------------------------------------------------------------------------------------------------------
  metric (ns)                            n       min    median       p99     p99.9     p99.99         max |  n>p99.9  n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
  publish->receive incl. queue      200000     104.7     155.0     255.2    7959.5     9462.2     37754.0 |      200        20
------------------------------------------------------------------------------------------------------------------------------------------
  samples=200000  warmup=20000  transferred=200000  lost=0  invalid(TSC-skew)=0  poll cost=21.3 ns  window=0.024 s
  measured backlog: 1.29 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
  measured window: minor_faults=0 major_faults=0 ctx_switches vol=5 invol=1
  core freq cpu3   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  core freq cpu1   (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
  this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).

SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
  json metric key                                what it measures                  n    median       p99     p99.9     p99.99         max
------------------------------------------------------------------------------------------------------------------------------------------
  Topic_cross_process_oneway_pingpong@64B        ONE-WAY latency             1000000     165.2     225.3     751.6     9522.3    512222.7
  Topic_cross_process_oneway_pingpong@1024B      ONE-WAY latency             1000000     310.4     410.5    1442.2    10674.6     18649.2
  Topic_cross_process_oneway_paced_stream@64B    ONE-WAY latency              200000     175.0     465.8    3741.5    14872.1     35399.5
  Topic_cross_process_oneway_paced_stream@1024B  ONE-WAY latency              200000     275.2     575.7    7097.9    17947.6     37904.1
  Topic_cross_process_unpaced_queue_delay@64B    QUEUE DELAY (not latency)    200000     155.0     255.2    7959.5     9462.2     37754.0
------------------------------------------------------------------------------------------------------------------------------------------
  Instrumentation is INCLUDED in every figure and never subtracted.
  `Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
  nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
  `RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
  the row to compare against any competitor publishing a halved ping-pong.
  `Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
  however long the message waited in the ring, so it scales with the achieved backlog
  (printed in its scenario block) and is NOT a latency.
  A figure without its payload size, ring depth and topology is not comparable to anything.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!  CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!!  No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!!  pingpong (1024 B) `one-way, offset-free`: p99.99 rests on 99 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!  unpaced_queue (64 B): measured backlog is only 1.29 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!!  unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-3.json
  keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
  NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
  will report it as missing and fail the gate until the window is refreshed —
  that is intended, and is why the key was not silently reused.
  Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
  7 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║        HORUS Robotics Message Types Benchmark                    ║
╠══════════════════════════════════════════════════════════════════╣
║  Testing real-world robotics message latency (REP 2014)          ║
╚══════════════════════════════════════════════════════════════════╝

Platform: AMD EPYC 7763 64-Core Processor (4 cores)
Iterations: 50000
Warmup: 5000

╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type    │ Size (bytes) │ Typical Rate │ Use Case                    ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel          │           16 │ 1000+ Hz     │ Velocity control commands   ║
║ Imu             │          296 │ 500+ Hz      │ IMU sensor fusion           ║
║ LaserScan       │         1480 │ 10-40 Hz     │ 2D lidar navigation         ║
║ JointCommand    │         1032 │ 500+ Hz      │ Manipulator control         ║
╚═════════════════════════════════════════════════════════════════════════════╝


[Topic] Running benchmarks — timing tx.send() enqueue cost only
        (cross-thread consumer drains in the background; no sample
         waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
  CmdVel              16 bytes │ median:      29ns │ p99:      60ns │ CV: 1.9834
  Imu                304 bytes │ median:      40ns │ p99:      90ns │ CV: 1.9723
  LaserScan         1480 bytes │ median:      89ns │ p99:     160ns │ CV: 1.2659
  JointCommand       928 bytes │ median:      80ns │ p99:     130ns │ CV: 1.7959

╔══════════════════════════════════════════════════════════════════════════════════════════╗
║           SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency           ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message      │   Size │ Median (ns) │ p99 (ns) │    CV   │ Misses │ Sustained rate       ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel       │     16 │          29 │       60 │  1.9834 │      0 │      6.03 M msg/s     ║
║ Imu          │    304 │          40 │       90 │  1.9723 │      0 │      3.66 M msg/s     ║
║ LaserScan    │   1480 │          89 │      160 │  1.2659 │      0 │      1.44 M msg/s     ║
║ JointCommand │    928 │          80 │      130 │  1.7959 │      0 │      2.99 M msg/s     ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.

╔══════════════════════════════════════════════════════════════════╗
║              REAL-TIME SUITABILITY ANALYSIS                      ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop       ║
║ guarantees: a real loop also pays the receive side, the node's   ║
║ compute, and the scheduler's jitter, none of which is measured   ║
║ here. The headroom is printed so a pass by four orders of        ║
║ magnitude cannot be read as a tight result.                      ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel    (1kHz  control): ~ PASS by 16667x — gate not discriminating (p99=60ns vs 1000000ns)
║ CmdVel    (10kHz control): ~ PASS by 1667x — gate not discriminating (p99=60ns vs 100000ns)
║ Imu       (500Hz fusion) : ~ PASS by 22222x — gate not discriminating (p99=90ns vs 2000000ns)
║ LaserScan (40Hz  lidar)  : ~ PASS by 156250x — gate not discriminating (p99=160ns vs 25000000ns)
║ JointCmd  (500Hz control): ~ PASS by 15385x — gate not discriminating (p99=130ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝

Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-3.json


Gated against a rolling window of recent main runs. Each metric's band is max(relative, center + k·sigma_observed, absolute floor), so it is derived from the noise the baseline itself measured — where a band is printed wider than the change you care about, this run did not detect a regression and could not have. Thresholds and their reasoning: RegressionPolicy in benchmarks/src/output.rs.
Commit: 5393b83

`safety.on_link_lost`, `import` and `deny_export` are safety controls.
The 2026-07-30 audit found the whole `[network]` table parsed and then
read by nothing, and `apply_network_config` was written to fix that --
then wired into ONE of the five paths that spawn a run child.

That one is `execute_multiple_files`, reached only when `horus run` is
handed more than one file. A bare `horus run`, a `horus.toml` run, a
directory run and a `[workspace]` run all resolve to a single executable.
On those the child environment came from `build_child_env()`, which sets
PATH, LD_LIBRARY_PATH and PYTHONPATH and nothing else, so
`NetConfig::default()` read an environment nobody had populated: `import`
stayed `Auto`, `deny_export` stayed empty, and the documented comms-loss
safe state never armed. The gap the two existing comments describe was
still open on every invocation a real single-binary robot makes.

`apply_network_config` becomes `network_env() -> Vec<(String, String)>`,
which `build_child_env` now extends with -- so every single-executable
path picks it up from the function they all already call, and no future
path can quietly miss it. `execute_workspace` builds its `Command` by
hand rather than through `build_child_env`, so it gets an explicit
`apply_network_config` call.

The operator still wins: `network_env` skips any key already present in
the environment, so an explicit `HORUS_NET_*=...` beats the manifest on
every path. That is asserted too.

Reverting the `build_child_env` line gives `left: None` for
HORUS_NET_ON_LINK_LOST -- the safety control simply absent.

`horus launch` still performs no translation; that is a separate gap and
the documentation says so.
`start_log_file_drain` was called by no shipped code path. A tree-wide
grep for the symbol returned four hits: the definition, its re-export,
and two lines of one integration test that invoked it by hand. So
`HORUS_LOG_FILE=1 horus run` wrote no `.horus/logs/horus.log`, and
`HORUS_LOG_DIR`, `HORUS_LOG_MAX_SIZE` and `HORUS_LOG_MAX_FILES` were dead
with it -- while the test that proves the drain works kept it green.

The env guard lives inside the function precisely so a call site can be
unconditional, and the function's own doc-comment calls the variable an
opt-in, which is only true if something always calls it. `Scheduler::new`
already does this kind of env-driven wiring twice, for HORUS_NET_ENABLED
and HORUS_RECORD_SESSION, so the drain goes beside them. A process that
does not set the variable pays one env lookup and spawns nothing.

Once per process, via a `OnceLock`: a program may build several
schedulers, and each drain thread tracks its own index into the same
global ring, so N of them would write every entry N times into one file.

This also keeps the promise for C++ and Python, which the doc-comment's
"or calling this function directly" escape hatch could not: the symbol
has no FFI export anywhere in the tree. Both bindings construct a core
`Scheduler` (horus_cpp/src/scheduler_ffi.rs:26,
horus_py/src/scheduler.rs:882), so both now get the drain.

The test does not call the drain. It sets the variable, builds a
`Scheduler` the way any program does, and asserts a log file appears; it
is alone in its binary because it writes process-global environment.
Reverting the wiring leaves the file absent and it fails.

An operator who sets a documented variable to get a post-incident log and
finds an empty disk is the exact failure that variable exists to prevent.
Copilot AI review requested due to automatic review settings September 4, 2026 23:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It introduces a real behavioral edge case in start_log_file_drain_once (caching the disabled state) and adds tests that can be environment-dependent/flaky unless process-global env is isolated and restored.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 12/12 changed files
  • Comments generated: 3
  • Review effort level: Lite

Comment thread horus_core/src/core/log_buffer.rs Resolved
Comment thread horus_manager/src/commands/run/mod.rs Resolved
`messages_total` is not a metric. `SubscriptionFreshness::refresh` watches
it as the "new data exists" signal behind `.subscribe_with_timeout()`, and
its answer drives `StalePolicy::SafeState` and `StalePolicy::Stop`.

Only `RingTopic::send` bumped it. `Topic::try_send` and
`Topic::send_blocking` go straight to the ring, so a topic published
entirely through either read as 0 messages and 0 Hz while carrying every
message -- and could safe-state or halt a subscriber node whose data was
arriving normally. That is exactly the failure the counter was introduced
to fix, reintroduced for backpressure-aware publishers, and these are the
two APIs the documentation points users to on critical topics.

The runtime states the invariant in both places that consume the counter:
"every publisher increments `messages_total` in the topic's SHM header on
every send, so watching that counter is a true 'new data exists' signal"
(scheduling/types.rs) and "incremented on every `send()` regardless of
backend" (header.rs). Neither is true.

Bumped once per public publish call, not inside `RingTopic::try_send`,
which `send_lossy_retry` calls up to ~70 times for one message.

`try_send` counts before the call, as `send()` does: an attempt count, and
a publisher hammering a full ring is alive. `send_blocking` does NOT count
a `NoBackpressure` refusal -- that is returned before the send is even
attempted, so nothing reached the ring, and counting it would report a
healthy flow on a topic delivering nothing. A `Timeout` did engage the
ring, so it counts like `send()`'s dropped message does.

Reverting both bumps reads 0 for a topic that carried ten messages.
A topic holds at most 16 participants. `register_role` builds a precise
error for the one that does not fit -- it names the limit, the count, and
what to do about it -- and all three call sites threw it away:
`recv_uninitialized` and `read_latest` were `if ....is_err() { return
None }`, `send_uninitialized` the same with `Err(msg)`. Those are the only
`ensure_consumer`/`ensure_producer` call sites in the tree, so that
message reached nothing, ever.

Because the role stays `Unregistered`, every later `recv()` re-enters the
same cold path and fails identically: `None` forever, from a subscriber
indistinguishable from an idle one.

Nothing anywhere defends the silence, and the identical 16-slot wall one
layer down is handled the opposite way on purpose -- rate-limited
`tracing::warn!` under "COMM-H1: never let 'all 16 slots live' become a
SILENT no-comms endpoint (worse than the pre-fix loud panic)". The rate
limiter for it sits in the same file, unused by this path. Meanwhile
header.rs defends a separate design decision by claiming the residual cost
"surfaces as the existing 'No available participant slots' error. Loud,
and it says what to do" -- a guarantee the code did not provide.

All three sites now report through that same limiter, so a hot loop warns
once a minute per topic rather than once ever.

The limiter is also keyed by reason now, not just topic. A topic can hit
two different walls -- its FanoutShm endpoint bitmap and the participant
table -- with different causes and different fixes, and one bucket let
whichever fired first silence the other for a minute. That is the masking
the per-topic keying was introduced to prevent, one level down.

The test fills the table from 20 threads (the participant-count leak is
what lets one process do that), asserts it really is full before asserting
anything about diagnosis, and captures `tracing` events through a
hand-rolled `Subscriber` so it needs no new dependency. Reverting the fix
prints nothing at all: "Warnings seen: []".
Copilot AI review requested due to automatic review settings September 5, 2026 00:33

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are a couple of concrete issues in the new logging/diagnostic paths (log warning formatting artifacts, and start_log_file_drain_once memoizing “disabled” forever if first called before HORUS_LOG_FILE is set) that should be corrected before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (2)

horus_core/src/communication/topic/dispatch.rs:2640

  • The warning message contains a large run of spaces ("receive nothing"), which will show up verbatim in logs and makes the message harder to read/search. This looks like an accidental formatting artifact from line wrapping.
        if should_report_endpoint_exhaustion(topic.name(), Exhausted::ParticipantTable) {
            tracing::warn!(
                "topic '{}': this subscriber is NOT registered and will receive                  nothing — {}",
                topic.name(),
                e
            );

horus_core/src/core/log_buffer.rs:660

  • OnceLock<Option<...>> memoizes the first call result. If Scheduler::new() runs once before HORUS_LOG_FILE is set (so start_log_file_drain() returns None), later schedulers in the same process can never start the drain even when the variable is set. That contradicts the “if HORUS_LOG_FILE asks for it” contract and can make log-file output depend on initialization order in long-lived processes/tests.
pub fn start_log_file_drain_once() -> bool {
    LOG_FILE_DRAIN.get_or_init(start_log_file_drain).is_some()
}
  • Files reviewed: 15/15 changed files
  • Comments generated: 1
  • Review effort level: Lite

`send_blocking_pod` ended in `.is_ok()`, discarding which
`SendBlockingError` came back, and both arms then reported
`HorusTimeoutError`. So the one error horus_core writes specifically to be
un-retryable arrived in Python as the one error callers retry.

The core is explicit about the distinction, twice: "Refusing is
deliberate: reporting success for a delivery nobody guaranteed is worse on
an e-stop path than failing loudly. It does not clear on retry -- it lasts
as long as the topology that caused it." The variant carries a four-line
message naming the fix. The binding replaced it with the exception whose
own doc-comment means the opposite -- "the buffer stayed full for the whole
timeout" -- and its private helper asserted, falsely, that a `false` return
meant the ring stayed full.

This is easy to reach and permanent: `detect_optimal_backend` resolves any
POD topic with a second subscriber to `PodShm`, so the moment a `horus
topic echo` or a monitor attaches to a POD e-stop topic, every
`send_blocking` starts raising forever -- as a "timeout", on the path the
method is documented for.

The variant is now threaded out of both arms and mapped to a new
`horus.HorusBackpressureError` carrying the core's own remediation text.
`HorusTimeoutError` keeps `Timeout` alone.

Also exposes `Topic.provides_backpressure`, which the core's doc-comment
instructs callers to assert on after the first send and which Python had
no way to ask. It is answered through a new
`horus_core::communication::topic::backend_provides_backpressure`, keyed
by backend NAME because a name is all a binding can see -- and
`provides_backpressure()` is now defined in terms of it too, so the two
cannot disagree. A test walks every `BackendMode` through an exhaustive
match, so a new backend stops compiling rather than silently defaulting to
"no backpressure".

Reverting the mapping fails the test at "must raise
HorusBackpressureError".
Copilot AI review requested due to automatic review settings September 5, 2026 00:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It introduces at least one likely CI-breaking Rust unit test dependency on a live Python interpreter (and minor log-message formatting issues), and the PR description should be updated to reflect the expanded user-facing scope.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

horus_core/src/scheduling/scheduler/mod.rs:629

  • PR title/description says this PR fixes four runtime defects, but the diff also includes additional user-facing changes (e.g., wiring HORUS_LOG_FILE via Scheduler::new, new HorusBackpressureError + provides_backpressure in Python, and topic staleness/endpoint-exhaustion reporting). Please update the PR description to enumerate these extra fixes/behavior changes so reviewers and release notes don’t miss the expanded scope.

horus_core/src/communication/topic/dispatch.rs:1895

  • The warning message contains a large run of extra spaces ("deliver nothing"), which makes logs harder to read and looks like an accidental formatting artifact. Collapse it to a single space.
                "topic '{}': this publisher is NOT registered and will deliver                  nothing — {}",

horus_core/src/communication/topic/dispatch.rs:2637

  • The warning message contains a large run of extra spaces ("receive nothing"), which makes logs harder to read and looks like an accidental formatting artifact. Collapse it to a single space.
                "topic '{}': this subscriber is NOT registered and will receive                  nothing — {}",
  • Files reviewed: 20/20 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread horus_py/src/topic.rs Resolved
`_setup_topics` documents a three-step type resolution whose middle step,
"auto-detect from SHM header", was unreachable code. It did:

    probe = Topic(topic_name, 64)
    type_name = getattr(probe, 'type_name', None)

Both halves were dead. `PyTopic` exposes no `type_name` — its getters are
`name`, `is_network_topic`, `endpoint`, `backend_type` — so the value was
always `None`. And the probe could not have worked even with that
attribute: `Topic(name, capacity)` builds a `Topic<GenericMessage>`, and
attaching 4364-byte GenericMessage to an existing typed ring is refused by
the size check in `RingTopic::new`, so it always raised into a bare `except
Exception`. Step 2 therefore never ran for any topic that has ever existed,
and the ~70-entry `_TYPE_NAME_MAP` built for it had exactly one reader,
inside the unreachable branch.

The size check is correct and stays: its comment records that a name-based
exemption once let a `Topic<GenericMessage>` join a `Topic<CmdVel>` ring and
"write megabytes past the end of the mapping". The question just cannot be
answered by attaching -- opening requires choosing a type, which is what the
caller is trying to find out.

So: `horus_core::communication::peek_topic_type_name(name)` reads the
header without attaching, exposed to Python as `_horus.peek_topic_type`.
`_detect_topic_type` calls it and maps through `_TYPE_NAME_MAP`, which now
has a reachable reader.

Second half: when the ring exists and holds a type this build cannot name,
`_setup_topics` raises with the topic, the ring's type and the concrete fix
(`subs=[Sub('scan', LaserScan)]`) instead of falling through to a generic
topic and letting the runtime's byte-count RuntimeError out -- which named
neither `subs=`, nor the type the ring holds, nor what to do.

Auto-detection was only ever scoped to subscriptions; the `pubs=` case is
unchanged, since a publisher creates the ring and there is nothing to read.

Tests both ends. The Rust test asserts the old probe's refusal still
happens (it must) and that the peek answers anyway; reverting the peek
gives `left: None`, which is what the old probe returned for every topic.
The Python test drives a real `Node` with a bare `subs=` string against a
typed ring; restoring the old probe fails it.
Copilot AI review requested due to automatic review settings September 5, 2026 00:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A new unit test is environment-dependent (fails if HORUS_NET_* is set), and there are also formatting issues in newly added warning strings that should be corrected.

Review details

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

horus_core/src/communication/topic/dispatch.rs:2640

  • This warning string contains a large run of spaces before "nothing"; that looks accidental and will produce messy log output.

horus_manager/src/commands/run/mod.rs:1329

  • This test asserts a precondition that certain HORUS_NET_* env vars are not set in the process environment; that makes it environment-dependent and can fail on CI or developer machines. Prefer temporarily clearing and restoring those variables within the test so it is hermetic.
        assert!(
            preset.is_empty(),
            "precondition: {:?} set in this environment would override the manifest",
            preset
        );

horus_core/src/communication/topic/dispatch.rs:1898

  • The warning string has a large run of spaces before "nothing", which will render oddly in logs and makes the message harder to read/search.
            tracing::warn!(
                "topic '{}': this publisher is NOT registered and will deliver                  nothing — {}",
                topic.name(),
                e
            );
  • Files reviewed: 23/23 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

**`post_test` skipped teardown exactly when teardown matters.** It sat
after an early `?` in the `Commands::Test` arm, so a failing `horus test`
never ran it, while `post_run` and `post_build` go through
`run_teardown_hooks` and run on both paths deliberately: "Skipping it when
the run failed would skip it in exactly the case where the hardware was
most likely left live." The hooks contract file calls `test` "the only
phase with teardown" and then pinned only the symmetry of the KEY NAMES, so
`post_test` kept success-only semantics while the other two were given real
teardown semantics. A `post_test = ["release-bench"]` was skipped precisely
when the tests aborted mid-run.

The new guard reads the dispatch in main.rs rather than the table: every
`post_` phase must go through `run_teardown_hooks` and none through the
plain `run_hooks`. Reverting the arm fails it on both halves.

**`horus doctor` recommended a variable that does nothing.** It printed
"use --net or HORUS_NET=1 to enable" and reported networking from
`HORUS_NET`, which nothing else in the tree reads. Following the advice
changed nothing except that `doctor` then printed "horus_net: enabled",
confirming a state that did not exist -- worse than inert. main.rs already
recorded this exact confusion when it fixed `--net`; the diagnostic handing
users the broken half of the same remedy was not fixed with it.

`doctor` now asks the question the BUILD asks, through a shared
`cargo_gen::net_capability_requested`, and names only routes that work.
`HORUS_NET` is retired: no reader remains.

**`horus install --plugin -t <workspace>` threw the name away.** The call
site collapsed it to `let local = target.is_some()`, and what was left
called `detect_or_select_workspace` -- which resolves the CURRENT
DIRECTORY's workspace, or prints a menu and blocks on stdin. Both are
things `horus install` documents it never does, and with no tty the read
returns empty and the command dies with "Invalid selection". The package
half of the same flag already resolved the name correctly; the bool was a
vestige of a `--local` flag that no longer exists, still named in the
doc-comment.

The ablation shows how completely the name vanished: with the bug the
install proceeds past the target entirely and fails with an unrelated "no
package registry is configured".
Copilot AI review requested due to automatic review settings September 5, 2026 01:13

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes safety-critical scheduling/degradation behavior and core runtime/IPC semantics across multiple crates, warranting final human review despite strong test coverage.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

horus_core/src/communication/topic/dispatch.rs:2640

  • The warning message has a large run of literal spaces ("receive nothing") which will show up in logs and looks like an accidental formatting artifact. It should be a single space so the message is readable and stable for any log parsing.

horus_core/src/communication/topic/dispatch.rs:1896

  • The warning message has a large run of literal spaces ("deliver nothing") which will show up in logs and looks like an accidental formatting artifact. It should be a single space so the message is readable and stable for any log parsing.
            tracing::warn!(
                "topic '{}': this publisher is NOT registered and will deliver                  nothing — {}",
                topic.name(),
  • Files reviewed: 27/27 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread horus_manager/src/commands/doctor.rs Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The installer’s rust-version gate still fails closed on awk errors despite being documented to fail open, and the new shm_base_dir_for helper should sanitize its input to avoid path-escape footguns.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

install.sh:571

  • The rust-version gate is documented here as needing to “fail OPEN”, but if ! awk ...; then will still fail CLOSED on any awk error (e.g., awk missing/unusable, unexpected version format), aborting the install with “toolchain too old” without having established that. To actually fail open, only treat awk exit status 1 as “too old”; treat other non-zero statuses as “unknown” and skip the check.

horus_sys/src/shm/mod.rs:261

  • shm_base_dir_for(namespace) accepts an arbitrary string and feeds it directly into PathBuf::join(format!("horus_{namespace}")). If a caller passes an absolute path or path separators, join can escape shm_parent_dir() (absolute path) or create nested paths, which is a footgun for any future use that forwards user input (e.g., horus clean). Since this module already has sanitize_namespace, consider applying it here as well so the named-namespace helper is as safe as generate_namespace().
  • Files reviewed: 72/72 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

`every_env_var_the_code_reads_is_documented` caught HORUS_COMPUTE_DIAG_CHILD,
which is correct — it is a HORUS_* variable read from src/. It is the child
half of the test that a compute-only program still drains its RT diagnostics
to stdout, read inside a `#[cfg(test)]` module and unreachable from a shipped
binary, so it goes in TEST_ONLY beside HORUS_CPP_FFI_EMIT_CHILD rather than
into the user-facing environment-variables page.
Copilot AI review requested due to automatic review settings September 5, 2026 22:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A few correctness/safety issues remain in changed code paths (fish prefix export update behavior, unsanitized namespace-to-path mapping, and one remaining signal-exit-code propagation site), and they should be fixed before approval.

Review details

Suppressed comments (4)

install.sh:961

  • In the fish rc path, this only checks whether any set -gx HORUS_PREFIX line exists, and leaves a stale prefix untouched on reinstall/upgrade. That means a moved prefix can keep exporting the old tree in fish shells, unlike the POSIX branch which is value-aware and replaces stale values.
    horus_sys/src/shm/mod.rs:261
  • shm_base_dir_for() currently interpolates the raw namespace into the path without sanitizing. That can drift from the actual on-disk namespace (which is always sanitize_namespaced via shm_namespace()), and it also allows path separators / .. components to escape the intended parent dir when used by cleanup code.
    horus_manager/src/commands/run/mod.rs:335
  • This still collapses signal deaths into exit code 1 on Unix. Elsewhere in this PR you introduced error_wrapper::shell_exit_code() specifically so 128+N exits (e.g. SIGKILL→137) and related diagnostics/hints stay reachable; this call site should use it as well.
    horus_manager/src/commands/doctor.rs:1820
  • This status line contains a large run of literal spaces before enable = ..., which will render as an awkward gap in horus doctor output. It looks unintentional and should be a single space.
  • Files reviewed: 73/73 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

…ostics

`RtExecutor::stop` ends with a final drain of the RT diagnostic ring, and says
why:

    Without it a run shorter than `RT_DIAG_POLL` — or one whose last deadline
    miss came in the final milliseconds, which is the interesting case — would
    exit with its diagnostics still sitting in the ring.

Compute, Event and AsyncIo had no such flush, though all three call
`start_rt_diag_drain` and all three queue through `honor_safe_state_request`.
So for a compute-only, event-only or async-only program the guarantee simply
did not hold: anything queued inside the last 25 ms — "PANICKED in
enter_safe_state", a final safing failure — was still in the ring when the
process exited. The same class partition that has caught this code repeatedly:
fixed on the RT path, left on the other three.

It also makes an existing warning true. When the drain thread cannot be
spawned, `start_rt_diag_drain` tells the operator diagnostics "will only be
flushed at shutdown" — with no shutdown flush on three of the four executors,
they were not flushed anywhere.

The drain is now one named `flush_rt_diag()` called from all four `stop()`
paths.

Found via a CI failure in
`a_compute_only_program_still_prints_its_safing_diagnostics`, which slept 200 ms
and assumed the 25 ms background drainer would have been scheduled inside that
window. On a loaded runner it was not, and the test reported "no drain thread
was running" against code that was working. With the flush the line is emitted
synchronously inside `stop()`, so the sleep is gone and no timing window is
involved.

The flush itself is deliberately NOT asserted on. The background drainer's
first poll runs when its thread starts — the same few microseconds as the
`rt_diag` under test — so an ordering assertion passes whether or not `stop()`
flushes. I wrote that assertion, ablated the flush, watched it pass three times
out of three, and removed it: a guard that cannot fail for the reason it states
is worse than no guard. Isolating the flush needs a way to suppress that
thread, which does not exist today. The test keeps the claim it can support —
the line reaches stdout — and the reasoning for the flush lives at
`flush_rt_diag`.

horus_core --lib: 2595 passed, 0 failed. (An earlier run of the same suite
showed 3 unrelated scheduler failures at load average ~70; all three pass in
isolation and the suite is clean at load 6.)
Copilot AI review requested due to automatic review settings September 5, 2026 22:45

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

There are at least two user-facing correctness issues in changed code (the installer’s “fail open” Rust version gate and a Python RuntimeError’s suggested fix being invalid in its own failure mode).

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

horus_py/horus/init.py:764

  • This RuntimeError is raised specifically when the ring holds a type this build cannot map to a Python class (_detect_topic_type returned None), but the suggested fix interpolates {ring_type} directly into Sub(..., {ring_type}), which is not valid Python unless a symbol of that name exists in scope (and in this failure mode it likely doesn’t). The error text should instead explain that bindings for the detected type are missing and show a syntactically valid placeholder for the required message class.
    install.sh:567
  • The comment above says this Rust version gate must “fail OPEN”, but as written it aborts the install whenever awk is missing or errors (any non-0 exit), not only when it has positively determined have < want. Under set -e, moving the awk call out of a conditional would also make non-zero exits fatal before you can interpret them.
  • Files reviewed: 73/73 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

…heir file

From a workspace sweep for the same shape of defect the docs audit
found: code that contradicts its own comments.

replay_from() built a name -> priority map from the recorded execution
order and then looked it up by node_id — the key of node_recordings,
a generated hex string (`format!("{:x}{:x}", unix_nanos, node_count)`)
that can never equal a node name. Every lookup missed, `unwrap_or(0)`
gave every node priority 0, and add_replay's stable sort left the
replay in HashMap iteration order: neither the recorded order nor the
same order twice. The module's whole claim is "replay with tick-perfect
determinism". The name comes back out of `<name>@<id>.horus`, and a
node that is genuinely absent from the recorded order now says so
instead of silently landing at 0 with everything else.

The existing tests could not catch it: a replay whose ordering is
arbitrary still ticks every node the right number of times. The new one
asserts the priorities.

hlog!'s doc said "The default level is DEFAULT_LEVEL (`info`, i.e.
`debug` is off)". DEFAULT_LEVEL is LEVEL_DEBUG, and three other places
in that file say so, including the constant's own doc and the test
nothing_is_suppressed_by_default. That is the doc a node author reads
before deciding a `hlog!(debug, ...)` in a tick loop is free — it is
not: unconfigured, every one runs format!, a timestamp and a serialise
into the shared buffer. The guard reads the constant and the macro's
doc block and requires them to agree; it lives in its own test file so
it cannot match its own text, and it fails when either side moves.

launch.rs still carried this branch's "INERT ... nothing reads it" doc
and a --dry-run line reading "IGNORED, see the warning at launch", both
made false by main's #194 — the merge took main's behaviour and left
its own description behind. The field is honoured, and what it does
(a separate shared-memory tree, so that node cannot see its siblings)
is now what the doc and the plan both say. run_launch and
print_launch_plan go through one function so they cannot disagree again.

node_namespace_construction was a test that computed `global/local/node`
from its own inputs and asserted the result equalled what it had just
computed — it called nothing from the module and described an identity
no process is given. Replaced with one that exercises the shipped
function.
deny_export was never consulted for a system topic. allow_export
handles `_horus.*` explicitly and its comment says
`deny_export = ["_horus.logs"]` "now actually works", with a green
test beside it asserting exactly that. Both were true of the guard
function and false of the process: allow_export's one production
caller filters self.readers, which is populated from the TopicRegistry,
and nothing registers `_horus.logs` or `_horus.presence` there. They
left the machine through send_system_topic, which never asked — every
Error and Warning line this node emits and its full node inventory,
multicast once per timer tick, to any host that announced an interest.
An operator who set deny_export got no change and a passing test
telling them otherwise. send_system_topic now asks. `_horus.estop`
does not come through it, so this cannot silence a fleet e-stop.

topic_name_from_c says "Every topic constructor starts here, which is
what keeps the two invariants the last-error channel rests on". The
two JsonWireMessage constructors did not: they called
CStr::from_ptr(name) directly — undefined behaviour on null, the exact
"segfault at best" case the guard closes — and never touched the error
slot, so horus_topic_last_error after one of them answered with
whatever the previous topic had failed for. json_wire is the transport
under every C++ service and action call. The new test enumerates the
constructors by hand rather than by grep, because a scan can only find
the ones that already look like the others, and it arms a stale reason
before each call so a constructor that neither clears nor sets its own
is caught.

While there: the four service and action constructors had the same
null-CStr::from_ptr UB with no doc claiming otherwise. Adjacent rather
than part of the same defect, but it is the same hazard on the same
public ABI, so they now refuse a null name too.

TransformFrame's wait_for_transform and tf_at_strict documented
`Raises: ValueError`. Both route through to_py_err, which maps Timeout
to HorusTimeoutError and Transform to HorusTransformError — built on
PyException, so neither is a ValueError subclass. The `except
ValueError` recovery path the docstring told you to write never caught
the timeout, and it left the node's tick() on exactly the case the
timeout parameter exists to handle. The test catches the documented
exception and also reads the docstring, so the two cannot drift apart
again.
Copilot AI review requested due to automatic review settings September 5, 2026 22:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 80 out of 80 changed files in this pull request and generated no new comments.

Suppressed comments (5)

horus_manager/src/commands/run/mod.rs:1

  • execute_workspace still uses status.code().unwrap_or(1), which collapses signal deaths on Unix into 1. This reintroduces the exact issue shell_exit_code() fixes elsewhere (unreachable 128+N/137/139 hinting and misleading reporting). Use crate::error_wrapper::shell_exit_code(&status) here as well, to keep exit-code behavior consistent across run paths.
    install.sh:1
  • The fish-shell branch only checks for the presence of set -gx HORUS_PREFIX and does not replace a stale value when the prefix changes. That contradicts the POSIX branch directly below (which is value-aware) and can leave a prefix install pointing at a deleted tree on fish. Make the fish branch value-aware too (delete/replace existing set -gx HORUS_PREFIX ... if it differs).
    horus_sys/src/shm/mod.rs:1
  • shm_base_dir_for now accepts an arbitrary &str and interpolates it directly into a path component. If namespace can contain path separators or .., callers (including cleanup code using remove_dir_all) could be tricked into targeting unintended paths. Consider validating/sanitizing namespace (e.g., reject /, \\, NUL, and .., and/or enforce an allowlist like [A-Za-z0-9_-]+) before constructing the PathBuf.
    horus_py/horus/init.py:1
  • The suggested fix subs=[Sub(<topic>, <ring_type>)] is not actionable as written: ring_type is a string like 'CmdVel', not a Python symbol, and this branch is specifically the case where the build cannot map that ring type to a Python class. Suggest a concrete, runnable remediation instead (e.g., instruct to import the message class (from horus or horus._horus) and declare subs=[Sub('<topic>', <MsgClass>)], and/or state that the Python binding for <ring_type> is missing and must be installed/upgraded).
    uninstall.sh:1
  • Rewriting rc files via > ${profile}.horus-new and mv can change the file mode/permissions (and potentially ownership/extended attrs), since the new file is created with the current umask. For shell rc files this can be a privacy/security regression (e.g., 600 → 644). Preserve the original metadata (e.g., capture the original mode and reapply it after rewrite, or prefer an in-place edit approach that keeps permissions intact).

… for good

`rt_diag` claims its slot by bumping `RT_DIAG_HEAD` FIRST, then marks the slot
odd, writes the bytes, and only then stores the completing even sequence. A
drain landing anywhere in that window got `None` from `read_slot`, hit
`continue`, and then ran `*tail = head` — stepping the tail permanently past a
line whose producer had done nothing wrong. Nobody could emit it afterwards:
the slot is skipped now and the tail never goes back.

`read_slot` returned one `None` for two opposite situations. A LAPPED slot has
been recycled by a later producer and really is gone. An IN-FLIGHT slot is
microseconds from being complete. The drain treated both as "skip it". They are
now distinguished — the sequence is derived from the monotonic claim index, so
`seq < want` is exactly "claimed, not finished yet" — and an in-flight slot
stops the drain instead of being discarded, leaving the line for the next one.

Bounded on purpose. A producer that panics between the head bump and its
completing store leaves a slot odd forever, and an unbounded retry would then
wedge the ring for the life of the process and cost every LATER diagnostic too
— strictly worse than the bug being fixed. Stalling twice on the same index is
taken as a dead producer: that one line is given up and draining continues.

This is what was actually behind the CI failure in
`a_compute_only_program_still_prints_its_safing_diagnostics`. The window is
nanoseconds on an idle machine, which is why it reproduced on a contended
runner and nowhere else, and why the previous commit's shutdown flush did not
help: by the time `stop()` flushed, a drain had already stepped over the line
and there was nothing left to find. The flush is still right — it is what makes
the last diagnostics reach stdout without waiting on a 25 ms poll — but it was
not the cause.

Two tests, both driving the ring by hand so they do not depend on thread
timing:
  - a drain must not advance the tail past an index whose producer has not
    finished writing (asserts on the tail, not on what got emitted, because the
    background drainer would race an "the next drain emits it" assertion)
  - repeated drains must get past a slot whose producer never completes, so the
    bound is real

They share the one process-global ring, and the second one's bounded skip steps
over the first one's in-flight slot, so they take a lock rather than racing each
other.

Ablation: skipping in-flight slots as before fails the first and leaves the
second green.

`read_slot`'s existing flood test asserted `.is_none()` for a recycled slot;
it now asserts `SlotRead::Lapped`, which is the thing it was always trying to
say.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Two user-facing install/cleanup helpers have correctness issues (fish HORUS_PREFIX updates and SHM namespace sanitization consistency) that should be fixed before merging.

Review details

Suppressed comments (2)

install.sh:961

  • In the fish rc path, the installer only checks for the presence of any "set -gx HORUS_PREFIX" line. If a user moved/changed their prefix, an existing line with the OLD value prevents writing the new value, so new shells keep exporting the stale prefix.
    horus_sys/src/shm/mod.rs:261
  • shm_base_dir_for() does not apply the same namespace sanitization/defaulting as shm_namespace()/shm_base_dir(). If a caller passes the raw HORUS_NAMESPACE value (e.g. containing '-' or other chars), this will point at a different on-disk directory than the one real processes use, so cleanup/sweeps can miss live namespaces (or create an unexpected new one).
  • Files reviewed: 80/80 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

…s shell rc

Not all 21 threads needed code: five describe install/uninstall regex problems
this branch already fixed (the `\?`-in-BRE portability bug and the unanchored
`HORUS_PREFIX` guards), and one describes a whitespace run in
`topic/dispatch.rs` that is no longer there. Those are answered on the threads.
Nine were live.

Severity first:

* `uninstall.sh` could replace a shell profile with an empty file. The rewrite
  is `{ grep -vF ... > "$profile.horus-new"; } || true`, and the comment above
  it correctly explains that `|| true` is needed because grep exits 1 when it
  selects no lines — the rc whose only content was our PATH line. But grep also
  exits 2 when it cannot READ the file, and the redirect creates the target
  either way, so `[ -f ]` could not tell those apart and the `mv` clobbered a
  profile it had failed to read. Now requires the profile to be readable and
  refuses to turn a non-empty profile into an empty one unless it genuinely
  held nothing else.

* `shm_base_dir_for` interpolated its namespace argument raw. It is public and
  exists precisely for namespaces that are not this process's own, and callers
  hand the result to `remove_dir_all` — so `../../../../etc` resolved to
  `/dev/shm/horus_../../../../etc`, outside the SHM parent entirely. The module
  already had `sanitize_namespace`, strict enough and used everywhere else;
  this was the one path that skipped it. Ablated: reverting the one-line fix
  turns the new test red with the escaping path in the message.

* `install.sh`'s fish arm tested only for the PRESENCE of `set -gx
  HORUS_PREFIX`, so a line naming a DIFFERENT prefix counted as "already
  configured" and a fish user who reinstalled elsewhere kept an export pointing
  at the tree the installer had just replaced. The POSIX arm was made
  value-aware for exactly this; fish was left behind.

* `start_log_file_drain_once` memoized the negative answer. `OnceLock::
  get_or_init` caches whatever the first call produced, `None` included, so a
  process that built a `Scheduler` before `HORUS_LOG_FILE` was set never wrote
  a log file again for its whole life. A `Mutex` keeps the at-most-one-drain
  guarantee without freezing "no".

* `status.code().unwrap_or(1)` reported every signal death — SIGSEGV, SIGKILL,
  the operator's Ctrl-C — as "exited with code 1". `shell_exit_code` exists for
  this and returns 128+N; this call site was missed when it was introduced.

* `bytes.len() as i32` wraps. A value longer than `i32::MAX` came back negative,
  and negative is the documented "no such param" return, so an overlong string
  was indistinguishable from a missing one. Saturates now.

* `horus doctor` printed fourteen consecutive spaces mid-sentence.

* `control_restart` waited a fixed 400 ms for the scheduler to start. That is a
  bet that the thread got scheduled inside the window; on a loaded runner it
  does not, and the test fails reporting 0 inits as though `RestartNode` were
  broken. Now a bounded poll — same upper bound, no assumption.

* `messages_total` — comment only, and deliberately. The reviewer is right that
  it advances even when `try_send` drops, and the comment defending that said a
  ring nobody drains is "one the subscriber's own staleness sees". That is
  circular: `SubscriptionFreshness::refresh` derives freshness from this counter
  and nothing else, so a publisher hammering a full ring keeps a subscriber
  "fresh" while it receives nothing — the watchdog cannot see the fault it
  watches for. Bumping on success would close it and is the right shape, but it
  moves the counter's meaning at seven call sites across two crates and changes
  when `StalePolicy::SafeState`/`Stop` fire on a live robot. That belongs in its
  own change with its own tests, not in a merge. The comment now records the
  limitation instead of denying it.

Verified: `horus_core --lib` 2597 passed, `horus_sys --lib` 225, `control_restart`
and `log_file_drain_wiring` green, `cargo fmt --check` and `clippy --all-targets
-D warnings` across horus_sys/horus_core/horus_manager clean, and both shell
scripts parse under `bash -n` (they are `#!/bin/bash`; a `sh -n` failure at
line 115 predates this and is a dash-vs-bash artifact, not a syntax error).
`registered.is_paused` is the one-tick `Miss::Skip` flag and is what
`reinit_pending_nodes` actually reads; the operator pause lives in
`node_controls` and is not read there at all. The comment blurred them, so it
justified lifting the operator pause with "would otherwise be skipped by the
very check that would have re-inited it" — which is not true of that check.

The code was already right to clear both. Lifting the operator pause is what
lets the node TICK once re-inited, not what permits the re-init: a wedged node
is usually paused before it is restarted, and without it the node comes back
initialised and still frozen.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

install.sh’s fish HORUS_PREFIX matcher uses \$\ as a literal $ instead of an end-of-line anchor, which can prevent replacing stale exports and reintroduce the prefix-move failure mode.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 80/80 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread install.sh Resolved

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core scheduling/safety behavior and multiple public surfaces across Rust/Python/C++ plus installer scripts, which warrants final domain-owner validation beyond automated review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

horus_manager/src/cargo_gen.rs:919

  • The generated Cargo.toml comment says driver deps are "from [hardware]", but this generator now feeds write_driver_deps from manifest.hardware_entries(), which merges both [hardware] and legacy [drivers]. As written, projects still using [drivers] will get the right deps but a misleading comment in the generated file.
  • Files reviewed: 80/80 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Seven conflicts, all resolved as unions rather than picks, plus one break that
existed in neither branch.

* `communication/mod.rs`, `topic/mod.rs` re-exports — both sides added names.
* `topic/metrics.rs` — this branch removes `recv_failures` from the internal
  `MigrationMetrics` (the real transport never increments it; only `MockTopic`
  does), #203 adds `send_retry_overruns`. Kept both changes: the field is gone
  and the counter is there.
* `topic/mod.rs` `metrics()` — same shape. This branch's hard `0` for
  recv-failures, with its reasoning, plus #203's overrun read.
* `rt_executor.rs` — `flush_rt_diag()` is this branch's wrapper around the very
  `rt_diag_drain(print_line)` main open-codes, so the wrapper stays and #203's
  `hlog_rt::flush()` is added beside it.
* `async_executor.rs`, `compute_executor.rs` — this branch starts the rt_diag
  drain so a safe-state panic on those executors is not queued into a ring
  nothing reads; #203 moved the spawn to `spawn_best_effort` so a non-real-time
  executor stops inheriting the tick thread's policy and reserved-core mask.
  Both are needed and neither replaces the other.
* `scheduler/mod.rs` — this branch wires `set_blackbox_hook`, without which
  horus_net's `record_external_event` dropped every peer-lost and desync event;
  #203 creates the `BbRing` the executors publish into. They reach the same
  recorder by different routes on purpose: the hook is for events raised OFF
  the tick path, the ring for events raised ON it.

The break neither branch had: #203 deleted `EstopTrigger::trigger` once every
caller had moved to `trigger_fmt`, and this branch added a NEW `trigger` caller
in `primitives.rs`. Both are correct in isolation and git merged them cleanly
into code that does not compile — E0599 at primitives.rs:102. Fixed by using
`trigger_fmt`/`format_args!`, which is what that site wanted anyway: it is the
RT panic path, and `format!` allocates on it.

Verified on the merged tree: `horus_core --lib` 2623 passed, `horus_sys --lib`
249, `cargo fmt --check` and `clippy --all-targets -D warnings` clean, and all
of #203's RT guards still hold — rt_helper_thread_isolation 5,
ready_dispatch_pool 2, rt_hlog_off_the_tick_path 4, blackbox_off_the_tick_path
2, recording_is_not_a_syscall_storm 1, rt_thread_status 13, control_restart 2.
`CLI contract (live docs)` clones horus-docs at a SHA pinned in this workflow,
deliberately — the job runs in a context with write access to this repo's
Actions cache, so building an arbitrary ref of another repository would be a
cross-repo escalation this repo's review never sees. Verifying different docs
means bumping the SHA here, in a reviewed PR.

horus-docs#17 is the documentation half of the same bidirectional audit this
branch is the code half of, and it merged as 9a1a0c1. Leaving the pin at
4a612b1 would validate this branch's runtime corrections against pages that
still describe what the runtime did before them — which is precisely the drift
the audit was run to find, reintroduced by the gate meant to catch it.

One commit, whose parent is the old pin. Verified against the pinned commit
itself rather than a local branch:

    HORUS_DOCS_DIR=<horus-docs at 9a1a0c1> \
      cargo test -p horus_manager --test docs_contract -- --include-ignored

14 passed, 0 failed.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new uninstall rc-file rewrite condition uses grep -cvF ... || echo 1, which breaks the “rc contained only our PATH line” cleanup case by contaminating the count output, so prefix PATH cleanup can silently fail.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 81/81 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread uninstall.sh Resolved
Comment thread horus_manager/src/cargo_gen.rs Resolved
Six CI jobs went red on one test, and it was mine:
`an_rc_file_that_is_only_the_prefix_path_line_is_emptied_not_skipped`. Code
Coverage, Cross-Process IPC Tests, CLI Integration Tests and Parity
(macOS ARM64) all run `completion_install_contract`, so a single shell bug
failed four leaves and two aggregates.

The guard I added to stop an unreadable profile being replaced with an empty
file also required "the result is non-empty, unless the profile held nothing
but our lines", and tested that second half with

    [ "$(grep -cvF "$prefix_path_line" "$profile" 2>/dev/null || echo 1)" = "0" ]

`grep -c` PRINTS "0" and EXITS 1 when the count is zero. So on the legitimate
case the `|| echo 1` also ran, the substitution became "0\n1", the comparison
failed, and the rewrite was skipped — leaving the stale PATH line in exactly
the file where removing it matters most. That test's doc comment already
records two earlier ways this line went wrong; this made a third.

`-r` was always the whole guard. An unreadable file is what `grep` exit 2 means
and what `[ -r ]` detects; given a readable profile, an empty result is
legitimate by construction and needs no second opinion. The extra condition
bought nothing and cost the case it was meant to protect.

Verified in all three directions, not just the one the test covers:

  * profile is only our line      -> emptied      (0 bytes)
  * profile has other content     -> ours removed, rest kept
  * profile is unreadable (000)   -> untouched    (36 bytes, contents intact)

and `completion_install_contract` is 26 passed, 0 failed.

Also: the generated Cargo header said "# Driver dependencies (from [hardware])"
for entries that `Manifest::hardware_entries()` merges out of BOTH `[hardware]`
and the legacy `[drivers]` table, so a user still on `[drivers]` would look for
a `[hardware]` table they do not have and conclude the generator invented the
dependency. It names both sources now.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two user-facing developer-experience issues remain (a copy/pastable-but-invalid subs=[Sub(...)] RuntimeError suggestion, and missing stubs for the newly exported peek_topic_type API).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 81/81 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread horus_py/horus/__init__.py Resolved
Comment thread horus_py/src/lib.rs Resolved
…nable

Two review findings, and I got the first one wrong the first time.

`peek_topic_type` is registered on the extension module (lib.rs:123) and
imported by `horus/__init__.py` as `_peek_topic_type`, but it was never declared
in `horus/_horus.pyi`. I previously resolved this thread as stale after
grepping `__init__.py` and finding the symbol there — the finding was about the
STUB, and those are different files. The stub is what a type checker reads, so
to mypy/pyright the function did not exist. Declared now, with a note that
`__init__.py` carries a None-returning fallback for builds without it.

The `RuntimeError` raised when a bare-string subscription collides with a typed
ring suggested `subs=[Sub('cmd_vel', CmdVel)]`. The shape is right — the second
argument is a class, not the string `peek_topic_type` returned — but the message
never said the class has to be imported, so following it literally raises
`NameError` for anyone who has not already got the type in scope. It now names
the import, and says what it means when this build has no such class: the ring
was written by a peer carrying a message type this install does not.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are two concrete correctness issues in changed code (awk split delimiter in install.sh and a fragile relative-path source read in hooks_contract.rs) that should be fixed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

horus_manager/tests/hooks_contract.rs:252

  • This test reads "src/main.rs" relative to the current working directory, but cargo test -p horus_manager does not guarantee the test binary runs with CWD set to the crate root (often it’s the repo root). That makes the test fail to find the file. Resolve the path via CARGO_MANIFEST_DIR like the other source-reading contract tests do.
  • Files reviewed: 81/81 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread install.sh Resolved
@starswallow

Copy link
Copy Markdown
Collaborator Author

Merged as part of the #216 merge stack, squashed onto main.

The benchmark-gate regression that blocked this PR is fixed in the stack. The only per-message additions on the ping-pong path were the two Relaxed fetch_adds behind TopicMetrics (messages_sent on send(), messages_received on recv()). Removing them in an interleaved A/B on the integration tree recovers −18 ns per round trip at 1024 B and −20 ns at 64 B; an atomic RMW on every send and recv is a lock xadd in the middle of the publish path. They now live as per-handle Cell<u64>s on RingTopic (which is deliberately !Sync), so your per-handle semantics and the MockTopic parity argument are unchanged, and TopicMetrics documents that a clone counts its own traffic.

Also folded in: the review follow-ups already on this branch, the #205 × #189 HORUS_SIM_MODE conflict resolution (your sim_mode_enabled(value) structure with the shared horus_sys::env vocabulary), and a rustdoc link fix (crate::testing::MockTopic is test-utils-gated, so it cannot be a doc link once #205 turns link resolution on).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants