Repository navigation
docs/ci: make the codebase readable and its documentation checkable - #205
starswallow wants to merge 7 commits into
Conversation
`cargo doc -p horus_core --open` rendered a front page with one module on it
(`terminal`), zero structs, and zero entries in the search index — for a
112k-line runtime crate. The standard first move for a Rust developer meeting a
new crate returned nothing.
Ten of the eleven modules carry `#[doc(hidden)]`, and that IS deliberate: the
crate note says "internal crate, use `horus::prelude`", and horus_core/README.md
has a whole "How to read the internals" section about it. The problem is the
escape hatch it documents:
RUSTDOCFLAGS="-Z unstable-options --document-hidden-items" \
cargo +nightly doc -p horus_core --no-deps --document-private-items
That needs nightly and an unstable flag, because `--document-private-items`
disables rustdoc's private-stripping pass and not its hidden-stripping pass.
(Verified here rather than assumed: with the flag, the front page gained exactly
one module — `utils`, the `pub(crate)` one — and none of the hidden ten.)
So the attributes are now `cfg_attr`'d behind a docs-only `internal-docs`
feature. Default output is byte-for-byte what it was, so the user-facing
boundary is untouched; a maintainer on a stable toolchain gets all eleven
modules and a populated search index with
cargo doc -p horus_core --no-deps --document-private-items \
--features internal-docs --open
The README now leads with that and keeps the nightly form as the no-feature
alternative.
CI documents horus_core a second time with the feature on. That is the half
worth keeping even if the rest is bikeshedded: rustdoc does not resolve links
inside a hidden module, so `-D warnings` had been holding roughly a quarter of
the crate and silently skipping the rest.
The same CI job gains a `Doctests` step, for a neighbouring reason: `grep -rn
-- '--doc' .github/workflows/` returned nothing, so `cargo test --lib` never ran
the examples and `cargo doc` only rendered them. 21 pass today. Its comment says
plainly that it is a thin gate — 193 of the workspace's 214 examples are fenced
```rust,ignore``` and are skipped — because a step that protects 21 examples
should not be described as protecting the examples.
…cealing
Documenting horus_core with its internals visible turned the `Documentation`
job red immediately, on four broken links that had accumulated unseen — rustdoc
does not resolve links inside a hidden module, so `-D warnings` never looked at
them. A fifth defect was visible all along and is fixed here because it is the
same failure: prose separated from the thing it describes.
* `depth_image.rs` documented `set_depth` as returning
`[`HorusError::OutOfRange`]`. That variant does not exist — the error
refactor folded it into `InvalidInput(ValidationError::…)` (error.rs:399
records the replacement) and the doc kept naming the deleted one. A
maintainer reading its `# Errors` section was told to match on a variant
they cannot write, and nothing in the build disagreed.
* `tensor_handle.rs` and `scheduler/mod.rs` each linked a private item from a
public doc comment (`pool_registry::global_pool`,
`rt::set_realtime_priority`). Both become plain code spans, which still tell
a reader where to look.
* `error.rs` linked `[`Error`]`, ambiguous between a type alias and a derive
macro in that scope. Disambiguated with `tyalias@`.
* `Scheduler` — the type a HORUS program is built around — had NO doc comment.
Its description, "Central orchestrator: holds nodes, drives the tick loop.",
was stranded one item higher, sitting on top of `PresenceEntry`'s own
comment, so a private struct rendered as the orchestrator and the public
type rendered blank.
Two variables in the same framework disagreed about the word "no". `HORUS_LOG_FILE=yes` was **false** — that site matched only `"true"` and `"1"`. `HORUS_SIM_MODE=no` was **true** — that site treated anything it did not recognise as an opt-in, so a deploy script saying "no" swapped every driver for an inert `SimStubNode`: actuators never commanded, sensors never read, and nothing said so. The comment already sitting there records the same class of bug being fixed once before, for `=false`; `no` and `off` were still wrong. `horus_sys::env::env_flag` is now the one answer. `1/true/yes/on` are true, `0/false/no/off` and empty are false, ASCII case-insensitive, whitespace trimmed — a value out of a YAML launch file or a shell here-doc usually carries some. An unrecognised value returns `None` rather than `false`, leaving the caller's default in force. `HORUS_SIM_MODE=maybe` therefore keeps simulation off instead of enabling it, which is the safe direction for a flag that disconnects the robot from its hardware — and it matches what the reference page already promises for `HORUS_SHM_PREFAULT`: "an unrecognised value keeps the default rather than meaning off". Routed: `HORUS_LOG_FILE`, `HORUS_SIM_MODE`, and the opt-out `HORUS_NET_ENABLED` (which accepted `0`/`false` but not `no`/`off`, so two of the four words that mean "off" everywhere else left networking running). Deliberately NOT routed: `HORUS_RT_ALLOW_CORE_SHARING` and `HORUS_WIN_REALTIME_CLASS`. Their exact vocabularies are published in horus-docs — the first is documented as case-sensitive on purpose — so changing them needs a paired documentation change rather than a drive-by here. The parsing half is a separate `parse_flag` so it can be tested without mutating process-global state: `set_var` is not safe against a concurrent `getenv` on another thread, and these test binaries run in parallel. The two divergences above are pinned as tests, by name.
`RuntimeParams`'s entire doc block — the description AND a working doctest — sat on `ParamChangeCallback`, a type alias declared between the comment and the struct it belonged to. So the parameter store rendered undocumented on docs.rs while a callback alias carried its example, and `cargo test --doc` reported the example as `params::ParamChangeCallback`. This is the same failure as the `Scheduler` one in the previous commit and it has the same cause: an item inserted between a doc comment and its target, with nothing in the build that can notice. Both are now attached to the right type; the doctest reports as `params::RuntimeParams (line 71)` and still passes.
CONTRIBUTING.md told a contributor that `readme_contract`'s Quick Start test is
run by nothing — "no workflow mentions `readme_contract`" — and handed them a
command to compile it by hand. docs/TESTING.md said the same of `docs_parity`:
"`grep -rn docs_parity .github/` returns nothing. The suite that exists to catch
*undocumented* new surface never runs."
Both are false. `docs-contract.yml` runs
cargo test -p horus_manager --test readme_contract -- --include-ignored
cargo test -p horus_manager --test docs_parity -- --ignored --nocapture
Each statement was true when written and was falsified by the commit that added
the invocation, without the prose being updated with it — which is the exact
failure TESTING.md's own closing rule warns about, in the mirror direction.
That direction is the one that costs a newcomer: it sends them to hand-run
suites CI already covers, and it teaches them the workflows cannot be trusted.
So the closing paragraph now says a table asserting something is *unrun* is
itself untested prose, and tells the reader to re-derive a row with
`grep -rn '<target>' .github/` before relying on it.
`cpp_toolchain_contract` really is unrun and keeps its row. So does the
`nextest` config — `grep -rn nextest .github/` still returns nothing — and
both were re-checked rather than carried over.
There was a problem hiding this comment.
🟡 Changes recommended
It introduces at least one concrete docs/CI-blocking issue (invalid rustdoc link disambiguator) and has test/doctest assertions that can be flaky due to assuming specific env vars are unset.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Improves maintainability and onboarding by making horus_core’s internal documentation renderable on stable (for maintainers), enforcing rustdoc/doctest checks in CI, and eliminating inconsistent boolean environment-variable parsing that previously caused unsafe behavior.
Changes:
- Add
internal-docsfeature to lift#[doc(hidden)]for maintainer-facing rustdoc, and run a second rustdoc pass in CI with link warnings enabled. - Introduce
horus_sys::env::{env_flag, parse_flag}as the single boolean env-var vocabulary and migrate key call sites (sim mode, log file, networking opt-out) to it. - Fix multiple documentation issues (misattached doc blocks, broken intra-doc links, and stale onboarding/testing docs).
File summaries
| File | Description |
|---|---|
| horus_sys/src/lib.rs | Exposes new env module (needs docs table update to mention it). |
| horus_sys/src/env/mod.rs | Adds shared parsing for boolean env vars plus tests/doctest. |
| horus_core/src/scheduling/scheduler/mod.rs | Uses env_flag for HORUS_NET_ENABLED and improves Scheduler docs. |
| horus_core/src/params.rs | Reattaches docs/doctest to RuntimeParams and documents ParamChangeCallback. |
| horus_core/src/memory/tensor_handle.rs | Adjusts doc wording to avoid brittle/broken link to global_pool. |
| horus_core/src/memory/depth_image.rs | Fixes broken error-variant link in set_depth docs. |
| horus_core/src/lib.rs | Gates #[doc(hidden)] behind internal-docs to make internals browsable on stable when desired. |
| horus_core/src/error.rs | Tweaks alias docs/linking (currently introduces an invalid rustdoc disambiguator). |
| horus_core/src/drivers/mod.rs | Switches HORUS_SIM_MODE parsing to env_flag. |
| horus_core/src/core/log_buffer.rs | Switches HORUS_LOG_FILE parsing to env_flag. |
| horus_core/README.md | Updates maintainer instructions to use internal-docs on stable. |
| horus_core/Cargo.toml | Adds internal-docs feature. |
| docs/TESTING.md | Corrects CI coverage claims for docs-related ignored tests; updates guidance about table staleness. |
| CONTRIBUTING.md | Corrects guidance about readme_contract being run in CI. |
| .github/workflows/ci.yml | Adds doctest job step; adds second cargo doc run for horus_core with internal-docs. |
Review details
Suppressed comments (1)
horus_sys/src/env/mod.rs:127
- This unit test assumes
HORUS_DEFINITELY_UNSET_XYZZYis unset, which can be false in some environments and make the test flaky. Generate a unique variable name instead (without mutating process-global env).
#[test]
fn env_flag_reports_none_for_an_unset_variable() {
assert_eq!(env_flag("HORUS_DEFINITELY_UNSET_XYZZY"), None);
}
- Files reviewed: 15/15 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Benchmark gateBaseline: rolling window of 8 run(s) on Result: PASS
|
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
40.0 ns | 40.0 ns | +0.0% | 65.0 ns | INCONCLUSIVE |
p95 |
50.0 ns | 50.0 ns | +0.0% | 100.0 ns | INCONCLUSIVE |
p99 |
69.0 ns | 69.0 ns | +0.0% | 169.0 ns | INCONCLUSIVE |
p99.9 |
80.0 ns | 80.0 ns | +0.0% | 330.0 ns | INCONCLUSIVE |
p99.99 |
— | 110.0 ns | — | — | N/A (samples) |
max |
15.92 µs | 14.79 µs | -7.1% | 31.85 µs | INCONCLUSIVE |
max_jitter |
15.91 µs | 14.78 µs | -7.1% | 31.83 µs | INCONCLUSIVE |
p99/median |
1.73x | 1.73x | +0.0% | 2.23x | INCONCLUSIVE |
p99.9/median |
2.00x | 2.00x | +0.0% | 3.00x | INCONCLUSIVE |
ns/msg |
170.3 ns | 170.6 ns | +0.1% | 220.3 ns | INCONCLUSIVE |
Adaptive_Imu@304B — 50000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
40.0 ns | 40.0 ns | +0.0% | 65.0 ns | INCONCLUSIVE |
p95 |
60.0 ns | 60.0 ns | +0.0% | 110.0 ns | INCONCLUSIVE |
p99 |
89.5 ns | 89.0 ns | -0.6% | 189.5 ns | INCONCLUSIVE |
p99.9 |
110.0 ns | 110.0 ns | +0.0% | 360.0 ns | INCONCLUSIVE |
p99.99 |
— | 140.0 ns | — | — | N/A (samples) |
max |
13.85 µs | 13.84 µs | -0.1% | 27.70 µs | INCONCLUSIVE |
max_jitter |
13.83 µs | 13.81 µs | -0.1% | 27.66 µs | INCONCLUSIVE |
p99/median |
2.24x | 2.23x | -0.6% | 2.80x | INCONCLUSIVE |
p99.9/median |
2.75x | 2.75x | +0.0% | 3.75x | INCONCLUSIVE |
ns/msg |
270.3 ns | 270.5 ns | +0.1% | 324.3 ns | INCONCLUSIVE |
Adaptive_JointCommand@928B — 50000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
80.0 ns | 80.0 ns | +0.0% | 105.0 ns | INCONCLUSIVE |
p95 |
100.0 ns | 100.0 ns | +0.0% | 150.0 ns | INCONCLUSIVE |
p99 |
130.0 ns | 130.0 ns | +0.0% | 230.0 ns | INCONCLUSIVE |
p99.9 |
160.0 ns | 160.0 ns | +0.0% | 410.0 ns | INCONCLUSIVE |
p99.99 |
— | 751.0 ns | — | — | N/A (samples) |
max |
15.53 µs | 16.68 µs | +7.4% | 31.07 µs | INCONCLUSIVE |
max_jitter |
15.48 µs | 16.63 µs | +7.4% | 30.97 µs | INCONCLUSIVE |
p99/median |
1.62x | 1.62x | +0.0% | 2.12x | INCONCLUSIVE |
p99.9/median |
2.00x | 2.00x | +0.0% | 3.00x | INCONCLUSIVE |
ns/msg |
335.0 ns | 332.9 ns | -0.6% | 402.0 ns | INCONCLUSIVE |
Adaptive_LaserScan@1480B — 10000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
89.0 ns | 89.0 ns | +0.0% | 114.0 ns | INCONCLUSIVE |
p95 |
110.0 ns | 110.0 ns | +0.0% | 160.0 ns | INCONCLUSIVE |
p99 |
150.0 ns | 140.0 ns | -6.7% | 250.0 ns | INCONCLUSIVE |
p99.9 |
170.0 ns | 170.0 ns | +0.0% | 420.0 ns | INCONCLUSIVE |
p99.99 |
— | 341.0 ns | — | — | N/A (samples) |
max |
13.15 µs | 11.82 µs | -10.1% | 26.30 µs | INCONCLUSIVE |
max_jitter |
13.09 µs | 11.76 µs | -10.1% | 26.18 µs | INCONCLUSIVE |
p99/median |
1.69x | 1.62x | -3.6% | 2.19x | INCONCLUSIVE |
p99.9/median |
1.91x | 2.00x | +4.7% | 2.91x | INCONCLUSIVE |
ns/msg |
689.0 ns | 689.8 ns | +0.1% | 826.8 ns | INCONCLUSIVE |
Topic_cross_process_oneway_paced_stream@1024B — 200000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
260.5 ns | 245.0 ns | -6.0% | 316.8 ns | INCONCLUSIVE |
p95 |
323.0 ns | 310.0 ns | -4.0% | 468.3 ns | INCONCLUSIVE |
p99 |
597.5 ns | 681.0 ns | +14.0% | 905.9 ns | INCONCLUSIVE |
p99.9 |
4.92 µs | 5.56 µs | +13.1% | 16.22 µs | INCONCLUSIVE |
p99.99 |
15.34 µs | 12.80 µs | -16.5% | 24.58 µs | INCONCLUSIVE |
max |
35.19 µs | 24.70 µs | -29.8% | 79.53 µs | INCONCLUSIVE |
max_jitter |
34.99 µs | 24.50 µs | -30.0% | 79.20 µs | INCONCLUSIVE |
p99/median |
2.33x | 2.62x | +12.4% | 3.91x | INCONCLUSIVE |
p99.9/median |
19.48x | 21.38x | +9.8% | 71.66x | INCONCLUSIVE |
ns/msg |
10.00 µs | 10.00 µs | +0.0% | 12.00 µs | INCONCLUSIVE |
Topic_cross_process_oneway_paced_stream@64B — 200000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
170.5 ns | 165.0 ns | -3.2% | 203.1 ns | INCONCLUSIVE |
p95 |
205.5 ns | 195.0 ns | -5.1% | 267.8 ns | INCONCLUSIVE |
p99 |
468.0 ns | 496.0 ns | +6.0% | 1.03 µs | INCONCLUSIVE |
p99.9 |
3.75 µs | 3.74 µs | -0.1% | 13.44 µs | INCONCLUSIVE |
p99.99 |
13.08 µs | 14.70 µs | +12.4% | 26.19 µs | INCONCLUSIVE |
max |
51.25 µs | 42.29 µs | -17.5% | 270.47 µs | INCONCLUSIVE |
max_jitter |
51.13 µs | 42.19 µs | -17.5% | 270.28 µs | INCONCLUSIVE |
p99/median |
2.70x | 3.01x | +11.3% | 5.43x | INCONCLUSIVE |
p99.9/median |
20.29x | 22.68x | +11.8% | 68.21x | INCONCLUSIVE |
ns/msg |
10.00 µs | 10.00 µs | +0.0% | 12.00 µs | INCONCLUSIVE |
Topic_cross_process_oneway_pingpong@1024B — 1000000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
278.0 ns | 271.0 ns | -2.5% | 305.8 ns | INCONCLUSIVE |
p95 |
361.0 ns | 355.0 ns | -1.7% | 415.1 ns | INCONCLUSIVE |
p99 |
408.5 ns | 395.0 ns | -3.3% | 508.5 ns | INCONCLUSIVE |
p99.9 |
1.40 µs | 1.42 µs | +1.1% | 1.96 µs | INCONCLUSIVE |
p99.99 |
10.48 µs | 10.64 µs | +1.5% | 15.72 µs | INCONCLUSIVE |
max |
27.51 µs | 20.65 µs | -24.9% | 94.84 µs | INCONCLUSIVE |
max_jitter |
27.31 µs | 20.46 µs | -25.1% | 94.60 µs | INCONCLUSIVE |
p99/median |
1.45x | 1.46x | +0.5% | 1.95x | INCONCLUSIVE |
p99.9/median |
4.97x | 5.23x | +5.3% | 7.16x | INCONCLUSIVE |
ns/msg |
344.5 ns | 339.5 ns | -1.5% | 413.4 ns | INCONCLUSIVE |
Topic_cross_process_oneway_pingpong@64B — 1000000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
150.0 ns | 150.0 ns | +0.0% | 179.7 ns | INCONCLUSIVE |
p95 |
180.0 ns | 180.0 ns | +0.0% | 230.0 ns | INCONCLUSIVE |
p99 |
200.0 ns | 200.0 ns | +0.0% | 300.0 ns | INCONCLUSIVE |
p99.9 |
728.5 ns | 842.0 ns | +15.6% | 1.27 µs | INCONCLUSIVE |
p99.99 |
7.78 µs | 7.76 µs | -0.3% | 11.67 µs | INCONCLUSIVE |
max |
23.44 µs | 18.62 µs | -20.6% | 60.24 µs | INCONCLUSIVE |
max_jitter |
23.33 µs | 18.50 µs | -20.7% | 60.09 µs | INCONCLUSIVE |
p99/median |
1.34x | 1.33x | -0.4% | 1.84x | INCONCLUSIVE |
p99.9/median |
4.79x | 5.71x | +19.2% | 7.76x | INCONCLUSIVE |
ns/msg |
200.6 ns | 199.2 ns | -0.7% | 250.6 ns | INCONCLUSIVE |
Topic_cross_process_unpaced_queue_delay@64B — 200000 samples
| metric | baseline | current | change | band (<=) | verdict |
|---|---|---|---|---|---|
median |
147.5 ns | 150.0 ns | +1.7% | 197.9 ns | INCONCLUSIVE |
p95 |
205.0 ns | 200.0 ns | -2.4% | 267.3 ns | INCONCLUSIVE |
p99 |
230.5 ns | 256.0 ns | +11.1% | 330.5 ns | INCONCLUSIVE |
p99.9 |
3.18 µs | 4.96 µs | +56.0% | 13.64 µs | INCONCLUSIVE |
p99.99 |
9.26 µs | 9.38 µs | +1.3% | 13.89 µs | INCONCLUSIVE |
max |
38.40 µs | 37.41 µs | -2.6% | 80.93 µs | INCONCLUSIVE |
max_jitter |
38.30 µs | 37.30 µs | -2.6% | 80.76 µs | INCONCLUSIVE |
p99/median |
1.60x | 1.73x | +8.0% | 2.10x | INCONCLUSIVE |
p99.9/median |
23.14x | 33.06x | +42.9% | 108.60x | INCONCLUSIVE |
ns/msg |
112.7 ns | 110.1 ns | -2.3% | 162.7 ns | INCONCLUSIVE |
Blocking metrics: median, p99/median. Report-only: p95, p99, p99.9, p99.99, max, max_jitter, p99.9/median, ns/msg. Tail metrics past p99.9 are advisory on a shared runner — one host preemption owns max.
Raw benchmark output
7949.7 18979.7 | 999 100
leg A->B (apparent) 1000000 100.6 150.5 230.6 1292.6 11882.2 34653.7 | 979 100
leg B->A (apparent) 1000000 99.8 140.3 200.4 490.7 8765.9 27840.8 | 984 100
responder turnaround 1000000 29.9 30.3 40.1 40.1 160.3 37619.1 | 379 87
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=12.7 ns window=0.398 s
measured window: minor_faults=0 major_faults=0 ctx_switches vol=78 invol=4
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
cross-core TSC offset theta_hat = +5.1 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7676 <-> pid 7687
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way, offset-free 1000000 195.1 270.7 395.4 1417.7 10564.3 18999.7 | 1000 100
round trip (RTT) 1000000 430.6 581.1 821.5 2975.2 21279.0 38029.7 | 1000 100
RTT halved (one-way) 1000000 215.3 290.5 410.8 1487.6 10639.5 19014.8 | 1000 100
leg A->B (apparent) 1000000 170.5 270.7 431.0 1593.1 19966.8 37679.2 | 995 100
leg B->A (apparent) 1000000 170.1 260.1 430.6 1392.3 9928.0 32539.6 | 994 100
responder turnaround 1000000 29.9 30.3 40.1 40.1 180.3 22651.7 | 907 98
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=40.1 ns window=0.679 s
measured window: minor_faults=0 major_faults=0 ctx_switches vol=134 invol=6
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
cross-core TSC offset theta_hat = +5.3 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7676 <-> pid 7689
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 119.8 170.1 510.7 2534.4 10669.3 21549.3 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=8.6 ns window=2.000 s
measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=393 invol=11
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.1 ns (add to correct).
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7676 <-> pid 7691
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 200.0 260.1 680.8 5560.0 12803.1 24705.3 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=20.4 ns window=2.000 s
measured backlog: 0.03 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=393 invol=13
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.3 ns (add to correct).
== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
!! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
!! time spent waiting in the ring. It is a backpressure/throughput figure.
!! Do not quote it as an IPC latency; that is the exact mistake the retired
!! `Topic_cross_process` metric shipped for months. Read `measured backlog`
!! below: ~1 message means the subscriber kept up and no queue formed.
json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7676 <-> pid 7693
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
publish->receive incl. queue 200000 110.0 150.1 260.5 4958.9 13484.3 21889.9 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=21.3 ns window=0.022 s
measured backlog: 1.36 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=4 invol=0
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +5.1 ns (add to correct).
SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
json metric key what it measures n median p99 p99.9 p99.99 max
------------------------------------------------------------------------------------------------------------------------------------------
Topic_cross_process_oneway_pingpong@64B ONE-WAY latency 1000000 150.1 200.0 811.3 7759.1 17416.8
Topic_cross_process_oneway_pingpong@1024B ONE-WAY latency 1000000 270.7 395.4 1417.7 10564.3 18999.7
Topic_cross_process_oneway_paced_stream@64B ONE-WAY latency 200000 170.1 510.7 2534.4 10669.3 21549.3
Topic_cross_process_oneway_paced_stream@1024B ONE-WAY latency 200000 260.1 680.8 5560.0 12803.1 24705.3
Topic_cross_process_unpaced_queue_delay@64B QUEUE DELAY (not latency) 200000 150.1 260.5 4958.9 13484.3 21889.9
------------------------------------------------------------------------------------------------------------------------------------------
Instrumentation is INCLUDED in every figure and never subtracted.
`Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
`RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
the row to compare against any competitor publishing a halved ping-pong.
`Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
however long the message waited in the ring, so it scales with the achieved backlog
(printed in its scenario block) and is NOT a latency.
A figure without its payload size, ring depth and topology is not comparable to anything.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.36 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-2.json
keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
will report it as missing and fail the gate until the window is refreshed —
that is intended, and is why the key was not silently reused.
Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
6 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║ HORUS Robotics Message Types Benchmark ║
╠══════════════════════════════════════════════════════════════════╣
║ Testing real-world robotics message latency (REP 2014) ║
╚══════════════════════════════════════════════════════════════════╝
Platform: AMD EPYC 7763 64-Core Processor (4 cores)
Iterations: 50000
Warmup: 5000
╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type │ Size (bytes) │ Typical Rate │ Use Case ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 1000+ Hz │ Velocity control commands ║
║ Imu │ 296 │ 500+ Hz │ IMU sensor fusion ║
║ LaserScan │ 1480 │ 10-40 Hz │ 2D lidar navigation ║
║ JointCommand │ 1032 │ 500+ Hz │ Manipulator control ║
╚═════════════════════════════════════════════════════════════════════════════╝
[Topic] Running benchmarks — timing tx.send() enqueue cost only
(cross-thread consumer drains in the background; no sample
waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
CmdVel 16 bytes │ median: 40ns │ p99: 69ns │ CV: 2.9034
Imu 304 bytes │ median: 40ns │ p99: 90ns │ CV: 2.6178
LaserScan 1480 bytes │ median: 89ns │ p99: 140ns │ CV: 1.3232
JointCommand 928 bytes │ median: 80ns │ p99: 130ns │ CV: 1.5270
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message │ Size │ Median (ns) │ p99 (ns) │ CV │ Misses │ Sustained rate ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 40 │ 69 │ 2.9034 │ 0 │ 5.86 M msg/s ║
║ Imu │ 304 │ 40 │ 90 │ 2.6178 │ 0 │ 3.70 M msg/s ║
║ LaserScan │ 1480 │ 89 │ 140 │ 1.3232 │ 0 │ 1.45 M msg/s ║
║ JointCommand │ 928 │ 80 │ 130 │ 1.5270 │ 0 │ 3.00 M msg/s ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.
╔══════════════════════════════════════════════════════════════════╗
║ REAL-TIME SUITABILITY ANALYSIS ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop ║
║ guarantees: a real loop also pays the receive side, the node's ║
║ compute, and the scheduler's jitter, none of which is measured ║
║ here. The headroom is printed so a pass by four orders of ║
║ magnitude cannot be read as a tight result. ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel (1kHz control): ~ PASS by 14493x — gate not discriminating (p99=69ns vs 1000000ns)
║ CmdVel (10kHz control): ~ PASS by 1449x — gate not discriminating (p99=69ns vs 100000ns)
║ Imu (500Hz fusion) : ~ PASS by 22222x — gate not discriminating (p99=90ns vs 2000000ns)
║ LaserScan (40Hz lidar) : ~ PASS by 178571x — gate not discriminating (p99=140ns vs 25000000ns)
║ JointCmd (500Hz control): ~ PASS by 15385x — gate not discriminating (p99=130ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝
Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-2.json
===== repetition 3/3 =====
HORUS cross-process benchmark — ONE-WAY LATENCY (headline) + unpaced queue delay
Two OS processes on two distinct physical cores over the real Topic SHM transport.
Every handle holds exactly one role, so the `role == Both` same-thread fast path that
produces the repo's headline send-cost figure is never taken. Round-trip rows are
labelled where they are a halved round trip. See the module header for the clock
decomposition and for why the `Topic_cross_process` JSON key was retired.
MACHINE — every nanosecond below is conditional on this block
------------------------------------------------------------------------------------------------------------------------------------------
cpu model : AMD EPYC 7763 64-Core Processor (2 physical / 4 logical cores)
kernel : 6.17.0-1022-azure (x86_64)
benchmark cores : cpu3 = process A (pinger / subscriber), cpu1 = process B (responder / publisher) [DistinctPhysical]
governor : cpu3=n/a cpu1=n/a driver=n/a
scaling range : n/a .. n/a (hw max n/a) turbo_disabled=unknown
clocksource : tsc tsc flags: [tsc, rdtscp, constant_tsc, tsc_reliable, nonstop_tsc]
TSC rate (calibrated) : 2445.523 MHz (0.4089 ns/cycle) — a WALL clock; NOT the core clock
clock instrumentation : 49 cycles = 20.0 ns for serialize()+rdtsc()+rdtscp(), INCLUDED in every sample and never subtracted
transparent_hugepage : [always] madvise never
sched_rt_runtime/period: 950000 / 1000000 us perf_event_paranoid=4
isolcpus=(none) nohz_full=(none) smt_active=1 mlockall(MCL_CURRENT)=true
------------------------------------------------------------------------------------------------------------------------------------------
running ping-pong (one-way latency), 64 B payload ...
running ping-pong (one-way latency), 1024 B payload ...
running paced stream (one-way latency), 64 B payload ...
running paced stream (one-way latency), 1024 B payload ...
running unpaced stream (QUEUE DELAY, not latency), 64 B payload ...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! READ THIS BEFORE QUOTING ANY NUMBER BELOW
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.16 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7715 <-> pid 7724
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way, offset-free 1000000 114.9 140.3 195.1 841.5 7759.1 18618.9 | 984 100
round trip (RTT) 1000000 270.3 320.6 420.8 1733.0 15989.2 37268.1 | 998 100
RTT halved (one-way) 1000000 135.1 160.3 210.4 866.5 7994.6 18634.1 | 998 100
leg A->B (apparent) 1000000 124.7 155.0 225.3 1377.2 13819.9 35990.7 | 984 100
leg B->A (apparent) 1000000 95.3 125.5 185.6 355.8 8731.1 37082.9 | 998 100
responder turnaround 1000000 29.9 30.3 40.1 40.1 170.1 32409.0 | 478 89
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=13.1 ns window=0.391 s
measured window: minor_faults=0 major_faults=0 ctx_switches vol=77 invol=1
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
cross-core TSC offset theta_hat = +14.7 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7715 <-> pid 7726
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way, offset-free 1000000 195.5 265.4 395.4 1372.3 10639.4 24915.3 | 997 100
round trip (RTT) 1000000 430.6 561.0 821.5 2845.2 21369.3 49860.9 | 997 100
RTT halved (one-way) 1000000 215.3 280.5 410.8 1422.6 10684.6 24930.5 | 997 100
leg A->B (apparent) 1000000 195.1 275.2 455.5 1477.4 20712.5 39066.1 | 1000 100
leg B->A (apparent) 1000000 165.6 245.8 426.1 1037.4 9933.3 49555.9 | 998 97
responder turnaround 1000000 29.9 30.3 40.1 40.1 160.3 22360.9 | 956 100
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=38.8 ns window=0.668 s
measured window: minor_faults=0 major_faults=0 ctx_switches vol=131 invol=4
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
cross-core TSC offset theta_hat = +14.7 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7715 <-> pid 7728
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 105.5 155.4 496.4 9582.8 67418.3 80452.3 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=8.6 ns window=2.000 s
measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=14
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +14.7 ns (add to correct).
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7715 <-> pid 7730
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 175.4 235.9 546.3 3641.8 12668.5 20362.5 | 199 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=20.9 ns window=2.000 s
measured backlog: 0.02 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=393 invol=15
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +14.7 ns (add to correct).
== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
!! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
!! time spent waiting in the ring. It is a backpressure/throughput figure.
!! Do not quote it as an IPC latency; that is the exact mistake the retired
!! `Topic_cross_process` metric shipped for months. Read `measured backlog`
!! below: ~1 message means the subscriber kept up and no queue formed.
json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7715 <-> pid 7732
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
publish->receive incl. queue 200000 95.3 125.5 256.0 7639.3 9382.5 41240.7 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=19.6 ns window=0.022 s
measured backlog: 1.16 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=4 invol=0
core freq cpu3 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
core freq cpu1 (measured window only): start=n/a end=n/a min=0MHz max=0MHz mean=0MHz spread=0.00% (n=0)
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +14.7 ns (add to correct).
SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
json metric key what it measures n median p99 p99.9 p99.99 max
------------------------------------------------------------------------------------------------------------------------------------------
Topic_cross_process_oneway_pingpong@64B ONE-WAY latency 1000000 140.3 195.1 841.5 7759.1 18618.9
Topic_cross_process_oneway_pingpong@1024B ONE-WAY latency 1000000 265.4 395.4 1372.3 10639.4 24915.3
Topic_cross_process_oneway_paced_stream@64B ONE-WAY latency 200000 155.4 496.4 9582.8 67418.3 80452.3
Topic_cross_process_oneway_paced_stream@1024B ONE-WAY latency 200000 235.9 546.3 3641.8 12668.5 20362.5
Topic_cross_process_unpaced_queue_delay@64B QUEUE DELAY (not latency) 200000 125.5 256.0 7639.3 9382.5 41240.7
------------------------------------------------------------------------------------------------------------------------------------------
Instrumentation is INCLUDED in every figure and never subtracted.
`Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
`RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
the row to compare against any competitor publishing a halved ping-pong.
`Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
however long the message waited in the ring, so it scales with the achieved backlog
(printed in its scenario block) and is NOT a latency.
A figure without its payload size, ring depth and topology is not comparable to anything.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! CPU governor is ?/?, not `performance`. RDTSC is a wall clock while the work costs a fixed number of CORE cycles, so every nanosecond below scales with a frequency the governor is free to change. iceoryx2's maintainers measured ~300 ns of difference from the governor alone.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.16 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-3.json
keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
will report it as missing and fail the gate until the window is refreshed —
that is intended, and is why the key was not silently reused.
Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
6 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║ HORUS Robotics Message Types Benchmark ║
╠══════════════════════════════════════════════════════════════════╣
║ Testing real-world robotics message latency (REP 2014) ║
╚══════════════════════════════════════════════════════════════════╝
Platform: AMD EPYC 7763 64-Core Processor (4 cores)
Iterations: 50000
Warmup: 5000
╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type │ Size (bytes) │ Typical Rate │ Use Case ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 1000+ Hz │ Velocity control commands ║
║ Imu │ 296 │ 500+ Hz │ IMU sensor fusion ║
║ LaserScan │ 1480 │ 10-40 Hz │ 2D lidar navigation ║
║ JointCommand │ 1032 │ 500+ Hz │ Manipulator control ║
╚═════════════════════════════════════════════════════════════════════════════╝
[Topic] Running benchmarks — timing tx.send() enqueue cost only
(cross-thread consumer drains in the background; no sample
waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
CmdVel 16 bytes │ median: 40ns │ p99: 69ns │ CV: 2.3771
Imu 304 bytes │ median: 40ns │ p99: 89ns │ CV: 2.3190
LaserScan 1480 bytes │ median: 89ns │ p99: 150ns │ CV: 1.9446
JointCommand 928 bytes │ median: 80ns │ p99: 130ns │ CV: 1.9443
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message │ Size │ Median (ns) │ p99 (ns) │ CV │ Misses │ Sustained rate ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 40 │ 69 │ 2.3771 │ 0 │ 5.87 M msg/s ║
║ Imu │ 304 │ 40 │ 89 │ 2.3190 │ 0 │ 3.69 M msg/s ║
║ LaserScan │ 1480 │ 89 │ 150 │ 1.9446 │ 0 │ 1.44 M msg/s ║
║ JointCommand │ 928 │ 80 │ 130 │ 1.9443 │ 0 │ 2.98 M msg/s ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.
╔══════════════════════════════════════════════════════════════════╗
║ REAL-TIME SUITABILITY ANALYSIS ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop ║
║ guarantees: a real loop also pays the receive side, the node's ║
║ compute, and the scheduler's jitter, none of which is measured ║
║ here. The headroom is printed so a pass by four orders of ║
║ magnitude cannot be read as a tight result. ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel (1kHz control): ~ PASS by 14493x — gate not discriminating (p99=69ns vs 1000000ns)
║ CmdVel (10kHz control): ~ PASS by 1449x — gate not discriminating (p99=69ns vs 100000ns)
║ Imu (500Hz fusion) : ~ PASS by 22472x — gate not discriminating (p99=89ns vs 2000000ns)
║ LaserScan (40Hz lidar) : ~ PASS by 166667x — gate not discriminating (p99=150ns vs 25000000ns)
║ JointCmd (500Hz control): ~ PASS by 15385x — gate not discriminating (p99=130ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝
Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-3.json
Gated against a rolling window of recent main runs. Each metric's band is max(relative, center + k·sigma_observed, absolute floor), so it is derived from the noise the baseline itself measured — where a band is printed wider than the change you care about, this run did not detect a regression and could not have. Thresholds and their reasoning: RegressionPolicy in benchmarks/src/output.rs.
Commit: 89fefd8
* `env` was missing from horus_sys's crate-level `## Modules` table. Correct and now listed — a module table that omits a module is worse than no table, because a reader treats it as exhaustive. * The doctest and the unit test both asserted that a specific variable was UNSET. True until someone's shell or a CI image disagrees. The doctest now exercises `parse_flag`, which is pure, and says why in one line. The unit test builds a name from `std::process::id()` so no other process can decide its outcome, and round-trips set/unset through it. * `tyalias@Error` was reported as an invalid disambiguator that "will fail to resolve". It is valid — rustdoc suggested that exact spelling in the error this fixed, and `cargo doc -D warnings` passes with it; the rendered page links to `type.Error.html` either way. Switched to `type@` anyway: it is equally correct, it is the spelling in the rustdoc book, and a future reader should not have to check which of the two is real.
`[`env`]` is ambiguous between the `env` module I had just added to the table and the `env!` macro, so the module table I added to fix a documentation gap turned the docs job red — the same failure class as the `[`Error`]` link two commits earlier, walked into while fixing its sibling. `mod@` resolves it. Caught by running the docs build rather than by CI, but only after the push, which is the honest order of events.
There was a problem hiding this comment.
🟡 Changes recommended
The new horus_sys::env unit test uses unsafe std::env::set_var/remove_var with an unsound safety justification under parallel test execution.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 15/15 changed files
- Comments generated: 1
- Review effort level: Lite
| /// `env_flag` is the thin half — it reads the variable and hands the value | ||
| /// to `parse_flag`. Asserting it on a name this process itself controls | ||
| /// keeps the check honest without depending on what the ambient | ||
| /// environment happens to hold. | ||
| #[test] | ||
| fn env_flag_delegates_to_parse_flag() { | ||
| // A name unique to this process, so a developer's shell or a CI image | ||
| // cannot decide the outcome. | ||
| let name = format!("HORUS_ENV_FLAG_SELFTEST_{}", std::process::id()); | ||
| assert_eq!(env_flag(&name), None, "an unset variable must not decide"); | ||
|
|
||
| // SAFETY: the name embeds this process's pid and is used by nothing | ||
| // else, so no other thread can be reading it concurrently. | ||
| unsafe { std::env::set_var(&name, "YES") }; | ||
| assert_eq!(env_flag(&name), Some(true)); | ||
| unsafe { std::env::set_var(&name, "off") }; | ||
| assert_eq!(env_flag(&name), Some(false)); | ||
| unsafe { std::env::set_var(&name, "banana") }; | ||
| assert_eq!(env_flag(&name), None); | ||
| unsafe { std::env::remove_var(&name) }; | ||
| } |
The stack caught the interaction between #205's internal-docs feature — which makes rustdoc resolve links inside the previously hidden modules — and #189's recv_failures doc, which linked crate::testing::MockTopic. That module is behind cfg(any(test, feature = "test-utils")), so the link is unresolved in a normal doc build. Plain text now. Verified: RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps --exclude horus_manager, and cargo doc -p horus_core --no-deps --features internal-docs, both pass.
, #213, #215) (#216) * fix(monitor): look for the plugin where horus install puts it Closes #184. `horus install horus-monitor` symlinks into `PluginRegistry::global_bin_dir()`, which is `paths::horus_dir()/bin` — the platform CONFIG dir, so `~/.config/horus/bin` on Linux. `which_monitor_binary` checked `~/.horus/bin` and nothing else, so the install succeeded and `horus monitor` then reported the plugin as not installed. The two halves sit about 800 lines apart in different files, and the comment on the path that was checked still called it "the plugin install location". The lookup now asks the registry for that directory rather than making a second hardcoded guess at it, and keeps `~/.horus/bin` behind it so an existing install goes on working. The candidate list is split out as `plugin_bin_dirs()` so the relationship can be asserted rather than assumed — the same shape as `run_python::python_cache_roots`, which exists because the package cache had this exact defect. Two tests: that the lookup searches the directory the installer writes, and that the legacy path is still searched. Verified non-vacuous: removing the registry lookup fails the first with exit 101. Three documentation pages carried this as a working recipe and now carry an `export PATH=...` workaround (softmata/horus-docs#16). Those lines should come out once this ships. * fix(control): make horus node restart actually restart the node `restart_node` sent PauseNode immediately followed by ResumeNode and printed "The scheduler will re-initialize this node on next tick". It did not. Both arms only call `NodeControls::set_paused` on a shared atomic; the code that re-runs `init()` is `reinit_pending_nodes`, which keys off `RegisteredNode::initialized`, and outside registration nothing in production ever set that back to false — the only two such assignments in the workspace are in `#[cfg(test)]` code. Four places in the runtime asserted the behaviour anyway: the clap help ("re-initialize without killing scheduler"), `restart_node`'s doc comment, an inline comment reading "Pause then resume triggers re-init in the scheduler", and `reinit_pending_nodes`'s own comment, "set by control commands". No control command set it. On a robot the operator restarts a wedged node, is told it worked, and moves on. Adds `ControlCommand::RestartNode { name }`, appended to the enum so a command serialised by an older CLI still decodes. The scheduler collects restart requests while draining the control topic — the loop holds a mutable borrow of `control_topic` and clearing the flag needs `self.nodes` — then clears `initialized` and lifts any operator pause, since a wedged node is often paused before being restarted and would otherwise be skipped by the very check that would have re-inited it. An unknown name logs rather than silently doing nothing. Tests drive the real control topic through `run_for`, not `tick_once`: the control topic is created in `run_with_filter`, so under `tick_once` it is None and no control command is polled at all. Worth knowing separately — operator control reaches a scheduler only while it is in `run()`. Verified non-vacuous: removing the RestartNode arm fails the test with exit 101 and `left: 1, right: 2`, which is precisely the pre-fix behaviour. Closes #184 is separate; this addresses the restart defect found by the bidirectional audit. * fix(safety): count consecutive deadline misses, not budget overruns `consecutive_misses` is the number the `max_deadline_misses` ceiling and the whole degradation ladder compare against, and nothing reset it from the deadline check. The reset lived in `NodeTimingState::record_tick`, reachable only through `check_budget`, which gave three wrong answers: - RT executor: `tick_node` calls `check_tick_budget` unconditionally and the deadline check afterwards, so a node woken 5 ms late that executes in microseconds cleared the run its own miss had just started. The counter was pinned at 1 forever. Neither `warn_after: 3` nor the ceiling could fire for the failure mode `check_deadline_from_release` was written to catch -- the one `primitives.rs` calls dominant. - Main loop: the call sits inside the budget-violation branch, so a clean tick never reaches it and the count is a lifetime total. - A node with `.deadline()` and no `.tick_budget()` skips the budget block entirely, so nothing cleared it on either path. `node_builder` derives a deadline from a budget and never the reverse, so this is an ordinary shape, and for it the per-node consecutive ceiling was a lifetime total wearing a safety threshold's name -- exactly what the per-node ceiling was introduced to end. A budget overrun and a deadline miss are different failures, so a tick that stayed inside its budget is no evidence its deadline was met. The reset moves to `SafetyMonitor::record_deadline_met`, called wherever the deadline check comes back clean on both dispatch paths. Also: a miss now clears `recovery_counter`. `record_successful_tick` means "did not panic", and both callers reach it on a tick already recorded as a miss in the same pass, so a node at RateReduced that missed EVERY deadline still accumulated `recovery_ticks` of "success" and was restored to the rate it had just proved it could not hold. The field's own comment already called itself a consecutive run. Two existing tests asserted the old contract by calling `check_budget` with an in-budget duration to mean "recovered"; they now say `record_deadline_met`, which is the thing they were reaching for. Each fix was reverted in turn to confirm the tests catch it: the in-budget run reads 1 instead of 10, the no-budget scheduler node reads 10 instead of 0, the RT node stays at 5 after meeting its deadline, and the never-recovering node is declared recovered at tick 8. * fix(safety): make the max_deadline_misses emergency stop reachable `max_deadline_misses` is documented as the per-node consecutive ceiling that stops the robot, defaults to 100, and could not fire at that value -- or at any value above 20 -- in any shipped build. `DegradationAction::Kill` sets `is_stopped`, and a stopped node is skipped forever on both dispatch paths. So the ladder's terminal rung, hardcoded at 20 consecutive misses, permanently retired the node before its count could climb any higher: `consecutive_misses` froze at 20 and the ceiling was unreachable. Nothing warned. `max_deadline_misses(n)` only clamps with `n.max(1)`, and the sole setter for the ladder is `#[cfg(test)]`, so no user build could raise the rung either. Two of the runtime's own defaults contradicted each other, and the unit tests only ever exercised the ceiling at 3 and 5 -- both below 20 -- which is why the interaction went unnoticed. A hardcoded constant must not silently cap a documented safety threshold, so the rung moves: `kill_after` is raised to the ceiling when the ceiling is higher. The graduated stages below it (warn at 3, half rate at 5, isolate at 10) are untouched, and an isolated node keeps ticking -- every tick on the RT executor, on a probe interval on the main loop -- so it can still reach the ceiling. Where the two now coincide the e-stop wins, because both paths record the miss, which evaluates the ceiling, before evaluating the ladder. Note this changes what a chronically missing node does under the default configuration: it is no longer killed at 20 while the robot runs on, it e-stops the robot at 100. That is what the knob, its default, the builder doc and five documentation pages all say it does. Both tests assert the ORDER of the two terminal events rather than the counter -- at this level nothing removes the node, so the count climbs regardless and the e-stop would fire on its own; what killed the ceiling is the `Kill` that lands first. Reverting to the hardcoded rung fails them at 20 for both the configured ceiling of 25 and the default of 100. * fix(run): apply [network] on every path that spawns a run child `safety.on_link_lost`, `import` and `deny_export` are safety controls. The 2026-07-30 audit found the whole `[network]` table parsed and then read by nothing, and `apply_network_config` was written to fix that -- then wired into ONE of the five paths that spawn a run child. That one is `execute_multiple_files`, reached only when `horus run` is handed more than one file. A bare `horus run`, a `horus.toml` run, a directory run and a `[workspace]` run all resolve to a single executable. On those the child environment came from `build_child_env()`, which sets PATH, LD_LIBRARY_PATH and PYTHONPATH and nothing else, so `NetConfig::default()` read an environment nobody had populated: `import` stayed `Auto`, `deny_export` stayed empty, and the documented comms-loss safe state never armed. The gap the two existing comments describe was still open on every invocation a real single-binary robot makes. `apply_network_config` becomes `network_env() -> Vec<(String, String)>`, which `build_child_env` now extends with -- so every single-executable path picks it up from the function they all already call, and no future path can quietly miss it. `execute_workspace` builds its `Command` by hand rather than through `build_child_env`, so it gets an explicit `apply_network_config` call. The operator still wins: `network_env` skips any key already present in the environment, so an explicit `HORUS_NET_*=...` beats the manifest on every path. That is asserted too. Reverting the `build_child_env` line gives `left: None` for HORUS_NET_ON_LINK_LOST -- the safety control simply absent. `horus launch` still performs no translation; that is a separate gap and the documentation says so. * fix(log): HORUS_LOG_FILE actually starts the file drain `start_log_file_drain` was called by no shipped code path. A tree-wide grep for the symbol returned four hits: the definition, its re-export, and two lines of one integration test that invoked it by hand. So `HORUS_LOG_FILE=1 horus run` wrote no `.horus/logs/horus.log`, and `HORUS_LOG_DIR`, `HORUS_LOG_MAX_SIZE` and `HORUS_LOG_MAX_FILES` were dead with it -- while the test that proves the drain works kept it green. The env guard lives inside the function precisely so a call site can be unconditional, and the function's own doc-comment calls the variable an opt-in, which is only true if something always calls it. `Scheduler::new` already does this kind of env-driven wiring twice, for HORUS_NET_ENABLED and HORUS_RECORD_SESSION, so the drain goes beside them. A process that does not set the variable pays one env lookup and spawns nothing. Once per process, via a `OnceLock`: a program may build several schedulers, and each drain thread tracks its own index into the same global ring, so N of them would write every entry N times into one file. This also keeps the promise for C++ and Python, which the doc-comment's "or calling this function directly" escape hatch could not: the symbol has no FFI export anywhere in the tree. Both bindings construct a core `Scheduler` (horus_cpp/src/scheduler_ffi.rs:26, horus_py/src/scheduler.rs:882), so both now get the drain. The test does not call the drain. It sets the variable, builds a `Scheduler` the way any program does, and asserts a log file appears; it is alone in its binary because it writes process-global environment. Reverting the wiring leaves the file absent and it fails. An operator who sets a documented variable to get a post-incident log and finds an empty disk is the exact failure that variable exists to prevent. * fix(topic): try_send and send_blocking move the staleness counter `messages_total` is not a metric. `SubscriptionFreshness::refresh` watches it as the "new data exists" signal behind `.subscribe_with_timeout()`, and its answer drives `StalePolicy::SafeState` and `StalePolicy::Stop`. Only `RingTopic::send` bumped it. `Topic::try_send` and `Topic::send_blocking` go straight to the ring, so a topic published entirely through either read as 0 messages and 0 Hz while carrying every message -- and could safe-state or halt a subscriber node whose data was arriving normally. That is exactly the failure the counter was introduced to fix, reintroduced for backpressure-aware publishers, and these are the two APIs the documentation points users to on critical topics. The runtime states the invariant in both places that consume the counter: "every publisher increments `messages_total` in the topic's SHM header on every send, so watching that counter is a true 'new data exists' signal" (scheduling/types.rs) and "incremented on every `send()` regardless of backend" (header.rs). Neither is true. Bumped once per public publish call, not inside `RingTopic::try_send`, which `send_lossy_retry` calls up to ~70 times for one message. `try_send` counts before the call, as `send()` does: an attempt count, and a publisher hammering a full ring is alive. `send_blocking` does NOT count a `NoBackpressure` refusal -- that is returned before the send is even attempted, so nothing reached the ring, and counting it would report a healthy flow on a topic delivering nothing. A `Timeout` did engage the ring, so it counts like `send()`'s dropped message does. Reverting both bumps reads 0 for a topic that carried ten messages. * fix(topic): tell the 17th participant why it has no comms A topic holds at most 16 participants. `register_role` builds a precise error for the one that does not fit -- it names the limit, the count, and what to do about it -- and all three call sites threw it away: `recv_uninitialized` and `read_latest` were `if ....is_err() { return None }`, `send_uninitialized` the same with `Err(msg)`. Those are the only `ensure_consumer`/`ensure_producer` call sites in the tree, so that message reached nothing, ever. Because the role stays `Unregistered`, every later `recv()` re-enters the same cold path and fails identically: `None` forever, from a subscriber indistinguishable from an idle one. Nothing anywhere defends the silence, and the identical 16-slot wall one layer down is handled the opposite way on purpose -- rate-limited `tracing::warn!` under "COMM-H1: never let 'all 16 slots live' become a SILENT no-comms endpoint (worse than the pre-fix loud panic)". The rate limiter for it sits in the same file, unused by this path. Meanwhile header.rs defends a separate design decision by claiming the residual cost "surfaces as the existing 'No available participant slots' error. Loud, and it says what to do" -- a guarantee the code did not provide. All three sites now report through that same limiter, so a hot loop warns once a minute per topic rather than once ever. The limiter is also keyed by reason now, not just topic. A topic can hit two different walls -- its FanoutShm endpoint bitmap and the participant table -- with different causes and different fixes, and one bucket let whichever fired first silence the other for a minute. That is the masking the per-topic keying was introduced to prevent, one level down. The test fills the table from 20 threads (the participant-count leak is what lets one process do that), asserts it really is full before asserting anything about diagnosis, and captures `tracing` events through a hand-rolled `Subscriber` so it needs no new dependency. Reverting the fix prints nothing at all: "Warnings seen: []". * fix(python): a refusal for want of backpressure is not a timeout `send_blocking_pod` ended in `.is_ok()`, discarding which `SendBlockingError` came back, and both arms then reported `HorusTimeoutError`. So the one error horus_core writes specifically to be un-retryable arrived in Python as the one error callers retry. The core is explicit about the distinction, twice: "Refusing is deliberate: reporting success for a delivery nobody guaranteed is worse on an e-stop path than failing loudly. It does not clear on retry -- it lasts as long as the topology that caused it." The variant carries a four-line message naming the fix. The binding replaced it with the exception whose own doc-comment means the opposite -- "the buffer stayed full for the whole timeout" -- and its private helper asserted, falsely, that a `false` return meant the ring stayed full. This is easy to reach and permanent: `detect_optimal_backend` resolves any POD topic with a second subscriber to `PodShm`, so the moment a `horus topic echo` or a monitor attaches to a POD e-stop topic, every `send_blocking` starts raising forever -- as a "timeout", on the path the method is documented for. The variant is now threaded out of both arms and mapped to a new `horus.HorusBackpressureError` carrying the core's own remediation text. `HorusTimeoutError` keeps `Timeout` alone. Also exposes `Topic.provides_backpressure`, which the core's doc-comment instructs callers to assert on after the first send and which Python had no way to ask. It is answered through a new `horus_core::communication::topic::backend_provides_backpressure`, keyed by backend NAME because a name is all a binding can see -- and `provides_backpressure()` is now defined in terms of it too, so the two cannot disagree. A test walks every `BackendMode` through an exhaustive match, so a new backend stops compiling rather than silently defaulting to "no backpressure". Reverting the mapping fails the test at "must raise HorusBackpressureError". * fix(python): make subs= type auto-detection actually run `_setup_topics` documents a three-step type resolution whose middle step, "auto-detect from SHM header", was unreachable code. It did: probe = Topic(topic_name, 64) type_name = getattr(probe, 'type_name', None) Both halves were dead. `PyTopic` exposes no `type_name` — its getters are `name`, `is_network_topic`, `endpoint`, `backend_type` — so the value was always `None`. And the probe could not have worked even with that attribute: `Topic(name, capacity)` builds a `Topic<GenericMessage>`, and attaching 4364-byte GenericMessage to an existing typed ring is refused by the size check in `RingTopic::new`, so it always raised into a bare `except Exception`. Step 2 therefore never ran for any topic that has ever existed, and the ~70-entry `_TYPE_NAME_MAP` built for it had exactly one reader, inside the unreachable branch. The size check is correct and stays: its comment records that a name-based exemption once let a `Topic<GenericMessage>` join a `Topic<CmdVel>` ring and "write megabytes past the end of the mapping". The question just cannot be answered by attaching -- opening requires choosing a type, which is what the caller is trying to find out. So: `horus_core::communication::peek_topic_type_name(name)` reads the header without attaching, exposed to Python as `_horus.peek_topic_type`. `_detect_topic_type` calls it and maps through `_TYPE_NAME_MAP`, which now has a reachable reader. Second half: when the ring exists and holds a type this build cannot name, `_setup_topics` raises with the topic, the ring's type and the concrete fix (`subs=[Sub('scan', LaserScan)]`) instead of falling through to a generic topic and letting the runtime's byte-count RuntimeError out -- which named neither `subs=`, nor the type the ring holds, nor what to do. Auto-detection was only ever scoped to subscriptions; the `pubs=` case is unchanged, since a publisher creates the ring and there is nothing to read. Tests both ends. The Rust test asserts the old probe's refusal still happens (it must) and that the peek answers anyway; reverting the peek gives `left: None`, which is what the old probe returned for every topic. The Python test drives a real `Node` with a bare `subs=` string against a typed ring; restoring the old probe fails it. * fix(cli): three flags that did not do what they said **`post_test` skipped teardown exactly when teardown matters.** It sat after an early `?` in the `Commands::Test` arm, so a failing `horus test` never ran it, while `post_run` and `post_build` go through `run_teardown_hooks` and run on both paths deliberately: "Skipping it when the run failed would skip it in exactly the case where the hardware was most likely left live." The hooks contract file calls `test` "the only phase with teardown" and then pinned only the symmetry of the KEY NAMES, so `post_test` kept success-only semantics while the other two were given real teardown semantics. A `post_test = ["release-bench"]` was skipped precisely when the tests aborted mid-run. The new guard reads the dispatch in main.rs rather than the table: every `post_` phase must go through `run_teardown_hooks` and none through the plain `run_hooks`. Reverting the arm fails it on both halves. **`horus doctor` recommended a variable that does nothing.** It printed "use --net or HORUS_NET=1 to enable" and reported networking from `HORUS_NET`, which nothing else in the tree reads. Following the advice changed nothing except that `doctor` then printed "horus_net: enabled", confirming a state that did not exist -- worse than inert. main.rs already recorded this exact confusion when it fixed `--net`; the diagnostic handing users the broken half of the same remedy was not fixed with it. `doctor` now asks the question the BUILD asks, through a shared `cargo_gen::net_capability_requested`, and names only routes that work. `HORUS_NET` is retired: no reader remains. **`horus install --plugin -t <workspace>` threw the name away.** The call site collapsed it to `let local = target.is_some()`, and what was left called `detect_or_select_workspace` -- which resolves the CURRENT DIRECTORY's workspace, or prints a menu and blocks on stdin. Both are things `horus install` documents it never does, and with no tty the read returns empty and the command dies with "Invalid selection". The package half of the same flag already resolved the name correctly; the bool was a vestige of a `--local` flag that no longer exists, still named in the doc-comment. The ablation shows how completely the name vanished: with the bug the install proceeds past the target entirely and fails with an unrelated "no package registry is configured". * test(log): fix a race in the drain wiring test that CI caught `log_drain_loop` snapshots `GLOBAL_LOG_BUFFER.write_idx()` when its thread starts and only writes entries newer than that snapshot. The test pushed its entries in one burst immediately after constructing the scheduler, so on a loaded machine the drain thread could reach the snapshot after the burst -- leaving nothing newer to write, and the test waiting out its full 10-second window for a drain that was working correctly. Pushing every polling round instead guarantees entries after the snapshot whenever the thread got going. Still non-vacuous: reverting the wiring in `Scheduler::new` fails it exactly as before. * fix(topic): give a participant registration back when its handle drops Nothing inside a living process ever returned one. The only `decrement_to_floor` calls on `publisher_count`/`subscriber_count` are in `reap_dead_participants_now`, which wants an expired lease AND a dead pid, and in `register_role`'s last-resort pass, which only runs once all 16 slots are occupied. `Drop for RingTopic` released spill keep-alives and FanoutShm endpoint bits and left the participant table untouched. So within one process the counts only grew -- and `detect_optimal_backend` reads them. A transient subscriber on a fresh thread (a worker pool, a scoped task, a one-shot diagnostic read) permanently converted a command topic from backpressured `SpscShm` to overwrite-oldest `PodShm`, with nothing alive to justify it and no path back for the life of the segment. It silently voided `try_send`'s full-ring `Err` and `send_blocking`'s backpressure on an e-stop path. The table also filled toward its 16-slot limit with registrations belonging to nothing. `horus_core/tests/participant_count_leak.rs` has documented this as OPEN, with an `#[ignore]`d test, since the naive fix was built and rejected. # Why the naive fix was unsafe, and what makes this one safe Releasing against the recorded slot INDEX is wrong: a slot is not owned for a handle's lifetime. `register_role`'s pass 3 reclaims slots whose thread has ended, so a stale index names somebody else's registration by the time it is given back. `ParticipantEntry` now carries a `generation`, bumped on every claim in `finish_claim` and recorded by the handle beside the slot, so a release can tell its own registration from a later one. A second hazard the earlier attempt did not name: an entry is keyed on (pid, thread), NOT on the handle, so several `Topic`s on one thread deliberately share one slot and one increment of `subscriber_count`. A (slot, generation) match alone would let the first to drop clear an entry the others are still using. The entry therefore also carries per-role handle counts -- per role, because one thread publishing and subscribing to the same topic is the supported `role = Both` fast path, and a single count could not express "the last subscriber left while a publisher stays". Every check refuses rather than guesses. Under-counting is the dangerous direction: `nothing_is_draining` lets a producer retire unread slots once `sub_count()` reads 0, so a lost registration becomes silent message loss on a subscriber that is still reading. Both new fields came out of the entry's existing padding, so it is still 24 bytes and the header is still 640; the static assertions on both are unchanged. # The downgrade this exposes, and why it is not taken With the counts finally able to fall, a two-subscriber broadcast topic whose second subscriber exits would migrate BACK to `SpscShm` -- and a backwards migration mid-stream drops whatever is already in the wider ring, because the migrator's drain has nothing to observe and always succeeds. Measured directly: with the downgrade allowed, the SURVIVING subscriber of two received 198 of 200 messages and ended up back on `SpscShm`, and the two multithread non-POD broadcast tests failed the same way the earlier attempt made them fail. So `detect_optimal_backend` now never narrows a running topic. Widening on a join stays immediate; narrowing has to wait for a drain that is real, and until then the wider path is both the safe answer and the one the runtime already took when the counts could not fall at all. Both directions are pinned by tests. The counts themselves are correct now regardless, which is what the participant table, `horus topic info` and the 16-slot ceiling all depend on. Verified: the previously-`#[ignore]`d test passes and is un-ignored; `sub_count` climbs to [1, 2, 3] with the release removed. The two multithread broadcast tests pass 10/10 at base timings (0.12-0.18s), and the full lib suite is green across six runs. * fix(topic): stop advertising a migration drain that is not performed `BackendMigrator` promised "Drain logic to ensure no in-flight message loss", and `perform_migration` said it "Drains in-flight operations before switching the backend mode. If the first drain attempt times out, retries once before failing." None of that is true. `drain_in_flight`'s own comment says why: "Nothing in `TopicHeader` counts in-flight operations -- there is no such field -- so this function has never had anything to observe. A wedged producer and an idle topic run the same number of iterations and both return `true`." It spins, yields three times, and returns `true` on every exit. So the retry and the `MigrationResult::Failed` it returned were unreachable code guarding a return value that no longer exists -- and callers wrote handlers for a case that could not occur, in mod.rs and in the test suite. The variant is gone with them. `drain_in_flight` becomes `settle_before_switch` and returns nothing, which is what it always did. The bound stays: that part was legitimate, and removing it made `send_blocking` block measurably longer under load. The guarantee itself is NOT restored here. A real one needs an in-flight counter incremented and decremented on the send and recv hot paths of a real-time framework -- a deliberate design decision with a measurable cost, which the existing comment argues should not be reintroduced by leaving a deadline around that only looks like it does the job. The doc-comments now say plainly that a message in flight across a switch can be lost; the topic's own tests already pre-warm backends "to avoid migration losses". This is also what the never-narrow rule added alongside the participant release depends on: narrowing a running topic is unsafe precisely because this drain is not real. Two tests pin what is actually true: the settle stays bounded, and an uncontended migration always completes. * fix(rt): a safing callback that panics must not cancel the safety action Two halves of one asymmetry: the executors got the isolation half of the main loop's fault handling and not the escalation half. **A panicking `shutdown()` cancelled the emergency stop that called it.** `BudgetPolicy::EmergencyStop` ran `let _ = node.node.shutdown()` bare, and BEFORE latching. `shutdown()` is user code on a node that is already failing, so an unwind out of it escaped past `estop.trigger`, past `running.store(false)` and into the loop's outer `catch_unwind`, whose only action is to stop the node. `get_state()` kept reporting Normal, no blackbox EmergencyStop was written, `PENDING_LOCAL_ESTOP` stayed empty so horus_net broadcast nothing, and the other RT nodes kept ticking -- exactly the outcome the comment three lines below it was written to prevent. The docs promise this e-stop unconditionally. Both `shutdown()` calls (Enforce and EmergencyStop) now go through `guard_fault_callback`, which primitives.rs already says "any future `&mut dyn Node` callback reached from an executor belongs behind". The EmergencyStop arm also latches and signals the thread BEFORE calling into the node, so the safety action no longer depends on user code returning. **A critical RT node that could not reach a safe state did not halt the robot.** `note_safing_failure` states the rule as a property of the system -- "a node that cannot safe itself is a safety failure -- escalate to a system emergency stop for critical nodes" -- and `is_critical_node` had exactly one caller, on the main loop. So the identical event halted the robot on a BestEffort node and merely stopped an RT one. Every RT node under a scheduler `.watchdog()` is a critical node, so the class most likely to be critical was the one the escalation omitted. `primitives::note_safing_failure` is the executor twin, wired into the Isolate and Kill rungs of `apply_degradation_action` and into the RT executor's `Miss::SafeMode` dispatch -- which called `enter_safe_state()` entirely bare, so a panic there only ever stopped the node. Ablations: with `shutdown()` bare and ahead of the latch the panic escapes the test itself; with isolation kept but escalation removed the panic is caught and no e-stop latches, which is the shipped behaviour. * fix: a watchdog is never tightened, and topic metrics count real traffic **`Scheduler::watchdog()` silently tightened the window.** The config stores whole milliseconds and `as_millis()` truncates, so `.watchdog(1500_u64.us())` stored 1 ms — and the graduated ladder latches a system-wide emergency stop past 3x the window, so it fired at 3 ms instead of the 4.5 ms asked for, with no diagnostic. The builder states the rule immediately above the line that broke it: "Round up to the 1ms floor the storage can represent rather than truncating ... a slightly-looser watchdog is a safety feature, and no watchdog is not ... silently changing a safety timeout is its own problem — the operator needs to know the value actually in force." That held for the sub-1 ms case and was inverted for every other one. Tightening is the direction that halts a healthy robot. Ceiling division now, across the whole range, and the warning fires for any value that had to change — not just the sub-millisecond one. An explicit `Duration::ZERO` is still an honoured disable. **Three of the four counters on the public `TopicMetrics` were dead.** `messages_sent` and `messages_received` moved only inside `send_with_content_logging` / `recv_with_content_logging` — both `#[cold]`, both reached only while the `horus monitor` TUI has set a topic's verbose flag. So accessors documented "Messages sent through this topic" read 0 in every normal run, however much traffic the topic carried. `MockTopic` — the stand-in users write their tests against — maintained both faithfully all along, with its own tests asserting on them. A double that reports what the real transport does not is the one defect a double cannot have. Both are counted on the ordinary paths now, beside the existing `messages_total` bump. That comment prices the SHM counter at ~28 ns, but this is a process-local uncontended relaxed increment on a counter only this handle writes; the cold paths no longer double-count. `recv_failures` had no increment anywhere in the transport and no honest meaning there — a `recv()` that returns `None` is an empty ring, not a failure. The dead field is gone from the transport-side struct, the snapshot reports 0 explicitly, and the accessor now says so and points at `missed_count()` for real consumer-side loss. It stays on `TopicMetrics` because `MockTopic` genuinely populates it from its fault injection, which is the whole point of the double. Ablations: restoring `as_millis()` reads 1 ms where 2 ms was asked for; removing the ordinary-path counts reads 0 for a topic that carried five messages. * fix(launch): --status told the truth about nothing, and namespace: lied twice **`horus launch --status` reported every node as running, forever.** `show_launch_status` colours a STATUS column with "running", "crashed" and "stopped" arms, and `LaunchProcessEntry::status` was assigned exactly once in production code -- `"running"` at construction -- and never mutated. Two of the three arms could not be reached. The monitor loop already knows: the same iteration that drops a dead node from the supervised vector logs `LaunchEvent::NodeCrashed`. Two halves of one workflow disagreed -- the JSONL event log knew, the session manifest the CLI reads did not. Worse, the refresh that followed re-associated by POSITION. Removing a dead node shifts every later index, so a surviving node's PID was written into the dead node's row: an operator asking whether the perception node crashed was told "running", beside a PID belonging to a different, healthy process. `refresh_session_manifest` is keyed by name, records `crashed` for a non-zero exit and `stopped` for a clean one, and puts a respawned node back to `running`. It is extracted so the property is testable without driving the supervisor loop. **A per-node `namespace:` did nothing, and said so nowhere.** It was parsed, composed into the name `--dry-run` PRINTS, and exported as `HORUS_NODE_NAMESPACE` -- which had exactly one occurrence in the whole repository, this write, and no reader in Rust, Python or C++. Its doc-comment promised "Namespace prefix for topics", which horus_core cannot do: there is no topic prefix or remap mechanism. The sibling `priority` export at least carries an explicit "this is currently a no-op" note. Giving it real meaning is a horus_core design decision (topic remapping), not a launcher change, and it cannot simply be "restored": composing it into the child's `HORUS_NAMESPACE` would put the node in a different shared-memory tree from its siblings and break the launch it belongs to. horus_sys states that granularity as "per robot" -- the file-level key, not this one. So the runtime stops lying while the semantics stay undecided: the export is gone, launching warns and says what to do instead, `--dry-run` shows the name the process is actually given rather than a composed identity nothing receives, and the field's doc-comment says it is inert and why. Ablation: with the positional refresh restored, a crashed node reads "running" and all three tests fail. * fix(python): an untyped topic accepts a message object, and the quick start runs **`Topic(X).send(instance)` and `node.send(name, instance)` disagreed.** `Topic("name")` carries `GenericMessage`, and `generic_message_from_py` ran `pythonize::depythonize` on the Python object — which cannot read an opaque `#[pyclass]`, i.e. every HORUS message class and every msggen-generated class. So it raised `TypeError: Failed to convert Python object: unsupported type CmdVel`, while `Node.send` worked, because it catches that refusal and retries the object as a dict. Two spellings of one operation disagreeing is the defect, so the conversion moves to where all three publish paths share it. `message_object_as_dict` is the Rust twin of `horus._message_to_dict`, used the same way: only after a direct conversion has been refused, so it needs no guess about which types are messages. An object with nothing to extract still raises. This is what `build_messages()` prints on success and what `horus/msggen/__init__.py` promises, so both now describe something that runs. Note serde derives on the generated Rust struct would NOT have helped: the failure is on the Python side of the boundary. **`build_messages()` printed two lines that both failed.** `from horus import <Name>` raises ImportError — a generated class is registered on the extension module and `horus/__init__.py` re-exports a fixed hard-coded list — so it prints `from horus._horus import <Name>` now, plus a send line. **msggen promised POD zero-copy it does not provide.** A generated class resolves through the generic backend, so it travels the MessagePack path. It works; it is not built-in-type latency, and the docstring says so. **`help(horus)`'s quick start raised on its second line.** It published an `Image` over a bare-string topic and called it zero-copy: a string argument builds a `GenericMessage` topic, so the send raised and the `recv()` / `to_numpy()` lines never ran. It uses the typed spelling now, `Topic(Image, endpoint=...)`, which the documentation already showed. The Common Mistakes entry claiming "`Topic(int)` or `Topic(42)` raises TypeError" was also half wrong: `int.__name__` extracts cleanly, so `Topic(int)` silently opens a generic topic named "int". It says which is which, and how to spell a pool-backed topic. These docstrings are not incidental — `python_api_docs.rs` exists to pin them, on the stated grounds that "its docstrings are the only API reference a user gets at the prompt". Ablation: with the fallback declining, the two message-object tests fail with the exact shipped error. * fix(drivers): honour `simulated`, and name an exec driver after its entry **`simulated = true` did nothing, and HORUS writes it itself.** `horus add <name> --driver --source sim` — and `--source simulated`, and `--source sim3d` — all emit `simulated = true` into horus.toml. The loader read only `sim`, so an entry the CLI generated constructed the REAL driver under `horus run --sim`: the exact failure `--sim` exists to prevent. `horus check` stayed quiet too, because `simulated` is a known field rather than an unknown one, and the loader's RESERVED list kept it away from the driver as well. The field's own doc-comment declares a purpose. It is an alias of `sim` now, extracted as `asks_for_simulation` so its truth table can be tested without a config file or the process-global environment — the same reason `sim_mode_enabled` is a free function. An explicit `sim` still wins. **An exec driver was named after its binary, not its manifest entry.** `ExecDriver::from_config` — the function a horus.toml exec entry actually takes — discarded the `[hardware.<name>]` key its caller already holds (and uses one line later) and derived the name from the binary's file stem. So two `[hardware]` entries pointing at one binary were indistinguishable in "Exec driver '{}' launched" while the scheduler knew them by their distinct keys. It also injected `HORUS_PARAM_*` and no name, so `HORUS_DRIVER_NAME` reached nothing on the live path: the only function that set it is `from_params`, whose sole caller is a unit test. The environment-surface contract asserts, in prose, that it "is injected into every user-written exec driver" — `docs_contract.rs:774` — which is now true and can stay. `load_exec_prefix` asserted the defect as the contract, with the comment "ExecDriver node name is derived from binary name". Updated: the driver is named after the entry, matching the name the scheduler uses. Ablations: without the alias, `simulated = true` reads as "not simulated"; with the file stem restored, the driver reads "rplidar_driver" where the operator wrote `front_lidar`. * fix(cpp): a param value must not come back silently clipped `horus_params_get_string` returned the number of bytes WRITTEN, so a caller had no way to tell a complete value from a truncated one and no way to size a buffer around it: `Params::get_string` handed back an ordinary `std::optional<std::string>` with a clipped value in it, indistinguishable from a whole one. It also returned that positive count for a NULL buffer — the write is guarded, the return was not — and split UTF-8 sequences. The contract this should have had is in the same file. `horus_topic_last_error` returns -1 for a null buffer or a zero length, walks the copy back to a char boundary because "half a UTF-8 sequence is not something to hand a caller that may re-encode it", and has a test asserting both. Its inputs are strictly LESS arbitrary than a param value: a topic name is length-checked before it gets there, while a param holds paths, URLs and calibration blobs. Nothing in `params.hpp`, `horus_c.h` or the shim states a length limit, so 1024 was an artefact of a stack buffer rather than a contract — which is why Rust, Python and C++ disagreed about what `get("key")` returned for one file. Only C++ clipped. The shim now returns the value's FULL length, so `>= buf_len` means "this was truncated, and here is the buffer size that holds it", and `Params::get_string` re-calls into a `std::string` sized from it. The header documents the convention. One layer up, this class's own constructor refuses to construct on a bad file precisely because "`get<T>(key, default)` gives the caller no way to tell those apart from the values in the file" — a silently clipped string was that same defect, delivered through that same `get<T>`. Ablation: with the old shim, a 2000-byte value reports 15, a null buffer returns 5 instead of -1, and the truncated length is wrong. * fix(install): a prefix install must export the variable it depends on `HORUS_PREFIX` is the whole interface to a relocated install. Both readers take it from the environment of the running `horus` process and from nowhere else — `version.rs` says so outright: "there is no second name to read, and no way to discover a prefix install from outside its own tree". `run_rust`'s cache roots and the version gate's state root are both derived from it. The installer never put it there. The rc file got a PATH line only, `horus env --init` writes proxy functions and no exports, and the prefix epilogue told the user to export `PATH` and `HORUS_SOURCE`. So for every prefix user the cache root fell back to `~/.horus/cache`, was deduped away as the legacy root — meaning the prefix cache was never searched — and the state root was `~/.horus`, holding none of that install's files. Which means the Rust half of that change landed only for a user who guessed the variable, and the two epilogue statements it was supposed to supersede were still true. `version.rs` records the intent it was meant to serve: the gate "was silently off for exactly the installs that most need it, and install.sh had to print a warning saying so". So install.sh writes `export HORUS_PREFIX` beside the PATH line — `set -gx` in config.fish, because fish has no `export` builtin and a POSIX line stops that file loading, the same trap this script already documents for PATH. With that done, the `export HORUS_SOURCE` instruction and the "version gate is inactive for a prefix install" warning are both false and go. The uninstall hint stays: nothing outside the prefix records where it was. Also: `generate_messages_from_yaml`'s docstring sat AFTER its `_HAS_YAML` raise, which made it a no-op expression rather than a docstring — `__doc__` was None and `help()` showed none of the YAML schema it documents. Moved above the guard, and it now also warns that `horus_py/src/custom_messages` is generator-owned: `build_messages()` rewrites that `mod.rs` to name only registered messages, so a file the YAML emitter drops there is declared by no `mod`, never compiled, and left behind by every later build. Ablation: with the rc line removed, a prefix install writes only `export PATH=...` and both new tests fail. * merge: drop the duplicated monitor lookup, keep main's #188 landed the same #184 fix while this branch carried its own, and the merge resolution kept this branch's copy. Main's is the one to keep: it calls `PluginRegistry::global_bin_dir()` directly rather than through a `plugin_bin_dirs()` helper that also searched `~/.horus/bin`. That fallback was searching a path nothing writes. Every `.horus/bin` writer in the tree is PROJECT-relative — `project_dir.join(".horus/bin")` in lockfile.rs, `PathBuf::from(".horus/bin")` in registry/install.rs. The HOME-relative `~/.horus` holds installer state, never plugin binaries, which is exactly what #188 established. So the helper and the test asserting the legacy path stayed searched both go. What did not land with #188 is a guard on the property, so the drift that caused #184 — two hardcoded copies of one path, 800 lines apart in two files — is still unpinned. `monitor_plugin_lookup_contract` reads the body of `which_monitor_binary` and asserts it resolves through the accessor and builds no home-relative plugin path of its own. `which_monitor_binary` is in the binary crate, so an integration test cannot call it; reading it catches the one thing that actually went wrong. * Initial plan * Harden CI shared-memory cleanup Co-authored-by: starswallow <237697722+starswallow@users.noreply.github.com> * test(cpp): cover the get_string re-call, which nothing exercised The Rust shim's contract is tested in c_api.rs, but the C++ wrapper's re-fetch — the branch that runs when a value does not fit the 1024-byte stack buffer — had no coverage on either side, and that is where an off-by-one lives. Sizes either side of the buffer and exactly on it (1023, 1024, 1025, 4096), plus a value made of three-byte characters so a naive cut would land mid-sequence. Verified by linking the real library and running it, not only by compiling: with the wrapper reverted to its single-call form the binary aborts on `*got == value`, because it builds a std::string from a length the buffer never held. * test(topic): stress the participant release under concurrent churn The release path guards on (slot, generation) and per-role handle counts, and those are read-modify-write sequences on shared memory that several threads reach at once. Reading the code cannot rule out an interleaving that double-decrements or strands a slot, and both directions are bad: over-counting pins a command topic on a lossy broadcast backend, while under-counting lets `nothing_is_draining` retire unread slots beneath a subscriber that is still reading. Eight threads churn a subscriber and a publisher each, forty rounds, against one topic — so it exercises the shared-entry case as well as the generation guard. When they are joined nothing is alive, so sub_count must be 0 and pub_count must be exactly the one handle this thread still holds. 6/6 runs. The sibling-handle direction is already pinned by `one_handle_leaving_does_not_deregister_its_siblings_on_the_same_thread`: releasing on the first handle instead of the last fails it with left: 0, right: 1 — the registration taken out from under a live reader. * fix(topic): count a handle taking a role its sibling already holds An under-count I introduced with the participant release, found by an adversarial pass over that commit and confirmed by the test added here. A participant entry is keyed on (pid, thread), and its ROLE BITS are shared by every handle on that thread. So "the entry already has the producer bit" says nothing about whether the CALLING handle holds it. The per-role handle count was incremented only when the bit was newly set on the entry, or when the handle had no slot yet: if old_role & role_bit == 0 || new_handle { counter.fetch_add(1) } Two handles on one thread that each publish and subscribe therefore counted 2 and 1 instead of 2 and 2: the second handle to take an already-set role was invisible, and the first Drop handed that role back while the other handle was still using it. That is the under-count direction, which is the dangerous one and the whole reason the release path refuses rather than guesses everywhere else: `nothing_is_draining` lets a producer retire unread slots once `sub_count()` reads 0, so a lost subscriber registration becomes silent message loss on a subscriber that is still reading. The new test shows exactly that — sub_count 0 while a live handle is mid-read. The condition was unnecessary as well as wrong: `ensure_role` returns early when the calling handle already holds the role, so reaching `register_role` means the handle is newly taking it and the increment is unconditional. That makes the `new_handle` parameter dead, so it is gone rather than left threaded through three signatures. * test(monitor): pin the opt-in on the project-local plugin lookup `monitor_plugin_lookup_contract` asserts two things: that the lookup resolves through `PluginRegistry::global_bin_dir()`, and that it builds no HOME-relative plugin path. Neither looks at the project-local branch, so deleting the `HORUS_ALLOW_LOCAL_PLUGINS` gate leaves both green. That gate is the thing standing between a cloned repository and arbitrary code execution. `.horus/bin/horus-monitor` is CWD-relative and was once checked unconditionally and BEFORE `PATH`, so `cd`-ing into a repo that ships that file and running `horus monitor` executed it directly — no trust gate, no signature check, none of the sandbox `plugins::executor` applies — and it shadowed a properly installed binary. Asserted now, and skipped cleanly if the local branch is ever removed outright, since that also satisfies the property. Ablation: with the gate dropped and the local lookup kept, the two existing tests still pass and this one fails. * fix(tests): stop leaking a shared-memory namespace per isolated child Two test files spawn a child process under a private `HORUS_NAMESPACE` — `errbuf_<pid>_<test>` and `hlogbench_<pid>_<test>` — to keep a process-wide log ring away from concurrent writers. The child materialises `/dev/shm/horus_<ns>/`, and nothing unlinks a namespace when a process exits, so every isolated child left a directory behind permanently. The names carry a PID, so they never collide and never get reused: they accumulate without bound across every run, local and CI alike. Measured on this machine: 3 runs of one of those files leak 9 directories. `/dev/shm` was at 5.0G / 88 entries when I looked, and a full `/dev/shm` is the documented cause of the rotating, unattributable test failures this workspace shows — the ones that pass in isolation and fail in a full run, teaching people to rerun instead of read. Another session independently hit the same thing and recorded it in #192's commit message. `cleanup_stale_shm()` does not cover this: it clears the topics and nodes directories of the DEFAULT namespace, which is not where a private one lives. Removed in the parent, because the child is the process that dies. Ablation: with the removal dropped and everything else identical, 3 runs leak 9 directories; with it, 0. The parent knew the namespace but had no way to name its directory: `shm_base_dir()` derives from `shm_namespace()`, which is fixed at startup and can only ever be the caller's own. Rather than restate the platform mapping in a test, `shm_base_dir_for(namespace)` names any namespace's directory, and `shm_base_dir()` is now defined in terms of it — which also removes the second copy of the `/dev/shm` vs `/tmp` vs `%TEMP%` mapping that sat beside `shm_parent_dir()`'s. * fix(topic): pool-backed try_send moves the staleness counter too An incomplete fix of my own, found by an adversarial pass over it. Wiring `messages_total` into the typed `Topic::try_send` missed `Image`, `PointCloud` and `DepthImage`: those have their own `try_send` impls that reach `ring.try_send` directly rather than through the typed one, so they never touched the counter. They are the worst three to have missed. `messages_total` is what `SubscriptionFreshness` watches behind `.subscribe_with_timeout()`, and that drives `StalePolicy::SafeState` and `Stop` — so a camera, lidar or depth publisher using `try_send` read as 0 messages and 0 Hz while delivering every frame, and could safe-state or halt a subscriber whose data was arriving normally. High-bandwidth sensor topics are exactly the ones a freshness watchdog is put on. The non-try path was already correct: `ring.send` bumps internally. Ablation: with the three bumps removed the counter reads 0 for five frames sent. * fix(control): horus node restart must reach a node on an executor Another incomplete fix of my own, and the same defect class the branch exists to close: a fix that cannot reach the nodes it is for. `process_control_commands` handled RestartNode by searching `Scheduler::nodes`. The class partition sets `self.nodes = groups.main_nodes` before the tick loop, so that vector holds only the main-thread group. Every RT, Compute, Event and AsyncIo node was therefore reported as unknown — and worse than unknown: the handler un-paused it first, so `horus node restart` on an RT node silently resumed it without re-initialising, which is a different operation from the one the operator asked for. Nodes with a `.rate()` are exactly the ones an operator restarts, and `.rate()` is what makes a node RT. `init()` needs `&mut dyn Node`, which the executor owns, so this crosses the same way the watchdog ladder's safing request does: the main thread raises a flag in the shared `NodeControlMap` — registered for all five execution groups — and the owning executor consumes it once per raise. `honor_restart_request` sits beside `honor_safe_state_request` and is called from all four executors. In `tick_node`, at the top and before the failure-policy gate: a suppressed node must still be restartable, since not running well is the reason to restart one. A restart also lifts an operator pause, because a wedged node is usually paused first and leaving it paused makes the restart a no-op at the next tick gate. Ablation: without the executor honouring it, init() is never re-run (left: 0, right: 1). * test: three of this branch's own guards did not guard An adversarial pass over the tests added here, not over the fixes. **The 17th-participant test had gone permanently vacuous.** It filled the table by spawning threads and letting them exit, relying on the very leak that a later commit on this same branch fixed — so once registrations were returned on Drop the table stopped filling and the test skipped itself: "the participant table did not fill (1/16)". The anti-vacuity guard did its job by reporting that rather than passing quietly, but nobody read the line. It now holds HOLDERS live subscribers open on a channel until the assertions are done, so the table is genuinely full. Reverting the fix gives "Warnings seen: []" again. **The monitor-lookup guard matched a comment.** It asserted the body of `which_monitor_binary` contains "global_bin_dir()" — and the function's own comment explains what `global_bin_dir()` is, so the assertion was satisfied by the prose describing the fix rather than by the call performing it. It strips comment lines now. Ablation: replacing the call with a hardcoded `~/.horus/bin` while leaving the comment intact fails both guards, where before it would have passed. **The Python backpressure test asserted a constructor.** It called `no_backpressure_err` directly, which is the easy half; the defect was that `send_blocking_pod` ended in `.is_ok()` and discarded WHICH error came back, so the SEND PATH reported every refusal as a timeout. The new test drives a real broadcast-backed topic — two subscribers on their own threads, because an entry is keyed on (pid, thread) — and reverting the mapping reproduces the shipped symptom precisely: `HorusTimeoutError ... after 21.102µs (deadline: 50ms)`, the zero-second "timeout" the documentation used to tell users to identify with a stopwatch. * test: anchor the safing-order guard on code, not on a comment `ca021a0e` picked this file up from the working tree mid-edit, before the anchor was fixed, so the compute half has been failing on the branch: it sliced the coordinator loop between `if ready_indices.is_empty() {` and the string "Begin recording" — which is a COMMENT, and this guard strips comments before matching precisely so it cannot be satisfied by prose. The anchor was therefore never found and the test panicked with its own "anchor is gone; this guard would cover nothing" message. That message is the anti-vacuity check doing its job: a guard whose anchor has moved covers nothing, and saying so beats passing. Re-anchored on `let now = Instant::now();`, which is code. Also reflows a `panic!` the formatter wanted on its own line. * fix: three more defects in this branch's own work **`ParticipantEntry::clear` published `active = 0` first.** That store is what makes a slot claimable — `claim_free_slot` scans for `active == 0` and CASes it — so every field written after it could be overwriting a NEW owner's. The ordering was harmless while the only non-test caller was `TopicHeader::initialize`, which runs single-threaded before `magic` is published; giving a registration back on Drop made it reachable from a live process on every last-handle release. `reap_dead_participants_now` writes the identity first and says why, in this same file — the sequence now lives in `clear()` once, where it belongs. **The participant-exhaustion warnings printed a garbled sentence.** Eighteen literal spaces in the middle: "will deliver nothing". A source-wrapped string literal whose continuation backslash was lost, so the indentation was baked into the message. These two lines are the ENTIRE user-visible payload of the commit whose point was that the 16-slot wall must never be silent, and a diagnostic that exists only to be read was printed malformed. **`install.sh` exported HORUS_PREFIX only when PATH was not yet configured.** Which is the one population that does not need it. A prefix install puts `${HORUS_PREFIX}/bin` on PATH, so every upgrade and re-install takes the "PATH already configured" arm and would never have been given the variable — and install.sh records this exact mistake 60 lines above, for the shell-integration block that used to live in that same arm. It is outside the conditional now, and value-aware: a stale line naming a different prefix is replaced rather than counted as already configured. The two new tests fail with the block back inside the else arm while the fresh-install tests still pass, which is why it was invisible. * fix: an error that asked for the impossible, and a stale doc-comment **The participant-exhaustion error named a remedy nobody can perform.** "...or increasing lease timeout to allow faster reclamation" — there is no env var, builder method, manifest key or public setter for the lease timeout anywhere in horus_core, horus_manager, horus_py or horus_cpp. The only setter is `#[cfg(test)]`, which is the same pattern this branch called a defect when `DegradationPolicy` had it. It was also backwards: a LONGER lease reclaims a dead process's slot more slowly, not faster. That text became user-visible when this branch routed the error into the log — the whole point of "tell the 17th participant why it has no comms" — so shipping advice that cannot be followed undoes the fix it belongs to. It now names what an operator can actually do, which this branch also made true: drop handles you no longer use, because a registration is returned when its last handle drops. It reports the real lease value rather than implying it is tunable. **`record_deadline_miss`'s doc-comment still described the process-wide ceiling.** "Triggers emergency stop only when the process-wide total crosses `max_deadline_misses`" — the function directly below it explains that the process-wide total is kept for the timing report only, and that it "used to be the e-stop trigger, and as a LIFETIME count across EVERY node that was indefensible on a real robot". The e-stop is gated on this node's CONSECUTIVE run. Not cosmetic: an operator sizes a fleet-wide lifetime budget very differently from a per-node run. * fix: three more declared-but-not-delivered defects **`horus doc --watch` was silently dropped.** The flag is declared "Watch for file changes and regenerate" and `run_extract` honours it first thing, but the dispatch guard listed every OTHER extract flag — so a bare `horus doc --watch` fell to the else arm, which opens rustdoc once and returns. The flag did nothing unless an extract flag happened to be passed alongside it. `watch` is in the guard now. **`benchmarks/README.md` named a backend the benchmark does not use.** The `CrossProc-1PMC` row said `SpmcShm`, and a note below credited "the SpmcShm path" with being least affected by the governor. `all_paths_latency.rs:1320` declares `expected_backend: "PodShm"` — a one-to-many POD topology resolves to broadcast, which is the very expectation the benchmark was corrected to stop asserting. No row uses `SpmcShm`, and the note now says so. **`MockTopic` counted recv ATTEMPTS as `messages_received`.** The real transport increments only when `recv()` hands a message back, so a test written against the double and re-run against a real topic disagreed the moment a poll came back empty — which is most polls on an idle topic. The double now counts deliveries, keeping its attempt counters for the fault-injection schedules. Sends stay attempt-counted, because that is what the transport reports too. A double that reports what the real thing does not is the one defect a double cannot have — the same argument that got these counters wired into the transport in the first place. * revert: restore four files the previous commit truncated cf23bb69 was meant to carry three fixes. It also swept in staged changes that were not mine and that I had not read: CONTRIBUTING.md 935 -> 303 lines, horus_core/README.md 528 -> 109, plus .gitignore and docs-contract.yml. All four are intact on main, so that commit was a regression against it. This restores all four to their content at cf23bb69~1, which matches main. The three intended fixes in that commit are untouched. I committed staged paths I had not inspected, in a worktree other sessions also write to. Staging is not authorship: the fix is to pass explicit paths to `git commit`, which is what this one does. * fix: two hints that could never fire, and a preferred section nobody read exit_code_hint documents that "signal-based exits (128+N) are translated into actionable hints", and its H053 arm says "Likely out of memory" - the most useful thing this module knows how to say. Every production caller reached it through `status.code().unwrap_or(1)`, and on Unix `code()` is None for a signal death, so an OOM-killed child took the `_ => {}` "too generic, no hint" branch instead. The 137, 139 and 128+N arms were reachable only from the unit tests that pass the numbers in as literals. shell_exit_code() reports a signal death the way a shell does, and the new test kills a real child rather than asserting against a constant. [hardware] is documented as the preferred spelling and [drivers] as legacy, but cargo_gen and pyproject_gen read `drivers` alone, so a driver declared the preferred way generated no Cargo dep and no PyPI dep - the legacy section did strictly more than the one the docs tell you to use. doctor and native_sync already merged the two by hand; hardware_entries() is now the single place that does it, and all four consumers go through it. * fix: --sim did nothing at all on the Python driver path horus_core's own comment states the failure this guards against: "the entry constructed the REAL driver under `horus run --sim`, which is the exact failure `--sim` exists to prevent". load_from honours that. load_config_entries - which is the whole of the Python path - had no simulation handling of any kind, so horus.drivers.load() instantiated the registered driver class for every entry, `sim = true` or not. Constructing a driver is where the serial port is opened and the motor enabled, so `--sim` was inert in Python while the same manifest under Rust got a stub. One predicate now backs both loaders: entry_is_simulated() folds the sim/simulated alias, HORUS_SIM_MODE and HORUS_SIM_TARGETS together. load_…
|
Merged as part of the #216 merge stack, squashed onto |
Five fixes from a maintainability audit of the tree, framed by one question: what does a human developer taking over from an AI actually pay?
Every claim below was measured or reproduced, and several audit findings are not here because checking them refuted them (noted at the end).
The headline: rustdoc rendered nothing
cargo doc -p horus_core --openproduced a front page with one module (terminal), zero structs, and zero entries in the search index — for a 112k-line runtime crate. The standard first move for a Rust developer meeting a new crate returned nothing.Ten of eleven modules carry
#[doc(hidden)], and that is deliberate — the crate note says "internal crate, usehorus::prelude", andhorus_core/README.mdhas a whole How to read the internals section. The problem is the escape hatch it documents needs nightly and an unstable flag:because
--document-private-itemsdisables rustdoc's private-stripping pass, not its hidden-stripping pass. Verified rather than assumed: with that flag the front page gained exactly one module —utils, thepub(crate)one — and none of the hidden ten.So the attributes are now
cfg_attr'd behind a docs-onlyinternal-docsfeature. Default output is unchanged (still one module — the user-facing boundary is intact); a maintainer on stable gets all eleven and a populated search index.--features internal-docs(maintainers, stable)That change immediately found four broken links
rustdoc does not resolve links inside a hidden module, so
-D warningshad been holding roughly a quarter of the crate and silently skipping the rest. CI now documents horus_core a second time with the feature on, which is the half of this PR worth keeping even if the rest is bikeshedded.The sharpest one:
set_depth's# Errorssection pointed atHorusError::OutOfRange, a variant the error refactor deleted. A maintainer was being told to match on something they cannot write, and nothing in the build disagreed.Two public types were documented as something else
Scheduler— the type a HORUS program is built around — had no doc comment at all. Its description sat one item higher, stranded onPresenceEntry, so a private struct rendered as the orchestrator.RuntimeParams's whole block, including a working doctest, sat on theParamChangeCallbacktype alias.Same cause both times: an item inserted between a comment and its target, with nothing in the build able to notice.
Doctests never ran
grep -rn -- '--doc' .github/workflows/returned nothing.--libexcludes them andcargo doconly renders them, so every///example was unverified. Added a step; 21 pass.Its comment says plainly that it is a thin gate — 193 of 214 examples are
rust,ignore— because a step protecting 21 examples should not be described as protecting the examples. Converting the type-check-only ignores tono_runis the follow-up, and needs this step to exist first.Two variables disagreed about the word "no"
HORUS_LOG_FILE=yeswas false.HORUS_SIM_MODE=nowas true — silently swapping every driver for an inertSimStubNode: actuators never commanded, sensors never read, nothing said so.horus_sys::env::env_flagis now the one answer (1/true/yes/on,0/false/no/off, case-insensitive, trimmed). An unrecognised value returnsNoneso the caller's default stands —HORUS_SIM_MODE=maybekeeps simulation off, the safe direction for a flag that disconnects hardware, and it matches what the reference page already promises forHORUS_SHM_PREFAULT.Deliberately not routed:
HORUS_RT_ALLOW_CORE_SHARINGandHORUS_WIN_REALTIME_CLASS. Their vocabularies are published in horus-docs (the first is documented case-sensitive on purpose), so changing them needs a paired docs change, not a drive-by.Two onboarding docs were actively lying
CONTRIBUTING.md told contributors
readme_contract's Quick Start test is run by nothing and handed them a manual command. TESTING.md saiddocs_parity"never runs". Both false —docs-contract.ymlruns both. Each was true when written and was falsified by the commit that added the invocation.That direction of rot is the one that costs a newcomer: it sends them to hand-run suites CI already covers, and teaches them the workflows can't be trusted. TESTING.md's closing rule now says so and tells the reader to re-derive a row before relying on it.
cpp_toolchain_contractand thenextestconfig really are unrun and keep their rows — re-checked, not carried over.Findings deliberately excluded
The audit's verification pass refuted 40 of 79 claims. Two I dropped after checking myself:
print_linein zero files." A file-level grep counting test modules. Productionprintln!in horus_core/horus_sys/horus_net is effectively zero against 228print_linecalls —src/here holds more test code (154k lines) than production (130k), so naive greps over it report mostly tests. The codebase is disciplined here.is_known_pypi"564 lines" is a 3-line binary search followed by a data table). Not reportable.Verification
fmt,clippy -D warnings,--no-default-features, CI's four cross-compile commands, and both rustdoc modes with-D warningsare clean.horus_core --lib2588,horus_sys --lib253 (+5 new), doctests 21.One
horus_corescheduler test failed mid-run and passed 3/3 in isolation and on a clean re-run — the rotating parallel-test contention this workstation shows under load, in a file this branch does not touch.