You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
merge stack: every open PR in one CI cycle (#189, #198, #205, #207, #210, #213, #215) - #216
Every open PR in one branch, so the queue drains in one CI cycle instead of seven.
Why a stack again
main requires seven contexts with strict: true, so each merge invalidates every
other open PR and the queue drains at one PR per full CI cycle — and every push
re-triggers a Copilot review. This repository has drained the queue this way
before (#119, #146, #174, #204); this is the same mechanism.
One, out of seven branches. horus_core/src/drivers/mod.rs: #205 routed HORUS_SIM_MODE through the new horus_sys::env vocabulary, while #189 moved the
predicate into the free function sim_mode_enabled(value) so its truth table is
testable without racing the process-global environment. Resolved by keeping #189's
structure and giving it #205's vocabulary:
sim_mode_enabled is now value.and_then(horus_sys::env::parse_flag).unwrap_or(false),
so 0/false/no/off/"" all disable simulation, 1/true/yes/on enable it, and an
unrecognised value (=maybe) leaves the default — OFF — in force.
The sim_mode_tests truth table was extended to the shared vocabulary and now
asserts agreement with parse_flag directly, which is the same vocabulary HORUS_NET_ENABLED reads in the scheduler.
Run Benchmarks failed on #189 at f5499e6c with a blocking median regression: Topic_cross_process_oneway_pingpong@1024B 278 ns → 310 ns (+11.5%), and @64B 150 ns → 165 ns (+10%). Empty-poll cost was unchanged between the two
runs (12.7/40.5 ns baseline vs 13.9/39.3 ns PR), so the cost was on the success
path.
The only per-message additions on that path were two Relaxedfetch_adds — messages_sent on send(), messages_received on recv() — added by this
branch's "topic metrics count real traffic" fix. An A/B on the integration tree
(1M samples × 6 pairs, interleaved, performance governor, turbo off) removing
them recovers −18 ns per round trip at 1024 B and −20 ns at 64 B; the merged
tree with them shows +3.6%/+13.0% over the same tree without them. An
uncontended lock xadd on every send and recv is a few nanoseconds of latency
added to the middle of the publish path, and it is a full barrier in a pipeline
that is otherwise measuring a ring-buffer write.
The fix, in this stack:
messages_sent / messages_received move from the Arc-shared MigrationMetrics (an AtomicU64 per message, required there because a Clone can run on another thread) onto RingTopic itself, as Cell<u64>s. RingTopic is deliberately !Sync (see its SAFETY note), so a Cell is sound, and a clone gets its own — the counters are per-handle and
the publish path pays one add instead of a lock xadd.
Consequently a Topic reports the traffic it carried, exactly like MockTopic — the double users write their tests against, which was the PR's
own stated bar. TopicMetrics documents this.
The MigrationMetrics counters that genuinely aggregate across handles
(send_failures, send_retry_overruns, migrations) stay atomic where they
are; only the two per-message traffic counters moved.
Fixes#197 (the stale shared-memory leak #198 hardens CI against).
Review fixes on top of the seven PRs
Copilot's review of this stack raised eight threads. Six were addressed in code,
two were answered with the pinned behaviour they conflict with:
fix(scheduling): a restart whose init() fails stops the node (three
threads) — honor_restart_request_with discarded init()'s Result and
treated only a panic as failure, so a driver whose device failed to reopen
was marked initialised and kept ticking while horus node restart reported
success. Err now clears initialized, stops the node, reports the error and
transitions the context to crashed — the rule reinit_pending_nodes already
applies. New test restart_whose_init_fails_stops_the_node; the
unknown-node test's fixed 400 ms wait becomes a bounded poll.
fix(topic): the metrics counters move on every public send/recv API
(two threads) — messages_sent counted only send() and messages_received only recv(), so try_send/send_blocking/try_recv
traffic read 0 while MockTopic, whose send() delegates to its try_send, counted it. Now every public API counts, the role=Both fast path
owns its count, and the new counters_move_on_every_public_send_and_recv_api
asserts a real topic and a mock report equal numbers.
horus_py auto-detect message — generated types are not re-exported from horus; the suggested import now points at horus._horus for them.
Answered, not changed:uninstall.sh's HORUS_PREFIX removal is
deliberate and pinned by the_prefix_cleanup_leaves_similarly_named_variables_alone, which
constructs exactly the case raised.
Maintenance commits on top
fix(deps): RUSTSEC-2026-0285 — RustSec published the advisory against rustls 0.23.43 on 2026-09-14, after main's last audit run, so every new PR
against main fails Security Audit today (it is strict for PRs targeting
main). Patch update to 0.23.45 (+ rustls-webpki 0.103.15); cargo audit is
clean again. No source changes.
fix(topic): the traffic counters live on RingTopic, not LocalState — the
benchmark fix below, after local_state_fits_four_cache_lines caught the
first cut.
#215's regression test is #[cfg(target_os = "macos")] under horus_core, and no macOS job in this repository builds horus_core's test targets: the macOS
legs build horus_types/horus_macros/horus_cpp_macros and horus_sys, and
the manager shell contracts. So the regression was compiled by nothing and run
by nothing. The last commit adds a step to parity-macos that runs exactly
that test, so the fix is verified and not merely compiled.
Validation on the stack
cargo fmt --all --check
cargo check --workspace --all-targets --exclude horus_py and cargo check -p horus_py --all-targets --no-default-features (the two
commands CI runs)
cargo test -p horus_core --lib — 2623 passed. It caught a real defect in
the first cut of the benchmark fix: the two counters were added to LocalState, whose 256-byte (four cache line) budget is asserted by local_state_fits_four_cache_lines. The counters live on RingTopic now
(also per-handle, also non-atomic), and the layout test is untouched. One
scheduler timing test failed in the parallel run under benchmark load and
passes 3/3 in isolation (wake_comes_from_the_publisher_not_from_the_backstop
— its own comments describe host-load sensitivity).
Closes#184.
`horus install horus-monitor` symlinks into `PluginRegistry::global_bin_dir()`,
which is `paths::horus_dir()/bin` — the platform CONFIG dir, so
`~/.config/horus/bin` on Linux. `which_monitor_binary` checked `~/.horus/bin`
and nothing else, so the install succeeded and `horus monitor` then reported the
plugin as not installed. The two halves sit about 800 lines apart in different
files, and the comment on the path that was checked still called it "the plugin
install location".
The lookup now asks the registry for that directory rather than making a second
hardcoded guess at it, and keeps `~/.horus/bin` behind it so an existing install
goes on working.
The candidate list is split out as `plugin_bin_dirs()` so the relationship can be
asserted rather than assumed — the same shape as `run_python::python_cache_roots`,
which exists because the package cache had this exact defect. Two tests: that the
lookup searches the directory the installer writes, and that the legacy path is
still searched. Verified non-vacuous: removing the registry lookup fails the
first with exit 101.
Three documentation pages carried this as a working recipe and now carry an
`export PATH=...` workaround (softmata/horus-docs#16). Those lines should come
out once this ships.
`restart_node` sent PauseNode immediately followed by ResumeNode and printed
"The scheduler will re-initialize this node on next tick". It did not. Both arms
only call `NodeControls::set_paused` on a shared atomic; the code that re-runs
`init()` is `reinit_pending_nodes`, which keys off `RegisteredNode::initialized`,
and outside registration nothing in production ever set that back to false — the
only two such assignments in the workspace are in `#[cfg(test)]` code.
Four places in the runtime asserted the behaviour anyway: the clap help
("re-initialize without killing scheduler"), `restart_node`'s doc comment, an
inline comment reading "Pause then resume triggers re-init in the scheduler",
and `reinit_pending_nodes`'s own comment, "set by control commands". No control
command set it. On a robot the operator restarts a wedged node, is told it
worked, and moves on.
Adds `ControlCommand::RestartNode { name }`, appended to the enum so a command
serialised by an older CLI still decodes. The scheduler collects restart
requests while draining the control topic — the loop holds a mutable borrow of
`control_topic` and clearing the flag needs `self.nodes` — then clears
`initialized` and lifts any operator pause, since a wedged node is often paused
before being restarted and would otherwise be skipped by the very check that
would have re-inited it. An unknown name logs rather than silently doing
nothing.
Tests drive the real control topic through `run_for`, not `tick_once`: the
control topic is created in `run_with_filter`, so under `tick_once` it is None
and no control command is polled at all. Worth knowing separately — operator
control reaches a scheduler only while it is in `run()`.
Verified non-vacuous: removing the RestartNode arm fails the test with exit 101
and `left: 1, right: 2`, which is precisely the pre-fix behaviour.
Closes#184 is separate; this addresses the restart defect found by the
bidirectional audit.
`consecutive_misses` is the number the `max_deadline_misses` ceiling and
the whole degradation ladder compare against, and nothing reset it from
the deadline check. The reset lived in `NodeTimingState::record_tick`,
reachable only through `check_budget`, which gave three wrong answers:
- RT executor: `tick_node` calls `check_tick_budget` unconditionally and
the deadline check afterwards, so a node woken 5 ms late that executes
in microseconds cleared the run its own miss had just started. The
counter was pinned at 1 forever. Neither `warn_after: 3` nor the
ceiling could fire for the failure mode `check_deadline_from_release`
was written to catch -- the one `primitives.rs` calls dominant.
- Main loop: the call sits inside the budget-violation branch, so a
clean tick never reaches it and the count is a lifetime total.
- A node with `.deadline()` and no `.tick_budget()` skips the budget
block entirely, so nothing cleared it on either path. `node_builder`
derives a deadline from a budget and never the reverse, so this is an
ordinary shape, and for it the per-node consecutive ceiling was a
lifetime total wearing a safety threshold's name -- exactly what the
per-node ceiling was introduced to end.
A budget overrun and a deadline miss are different failures, so a tick
that stayed inside its budget is no evidence its deadline was met. The
reset moves to `SafetyMonitor::record_deadline_met`, called wherever the
deadline check comes back clean on both dispatch paths.
Also: a miss now clears `recovery_counter`. `record_successful_tick`
means "did not panic", and both callers reach it on a tick already
recorded as a miss in the same pass, so a node at RateReduced that
missed EVERY deadline still accumulated `recovery_ticks` of "success"
and was restored to the rate it had just proved it could not hold. The
field's own comment already called itself a consecutive run.
Two existing tests asserted the old contract by calling `check_budget`
with an in-budget duration to mean "recovered"; they now say
`record_deadline_met`, which is the thing they were reaching for.
Each fix was reverted in turn to confirm the tests catch it: the
in-budget run reads 1 instead of 10, the no-budget scheduler node reads
10 instead of 0, the RT node stays at 5 after meeting its deadline, and
the never-recovering node is declared recovered at tick 8.
`max_deadline_misses` is documented as the per-node consecutive ceiling
that stops the robot, defaults to 100, and could not fire at that value
-- or at any value above 20 -- in any shipped build.
`DegradationAction::Kill` sets `is_stopped`, and a stopped node is
skipped forever on both dispatch paths. So the ladder's terminal rung,
hardcoded at 20 consecutive misses, permanently retired the node before
its count could climb any higher: `consecutive_misses` froze at 20 and
the ceiling was unreachable. Nothing warned. `max_deadline_misses(n)`
only clamps with `n.max(1)`, and the sole setter for the ladder is
`#[cfg(test)]`, so no user build could raise the rung either.
Two of the runtime's own defaults contradicted each other, and the unit
tests only ever exercised the ceiling at 3 and 5 -- both below 20 --
which is why the interaction went unnoticed.
A hardcoded constant must not silently cap a documented safety
threshold, so the rung moves: `kill_after` is raised to the ceiling when
the ceiling is higher. The graduated stages below it (warn at 3, half
rate at 5, isolate at 10) are untouched, and an isolated node keeps
ticking -- every tick on the RT executor, on a probe interval on the
main loop -- so it can still reach the ceiling. Where the two now
coincide the e-stop wins, because both paths record the miss, which
evaluates the ceiling, before evaluating the ladder.
Note this changes what a chronically missing node does under the default
configuration: it is no longer killed at 20 while the robot runs on, it
e-stops the robot at 100. That is what the knob, its default, the
builder doc and five documentation pages all say it does.
Both tests assert the ORDER of the two terminal events rather than the
counter -- at this level nothing removes the node, so the count climbs
regardless and the e-stop would fire on its own; what killed the ceiling
is the `Kill` that lands first. Reverting to the hardcoded rung fails
them at 20 for both the configured ceiling of 25 and the default of 100.
`safety.on_link_lost`, `import` and `deny_export` are safety controls.
The 2026-07-30 audit found the whole `[network]` table parsed and then
read by nothing, and `apply_network_config` was written to fix that --
then wired into ONE of the five paths that spawn a run child.
That one is `execute_multiple_files`, reached only when `horus run` is
handed more than one file. A bare `horus run`, a `horus.toml` run, a
directory run and a `[workspace]` run all resolve to a single executable.
On those the child environment came from `build_child_env()`, which sets
PATH, LD_LIBRARY_PATH and PYTHONPATH and nothing else, so
`NetConfig::default()` read an environment nobody had populated: `import`
stayed `Auto`, `deny_export` stayed empty, and the documented comms-loss
safe state never armed. The gap the two existing comments describe was
still open on every invocation a real single-binary robot makes.
`apply_network_config` becomes `network_env() -> Vec<(String, String)>`,
which `build_child_env` now extends with -- so every single-executable
path picks it up from the function they all already call, and no future
path can quietly miss it. `execute_workspace` builds its `Command` by
hand rather than through `build_child_env`, so it gets an explicit
`apply_network_config` call.
The operator still wins: `network_env` skips any key already present in
the environment, so an explicit `HORUS_NET_*=...` beats the manifest on
every path. That is asserted too.
Reverting the `build_child_env` line gives `left: None` for
HORUS_NET_ON_LINK_LOST -- the safety control simply absent.
`horus launch` still performs no translation; that is a separate gap and
the documentation says so.
`start_log_file_drain` was called by no shipped code path. A tree-wide
grep for the symbol returned four hits: the definition, its re-export,
and two lines of one integration test that invoked it by hand. So
`HORUS_LOG_FILE=1 horus run` wrote no `.horus/logs/horus.log`, and
`HORUS_LOG_DIR`, `HORUS_LOG_MAX_SIZE` and `HORUS_LOG_MAX_FILES` were dead
with it -- while the test that proves the drain works kept it green.
The env guard lives inside the function precisely so a call site can be
unconditional, and the function's own doc-comment calls the variable an
opt-in, which is only true if something always calls it. `Scheduler::new`
already does this kind of env-driven wiring twice, for HORUS_NET_ENABLED
and HORUS_RECORD_SESSION, so the drain goes beside them. A process that
does not set the variable pays one env lookup and spawns nothing.
Once per process, via a `OnceLock`: a program may build several
schedulers, and each drain thread tracks its own index into the same
global ring, so N of them would write every entry N times into one file.
This also keeps the promise for C++ and Python, which the doc-comment's
"or calling this function directly" escape hatch could not: the symbol
has no FFI export anywhere in the tree. Both bindings construct a core
`Scheduler` (horus_cpp/src/scheduler_ffi.rs:26,
horus_py/src/scheduler.rs:882), so both now get the drain.
The test does not call the drain. It sets the variable, builds a
`Scheduler` the way any program does, and asserts a log file appears; it
is alone in its binary because it writes process-global environment.
Reverting the wiring leaves the file absent and it fails.
An operator who sets a documented variable to get a post-incident log and
finds an empty disk is the exact failure that variable exists to prevent.
`messages_total` is not a metric. `SubscriptionFreshness::refresh` watches
it as the "new data exists" signal behind `.subscribe_with_timeout()`, and
its answer drives `StalePolicy::SafeState` and `StalePolicy::Stop`.
Only `RingTopic::send` bumped it. `Topic::try_send` and
`Topic::send_blocking` go straight to the ring, so a topic published
entirely through either read as 0 messages and 0 Hz while carrying every
message -- and could safe-state or halt a subscriber node whose data was
arriving normally. That is exactly the failure the counter was introduced
to fix, reintroduced for backpressure-aware publishers, and these are the
two APIs the documentation points users to on critical topics.
The runtime states the invariant in both places that consume the counter:
"every publisher increments `messages_total` in the topic's SHM header on
every send, so watching that counter is a true 'new data exists' signal"
(scheduling/types.rs) and "incremented on every `send()` regardless of
backend" (header.rs). Neither is true.
Bumped once per public publish call, not inside `RingTopic::try_send`,
which `send_lossy_retry` calls up to ~70 times for one message.
`try_send` counts before the call, as `send()` does: an attempt count, and
a publisher hammering a full ring is alive. `send_blocking` does NOT count
a `NoBackpressure` refusal -- that is returned before the send is even
attempted, so nothing reached the ring, and counting it would report a
healthy flow on a topic delivering nothing. A `Timeout` did engage the
ring, so it counts like `send()`'s dropped message does.
Reverting both bumps reads 0 for a topic that carried ten messages.
A topic holds at most 16 participants. `register_role` builds a precise
error for the one that does not fit -- it names the limit, the count, and
what to do about it -- and all three call sites threw it away:
`recv_uninitialized` and `read_latest` were `if ....is_err() { return
None }`, `send_uninitialized` the same with `Err(msg)`. Those are the only
`ensure_consumer`/`ensure_producer` call sites in the tree, so that
message reached nothing, ever.
Because the role stays `Unregistered`, every later `recv()` re-enters the
same cold path and fails identically: `None` forever, from a subscriber
indistinguishable from an idle one.
Nothing anywhere defends the silence, and the identical 16-slot wall one
layer down is handled the opposite way on purpose -- rate-limited
`tracing::warn!` under "COMM-H1: never let 'all 16 slots live' become a
SILENT no-comms endpoint (worse than the pre-fix loud panic)". The rate
limiter for it sits in the same file, unused by this path. Meanwhile
header.rs defends a separate design decision by claiming the residual cost
"surfaces as the existing 'No available participant slots' error. Loud,
and it says what to do" -- a guarantee the code did not provide.
All three sites now report through that same limiter, so a hot loop warns
once a minute per topic rather than once ever.
The limiter is also keyed by reason now, not just topic. A topic can hit
two different walls -- its FanoutShm endpoint bitmap and the participant
table -- with different causes and different fixes, and one bucket let
whichever fired first silence the other for a minute. That is the masking
the per-topic keying was introduced to prevent, one level down.
The test fills the table from 20 threads (the participant-count leak is
what lets one process do that), asserts it really is full before asserting
anything about diagnosis, and captures `tracing` events through a
hand-rolled `Subscriber` so it needs no new dependency. Reverting the fix
prints nothing at all: "Warnings seen: []".
`send_blocking_pod` ended in `.is_ok()`, discarding which
`SendBlockingError` came back, and both arms then reported
`HorusTimeoutError`. So the one error horus_core writes specifically to be
un-retryable arrived in Python as the one error callers retry.
The core is explicit about the distinction, twice: "Refusing is
deliberate: reporting success for a delivery nobody guaranteed is worse on
an e-stop path than failing loudly. It does not clear on retry -- it lasts
as long as the topology that caused it." The variant carries a four-line
message naming the fix. The binding replaced it with the exception whose
own doc-comment means the opposite -- "the buffer stayed full for the whole
timeout" -- and its private helper asserted, falsely, that a `false` return
meant the ring stayed full.
This is easy to reach and permanent: `detect_optimal_backend` resolves any
POD topic with a second subscriber to `PodShm`, so the moment a `horus
topic echo` or a monitor attaches to a POD e-stop topic, every
`send_blocking` starts raising forever -- as a "timeout", on the path the
method is documented for.
The variant is now threaded out of both arms and mapped to a new
`horus.HorusBackpressureError` carrying the core's own remediation text.
`HorusTimeoutError` keeps `Timeout` alone.
Also exposes `Topic.provides_backpressure`, which the core's doc-comment
instructs callers to assert on after the first send and which Python had
no way to ask. It is answered through a new
`horus_core::communication::topic::backend_provides_backpressure`, keyed
by backend NAME because a name is all a binding can see -- and
`provides_backpressure()` is now defined in terms of it too, so the two
cannot disagree. A test walks every `BackendMode` through an exhaustive
match, so a new backend stops compiling rather than silently defaulting to
"no backpressure".
Reverting the mapping fails the test at "must raise
HorusBackpressureError".
`_setup_topics` documents a three-step type resolution whose middle step,
"auto-detect from SHM header", was unreachable code. It did:
probe = Topic(topic_name, 64)
type_name = getattr(probe, 'type_name', None)
Both halves were dead. `PyTopic` exposes no `type_name` — its getters are
`name`, `is_network_topic`, `endpoint`, `backend_type` — so the value was
always `None`. And the probe could not have worked even with that
attribute: `Topic(name, capacity)` builds a `Topic<GenericMessage>`, and
attaching 4364-byte GenericMessage to an existing typed ring is refused by
the size check in `RingTopic::new`, so it always raised into a bare `except
Exception`. Step 2 therefore never ran for any topic that has ever existed,
and the ~70-entry `_TYPE_NAME_MAP` built for it had exactly one reader,
inside the unreachable branch.
The size check is correct and stays: its comment records that a name-based
exemption once let a `Topic<GenericMessage>` join a `Topic<CmdVel>` ring and
"write megabytes past the end of the mapping". The question just cannot be
answered by attaching -- opening requires choosing a type, which is what the
caller is trying to find out.
So: `horus_core::communication::peek_topic_type_name(name)` reads the
header without attaching, exposed to Python as `_horus.peek_topic_type`.
`_detect_topic_type` calls it and maps through `_TYPE_NAME_MAP`, which now
has a reachable reader.
Second half: when the ring exists and holds a type this build cannot name,
`_setup_topics` raises with the topic, the ring's type and the concrete fix
(`subs=[Sub('scan', LaserScan)]`) instead of falling through to a generic
topic and letting the runtime's byte-count RuntimeError out -- which named
neither `subs=`, nor the type the ring holds, nor what to do.
Auto-detection was only ever scoped to subscriptions; the `pubs=` case is
unchanged, since a publisher creates the ring and there is nothing to read.
Tests both ends. The Rust test asserts the old probe's refusal still
happens (it must) and that the peek answers anyway; reverting the peek
gives `left: None`, which is what the old probe returned for every topic.
The Python test drives a real `Node` with a bare `subs=` string against a
typed ring; restoring the old probe fails it.
**`post_test` skipped teardown exactly when teardown matters.** It sat
after an early `?` in the `Commands::Test` arm, so a failing `horus test`
never ran it, while `post_run` and `post_build` go through
`run_teardown_hooks` and run on both paths deliberately: "Skipping it when
the run failed would skip it in exactly the case where the hardware was
most likely left live." The hooks contract file calls `test` "the only
phase with teardown" and then pinned only the symmetry of the KEY NAMES, so
`post_test` kept success-only semantics while the other two were given real
teardown semantics. A `post_test = ["release-bench"]` was skipped precisely
when the tests aborted mid-run.
The new guard reads the dispatch in main.rs rather than the table: every
`post_` phase must go through `run_teardown_hooks` and none through the
plain `run_hooks`. Reverting the arm fails it on both halves.
**`horus doctor` recommended a variable that does nothing.** It printed
"use --net or HORUS_NET=1 to enable" and reported networking from
`HORUS_NET`, which nothing else in the tree reads. Following the advice
changed nothing except that `doctor` then printed "horus_net: enabled",
confirming a state that did not exist -- worse than inert. main.rs already
recorded this exact confusion when it fixed `--net`; the diagnostic handing
users the broken half of the same remedy was not fixed with it.
`doctor` now asks the question the BUILD asks, through a shared
`cargo_gen::net_capability_requested`, and names only routes that work.
`HORUS_NET` is retired: no reader remains.
**`horus install --plugin -t <workspace>` threw the name away.** The call
site collapsed it to `let local = target.is_some()`, and what was left
called `detect_or_select_workspace` -- which resolves the CURRENT
DIRECTORY's workspace, or prints a menu and blocks on stdin. Both are
things `horus install` documents it never does, and with no tty the read
returns empty and the command dies with "Invalid selection". The package
half of the same flag already resolved the name correctly; the bool was a
vestige of a `--local` flag that no longer exists, still named in the
doc-comment.
The ablation shows how completely the name vanished: with the bug the
install proceeds past the target entirely and fails with an unrelated "no
package registry is configured".
`log_drain_loop` snapshots `GLOBAL_LOG_BUFFER.write_idx()` when its thread
starts and only writes entries newer than that snapshot. The test pushed
its entries in one burst immediately after constructing the scheduler, so
on a loaded machine the drain thread could reach the snapshot after the
burst -- leaving nothing newer to write, and the test waiting out its full
10-second window for a drain that was working correctly.
Pushing every polling round instead guarantees entries after the snapshot
whenever the thread got going. Still non-vacuous: reverting the wiring in
`Scheduler::new` fails it exactly as before.
Nothing inside a living process ever returned one. The only
`decrement_to_floor` calls on `publisher_count`/`subscriber_count` are in
`reap_dead_participants_now`, which wants an expired lease AND a dead pid,
and in `register_role`'s last-resort pass, which only runs once all 16
slots are occupied. `Drop for RingTopic` released spill keep-alives and
FanoutShm endpoint bits and left the participant table untouched.
So within one process the counts only grew -- and `detect_optimal_backend`
reads them. A transient subscriber on a fresh thread (a worker pool, a
scoped task, a one-shot diagnostic read) permanently converted a command
topic from backpressured `SpscShm` to overwrite-oldest `PodShm`, with
nothing alive to justify it and no path back for the life of the segment.
It silently voided `try_send`'s full-ring `Err` and `send_blocking`'s
backpressure on an e-stop path. The table also filled toward its 16-slot
limit with registrations belonging to nothing.
`horus_core/tests/participant_count_leak.rs` has documented this as OPEN,
with an `#[ignore]`d test, since the naive fix was built and rejected.
# Why the naive fix was unsafe, and what makes this one safe
Releasing against the recorded slot INDEX is wrong: a slot is not owned for
a handle's lifetime. `register_role`'s pass 3 reclaims slots whose thread
has ended, so a stale index names somebody else's registration by the time
it is given back. `ParticipantEntry` now carries a `generation`, bumped on
every claim in `finish_claim` and recorded by the handle beside the slot,
so a release can tell its own registration from a later one.
A second hazard the earlier attempt did not name: an entry is keyed on
(pid, thread), NOT on the handle, so several `Topic`s on one thread
deliberately share one slot and one increment of `subscriber_count`. A
(slot, generation) match alone would let the first to drop clear an entry
the others are still using. The entry therefore also carries per-role
handle counts -- per role, because one thread publishing and subscribing to
the same topic is the supported `role = Both` fast path, and a single count
could not express "the last subscriber left while a publisher stays".
Every check refuses rather than guesses. Under-counting is the dangerous
direction: `nothing_is_draining` lets a producer retire unread slots once
`sub_count()` reads 0, so a lost registration becomes silent message loss
on a subscriber that is still reading.
Both new fields came out of the entry's existing padding, so it is still 24
bytes and the header is still 640; the static assertions on both are
unchanged.
# The downgrade this exposes, and why it is not taken
With the counts finally able to fall, a two-subscriber broadcast topic
whose second subscriber exits would migrate BACK to `SpscShm` -- and a
backwards migration mid-stream drops whatever is already in the wider ring,
because the migrator's drain has nothing to observe and always succeeds.
Measured directly: with the downgrade allowed, the SURVIVING subscriber of
two received 198 of 200 messages and ended up back on `SpscShm`, and the
two multithread non-POD broadcast tests failed the same way the earlier
attempt made them fail.
So `detect_optimal_backend` now never narrows a running topic. Widening on
a join stays immediate; narrowing has to wait for a drain that is real, and
until then the wider path is both the safe answer and the one the runtime
already took when the counts could not fall at all. Both directions are
pinned by tests.
The counts themselves are correct now regardless, which is what the
participant table, `horus topic info` and the 16-slot ceiling all depend on.
Verified: the previously-`#[ignore]`d test passes and is un-ignored;
`sub_count` climbs to [1, 2, 3] with the release removed. The two
multithread broadcast tests pass 10/10 at base timings (0.12-0.18s), and
the full lib suite is green across six runs.
`BackendMigrator` promised "Drain logic to ensure no in-flight message
loss", and `perform_migration` said it "Drains in-flight operations before
switching the backend mode. If the first drain attempt times out, retries
once before failing."
None of that is true. `drain_in_flight`'s own comment says why: "Nothing in
`TopicHeader` counts in-flight operations -- there is no such field -- so
this function has never had anything to observe. A wedged producer and an
idle topic run the same number of iterations and both return `true`." It
spins, yields three times, and returns `true` on every exit.
So the retry and the `MigrationResult::Failed` it returned were unreachable
code guarding a return value that no longer exists -- and callers wrote
handlers for a case that could not occur, in mod.rs and in the test suite.
The variant is gone with them.
`drain_in_flight` becomes `settle_before_switch` and returns nothing, which
is what it always did. The bound stays: that part was legitimate, and
removing it made `send_blocking` block measurably longer under load.
The guarantee itself is NOT restored here. A real one needs an in-flight
counter incremented and decremented on the send and recv hot paths of a
real-time framework -- a deliberate design decision with a measurable cost,
which the existing comment argues should not be reintroduced by leaving a
deadline around that only looks like it does the job. The doc-comments now
say plainly that a message in flight across a switch can be lost; the
topic's own tests already pre-warm backends "to avoid migration losses".
This is also what the never-narrow rule added alongside the participant
release depends on: narrowing a running topic is unsafe precisely because
this drain is not real.
Two tests pin what is actually true: the settle stays bounded, and an
uncontended migration always completes.
Two halves of one asymmetry: the executors got the isolation half of the
main loop's fault handling and not the escalation half.
**A panicking `shutdown()` cancelled the emergency stop that called it.**
`BudgetPolicy::EmergencyStop` ran `let _ = node.node.shutdown()` bare, and
BEFORE latching. `shutdown()` is user code on a node that is already
failing, so an unwind out of it escaped past `estop.trigger`, past
`running.store(false)` and into the loop's outer `catch_unwind`, whose only
action is to stop the node. `get_state()` kept reporting Normal, no
blackbox EmergencyStop was written, `PENDING_LOCAL_ESTOP` stayed empty so
horus_net broadcast nothing, and the other RT nodes kept ticking -- exactly
the outcome the comment three lines below it was written to prevent. The
docs promise this e-stop unconditionally.
Both `shutdown()` calls (Enforce and EmergencyStop) now go through
`guard_fault_callback`, which primitives.rs already says "any future
`&mut dyn Node` callback reached from an executor belongs behind". The
EmergencyStop arm also latches and signals the thread BEFORE calling into
the node, so the safety action no longer depends on user code returning.
**A critical RT node that could not reach a safe state did not halt the
robot.** `note_safing_failure` states the rule as a property of the system
-- "a node that cannot safe itself is a safety failure -- escalate to a
system emergency stop for critical nodes" -- and `is_critical_node` had
exactly one caller, on the main loop. So the identical event halted the
robot on a BestEffort node and merely stopped an RT one. Every RT node
under a scheduler `.watchdog()` is a critical node, so the class most
likely to be critical was the one the escalation omitted.
`primitives::note_safing_failure` is the executor twin, wired into the
Isolate and Kill rungs of `apply_degradation_action` and into the RT
executor's `Miss::SafeMode` dispatch -- which called `enter_safe_state()`
entirely bare, so a panic there only ever stopped the node.
Ablations: with `shutdown()` bare and ahead of the latch the panic escapes
the test itself; with isolation kept but escalation removed the panic is
caught and no e-stop latches, which is the shipped behaviour.
**`Scheduler::watchdog()` silently tightened the window.** The config
stores whole milliseconds and `as_millis()` truncates, so
`.watchdog(1500_u64.us())` stored 1 ms — and the graduated ladder latches a
system-wide emergency stop past 3x the window, so it fired at 3 ms instead
of the 4.5 ms asked for, with no diagnostic.
The builder states the rule immediately above the line that broke it:
"Round up to the 1ms floor the storage can represent rather than truncating
... a slightly-looser watchdog is a safety feature, and no watchdog is not
... silently changing a safety timeout is its own problem — the operator
needs to know the value actually in force." That held for the sub-1 ms case
and was inverted for every other one. Tightening is the direction that
halts a healthy robot.
Ceiling division now, across the whole range, and the warning fires for any
value that had to change — not just the sub-millisecond one. An explicit
`Duration::ZERO` is still an honoured disable.
**Three of the four counters on the public `TopicMetrics` were dead.**
`messages_sent` and `messages_received` moved only inside
`send_with_content_logging` / `recv_with_content_logging` — both `#[cold]`,
both reached only while the `horus monitor` TUI has set a topic's verbose
flag. So accessors documented "Messages sent through this topic" read 0 in
every normal run, however much traffic the topic carried.
`MockTopic` — the stand-in users write their tests against — maintained
both faithfully all along, with its own tests asserting on them. A double
that reports what the real transport does not is the one defect a double
cannot have. Both are counted on the ordinary paths now, beside the
existing `messages_total` bump. That comment prices the SHM counter at
~28 ns, but this is a process-local uncontended relaxed increment on a
counter only this handle writes; the cold paths no longer double-count.
`recv_failures` had no increment anywhere in the transport and no honest
meaning there — a `recv()` that returns `None` is an empty ring, not a
failure. The dead field is gone from the transport-side struct, the
snapshot reports 0 explicitly, and the accessor now says so and points at
`missed_count()` for real consumer-side loss. It stays on `TopicMetrics`
because `MockTopic` genuinely populates it from its fault injection, which
is the whole point of the double.
Ablations: restoring `as_millis()` reads 1 ms where 2 ms was asked for;
removing the ordinary-path counts reads 0 for a topic that carried five
messages.
…ed twice
**`horus launch --status` reported every node as running, forever.**
`show_launch_status` colours a STATUS column with "running", "crashed" and
"stopped" arms, and `LaunchProcessEntry::status` was assigned exactly once
in production code -- `"running"` at construction -- and never mutated. Two
of the three arms could not be reached.
The monitor loop already knows: the same iteration that drops a dead node
from the supervised vector logs `LaunchEvent::NodeCrashed`. Two halves of
one workflow disagreed -- the JSONL event log knew, the session manifest
the CLI reads did not.
Worse, the refresh that followed re-associated by POSITION. Removing a dead
node shifts every later index, so a surviving node's PID was written into
the dead node's row: an operator asking whether the perception node crashed
was told "running", beside a PID belonging to a different, healthy process.
`refresh_session_manifest` is keyed by name, records `crashed` for a
non-zero exit and `stopped` for a clean one, and puts a respawned node back
to `running`. It is extracted so the property is testable without driving
the supervisor loop.
**A per-node `namespace:` did nothing, and said so nowhere.** It was parsed,
composed into the name `--dry-run` PRINTS, and exported as
`HORUS_NODE_NAMESPACE` -- which had exactly one occurrence in the whole
repository, this write, and no reader in Rust, Python or C++. Its
doc-comment promised "Namespace prefix for topics", which horus_core cannot
do: there is no topic prefix or remap mechanism. The sibling `priority`
export at least carries an explicit "this is currently a no-op" note.
Giving it real meaning is a horus_core design decision (topic remapping),
not a launcher change, and it cannot simply be "restored": composing it
into the child's `HORUS_NAMESPACE` would put the node in a different
shared-memory tree from its siblings and break the launch it belongs to.
horus_sys states that granularity as "per robot" -- the file-level key, not
this one.
So the runtime stops lying while the semantics stay undecided: the export
is gone, launching warns and says what to do instead, `--dry-run` shows the
name the process is actually given rather than a composed identity nothing
receives, and the field's doc-comment says it is inert and why.
Ablation: with the positional refresh restored, a crashed node reads
"running" and all three tests fail.
… start runs
**`Topic(X).send(instance)` and `node.send(name, instance)` disagreed.**
`Topic("name")` carries `GenericMessage`, and `generic_message_from_py` ran
`pythonize::depythonize` on the Python object — which cannot read an opaque
`#[pyclass]`, i.e. every HORUS message class and every msggen-generated
class. So it raised `TypeError: Failed to convert Python object: unsupported
type CmdVel`, while `Node.send` worked, because it catches that refusal and
retries the object as a dict.
Two spellings of one operation disagreeing is the defect, so the conversion
moves to where all three publish paths share it. `message_object_as_dict` is
the Rust twin of `horus._message_to_dict`, used the same way: only after a
direct conversion has been refused, so it needs no guess about which types
are messages. An object with nothing to extract still raises.
This is what `build_messages()` prints on success and what
`horus/msggen/__init__.py` promises, so both now describe something that
runs. Note serde derives on the generated Rust struct would NOT have helped:
the failure is on the Python side of the boundary.
**`build_messages()` printed two lines that both failed.** `from horus
import <Name>` raises ImportError — a generated class is registered on the
extension module and `horus/__init__.py` re-exports a fixed hard-coded list
— so it prints `from horus._horus import <Name>` now, plus a send line.
**msggen promised POD zero-copy it does not provide.** A generated class
resolves through the generic backend, so it travels the MessagePack path.
It works; it is not built-in-type latency, and the docstring says so.
**`help(horus)`'s quick start raised on its second line.** It published an
`Image` over a bare-string topic and called it zero-copy: a string argument
builds a `GenericMessage` topic, so the send raised and the `recv()` /
`to_numpy()` lines never ran. It uses the typed spelling now,
`Topic(Image, endpoint=...)`, which the documentation already showed.
The Common Mistakes entry claiming "`Topic(int)` or `Topic(42)` raises
TypeError" was also half wrong: `int.__name__` extracts cleanly, so
`Topic(int)` silently opens a generic topic named "int". It says which is
which, and how to spell a pool-backed topic.
These docstrings are not incidental — `python_api_docs.rs` exists to pin
them, on the stated grounds that "its docstrings are the only API reference
a user gets at the prompt".
Ablation: with the fallback declining, the two message-object tests fail
with the exact shipped error.
…ntry
**`simulated = true` did nothing, and HORUS writes it itself.** `horus add
<name> --driver --source sim` — and `--source simulated`, and `--source
sim3d` — all emit `simulated = true` into horus.toml. The loader read only
`sim`, so an entry the CLI generated constructed the REAL driver under
`horus run --sim`: the exact failure `--sim` exists to prevent. `horus
check` stayed quiet too, because `simulated` is a known field rather than an
unknown one, and the loader's RESERVED list kept it away from the driver as
well. The field's own doc-comment declares a purpose.
It is an alias of `sim` now, extracted as `asks_for_simulation` so its truth
table can be tested without a config file or the process-global environment
— the same reason `sim_mode_enabled` is a free function. An explicit `sim`
still wins.
**An exec driver was named after its binary, not its manifest entry.**
`ExecDriver::from_config` — the function a horus.toml exec entry actually
takes — discarded the `[hardware.<name>]` key its caller already holds (and
uses one line later) and derived the name from the binary's file stem. So
two `[hardware]` entries pointing at one binary were indistinguishable in
"Exec driver '{}' launched" while the scheduler knew them by their distinct
keys.
It also injected `HORUS_PARAM_*` and no name, so `HORUS_DRIVER_NAME` reached
nothing on the live path: the only function that set it is `from_params`,
whose sole caller is a unit test. The environment-surface contract asserts,
in prose, that it "is injected into every user-written exec driver" —
`docs_contract.rs:774` — which is now true and can stay.
`load_exec_prefix` asserted the defect as the contract, with the comment
"ExecDriver node name is derived from binary name". Updated: the driver is
named after the entry, matching the name the scheduler uses.
Ablations: without the alias, `simulated = true` reads as "not simulated";
with the file stem restored, the driver reads "rplidar_driver" where the
operator wrote `front_lidar`.
`horus_params_get_string` returned the number of bytes WRITTEN, so a caller
had no way to tell a complete value from a truncated one and no way to size
a buffer around it: `Params::get_string` handed back an ordinary
`std::optional<std::string>` with a clipped value in it, indistinguishable
from a whole one. It also returned that positive count for a NULL buffer —
the write is guarded, the return was not — and split UTF-8 sequences.
The contract this should have had is in the same file.
`horus_topic_last_error` returns -1 for a null buffer or a zero length,
walks the copy back to a char boundary because "half a UTF-8 sequence is not
something to hand a caller that may re-encode it", and has a test asserting
both. Its inputs are strictly LESS arbitrary than a param value: a topic
name is length-checked before it gets there, while a param holds paths, URLs
and calibration blobs.
Nothing in `params.hpp`, `horus_c.h` or the shim states a length limit, so
1024 was an artefact of a stack buffer rather than a contract — which is why
Rust, Python and C++ disagreed about what `get("key")` returned for one
file. Only C++ clipped.
The shim now returns the value's FULL length, so `>= buf_len` means "this
was truncated, and here is the buffer size that holds it", and
`Params::get_string` re-calls into a `std::string` sized from it. The header
documents the convention.
One layer up, this class's own constructor refuses to construct on a bad
file precisely because "`get<T>(key, default)` gives the caller no way to
tell those apart from the values in the file" — a silently clipped string
was that same defect, delivered through that same `get<T>`.
Ablation: with the old shim, a 2000-byte value reports 15, a null buffer
returns 5 instead of -1, and the truncated length is wrong.
`HORUS_PREFIX` is the whole interface to a relocated install. Both readers
take it from the environment of the running `horus` process and from nowhere
else — `version.rs` says so outright: "there is no second name to read, and
no way to discover a prefix install from outside its own tree".
`run_rust`'s cache roots and the version gate's state root are both derived
from it.
The installer never put it there. The rc file got a PATH line only, `horus
env --init` writes proxy functions and no exports, and the prefix epilogue
told the user to export `PATH` and `HORUS_SOURCE`. So for every prefix user
the cache root fell back to `~/.horus/cache`, was deduped away as the legacy
root — meaning the prefix cache was never searched — and the state root was
`~/.horus`, holding none of that install's files.
Which means the Rust half of that change landed only for a user who guessed
the variable, and the two epilogue statements it was supposed to supersede
were still true. `version.rs` records the intent it was meant to serve: the
gate "was silently off for exactly the installs that most need it, and
install.sh had to print a warning saying so".
So install.sh writes `export HORUS_PREFIX` beside the PATH line — `set -gx`
in config.fish, because fish has no `export` builtin and a POSIX line stops
that file loading, the same trap this script already documents for PATH.
With that done, the `export HORUS_SOURCE` instruction and the "version gate
is inactive for a prefix install" warning are both false and go. The
uninstall hint stays: nothing outside the prefix records where it was.
Also: `generate_messages_from_yaml`'s docstring sat AFTER its `_HAS_YAML`
raise, which made it a no-op expression rather than a docstring —
`__doc__` was None and `help()` showed none of the YAML schema it documents.
Moved above the guard, and it now also warns that
`horus_py/src/custom_messages` is generator-owned: `build_messages()`
rewrites that `mod.rs` to name only registered messages, so a file the YAML
emitter drops there is declared by no `mod`, never compiled, and left behind
by every later build.
Ablation: with the rc line removed, a prefix install writes only
`export PATH=...` and both new tests fail.
Main gained #182 and #183 (this session's earlier PRs, now merged) plus
#188 and #190 from other work. The only content conflict was
`which_monitor_binary` in horus_manager/src/main.rs, where #188 landed the
same fix this branch already carried as b20e9f7.
Resolved to this branch's version, which is a strict superset: it resolves
through `plugin_bin_dirs()` — covering `global_bin_dir()`, what #188 uses,
AND the legacy ~/.horus/bin — and additionally gates the project-local
.horus/bin behind HORUS_ALLOW_LOCAL_PLUGINS, so a cloned repository cannot
shadow an installed binary with an untrusted one. Nothing from #188 is lost.
#188 landed the same #184 fix while this branch carried its own, and the
merge resolution kept this branch's copy. Main's is the one to keep: it
calls `PluginRegistry::global_bin_dir()` directly rather than through a
`plugin_bin_dirs()` helper that also searched `~/.horus/bin`.
That fallback was searching a path nothing writes. Every `.horus/bin`
writer in the tree is PROJECT-relative — `project_dir.join(".horus/bin")`
in lockfile.rs, `PathBuf::from(".horus/bin")` in registry/install.rs. The
HOME-relative `~/.horus` holds installer state, never plugin binaries,
which is exactly what #188 established. So the helper and the test
asserting the legacy path stayed searched both go.
What did not land with #188 is a guard on the property, so the drift that
caused #184 — two hardcoded copies of one path, 800 lines apart in two
files — is still unpinned. `monitor_plugin_lookup_contract` reads the body
of `which_monitor_binary` and asserts it resolves through the accessor and
builds no home-relative plugin path of its own. `which_monitor_binary` is
in the binary crate, so an integration test cannot call it; reading it
catches the one thing that actually went wrong.
One content conflict, in `send_blocking_pod`, where both sides were right
about different things:
* main (#192) replaced `.lock().expect("lock")` with `topic_lock()`, so a
poisoned lock raises a Python error instead of aborting the interpreter;
* this branch stopped collapsing the result with `.is_ok()`, so a caller
can still tell `NoBackpressure` — deliberate and un-retryable — from a
genuine `Timeout`.
They compose, and the resolution keeps both.
The merge also left five sites in the Generic and pool-backed arms still on
`.lock().expect("topic lock poisoned")`: #192 converted 189 call sites, and
these five survived only because this branch had modified those same
regions, so the merge preferred our side. That is a defect that exists in
neither contributing branch — panicking on a poisoned lock in exactly the
arms the other 189 no longer do. Converted, with `send_blocking` keeping its
`Result` whole.
The Rust shim's contract is tested in c_api.rs, but the C++ wrapper's
re-fetch — the branch that runs when a value does not fit the 1024-byte
stack buffer — had no coverage on either side, and that is where an
off-by-one lives.
Sizes either side of the buffer and exactly on it (1023, 1024, 1025, 4096),
plus a value made of three-byte characters so a naive cut would land
mid-sequence.
Verified by linking the real library and running it, not only by compiling:
with the wrapper reverted to its single-call form the binary aborts on
`*got == value`, because it builds a std::string from a length the buffer
never held.
The release path guards on (slot, generation) and per-role handle counts,
and those are read-modify-write sequences on shared memory that several
threads reach at once. Reading the code cannot rule out an interleaving that
double-decrements or strands a slot, and both directions are bad:
over-counting pins a command topic on a lossy broadcast backend, while
under-counting lets `nothing_is_draining` retire unread slots beneath a
subscriber that is still reading.
Eight threads churn a subscriber and a publisher each, forty rounds, against
one topic — so it exercises the shared-entry case as well as the generation
guard. When they are joined nothing is alive, so sub_count must be 0 and
pub_count must be exactly the one handle this thread still holds. 6/6 runs.
The sibling-handle direction is already pinned by
`one_handle_leaving_does_not_deregister_its_siblings_on_the_same_thread`:
releasing on the first handle instead of the last fails it with left: 0,
right: 1 — the registration taken out from under a live reader.
An under-count I introduced with the participant release, found by an
adversarial pass over that commit and confirmed by the test added here.
A participant entry is keyed on (pid, thread), and its ROLE BITS are shared
by every handle on that thread. So "the entry already has the producer bit"
says nothing about whether the CALLING handle holds it. The per-role handle
count was incremented only when the bit was newly set on the entry, or when
the handle had no slot yet:
if old_role & role_bit == 0 || new_handle { counter.fetch_add(1) }
Two handles on one thread that each publish and subscribe therefore counted
2 and 1 instead of 2 and 2: the second handle to take an already-set role
was invisible, and the first Drop handed that role back while the other
handle was still using it.
That is the under-count direction, which is the dangerous one and the whole
reason the release path refuses rather than guesses everywhere else:
`nothing_is_draining` lets a producer retire unread slots once `sub_count()`
reads 0, so a lost subscriber registration becomes silent message loss on a
subscriber that is still reading. The new test shows exactly that — sub_count
0 while a live handle is mid-read.
The condition was unnecessary as well as wrong: `ensure_role` returns early
when the calling handle already holds the role, so reaching `register_role`
means the handle is newly taking it and the increment is unconditional. That
makes the `new_handle` parameter dead, so it is gone rather than left
threaded through three signatures.
`monitor_plugin_lookup_contract` asserts two things: that the lookup
resolves through `PluginRegistry::global_bin_dir()`, and that it builds no
HOME-relative plugin path. Neither looks at the project-local branch, so
deleting the `HORUS_ALLOW_LOCAL_PLUGINS` gate leaves both green.
That gate is the thing standing between a cloned repository and arbitrary
code execution. `.horus/bin/horus-monitor` is CWD-relative and was once
checked unconditionally and BEFORE `PATH`, so `cd`-ing into a repo that
ships that file and running `horus monitor` executed it directly — no trust
gate, no signature check, none of the sandbox `plugins::executor` applies —
and it shadowed a properly installed binary.
Asserted now, and skipped cleanly if the local branch is ever removed
outright, since that also satisfies the property.
Ablation: with the gate dropped and the local lookup kept, the two existing
tests still pass and this one fails.
…c-shared atomic
The benchmark gate blocked #189 on a +11.5% median regression of
Topic_cross_process_oneway_pingpong@1024B. The only per-message additions
on that path were two Relaxed fetch_adds (send and recv); an A/B on this
box removes 18-49 ns per round trip when they are dropped, and an
uncontended lock xadd serialises a pipeline that is otherwise measuring a
ring-buffer write.
The counters move to LocalState, which is per-handle by the crate's
thread-ownership contract, so they are plain u64 increments and each
handle reports the traffic it carried — matching MockTopic, the double
users write their tests against. The Arc-shared MigrationMetrics keeps the
counters that genuinely aggregate across clones.
#215's macos_reader_opens_a_live_posix_topic is #[cfg(target_os = "macos")]
under horus_core, and no macOS job in this repository builds horus_core's
test targets — the macOS legs build horus_types/horus_macros/
horus_cpp_macros and horus_sys. The new step runs exactly that test, so the
fix is verified rather than only compiled.
local_state_fits_four_cache_lines caught the first cut: two u64 fields
grew LocalState from 256 to 264 bytes, and that 256-byte budget is a
performance contract, not a style rule. RingTopic is per-handle too (a
Clone builds a fresh one), so the counters move there as Cell<u64>s —
same one add per message, same per-handle semantics, no layout change.
fmt and clippy clean; local_state_fits_four_cache_lines and
topic_metrics_count_traffic_without_the_monitor_attached pass.
…n-level handshakes
RustSec published RUSTSEC-2026-0285 on 2026-09-14, after main's last
audit run, so every new PR against main fails Security Audit today.
Patch update: rustls 0.23.43 -> 0.23.45 (and rustls-webpki
0.103.13 -> 0.103.15). No source changes; cargo audit is clean again.
Blocking metrics: median, p99/median. Report-only: p95, p99, p99.9, p99.99, max, max_jitter, p99.9/median, ns/msg. Tail metrics past p99.9 are advisory on a shared runner — one host preemption owns max.
Raw benchmark output
------------------------
one-way, offset-free 1000000 443.0 523.4 633.0 3350.2 7879.0 26533.5 | 1000 100
round trip (RTT) 1000000 909.5 1070.4 1291.2 6881.3 15809.4 53104.4 | 1000 100
RTT halved (one-way) 1000000 454.8 535.2 645.6 3440.6 7904.7 26552.2 | 1000 100
leg A->B (apparent) 1000000 420.8 496.5 673.0 4926.6 14922.5 52385.3 | 1000 99
leg B->A (apparent) 1000000 406.9 529.5 641.7 1555.6 10075.9 35559.4 | 1000 100
responder turnaround 1000000 21.7 23.5 24.3 33.9 68.7 16024.2 | 997 100
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=27.4 ns window=1.126 s
measured window: minor_faults=1 major_faults=0 ctx_switches vol=222 invol=7
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3599MHz max=3601MHz mean=3600MHz spread=0.03% (n=222)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3591MHz max=3605MHz mean=3600MHz spread=0.39% (n=222)
frequency-normalised `one-way, offset-free`: median 1884 core cycles, p99.9 12061 core cycles at f_core=3.60 GHz
cross-core TSC offset theta_hat = -16.5 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7336 <-> pid 7348
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 193.0 265.2 341.7 4866.6 10898.4 19583.1 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=3.5 ns window=2.000 s
measured backlog: 0.03 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=16
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3599MHz max=3601MHz mean=3600MHz spread=0.06% (n=394)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3599MHz max=3607MHz mean=3600MHz spread=0.21% (n=394)
frequency-normalised `one-way (assumes TSC coherence)`: median 955 core cycles, p99.9 17520 core cycles at f_core=3.60 GHz
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +16.5 ns (add to correct).
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7336 <-> pid 7351
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 353.0 513.0 805.2 5120.5 10449.8 19593.5 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=14.3 ns window=2.000 s
measured backlog: 0.05 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=13
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3599MHz max=3601MHz mean=3600MHz spread=0.05% (n=394)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3594MHz max=3605MHz mean=3600MHz spread=0.32% (n=394)
frequency-normalised `one-way (assumes TSC coherence)`: median 1847 core cycles, p99.9 18434 core cycles at f_core=3.60 GHz
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = -16.5 ns (add to correct).
== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
!! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
!! time spent waiting in the ring. It is a backpressure/throughput figure.
!! Do not quote it as an IPC latency; that is the exact mistake the retired
!! `Topic_cross_process` metric shipped for months. Read `measured backlog`
!! below: ~1 message means the subscriber kept up and no queue formed.
json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7336 <-> pid 7353
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
publish->receive incl. queue 200000 135.6 306.9 507.8 9808.9 15111.2 55326.8 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=22.2 ns window=0.042 s
measured backlog: 1.47 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=9 invol=1
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3600MHz max=3600MHz mean=3600MHz spread=0.00% (n=9)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3600MHz max=3601MHz mean=3600MHz spread=0.02% (n=9)
frequency-normalised `publish->receive incl. queue`: median 1105 core cycles, p99.9 35312 core cycles at f_core=3.60 GHz
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +16.5 ns (add to correct).
SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
json metric key what it measures n median p99 p99.9 p99.99 max
------------------------------------------------------------------------------------------------------------------------------------------
Topic_cross_process_oneway_pingpong@64B ONE-WAY latency 1000000 284.3 367.8 1447.7 7313.0 27569.5
Topic_cross_process_oneway_pingpong@1024B ONE-WAY latency 1000000 523.4 633.0 3350.2 7879.0 26533.5
Topic_cross_process_oneway_paced_stream@64B ONE-WAY latency 200000 265.2 341.7 4866.6 10898.4 19583.1
Topic_cross_process_oneway_paced_stream@1024B ONE-WAY latency 200000 513.0 805.2 5120.5 10449.8 19593.5
Topic_cross_process_unpaced_queue_delay@64B QUEUE DELAY (not latency) 200000 306.9 507.8 9808.9 15111.2 55326.8
------------------------------------------------------------------------------------------------------------------------------------------
Instrumentation is INCLUDED in every figure and never subtracted.
`Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
`RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
the row to compare against any competitor publishing a halved ping-pong.
`Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
however long the message waited in the ring, so it scales with the achieved backlog
(printed in its scenario block) and is NOT a latency.
A figure without its payload size, ring depth and topology is not comparable to anything.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! DVFS is live: scaling_min_freq (800MHz) != scaling_max_freq (2300MHz). Pin them equal before quoting any absolute figure.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! pingpong (64 B): 3 MINOR PAGE FAULTS inside the measured window — a pre-fault was missed, and those faults are in the tail figures.
!! pingpong (1024 B): 1 MINOR PAGE FAULTS inside the measured window — a pre-fault was missed, and those faults are in the tail figures.
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.47 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-2.json
keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
will report it as missing and fail the gate until the window is refreshed —
that is intended, and is why the key was not silently reused.
Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
8 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║ HORUS Robotics Message Types Benchmark ║
╠══════════════════════════════════════════════════════════════════╣
║ Testing real-world robotics message latency (REP 2014) ║
╚══════════════════════════════════════════════════════════════════╝
Platform: INTEL(R) XEON(R) PLATINUM 8573C (4 cores)
Iterations: 50000
Warmup: 5000
╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type │ Size (bytes) │ Typical Rate │ Use Case ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 1000+ Hz │ Velocity control commands ║
║ Imu │ 296 │ 500+ Hz │ IMU sensor fusion ║
║ LaserScan │ 1480 │ 10-40 Hz │ 2D lidar navigation ║
║ JointCommand │ 1032 │ 500+ Hz │ Manipulator control ║
╚═════════════════════════════════════════════════════════════════════════════╝
[Topic] Running benchmarks — timing tx.send() enqueue cost only
(cross-thread consumer drains in the background; no sample
waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
CmdVel 16 bytes │ median: 22ns │ p99: 61ns │ CV: 2.2656
Imu 304 bytes │ median: 34ns │ p99: 65ns │ CV: 1.4190
LaserScan 1480 bytes │ median: 56ns │ p99: 104ns │ CV: 1.8756
JointCommand 928 bytes │ median: 65ns │ p99: 95ns │ CV: 0.8054
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message │ Size │ Median (ns) │ p99 (ns) │ CV │ Misses │ Sustained rate ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 22 │ 61 │ 2.2656 │ 0 │ 8.49 M msg/s ║
║ Imu │ 304 │ 34 │ 65 │ 1.4190 │ 0 │ 6.06 M msg/s ║
║ LaserScan │ 1480 │ 56 │ 104 │ 1.8756 │ 0 │ 1.89 M msg/s ║
║ JointCommand │ 928 │ 65 │ 95 │ 0.8054 │ 0 │ 4.56 M msg/s ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.
╔══════════════════════════════════════════════════════════════════╗
║ REAL-TIME SUITABILITY ANALYSIS ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop ║
║ guarantees: a real loop also pays the receive side, the node's ║
║ compute, and the scheduler's jitter, none of which is measured ║
║ here. The headroom is printed so a pass by four orders of ║
║ magnitude cannot be read as a tight result. ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel (1kHz control): ~ PASS by 16393x — gate not discriminating (p99=61ns vs 1000000ns)
║ CmdVel (10kHz control): ~ PASS by 1639x — gate not discriminating (p99=61ns vs 100000ns)
║ Imu (500Hz fusion) : ~ PASS by 30769x — gate not discriminating (p99=65ns vs 2000000ns)
║ LaserScan (40Hz lidar) : ~ PASS by 240385x — gate not discriminating (p99=104ns vs 25000000ns)
║ JointCmd (500Hz control): ~ PASS by 21053x — gate not discriminating (p99=95ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝
Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-2.json
===== repetition 3/3 =====
HORUS cross-process benchmark — ONE-WAY LATENCY (headline) + unpaced queue delay
Two OS processes on two distinct physical cores over the real Topic SHM transport.
Every handle holds exactly one role, so the `role == Both` same-thread fast path that
produces the repo's headline send-cost figure is never taken. Round-trip rows are
labelled where they are a halved round trip. See the module header for the clock
decomposition and for why the `Topic_cross_process` JSON key was retired.
MACHINE — every nanosecond below is conditional on this block
------------------------------------------------------------------------------------------------------------------------------------------
cpu model : INTEL(R) XEON(R) PLATINUM 8573C (2 physical / 4 logical cores)
kernel : 6.17.0-1022-azure (x86_64)
benchmark cores : cpu3 = process A (pinger / subscriber), cpu1 = process B (responder / publisher) [DistinctPhysical]
governor : cpu3=performance cpu1=performance driver=intel_cpufreq
scaling range : 800MHz .. 2300MHz (hw max 2300MHz) turbo_disabled=true
clocksource : tsc tsc flags: [tsc, rdtscp, constant_tsc, tsc_reliable, nonstop_tsc]
TSC rate (calibrated) : 2300.160 MHz (0.4348 ns/cycle) — a WALL clock; NOT the core clock
clock instrumentation : 28 cycles = 12.2 ns for serialize()+rdtsc()+rdtscp(), INCLUDED in every sample and never subtracted
transparent_hugepage : [always] madvise never
sched_rt_runtime/period: 950000 / 1000000 us perf_event_paranoid=4
isolcpus=(none) nohz_full=(none) smt_active=1 mlockall(MCL_CURRENT)=true
------------------------------------------------------------------------------------------------------------------------------------------
running ping-pong (one-way latency), 64 B payload ...
running ping-pong (one-way latency), 1024 B payload ...
running paced stream (one-way latency), 64 B payload ...
running paced stream (one-way latency), 1024 B payload ...
running unpaced stream (QUEUE DELAY, not latency), 64 B payload ...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! READ THIS BEFORE QUOTING ANY NUMBER BELOW
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! DVFS is live: scaling_min_freq (800MHz) != scaling_max_freq (2300MHz). Pin them equal before quoting any absolute figure.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! pingpong (64 B): 3 MINOR PAGE FAULTS inside the measured window — a pre-fault was missed, and those faults are in the tail figures.
!! pingpong (1024 B): 1 MINOR PAGE FAULTS inside the measured window — a pre-fault was missed, and those faults are in the tail figures.
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.49 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7374 <-> pid 7383
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way, offset-free 1000000 238.7 308.2 403.5 1394.7 7334.7 24523.1 | 1000 100
round trip (RTT) 1000000 499.1 639.1 829.5 3441.5 14741.6 49068.8 | 1000 100
RTT halved (one-way) 1000000 249.5 319.5 414.8 1720.7 7370.8 24534.4 | 1000 100
leg A->B (apparent) 1000000 235.6 340.0 457.4 549.5 13640.8 48668.8 | 998 100
leg B->A (apparent) 1000000 226.1 322.6 416.5 460.8 8609.8 37493.0 | 987 100
responder turnaround 1000000 20.9 22.6 23.5 27.8 58.3 16337.1 | 929 94
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=7.4 ns window=0.684 s
measured window: minor_faults=3 major_faults=0 ctx_switches vol=134 invol=7
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3599MHz max=3601MHz mean=3600MHz spread=0.03% (n=134)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3594MHz max=3606MHz mean=3600MHz spread=0.34% (n=134)
frequency-normalised `one-way, offset-free`: median 1110 core cycles, p99.9 5021 core cycles at f_core=3.60 GHz
cross-core TSC offset theta_hat = +8.7 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process ping-pong -- ONE-WAY LATENCY (2 processes, 2 topics, 1 role per handle) ==
json metric key: `Topic_cross_process_oneway_pingpong` (identity used by the CI gate: `Topic_cross_process_oneway_pingpong@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7374 <-> pid 7385
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way, offset-free 1000000 433.4 544.3 647.3 3012.4 7852.9 27925.4 | 1000 100
round trip (RTT) 1000000 889.5 1111.2 1318.2 6063.9 15760.6 55873.5 | 1000 100
RTT halved (one-way) 1000000 444.8 555.6 659.1 3032.0 7880.3 27936.7 | 1000 100
leg A->B (apparent) 1000000 413.0 546.0 689.5 2181.6 14848.5 55291.8 | 1000 100
leg B->A (apparent) 1000000 401.7 543.4 648.7 858.2 8393.3 41891.0 | 998 100
responder turnaround 1000000 20.9 22.6 23.5 28.7 58.3 17356.2 | 846 77
------------------------------------------------------------------------------------------------------------------------------------------
samples=1000000 warmup=200000 transferred=2000000 lost=0 invalid(TSC-skew)=0 poll cost=27.4 ns window=1.161 s
measured window: minor_faults=1 major_faults=0 ctx_switches vol=229 invol=10
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3600MHz max=3601MHz mean=3600MHz spread=0.04% (n=229)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3600MHz max=3601MHz mean=3600MHz spread=0.03% (n=229)
frequency-normalised `one-way, offset-free`: median 1960 core cycles, p99.9 10845 core cycles at f_core=3.60 GHz
cross-core TSC offset theta_hat = +1.3 ns (clock B - clock A). The `one-way, offset-free` row does not depend on it.
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7374 <-> pid 7387
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 196.5 271.3 338.2 2152.9 9861.1 60975.8 | 199 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=3.0 ns window=2.000 s
measured backlog: 0.03 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=13
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3599MHz max=3601MHz mean=3600MHz spread=0.04% (n=394)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3598MHz max=3604MHz mean=3600MHz spread=0.16% (n=394)
frequency-normalised `one-way (assumes TSC coherence)`: median 977 core cycles, p99.9 7750 core cycles at f_core=3.60 GHz
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +8.7 ns (add to correct).
== cross-process paced stream -- ONE-WAY LATENCY (10000 ns publish period, publisher -> subscriber, direct) ==
json metric key: `Topic_cross_process_oneway_paced_stream` (identity used by the CI gate: `Topic_cross_process_oneway_paced_stream@1024B`)
payload Probe1k = 1024 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7374 <-> pid 7389
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
one-way (assumes TSC coherence) 200000 366.1 535.6 926.9 5036.2 10166.2 23008.8 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=14.3 ns window=2.000 s
measured backlog: 0.05 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=394 invol=16
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3600MHz max=3603MHz mean=3600MHz spread=0.09% (n=394)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3595MHz max=3606MHz mean=3600MHz spread=0.31% (n=394)
frequency-normalised `one-way (assumes TSC coherence)`: median 1928 core cycles, p99.9 18130 core cycles at f_core=3.60 GHz
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +1.3 ns (add to correct).
== cross-process unpaced stream -- QUEUE DELAY / BACKPRESSURE, NOT LATENCY (publisher unthrottled; each sample includes its wait in the ring) ==
!! Every number in this block is a PUBLISH-TO-RECEIVE DELAY that INCLUDES
!! time spent waiting in the ring. It is a backpressure/throughput figure.
!! Do not quote it as an IPC latency; that is the exact mistake the retired
!! `Topic_cross_process` metric shipped for months. Read `measured backlog`
!! below: ~1 message means the subscriber kept up and no queue formed.
json metric key: `Topic_cross_process_unpaced_queue_delay` (identity used by the CI gate: `Topic_cross_process_unpaced_queue_delay@64B`)
payload Probe64 = 64 B (POD=true) | ring capacity 256 slots | topology 1P/1C | cpu3 <-> cpu1 | pid 7374 <-> pid 7392
------------------------------------------------------------------------------------------------------------------------------------------
metric (ns) n min median p99 p99.9 p99.99 max | n>p99.9 n>p99.99
------------------------------------------------------------------------------------------------------------------------------------------
publish->receive incl. queue 200000 138.3 275.6 409.5 8354.2 10498.4 68830.9 | 200 20
------------------------------------------------------------------------------------------------------------------------------------------
samples=200000 warmup=20000 transferred=200000 lost=0 invalid(TSC-skew)=0 poll cost=38.3 ns window=0.037 s
measured backlog: 1.49 messages deep on the median sample (ring holds 256); delivered 100.0% of transferred
measured window: minor_faults=0 major_faults=0 ctx_switches vol=8 invol=0
core freq cpu3 (measured window only): start=3600MHz end=3600MHz min=3600MHz max=3600MHz mean=3600MHz spread=0.00% (n=8)
core freq cpu1 (measured window only): start=3600MHz end=3600MHz min=3600MHz max=3600MHz mean=3600MHz spread=0.01% (n=8)
frequency-normalised `publish->receive incl. queue`: median 992 core cycles, p99.9 30075 core cycles at f_core=3.60 GHz
this row assumes cross-core TSC coherence; theta_hat measured in the ping-pong at this size = +8.7 ns (add to correct).
SUMMARY — one row per published JSON metric (nanoseconds, wall clock)
------------------------------------------------------------------------------------------------------------------------------------------
json metric key what it measures n median p99 p99.9 p99.99 max
------------------------------------------------------------------------------------------------------------------------------------------
Topic_cross_process_oneway_pingpong@64B ONE-WAY latency 1000000 308.2 403.5 1394.7 7334.7 24523.1
Topic_cross_process_oneway_pingpong@1024B ONE-WAY latency 1000000 544.3 647.3 3012.4 7852.9 27925.4
Topic_cross_process_oneway_paced_stream@64B ONE-WAY latency 200000 271.3 338.2 2152.9 9861.1 60975.8
Topic_cross_process_oneway_paced_stream@1024B ONE-WAY latency 200000 535.6 926.9 5036.2 10166.2 23008.8
Topic_cross_process_unpaced_queue_delay@64B QUEUE DELAY (not latency) 200000 275.6 409.5 8354.2 10498.4 68830.9
------------------------------------------------------------------------------------------------------------------------------------------
Instrumentation is INCLUDED in every figure and never subtracted.
`Topic_cross_process_oneway_pingpong` is the true one-way figure: it assumes
nothing about cross-core TSC synchronisation and excludes the responder's turnaround.
`RTT halved (one-way)` in the per-scenario tables is a ROUND TRIP DIVIDED BY TWO; it is
the row to compare against any competitor publishing a halved ping-pong.
`Topic_cross_process_unpaced_queue_delay` is a BACKPRESSURE figure: it includes
however long the message waited in the ring, so it scales with the achieved backlog
(printed in its scenario block) and is NOT a latency.
A figure without its payload size, ring depth and topology is not comparable to anything.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! THE SAME WARNINGS AGAIN — they apply to every number above
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! DVFS is live: scaling_min_freq (800MHz) != scaling_max_freq (2300MHz). Pin them equal before quoting any absolute figure.
!! No isolcpus= on the kernel command line: the benchmark cores are shared with everything else on this box. The involuntary context-switch count in each scenario block is the mechanism behind the far tail.
!! pingpong (64 B): 3 MINOR PAGE FAULTS inside the measured window — a pre-fault was missed, and those faults are in the tail figures.
!! pingpong (1024 B): 1 MINOR PAGE FAULTS inside the measured window — a pre-fault was missed, and those faults are in the tail figures.
!! paced_stream (64 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! paced_stream (1024 B) `one-way (assumes TSC coherence)`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!! unpaced_queue (64 B): measured backlog is only 1.49 messages — the subscriber kept up, so the ring never filled and `Topic_cross_process_unpaced_queue_delay` is NOT a backpressure figure on this run. It is a publish-to-receive delay with no queueing in it, and it is still not a latency (see the paced and ping-pong rows for that).
!! unpaced_queue (64 B) `publish->receive incl. queue`: p99.99 rests on 20 observations. Below ~100 exceedances it is a max wearing a percentile's name; +/-5% needs ~400 (--iterations 4000000).
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Gate report written to /home/runner/work/horus/horus/benchmarks/gate-reports/cross_process-3.json
keys: Topic_cross_process_oneway_pingpong@64B, Topic_cross_process_oneway_pingpong@1024B, Topic_cross_process_oneway_paced_stream@64B, Topic_cross_process_oneway_paced_stream@1024B, Topic_cross_process_unpaced_queue_delay@64B
NOTE: `Topic_cross_process` is retired. A baseline window that still holds it
will report it as missing and fail the gate until the window is refreshed —
that is intended, and is why the key was not silently reused.
Raw samples are omitted (pass --raw-in-json, or --raw-dir for a lossless cycle dump).
8 warning(s) apply to these numbers and are NOT in this file; use --detail-json.
╔══════════════════════════════════════════════════════════════════╗
║ HORUS Robotics Message Types Benchmark ║
╠══════════════════════════════════════════════════════════════════╣
║ Testing real-world robotics message latency (REP 2014) ║
╚══════════════════════════════════════════════════════════════════╝
Platform: INTEL(R) XEON(R) PLATINUM 8573C (4 cores)
Iterations: 50000
Warmup: 5000
╔═════════════════════════════════════════════════════════════════════════════╗
║ Message Type │ Size (bytes) │ Typical Rate │ Use Case ║
╠═════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 1000+ Hz │ Velocity control commands ║
║ Imu │ 296 │ 500+ Hz │ IMU sensor fusion ║
║ LaserScan │ 1480 │ 10-40 Hz │ 2D lidar navigation ║
║ JointCommand │ 1032 │ 500+ Hz │ Manipulator control ║
╚═════════════════════════════════════════════════════════════════════════════╝
[Topic] Running benchmarks — timing tx.send() enqueue cost only
(cross-thread consumer drains in the background; no sample
waits for delivery, so these are NOT end-to-end latencies)
─────────────────────────────────────────────────
CmdVel 16 bytes │ median: 23ns │ p99: 61ns │ CV: 1.5174
Imu 304 bytes │ median: 35ns │ p99: 66ns │ CV: 1.4158
LaserScan 1480 bytes │ median: 55ns │ p99: 95ns │ CV: 1.0354
JointCommand 928 bytes │ median: 57ns │ p99: 100ns │ CV: 0.9971
╔══════════════════════════════════════════════════════════════════════════════════════════╗
║ SUMMARY BY MESSAGE TYPE — send() ENQUEUE COST, not end-to-end latency ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ Message │ Size │ Median (ns) │ p99 (ns) │ CV │ Misses │ Sustained rate ║
╠══════════════════════════════════════════════════════════════════════════════════════════╣
║ CmdVel │ 16 │ 23 │ 61 │ 1.5174 │ 0 │ 8.50 M msg/s ║
║ Imu │ 304 │ 35 │ 66 │ 1.4158 │ 0 │ 6.07 M msg/s ║
║ LaserScan │ 1480 │ 55 │ 95 │ 1.0354 │ 0 │ 1.90 M msg/s ║
║ JointCommand │ 928 │ 57 │ 100 │ 0.9971 │ 0 │ 4.53 M msg/s ║
╚══════════════════════════════════════════════════════════════════════════════════════════╝
Sustained rate is wall-clock over the measured loop, so it includes the batch
drain-waits. It is a publish rate for this producer/consumer pair, not a bus limit.
╔══════════════════════════════════════════════════════════════════╗
║ REAL-TIME SUITABILITY ANALYSIS ║
╠══════════════════════════════════════════════════════════════════╣
║ These gates grade the p99 of the send() ENQUEUE COST against one ║
║ whole control period. They are not end-to-end control-loop ║
║ guarantees: a real loop also pays the receive side, the node's ║
║ compute, and the scheduler's jitter, none of which is measured ║
║ here. The headroom is printed so a pass by four orders of ║
║ magnitude cannot be read as a tight result. ║
╠══════════════════════════════════════════════════════════════════╣
║ CmdVel (1kHz control): ~ PASS by 16393x — gate not discriminating (p99=61ns vs 1000000ns)
║ CmdVel (10kHz control): ~ PASS by 1639x — gate not discriminating (p99=61ns vs 100000ns)
║ Imu (500Hz fusion) : ~ PASS by 30303x — gate not discriminating (p99=66ns vs 2000000ns)
║ LaserScan (40Hz lidar) : ~ PASS by 263158x — gate not discriminating (p99=95ns vs 25000000ns)
║ JointCmd (500Hz control): ~ PASS by 20000x — gate not discriminating (p99=100ns vs 2000000ns)
╚══════════════════════════════════════════════════════════════════╝
Results written to: /home/runner/work/horus/horus/benchmarks/gate-reports/robotics-3.json
Gated against a rolling window of recent main runs. Each metric's band is max(relative, center + k·sigma_observed, absolute floor), so it is derived from the noise the baseline itself measured — where a band is printed wider than the change you care about, this run did not detect a regression and could not have. Thresholds and their reasoning: RegressionPolicy in benchmarks/src/output.rs. Commit: bae6fdc
run_doctor discovers the project by searching upward and passes that context to the other checks, but this reloads horus.toml relative to the current directory. From a project subdirectory with no local manifest, it falls back to Default, so a parent manifest with enable = ["net"] is reported as networking-disabled. Pass the detected manifest/root into check_network (and preserve its parse error) instead of resolving this path from cwd.
Run command ignores preferred hardware entries
horus_manager/src/manifest.rs:1282
This centralizes [hardware]/legacy [drivers] only for the manager generators and doctor, but commands/run/features.rs:125-136 still iterates manifest.drivers directly. Consequently horus run ignores registry-required Cargo features for a driver declared with the preferred [hardware] spelling, so the two manifest spellings still behave differently depending on the command. Route that consumer through hardware_entries() as well.
The stack caught the interaction between #205's internal-docs feature —
which makes rustdoc resolve links inside the previously hidden modules —
and #189's recv_failures doc, which linked crate::testing::MockTopic.
That module is behind cfg(any(test, feature = "test-utils")), so the link
is unresolved in a normal doc build. Plain text now.
Verified: RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps
--exclude horus_manager, and cargo doc -p horus_core --no-deps --features
internal-docs, both pass.
Doctor resolves manifest from the current directory
horus_manager/src/commands/doctor.rs:1880
This reloads horus.toml relative to the current working directory and silently falls back to an empty manifest. run_doctor() already resolves a ProjectContext (including its project root/manifest), so invoking horus doctor from a project subdirectory can now report networking as disabled even when the project root enables net (and can read an unrelated cwd manifest). Pass the resolved context or manifest into this check instead.
horus run still has a hardware-config consumer that bypasses this helper: commands/run/features.rs::DriverConfig::from_manifest iterates manifest.drivers directly. Consequently a driver declared only under the preferred [hardware] table is still omitted from the run-time feature/driver configuration even though cargo generation, syncing, and doctor now use hardware_entries(). Route that consumer through the merged map as well.
honor_restart_request_with discarded init()'s Result and treated only a
panic as failure, so a driver whose device failed to reopen was marked
initialised and kept ticking while `horus node restart` reported success.
reinit_pending_nodes already refuses exactly this shape for a node still
waiting on its first init; the executor path now matches: Err clears
initialized, stops the node, emits a diagnostic naming the error, and
transitions the node's context to crashed.
Found by Copilot's review of the stack (three threads, one per call site:
control.rs doc, primitives.rs implementation, types.rs field doc).
control_restart.rs: the unknown-node test's fixed 400 ms wait becomes the
bounded poll the test above it uses, and
restart_whose_init_fails_stops_the_node pins the new behaviour.
messages_sent counted only send() and messages_received only recv(), so a
topic driven through try_send/send_blocking/try_recv read 0 while MockTopic
— whose send() delegates to its try_send — counted the same traffic. That
is the 'double that reports what the real transport does not' defect these
counters exist to prevent, one API surface over.
- Sent: send, try_send and send_blocking (the pool-backed try_send paths
included) each count one attempt; a send_blocking NoBackpressure refusal
that never reaches the ring does not.
- Received: try_recv is the funnel for every non-fast-path delivery, so it
counts there; the role=Both POD fast path counts its own; recv() and
recv_with_content_logging() no longer double count.
- horus_py's auto-detect message now sends generated types to
horus._horus, where they actually live (they are not re-exported from
horus).
- New parity test runs the same sequence through a real Topic and a
MockTopic and asserts equal counters.
Found by Copilot's review of the stack.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Every open PR in one branch, so the queue drains in one CI cycle instead of seven.
Why a stack again
mainrequires seven contexts withstrict: true, so each merge invalidates everyother open PR and the queue drains at one PR per full CI cycle — and every push
re-triggers a Copilot review. This repository has drained the queue this way
before (#119, #146, #174, #204); this is the same mechanism.
Contents
release.yml)The one conflict
One, out of seven branches.
horus_core/src/drivers/mod.rs: #205 routedHORUS_SIM_MODEthrough the newhorus_sys::envvocabulary, while #189 moved thepredicate into the free function
sim_mode_enabled(value)so its truth table istestable without racing the process-global environment. Resolved by keeping #189's
structure and giving it #205's vocabulary:
sim_mode_enabledis nowvalue.and_then(horus_sys::env::parse_flag).unwrap_or(false),so
0/false/no/off/""all disable simulation,1/true/yes/onenable it, and anunrecognised value (
=maybe) leaves the default — OFF — in force.sim_mode_teststruth table was extended to the shared vocabulary and nowasserts agreement with
parse_flagdirectly, which is the same vocabularyHORUS_NET_ENABLEDreads in the scheduler.Benchmark gate: #189 was blocked, and why
Run Benchmarksfailed on #189 atf5499e6cwith a blocking median regression:Topic_cross_process_oneway_pingpong@1024B278 ns → 310 ns (+11.5%), and@64B150 ns → 165 ns (+10%). Empty-poll cost was unchanged between the tworuns (12.7/40.5 ns baseline vs 13.9/39.3 ns PR), so the cost was on the success
path.
The only per-message additions on that path were two
Relaxedfetch_adds —messages_sentonsend(),messages_receivedonrecv()— added by thisbranch's "topic metrics count real traffic" fix. An A/B on the integration tree
(1M samples × 6 pairs, interleaved, performance governor, turbo off) removing
them recovers −18 ns per round trip at 1024 B and −20 ns at 64 B; the merged
tree with them shows +3.6%/+13.0% over the same tree without them. An
uncontended
lock xaddon every send and recv is a few nanoseconds of latencyadded to the middle of the publish path, and it is a full barrier in a pipeline
that is otherwise measuring a ring-buffer write.
The fix, in this stack:
messages_sent/messages_receivedmove from theArc-sharedMigrationMetrics(anAtomicU64per message, required there because aClonecan run on another thread) ontoRingTopicitself, asCell<u64>s.RingTopicis deliberately!Sync(see its SAFETY note), so aCellis sound, and a clone gets its own — the counters are per-handle andthe publish path pays one
addinstead of alock xadd.Topicreports the traffic it carried, exactly likeMockTopic— the double users write their tests against, which was the PR'sown stated bar.
TopicMetricsdocuments this.MigrationMetricscounters that genuinely aggregate across handles(
send_failures,send_retry_overruns,migrations) stay atomic where theyare; only the two per-message traffic counters moved.
Fixes #197 (the stale shared-memory leak #198 hardens CI against).
Review fixes on top of the seven PRs
Copilot's review of this stack raised eight threads. Six were addressed in code,
two were answered with the pinned behaviour they conflict with:
fix(scheduling): a restart whose init() fails stops the node(threethreads) —
honor_restart_request_withdiscardedinit()'sResultandtreated only a panic as failure, so a driver whose device failed to reopen
was marked initialised and kept ticking while
horus node restartreportedsuccess.
Errnow clearsinitialized, stops the node, reports the error andtransitions the context to crashed — the rule
reinit_pending_nodesalreadyapplies. New test
restart_whose_init_fails_stops_the_node; theunknown-node test's fixed 400 ms wait becomes a bounded poll.
fix(topic): the metrics counters move on every public send/recv API(two threads) —
messages_sentcounted onlysend()andmessages_receivedonlyrecv(), sotry_send/send_blocking/try_recvtraffic read 0 while
MockTopic, whosesend()delegates to itstry_send, counted it. Now every public API counts, the role=Both fast pathowns its count, and the new
counters_move_on_every_public_send_and_recv_apiasserts a real topic and a mock report equal numbers.
horus_pyauto-detect message — generated types are not re-exported fromhorus; the suggested import now points athorus._horusfor them.uninstall.sh'sHORUS_PREFIXremoval isdeliberate and pinned by
the_prefix_cleanup_leaves_similarly_named_variables_alone, whichconstructs exactly the case raised.
Maintenance commits on top
fix(deps): RUSTSEC-2026-0285— RustSec published the advisory againstrustls 0.23.43on 2026-09-14, after main's last audit run, so every new PRagainst
mainfailsSecurity Audittoday (it is strict for PRs targetingmain). Patch update to 0.23.45 (+
rustls-webpki0.103.15);cargo auditisclean again. No source changes.
fix(topic): the traffic counters live on RingTopic, not LocalState— thebenchmark fix below, after
local_state_fits_four_cache_linescaught thefirst cut.
docs(topic): MockTopic is cfg-gated, so the rustdoc link cannot resolve—the stack caught what the individual PRs could not: docs/ci: make the codebase readable and its documentation checkable #205 turns on rustdoc link
resolution inside the previously hidden modules, and Fix 31 runtime/documentation divergences found by a bidirectional audit #189's
recv_failuresdoc linked
crate::testing::MockTopic, which only exists undertest-utils. Plain text now; bothcargo doccommands CI runs pass with-D warnings.One CI gap this stack closes
#215's regression test is
#[cfg(target_os = "macos")]underhorus_core, andno macOS job in this repository builds horus_core's test targets: the macOS
legs build
horus_types/horus_macros/horus_cpp_macrosandhorus_sys, andthe manager shell contracts. So the regression was compiled by nothing and run
by nothing. The last commit adds a step to
parity-macosthat runs exactlythat test, so the fix is verified and not merely compiled.
Validation on the stack
cargo fmt --all --checkcargo check --workspace --all-targets --exclude horus_pyandcargo check -p horus_py --all-targets --no-default-features(the twocommands CI runs)
cargo test -p horus_core --lib— 2623 passed. It caught a real defect inthe first cut of the benchmark fix: the two counters were added to
LocalState, whose 256-byte (four cache line) budget is asserted bylocal_state_fits_four_cache_lines. The counters live onRingTopicnow(also per-handle, also non-atomic), and the layout test is untouched. One
scheduler timing test failed in the parallel run under benchmark load and
passes 3/3 in isolation (
wake_comes_from_the_publisher_not_from_the_backstop— its own comments describe host-load sensitivity).
cargo audit— clean after therustlspatch bump.cargo clippy -p horus_core --all-targets -- -D warnings— clean.cargo test -p horus_sys --lib, workspace doctests (including the newinternal-docsdocumentation gate from docs/ci: make the codebase readable and its documentation checkable #205)Full CI is the arbiter for everything else.