Skip to content

Autopilot: recurring unexplained CI failure on main #197

Closed

Description

@github-actions

Autopilot triage — 25 failed run(s) on main

Unexplained, single occurrence (watching, not reporting)

Recurring unexplained failures — these need attention

Integration Tests — failed 2 times

log tail
2026-09-02T22:13:21.6507480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-02T22:13:21.6636031Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-02T22:13:21.6764480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-02T22:13:21.6894010Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-02T22:13:21.7022552Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-02T22:13:21.7159254Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-02T22:13:22.7297490Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-02T22:13:38.0814144Z [multithread] allocs: 20931647, failures: 0, releases: 20931641, corruptions: 0
2026-09-02T22:15:51.0106105Z thread 'test_slot_reuse_after_release' (104204) panicked at horus_core/tests/tensor_pool_concurrent.rs:145:62:
2026-09-02T22:15:51.0118950Z test test_slot_reuse_after_release ... FAILED
2026-09-02T22:15:51.0119578Z failures:
2026-09-02T22:15:51.0119884Z failures:
2026-09-02T22:15:51.0120986Z test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
2026-09-02T22:15:51.0159639Z ##[error]Process completed with exit code 101.

Multi-Platform — failed 6 times

log tail
2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
2026-09-01T21:21:44.3281997Z failures:
2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
2026-09-01T21:21:44.3285571Z failures:
2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.

Docs Contract — failed 11 times

log tail
2026-09-01T09:15:07.2539488Z   content/docs/troubleshooting.mdx:939  src/lib.rs:298:29: error[E0609]: no field 'topic' on type '&mut docblock_578::_DocSelf': unknown field
2026-09-01T09:15:07.2540508Z   content/docs/troubleshooting.mdx:950  src/lib.rs:313:20: error[E0609]: no field 'topic' on type '&mut docblock_579::_DocSelf': unknown field
2026-09-01T09:15:07.2541511Z   content/docs/troubleshooting.mdx:971  src/lib.rs:344:13: error[E0609]: no field 'sensor' on type '&mut docblock_581::_DocSelf': unknown field
2026-09-01T09:15:07.2542509Z   content/docs/troubleshooting.mdx:982  src/lib.rs:360:21: error[E0609]: no field 'sensor' on type '&mut docblock_582::_DocSelf': unknown field
2026-09-01T09:15:07.2543584Z   content/docs/troubleshooting.mdx:992  src/lib.rs:373:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
2026-09-01T09:15:07.2544812Z   content/docs/troubleshooting.mdx:1052  src/lib.rs:386:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
2026-09-01T09:15:07.2548898Z thread 'documented_rust_examples_compile' (9229) panicked at horus_manager/tests/docs_examples.rs:1241:9:
2026-09-01T09:15:07.2550430Z       src/lib.rs:203:16: error[E0061]: this function takes 4 arguments but 3 arguments were supplied
2026-09-01T09:15:07.2551475Z       src/lib.rs:203:16: error[E0277]: the '?' operator can only be applied to values that implement 'Try': the '?' operator cannot be applied to type 'horus::types::OccupancyGrid'
2026-09-01T09:15:07.2561017Z test documented_rust_examples_compile ... FAILED
2026-09-01T09:15:07.2561315Z failures:
2026-09-01T09:15:07.2561506Z failures:
2026-09-01T09:15:07.2562163Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 44.80s
2026-09-01T09:15:07.2573183Z ##[error]Process completed with exit code 101.

Feature Matrix — failed 3 times

log tail
2026-08-24T03:47:14.1689955Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
2026-08-24T03:47:14.1691672Z 5030 |                     severity: crate::error::Severity::Fatal,
2026-08-24T03:47:14.1888024Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
2026-08-24T03:47:14.1889341Z 592 | ...                   severity: crate::error::Severity::Fatal,
2026-08-24T03:47:14.2028550Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
2026-08-24T03:47:14.2030352Z 389 | ...                   severity: crate::error::Severity::Fatal,
2026-08-24T03:47:14.2056749Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
2026-08-24T03:47:14.2058187Z 280 | ...                   severity: crate::error::Severity::Fatal,
2026-08-24T03:47:14.2091563Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
2026-08-24T03:47:14.2092779Z 344 | ...                   severity: crate::error::Severity::Fatal,
2026-08-24T03:47:15.1737143Z ##[error]Process completed with exit code 101.

Before changing any code

Most red CI in this repo is environmental. The triage above already
filtered the failures it could explain, but confirm the remainder is
real before touching source:

  • Timing, throughput and tick-rate assertions are only meaningful with
    --release. In debug they fail for no reason.
  • --test-threads=1 is required; the tests share one SHM namespace.
  • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
    tests fail in a rotating pattern.
  • Without --no-fail-fast, cargo stops at the first failing binary and
    you see one failure per run.
  • A bare exit code 1 with no output is usually the /tmp per-user
    quota, not your change. df looks fine because it is a quota.
  • Detection of the concurrency bugs here is probabilistic. Passing once
    is not evidence of a fix.

The rule that matters most

Do not claim a fix without an ablation: show the test FAILS without the
change and passes with it. A test that passes both ways proves nothing,
and that is the most common way work here goes wrong. If you cannot
construct a failing case, say so and leave the finding documented
rather than shipping a change you cannot demonstrate.

For anything about ordering or interleaving, prefer a loom model
(horus_core/tests/loom_*.rs). It explores exhaustively and models weak
memory ordering, which x86 cannot exhibit. Every loom model here
documents the mutation that turns it red — yours must too.

Scope

Fix only what this issue reports. If you find something else, say so
separately rather than widening the change. Do not add scratch or report
.md files to this repo; it is public. Do not add co-authorship
trailers. Explain the mechanism in the commit message, with the numbers
you actually measured.

Activity

  1. linked a pull request that will close this issueHarden CI against stale shared-memory state #198on Sep 5, 2026
  2. github-actions commented on Sep 6, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 12 times

    log tail
    2026-09-06T08:26:42.6302944Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-06T08:26:42.6304026Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-06T08:26:42.6305104Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-06T08:26:42.6306176Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-06T08:26:42.6307312Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-06T08:26:42.6308839Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-06T08:26:42.6312973Z thread 'documented_rust_examples_compile' (9145) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-06T08:26:42.6314632Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-06T08:26:42.6315631Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-06T08:26:42.6325420Z test documented_rust_examples_compile ... FAILED
    2026-09-06T08:26:42.6325716Z failures:
    2026-09-06T08:26:42.6325896Z failures:
    2026-09-06T08:26:42.6326538Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 43.02s
    2026-09-06T08:26:42.6339141Z ##[error]Process completed with exit code 101.
    

    Integration Tests — failed 2 times

    log tail
    2026-09-02T22:13:21.6507480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6636031Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6764480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6894010Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7022552Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7159254Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:22.7297490Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:38.0814144Z [multithread] allocs: 20931647, failures: 0, releases: 20931641, corruptions: 0
    2026-09-02T22:15:51.0106105Z thread 'test_slot_reuse_after_release' (104204) panicked at horus_core/tests/tensor_pool_concurrent.rs:145:62:
    2026-09-02T22:15:51.0118950Z test test_slot_reuse_after_release ... FAILED
    2026-09-02T22:15:51.0119578Z failures:
    2026-09-02T22:15:51.0119884Z failures:
    2026-09-02T22:15:51.0120986Z test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
    2026-09-02T22:15:51.0159639Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 6 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Feature Matrix — failed 2 times

    log tail
    2026-08-24T03:47:14.1689955Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.1691672Z 5030 |                     severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.1888024Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.1889341Z 592 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2028550Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2030352Z 389 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2056749Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2058187Z 280 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2091563Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2092779Z 344 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:15.1737143Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  3. github-actions commented on Sep 7, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 12 times

    log tail
    2026-09-07T09:00:08.6433761Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-07T09:00:08.6434852Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-07T09:00:08.6435955Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-07T09:00:08.6437405Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-07T09:00:08.6438582Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-07T09:00:08.6439900Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-07T09:00:08.6444069Z thread 'documented_rust_examples_compile' (9167) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-07T09:00:08.6445727Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-07T09:00:08.6447120Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-07T09:00:08.6456735Z test documented_rust_examples_compile ... FAILED
    2026-09-07T09:00:08.6457025Z failures:
    2026-09-07T09:00:08.6457211Z failures:
    2026-09-07T09:00:08.6457852Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 40.89s
    2026-09-07T09:00:08.6469216Z ##[error]Process completed with exit code 101.
    

    Integration Tests — failed 2 times

    log tail
    2026-09-02T22:13:21.6507480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6636031Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6764480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6894010Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7022552Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7159254Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:22.7297490Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:38.0814144Z [multithread] allocs: 20931647, failures: 0, releases: 20931641, corruptions: 0
    2026-09-02T22:15:51.0106105Z thread 'test_slot_reuse_after_release' (104204) panicked at horus_core/tests/tensor_pool_concurrent.rs:145:62:
    2026-09-02T22:15:51.0118950Z test test_slot_reuse_after_release ... FAILED
    2026-09-02T22:15:51.0119578Z failures:
    2026-09-02T22:15:51.0119884Z failures:
    2026-09-02T22:15:51.0120986Z test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
    2026-09-02T22:15:51.0159639Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 6 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Feature Matrix — failed 2 times

    log tail
    2026-08-24T03:47:14.1689955Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.1691672Z 5030 |                     severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.1888024Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.1889341Z 592 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2028550Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2030352Z 389 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2056749Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2058187Z 280 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2091563Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2092779Z 344 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:15.1737143Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  4. github-actions commented on Sep 8, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 13 times

    log tail
    2026-09-08T08:41:37.0464375Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-08T08:41:37.0465476Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-08T08:41:37.0466565Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-08T08:41:37.0467641Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-08T08:41:37.0468779Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-08T08:41:37.0470066Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-08T08:41:37.0474286Z thread 'documented_rust_examples_compile' (9222) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-08T08:41:37.0475998Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-08T08:41:37.0477010Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-08T08:41:37.0486393Z test documented_rust_examples_compile ... FAILED
    2026-09-08T08:41:37.0486675Z failures:
    2026-09-08T08:41:37.0486859Z failures:
    2026-09-08T08:41:37.0487488Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 43.72s
    2026-09-08T08:41:37.0500072Z ##[error]Process completed with exit code 101.
    

    Integration Tests — failed 2 times

    log tail
    2026-09-02T22:13:21.6507480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6636031Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6764480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6894010Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7022552Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7159254Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:22.7297490Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:38.0814144Z [multithread] allocs: 20931647, failures: 0, releases: 20931641, corruptions: 0
    2026-09-02T22:15:51.0106105Z thread 'test_slot_reuse_after_release' (104204) panicked at horus_core/tests/tensor_pool_concurrent.rs:145:62:
    2026-09-02T22:15:51.0118950Z test test_slot_reuse_after_release ... FAILED
    2026-09-02T22:15:51.0119578Z failures:
    2026-09-02T22:15:51.0119884Z failures:
    2026-09-02T22:15:51.0120986Z test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
    2026-09-02T22:15:51.0159639Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 5 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Feature Matrix — failed 2 times

    log tail
    2026-08-24T03:47:14.1689955Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.1691672Z 5030 |                     severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.1888024Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.1889341Z 592 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2028550Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2030352Z 389 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2056749Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2058187Z 280 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2091563Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2092779Z 344 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:15.1737143Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  5. github-actions commented on Sep 9, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 13 times

    log tail
    2026-09-09T08:41:55.2058253Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-09T08:41:55.2059347Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-09T08:41:55.2060436Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-09T08:41:55.2061531Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-09T08:41:55.2062689Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-09T08:41:55.2063994Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-09T08:41:55.2068129Z thread 'documented_rust_examples_compile' (5303) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-09T08:41:55.2069796Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-09T08:41:55.2070802Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-09T08:41:55.2080652Z test documented_rust_examples_compile ... FAILED
    2026-09-09T08:41:55.2080943Z failures:
    2026-09-09T08:41:55.2081126Z failures:
    2026-09-09T08:41:55.2081770Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 44.14s
    2026-09-09T08:41:55.2092518Z ##[error]Process completed with exit code 101.
    

    Integration Tests — failed 2 times

    log tail
    2026-09-02T22:13:21.6507480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6636031Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6764480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6894010Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7022552Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7159254Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:22.7297490Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:38.0814144Z [multithread] allocs: 20931647, failures: 0, releases: 20931641, corruptions: 0
    2026-09-02T22:15:51.0106105Z thread 'test_slot_reuse_after_release' (104204) panicked at horus_core/tests/tensor_pool_concurrent.rs:145:62:
    2026-09-02T22:15:51.0118950Z test test_slot_reuse_after_release ... FAILED
    2026-09-02T22:15:51.0119578Z failures:
    2026-09-02T22:15:51.0119884Z failures:
    2026-09-02T22:15:51.0120986Z test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
    2026-09-02T22:15:51.0159639Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 5 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Feature Matrix — failed 2 times

    log tail
    2026-08-24T03:47:14.1689955Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.1691672Z 5030 |                     severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.1888024Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.1889341Z 592 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2028550Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2030352Z 389 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2056749Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2058187Z 280 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:14.2091563Z error[E0559]: variant 'BlackBoxEvent::NodeError' has no field named 'severity'
    2026-08-24T03:47:14.2092779Z 344 | ...                   severity: crate::error::Severity::Fatal,
    2026-08-24T03:47:15.1737143Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  6. github-actions commented on Sep 10, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 14 times

    log tail
    2026-09-10T08:43:30.7532920Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-10T08:43:30.7534124Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-10T08:43:30.7535229Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-10T08:43:30.7536314Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-10T08:43:30.7537473Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-10T08:43:30.7538823Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-10T08:43:30.7542798Z thread 'documented_rust_examples_compile' (5288) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-10T08:43:30.7544758Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-10T08:43:30.7545769Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-10T08:43:30.7555277Z test documented_rust_examples_compile ... FAILED
    2026-09-10T08:43:30.7555575Z failures:
    2026-09-10T08:43:30.7555760Z failures:
    2026-09-10T08:43:30.7556412Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 40.34s
    2026-09-10T08:43:30.7566610Z ##[error]Process completed with exit code 101.
    

    Integration Tests — failed 2 times

    log tail
    2026-09-02T22:13:21.6507480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6636031Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6764480Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.6894010Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7022552Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:21.7159254Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:22.7297490Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-02T22:13:38.0814144Z [multithread] allocs: 20931647, failures: 0, releases: 20931641, corruptions: 0
    2026-09-02T22:15:51.0106105Z thread 'test_slot_reuse_after_release' (104204) panicked at horus_core/tests/tensor_pool_concurrent.rs:145:62:
    2026-09-02T22:15:51.0118950Z test test_slot_reuse_after_release ... FAILED
    2026-09-02T22:15:51.0119578Z failures:
    2026-09-02T22:15:51.0119884Z failures:
    2026-09-02T22:15:51.0120986Z test result: FAILED. 4 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.14s
    2026-09-02T22:15:51.0159639Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 5 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  7. github-actions commented on Sep 11, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 15 times

    log tail
    2026-09-11T08:38:02.9365163Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-11T08:38:02.9366262Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-11T08:38:02.9367346Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-11T08:38:02.9368429Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-11T08:38:02.9369863Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-11T08:38:02.9371172Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-11T08:38:02.9375149Z thread 'documented_rust_examples_compile' (5203) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-11T08:38:02.9376818Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-11T08:38:02.9377808Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-11T08:38:02.9387493Z test documented_rust_examples_compile ... FAILED
    2026-09-11T08:38:02.9387792Z failures:
    2026-09-11T08:38:02.9387977Z failures:
    2026-09-11T08:38:02.9388623Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 43.77s
    2026-09-11T08:38:02.9399634Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 5 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  8. github-actions commented on Sep 12, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 16 times

    log tail
    2026-09-12T08:25:18.0501323Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-12T08:25:18.0502125Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-12T08:25:18.0502925Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-12T08:25:18.0503720Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-12T08:25:18.0504567Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-12T08:25:18.0505611Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-12T08:25:18.0508588Z thread 'documented_rust_examples_compile' (4994) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-12T08:25:18.0509891Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-12T08:25:18.0510665Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-12T08:25:18.0517982Z test documented_rust_examples_compile ... FAILED
    2026-09-12T08:25:18.0518207Z failures:
    2026-09-12T08:25:18.0518359Z failures:
    2026-09-12T08:25:18.0518863Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 35.69s
    2026-09-12T08:25:18.0528339Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 5 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  9. github-actions commented on Sep 13, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 16 times

    log tail
    2026-09-13T08:59:44.2112865Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-13T08:59:44.2113896Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-13T08:59:44.2114917Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-13T08:59:44.2115945Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-13T08:59:44.2117029Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-13T08:59:44.2118265Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-13T08:59:44.2122278Z thread 'documented_rust_examples_compile' (5261) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-13T08:59:44.2123981Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-13T08:59:44.2124954Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-13T08:59:44.2134668Z test documented_rust_examples_compile ... FAILED
    2026-09-13T08:59:44.2134967Z failures:
    2026-09-13T08:59:44.2135161Z failures:
    2026-09-13T08:59:44.2135810Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 45.21s
    2026-09-13T08:59:44.2146356Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 5 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  10. github-actions commented on Sep 14, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 17 times

    log tail
    2026-09-14T09:39:04.1631658Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-14T09:39:04.1632741Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-14T09:39:04.1633824Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-14T09:39:04.1634905Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-14T09:39:04.1636051Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-14T09:39:04.1637564Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-14T09:39:04.1641557Z thread 'documented_rust_examples_compile' (9180) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-14T09:39:04.1643417Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-14T09:39:04.1644417Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-14T09:39:04.1654122Z test documented_rust_examples_compile ... FAILED
    2026-09-14T09:39:04.1654412Z failures:
    2026-09-14T09:39:04.1654591Z failures:
    2026-09-14T09:39:04.1655218Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 50.08s
    2026-09-14T09:39:04.1666864Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 4 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  11. github-actions commented on Sep 15, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 17 times

    log tail
    2026-09-15T09:17:02.1033709Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-15T09:17:02.1034781Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-15T09:17:02.1036166Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-15T09:17:02.1037261Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-15T09:17:02.1038397Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-15T09:17:02.1039692Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-15T09:17:02.1043663Z thread 'documented_rust_examples_compile' (9170) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-15T09:17:02.1045717Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-15T09:17:02.1046741Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-15T09:17:02.1056263Z test documented_rust_examples_compile ... FAILED
    2026-09-15T09:17:02.1056543Z failures:
    2026-09-15T09:17:02.1056719Z failures:
    2026-09-15T09:17:02.1057347Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 48.23s
    2026-09-15T09:17:02.1069602Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 4 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  12. github-actions commented on Sep 16, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 17 times

    log tail
    2026-09-16T09:07:21.7992317Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-16T09:07:21.7993168Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-16T09:07:21.7994017Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-16T09:07:21.7994854Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-16T09:07:21.7995746Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-16T09:07:21.7996739Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-16T09:07:21.7999923Z thread 'documented_rust_examples_compile' (5010) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-16T09:07:21.8001282Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-16T09:07:21.8002069Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-16T09:07:21.8009863Z test documented_rust_examples_compile ... FAILED
    2026-09-16T09:07:21.8010126Z failures:
    2026-09-16T09:07:21.8010304Z failures:
    2026-09-16T09:07:21.8010878Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 40.20s
    2026-09-16T09:07:21.8021302Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 4 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  13. github-actions commented on Sep 17, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 17 times

    log tail
    2026-09-17T09:16:52.1282757Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-17T09:16:52.1283853Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-17T09:16:52.1284954Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-17T09:16:52.1286045Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-17T09:16:52.1287205Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-17T09:16:52.1288641Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-17T09:16:52.1292699Z thread 'documented_rust_examples_compile' (5212) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-17T09:16:52.1294378Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-17T09:16:52.1295395Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-17T09:16:52.1305074Z test documented_rust_examples_compile ... FAILED
    2026-09-17T09:16:52.1305366Z failures:
    2026-09-17T09:16:52.1305549Z failures:
    2026-09-17T09:16:52.1306202Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 42.39s
    2026-09-17T09:16:52.1316698Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 4 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  14. github-actions commented on Sep 18, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 17 times

    log tail
    2026-09-18T08:49:16.8361013Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-18T08:49:16.8361842Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-18T08:49:16.8362653Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-18T08:49:16.8363459Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-18T08:49:16.8364322Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-18T08:49:16.8365294Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-18T08:49:16.8368304Z thread 'documented_rust_examples_compile' (4995) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-18T08:49:16.8369768Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-18T08:49:16.8370650Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-18T08:49:16.8377942Z test documented_rust_examples_compile ... FAILED
    2026-09-18T08:49:16.8378179Z failures:
    2026-09-18T08:49:16.8378335Z failures:
    2026-09-18T08:49:16.8379000Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 36.67s
    2026-09-18T08:49:16.8388339Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 4 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

  15. github-actions commented on Sep 19, 2026

    @github-actions
    ContributorAuthor

    Autopilot triage — 25 failed run(s) on main

    Unexplained, single occurrence (watching, not reporting)

    Recurring unexplained failures — these need attention

    Docs Contract — failed 17 times

    log tail
    2026-09-19T08:35:13.1678222Z   content/docs/troubleshooting.mdx:1176  src/lib.rs:1288:29: error[E0609]: no field 'topic' on type '&mut docblock_596::_DocSelf': unknown field
    2026-09-19T08:35:13.1678823Z   content/docs/troubleshooting.mdx:1187  src/lib.rs:1303:20: error[E0609]: no field 'topic' on type '&mut docblock_597::_DocSelf': unknown field
    2026-09-19T08:35:13.1679422Z   content/docs/troubleshooting.mdx:1208  src/lib.rs:1334:13: error[E0609]: no field 'sensor' on type '&mut docblock_599::_DocSelf': unknown field
    2026-09-19T08:35:13.1680153Z   content/docs/troubleshooting.mdx:1219  src/lib.rs:10:21: error[E0609]: no field 'sensor' on type '&mut docblock_600::_DocSelf': unknown field
    2026-09-19T08:35:13.1680775Z   content/docs/troubleshooting.mdx:1229  src/lib.rs:23:5: error[E0433]: cannot find module or crate 'thread' in this scope: use of unresolved module or unlinked crate 'thread'
    2026-09-19T08:35:13.1681474Z   content/docs/troubleshooting.mdx:1303  src/lib.rs:36:37: error[E0424]: expected value, found module 'self': 'self' value is a keyword only available in methods with a 'self' parameter
    2026-09-19T08:35:13.1683622Z thread 'documented_rust_examples_compile' (4975) panicked at horus_manager/tests/docs_examples.rs:1241:9:
    2026-09-19T08:35:13.1684643Z       src/lib.rs:181:43: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-19T08:35:13.1685190Z       src/lib.rs:182:41: error[E0277]: '?' couldn't convert the error to 'HorusError': the trait 'From<&str>' is not implemented for 'HorusError'
    2026-09-19T08:35:13.1690469Z test documented_rust_examples_compile ... FAILED
    2026-09-19T08:35:13.1690634Z failures:
    2026-09-19T08:35:13.1690740Z failures:
    2026-09-19T08:35:13.1691104Z test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 26 filtered out; finished in 34.99s
    2026-09-19T08:35:13.1699353Z ##[error]Process completed with exit code 101.
    

    Multi-Platform — failed 4 times

    log tail
    2026-09-01T21:21:43.1082262Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1929780Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1985939Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1986835Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.1991250Z [ERROR] [unknown] Fatal init: Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.1991532Z Failed to initialize node 'panic_init_node': Node error: Node 'panic_init_node' panicked during init
    2026-09-01T21:21:43.3214472Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.3813284Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:43.5842481Z Warning: Failed to set signal handler: Ctrl-C error: Ctrl-C signal handler already registered
    2026-09-01T21:21:44.3281997Z failures:
    2026-09-01T21:21:44.3282673Z thread 'communication::topic::tests::broadcast_subscriber_join_midstream' (9763) panicked at horus_core/src/communication/topic/tests.rs:7163:5:
    2026-09-01T21:21:44.3285571Z failures:
    2026-09-01T21:21:44.3286083Z test result: FAILED. 2460 passed; 1 failed; 4 ignored; 0 measured; 4 filtered out; finished in 72.72s
    2026-09-01T21:22:23.5884978Z ##[error]Process completed with exit code 101.
    

    Before changing any code

    Most red CI in this repo is environmental. The triage above already
    filtered the failures it could explain, but confirm the remainder is
    real before touching source:

    • Timing, throughput and tick-rate assertions are only meaningful with
      --release. In debug they fail for no reason.
    • --test-threads=1 is required; the tests share one SHM namespace.
    • Run rm -rf /dev/shm/horus_* first. Leaked namespaces make unrelated
      tests fail in a rotating pattern.
    • Without --no-fail-fast, cargo stops at the first failing binary and
      you see one failure per run.
    • A bare exit code 1 with no output is usually the /tmp per-user
      quota, not your change. df looks fine because it is a quota.
    • Detection of the concurrency bugs here is probabilistic. Passing once
      is not evidence of a fix.

    The rule that matters most

    Do not claim a fix without an ablation: show the test FAILS without the
    change and passes with it. A test that passes both ways proves nothing,
    and that is the most common way work here goes wrong. If you cannot
    construct a failing case, say so and leave the finding documented
    rather than shipping a change you cannot demonstrate.

    For anything about ordering or interleaving, prefer a loom model
    (horus_core/tests/loom_*.rs). It explores exhaustively and models weak
    memory ordering, which x86 cannot exhibit. Every loom model here
    documents the mutation that turns it red — yours must too.

    Scope

    Fix only what this issue reports. If you find something else, say so
    separately rather than widening the change. Do not add scratch or report
    .md files to this repo; it is public. Do not add co-authorship
    trailers. Explain the mechanism in the commit message, with the numbers
    you actually measured.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

autopilotOpened by the autopilot

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions