subscriptions-channels.yml fails on self-hosted Linux runners: no docker.sock (needs DOCKER_HOST env like smoke workflow)

[aacruzgon]

Summary

subscriptions-channels.yml fails on the current [self-hosted, Linux] runner pool: every testcontainer-based test panics immediately with

Failed to start PostgreSQL container: Client(Init(SocketNotFoundError("/var/run/docker.sock")))

The runners no longer expose a local Docker socket. subscriptions-smoke.yml adapted to this by setting DOCKER_HOST: ${{ secrets.DOCKER_HOST }} (plus DOCKER_HOST_IP) so testcontainers talk to a remote Docker host, but subscriptions-channels.yml never got that env block — its jobs run with no Docker configuration at all, so testcontainers falls back to /var/run/docker.sock and fails.

Evidence

• Run 30062816042 (dispatched 2026-07-24 against feat/cluster-capable-state): all 15 Unit tests (<matrix>) jobs failed identically — 125 tests failed in 0.02s, every panic at crates/persistence/tests/postgres_tests.rs:761 with the SocketNotFoundError above (1,875 occurrences across the log).
• A --failed re-run reproduced the exact same failure on all 15 jobs, ruling out a single bad runner.
• The failing set includes long-standing tests (e.g. postgres_integration_update_resource), not just newly added ones — the branch's code is not the trigger.
• The same persistence test suites passed in the main CI run (30062676960) on this branch; CI's Test Rust job uses unpinned [self-hosted] runners, so it can land on a machine that still has a Docker socket.

Why this is infra drift, not branch breakage

• The unit-tests job's cargo test ... -p helios-persistence list dates back to the WebSocket channel work (feat(subscriptions): implement WebSocket channel — Phase 2 #62, April 2026).
• The workflow's last green run was 2026-05-01 on feature/subscriptions (run 25202533588).
• Nothing on feat/cluster-capable-state (PR feat(cluster): make HFS cluster-capable behind a load balancer #269) touches this workflow; the failure will reproduce on any branch dispatched today.

Proposed fix

Add the same Docker env wiring subscriptions-smoke.yml uses (.github/workflows/subscriptions-smoke.yml:20-21) to subscriptions-channels.yml:

env:
  DOCKER_HOST: ${{ secrets.DOCKER_HOST }}
  DOCKER_HOST_IP: ${{ secrets.DOCKER_HOST_IP }}

testcontainers-rs honors DOCKER_HOST, and get_host() in the tests resolves to the remote Docker host, so the persistence/subscriptions testcontainer suites should work the same way the smoke workflow's do. Alternatively (or additionally), the runner pool could restore a local Docker socket, which would also un-break any other testcontainer consumer that pins [self-hosted, Linux].

Impact

• subscriptions-channels.yml is workflow_dispatch-only, so this doesn't block PRs, but it makes the channel integration matrix impossible to run — including as a validation gate for subscriptions changes (it was part of the post-merge validation sweep for PR feat(cluster): make HFS cluster-capable behind a load balancer #269, where everything else passed).
• Any other workflow that pins [self-hosted, Linux] and uses testcontainers without DOCKER_HOST will fail the same way.

[aacruzgon]

Additional finding on why only the unit-test jobs went red in run 30062816042 — the same Docker outage affects the green jobs too, just differently:

• Unit tests (failed): only job running -p helios-persistence; postgres_tests.rs:761 uses .expect(...) with no Docker guard → hard panic.
• Rest-Hook / WebSocket / Messaging (passed legitimately): zero testcontainers usage — in-process endpoints, no Docker needed.
• Email (passed misleadingly): email_integration.rs needs a mailpit container but its skip_if_no_docker! guard (line ~198) silently returns early when /var/run/docker.sock is absent and DOCKER_HOST is unset. The Email jobs show 5 passed ... finished in 0.01s — the delivery tests never ran. The guard's comment ("CI runners always have Docker so this is purely a developer-experience aid") no longer holds on the current runner pool.

So the workflow currently has a silent coverage gap in addition to the red jobs: all 32 Email matrix jobs report green without exercising email delivery. Setting DOCKER_HOST (the proposed fix) would both un-break the unit tests and make the email tests actually run again, since the skip guard also honors DOCKER_HOST.

[aacruzgon]

One more coverage detail: cargo test fails fast across test binaries, so in the failed unit-test jobs the run aborted at postgres_tests.rs and never executed the helios-subscriptions test binaries at all (rehydrate_integration, cluster_engine, subscriptions_cluster_pg — zero Running lines for them in any failed job log). subscriptions_cluster_pg.rs also uses testcontainers with no skip guard, so it would fail the same way once Postgres is un-broken unless DOCKER_HOST is set.

Also for completeness: the Elasticsearch/MongoDB integration tests are not affected — they're compiled out behind the elasticsearch/mongodb features, which this workflow doesn't enable; only their non-Docker unit tests ran.

Suggested hardening alongside the DOCKER_HOST fix: make the email guard (and any future availability guard) fail rather than silently pass when CI is set — Rust's libtest has no runtime skip outcome, so guards must choose, and in CI the right choice is loud failure. Optionally add --no-fail-fast to the unit-test step so one broken suite doesn't mask the rest.