Skip to content

tests: make the Docker integration suite build and boot again - #506

Open
TeoSlayer wants to merge 2 commits into
mainfrom
fix/integration-harness-build
Open

TeoSlayer wants to merge 2 commits into
mainfrom
fix/integration-harness-build

Conversation

@TeoSlayer

@TeoSlayer TeoSlayer commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The compose-based suite in tests/integration/local has not been able to start since rendezvous and gateway left this repository (a6f97f5, 2026-05-27). Each test brought the stack up with output discarded, polled for 60 s and failed at "agents did not register", so every run reported setup failures and nothing about the product.

With this change the image builds, the stack boots in about 7 s, and tests run. On main (macOS, 8 workers, 12 minutes): 150 tests, 109 pass, 41 fail.

What was broken

Each problem hid the next one.

# Problem Change
1 Dockerfile.multi built ./cmd/rendezvous and ./cmd/gateway, which no longer exist Build the rendezvous server from the module version go.mod pins. Drop the gateway build
2 No main package for pilot-gateway exists anywhere (the gateway repo is a library and is archived) Remove the ten test_gateway_* scripts and their compose file
3 The registry's /api/stats now requires an admin token, so the rendezvous health check never passed and agents never started All 217 uses only read total_nodes; point them at the unauthenticated /api/public-stats
4 macOS has no timeout(1); 38 scripts call it on the host and reported "command not found" as a product failure run-all.sh puts shims/timeout on PATH when the command is missing
5 run-all.sh ran test_cli.sh on the host. That script is the in-container driver: it starts a daemon on /tmp/pilot.sock, and its cleanup runs pkill -9 -f pilot-daemon and deletes that socket, so a full run killed the developer's own daemon Exclude it; _lib.sh no longer runs pkill on the host
6 QUICK.txt listed 17 scripts that do not exist Pruned

The 41 remaining failures

These are not addressed here. They are what the suite reports now that it runs.

Cause Count Tests
pilotctl send-file output no longer has .data.ack 12 send_file_* (4), size_file_* (4), mixed_traffic_burst, stress_edge, rendezvous_restart_midflight, sender_sigkill_midfile
pilotctl inbox lists preview, not data 7 message_text, message_json, message_binary, message_payload_integrity, chain_abc_message, mesh_3agents_crosstraffic, inbox_ordering
Relay fallback with the direct path blocked 3 force_relay_send_message, force_relay_send_file, force_relay_pubsub. A real daemon problem: see #505. The scripts also send 10 s after the partition, before the daemon has moved to the relay
Compose stack did not come up 2 nat_asymmetric_routing, nat_dual_symmetric
Need individual triage (CLI, registry or policy behaviour changed since May) 17 beacon_ping, broadcast, nat_egress_443_only, nat_hairpin, nat_partition_post_reg, nat_stateful_firewall, net_join_deny_per_network, policy_connect_tag, policy_cycle_evict, policy_datagram_score, register_identity_new_endpoint, rendezvous_beacon_split, resilience, sec_beacon_amplification, service_agent, trust_revoke, type_routed_service

Not covered

  • Dockerfile (the CLI + Python SDK image behind make test, test-cli, test-sdk, build) still builds sdk/cgo, sdk/python and ./cmd/gateway, none of which are in this repository. Those make targets remain broken.
  • tests/integration/*.sh holds 48 older copies of scripts that also exist under local/. Untouched.
  • Tests still discard the output of compose up, so the next build break will again look like a registration timeout.

Test Plan

  • docker build -f tests/integration/local/Dockerfile.multi succeeds; the image contains pilot-rendezvous, pilot-daemon, pilotctl
  • ./run-all.sh -j 8 over all 150 scripts: 90 pass before the timeout shim, 109 with it
  • ./run-all.sh --list no longer includes test_cli.sh or any gateway script
  • A full run leaves a daemon running on the host untouched
  • Not run on Linux

🤖 Generated with Claude Code

The compose-based suite under tests/integration/local had not been able
to start since rendezvous and gateway left this repository (a6f97f5,
2026-05-27). Every test brought the stack up with its output discarded,
polled for 60 s and failed at "agents did not register", so a run
reported setup failures only.

What was broken, in the order each one hid the next:

- Dockerfile.multi built ./cmd/rendezvous and ./cmd/gateway, which no
  longer exist. The rendezvous server is now built from the module
  version go.mod pins. pilot-gateway is not built: no main package for
  it exists anywhere, so the ten test_gateway_* scripts and their
  compose file are removed.
- The registry's /api/stats now requires an admin token, so the
  rendezvous health check never passed and the agents never started.
  All 217 uses only read total_nodes; they use /api/public-stats.
- On macOS there is no timeout(1). 38 scripts call it on the host and
  reported the "command not found" as a product failure. run-all.sh
  puts a small shim on PATH when the command is missing.
- run-all.sh ran test_cli.sh on the host. That script is the driver for
  the in-container image and begins with `pkill -9 -f pilot-daemon`, so
  a full run killed the developer's own daemon. It is excluded, and
  _lib.sh no longer runs pkill on the host.
- QUICK.txt listed 17 scripts that do not exist.

Result on main (macOS, 8 workers, 12 min): 150 tests, 109 pass, 41 fail.
The failures are real output from running tests now; they are listed by
cause in the pull request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its cleanup, not its start, runs pkill and removes /tmp/pilot.sock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant