daemon: much less CPU under send bursts

[TeoSlayer]

Summary

Profiling two local daemons under bursts of pilotctl send-message (32- and 256-way, loopback, no network loss) showed the sending daemon spending ~85% of its CPU in the Go scheduler (waking and parking threads) and 73% of everything it allocated on info replies. Three changes:

1. Dials stop polling every 10 ms. SYN-ACK already signalled conn.DialCh; RST and shutdown now do too, so the ticker is only a 250 ms backstop. With hundreds of dials in flight the 10 ms tickers alone kept the scheduler busy.
2. Concurrent info requests share one reply for 200 ms. Every pilotctl send-message asks for info (to see whether a session with the peer exists), and the reply serialises every peer and connection row — so N concurrent sends cost N builds over a table that itself grows with N. Requests that arrive during a build wait for it rather than starting their own; this also stops a burst from issuing one registry lookup per request when the network-membership cache has just expired. The TTL is a package var so tests can turn reuse off.
3. The daemon caps itself at 4 OS threads unless GOMAXPROCS is set. One thread per core (16 on the test machine, 64 on the service-agent VM) is what made the scheduler churn expensive.

Measured (sender daemon, CPU per message)

Burst	main	this branch	of which: branch at 16 threads
32 concurrent × 8	4.53 ms	1.37 ms (−70%)	1.56 ms
256 concurrent × 4	24.06 ms	2.62 ms (−89%)	3.53 ms

Memory right after the burst: 48 → 35 MB (32-way), 65 → 56 MB (256-way). The receiving daemon is unchanged (~1.2 ms per message).

Not changed here, found on the way: at 256-way, 40–46% of sends fail on both builds. The receiver drops SYNs past its per-source limit (10/s sustained, burst 64), and each refused dial retries for seconds. Connection-per-message is the underlying cost.

Functionality

• New tests: TestInfoReplyIsSharedWithinTTLThenRebuilt, TestInfoReplyReuseCanBeTurnedOff, TestResetWakesAWaitingDial (the last fails on main).
• go test ./pkg/... ./cmd/... ./internal/... -short: pass. Full go test -parallel 4 -count=1 ./tests/: pass (291 s).
• Docker suite (harness from tests: make the Docker integration suite build and boot again #506): all 109 of 109 tests that pass on main pass on this branch. The image was built from a checkout of this branch with tests: make the Docker integration suite build and boot again #506 merged and the tests ran from it, so the NAT tests' compose up --build rebuilt the same code. (An earlier run reported here had built from the harness checkout, i.e. main, after the first NAT test; corrected.)
• Behaviour to know: two pilotctl info calls within 200 ms show the same numbers. GOMAXPROCS still overrides the thread cap.

🤖 Generated with Claude Code