Mesh-LLM v1: relay-gated direct-iroh inference between users (WAN)

[tlongwell-block]

Mesh-LLM v1: relay-gated direct-iroh inference between users

Real LLM inference between two users on the same Sprout relay, gated purely by relay access. One user shares compute (serves a model); another runs an agent against it. Discovery, connection, and hole-punch signaling all flow through relay events — no public iroh relays, no out-of-band coordination.

How it works

1. Discovery — a serving node publishes 24620 status reports; the relay validates membership and projects each into a per-reporter relay-signed 30621. Non-members can't be discovered.
2. Connect — a consuming node publishes a 24621 connect-request naming a target. The relay validates both parties are members.
3. Call-me-now — on an accepted request the relay emits a paired, relay-signed 24622 to both endpoints; each dials the other. Fails closed on non-membership, self-target, or malformed requests.
4. Inference — the consumer's local OpenAI-compatible endpoint proxies to the served model over the direct iroh link.

Lanes

• Relay (crates/sprout-relay): mesh kinds, status-report ingress → 30621 projection, membership-gated signaling → paired 24622. Dead iroh_relay admission module removed.
• Desktop (desktop/): in-process mesh node wiring, the 24620/24621/24622 UI flow, create-agent fail-closed on missing reporter_pubkey/inviteToken, 5s since-cushion on call-me-now subscribe. Mesh ports env-overridable (SPROUT_MESH_API_PORT/SPROUT_MESH_CONSOLE_PORT, default 9337/3131, fail-closed) so two nodes can run on one host for dev/test.

Verification

• Relay full suite: 271 passed / 0 failed, including the two mesh_signaling invariants (accepted pair → two relay-signed 24622; self-target → zero emit).
• Desktop: cargo check/test (mesh_llm), pnpm typecheck, fmt, file-size gate all green. The fail-closed + relay-client wiring was kept under size limits via a clean split (relayMeshSignaling.ts + startRelayMeshClientForTarget.ts) — no override bump.
• Relay lane independently fresh-eyes reviewed by Dawn + Perci → 9/10, all must-fixes addressed.

⚠️ PENDING: live two-machine inference proof

The relay-backed two-machine desktop flow is new and has not been run end-to-end. It requires two desktop GUI nodes + a served model, which can't be driven headless. Run before merge using the runbook below (single host OK thanks to the env-port seam):

Setup (single host OK with the env-port seam):

• Start a relay (dev) and add both identities as members: sprout-admin add-member --pubkey <A_HEX> --role member (and B).
• Launch two desktops against the relay. A on default ports; B with SPROUT_MESH_API_PORT=9338 SPROUT_MESH_CONSOLE_PORT=3132 bin/pnpm --dir desktop tauri dev.

Drive:

1. A: Settings → Compute → Share compute → pick a small model (e.g. jc-builds/SmolLM2-135M-Instruct-Q4_K_M-GGUF:Q4_K_M) → toggle Share → wait for Active.
2. B: Create Agent → toggle "Run on relay mesh" → A's target appears as model — device (≤~20s after A is Active) → select it → create. This starts B's client, publishes 24621, the relay emits paired 24622, both nodes dial.
3. Inference proof on B: curl -s http://127.0.0.1:9338/v1/chat/completions -H 'content-type: application/json' -d '{"model":"<id>","messages":[{"role":"user","content":"Reply with exactly one word: PONG"}],"max_tokens":128,"temperature":0}' → non-empty completion while the model is served only on A.

Passing condition: non-empty assistant content from B's local mesh endpoint, served by A, with discovery+connect gated purely by relay membership.

Passing condition: a curl chat completion against the consumer's local :9337 (or override port) returns non-empty content while the model is served only on the other node.

Connectivity: WAN via STUN + relay-signaled hole-punch

Disabling iroh relays (RelayPolicy::ExplicitlyDisabled) turns off the iroh relay transport (no *.iroh.link traffic, no public-relay leak) but keeps raw STUN on for public-address discovery. The STUN-discovered public address is injected into the invite token / EndpointAddr, exchanged via the relay-signaled 24621/24622 flow, and used to hole-punch directly over UDP — so this works over WAN, not just LAN. Our Sprout relay performs the address-exchange coordination that iroh's relay would otherwise do; STUN is only a "what's my public IP" lookup, not a data path.

Residual limit (honest): with iroh relays off there is no relay transport fallback, so two peers both behind symmetric NATs may fail to hole-punch (the case iroh relays normally cover). Works for the common cases (≥1 side cone-NAT / port-forwarded / server). If symmetric-both-ends becomes a real constraint, allow a self-hosted relay as a fallback — follow-up.

⚠️ Dependency caveat (must resolve before merge)

desktop/src-tauri/Cargo.toml pins mesh-llm-sdk to a personal fork (tlongwell-block/mesh-llm@bc2f1106) as temporary e2e wiring. The two SDK knobs it carries (disable_iroh_relays, runtime EmbeddedNodeHandle::join_token) sit on the unmerged upstream SDK stack — companion upstream PR Mesh-LLM/mesh-llm#782 (base micn/sprout-embedded-serve-sdk = PR #736; mergeable). Before this merges, repoint the dep to a Block-owned source or an upstream tag once #782/#736 land upstream.

This SHA incorporates a line-by-line review (Perci) that caught two real blockers, now fixed: (1) disable_iroh_relays skips the 5s endpoint.online() wait via RelayPolicy::ExplicitlyDisabled — no startup tax with relays off; (2) the passive/client runtime loop handles RuntimeControlRequest::Join, so join_token() works in the mode Sprout's relay-mesh client actually uses. The desktop lib compiles and the mesh tests pass against this SHA.

Relationship to main's private-relay work (why this removes hydrate_private_relay_config)

main (#798) began a private iroh relay path (hydrate_private_relay_config fetches an iroh_relay_url from the relay's NIP-11 + a NIP-98 bearer). That's a good future default — a hosted relay traverses symmetric NAT, which this PR's direct-STUN path can't. But it is not yet wired end-to-end: the relay's NIP-11 doesn't advertise iroh_relay_url and the restricted iroh-relay runtime lane is unfinished (iroh_relay.rs is scaffold). So the desktop hydration would fail closed before mesh start.

This PR therefore takes the direct-STUN + relay-signaled call-me-now path as the working WAN-today solution, and removes the incomplete private-relay hydration to keep the connectivity model coherent. The clean follow-up (when the relay side exists): make transport explicit config — private_iroh_relay default when NIP-11 advertises one, direct_stun_signaling (this PR) as the opt-in/fallback. Decided with Max + Perci; see thread.