Skip to content

Grid routing ranks by ADVERTISED total VRAM, not live free — and registry/gossip have no join key #2228

Description

@joelteply

Two findings from mapping the routing seam. They share one root: the node registry and the capacity gossip were built as separate worlds and have never met.

1. Live bug — routing picks the biggest CARD, not the most FREE node

modules/grid/router.rs::find_gpu_node ranks candidates with gpu_vram(node):

fn gpu_vram(node: &GridNode) -> u64 {
    node.capabilities.iter().filter_map(|c| match c {
        NodeCapability::Compute { vram_mb, .. } => *vram_mb,
        _ => None,
    }).max().unwrap_or(0)
}

NodeCapability::Compute { vram_mb } is what the node advertised at registration — a static spec-sheet total. So a 5090 that is currently running a game outranks an idle 16GB box. Routing sends work to the machine with the biggest card rather than the one that can actually take it.

This is the same spec-sheet-vs-live-reading failure that capacity/grid_budget.rs::a_full_big_card_offers_its_free_bytes_not_its_total exists to prevent — except that guard is on the new projection, and this is production routing today.

Related: capability routing only fires at all when the local node has no GPU whatsoever

if requires_gpu(command) && !self.local_has_gpu { ... }

so a box with a small card never routes out even when a peer could serve far better. Routing is binary (have-GPU / don't), not fit-based.

Also worth flagging while here: requires_gpu(command) is a hardcoded command-prefix list (genome/train, plasticity/, ai/*), which is the hardcoded-command-list anti-pattern CLAUDE.md forbids. A command should declare its own requirements rather than be enumerated in the router.

2. Structural blocker — no join key between the two capacity sources

Source Keyed by Capacity number Freshness
NodeRegistry / GridNode node_id — a transport identity (Tailscale IP) NodeCapability::Compute { vram_mb } advertised at registration, static
capacity::gossip / CapacityOffer airc PeerId (Uuid) gpu_free_bytes_live live, with silence-eviction

GridNode carries node_id, node_name, addresses, capabilities, trust_level, last_seen, latency_ms — and no PeerId, no UUID. rg 'peer_id|PeerId|uuid|Uuid' modules/grid/node.rs returns nothing.

So the live capacity readings cannot be attached to the nodes the router is choosing between. Fixing (1) properly is blocked on this.

The fix, and the one to avoid

Avoid: fuzzy-joining on IP or node name. It would look correct and mismatch silently in the field — worse than the current bug, which is at least honestly dumb.

Do: give GridNode the peer's airc PeerId, set where the node is registered/discovered. Then the two sources join cleanly and each answers exactly one question:

  • registry — identity, trust, addresses, what hardware exists (durable)
  • gossipwhat is free right now (ephemeral)

Routing then filters by trust/reachability from the registry and ranks by live capacity from gossip. One source per question instead of two sources for one.

airc_core::PeerId is already a newtype over Uuid, so there is an in-tree precedent to copy rather than a type to design.


Found while building capacity/grid_budget.rs (see docs/architecture/GRID-ELASTIC-CAPABILITY.md). The grid-elastic capability work is blocked on the join key — on identity, not on policy.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions