Skip to content

perf(nodes): region-filtered /api/nodes cold build takes ~1 min per region set (build per IATA once, union per request) #618

Closed

Description

@dborup

Problem

A region-filtered Nodes request (/api/nodes?region=…) takes about a minute the first time each region set is requested after a restart. During that time the Nodes page shows nothing. Behind the reverse proxy the request can time out, and the page shows a 502 with no visible error. A region-group quick pick (#565) or any new combination of codes is a new set, so it is cold again.

Measured from the [nodes-region] … full build log lines:

Instance Retention / DB Cold build per region set
Production (2 vCPU / 3.8 GB) about 67.8M observation ids RKE 70.2 s, RNN 71.3 s
Canary (8 GB) 7 d retention RNN 53.4 s, AAL,AAR 52.6 s, AAR 34.2 s
Either, warm about 6 ms

This is pre-existing: cmd/server/nodes_region_cache.go has not changed between v0.2.5 and v0.2.6. #612 (#488 M2) touched it only after v0.2.6.

Cause

scanNodeRegionKeys does a full history scan per region set: the region's observers ⋈ observations ⋈ ADVERT transmissions. The cache is keyed by the joined set of codes, so N quick picks or combinations cost N full scans of the whole retained history.

Proposal

Build the heard membership per IATA code in one shared scan:

  • one pass over ADVERT observations grouped by observer IATA;
  • serve any region set as the union of the per-code key sets;
  • keep the existing watermark/delta and rebuild semantics per code, and the singleflight.

This is the same shape #488 M2's placement aggregate already uses (one shared full scan for all regions). Reuse that machinery instead of duplicating it where possible.

Also:

  • Warm the per-code sets in the background after startup, outside any request and outside PacketStore.mu, so the first user request does not pay for the scan.
  • Bound memory: about 15 codes × a few thousand keys.

Acceptance

  • After warmup, any region set (single, multi, quick-pick group) answers in milliseconds.
  • The cold path (warmup not finished) never blocks longer than today, and concurrent identical requests share one build.
  • The result is identical to today's per-set build for every set: a property test over the fixture plus a generated large DB.
  • Benchmark: cold all-codes build vs the sum of today's per-set builds on the generator DB (nodes: investigate direct-evidence regional placement without regressing historical cache #488), plus the warm union cost.
  • No SQL under PacketStore.mu; the server stays read-only; no new map[string]interface{}.

Activity

  1. dborup commented on Oct 26, 2026

    @dborup
    OwnerAuthor

    Fixed by #619 (merged as 960c910cc2270e489b1fb92c404e8c8058115da0).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions