perf: serve historical metric charts from pre-aggregated buckets - #14
Conversation
Long chart ranges aggregated raw samples inside SQLite: a 7-day interface chart
scanned ~484k rows and cost ~112 ms, a 30-day one ~486 ms, and the cost grew
linearly with every day of history because nothing pruned the raw tables. The
GROUP BY bucket expression cannot be indexed, so no index fixes this shape.
Two 15-minute bucket tables (system_metric_buckets, interface_metric_buckets)
are now upserted from the raw rows of their own quarter hour on the collector's
10 s tick, in the same transaction — recompute-based, so re-running converges
instead of double counting. The interface buckets additionally materialize the
peak of the router-wide per-instant summed rate: the chart's combined spike
cannot be reconstructed from per-interface maxima (a sum overstates it, a max
understates it).
/metrics/{system,interfaces} read buckets for 24h/7d/30d and keep raw reads for
1h/6h, where the window scan is already cheaper; an uncovered (e.g. freshly
upgraded) range falls back to raw so a chart never goes empty. A one-off
chunked backfill builds buckets from existing raw history at first start, and
hourly batched retention prunes raw rows after 2 days and buckets after 400 —
both overridable through settings. Raw window predicates avoid per-row
strftime so the composite (router_id, timestamp) indexes apply as range
searches.
The CI run caught the buckets comparison drifting by one point (96 vs 95) at some positions within the 15-minute grid. Two contributing edges: - the window predicate had been switched from integer epoch comparison to text timestamp comparison; SQLite collates bare timestamp comparisons as BINARY, so any text mismatch against the datetime()-derived bound moves a row across the bound silently. Restored the epoch-seconds form. - the seeded block ran to wall-clock now, and the test asks the bucket path and the raw path ~0.2 s apart — each request derives its own startTime, and the newest samples sat exactly on that moving edge. Seeding now stops one bucket before now, so the newest sample's bucket cannot straddle the request drift.
The bucket query admits the window-straddling bucket (its -900 s allowance), while the raw path drops it once its samples fall under the request-specific startTime. A swept simulation of all grid phases (129 positions across the 15-minute grid) confirmed the two paths diverge whenever any sample lands in the window's first bucket: seeding now starts two buckets past the floored window start, and every phase matches.
- interface buckets no longer store a router-wide summed peak and treat it as the selection's: the exact combined peak of a proper subset is not reconstructable from per-interface aggregates, so the read side takes the max of the selected interfaces' own peaks (exact for one interface, a never-inventing lower bound for a subset); the schema comment states this - the recombined interface mean now divides by instants per quarter hour (max of per-interface sample counts — every tick samples every interface) instead of collapsing the two-level aggregation into one max(samples) across display buckets, which inflated averages ~4x on 7d and ~16x on 30d - summed-peak CTEs group by (router_id, timestamp), so an unscoped backfill can no longer mix two routers' instants - system bucket rows carry temperature/voltage non-NULL counts; a display bucket weights those averages by them, matching SQLite avg() semantics over NULLs - ts_epoch is aggregated with max() in both bucket queries (it was a bare column under GROUP BY, returning an arbitrary row's value) - bucket-path coverage is checked per selected interface, not for any name of the router, so a partially bucketed range cannot serve a half-empty chart - the collector's bucket recompute error is no longer discarded: the tick's raw rows roll back with it, so raw and buckets never drift apart silently Equivalence test adapted honestly: the system path asserts full equality; the interface subset path asserts exact averages plus the never-invent bound, and a new single-interface test asserts full equality where the bound coincides with the truth.
countInterfaceBucketsInRange now counts DISTINCT selected names and the handler requires coverage of every name in the selection: one bucketed interface out of two used to switch the range to the bucket path and sum a partial selection while looking complete. Raw serves such requests until the backfill catches up. Also: the scan helpers own their rows.Close (the handlers' deferred Close only fires on their own error paths), and the README peak wording matches the per-interface storage decision from the previous commit.
|
Review addressed in aeb4ccf + b30ece1; gate green on the final push. Adopted (9 of 12 findings): mean now divides by per-quarter-hour instants ( Design decision changed under review: the router-wide summed peak is removed from interface buckets. For a proper subset it overstated the chart's combined spike (the collector samples every interface, so selections are almost always subsets and the default UI selects 2 of N). The read side now reports the max of the selected interfaces' own peaks: exact for one interface, a strict lower bound for a subset — it can understate a genuine simultaneous spike but never manufacture one. The equivalence test says the same thing: system path asserts full equality; interface subset path asserts exact means + the never-invent bound; a new single-interface test asserts full equality where the bound coincides with the truth. False positive (1 finding): the "placeholders vs args mismatch" in the interface recompute — verified by probe, the window predicate is declared once in Not adopted (2 findings): the |
1 participant
Closes #8 (items 1, 2 and the read-path parts of 5; sargable rollup predicates stay open).
What
Two pre-aggregated 15-minute bucket tables, upserted from the raw rows of their own quarter hour on the collector's existing 10 s tick, in the same transaction:
system_metric_buckets— samples, avg/max CPU, avg memory %, avg bytes used, max bytes total, avg/max temperature, avg/min/max voltage;interface_metric_buckets— per interface: sum/max/avg of rx/tx rates, plus the peak of the router-wide per-instant summed rate (*_sum_bps_max). The last one matters: the chart's two-level aggregation peaks over per-sample totals, and that cannot be reconstructed from per-interface maxima (a sum of peaks invents a combined spike, a max of peaks understates it), so it is materialized at write time./api/v1/metrics/{system,interfaces}read buckets for 24 h/7 d/30 d and keep the raw tables for 1 h/6 h, where the window scan is already the cheaper query. Coverage is checked per router/range, so an uncovered range (right after an upgrade, before the backfill) falls back to raw and a chart never goes silently empty. The JSON shape is unchanged; both scan paths emit identical column sets.A one-off chunked backfill (day-sized chunks, marker in
app_settings, versioned so a layout change can re-trigger it) builds buckets from existing raw history at first start; interrupted runs resume. Hourly batched retention prunes raw rows after 2 days (buckets are final once a quarter hour closes — raw only needs to outlive bucket openness) and buckets after 400 days, both overridable through settings; batches are oldest-first over the rowid order so the single SQLite writer is never held past other workers' busy timeout.Measured
Same-shape measurements against a synthetic 30-day database (2.07 M interface rows, 259 k system rows):
Chart cost stops depending on history depth. The backfill also matters for the numbers to survive the first 30 days on a new install: raw retention stops at 2 days, the long horizon lives in buckets.
Verification
go test ./internal/...— 6/6 packages, including: aggregation correctness from a handful of raw rows; upsert idempotency (recomposing the same window twice does not double count); retention (old rows pruned, recent kept, bucket retention never below raw); handler-from-buckets output matching the raw derivation point for point.check-identifiers.cjsclean,gofmt/go vetclean.Not in this PR
The sargability fixes for the rollup statements (
DATE(record_date)wrapping,OR router_id IS NULL) and the unbounded all-time fallbacks inanalytics.go— different query family, tracked separately in #8; they were measured at 5–6x and 700–800 ms respectively and deserve their own review.