Skip to content
This repository was archived by the owner on May 11, 2026. It is now read-only.

perf: add materialized view for aircraft stats dashboard#13

Open
okane16 wants to merge 6 commits into
mainfrom
perf/aircraft-stats-mv
Open

perf: add materialized view for aircraft stats dashboard#13
okane16 wants to merge 6 commits into
mainfrom
perf/aircraft-stats-mv

Conversation

@okane16
Copy link
Copy Markdown
Contributor

@okane16 okane16 commented Apr 8, 2026

Summary

The aircraftSpeedAltitudeByType dashboard query is the #1 slowest query in production, scanning 63.9M rows (1.66 GB) on every page load to produce 30 aggregated rows. This PR adds an incremental AggregatingMergeTree materialized view that pre-computes category-level stats at insert time, reducing the dashboard query from a full table scan to reading ~41 pre-aggregated rows.

Changes

  • New file: moose/app/ingest/aircraft_stats_mv.ts — defines AircraftStatsByCategoryTable (AggregatingMergeTree) and AircraftStatsByCategoryMV (incremental MV over AircraftTrackingProcessedTable)
  • Updated: moose/app/apis/aircraftSpeedAltitudeByType.ts — fast path reads from the MV target table for the default (unfiltered) dashboard load; falls back to the full table scan only when altitude/speed range filters are active
  • Updated: moose/app/index.ts — exports the new MV resources
  • Bump: moose-lib and moose-cli from 0.6.493 → 0.6.495 (fixes hosting build version mismatch)

Benchmark Results

Benchmarked on production ClickHouse (main branch, 63.9M rows):

Metric Baseline (full scan) Candidate (MV) Improvement
Duration run 1 6,748 ms 86 ms 78x
Duration run 2 5,730 ms 233 ms 25x
Duration run 3 8,437 ms 123 ms 69x
Avg duration 6,972 ms 147 ms 47x faster
Granules read 7,819 / 7,819 5 / 5 1,564x fewer
Result rows 30 30 Parity ✅

EXPLAIN Comparison

Baseline:

ReadFromMergeTree (AircraftTrackingProcessedTable)
  PrimaryKey Condition: true
  Parts: 13/13
  Granules: 7819/7819    ← full scan

Candidate:

ReadFromMergeTree (AircraftStatsByCategoryTable)
  PrimaryKey Condition: true
  Parts: 5/5
  Granules: 5/5           ← 41 pre-aggregated rows

Caveats

  • The MV only pre-aggregates data inserted after the MV is created. A one-time backfill INSERT INTO ... SELECT is recommended after merge to cover historical data.
  • Dashboard requests with altitude/speed range filters still hit the full table. These filters do not appear in the production slow query log (all top-10 slow queries are the unfiltered default).
  • The MV is additive — no existing tables or data are modified.

Test Plan

  • TypeScript compiles cleanly (npx tsc --noEmit)
  • moose ls detects the new table and MV
  • Preview deployment builds and deploys successfully
  • Benchmark shows 47x improvement on production data
  • After merge: run backfill INSERT to populate MV with historical data
  • Verify dashboard loads correctly via the MV fast path

Made with Cursor

okane16 added 2 commits April 8, 2026 13:47
The aircraftSpeedAltitudeByType query scans 63M+ rows (1.6 GB) on every
dashboard load, taking 3.5-4.7 seconds. Add an AggregatingMergeTree
target table populated by an incremental MaterializedView that
pre-aggregates stats by category at insert time.

The API now reads ~10 pre-aggregated rows (fast path) for the default
dashboard view, falling back to the full scan only when altitude/speed
range filters are active.

Made-with: Cursor
Fix hosting build failure caused by version mismatch between the
locked moose-lib (0.6.493) and the CI-resolved moose-cli (0.6.495).

Made-with: Cursor
@fiveonefour-hosting
Copy link
Copy Markdown

fiveonefour-hosting Bot commented Apr 8, 2026

The latest updates on your projects. Learn more about Boreal.

Project Deployment Logs Base URL Updated (UTC)
planes ⏹️ Terminated Logs https://514-demos-planes-perf-airc-f6378.boreal.cloud 2026-04-16 21:35:01

@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 8, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
planes Ready Ready Preview, Comment Apr 9, 2026 4:38pm

Request Review

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant