Skip to content

✨ perf: progressively load the dashboard shell and live panels - #16457

Merged
hivecommons-hive[bot] merged 5 commits into
v5from
perf/dashboard-progressive-loading-20261103
Nov 3, 2026
Merged

hivecommons-hive[bot] merged 5 commits into
v5from
perf/dashboard-progressive-loading-20261103

Conversation

@clubanderson

Copy link
Copy Markdown
Member

Change

Show the dashboard shell before downloading the application script; serve the exact classic-script bytes through a content-addressed, privately cached deferred asset with SRI and an explicit reload recovery state.

Prioritize the selected and visible panels, cap background work at two concurrent jobs through decoding, coalesce overlapping polls and hidden-tab snapshots, render one status section per idle turn, and restore navigation without awaiting every API. Preserve auth/session guards and mutation snapshot floors.

Evidence and verification

Authenticated Chrome baseline: approximately 1.2 MB compressed HTML, 9.1 seconds download, 75 requests in 45 seconds. Repository-byte extraction reduces the initial compressed shell to approximately 154 KB; the application script remains a separate cacheable download. This is a byte comparison, not a post-deployment timing claim.

Added HTTP cache/gzip/HEAD/304/CSP/recovery tests plus executable progressive-loading regressions. Pre-commit review identified and fixed a queued tracker redraw race. Canonical dashboard regenerated; Go formatting clean. Per AGENTS.md, CI is the sole test/build/lint verdict. No deployment or live speedup claimed.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@kubestellar-prow kubestellar-prow Bot added dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Nov 3, 2026
@kubestellar-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hivecommons-hive

hivecommons-hive Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Hive status — updated in place; edits do not notify. Full history is in the Activity Monitor.

clubanderson this PR is held at stage/071-needs-author and the hive will not merge it yet.

Failing checks:

  • test (dashboard shuffle 3/3)

Next: the hive does not push to your branch. Push a new head (or reply here); CI and the review swarm re-run automatically and the PR returns to the normal path.

Review findings from reviewer (folded here because this head already carries its hive review):

Read the tree at the PR head (not just the diff). The externalization is sound; nothing blocking.

Verified

  • webstatic/dashboard_script.go:84-116 — the marker occurs exactly once in static/index.html (line 8683), and it is the document's last <script> (only two exist: 7056 and 8682), so the "no later script" guard holds and defer cannot reorder execution. The extracted body contains no document.write/document.currentScript.
  • Auth: /assets/ is not in isPublicPath (server.go:2377-2440), so the content-addressed script sits behind the same authenticate wrapper as /{$}. CSP: script-src-elem 'self' 'sha256-…' already admits the same-origin external script, and the inline bootstrap is hashed because setBrandedIndex(branded) (server.go:1719) receives the post-extraction bytes.
  • Go 1.22 mux specificity means GET /assets/{script} wins over the file server; the handler 404s anything but its exact path (dashboard_script.go:130), so no filesystem traversal surface.
  • Deferred execution: the only readyState === 'loading' branch in the main script (index.html:11538) falls through to queueMicrotask(start) when run deferred, which is the same behaviour.
  • refresh.js:144 dashboardStatusCanRender + Map-overwrite in dashboardRenderSection (progressive.js:144) means a queued older snapshot is dropped rather than painted over a newer one; the mutation seq floor is preserved.

Low

  • refresh.js:150 _deferredLoadPromise is now write-only: _afterPaint no longer awaits it and shimmy-handlers.js:92 is the sole remaining reference. Harmless, but dead state — worth dropping or noting why it stays.
  • dashboard_script.go:44 — the bootstrap's runtime path (tested at dashboard_script_test.go:52) shows a non-dismissable "could not be loaded" dialog for any uncaught top-level exception during the main script's initial execution, not just a fetch failure. A deterministic throw becomes a reload loop with no escape (cancel is suppressed). Before this PR the same throw left a partially initialised page, so this is a UX trade-off rather than a regression; the dialog copy could say "could not finish starting" to match.

Looks correct to me.

Review findings from reviewer-2 (folded here because this head already carries its hive review):

docs-currency review (read at head 8e1b32b)

1. New GET /assets/{script} route is not in the API reference — src/pkg/dashboard/server.go:1709 registers s.mux.Handle("GET /assets/{script}", mainScript). src/docs/api-reference.md:3 says "update this page in the same change that adds or renames a route", and the PR already edits src/docs/api-reference/snapshots-and-style.md (line-number bump only, line 9) without adding the row. The comparable asset route /branding/custom.css is documented there at server.go:1733. The route is not in isPublicPath (server.go:2377), so it is dashboard auth/session.

| `GET` | `/branding/custom.css` | Dashboard auth/session | Operator branding stylesheet override, read per request (see [branding](../branding.md)) | `pkg/dashboard/server.go:1733` |
| `GET` | `/assets/{script}` | Dashboard auth/session | Content-addressed dashboard application script (`/assets/dashboard-main.<sha256>.js`, SRI-pinned, `Cache-Control: private, max-age=31536000, immutable`, ETag/304, gzip); only the exact hashed path is served, anything else is 404 | `pkg/dashboard/server.go:1709` |

and bump Snapshots and style … | 5 | → 6 at src/docs/api-reference.md:14 and 5 routes → 6 routes at :35.

2. src/docs/branding.md:40-48 now describes the old shape — "the document is ~1.3 MB of inline HTML/CSS/JS … The index document carries a precomputed gzip body and a strong ETag". After webstatic.ExternalizeDashboardScript (server.go:1705) the branded JS is served from the separate hashed asset whose URL changes with branding (dashboard_script.go:68-69). Suggested rewording for line 45-46: "The index shell and the content-addressed main script each carry a precomputed gzip body and a strong ETag; the script's URL changes whenever branding changes, so a restart invalidates old clients' cached script."

3. Recovery state hides a root that does not exist (low) — dashboard_script.go:34 looks up #hive-dashboard-root to hide/inert the shell; static/index.html at head has no element with that id (0 matches), and dashboard_script_test.go:377 fabricates one in the stub. The if (root) guard means no crash, but the "hide the live shell behind the dialog" half of the recovery state the body describes is dead code. Either add id="hive-dashboard-root" to the shell's top-level container in static-src or drop the lookup and the test's fake root.

Evidence bundle `#164578e1b32b` sha256:84380337c8f2c89dfffb67319073b50a9bef06e0207a49626ee1d56e7dfc9f85 — download from the dashboard or `hivectl review evidence #16457`.

Review findings from adjudicator (folded here because this head already carries its hive review):

Correctness review (read at head 8e1b32b; progressive_loading.test.js run locally, 19/19 pass; Go asset handler sits behind s.authenticate at server.go:1739 and CSP script-src-elem already carries 'self' at webstatic/csp_script_src.go:92, so the externalized script is auth-gated and CSP-permitted).

1. Two-slot background queue has head-of-line blocking with no slot timeout (medium)

progressive.js:1 caps background work at DASHBOARD_BACKGROUND_CONCURRENCY = 2, and dashboardScheduleWork (progressive.js:107-108) returns the existing in-flight promise for a repeated key instead of starting a new fetch. A slot is only released in finished() (progressive.js:92-96) when the job settles. Almost none of the panel fetchers use a timeout — the whole generated index.html contains three AbortController uses — and the Go server has a ReadTimeout but no WriteTimeout (server.go:1741-1749), so a handler that stalls holds the response open indefinitely.

Consequences vs. the pre-PR behaviour (independent setInterval polls, history-sparklines.js:30 old):

  • one stalled /api/... response freezes that panel's poll (dashboardRunInterval → coalesced to the hung promise) rather than the next tick retrying;
  • two stalled responses freeze every background panel on the page (cost, approvals, review queue, gh-auth banner, …) until the hung sockets die.

Suggested fix — release the slot (and the coalescing key) after a ceiling while letting the late job still settle its own promise:

--- a/src/pkg/dashboard/static-src/js/progressive.js
+++ b/src/pkg/dashboard/static-src/js/progressive.js
@@
     const DASHBOARD_BACKGROUND_DELAY_MS = 350;
+    // A stalled response must not pin a slot (or its coalescing key) forever.
+    const DASHBOARD_SLOT_TIMEOUT_MS = 20000;
@@
         job.running = true;
         _dashboardBackgroundRunning++;
-        function finished() {
-          _dashboardBackgroundJobs.delete(job.key);
-          _dashboardBackgroundRunning--;
-          dashboardWakeBackground();
-        }
+        let released = false;
+        function finished() {
+          if (released) return;
+          released = true;
+          clearTimeout(slotTimer);
+          if (_dashboardBackgroundJobs.get(job.key) === job) _dashboardBackgroundJobs.delete(job.key);
+          _dashboardBackgroundRunning--;
+          dashboardWakeBackground();
+        }
+        const slotTimer = setTimeout(function() {
+          console.warn('dashboard work ' + job.key + ' exceeded ' + DASHBOARD_SLOT_TIMEOUT_MS + 'ms; releasing its slot');
+          finished();
+        }, DASHBOARD_SLOT_TIMEOUT_MS);

A test for "a never-settling job releases its slot and a later tick for the same key starts a fresh flight" would close the gap in progressive_loading.test.js (none of the 19 cases exercise a job that does not settle).

2. Selected-section priority misses hashes with a query string or alias (low)

dashboardWorkPriority (progressive.js:52) compares window.location.hash === '#section-' + id literally, but the dashboard's own hash parser _ocSectionFromHash strips ?… (h.split('?')[0]) and maps aliases such as #throughput → pr-throughput-section and #review-queue → review-queue-section (generated index.html _ocSectionFromHash). With e.g. #section-overview-section?project=x or #throughput, the selected section never gets priority 0 and competes as an ordinary visible/background panel. Minimal fix for the query-string case:

        if (window.location.hash.split('?')[0] === '#section-' + id) return 0;

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Open PR overlap check

Base branch: v5

Conflicts are checked with every PR merged onto the current tip of v5, so a PR that only needs a rebase is not reported as conflicting with its siblings.

No conflicting open PRs or same-file overlaps were found.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

Post-PR integration review found a defer-specific initialization-order bug: the modal observer could run while later main-script constants were still in their temporal dead zone. Fixed in 6e32aa8 by deferring its non-loading startup to a microtask; added executable regressions for loading, interactive, and complete ready states using the production initializer/helper order. The earlier hidden-tab tracker redraw race also has an executable regression. CI-reported source citation drift was regenerated with the prescribed fixer. No local test/build/lint execution and no hosted deployment or measured live speedup claimed.

Signed-off-by: Andy Anderson <andy@clubanderson.com>
@clubanderson

Copy link
Copy Markdown
Member Author

CI iteration: the new executable progressive-loading suite, existing idle behavior, lint, and reviewed modal initialization regressions passed. Corrected four integration contracts from failed shards: asset registration is now a literal static route with exact-hash validation; interval replacement/cleanup remains directly guardable; review and throughput wiring assertions now pin the queued form instead of the removed burst form. Added real mux/unknown-hash/HEAD/304/method regressions and an executable five-panel bootstrap concurrency test. Source citations regenerated after final route wiring. Waiting for full exact-head CI and coverage rather than bypassing failed gates.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read against head d87dea3 (the push that fixes the three test (dashboard …) failures on 6e32aa8); no defects found in the parts that carry real risk. CI for this head was still running when I looked.

What I verified in the tree, not just the diff:

  • Auth boundary. /assets/dashboard-main.<sha256>.js is registered on s.mux (server.go:1709) behind the same authenticate → roleEnforcement → securityHeaders chain as GET /{$} (server.go:1738), and /assets/ is not in isPublicPath (server.go:2377+). Browser auth is session-cookie / proxy-header based (server.go:2030-2075), which a same-origin <script src> carries, so a client that could load the shell can load the script. No new unauthenticated surface.
  • CSP. ExternalizeDashboardScript runs before NewIndexDocument(branded) and before the bytes are handed to setBrandedIndex, so the new inline bootstrap listener is hashed into script-src-elem from the served bytes, and webstatic/csp_script_src.go:92 already seeds the list with 'self', which the external script needs. SRI integrity is the SHA-256 of the exact extracted body (dashboard_script.go:97-103).
  • Fail-safe extraction. dashboard_script.go:87-96: a marker count ≠ 1, a missing </script>, or any later <script falls back to the original inline document with a warn log (server.go:1712), so a future generator change degrades to today's behaviour rather than a broken shell. ServeHTTP 404s any path other than its own (dashboard_script.go:135-139), so GET /assets/{script} is not a file-server.
  • Interval/scheduler behaviour. history-sparklines.js:20-45 still clears and re-creates the timer per key; dashboardRunInterval only routes keys present in DASHBOARD_TASK_SECTIONS through dashboardScheduleWork, which dedupes by key so a slow fetch does not stack. progressive.js:113-135 drops a queued render whose snapshot fails dashboardStatusCanRender (refresh.js:144-148), so the mutation seq floor from noteMutationSnapshotFloor is honoured for deferred paints as well as immediate ones.
  • d87dea3 moves dashboardClearInterval adjacent to the new setInterval wrapper to satisfy TestDashboardIntervalsAreGuardedOrDocumented, and updates the two string-wiring tests to the new [key, fn] task tuples that index.html:67704 actually contains.

One thing to be aware of (info, not a defect): the asset URL embeds the hash of the branded bytes, so two processes with different branding/kiosk settings behind one hostname serve different URLs. The bootstrap at dashboard_script.go:21-63 handles that with the reload dialog as the body says, but a reload against a round-robin pair can loop; if that topology exists anywhere, a sticky-session note in the operator docs would help.

PR body vs diff: consistent — the ~154 KB shell figure is reproduced by TestDashboardScriptRealIndexShellBudget, and no runtime speedup is claimed. Looks correct to me; I have not approved or merged anything.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@clubanderson

Copy link
Copy Markdown
Member Author

Final live-update consistency fix: agent-status SSE now coalesces the cost-table redraw as well as agent tables, rather than rebuilding the hidden cost table on every incoming frame. Added a production-handler regression proving a hidden ten-frame burst performs zero table rebuilds, then one latest-state rebuild per table on resume. Hive bot integration of current v5 was fast-forwarded and preserved, including unrelated tracker/header improvements. All merges remain gated on the actual latest source head.

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency — delta re-review at 935ddb0 (delta = merge of v5 abf91a8 #16454; no new author commits). Regenerated static/index.html from static-src/ at head and it is byte-identical, so the merge resolution of the generated artifact is clean. Verdict: approve, with two doc gaps the author may want to close:

  1. New served route is missing from the API reference. pkg/dashboard/server.go:1709 registers GET /assets/{script} (the content-addressed /assets/dashboard-main.<sha256>.js, Cache-Control: private, max-age=31536000, immutable, ETag/304, 404 for any other name — webstatic/dashboard_script.go:130-145). Its sibling GET /branding/custom.css (server.go:1733) is documented in docs/api-reference/snapshots-and-style.md:9, which this PR already edits; the new route is not. Suggested row:
| `GET` | `/assets/{script}` | Dashboard auth/session | The dashboard's main application script, externalized from the index document as a content-addressed deferred asset (`/assets/dashboard-main.<sha256>.js`, SRI `sha256-…`, `Cache-Control: private, max-age=31536000, immutable`, `ETag` + 304); any other name is `404` | `pkg/dashboard/server.go:1709` |
  1. docs/dashboard-source-layout.md is now stale against this PR. Line 3 ("the single document the spoke dashboard serves") and line 24 ("the served bytes are identical") no longer hold: ExternalizeDashboardScript (webstatic/dashboard_script.go:81-111) splits the main script out at serve time, keyed on the literal first line of static-src/js/core.js:1 (pinned by TestDashboardScriptRealIndexShellBudget, dashboard_script_test.go:107). The JS areas list (lines 188-207) also omits the new progressive area added to manifest.txt:61. Suggested: add progressive after layout on line 188 and a sentence under "Why a generated artifact" noting that the committed index.html is still the canonical source document, but the server externalizes the final script (marker = js/core.js:1) so editing that first comment line changes the shell/script boundary.

Previous finding [correctness] webstatic/dashboard_script.go:26 (top-level throw surfaces as fatal load dialog): still present, unchanged by the delta; it was advisory under the prior approve.

— hive: agent=reviewer-2 backend=copilot model=claude-fable-5.1 copilot=1.0.88

@hivecommons-hive
hivecommons-hive Bot merged commit 6b9d9cf into v5 Nov 3, 2026
81 of 83 checks passed
@kubestellar-prow
kubestellar-prow Bot deleted the perf/dashboard-progressive-loading-20261103 branch November 3, 2026 19:27
@hivecommons-hive

Copy link
Copy Markdown
Contributor

merged by Hive trusted-author auto-merge: author @clubanderson holds role owner and GitHub permission admin on repo; CI green at 8e1b32b.

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the DCO. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant