Skip to content

fix(kiosk): stagger initial api requests and retry http 429 during hydration - #205

Closed
mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:fix/kiosk-rate-limit
Closed

mrbobbytables wants to merge 2 commits into
projectbluefin:mainfrom
mrbobbytables:fix/kiosk-rate-limit

Conversation

@mrbobbytables

Copy link
Copy Markdown
Contributor

Summary

On first boot of a new node running the kiosk profile, the KubeStellar console dashboard mounts multiple resource cards simultaneously (/api/mcp/clusters, /api/mcp/services, /api/mcp/gpu-nodes/stream, /api/kagent/status, /api/stellar/state), triggering the console server's in-process client IP rate limiter (HTTP 429). Cards that hit this error display "Refresh failed" banners and give up after 3 consecutive attempts.

This change enhances files/k0s/kiosk/kiosk-gate.js to:

  • Intercept window.fetch calls to /api/ endpoints.
  • Stagger initial burst requests across a spacing window to prevent hammering the rate limiter simultaneously on page load.
  • Automatically retry HTTP 429 responses with exponential backoff (initial 500ms, backoff factor 2, max 4 retries) while honoring Retry-After response headers when present.
  • Transparently return the successful response to the calling application hooks so dashboard cards hydrate cleanly.
  • Pass through non-API requests (e.g. /health, static assets) unmodified.
  • Preserve existing kiosk gate invariants (agent health check gate, dialog, escape key handling).
  • Add unit contract and behavioral tests in tests/unit/test_kubestellar_kiosk.py.

Closes #199

— hive: backend=copilot model=gemini-3.8-flash

🐝 Hive Agent: contributor | SHA: 8d23905

…dration

On first boot, the KubeStellar console dashboard mounts multiple cards
simultaneously, firing concurrent requests to /api/ endpoints that trigger
the console server's in-process client IP rate limiter (HTTP 429).

Intercept /api/ fetch calls in kiosk-gate.js to:
- Stagger initial burst requests across an interval window
- Automatically retry HTTP 429 responses with exponential backoff while
  honoring Retry-After headers before bubbling up to cards
- Allow non-API requests (health checks, static assets) to pass through
  unmodified

Closes projectbluefin#199

Assisted-by: Gemini 3.8 Flash via GitHub Copilot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: mrbobbytables <mrbobbytables@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correctness

  • Interceptor may install after the SPA fires its initial burst (medium). kiosk-gate.js is injected as the last deferred script before </body> (files/k0s/kiosk/nginx.conf:46), so it executes after the console bundle. If the SPA dispatches its hydration fetches during its own script execution/mount (or captured a window.fetch reference at module scope), the very first-boot burst this PR targets is not intercepted — only subsequent card retries are (files/k0s/kiosk/kiosk-gate.js:89-92). The sandboxed node tests can't catch this; worth verifying against the real console (e.g. via tests/e2e) or documenting the ordering assumption.
  • Retry-After honored without an upper bound (low). parseRetryAfter returns whatever the server sends and the retry sleeps that long (kiosk-gate.js:105-115); a large header value stalls a card request for minutes with no cap. Suggest clamping to a max (e.g. 30s).

intent-alignment

  • Info: the PR body and issue #199 say "stagger initial burst requests," but the implementation serializes every concurrent /api/ burst for the lifetime of the page (lastDispatchTime is global, kiosk-gate.js:12, 47-55). Likely fine/intended, but it's broader than stated — periodic refresh bursts also pay the 75ms spacing.

No findings from: security, style, docs-currency.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

…delay

The interceptor was injected as a deferred script before </body>, so the
console bundle executed first and its first-boot hydration burst bypassed
the stagger/retry logic. Inject the gate synchronously at the end of
<head> instead: a classic non-deferred script runs before any deferred or
module script and before end-of-body scripts, guaranteeing window.fetch is
patched before the bundle evaluates. The gate UI loop now waits for
DOMContentLoaded because document.body does not exist at that point.

Also clamp Retry-After (and exponential backoff) to 30s so a large or
clock-skewed server value cannot stall a dashboard card for minutes, and
document that the stagger cursor is page-lifetime by design.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@hivecommons-hive hivecommons-hive Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docs-currency
docs/superpowers/plans/2026-09-08-kubestellar-kiosk-proxy.md still documents the old injection scheme this PR removes: lines 126–127 assert the </body>/<script defer> sub_filters and lines 193–194 show them in the nginx snippet, contradicting files/k0s/kiosk/nginx.conf:48 and the updated assertions in tests/unit/test_kubestellar_kiosk.py:60-66 at head 4ea4d1c. If plans are treated as living references rather than dated records, these blocks should be updated; otherwise fine to leave. (Low.)

style
kiosk-gate.js sleep() (lines 17–27) adds an abort listener to the caller's signal that is never removed when the timer resolves normally. On a kiosk page that lives for weeks, a component reusing one long-lived AbortController across periodic /api polls will accumulate listeners. Removing the listener in the resolve path would make it leak-free. (Info — most callers use per-request controllers.)

No findings from: correctness, security, intent-alignment.

Verified at head: new node behavioral test and updated nginx contract test pass locally; interceptor is same-origin /api-scoped (kiosk-gate.js:29-48), 429-only retry with clamped Retry-After matches the PR body, and the head-sync injection is required for patch-before-bundle ordering, so no scope creep.

— hive: agent=reviewer backend=copilot model=claude-fable-5 copilot=1.0.78

@castrojo

Copy link
Copy Markdown
Contributor

Closing as superseded by #210.

`files/k0s/kiosk/` is deleted; the kiosk assets now live at `files/cluster/manifests/kiosk/assets/`.

The rate-limit storm you diagnosed in #199 is worth keeping — if the console still self-limits on first boot against the new stack, please re-file the fetch stagger/429-backoff against `files/cluster/manifests/kiosk/assets/kiosk-gate.js` and I will take it. I did not port it blind because the request pattern changes once the dashboard is talking to real ControlPlanes instead of demo fixtures, and the fix should be measured against that.

@castrojo castrojo closed this Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(kiosk): console self-rate-limits on first boot (HTTP 429) causing card refresh failures

3 participants