Skip to content

Fix evalbench dashboard outage: render timeouts, precompute crash-loop, duplicate precompute writers - #579

Merged
prernakakkar-google merged 7 commits into
mainfrom
fix/mesop-dashboard-render-timeout
Aug 25, 2026
Merged

Fix evalbench dashboard outage: render timeouts, precompute crash-loop, duplicate precompute writers#579
prernakakkar-google merged 7 commits into
mainfrom
fix/mesop-dashboard-render-timeout

Conversation

@prernakakkar-google

Copy link
Copy Markdown
Collaborator

Summary
The dashboard at evalbench-463971716997.cr.gclb.goog was returning 500 on every /ui render (120s, the request timeout). Three independent causes, all fixed here. Post-deploy it renders in ~1.8–3.3s.

Root causes

  1. The render path stat'd the whole results mount, three times per page.
    Each render ran [d for d in os.listdir(p) if os.path.isdir(...)] in three places. os.path.isdir can't reuse anything os.listdir learned, so it costs one FUSE round trip per entry — ~25k entries × 3 walks ≈ 75k round trips per render, which alone exceeded the timeout.

  2. Precompute was crash-looping and had processed 695 of 24,969 runs.
    A run whose summary.csv lacked metric_name raised past the batch loop without being added to batch_processed, so it was rescanned on every pass, forever. The backlog never drained, and the cache the UI depends on never grew.

  3. Every Cloud Run instance ran its own precompute against the same bucket.
    maxScale=20, and supervisord starts the precompute loop in every instance — five were observed running identical passes over the same 22k directories. Beyond ~5× wasted work (dominated by one Gemini call per run), both caches are rewritten whole, so a later write silently discards an earlier one's rows while processed_dirs.json still records those runs as done. Anything lost that way is never reconsidered and never appears in trends again.

Changes

  1. viewer/run_index.py (new) — shared os.scandir-based listing with a 60s TTL cache. scandir carries the dirent type through, so is_dir() needs no second trip. Callers in main.py, trends.py, and precompute_dataset_quality.py; the clear-cache handler invalidates it.
  2. viewer/precompute_trends.py — return None was overloaded, meaning both "artifacts not written yet, retry" and "this threw"; the second is what jammed the queue. Now split: ParserError/EmptyDataError/OSError still defer (a run caught mid-write or a transient FUSE read deserves a retry), while anything else raises UnparsableRun and is quarantined as processed. Also fixes a second instance of the same bug on a missing run_time column.
  3. viewer/precompute_lease.py (new) + run_precompute.py — single-writer lease. Acquiring writes a holder ID and wall-clock timestamp, waits out the competing-write window, then reads back; a GCS whole-file rewrite has exactly one winner and reads are strongly consistent after it, so whoever reads their own ID back holds it. Renewed every 30s against a 120s TTL, so long passes keep the lease but a killed holder's lease is taken over — which matters, since a large backlog can get precompute SIGKILLed mid-pass. Fails open: if the lease can't be written, precompute runs unguarded, because a dashboard that silently stops updating is worse than duplicated work.
  4. ai_summary windowing — it was 95.8% of trends_cache.csv, on track for ~93MB read on every render once the backlog drained. Now kept inline only for the last 7 days and written to a per-run ai_summary.txt sidecar, which the detail view reads. Existing summaries are backfilled to sidecars before being blanked (~3.3MB → 209KB measured; ~93MB → ~5MB projected).
  5. supervisord_cloudrun.conf — PRECOMPUTE_WORKERS=12, PRECOMPUTE_INTERVAL=600. Precompute stays in the serving container so trends keep auto-updating.

@prernakakkar-google

Copy link
Copy Markdown
Collaborator Author

/gcbrun

Comment thread viewer/main.py Fixed
prernakakkar-google and others added 5 commits August 24, 2026 13:52
…rt' and 'import from''

Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…eCloudPlatform/evalbench into fix/mesop-dashboard-render-timeout
Comment thread viewer/precompute_trends.py Fixed
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
@prernakakkar-google

Copy link
Copy Markdown
Collaborator Author

/gcbrun

@helloeve helloeve left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

overall LGTM! Left a few non-blocking comments.

Comment thread viewer/precompute_lease.py
Comment thread viewer/precompute_trends.py
@prernakakkar-google
prernakakkar-google merged commit 1f23314 into main Aug 25, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants