Fix evalbench dashboard outage: render timeouts, precompute crash-loop, duplicate precompute writers - #579
Merged
Conversation
prernakakkar-google
requested review from
IsmailMehdi and
helloeve
as code owners
August 24, 2026 13:51
Collaborator
Author
|
/gcbrun |
…rt' and 'import from'' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
…eCloudPlatform/evalbench into fix/mesop-dashboard-render-timeout
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Collaborator
Author
|
/gcbrun |
helloeve
approved these changes
Aug 24, 2026
helloeve
left a comment
Collaborator
There was a problem hiding this comment.
overall LGTM! Left a few non-blocking comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The dashboard at evalbench-463971716997.cr.gclb.goog was returning 500 on every /ui render (120s, the request timeout). Three independent causes, all fixed here. Post-deploy it renders in ~1.8–3.3s.
Root causes
The render path stat'd the whole results mount, three times per page.
Each render ran [d for d in os.listdir(p) if os.path.isdir(...)] in three places. os.path.isdir can't reuse anything os.listdir learned, so it costs one FUSE round trip per entry — ~25k entries × 3 walks ≈ 75k round trips per render, which alone exceeded the timeout.
Precompute was crash-looping and had processed 695 of 24,969 runs.
A run whose summary.csv lacked metric_name raised past the batch loop without being added to batch_processed, so it was rescanned on every pass, forever. The backlog never drained, and the cache the UI depends on never grew.
Every Cloud Run instance ran its own precompute against the same bucket.
maxScale=20, and supervisord starts the precompute loop in every instance — five were observed running identical passes over the same 22k directories. Beyond ~5× wasted work (dominated by one Gemini call per run), both caches are rewritten whole, so a later write silently discards an earlier one's rows while processed_dirs.json still records those runs as done. Anything lost that way is never reconsidered and never appears in trends again.
Changes