Skip to content

fix(evals): keep large benchmark viewers bounded - #478

Merged
aamir merged 1 commit into
mainfrom
fix/bounded-benchmark-viewer
Sep 1, 2026
Merged

fix(evals): keep large benchmark viewers bounded#478
aamir merged 1 commit into
mainfrom
fix/bounded-benchmark-viewer

Conversation

@aamir

@aamir aamir commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Large trace-derived evals now finish packaging instead of crashing while generating their optional HTML review viewer. Small benchmarks remain self-contained; large benchmarks retain capture bodies in their existing sidecar files and load them through understudy traces serve --benchmark.

Root cause

The viewer embedded every normalized capture and serialized the entire payload into one JavaScript string. The reproduced workload's 7,036 captures totaled more than Node's roughly 512 MiB single-string limit, so build-benchmark stopped before writing its final manifest and goal event even though the core eval artifacts had already been created.

What changed

  • Record each capture sidecar's serialized byte size while writing it.
  • Inline capture bodies only for benchmarks with at most 250 captures and 16 MiB of capture data.
  • Keep large viewers bounded by embedding sidecar references and showing the existing local-server command.
  • Cover both byte-count and capture-count limits with a regression test.

Validation

  • node --test tests/trace-foundry.test.mjs — 62/62 passed.
  • Production-scale reconstruction — 3,728 tasks and 7,036 captures produced a 25,375,154-byte viewer with capture bodies omitted.
  • npm run typecheck
  • npm run skills:validate — 43 public skills passed.
  • npm run package:smoke
  • git diff --check

Compound Engineering


Devin Review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-01T02:08:56.416257Z 02c8522 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@aamir
aamir merged commit d1ce244 into main Sep 1, 2026
15 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant