Skip to content

perf(report): avoid copying accumulated Markdown file results - #1814

Open
jxnxts wants to merge 2 commits into
e2e-inc:mainfrom
jxnxts:jxnxts/markdown-grouping-1802
Open

jxnxts wants to merge 2 commits into
e2e-inc:mainfrom
jxnxts:jxnxts/markdown-grouping-1802

Conversation

@jxnxts

@jxnxts jxnxts commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

What and why

  • fileGroups in packages/e2e/src/report/markdown.ts copied every earlier entry of a target/file group each time it added one, so a group of n results did O(n²) copying. Each group now owns one array that results are pushed into.
  • The comparator, result order, 400-row display cap, 60,000-character limit and rendered bytes are unchanged. No public API, schema or option changed. The changeset is a patch line, as the triage comment on Performance: avoid repeatedly copying Markdown result groups #1802 asked for when output is identical.
const grouped = byFile.get(entry.result.file);
if (grouped) grouped.push(entry);
else byFile.set(entry.result.file, [entry]);
  • No tests are added: packages/e2e/tests/unit/report-markdown.test.ts is unchanged from main.

Issue

Fixes #1802

Verified

Ran it locally: yes

  • Built renderMarkdownReport from main b56e8ed5 and from this change, then compared output for ten schema-valid workloads: identical bytes. An independent reviewer checked another 49 frozen cases, including stored CLI reports, truncation and shared references, with identical output and unchanged inputs. Since b56e8ed5, main has changed only import paths in these two files; the grouping hunks apply with zero fuzz.
  • Ran both built CLIs with the repository's synthetic engine on two targets with --repeat-each 2. Both wrote schema-valid reports and summaries with 14 selected, 12 executed, 8 passed, 4 planted failures and 2 skipped, and both exited 1 because of the planted failures. No browser or model was used.
  • Both checks above ran on the earlier let/if form of the loop, before this revision. The shorter form does the same work, because a bucket is never empty once it exists.

Elapsed milliseconds per full render of the built renderer, from that b56e8ed5 comparison (Node 24.22.2, Apple M3 Pro, eight warmups, nine alternating paired samples, medians):

Workload main this change
100 results, one file 0.311 0.309
1,000 results, one file 2.831 2.281
10,000 results, one file 57.078 21.394
1,000 results, 100 files 2.312 2.305
10,000 results, 100 files 21.777 22.042

These numbers are renderer time only. Dense one-file groups improve; small and spread-out reports show no consistent difference. They say nothing about suite, CI or memory.

pnpm bench:ab was not run. It invokes e2e run --reporter json (scripts/bench-ab.ts:187), replacing the built-in Markdown reporter, and strips GITHUB_* (scripts/bench-ab.ts:152), so the configured GitHub reporter also returns before rendering. It does not exercise the Markdown renderer.

This revision on Node 24.22.2 (local commit 0d710d2f): the Markdown and reporter tests (report-markdown.test.ts, reporters.test.ts and the GitHub reporter.test.ts), pnpm build and pnpm check passed. The first full pnpm test ran at a load average of 26 to 30 and failed 5 tests in mcp-test-run.test.ts and screenshot-assertions.test.ts. Run alone, screenshot-assertions passed, while mcp-test-run failed first on a tap that timed out after 235 ms, then on a session-slot cascade. The second full run, at a load of about 5, passed every e2e test but failed 27 web CDP and download tests, because the Playwright Chromium binary was missing until partway through that run. With the browser confirmed present, the four affected web files passed (32 tests), and one more full pnpm test exited 0: e2e 4,673 passed, 11 skipped; web 832 passed, 1 skipped; every later package and test:scripts passed.

View guided diff

Append each result to one array per target/file group instead of copying
every earlier entry on each append. The comparator, result order, display
cap and rendered output are unchanged.

Fixes e2e-inc#1802
@jxnxts
jxnxts requested a review from okwasniewski as a code owner November 15, 2026 19:36
@github-actions github-actions Bot added the size:S Up to 29 changed lines, not counting tests, fixtures, or the lockfile label Nov 15, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 3 files

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

@jxnxts

jxnxts commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

CI is green on a9db22b, and the independent review found no blocking issues. Local build, checks and the full test suite passed. The account does not have permission to add the Ready for Human Review label, so applying it needs a maintainer. No merge has been requested.

@okwasniewski okwasniewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Automated review.

Thanks, the fix is right. A randomized comparison of renderMarkdownReport on main vs this branch (1500 documents, up to 3000 results, several targets, repeats, hitting the display cap and the char limit) gave byte-identical output. CI is green.

Two changes before merge:

  1. Please drop the two new tests and the freeze helper. The existing tests already fail on every mutation of the changed lines (dropped push, reversed order, fresh array each time). The new tests mostly pin things this PR does not touch: input immutability, which the renderer never broke, and the file/target tie-break, which is unchanged.
  2. Make the changeset a user-facing line, e.g. "The Markdown summary renders faster for files with many results. Output is unchanged."

Optional: const grouped = byFile.get(entry.result.file); if (grouped) grouped.push(entry); else byFile.set(entry.result.file, [entry]); is shorter than the let/if block.

…geset

Remove the two added tests and the freeze helper, as the existing
Markdown tests already cover the changed lines. Reword the changeset for
users and shorten the grouping loop; output and order are unchanged.
@github-actions github-actions Bot added size:XS Up to 9 changed lines, not counting tests, fixtures, or the lockfile and removed size:S Up to 29 changed lines, not counting tests, fixtures, or the lockfile labels Nov 15, 2026
@jxnxts

jxnxts commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. Both changes are in:

  • Dropped the two new tests and the freeze helper. report-markdown.test.ts is now identical to main.
  • The changeset now reads "The Markdown summary renders faster for files with many results. Output is unchanged."
  • Took the optional if (grouped) grouped.push(entry); else byFile.set(...) form.

Locally on Node 24.22.2, pnpm build, pnpm check and the Markdown and GitHub reporter tests pass, and a full pnpm test passes. Two earlier full runs failed on this machine: one under heavy load in MCP session and screenshot tests, and one while the Playwright Chromium binary was missing.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 3 files (changes from recent commits).

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

View guided diff | Re-trigger cubic

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS Up to 9 changed lines, not counting tests, fixtures, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance: avoid repeatedly copying Markdown result groups

2 participants