Skip to content

benchmark-memory outgrew its 60-minute budget: raise it or shrink the suite? #627

Open

Description

@zackees

Owner decision needed (CLAUDE.md rule 7). Found while fixing the stuck weekly benchmarks in #626.

ci/check_benchmark_memory_workflow.py enforces build-and-measure timeout-minutes == 60 as a hard budget (with negative controls for 61 and 45). The suite no longer fits:

run date run external memory suite step
33315579361 08-30 43m11s, passed
33820641684 09-04 48m52s, passed
35512774807 09-20 cut off at 52m56s by the 60 min job timeout (job 1h0m22s)

The Bun row (commit 2f2791f, 09-03) made it five allocators. Builds add about 7 minutes (native libs 6m08s, suite 54s). The 09-06 and 09-13 runs were cut off the same way.

#626 fixes the separate cause of the 09-27, 10-04 and 10-25 cancellations, a cron collision with benchmark-latency in the shared concurrency group, and leaves the timeout at 60.

Options:

  1. Raise the budget to about 90 minutes, which gives roughly 25% headroom over the measured trend, and update the checker's bounds.
  2. Keep 60 and cut the suite's blocks or cells (for example, fewer blocks per allocator) to fit.
  3. Split the suite across two jobs.

Also worth doing in any case: the suite prints nothing while it runs, so a hang can't be told apart from slowness. A per-block progress line would fix that.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions