Skip to content

fix(ci): cap back-end jest workers to prevent OOM on CI runner#828

Merged
diegomayorga-dept merged 2 commits into
developmentfrom
fix/backend-jest-oom-cdk-workers
Jul 6, 2026
Merged

fix(ci): cap back-end jest workers to prevent OOM on CI runner#828
diegomayorga-dept merged 2 commits into
developmentfrom
fix/backend-jest-oom-cdk-workers

Conversation

@diegomayorga-dept

Copy link
Copy Markdown
Contributor

Title*

fix(ci): cap back-end jest workers to prevent OOM on CI runner

Type of Change*

  • Bug fix

Description

The Build & Deploy Back-End job in cicd-release-quality (and the sandbox/UAT release pipelines) started failing deterministically on development after PR #823 (fix(EGV-206): upgrade aws-cdk-lib), which bumped aws-cdk-lib from ^2.176.0 to ^2.260.0.

Root cause is an out-of-memory kill, not a test failure:

  • The packages/back-end/test/infra/** suites synthesize full CDK stacks (e.g. easy-genomics-nested-stack, auth-nested-stack), each consuming ~1.4 GB of heap.
  • Back-end Jest runs with default parallelism (cpuCount - 1 → 3 workers on the 4-vCPU ubuntu-latest runner), each worker inheriting NODE_OPTIONS=--max-old-space-size=8192 and running with v8 coverage enabled.
  • The larger aws-cdk-lib raised each synth's footprint just enough that the aggregate RSS across the parallel workers crossed the 16 GB runner limit. The OS OOM-killer then SIGKILLed the process group — which is why the run exits 1 with no Jest summary, output cut off mid-flush, and no V8 "Reached heap limit" message.

This is why the failure was deterministic and re-running the pipeline did not help.

The fix reduces peak memory without touching application code, all via the projen source of truth (.projenrc.ts) which regenerates the back-end config:

  • maxWorkers: '50%' — caps concurrent Jest workers so the heavy infra synth suites don't all run at once (2 workers on the CI runner instead of 3).
  • --workerIdleMemoryLimit=2GB — recycles a worker once it exceeds 2 GB, freeing accumulated coverage/synth memory between suites instead of letting it grow for the worker's lifetime.

Testing*

  • Ran the full back-end suite locally with coverage enabled and the new flags. Before the fix the run was OOM-killed (exit 1, no summary); after the fix it completes cleanly: 66/66 test suites passed, 491/491 tests passed, exit 0, with per-worker heap bounded at ~1.4 GB (workers recycle at the 2 GB idle limit rather than accumulating).
  • Verified projen regenerated the config correctly:
    • packages/back-end/package.jsonjest.maxWorkers: "50%"
    • packages/back-end/.projen/tasks.jsonjest --workerIdleMemoryLimit=2GB --passWithNoTests --updateSnapshot
  • No test code changed; all existing tests still pass.

Impact

  • CI/build only — unblocks the back-end build/deploy pipeline. No runtime, infrastructure, or application behavior changes.
  • Back-end test wall-clock may increase slightly on CI due to reduced parallelism, in exchange for a stable, non-OOM run.
  • No new dependencies.
  • Follow-up (out of scope): the underlying driver is the ever-growing infra synth footprint (same trend behind the nested-stack resource-limit work). Splitting the nested stack remains the durable path; this change buys headroom now.

Additional Information

  • Changed files: .projenrc.ts (source), plus the two projen-generated files packages/back-end/package.json and packages/back-end/.projen/tasks.json. Regenerate with pnpm exec projen if editing .projenrc.ts.
  • --workerIdleMemoryLimit is passed via extraCliOptions because it is not exposed in projen's JestConfigOptions type; maxWorkers is set directly in jestConfig.

Checklist*

  • No new errors or warnings have been introduced.
  • All tests pass successfully and new tests added as necessary.
  • Documentation has been updated accordingly. (N/A — CI config change)
  • Code adheres to the coding and style guidelines of the project.
  • Code has been commented in particularly hard-to-understand areas.

@diegomayorga-dept diegomayorga-dept merged commit dd2cb84 into development Jul 6, 2026
2 checks passed
@diegomayorga-dept diegomayorga-dept deleted the fix/backend-jest-oom-cdk-workers branch July 6, 2026 19:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants