Skip to content

Change CI to two fixed seed runs - #305

Merged
skyw merged 2 commits into
mainfrom
skyw/update_ci_test_randomness
Sep 29, 2026
Merged

skyw merged 2 commits into
mainfrom
skyw/update_ci_test_randomness

Conversation

@skyw

@skyw skyw commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

We have been using two rounds of test in CI, one with fixed seed and one without. As we adding more complex tests, the error margin starts vary more with different seed. If we keep increasing threshold to allow maximum variance caused by random seed to pas, it can start gating real errors and defeat the purpose of having multiple rounds in CI.

The decision is to change to two rounds both with fixed seed. And more rounds can be added trivially by just adding more seed into the loop.

Signed-off-by: Hao Wu <skyw@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@skyw

skyw commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 4c2847d

@greptile-apps

greptile-apps Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Medium risk] Changes test execution strategy from random to fixed seeds.

The PR appears safe to merge; the previously reported seed mismatch is fixed and no new actionable issue was established.

Findings

  1. P2 Documented CI seeds differ ▶

Summary

The PR replaces the random L0 GPU test pass with a second fixed-seed pass and updates contributor guidance to name the CI seeds.

Reviews (2) · Last reviewed commit: "update docs"

Comment thread CLAUDE.md Outdated
Signed-off-by: Hao Wu <skyw@nvidia.com>
@skyw
skyw requested a review from mkhona-nvidia September 29, 2026 20:21
@skyw

skyw commented Sep 29, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test b085453

@skyw
skyw enabled auto-merge (squash) September 29, 2026 21:13
@github-actions

Copy link
Copy Markdown

Test Results

   99 files  +  2    198 suites  +7   2m 10s ⏱️ -3s
1 464 tests ±  0  1 464 ✅ ±  0  0 💤 ±0  0 ❌ ±0 
3 507 runs  +163  3 507 ✅ +163  0 💤 ±0  0 ❌ ±0 

Results for commit b085453. ± Comparison against base commit 840ae70.

@skyw
skyw merged commit 8b775b5 into main Sep 29, 2026
31 checks passed
@codecov

codecov Bot commented Sep 29, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@skyw
skyw deleted the skyw/update_ci_test_randomness branch September 29, 2026 21:14

This branch was successfully deployed

2 active deployments
public — b0854537 Deployed Sep 29, 2026 by copy-pr-bot[bot] via release / finalize / notify #269
test — b0854537 Deployed Sep 29, 2026 by copy-pr-bot[bot] via cicd-wait-in-queue #1059
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants