-
Notifications
You must be signed in to change notification settings - Fork 12
ci: Move realm server tests into shards #4269
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
797772e
Run chunks of realm-server tests on ci
IanCal f1139a3
Run multiple modules as a group in CI
IanCal 3cc9065
Call original test command
IanCal ebd54d2
Merge branch 'main' into server/group-tests-continued
backspace 7c01d8d
Fix test
backspace 2e79398
Fix lint errors
backspace 5527bb7
Replace manual sharding with script
backspace 3def21c
Add reporting
backspace 633390c
Add missed script
backspace 08b469d
Fix lint error
backspace d1d5c38
Merge remote-tracking branch 'origin/main' into server/group-tests-co…
backspace 63a1472
Restore deleted paragraph
backspace b466d12
Merge remote-tracking branch 'origin/main' into server/group-tests-co…
backspace a9d868c
Remove test shards lintinng
backspace 0e6e753
Change to six shards
backspace 191a8cd
Add deliberate test failure
backspace 6af24f2
Remove custom shard name
backspace 5ad24c3
Remove deliberate test failure
backspace 8520c5d
Change matrix variables to match
backspace 48365ea
Fix ordering
backspace 521e7e6
Add multi-file test explanation to AGENTS.md
backspace 49e2c98
Add filter on merge realm server reports job
backspace eace2d9
Fix grouping by name
backspace File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -116,6 +116,8 @@ | |||||
| `pnpm test` | ||||||
| - Run a single module: | ||||||
| `TEST_MODULE=card-endpoints-test.ts pnpm test-module` | ||||||
| - Run a list of modules: | ||||||
| `TEST_MODULES=card-endpoints-test.ts|another-module-test.ts pnpm test` | ||||||
|
||||||
| `TEST_MODULES=card-endpoints-test.ts|another-module-test.ts pnpm test` | |
| `TEST_MODULES="card-endpoints-test.ts|another-module-test.ts" pnpm test` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,111 @@ | ||
| #!/usr/bin/env node | ||
| /* eslint-env node */ | ||
| 'use strict'; | ||
|
|
||
| // QUnit JUnit XML reporter for CI. | ||
| // | ||
| // Usage: qunit --reporter junit-reporter.js ... | ||
| // Or: qunit --require ./scripts/junit-reporter.js --reporter console ... | ||
| // | ||
| // Set JUNIT_OUTPUT_FILE to control the output path (default: junit/realm-server.xml). | ||
| // When used as a --require module, this attaches alongside the default console | ||
| // reporter so you get both terminal output and a JUnit file. | ||
|
|
||
| const fs = require('node:fs'); // eslint-disable-line @typescript-eslint/no-var-requires | ||
| const path = require('node:path'); // eslint-disable-line @typescript-eslint/no-var-requires | ||
| const QUnit = require('qunit'); // eslint-disable-line @typescript-eslint/no-var-requires | ||
|
|
||
| const outputFile = | ||
| process.env.JUNIT_OUTPUT_FILE || | ||
| path.join(process.cwd(), '..', '..', 'junit', 'realm-server.xml'); | ||
|
|
||
| const suites = new Map(); // moduleName -> { tests, failures, errors, time, testCases } | ||
|
|
||
| function escapeXml(str) { | ||
| return String(str) | ||
| .replace(/&/g, '&') | ||
| .replace(/</g, '<') | ||
| .replace(/>/g, '>') | ||
| .replace(/"/g, '"') | ||
| .replace(/'/g, '''); | ||
| } | ||
|
|
||
| QUnit.on('testEnd', (data) => { | ||
| const moduleName = data.module || 'default'; | ||
| const testName = data.name || 'unknown'; | ||
| const runtime = (data.runtime || 0) / 1000; // ms → seconds | ||
| const status = data.status; // passed, failed, skipped, todo | ||
|
|
||
| if (!suites.has(moduleName)) { | ||
| suites.set(moduleName, { | ||
| tests: 0, | ||
| failures: 0, | ||
| errors: 0, | ||
| skipped: 0, | ||
| time: 0, | ||
| testCases: [], | ||
| }); | ||
| } | ||
|
|
||
| const suite = suites.get(moduleName); | ||
| suite.tests++; | ||
| suite.time += runtime; | ||
|
|
||
| let caseXml = ` <testcase classname="${escapeXml(moduleName)}" name="${escapeXml(testName)}" time="${runtime.toFixed(3)}"`; | ||
|
|
||
| if (status === 'failed') { | ||
| suite.failures++; | ||
| const messages = (data.errors || []) | ||
| .map((e) => { | ||
| let msg = e.message || ''; | ||
| if (e.actual !== undefined && e.expected !== undefined) { | ||
| msg += `\nExpected: ${JSON.stringify(e.expected)}\nActual: ${JSON.stringify(e.actual)}`; | ||
| } | ||
| if (e.stack) { | ||
| msg += `\n${e.stack}`; | ||
| } | ||
| return msg; | ||
| }) | ||
| .join('\n---\n'); | ||
| caseXml += `>\n <failure message="${escapeXml((data.errors?.[0]?.message || 'test failed').slice(0, 200))}">${escapeXml(messages)}</failure>\n </testcase>`; | ||
| } else if (status === 'skipped' || status === 'todo') { | ||
| suite.skipped++; | ||
| caseXml += `>\n <skipped />\n </testcase>`; | ||
| } else { | ||
| caseXml += ` />`; | ||
| } | ||
|
|
||
| suite.testCases.push(caseXml); | ||
| }); | ||
|
|
||
| QUnit.on('runEnd', () => { | ||
| const suitesXml = []; | ||
| let totalTests = 0; | ||
| let totalFailures = 0; | ||
| let totalErrors = 0; | ||
| let totalSkipped = 0; | ||
| let totalTime = 0; | ||
|
|
||
| for (const [name, suite] of suites) { | ||
| totalTests += suite.tests; | ||
| totalFailures += suite.failures; | ||
| totalErrors += suite.errors; | ||
| totalSkipped += suite.skipped; | ||
| totalTime += suite.time; | ||
|
|
||
| suitesXml.push( | ||
| ` <testsuite name="${escapeXml(name)}" tests="${suite.tests}" failures="${suite.failures}" errors="${suite.errors}" skipped="${suite.skipped}" time="${suite.time.toFixed(3)}">\n${suite.testCases.join('\n')}\n </testsuite>`, | ||
| ); | ||
| } | ||
|
|
||
| const xml = [ | ||
| `<?xml version="1.0" encoding="UTF-8"?>`, | ||
| `<testsuites name="Realm Server Tests" tests="${totalTests}" failures="${totalFailures}" errors="${totalErrors}" skipped="${totalSkipped}" time="${totalTime.toFixed(3)}">`, | ||
| ...suitesXml, | ||
| `</testsuites>`, | ||
| ].join('\n'); | ||
|
|
||
| const dir = path.dirname(outputFile); | ||
| fs.mkdirSync(dir, { recursive: true }); | ||
| fs.writeFileSync(outputFile, xml, 'utf8'); | ||
| }); |
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This step hardcodes the total shard count as
6even though the matrix already definesshardTotal. To avoid drift if the shard count changes later, pass${{ matrix.shardTotal }}toshard-test-modules.jsinstead of a constant.