Skip to content

feat: automate runner runtime updates #149

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
svelderrainruiz merged 1 commit into from
Aug 21, 2025
Merged

feat: automate runner runtime updates #149

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions .github/workflows/update-runner-runtime.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
name: Update Runner Runtime

on:
workflow_dispatch:
inputs:
runner_key:
description: 'Key for the runner entry in requirements files'
required: true
runner_label:
description: 'New runner_label value'
required: false
runner_type:
description: 'New runner_type value'
required: false
skip_dry_run:
description: 'Set skip_dry_run for the runner'
required: false
type: boolean
default: false

jobs:
update:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}

- name: Update requirements
shell: bash
env:
RUNNER_KEY: ${{ inputs.runner_key }}
RUNNER_LABEL: ${{ inputs.runner_label }}
RUNNER_TYPE: ${{ inputs.runner_type }}
SKIP_DRY_RUN: ${{ inputs.skip_dry_run }}
run: |
node - <<'NODE'
const fs = require('fs');
const key = process.env.RUNNER_KEY;
const label = process.env.RUNNER_LABEL;
const type = process.env.RUNNER_TYPE;
const skipInput = process.env.SKIP_DRY_RUN;
const files = fs.readdirSync(process.cwd()).filter(f => f.startsWith('requirements') && f.endsWith('.json'));
let found = false;
for (const file of files) {
const json = JSON.parse(fs.readFileSync(file, 'utf8'));
if (json.runners && json.runners[key]) {
if (label) json.runners[key].runner_label = label;
if (type) json.runners[key].runner_type = type;
if (skipInput !== '') json.runners[key].skip_dry_run = skipInput === 'true';
fs.writeFileSync(file, JSON.stringify(json, null, 2) + '\n');
fs.appendFileSync(process.env.GITHUB_ENV, `REQ_FILE=${file}\n`);
found = true;
console.log(`Updated ${file}`);
break;
}
}
if (!found) {
throw new Error(`Runner ${key} not found`);
}
NODE

- name: Commit changes
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git add "$REQ_FILE"
git commit -m "chore: update runner runtime for ${{ inputs.runner_key }}" || echo "No changes to commit"
git push

- name: Validate
run: |
npm run derive:registry
npm run check:traceability || true
actionlint
70 changes: 2 additions & 68 deletions artifacts/linux/requirements-summary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,78 +1,12 @@
### Requirement Summary
| Requirement ID | Description | Owner | Total Tests | Passed | Failed | Skipped | Pass Rate (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| REQ-023 | Parser ingests JUnit XML artifacts starting at the testsuites root and iterating through nested suites and testcases. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-024 | Top-level testsuites attributes name, tests, errors, failures, disabled, and time are captured for summary reporting. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-025 | Each testsuite records attributes including name, tests, errors, failures, hostname, id, skipped, disabled, package, and time. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-026 | Suite properties are extracted as name/value pairs for environment details. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-027 | Testcase attributes name, status, classname, assertions, time, and any skip message are preserved. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-028 | Requirement identifiers embedded in testcase names are detected and associated with the test. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-029 | Test results are aggregated by requirement and by suite to count passed, failed, and skipped cases. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-030 | Traceability matrix links requirement IDs to testcases with status, execution time, host properties, and skipped reasons. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-031 | Parsing logic validates presence of required fields and reports missing or malformed data. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-032 | Parser tolerates and retains unknown attributes for future extensibility. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-033 | Tests ending with SelfHosted.Workflow.Tests.ps1 execute only in dry run mode unless the workflow targets a self-hosted Windows runner labeled self-hosted-windows-lv. | | 1 | 1 | 0 | 0 | 100.00 |
| Unmapped | | | 48 | 48 | 0 | 0 | 100.00 |
| Unmapped | | | 4 | 4 | 0 | 0 | 100.00 |

### Requirement Testcases
| Requirement ID | Test ID | Status |
| --- | --- | --- |
| REQ-023 | parses-nested-junit-structures | Passed |
| REQ-024 | captures-root-testsuites-attributes | Passed |
| REQ-025 | captures-testsuite-attributes | Passed |
| REQ-026 | captures-suite-properties | Passed |
| REQ-027 | captures-testcase-attributes-and-skipped-message | Passed |
| REQ-028 | extracts-requirement-identifiers | Passed |
| REQ-029 | aggregates-status-by-requirement-and-suite | Passed |
| REQ-030 | builds-traceability-matrix-with-skipped-reasons | Passed |
| REQ-031 | validates-missing-fields | Passed |
| REQ-032 | preserves-unknown-attributes | Passed |
| REQ-033 | throws-error-for-malformed-xml | Passed |
| Unmapped | associates-classname-with-requirement | Passed |
| Unmapped | buildissuebranchname-formats-branch-name | Passed |
| Unmapped | buildissuebranchname-rejects-non-numeric-input | Passed |
| Unmapped | buildprofile1.iconeditor.addtokentolabview.dispatcher.dry-runs-add-token-to-labview-with-expected-arguments- | Passed |
| Unmapped | buildprofile1.iconeditor.applyvipc.dispatcher.dry-runs-apply-vipc-with-expected-arguments- | Passed |
| Unmapped | buildprofile1.iconeditor.buildvipackage.dispatcher.dry-runs-build-vi-package-with-expected-arguments- | Passed |
| Unmapped | buildprofile1.iconeditor.closelabview.dispatcher.dry-runs-close-labview-with-expected-arguments- | Passed |
| Unmapped | buildsummary-splits-totals-by-os | Passed |
| Unmapped | collecttestcases-captures-requirement-property | Passed |
| Unmapped | collecttestcases-uses-evidence-property-and-falls-back-to-directory-scan | Passed |
| Unmapped | collecttestcases-uses-machine-name-property-for-owner | Passed |
| Unmapped | computestatuscounts-tallies-test-statuses | Passed |
| Unmapped | detects-downloaded-artifacts-path | Passed |
| Unmapped | dispatchers-and-parameters-include-descriptions | Passed |
| Unmapped | errors-when-strict-unmapped-mode-enabled | Passed |
| Unmapped | escapemarkdown-escapes-special-characters | Passed |
| Unmapped | escapemarkdown-leaves-plain-text-untouched | Passed |
| Unmapped | fails-when-commit-lacks-requirement-reference | Passed |
| Unmapped | fails-when-requirement-lacks-test-coverage | Passed |
| Unmapped | fails-when-tests-are-unmapped | Passed |
| Unmapped | fails-when-tests-reference-unknown-requirements | Passed |
| Unmapped | formaterror-handles-plain-objects | Passed |
| Unmapped | formaterror-handles-primitives | Passed |
| Unmapped | formaterror-handles-real-error-objects | Passed |
| Unmapped | formaterror-handles-unstringifiable-values | Passed |
| Unmapped | generate-ci-summary-features | Passed |
| Unmapped | groups-owners-and-includes-requirements-and-evidence | Passed |
| Unmapped | grouptomarkdown-omits-numeric-identifiers | Passed |
| Unmapped | grouptomarkdown-supports-optional-limit-for-truncation | Passed |
| Unmapped | handles-root-level-testcases | Passed |
| Unmapped | handles-zipped-junit-artifacts | Passed |
| Unmapped | ignores-stale-junit-files-outside-artifacts-path | Passed |
| Unmapped | loadrequirements-logs-warning-on-invalid-json | Passed |
| Unmapped | loadrequirements-merges-multiple-files | Passed |
| Unmapped | loadrequirements-warns-and-skips-invalid-entries | Passed |
| Unmapped | logs-a-warning-when-no-junit-files-are-found | Passed |
| Unmapped | partitions-requirement-groups-by-runner\_type | Passed |
| Unmapped | passes-with-coverage-and-requirement-reference | Passed |
| Unmapped | requirementssummarytomarkdown-escapes-pipes-in-description | Passed |
| Unmapped | skips-invalid-junit-files-and-still-generates-summary | Passed |
| Unmapped | summarytomarkdown-handles-no-tests | Passed |
| Unmapped | summarytomarkdown-sorts-os-alphabetically-and-escapes-special-characters | Passed |
| Unmapped | throws-when-no-junit-files-found-and-strict-mode-enabled | Passed |
| Unmapped | uses-latest-artifact-directory-when-multiple-are-present | Passed |
| Unmapped | warns-when-all-tests-are-unmapped | Passed |
| Unmapped | writeerrorsummary-appends-error-details-to-summary-file | Passed |
| Unmapped | writeerrorsummary-skips-summary-file-for-non-error-throws | Passed |
| Unmapped | writes-outputs-to-os-specific-directory | Passed |
| Unmapped | buildprofile1.iconeditor.closelabview.dispatcher.dry-runs-close-labview-with-expected-arguments- | Passed |
4 changes: 2 additions & 2 deletions artifacts/linux/summary-standard.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
### Test Summary
| OS | Passed | Failed | Skipped | Duration (s) | Pass Rate (%) |
| --- | --- | --- | --- | --- | --- |
| overall | 59 | 0 | 0 | 21.58 | 100.00 |
| linux | 59 | 0 | 0 | 21.58 | 100.00 |
| overall | 4 | 0 | 0 | 0.67 | 100.00 |
| linux | 4 | 0 | 0 | 0.67 | 100.00 |

_For detailed per-test information, see [traceability-standard.md](traceability-standard.md)._
17 changes: 3 additions & 14 deletions artifacts/linux/summary.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,12 @@
### Test Summary
| OS | Passed | Failed | Skipped | Duration (s) | Pass Rate (%) |
| --- | --- | --- | --- | --- | --- |
| overall | 59 | 0 | 0 | 21.58 | 100.00 |
| linux | 59 | 0 | 0 | 21.58 | 100.00 |
| overall | 4 | 0 | 0 | 0.67 | 100.00 |
| linux | 4 | 0 | 0 | 0.67 | 100.00 |

### Requirement Summary
| Requirement ID | Description | Owner | Total Tests | Passed | Failed | Skipped | Pass Rate (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| REQ-023 | Parser ingests JUnit XML artifacts starting at the testsuites root and iterating through nested suites and testcases. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-024 | Top-level testsuites attributes name, tests, errors, failures, disabled, and time are captured for summary reporting. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-025 | Each testsuite records attributes including name, tests, errors, failures, hostname, id, skipped, disabled, package, and time. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-026 | Suite properties are extracted as name/value pairs for environment details. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-027 | Testcase attributes name, status, classname, assertions, time, and any skip message are preserved. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-028 | Requirement identifiers embedded in testcase names are detected and associated with the test. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-029 | Test results are aggregated by requirement and by suite to count passed, failed, and skipped cases. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-030 | Traceability matrix links requirement IDs to testcases with status, execution time, host properties, and skipped reasons. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-031 | Parsing logic validates presence of required fields and reports missing or malformed data. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-032 | Parser tolerates and retains unknown attributes for future extensibility. | | 1 | 1 | 0 | 0 | 100.00 |
| REQ-033 | Tests ending with SelfHosted.Workflow.Tests.ps1 execute only in dry run mode unless the workflow targets a self-hosted Windows runner labeled self-hosted-windows-lv. | | 1 | 1 | 0 | 0 | 100.00 |
| Unmapped | | | 48 | 48 | 0 | 0 | 100.00 |
| Unmapped | | | 4 | 4 | 0 | 0 | 100.00 |

_For detailed per-test information, see [traceability.md](traceability.md)._
116 changes: 2 additions & 114 deletions artifacts/linux/traceability-standard.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,122 +1,10 @@
### Test Traceability Matrix

#### REQ-023 (100% passed)
#### Unmapped (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-023 | parses-nested-junit-structures | Passed | 0.023 | | |

#### REQ-024 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-024 | captures-root-testsuites-attributes | Passed | 0.007 | | |

#### REQ-025 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-025 | captures-testsuite-attributes | Passed | 0.012 | | |

#### REQ-026 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-026 | captures-suite-properties | Passed | 0.001 | | |

#### REQ-027 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-027 | captures-testcase-attributes-and-skipped-message | Passed | 0.001 | | |

#### REQ-028 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-028 | extracts-requirement-identifiers | Passed | 0.011 | | |

#### REQ-029 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-029 | aggregates-status-by-requirement-and-suite | Passed | 0.002 | | |

#### REQ-030 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-030 | builds-traceability-matrix-with-skipped-reasons | Passed | 0.002 | | |

#### REQ-031 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-031 | validates-missing-fields | Passed | 0.001 | | |

#### REQ-032 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-032 | preserves-unknown-attributes | Passed | 0.001 | | |

#### REQ-033 (100% passed)

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| REQ-033 | throws-error-for-malformed-xml | Passed | 0.003 | | |

<details><summary>Unmapped (100% passed)</summary>

| Requirement | Test ID | Status | Duration (s) | Owner | Evidence |
| --- | --- | --- | --- | --- | --- |
| Unmapped | associates-classname-with-requirement | Passed | 0.028 | | |
| Unmapped | buildissuebranchname-formats-branch-name | Passed | 0.001 | | |
| Unmapped | buildissuebranchname-rejects-non-numeric-input | Passed | 0.002 | | |
| Unmapped | buildprofile1.iconeditor.addtokentolabview.dispatcher.dry-runs-add-token-to-labview-with-expected-arguments- | Passed | 0.482 | | |
| Unmapped | buildprofile1.iconeditor.applyvipc.dispatcher.dry-runs-apply-vipc-with-expected-arguments- | Passed | 0.042 | | |
| Unmapped | buildprofile1.iconeditor.buildvipackage.dispatcher.dry-runs-build-vi-package-with-expected-arguments- | Passed | 0.106 | | |
| Unmapped | buildprofile1.iconeditor.closelabview.dispatcher.dry-runs-close-labview-with-expected-arguments- | Passed | 0.036 | | |
| Unmapped | buildsummary-splits-totals-by-os | Passed | 0.002 | | |
| Unmapped | collecttestcases-captures-requirement-property | Passed | 0.017 | | |
| Unmapped | collecttestcases-uses-evidence-property-and-falls-back-to-directory-scan | Passed | 0.015 | | |
| Unmapped | collecttestcases-uses-machine-name-property-for-owner | Passed | 0.023 | | |
| Unmapped | computestatuscounts-tallies-test-statuses | Passed | 0.000 | | |
| Unmapped | detects-downloaded-artifacts-path | Passed | 1.200 | | |
| Unmapped | dispatchers-and-parameters-include-descriptions | Passed | 0.009 | | |
| Unmapped | errors-when-strict-unmapped-mode-enabled | Passed | 1.251 | | |
| Unmapped | escapemarkdown-escapes-special-characters | Passed | 0.001 | | |
| Unmapped | escapemarkdown-leaves-plain-text-untouched | Passed | 0.000 | | |
| Unmapped | fails-when-commit-lacks-requirement-reference | Passed | 1.259 | | |
| Unmapped | fails-when-requirement-lacks-test-coverage | Passed | 1.342 | | |
| Unmapped | fails-when-tests-are-unmapped | Passed | 1.237 | | |
| Unmapped | fails-when-tests-reference-unknown-requirements | Passed | 1.214 | | |
| Unmapped | formaterror-handles-plain-objects | Passed | 0.000 | | |
| Unmapped | formaterror-handles-primitives | Passed | 0.000 | | |
| Unmapped | formaterror-handles-real-error-objects | Passed | 0.005 | | |
| Unmapped | formaterror-handles-unstringifiable-values | Passed | 0.001 | | |
| Unmapped | generate-ci-summary-features | Passed | 0.039 | | |
| Unmapped | groups-owners-and-includes-requirements-and-evidence | Passed | 1.376 | | |
| Unmapped | grouptomarkdown-omits-numeric-identifiers | Passed | 0.002 | | |
| Unmapped | grouptomarkdown-supports-optional-limit-for-truncation | Passed | 0.001 | | |
| Unmapped | handles-root-level-testcases | Passed | 0.001 | | |
| Unmapped | handles-zipped-junit-artifacts | Passed | 0.941 | | |
| Unmapped | ignores-stale-junit-files-outside-artifacts-path | Passed | 1.003 | | |
| Unmapped | loadrequirements-logs-warning-on-invalid-json | Passed | 0.013 | | |
| Unmapped | loadrequirements-merges-multiple-files | Passed | 0.014 | | |
| Unmapped | loadrequirements-warns-and-skips-invalid-entries | Passed | 0.004 | | |
| Unmapped | logs-a-warning-when-no-junit-files-are-found | Passed | 1.165 | | |
| Unmapped | partitions-requirement-groups-by-runner\_type | Passed | 0.966 | | |
| Unmapped | passes-with-coverage-and-requirement-reference | Passed | 1.244 | | |
| Unmapped | requirementssummarytomarkdown-escapes-pipes-in-description | Passed | 0.001 | | |
| Unmapped | skips-invalid-junit-files-and-still-generates-summary | Passed | 1.497 | | |
| Unmapped | summarytomarkdown-handles-no-tests | Passed | 0.001 | | |
| Unmapped | summarytomarkdown-sorts-os-alphabetically-and-escapes-special-characters | Passed | 0.001 | | |
| Unmapped | throws-when-no-junit-files-found-and-strict-mode-enabled | Passed | 0.829 | | |
| Unmapped | uses-latest-artifact-directory-when-multiple-are-present | Passed | 0.989 | | |
| Unmapped | warns-when-all-tests-are-unmapped | Passed | 1.412 | | |
| Unmapped | writeerrorsummary-appends-error-details-to-summary-file | Passed | 0.007 | | |
| Unmapped | writeerrorsummary-skips-summary-file-for-non-error-throws | Passed | 0.006 | | |
| Unmapped | writes-outputs-to-os-specific-directory | Passed | 1.730 | | |

</details>
| Unmapped | buildprofile1.iconeditor.closelabview.dispatcher.dry-runs-close-labview-with-expected-arguments- | Passed | 0.036 | | |
Loading
Loading