Create L1_L2_CoverageReport#217
Open
nhanasi wants to merge 7 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new GitHub Actions workflow intended to generate L1 (unit) and L2 (functional) coverage metrics on PRs, publish artifacts, regenerate coverage markdown docs, and post/update a PR comment with a summary.
Changes:
- Introduces a multi-job PR workflow to compute and upload L1 and L2 coverage-related artifacts.
- Regenerates L1/L2 coverage markdown files in-repo and (for same-repo PRs) commits them back to the PR branch.
- Upserts a sticky PR comment containing the latest coverage summary.
Comment on lines
+141
to
+144
| - name: Build and run L2 tests | ||
| run: | | ||
| docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib: && sh run_l2.sh" | ||
|
|
Comment on lines
+246
to
+250
| - name: Checkout code | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
Comment on lines
+269
to
+272
| GENERATED_DATE="$(date -u +%Y-%m-%d)" | ||
|
|
||
| cat > test/functional-tests/docs/L1_Test_Coverage.md << EOF | ||
| # Remote Debugger L1 Coverage Report |
Comment on lines
+7
to
+9
| permissions: | ||
| contents: write | ||
| pull-requests: write |
Comment on lines
+240
to
+244
| update-docs: | ||
| name: Update L1/L2 coverage markdowns | ||
| runs-on: ubuntu-latest | ||
| needs: [l1-coverage, l2-coverage] | ||
|
|
Comment on lines
+269
to
+271
| GENERATED_DATE="$(date -u +%Y-%m-%d)" | ||
|
|
||
| cat > test/functional-tests/docs/L1_Test_Coverage.md << EOF |
| test_functions=$(grep -R "^def test_" remote_debugger/test/functional-tests/tests/test_*.py | wc -l | tr -d ' ') | ||
| test_functions_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*def test_" remote_debugger/test/functional-tests/tests/test_*.py | wc -l | tr -d ' ' || true) | ||
|
|
||
| src_functions=$(grep -R "^[A-Za-z_][A-Za-z0-9_[:space:]\*]*([^;]*)[[:space:]]*{" remote_debugger/src/*.c | wc -l | tr -d ' ') |
Comment on lines
+165
to
+170
| c_api_scenarios=$(grep -n "^[[:space:]]*Scenario:" remote_debugger/test/functional-tests/features/rrd_c_api_upload.feature | wc -l | tr -d ' ') | ||
| c_api_tests=$(grep -n "^def test_" remote_debugger/test/functional-tests/tests/test_rrd_c_api_upload.py | wc -l | tr -d ' ') | ||
| c_api_gap=$((c_api_scenarios - c_api_tests)) | ||
| if [ "$c_api_gap" -lt 0 ]; then | ||
| c_api_gap=0 | ||
| fi |
Comment on lines
+412
to
+414
| - name: Upsert PR coverage comment | ||
| uses: actions/github-script@v7 | ||
| env: |
Comment on lines
+15
to
+16
| container: | ||
| image: ghcr.io/rdkcentral/docker-rdk-ci:latest |
Comment on lines
+127
to
+130
| - name: Pull required docker images | ||
| run: | | ||
| docker pull ghcr.io/rdkcentral/docker-device-mgt-service-test/mockxconf:latest | ||
| docker pull ghcr.io/rdkcentral/docker-device-mgt-service-test/native-platform:latest |
Code Coverage Summary |
Code Coverage Summary |
|
|
||
| - name: Build and run L2 tests | ||
| run: | | ||
| docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib: && sh run_l2.sh" |
Comment on lines
+165
to
+170
| c_api_scenarios=$(grep -n "^[[:space:]]*Scenario:" remote_debugger/test/functional-tests/features/rrd_c_api_upload.feature | wc -l | tr -d ' ') | ||
| c_api_tests=$(grep -n "^def test_" remote_debugger/test/functional-tests/tests/test_rrd_c_api_upload.py | wc -l | tr -d ' ') | ||
| c_api_gap=$((c_api_scenarios - c_api_tests)) | ||
| if [ "$c_api_gap" -lt 0 ]; then | ||
| c_api_gap=0 | ||
| fi |
Comment on lines
+269
to
+271
| GENERATED_DATE="$(date -u +%Y-%m-%d)" | ||
|
|
||
| cat > test/functional-tests/docs/L1_Test_Coverage.md << EOF |
Comment on lines
+412
to
+416
| - name: Upsert PR coverage comment | ||
| uses: actions/github-script@v7 | ||
| env: | ||
| COMMENT_PATH: /tmp/pr_coverage_comment.md | ||
| with: |
Comment on lines
+272
to
+276
| # Remote Debugger L1 Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** \`remotedebugger\` | ||
| **Source:** PR pipeline coverage run |
Comment on lines
+298
to
+302
| # Remote Debugger L2 Test Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** \`remotedebugger\` | ||
| **Test Suite:** \`test/functional-tests\` |
Comment on lines
+7
to
+9
| permissions: | ||
| contents: write | ||
| pull-requests: write |
Comment on lines
+15
to
+16
| container: | ||
| image: ghcr.io/rdkcentral/docker-rdk-ci:latest |
Code Coverage Summary |
Comment on lines
+7
to
+9
| permissions: | ||
| contents: write | ||
| pull-requests: write |
Comment on lines
+240
to
+244
| update-docs: | ||
| name: Update L1/L2 coverage markdowns | ||
| runs-on: ubuntu-latest | ||
| needs: [l1-coverage, l2-coverage] | ||
|
|
Comment on lines
+31
to
+35
| COV_FILE="src/unittest/coverage.info" | ||
| if [ ! -f "$COV_FILE" ]; then | ||
| echo "coverage.info was not generated at $COV_FILE" | ||
| exit 1 | ||
| fi |
| test_functions=$(grep -R "^def test_" remote_debugger/test/functional-tests/tests/test_*.py | wc -l | tr -d ' ') | ||
| test_functions_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*def test_" remote_debugger/test/functional-tests/tests/test_*.py | wc -l | tr -d ' ' || true) | ||
|
|
||
| src_functions=$(grep -R "^[A-Za-z_][A-Za-z0-9_[:space:]\*]*([^;]*)[[:space:]]*{" remote_debugger/src/*.c | wc -l | tr -d ' ') |
Comment on lines
+272
to
+276
| # Remote Debugger L1 Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** `remotedebugger` | ||
| **Source:** PR pipeline coverage run |
Comment on lines
+292
to
+294
| - Coverage is generated by running `sh run_ut.sh --enable-cov`. | ||
| - Source artifact: `src/unittest/coverage.info`. | ||
| - This file is updated automatically on pull requests. |
Comment on lines
+298
to
+302
| # Remote Debugger L2 Test Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** `remotedebugger` | ||
| **Test Suite:** `test/functional-tests` |
| name: l1-coverage-artifacts | ||
| path: | | ||
| src/unittest/coverage.info | ||
| /tmp/coverage_summary.txt |
Comment on lines
+328
to
+330
| 1. `rrd_c_api_upload.feature` scenarios: $C_API_SCENARIOS | ||
| `test_rrd_c_api_upload.py` test functions: $C_API_TESTS | ||
| **Current gap:** $C_API_GAP |
Comment on lines
+7
to
+9
| permissions: | ||
| contents: write | ||
| pull-requests: write |
Comment on lines
+134
to
+135
| docker run -d --name mockxconf -p 50050:50050 -p 50051:50051 -p 50052:50052 -p 50054:50054 -v ${{ github.workspace }}:/mnt/L2_CONTAINER_SHARED_VOLUME ghcr.io/rdkcentral/docker-device-mgt-service-test/mockxconf:latest | ||
| docker run -d --name native-platform --link mockxconf -v ${{ github.workspace }}:/mnt/L2_CONTAINER_SHARED_VOLUME ghcr.io/rdkcentral/docker-device-mgt-service-test/native-platform:latest |
Comment on lines
+165
to
+170
| c_api_scenarios=$(grep -n "^[[:space:]]*Scenario:" remote_debugger/test/functional-tests/features/rrd_c_api_upload.feature | wc -l | tr -d ' ') | ||
| c_api_tests=$(grep -n "^def test_" remote_debugger/test/functional-tests/tests/test_rrd_c_api_upload.py | wc -l | tr -d ' ') | ||
| c_api_gap=$((c_api_scenarios - c_api_tests)) | ||
| if [ "$c_api_gap" -lt 0 ]; then | ||
| c_api_gap=0 | ||
| fi |
Comment on lines
+269
to
+271
| GENERATED_DATE="$(date -u +%Y-%m-%d)" | ||
|
|
||
| cat > test/functional-tests/docs/L1_Test_Coverage.md <<EOF |
Comment on lines
+272
to
+276
| # Remote Debugger L1 Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** `remotedebugger` | ||
| **Source:** PR pipeline coverage run |
Comment on lines
+298
to
+302
| # Remote Debugger L2 Test Coverage Report | ||
|
|
||
| **Generated:** $GENERATED_DATE | ||
| **Component:** `remotedebugger` | ||
| **Test Suite:** `test/functional-tests` |
Code Coverage Summary |
2 similar comments
Code Coverage Summary |
Code Coverage Summary |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (7)
.github/workflows/L1_L2_CoverageReport.yml:9
- Workflow-level
contents: writegrants a write-scopedGITHUB_TOKENto all jobs that execute PR-provided scripts (e.g.,run_ut.sh,cov_build.sh,run_l2.sh). This increases blast radius if a PR branch is compromised. Also, PR comment creation uses the Issues API (notpull-requests), soissuespermission is the relevant scope. Prefer least-privilege at workflow level and grant write only to the job that pushes/docs/comments.
permissions:
contents: write
pull-requests: write
.github/workflows/L1_L2_CoverageReport.yml:244
update-docsneeds elevated token scopes to push commits and create/update the PR comment. With workflow-level permissions reduced, explicitly grantcontents: writeandissues: writeto this job only (principle of least privilege).
update-docs:
name: Update L1/L2 coverage markdowns
runs-on: ubuntu-latest
needs: [l1-coverage, l2-coverage]
.github/workflows/L1_L2_CoverageReport.yml:250
- In
pull_requestworkflows,actions/checkoutdefaults to the merge ref (refs/pull/*/merge). Committing on top of that merge commit and pushing togithub.head_refis typically rejected as non-fast-forward. Check out the PR head SHA/branch explicitly before committing.
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
.github/workflows/L1_L2_CoverageReport.yml:143
$LD_LIBRARY_PATHis expanded on the runner before being passed into the container (because it is inside a double-quotedbash -cstring). Escape the$so the variable is expanded insidenative-platform, preserving the container's existingLD_LIBRARY_PATH.
docker exec -i native-platform /bin/bash -c "cd /mnt/L2_CONTAINER_SHARED_VOLUME/remote_debugger && sh cov_build.sh && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/x86_64-linux-gnu:/lib/aarch64-linux-gnu:/usr/local/lib: && sh run_l2.sh"
.github/workflows/L1_L2_CoverageReport.yml:159
- With
set -euo pipefail, thegrep | wc -lpipelines will fail the step when there are zero matches (grep exits 1) or when the globbed files don't exist. This makes the workflow flaky (e.g., if there are temporarily noScenario:lines). Make these counts robust and default to 0.
feature_files=$(find remote_debugger/test/functional-tests/features -name '*.feature' | wc -l | tr -d ' ')
feature_scenarios=$(grep -R "^[[:space:]]*Scenario:" remote_debugger/test/functional-tests/features/*.feature | wc -l | tr -d ' ')
feature_scenarios_disabled=$(grep -R "^[[:space:]]*#[[:space:]]*Scenario:" remote_debugger/test/functional-tests/features/*.feature | wc -l | tr -d ' ' || true)
test_files=$(find remote_debugger/test/functional-tests/tests -maxdepth 1 -name 'test_*.py' | wc -l | tr -d ' ')
.github/workflows/L1_L2_CoverageReport.yml:416
- On fork-based PRs,
pull_requestworkflows run with a read-onlyGITHUB_TOKEN, soissues.createComment/issues.updateCommentwill typically 403 and fail the job. If you want forks to pass CI, skip the PR comment upsert when the PR comes from a fork (or move commenting to apull_request_targetworkflow).
- name: Upsert PR coverage comment
uses: actions/github-script@v7
env:
COMMENT_PATH: /tmp/pr_coverage_comment.md
with:
.github/workflows/L1_L2_CoverageReport.yml:16
- Using floating
:latesttags for the CI container and test images makes runs non-reproducible and increases supply-chain risk (image content can change without any repo change). Prefer pinning to an immutable digest (or at least a versioned tag) fordocker-rdk-ci,mockxconf, andnative-platform.
container:
image: ghcr.io/rdkcentral/docker-rdk-ci:latest
Comment on lines
+232
to
+239
| - name: Upload L2 artifacts | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: l2-coverage-artifacts | ||
| path: | | ||
| /tmp/L2_TEST_RESULTS | ||
| /tmp/l2_metrics.env | ||
|
|
Code Coverage Summary |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.