Add CRCR downstream health metrics and critical tests with HUD metrics (L1/L2) by jewelkm89 · Pull Request #9 · pytorch/crcr-test

jewelkm89 · 2026-06-30T13:28:09Z

Summary

Turns pytorch/crcr-test into a downstream CRCR test repo. This repo contains test workflows and validators that run on live repository_dispatch events from the relay.

On upstream PyTorch PR opened/reopened, workflows validate dispatch delivery, PyTorch checkout fidelity, and (for L2) relay callbacks. Health check outcomes are sent to HUD via the existing completed callback test_results field (each integration check maps to passed/failed/total).

On synchronize and push, L1 runs a light probe (payload + checkout only, no callbacks). L2 is skipped. This tiered model reduces relay HTTP 429 rate-limit hits (#8) while keeping dispatch coverage on high-volume events.

flowchart TB
    PT[pytorch/pytorch PR or push] --> APP[CRCR GitHub App]
    APP --> RD[repository_dispatch to crcr-test]

    RD --> L1C{PR closed?}
    RD --> L2C{PR closed?}

    L1C -->|yes| L1X[L1 cancel-workflow]
    L1C -->|no| L1A{opened or reopened?}
    L1A -->|yes| L1F[L1 full probe + HUD]
    L1A -->|no| L1L[L1 light probe no callbacks]

    L2C -->|yes| L2X[L2 cancel-workflow]
    L2C -->|no| L2A{opened or reopened?}
    L2A -->|yes| L2[L2 full probe + HUD]
    L2A -->|no| L2S[L2 skipped]

    L1F --> RELAY[Callback lambda]
    L2 --> RELAY
    RELAY --> HUD[HUD crcr-test dashboard]

    OFF[Layer 0 offline unit tests] -.-> L1F
    OFF -.-> L2

Test layers overview

	L0 — Offline contract tests	L1 — Dispatch probe	L2 — Callback + HUD probe
Purpose	Keep validators/fixtures correct	Prove relay dispatches a usable payload	Prove callbacks work and HUD gets metrics
Workflow	`crcr-unit-tests.yml`	`crcr-dispatch-receiver.yml`	`oot-l2-ci.yml`
Trigger	Push / PR to `crcr-test`	Live `repository_dispatch` (`pull_request` + `push`)	Live `repository_dispatch` (`pull_request`, `opened`/`reopened` only)
Touches CRCR relay?	No	Yes (receives dispatch)	Yes (dispatch + callbacks)
Pushed to HUD?	No	Yes on `opened`/`reopened` only	Yes on `opened`/`reopened` only
When it runs	On changes to test code	Every upstream PyTorch PR/push when relay fires	Upstream PR `opened`/`reopened` only

Event filtering (relay rate limits)

The callback Lambda enforces a per-repo sliding-window limit (~20/min in prod today; 60/min after test-infra#8203 deploy). A full L1+L2 probe costs 4 callbacks. synchronize is high volume and was tripping 429s when every event sent full callbacks.

Event	L1	L2	Callbacks	HUD rows
`opened`, `reopened`	Full + HUD	Full + HUD	4	2
`synchronize`	Light (validate + checkout)	Skipped	0	0
`closed`	Cancel job	Cancel job	0	0
`push`	Light (validate + checkout)	N/A	0	0

Light probe still validates dispatch payload, checks out PyTorch, verifies SHA, and uploads health-report.json. Callback coverage remains on low-volume opened/reopened where the callback code path is identical.

Tests and checks by layer

L0 — Offline (`crcr-unit-tests.yml`)

#	Test	What it validates
1	PR opened fixture	Dispatch payload shape for `opened`
2	PR synchronize fixture	Payload shape for `synchronize`
3	PR closed fixture	Payload shape for `closed`
4	Push / ciflow tag fixture	Payload shape for `push`
5	Reject missing `delivery_id`	Validator catches bad payload
6	Reject wrong upstream repo	Must be `pytorch/pytorch`
7	Reject invalid SHA	40-char hex required
8	Callback `in_progress` shape	L2 wire format
9	Callback `completed` shape	L2 wire format + `conclusion`
10	Callback rejects missing `conclusion`	Completed must have conclusion
11	Checkout SHA mismatch detection	`validate_pytorch_checkout.py` logic
12	Health report to `test_results` mapping	passed/failed/total counts
13	Healthy / unhealthy report	`healthy`, `conclusion` fields
14	All fixtures through validator	End-to-end fixture sanity

L1 — Live dispatch probe (`crcr-dispatch-receiver.yml`)

#	Check	Full (`opened`/`reopened`)	Light (`synchronize`/`push`)
1	`validate_dispatch_payload`	Yes	Yes
2	`in_progress` callback	Yes	No
3	Checkout PyTorch @ dispatched SHA	Yes	Yes
4	`validate_pytorch_checkout`	Yes	Yes
5	Assert `delivery_id`	Yes	Yes
6	`completed` callback + HUD metrics	Yes	No
—	`cancel-workflow` (PR closed)	Cancel only	Cancel only

L2 — Live callback probe (`oot-l2-ci.yml`)

Runs only on opened/reopened.

#	Check	What it validates
1	`validate_dispatch_payload`	Same L1 contract on live payload
2	`in_progress` callback	State DISPATCHED to IN_PROGRESS
3	Checkout PyTorch @ PR head SHA	PR ref fetchable
4	`validate_pytorch_checkout`	Checkout matches head SHA
5	Smoke checks	README, .github, Python (deterministic)
6	`write_health_report` + `completed` callback	`test_results` + `conclusion` to HUD
7	`completed_callback` in final report	Completed callback succeeded
—	`cancel-workflow` (PR closed)	L2 build skipped on `closed`

One upstream event — what runs?

Upstream event	L0	L1	L2	Callbacks	HUD rows
PR `opened` / `reopened`	—	`l1-critical` full	`l2-critical`	4	2
PR `synchronize`	—	`l1-light`	—	0	0
PR `closed`	—	`cancel-workflow`	`cancel-workflow`	0	0
Push / ciflow tag	—	`l1-light`	—	0	0
Push/PR to crcr-test	`crcr-unit-tests`	—	—	0	0

Shared components by layer

Component	L0	L1	L2
`validate_dispatch_payload.py`	Fixtures	Live payload	Live payload
`validate_pytorch_checkout.py`	Unit test	Live checkout	Live checkout
`validate_callback_payload.py`	Unit test	—	—
`write_health_report.py`	Unit test	Live to HUD (full only)	Live to HUD
`cross-repo-ci-relay-callback`	—	Full probe only	Yes

Key changes

tests/ — payload validators, JSON fixtures, write_health_report.py, and 16 unit tests
write_health_report.py — shared health reporter used by both L1 and L2 probes; aggregates step outcomes into checks, maps them to HUD test_results, and feeds the completed callback (separate report and HUD row per probe)
crcr-dispatch-receiver.yml — L1 probe: full on opened/reopened, light on synchronize/push
oot-l2-ci.yml — L2 probe: opened/reopened only; deterministic checks + callbacks
crcr-unit-tests.yml — offline contract tests on push/PR to this repo
scripts/ — run-local-tests.sh and trigger-test-dispatch.sh for local/fork testing
README — documents probe model, event filtering, triggers, and HUD metric mapping

CRCR health is measured when the relay dispatches; offline unit tests guard validator/fixture drift on merge.

HUD health metrics

L1 and L2 full probes build a health report from step outcomes, then send it on the completed callback:

Step outcome	HUD `test_results`
`success`	`passed_tests`
`failure` / `cancelled`	`failed_tests`
`skipped`	`skipped_tests`

conclusion is success only when all checks pass. Per-check names are in the health-report.json workflow artifact. Verify rows at https://hud.pytorch.org/crcr/pytorch/crcr-test

Test plan

Pre-merge (offline):

cd crcr-test
./scripts/run-local-tests.sh

Pre-merge (CI):

crcr-unit-tests workflow passes on this PR

Post-merge (live CRCR):

Upstream PyTorch PR opened triggers L1 Health (full) and L2 Health workflows
Upstream PyTorch PR synchronize triggers L1 light only (no L2, no callbacks)
completed callback succeeds on opened/reopened (HTTP 2xx from relay)
HUD row shows passed_tests / failed_tests / total_tests matching probe checks
health-report.json artifact uploaded per run

Authored with an AI assistant.

Introduce payload validators, JSON fixtures, and unit tests that run on push/PR to this repo. Enhance the dispatch-receiver and L2 workflows to assert payload shape, PyTorch checkout SHA, and deterministic relay callbacks instead of random pass/fail simulation. Test Plan: ``` python3 -m unittest discover -s tests -p 'test_*.py' -v ``` Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>

Remove any scheduled-monitor direction from docs and strengthen L1/L2 workflows to record actual step outcomes, emit health-report.json artifacts per dispatch run, and assert closed-event cancel behavior. Test Plan: ``` python3 -m unittest discover -s tests -p 'test_*.py' -v ``` Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>

Test Plan: ``` ./scripts/run-local-tests.sh ``` Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>

Map each integration check to workflow.test_results (passed/failed/total) and post through the existing cross-repo-ci-relay-callback action. L1 now reports in_progress/completed callbacks as well as L2. Test Plan: ``` python3 -m unittest discover -s tests -p 'test_*.py' -v ``` Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>

jewelkm89 · 2026-07-06T10:05:40Z

Hi @atalman, can you please review the tests added for critical test check

Default to the current gh repo with fallback to pytorch/crcr-test instead of a hardcoded personal fork. CRCR_TEST_REPO still overrides explicitly. Test plan: ``` ./scripts/trigger-test-dispatch.sh --help # N/A; script has no --help git diff scripts/trigger-test-dispatch.sh ``` Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>

L1 runs a light probe (payload + checkout, no callbacks) on synchronize and push. L2 runs only on opened/reopened. Full L1+L2 HUD probes remain on low-volume PR events. Document tiered coverage in README. Test plan: ``` ./scripts/run-local-tests.sh ``` Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>

atalman · 2026-07-07T14:51:11Z

@@ -1,12 +1,12 @@
-name: OOT L2 CI


Lets change OOT -> CRCR here as well

atalman

A few concise inline notes — no blockers. Top two: add a positive-path checkout test (#1) and de-duplicate the health-report invocation (#2).

atalman · 2026-07-07T15:19:23Z

+
+
+class TestPytorchCheckout(unittest.TestCase):
+    def test_checkout_matches_dispatch_sha(self) -> None:


Name says matches but this only tests the mismatch path (asserts non-zero + "checkout SHA mismatch"). The happy path (HEAD == dispatched SHA -> exit 0) is never covered, so a validator that always reported mismatch would still pass. Add a positive case.

atalman · 2026-07-07T15:19:23Z

+          fi
+          echo "delivery_id=${DELIVERY_ID}"
+
+      - name: L1 — build health metrics for HUD


write_health_report.py is invoked twice per workflow with a near-identical --check list (also L1 line ~175, and L2 lines ~100/~128). Adding/renaming a check means editing 4 blocks that will drift — consider factoring the check list out.

atalman · 2026-07-07T15:19:23Z

+
+  l2-critical:
+    if: >-
+      github.event.client_payload.payload.action != 'closed' &&


Redundant: contains(["opened","reopened"], action) already excludes closed, so action != 'closed' && is dead. Drop it.

atalman · 2026-07-07T15:19:24Z

+            --check "delivery_id=${{ steps.delivery.outcome }}" \
+            --strict
+
+      - name: L1 — write full health report with callback outcome


This strict report overwrites health-report.json with 6 checks (incl. completed_callback), but HUD test_results came from the earlier health_metrics step (5 checks). Artifact total (6) won't match HUD total_tests (5) — worth a comment noting it's inherent to reporting the callback's own outcome.

atalman · 2026-07-07T15:19:24Z

+
+    expect_status = sys.argv[1]
+    if expect_status not in ALLOWED_STATUSES:
+        print(f"::error::status arg must be in_progress or completed", file=sys.stderr)


f-string with no placeholder — drop the f.

meta-cla Bot added the cla signed label Jun 30, 2026

jewelkm89 marked this pull request as draft June 30, 2026 13:31

jewelkm89 and others added 3 commits July 6, 2026 10:15

Add local test and manual dispatch helper scripts.

aae7b37

Test Plan: ``` ./scripts/run-local-tests.sh ``` Authored with an AI assistant. Co-authored-by: Cursor <cursoragent@cursor.com>

jewelkm89 changed the title ~~Add L1/L2 critical integration tests for CRCR~~ Add CRCR downstream health probes with HUD metrics (L1/L2) Jul 6, 2026

jewelkm89 marked this pull request as ready for review July 6, 2026 08:26

jewelkm89 requested review from atalman, can-gaa-hou and subinz1 July 6, 2026 09:51

jewelkm89 changed the title ~~Add CRCR downstream health probes with HUD metrics (L1/L2)~~ Add CRCR downstream health metrics and critical tests with HUD metrics (L1/L2) Jul 6, 2026

jewelkm89 and others added 2 commits July 6, 2026 15:50

atalman reviewed Jul 7, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add CRCR downstream health metrics and critical tests with HUD metrics (L1/L2)#9

Add CRCR downstream health metrics and critical tests with HUD metrics (L1/L2)#9
jewelkm89 wants to merge 6 commits into
pytorch:mainfrom
jewelkm89:crcr-critical-tests

jewelkm89 commented Jun 30, 2026 •

edited

Loading

Uh oh!

jewelkm89 commented Jul 6, 2026

Uh oh!

atalman Jul 7, 2026

Uh oh!

atalman left a comment

Uh oh!

atalman Jul 7, 2026

Uh oh!

atalman Jul 7, 2026

Uh oh!

atalman Jul 7, 2026

Uh oh!

atalman Jul 7, 2026

Uh oh!

atalman Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		class TestPytorchCheckout(unittest.TestCase):
		def test_checkout_matches_dispatch_sha(self) -> None:

Uh oh!

Conversation

jewelkm89 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test layers overview

Event filtering (relay rate limits)

Tests and checks by layer

L0 — Offline (crcr-unit-tests.yml)

L1 — Live dispatch probe (crcr-dispatch-receiver.yml)

L2 — Live callback probe (oot-l2-ci.yml)

One upstream event — what runs?

Shared components by layer

Key changes

HUD health metrics

Test plan

Uh oh!

jewelkm89 commented Jul 6, 2026

Uh oh!

atalman Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

atalman Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

atalman Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

atalman Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

atalman Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

atalman Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jewelkm89 commented Jun 30, 2026 •

edited

Loading

L0 — Offline (`crcr-unit-tests.yml`)

L1 — Live dispatch probe (`crcr-dispatch-receiver.yml`)

L2 — Live callback probe (`oot-l2-ci.yml`)