[Szer]

What this adds

.github/workflows/workflow-canary.yml — a scheduled sentinel that watches the repo's scheduled workflows via gh run list and pages PagerDuty (Events API v2, EU host) when one fails or stalls. Plain curl + gh + jq; no LLM, no VPN. Hourly cron (50 * * * *), workflow_dispatch for manual runs.

Watched workflows (static table in the workflow — add a workflow = add a row)

Workflow file	Display name	Stall threshold
monitor.yml	Monitor	5h30m
project.yml	Daily Project Assessment	26h
product.yml	Product Analysis	98h
vahter-mirror-reconcile.yml	Vahter: Mirror Reconcile	26h

Classification

• failed — the latest completed run (any event, cancelled skipped) concluded failure / timed_out / startup_failure. A manual-dispatch success counts as healthy for the failed check.
• stalled — judged on schedule-event runs only: no non-cancelled schedule run started/completed within the threshold. In-progress/queued schedule runs count as fired; a run cancelled via aks-vpn sharing does not count as fresh.
• healthy — otherwise.

Manual-dispatch handling decision: the failed check uses the latest completed run across events (so a manual-dispatch success makes the failed check green), while the stall clock resets only on schedule-event runs.

PagerDuty

• One dedup_key per workflow: bots/<workflow-file>; trigger on failed/stalled, resolve sent unconditionally on healthy (resolving a dedup with no open incident is harmless). PD service has auto_resolve_timeout = null, so every trigger has a matching resolve path.
• Severity per dedup key: warning normally; if ≥2 of the three agent workflows (monitor, project, product) are failed/stalled in the same poll, their triggers carry severity: critical. Mirror Reconcile is warning only and never part of the critical count. When the count drops back to one, the next trigger for that key returns to warning.
• Payload: source = Szer/bots/<file>, one-line summary (display name + state + run URL or "no run in N hours"), component = file, group = github-actions, custom_details = latest run id / conclusion / created_at / threshold.
• Secret PAGERDUTY_ROUTING_KEY; if unset, prints PAGERDUTY_ROUTING_KEY not set, skipping and exits 0. Non-2xx from PD prints the response body and fails the step; no blind retries.
• Canary's own failure: last step if: failure() triggers bots/workflow-canary (warning); the first step of every run resolves that key.

How to test

• Dry run: gh workflow run workflow-canary.yml --repo Szer/bots --ref <branch> -f dry_run=true — prints the classification table and every event that would be sent; sends nothing.
• Warning roundtrip: dispatch with test_event=warning-roundtrip, dry_run=false — sends one warning trigger + resolve on bots/canary-selftest. Never sends critical from a test path.
• Simulate the critical path: on a dry_run=true dispatch, temporarily lower one agent workflow's threshold in the watch table (e.g. set monitor.yml to 60|1m) so it goes stalled alongside another, and observe severity=critical in the printed would-send lines. Revert before merging.

Notes

• permissions: actions: read only; no checkout, no contents.
• timeout-minutes: 10, runs-on: ubuntu-latest.
• Scheduled runs send for real (inputs default empty → live mode); workflow_dispatch defaults to dry_run=true.
• Not registered in .github/bots.yml; no existing workflow touched. Optional in-workflow if: failure() hooks in the agent workflows are a separate follow-up.

🤖 Generated with Claude Code

[Szer]

Dry-run classification evidence

gh workflow run from the PR branch is blocked by GitHub: the dispatch API returns 404 {"message":"Not Found"} because the workflow does not exist on the default branch yet (GitHub only dispatches workflows that exist on main; ours is only on ci/workflow-canary-pagerduty). Same 404 by file name and by display name.

So I ran the exact classify step script (extracted verbatim from this workflow's run: block) locally against the live Szer/bots Actions API at 2026-09-02T12:56:13Z. This is what the canary would have printed on a dry_run=true dispatch:

=== workflow canary classification 2026-09-02T12:56:13Z ===
workflow                     | display                  | state    | run_id      | conclusion     | created_at           | threshold
monitor.yml                  | Monitor                  | healthy  | 33631264938 | success        | 2026-09-02T12:40:40Z | 5h30m
project.yml                  | Daily Project Assessment | healthy  | 33625521447 | success        | 2026-09-02T11:36:55Z | 26h
product.yml                  | Product Analysis         | failed   | 33497518479 | failure        | 2026-09-01T10:27:40Z | 98h
vahter-mirror-reconcile.yml  | Vahter: Mirror Reconcile | healthy  | 33603549937 | success        | 2026-09-02T07:26:06Z | 26h
agent_bad=1 critical=false

Would-send events (dry-run output of the send step):

[dry-run] would resolve bots/monitor.yml (Monitor)
[dry-run] would resolve bots/project.yml (Daily Project Assessment)
[dry-run] would trigger bots/product.yml severity=warning summary=Product Analysis: failed — https://github.com/Szer/bots/actions/runs/33497518479
[dry-run] would resolve bots/vahter-mirror-reconcile.yml (Vahter: Mirror Reconcile)

Not all four are healthy today. product.yml is failed: its latest completed run (09-01T10:27Z, pre-fix b386ecd) concluded failure in the Run product agent step (openai/codex-action@v1 — the same 08-21→09-02 breakage fixed by #449, which merged 09-02T11:33Z). Its next schedule run is Friday 09-05; until then the canary will classify it failed and send a warning trigger for bots/product.yml each poll. No action taken — reporting only, per the test plan.

Monitor and Project show as healthy because the latest completed run across events is the post-fix manual dispatch (#449) — the designed "manual-dispatch success counts as healthy for the failed check" behavior; their stall clocks are fine (monitor's 12:40 schedule run is post-fix and green).

The critical path and mirror-exclusion rules were verified locally with a stubbed gh: with 2+ of monitor/project/product bad the triggers carry severity=critical; vahter-mirror-reconcile.yml stays warning even when it is also bad (verified all-4-bad case: agent_bad=3, mirror still warning).

Post-merge, the hourly schedule will run for real; a dry_run=true dispatch from main will print the same table in the run log.