Skip to content

feat(fizruk): per-game status details, starting with Factorio research #516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Szer merged 1 commit into from
Sep 25, 2026
Merged

feat(fizruk): per-game status details, starting with Factorio research #516

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions .github/canary-pagerduty-handoff.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Handoff: PagerDuty service + routing key for the GitHub Actions workflow canary

**For:** the agent doing terraform work in `~/git/my-infra`.
**From:** the canary-implementer in `Szer/bots` (this repo).
**Status:** awaiting your terraform work — do not start the bots-side canary until the items in **"What I need back"** below are answered.

---

## 1. Why this exists (context)

All scheduled agent workflows in `Szer/bots` silently failed for ~1.5 weeks (08-21 → 09-02):
`openai/codex-action@v1` moved its tag to a commit (86365089) that rejects our
`codex-args` network override under `safety-strategy: drop-sudo`, breaking every
`codex-action` invocation (Monitor 4h, Project daily, Product Tue/Fri, SRE). The runs
went **red but nobody was notified** — there is no human-notification path for a
failed/stalled workflow in this repo.

We are building a **canary**: an independent sentinel workflow in `Szer/bots` that runs
on its own schedule, checks `gh run list` for every scheduled workflow (failed OR
stalled = no run in longer than its cadence), and on a problem calls **PagerDuty Events
API v2** (`POST https://events.pagerduty.com/v2/enqueue`, plain `curl`, `dedup_key` +
auto-`resolve` on recovery) so one incident pages once and self-closes.

The sentinel needs a **PagerDuty routing key**. That key is provisioned by PagerDuty for
an **Events API v2 integration** on a PagerDuty service. The PagerDuty configuration
lives in your terraform repo (`~/git/my-infra`). This document is the exact work order.

## 2. What I need you to do (terraform side)

1. **Explore `~/git/my-infra/terraform/`** — read `pagerduty.tf`, `alerts-monitoring.tf`,
`provider.tf`, `outputs.tf`, and any module dir they reference. Report the existing
PagerDuty shape: what services exist, what integrations each has, what escalation
policies exist, and how outputs/secrets are conventionally declared.

2. **Decide: new dedicated service vs reuse.** My recommendation is a **dedicated
"GitHub Actions"** service with its own Events API v2 integration, so GitHub
workflow alerts are a separate, quiet channel from your infra alerts. If you find a
strong reason to reuse an existing service instead (e.g., an existing Events API v2
routing key that already pages the right people), say so and justify — but a
dedicated key is preferred.

3. **Implement terraform** (follow the repo's existing conventions — modules, naming,
tags, provider config, output style):
- (if new) a `pagerduty_service` for "GitHub Actions";
- a `pagerduty_service_integration` with `type = "events_api_v2_inbound_integration"`.
The `integration_key` is **auto-generated by PagerDuty** (read-only attribute);
- expose the routing key as a **sensitive** terraform output (e.g.
`output "github_actions_routing_key" { value = <integration_key> ; sensitive = true }`);
- attach/reuse an **escalation policy** so the incident actually pages the right
people — confirm which one, or flag if a new one is needed.

4. **Do NOT run terraform locally.** The repo's CI does `plan` on PR and `apply` on
merge. Open a my-infra PR that plans cleanly.

5. **Security:** never print the routing key anywhere — not in logs, PR title/body,
comments, or commits. The key will live in terraform state (sensitive attribute);
that is acceptable. You do **not** need to deliver the key value to me — the repo
owner will copy it into a GitHub secret (see below). You only need to tell me *how to
retrieve it*.

6. **Do not touch anything outside terraform** in my-infra. Do not modify the bots repo.
Do not re-delegate.

## 3. What I need back (the blocking questions for my side)

Answer these explicitly in your final report — the bots-side canary cannot proceed
until they are answered:

1. **Approach chosen** — new dedicated service, or reused existing one (name + PD
service ID either way).
2. **Integration type** — confirm `events_api_v2_inbound_integration` and the exact
resource names.
3. **How to retrieve the routing key** — exact command (e.g. `terraform output -raw
github_actions_routing_key` after apply) or the PagerDuty console path
(Service → Integrations → Events API v2 → Integration Key), so the owner can store it
as GitHub secret **`PAGERDUTY_ROUTING_KEY`** in `Szer/bots`.
4. **Escalation policy** — which policy the service uses, and who it pages (or "default",
if that's the convention).
5. **my-infra PR link/number** and its plan status.
6. **Anything that surprises you** in the existing PD setup that affects this (plan
limits, an existing routing key that should be used instead, existing Grafana→PD
alerting to align with, etc.).

## 4. Sequence after you finish (for the owner / bots side — not your job)

1. Owner applies your PR (via merge), retrieves the key, adds it as the
`PAGERDUTY_ROUTING_KEY` secret in `Szer/bots`.
2. The bots-side agent (separate handoff) adds the sentinel workflow + optional
in-workflow failure hooks, then tests with a dry-run and a forced-failure run.

Do not start step 2 until your answers land.

---

## 5. Answers from the my-infra side (2026-09-02)

Work done by the trusted agent that holds `~/git/my-infra`. The bots-side canary may proceed once the owner confirms the secret is set.

1. **Approach chosen:** NEW dedicated PagerDuty service **"GitHub Actions"** (`pagerduty_service.github_actions`). Reuse was rejected: the only candidate, "Azure Monitor", is hard-wired to constant low urgency and semantically wrong for GitHub. PD service ID is assigned on apply; retrieve it from the PD console (Services → GitHub Actions) if needed, it is not required for the Events API.
2. **Integration:** `pagerduty_service_integration.github_actions_events_v2`, `type = "events_api_v2_inbound_integration"`, display name "GitHub Actions Events API V2". Routing key exposed as sensitive output `pagerduty_github_actions_routing_key`.
3. **How to retrieve the key (owner only, after apply):**
`gh secret set PAGERDUTY_ROUTING_KEY --repo Szer/bots --body "$(terraform -chdir=terraform output -raw pagerduty_github_actions_routing_key)"` run from `~/git/my-infra` (reads state only, applies nothing). Console fallback: PagerDuty → Services → GitHub Actions → Integrations → Events API V2 → Integration Key.
4. **Escalation policy:** the existing single policy "Default" (`pagerduty_escalation_policy.default`): one rule, 30 min, targets the owner's user directly. No teams or schedules exist on this PD plan (Teams returns HTTP 402).
5. **my-infra PR:** https://github.com/Szer/my-infra/pull/146 — plan check green, `Plan: 2 to add, 0 to change, 0 to destroy`, no existing resource touched. Merge = CI apply (owner merges; PR is mergeable and CLEAN).
6. **Things that affect the canary design (BINDING for the bots side):**
- The service uses **severity_based urgency**. The canary chooses per event: `"severity": "warning"` → low urgency → silent mobile push; `"severity": "critical"` → high urgency → loud page. Owner's notification model: push for "worth knowing", page only for "must act now". Recommended mapping: a single failed/stalled workflow = `warning`; ALL scheduled agent workflows failed or stalled at once (the 08-21 pattern) = `critical`.
- PD account is **EU region**. `https://events.pagerduty.com/v2/enqueue` routes cross-region fine, but `https://events.eu.pagerduty.com/v2/enqueue` is what the in-cluster Alertmanager uses; either works, prefer the EU host for consistency.
- Precedent to copy: `Szer/my-infra/.github/workflows/terraform-apply.yml` lines ~150-176 already does an `if: failure()` curl to Events API v2 with a repo secret and no-ops quietly when the secret is unset. Same shape, same silence-on-missing-secret behavior expected.
- Use one `dedup_key` per monitored workflow (e.g. `gh-actions/<workflow-file>`) and send `event_action: resolve` on recovery so incidents self-close. The service has `auto_resolve_timeout = null`, so nothing resolves unless the canary says so.
- Payload `source` should identify the repo + workflow; `component`/`group` are free-form and show up in the PD mobile UI, use them.
- The secret name `PAGERDUTY_ROUTING_KEY` in Szer/bots is free (no PagerDuty secret exists there today).
74 changes: 74 additions & 0 deletions .github/canary-sentinel-handoff.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# Handoff: build the scheduled-workflow canary in Szer/bots

**For:** the agent implementing the bots-side sentinel (this repo only).
**From:** the agent that did the PagerDuty terraform in my-infra.
**Status:** ALL prerequisites are DONE. Start now. Background and my answers to the original work order are in `.github/canary-pagerduty-handoff.md` (sections 1 and 5).

## 1. What already exists (do not redo, do not touch)

- PagerDuty service **"GitHub Actions"** with an Events API v2 integration is live (my-infra PR #146, applied 2026-09-02, run 33629978524 green).
- Repo secret **`PAGERDUTY_ROUTING_KEY`** is set in Szer/bots (2026-09-02 12:30 UTC). Never print, echo, mask-test, or log it. If a run shows it is missing, no-op quietly and print one line saying so; never fail the canary because of a missing secret.
- The service has `auto_resolve_timeout = null`: an incident stays open until YOU send `event_action: resolve`. There is no time-based cleanup. Every trigger path must have a matching resolve path.
- Urgency is **severity_based**. `severity: warning` = low urgency = silent mobile push. `severity: critical` = high urgency = loud page. This mapping is BINDING (owner's notification model: push for "worth knowing", page only for "must act now").
- The my-infra repo is out of scope. Do not open it, do not reference its files.

## 2. What to build

One new workflow, `.github/workflows/workflow-canary.yml` (name it `Workflow Canary`), on its own schedule, that:

1. Lists recent runs of every scheduled workflow in this repo via `gh run list` with `GITHUB_TOKEN` (`permissions: actions: read` is enough; add `contents: read` if you check out the repo).
2. Classifies each workflow as `healthy`, `failed` (latest completed scheduled run concluded `failure`, `timed_out`, or `startup_failure`), or `stalled` (no completed run within its expected cadence + grace).
3. Sends PagerDuty Events API v2 events: `trigger` for failed/stalled, `resolve` when healthy again. Endpoint `https://events.eu.pagerduty.com/v2/enqueue` (EU account; the global host also routes, but prefer EU).
4. Uses one stable `dedup_key` per workflow, e.g. `bots/<workflow-file-name>`, so repeated polls update one incident and recovery closes exactly that incident. Resolving a dedup key that has no open incident is harmless; do it unconditionally on healthy.

Plain `curl` + `gh` + `jq`. No `openai/codex-action`, no LLM, no VPN, no `aks-vpn` concurrency group. The canary must be dumber than the things it watches.

## 3. Workflows to watch (ground truth from `.github/workflows/`, 2026-09-02)

| Workflow file | `name:` | cron (UTC) | Cadence | Suggested stall threshold |
|---|---|---|---|---|
| `monitor.yml` | Monitor | `20 */4 * * *` | 4h | 5h30m |
| `project.yml` | Daily Project Assessment | `37 4 * * *` | 24h | 26h |
| `product.yml` | Product Analysis | `15 10 * * 2,5` | Tue/Fri (up to 4d gap) | 4d + 2h = 98h |
| `vahter-mirror-reconcile.yml` | Vahter: Mirror Reconcile | `10 7 * * *` | 24h | 26h |

Keep this as a static table in the workflow (a small JSON or bash array with file, threshold), not cron parsing. Adding a workflow = adding a row. Grace must be at least 60 min: GitHub delays scheduled runs by minutes to over an hour under load, and schedules are silently disabled after 60 days of repo inactivity (stall detection covers that case, which is why it is mandatory).

Notes per workflow:
- **`vahter-mirror-reconcile.yml` fails BY DESIGN** as its signal (it fails the run when a mirror sync PR is stuck or check-less). A failure there is a real "someone must look" condition and today it notifies nobody. Treat it exactly like the others: warning.
- Monitor, Project, Product share the `aks-vpn` concurrency group, so a run can conclude `cancelled`. Treat `cancelled` as neither failed nor healthy: it does not trigger, and it does not count as a fresh healthy run for stall purposes. Use the latest run whose conclusion is `success`/`failure`/`timed_out`/`startup_failure`.
- Filter to `--event schedule` when judging cadence. A manual `workflow_dispatch` success may be counted as healthy for the failed check but should not reset the stall clock unless you decide otherwise and say so in the PR.
- An unloadable workflow (bad YAML) produces a jobless `startup_failure` with no error text in the API, or no run at all. Both must surface: `startup_failure` via the failed path, no-run via the stalled path.

## 4. Severity rules (BINDING)

- One workflow failed or stalled: `severity: warning` (silent push).
- Two or more of the three agent workflows (Monitor, Project, Product) failed or stalled in the same poll: `severity: critical` (loud page). This is the 2026-08-21 pattern, when a single upstream action change killed all of them at once.
- Send severity per dedup key. When the count drops back to one, the next trigger for that key should carry `warning` again (PagerDuty updates urgency on re-trigger with a new severity).
- Payload: `source` = `Szer/bots/<workflow-file>`, `summary` = one line with workflow name, state, and the run URL or "no run in N hours", `component` = workflow file, `group` = `github-actions`, `custom_details` = latest run id, conclusion, created_at, threshold. These fields show in the PD mobile UI; use them.

## 5. Canary schedule and self-reporting

- Cron: hourly at an odd minute, e.g. `50 * * * *`. Also `workflow_dispatch` with inputs `dry_run` (boolean, default `true`: print every event that would be sent, send nothing) and `test_event` (choice: `none` / `warning-roundtrip`: send one `warning` trigger with dedup key `bots/canary-selftest` and immediately resolve it). Never send `critical` from a test path.
- The canary's own failure is the remaining blind spot. Mitigate cheaply: add `if: failure()` last step that sends a `warning` trigger with dedup key `bots/workflow-canary` (same secret), and resolve that key at the start of every successful run. If PagerDuty itself is unreachable nothing can help; accept it.
- `timeout-minutes: 10`. Runner `ubuntu-latest`. `gh`, `jq`, `curl` are preinstalled.
- On any non-2xx from PagerDuty, print the response body (it never contains the key) and fail the step. Do not retry blindly.

## 6. Repo rules that WILL fail your PR if ignored

- **Comment gate** (`.comment-ratio.conf`, enforced by `.githooks/pre-commit` and `comment-lint.yml`): YAML ratio 0.30 with floor 5, and **no comment block longer than 2 lines anywhere, ever**. Existing workflows have long headers; those predate the gate. Yours cannot. Comments = constraints only, no narrative, no process history. Run `make install-hooks` once so the pre-commit hook checks you locally.
- **actionlint** runs on every workflow via `lint-workflows.yml`. Run `actionlint .github/workflows/workflow-canary.yml` locally before pushing if it is installed; otherwise rely on CI and read the failure.
- Do not add the canary to `.github/bots.yml` (that registry is for bots, not workflows) and do not edit any existing workflow. Optional in-workflow `if: failure()` hooks in the agent workflows are a separate follow-up PR, not this one.
- Do not run any real-Telegram or paid-LLM test workflows (`*-real-test.yml`). Not related, listed because agents keep tripping on it.
- PR title style: `CI: workflow canary → PagerDuty for failed/stalled scheduled workflows`. Keep the PR body factual: what is watched, thresholds, severity rules, how to test, how to add a workflow.

## 7. Test plan (do this, report the evidence)

1. Push the branch, open the PR, confirm `lint-workflows` and `comment-lint` are green.
2. `workflow_dispatch` the canary from the PR branch with `dry_run=true`: paste the classification table it printed for all four workflows into the PR. All four should be healthy today (Monitor and Project were re-verified green on 2026-09-02; check Product and Mirror Reconcile yourself and report what you see).
3. `workflow_dispatch` with `test_event=warning-roundtrip`, `dry_run=false`: report the two HTTP responses (status only, no bodies with keys). The owner confirms a silent push arrived on the phone and that the incident auto-closed. Do not proceed to merge until the owner confirms.
4. Do NOT force a real workflow to fail to test the critical path. Describe in the PR how to test it by hand if the owner wants to (e.g. temporarily set one threshold to 1 minute on a branch dispatch with `dry_run=true`).

## 8. Report back

State: PR number + CI conclusions, the dry-run classification table, the roundtrip HTTP statuses, every deviation from sections 2 to 6 with the reason, and anything in the run history that surprised you (e.g. a workflow that is already stalled today).
Loading
Loading