feat(triage): experimental Flue-based issue triage#1090
Conversation
Adds two complementary triage paths under .flue/: Phase 1: Worker-deployed auto-labeller (agents/triage-label.ts). Receives GitHub issues.opened webhooks, verifies HMAC against raw bytes via Web Crypto, classifies the issue with kimi-k2.6 through our existing Cloudflare AI Gateway (same path /bonk uses), and applies bug/question/documentation + area/* labels with a short summary comment. Filter-then-apply guards against hallucinated label names. DRY_RUN env var skips writes for first deploys. Phase 2: GH-Actions-driven reproduction (agents/repro-issue.ts + .github/workflows/auto-repro.yml). Fires only when a maintainer applies the triage:reproduce label, never on every new issue. Uses local() sandbox so the agent's bash tool gets real pnpm/git/ gh/node access on the runner. Tries to reproduce as a vitest test or repro script and posts a single summary comment with one of reproduced/not-reproduced/repro-skipped labels. Never pushes branches, commits, or attempts fixes. Local prototype runner (scripts/run-local.ts) wraps flue run triage-issue with real issue payloads (5 fixtures from current queue) so the prompt can be iterated without deploying anything. All model traffic routes through CF_AI_GATEWAY_* secrets shared with bonk/review for unified cost tracking. Not deployed yet -- needs a Discussion before any actual rollout.
|
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-playground | 61ba8dc | May 26 2026, 04:19 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
docs | 61ba8dc | May 26 2026, 04:19 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-perf-coordinator | efd0efc | May 18 2026, 11:23 AM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-cache | 61ba8dc | May 26 2026, 04:19 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-i18n | efd0efc | May 18 2026, 11:23 AM |
Scope checkThis PR changes 6,530 lines across 20 files. Large PRs are harder to review and more likely to be closed without review. If this scope is intentional, no action needed. A maintainer will review it. If not, please consider splitting this into smaller PRs. See CONTRIBUTING.md for contribution guidelines. |
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/blocks
@emdash-cms/cloudflare
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
|
/review |
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
| className: "TriageLabel", | ||
| sqlite: true, | ||
| }), | ||
| GITHUB_WEBHOOK_SECRET: process.env.GITHUB_WEBHOOK_SECRET ?? "dev-secret-rotate-me", |
There was a problem hiding this comment.
HIGH — fail-open default for webhook secret.
When process.env.GITHUB_WEBHOOK_SECRET is unset, the binding is initialised to the literal string "dev-secret-rotate-me". Combined with triage-label.ts:226 which only refuses the request when secret is falsy, this means a deploy that forgets to set the secret will happily verify HMACs against a value now public in this PR. Anyone reading the source can forge issues.opened webhooks against the deployed Worker.
This is the kind of footgun that bites the first time someone runs alchemy deploy against a real stage without the secret bound. The comment two lines above is the only thing stopping it.
Fix: default to "" (or omit the binding) so the request handler's if (!secret) branch refuses the request, instead of a plausible-looking sentinel that passes the existence check.
| lines.push(""); | ||
| lines.push("```"); | ||
| lines.push(repro.notes); | ||
| lines.push("```"); |
There was a problem hiding this comment.
MEDIUM — fenced-code-block escape via model output.
repro.notes is the agent's transcript of bash commands and their captured output. Whenever the agent runs pnpm test or grep against a markdown file (highly likely on this monorepo — docs/, README.md, CONTRIBUTING.md, every plugin's docs), the captured output frequently contains a literal triple-backtick. Wrapping it in ``` ... ``` then breaks out of the fence and the rest of the comment renders as raw markdown / executes any nested formatting.
This isn't theoretical — the very reproduce skill encourages running grep/test commands whose stdout includes the repo's markdown. First time the model captures a doc snippet, the comment is mangled.
Fix: use a four-backtick (or longer) fence, or replace \u0060\u0060\u0060 runs inside repro.notes before interpolation. The simplest: find the longest backtick run in notes, fence with one more.
| // surface. Override with FLUE_REPRO_MODEL for experiments. | ||
| model: process.env.FLUE_REPRO_MODEL ?? "cloudflare-ai-gateway/claude-opus-4-7", | ||
| }); | ||
| const reproSession = await reproHarness.session(); |
There was a problem hiding this comment.
MEDIUM — prompt injection from issue body into shell-equipped agent.
The repro session uses local() so its bash tool has real pnpm / gh / git / node against the runner's filesystem, and GH_TOKEN is explicitly injected on line 132. Then reproSession.skill("reproduce", { args: { ..., issueTitle, issueBody, triage } }) feeds the entirely-attacker-controlled issue body into the prompt context.
The "strict guardrails" called out in the PR description and in SKILL.md ("no commits, no fix attempts, no curl") are prompt-level — they are advisory text the model can be argued out of by a sufficiently clever issue body. With write-scoped GH_TOKEN in env, a successful jailbreak can call gh to comment on any issue/PR in the repo, apply or remove arbitrary labels, close issues, etc. (The token can't push to branches because the workflow grants contents: read, which is the one real guardrail.)
The trigger is gated on triage:reproduce being applied by a maintainer, which is the main mitigation. But the threat model is: open a malicious issue, wait for a triager to mistakenly request reproduction. Worth being explicit about this in the README / Discussion before enabling, and considering whether the runner should run with a token that only has issues:write on the target issue (e.g. a finer-grained PAT or App installation scoped via headers).
| ? "reproduced" | ||
| : "not-reproduced"; | ||
| try { | ||
| await addLabels(githubToken, owner, repo, issueNumber, [resultLabel]); |
There was a problem hiding this comment.
LOW — comment claims the workflow creates labels lazily; nothing does.
Lines 165-169 say "these labels don't exist yet; the workflow creates them lazily" but neither auto-repro.yml nor this file calls POST /repos/{owner}/{repo}/labels to create them. addLabels calls POST /repos/{owner}/{repo}/issues/{n}/labels, which returns 422 if the label doesn't exist on the repo. The try/catch on line 175 swallows this so it's not fatal, but the first N runs will silently fail to apply any result label until someone manually creates reproduced, not-reproduced, repro-skipped in the repo's Labels settings.
Fix options: (a) precreate the three labels as part of the workflow's setup step using gh label create --force, or (b) update the comment to say "these labels must be precreated" so the next person reading the code knows.
| // Set to "true" to skip HMAC verification. DEV ONLY — never enable in | ||
| // production. Used for local `flue run` prototyping where we can't | ||
| // easily sign the synthetic payload. | ||
| SKIP_HMAC?: string; |
There was a problem hiding this comment.
LOW — SKIP_HMAC is read directly from env with no compile-time gate.
SKIP_HMAC === "true" bypasses HMAC verification entirely. The comment says "DEV ONLY — never enable in production" and alchemy.run.ts doesn't currently expose SKIP_HMAC as a binding, so it can't be turned on via the standard deploy path. But a future contributor copy-pasting a binding row or a wrangler secret put SKIP_HMAC true would silently disable auth on the live Worker, with no log telemetry beyond a single warn line that gets buried under request volume.
Fix: gate this on a build-time signal that can't be flipped after deploy — e.g. only consult the flag when globalThis.process?.env?.NODE_ENV === "development" (or equivalent for the Workers runtime), or just delete it. The local prototype path (scripts/run-local.ts) doesn't hit this code path anyway — it calls triage-issue.ts directly without the webhook agent.
| } | ||
| lines.push(""); | ||
| lines.push(`> ${triage.summary}`); | ||
| return lines.join("\n"); |
There was a problem hiding this comment.
LOW — triage.summary interpolated into a blockquote line with no newline-stripping.
The schema allows up to 200 chars and \n is a valid character in a v.string(). If the model returns a multi-line summary, only the first line is in the blockquote — the rest renders as bare body text, breaking the formatted comment.
Not a security issue (GitHub comments don't render raw HTML), but the auto-comment loses its structured look the moment the model decides to return two sentences. Strip \r?\n from triage.summary before rendering, or replace with a space.
| async function loadFixture(arg: string, live: boolean): Promise<Fixture> { | ||
| if (live) { | ||
| const raw = execSync( | ||
| `gh issue view ${arg} --repo emdash-cms/emdash --json number,title,body,labels`, |
There was a problem hiding this comment.
LOW — unvalidated arg interpolated into gh command.
gh issue view ${arg} where arg is args.filter((a) => !a.startsWith("--"))[i]. Nothing enforces that arg is a number — pnpm prototype '1 && rm -rf /' (or anything with shell metachars) is passed through execSync as-is. This is a local dev tool with no security boundary, but the same arg is then used to construct issue-${arg}.json and would happily traverse with ../.
Fix: assert /^\d+$/.test(arg) before either call, fail with a clear message otherwise.
|
Review posted successfully with 7 line-anchored comments. Summary of findings:
|
HIGH: - alchemy.run.ts: default GITHUB_WEBHOOK_SECRET to "" (was a sentinel string that would have fail-open authenticated forged webhooks). The empty default makes the agent's `if (!secret)` guard fail closed. MEDIUM: - repro-issue.ts: dynamic-length backtick fence for `repro.notes` in the rendered comment. Captured bash output frequently contains triple backticks (markdown docs, README snippets, etc.) which would otherwise break out of the code fence. - README + SKILL.md: document the threat model for the Phase 2 repro agent. The maintainer label gate is the real security boundary, not the prompt-level guardrails. Add explicit hard prohibitions to the skill (no git push, no PR writes, no label management, no curl, etc.) and call out the worst-case capabilities of a successful jailbreak. LOW: - auto-repro.yml: precreate the three result labels (reproduced / not-reproduced / repro-skipped) idempotently with gh label create --force. Update the misleading comment in repro-issue.ts that implied the workflow did this already. - triage-label.ts: remove SKIP_HMAC entirely. The local prototype path uses triage-issue.ts (no webhook) so the bypass was never reachable in dev, and a wrangler secret put SKIP_HMAC true on a live deploy would silently disable verification. - triage-label.ts: collapse newlines in triage.summary before rendering into the blockquote, so a multi-line model summary doesn't leak text outside the > prefix. - run-local.ts: validate issueNumber matches /^\d+$/ before interpolating into the gh command and the fixture path.
Mirrors the .flue/agents/issue-triage.ts pattern in withastro/astro:
two distinct tokens, separating the agent's bash sandbox capability
from the orchestrator's write capability.
- AGENT_GH_TOKEN (workflow's default GITHUB_TOKEN, scoped to
contents: read + issues: read) is the only token passed into
local({ env: { GH_TOKEN: ... } }). The agent's bash can clone the
repo and read issues, but cannot comment, label, or close anything.
- ORCHESTRATOR_GH_TOKEN (a GitHub App installation token minted by
actions/create-github-app-token) is read by the TS orchestrator
from process.env and used for postIssueComment and addLabels. It
never enters the sandbox env, so a complete jailbreak of the
agent's bash cannot escalate to comment or label writes.
Workflow changes:
- Job permissions narrowed: contents: read, issues: read (was
contents: read, issues: write).
- App token minted via actions/create-github-app-token@v3.2.0 with
the existing APP_ID / APP_PRIVATE_KEY secrets (same pattern
release.yml and auto-format.yml use).
- Label precreation step now uses the app token; agent step gets
both tokens but only forwards AGENT_GH_TOKEN to the sandbox.
README threat model rewritten to reflect the new split: lists the
exact capabilities a jailbroken sandbox retains (read-only operations
across the repo + arbitrary runner network egress) versus what is
now structurally impossible (comment, label, branch push, secret
exfiltration of the orchestrator token or AI Gateway credentials).
Also drops three inaccurate 'all model traffic goes through the AI
Gateway' code comments. The gateway is a binding option on
env.AI.run(), not a routing layer.
ascorbic
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
zizmor flagged the create-github-app-token step as minting an unscoped token. By default the action issues a token with every permission the App's installation has been granted, which is broader than the orchestrator needs. Add explicit `repositories: emdash` and `permission-issues: write` inputs so the minted token can only post comments and apply labels on this one repo. Read of issue bodies is implied by the installation; we don't need permission-contents (the workflow's default GITHUB_TOKEN handles the checkout).
| # title/body cleanly (they may contain quotes, backticks, etc.). | ||
| PAYLOAD=$(jq -nc \ | ||
| --argjson n "$ISSUE_NUMBER" \ | ||
| --arg t "$ISSUE_TITLE" \ | ||
| --arg b "$ISSUE_BODY" \ |
| const result = spawnSync( | ||
| "npx", | ||
| ["flue", "run", "triage-issue", "--target", "node", "--id", id, "--payload", payload], | ||
| { | ||
| cwd: FLUE_DIR, | ||
| env: process.env, | ||
| encoding: "utf8", | ||
| }, | ||
| ); |
| if (apply && !process.env.GITHUB_TOKEN && !process.env.GH_TOKEN) { | ||
| console.error("--apply requires GITHUB_TOKEN"); | ||
| process.exit(2); | ||
| } |
| // - GitHub Actions in Phase 2, when a maintainer applies a `triage:run` | ||
| // label to an existing issue and we want a fresh classification. | ||
| // - The webhook agent (agents/triage-label.ts), which calls into the | ||
| // same `classifyIssue` core function after verifying the signature. |
| // The deployed Worker uses the Workers AI binding directly (no | ||
| // gateway hop) since we're already running on Cloudflare. The | ||
| // AI Gateway is for off-CF callers like the GH Actions runner. |
| // Registers the Cloudflare Workers AI binding with our AI Gateway so all | ||
| // model calls (kimi, etc.) flow through gateway logging + cost tracking, | ||
| // same surface bonk.yml and review.yml use. Without this, the binding |
| }; | ||
| if (token) headers.Authorization = `token ${token}`; |
| @@ -0,0 +1,86 @@ | |||
| # Flue triage experiment | |||
|
|
|||
| **Status:** prototype, not deployed. See the EmDash Discussion for design context (TBD link). | |||
| if (repro.suggestedNextStep.trim().length > 0) { | ||
| lines.push(""); | ||
| lines.push(`> ${repro.suggestedNextStep}`); | ||
| } |
| "verbatimModuleSyntax": true, | ||
| "noEmit": true, | ||
| "allowImportingTsExtensions": false, | ||
| "types": ["@cloudflare/workers-types"] |
Real bugs: - workflow: ran 'cd .flue' before 'flue run', which made the local() sandbox start in .flue/ -- skills resolved against the wrong cwd and 'pnpm test' would have hit the wrong package root. Run from the repo root via 'pnpm --dir .flue exec flue run --root .flue', and pin agent + sandbox cwd to GITHUB_WORKSPACE explicitly. - run-local.ts: 'npx flue' could pull a different Flue version than the lockfile-pinned one. Swap to 'pnpm exec flue'. - run-local.ts: --apply accepted either GITHUB_TOKEN or GH_TOKEN but the spawned agent only read GITHUB_TOKEN. Normalise to GITHUB_TOKEN before spawn so a user-set GH_TOKEN works end-to-end. - repro-issue.ts: suggestedNextStep was rendered into a blockquote without newline-stripping, same bug class as the summary fix in triage-label.ts. Add module-scope NEWLINE_RUN_RE and apply it. - tsconfig.json: explicit 'types' list excluded @types/node, which agents and scripts need (process.env, node:child_process, etc.). Add 'node' to the types array and declare @types/node in deps. Doc / comment accuracy: - triage-issue.ts: comment referenced 'triage:run' label and a Phase 2 classification flow that doesn't exist. Replace with the accurate description -- Phase 2's repro-issue.ts calls classifyIssue() directly. - triage-label.ts: comment claimed 'no gateway hop' for the deployed Worker, but app.ts wires the gateway through env.AI.run when CLOUDFLARE_GATEWAY_ID is set. Reword to describe the actual logic. - app.ts: top-comment over-claimed 'all model calls flow through the gateway'; it's conditional on the binding being set. Qualify. - README: replace 'TBD link' placeholder with the actual PR link. Style: - lib/github.ts: switch 'Authorization: token' to 'Authorization: Bearer' and add 'X-GitHub-Api-Version: 2022-11-28' to match GitHub's current preferred header style and how other workflows in this repo call the API. Build hygiene: - Add .flue/pnpm-workspace.yaml so pnpm v11's strict build-script gate (verifyDepsBeforeRun + strictDepBuilds at the repo root) can be satisfied without --ignore-workspace. Drops that flag from the README install instructions.
What does this PR do?
Adds an experimental Flue-based issue triage system. Two complementary paths, both opt-in, neither deployed. CI tooling only -- nothing published, no behaviour change in any package.
Phase 1: Worker-deployed auto-labeller (
.flue/agents/triage-label.ts)Receives
issues.openedwebhooks, verifies HMAC against raw bytes via Web Crypto, classifies the issue with kimi-k2.6, applies labels, and posts a structured summary comment. Filter-then-apply guard against hallucinated label names.DRY_RUNenv var skips writes for first deploys. Schema enforces:kind(bug/enhancement/documentation/question),severity(low/medium/high/critical), 0-3area/*labels,reproduciblebool,dataLossRiskbool, one-sentencesummary.Phase 2: GH-Actions-driven reproduction (
.flue/agents/repro-issue.ts+.github/workflows/auto-repro.yml)Fires only when a maintainer applies the
triage:reproducelabel, never on every new issue. Uses Flue'slocal()sandbox so the agent's bash tool gets realpnpm/git/gh/nodeaccess on the runner. Tries to reproduce as a vitest test or repro script, posts a single summary comment, appliesreproduced/not-reproduced/repro-skipped. Strict guardrails: no branch pushes, no commits, no fix attempts.Local prototype runner (
.flue/scripts/run-local.ts)Wraps
flue run triage-issuewith real issue payloads. 5 fixture issues from the current queue (#1021, #1042, #1046, #1049, #1080) so the prompt can be iterated without deploying anything.Model traffic uses the existing
CF_AI_GATEWAY_*secrets shared withbonk.ymlandreview.ymlfor unified cost tracking. No new auth surface, no new provider keys.Why two phases: Phase 1 is cheap (~$0/issue on Workers AI), fast (~5s), and conservative (label + comment only) -- safe to run on every new issue. Phase 2 is expensive (Opus on a 30-min runner) and powerful (real shell, can write tests) -- runs only on explicit maintainer opt-in.
Not deployed. This PR adds the scaffolding only. No
alchemy deployworkflow, no GH webhook registered, noGITHUB_WEBHOOK_SECRETset. Follow-up PRs will wire up the deploy plumbing and a first synthetic-issue test once we agree on the shape here.Threat model
The repro agent (Phase 2) feeds attacker-controlled issue bodies into a prompt context with a
bash-equipped sandbox. Anyone can file an issue. The agent's "do not push, do not commit, do not curl" guardrails inskills/reproduce/SKILL.mdare prompt-level only.What blocks real abuse (in order of strength):
issue-triagesetup:AGENT_GH_TOKEN(workflow's defaultGITHUB_TOKEN, scoped tocontents: read, issues: read) is the only token passed into thelocal()sandbox env. A jailbroken agent's bash cangh issue view,git clone, read the repo -- and nothing else.ORCHESTRATOR_GH_TOKEN(a GitHub App installation token minted byactions/create-github-app-token) holdsissues: write. Stays in the TS orchestrator'sprocess.env; never crosses into the sandbox. All comment and label writes go through this. A jailbroken agent's bash cannot impersonate the bot.issues.labeledwithlabel.name == 'triage:reproduce'. A maintainer has to apply that label before the agent ever sees the body. First boundary -- don't applytriage:reproduceto an issue you wouldn't drop a fresh Opus into.contents: readpermission on the job. No branch pushes, even from a jailbroken agent.Capabilities a jailbroken sandbox retains: read-only operations across the repo, arbitrary runner network egress. Structurally impossible: comment / label / close any issue or PR, push to any branch, exfiltrate the orchestrator token or AI Gateway credentials (
local()filters host env by default).Follow-up worth considering:
step-security/harden-runnerfor an egress allowlist (defense in depth against the "agent curls attacker.com from the runner" residual).Closes #
Type of change
Checklist
pnpm typecheckpassespnpm lintpasses (0 diagnostics in the new files)pnpm testpasses (or targeted tests for my change) -- no test surface changedpnpm formathas been runmessages.pochanges except in translation PRs -- a workflow extracts catalogs on merge tomain. (N/A -- no admin UI changes.)AI-generated code disclosure
Review history
efd0efc1-- initial scaffold (Phase 1 + 2 + local runner, 5 issue fixtures)b453ae95-- review feedback: HIGH webhook-secret fail-open fix, MEDIUM fence-escape in repro comment, MEDIUM threat model docs, LOW label precreation, LOWSKIP_HMACremoval, LOW newline strip on summary, LOW arg validation on local runnerfd81927e-- two-token split for prompt-injection mitigation (sandbox gets read-onlyGITHUB_TOKEN; orchestrator uses an app token for writes); narrowed workflow permissions; removed three inaccurate "all model traffic goes through the gateway" code comments916b32b4-- mergemain(53 commits, no conflicts)Remaining before ready-for-review
pnpm prototypeagainst a fixture issue (blocked on local CF gateway env)Validate PRbadge -- compliance check actually passed onb453ae95; needs a re-trigger against the current headTry this PR
Open a fresh playground →
A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.
Tracks
feat/flue-triage. Updated automatically when the playground redeploys.