feat(ci): accept a wizard-app gateway token under --ci by brandonleung · Pull Request #1225 · PostHog/wizard

brandonleung · 2026-09-08T18:14:34Z

Problem

The CI smoke test runs --ci --api-key phx_, and the gateway-token mint refuses a personal API key. CI works today only because that refusal falls through to the legacy gateway, so it breaks when the legacy product is switched off.

Changes

The key-prefix guard accepts a wizard-application pha_ under --ci, matched at the start of the string, so the smoke test can carry the credential the mint actually accepts.

The token itself comes from the management command in the paired posthog PR, and still needs rotating into the CI secret.

Test plan

pnpm test. The guard's warning is asserted present for every other prefix and absent for this one.

github-actions · 2026-09-08T18:15:20Z

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

/wizard-ci all

Test all apps in a directory:

/wizard-ci ai-observability
/wizard-ci basic-integration
/wizard-ci mcp-analytics
/wizard-ci replay-vision
/wizard-ci revenue
/wizard-ci self-driving
/wizard-ci warehouse
/wizard-ci warehouse-seeded

Test an individual app:

/wizard-ci ai-observability/anthropic
/wizard-ci ai-observability/google-adk
/wizard-ci ai-observability/groq

Show more apps

/wizard-ci ai-observability/manual-capture
/wizard-ci ai-observability/openai
/wizard-ci ai-observability/openai-agents
/wizard-ci ai-observability/opentelemetry
/wizard-ci ai-observability/vercel-ai
/wizard-ci basic-integration/android
/wizard-ci basic-integration/angular
/wizard-ci basic-integration/astro
/wizard-ci basic-integration/django
/wizard-ci basic-integration/fastapi
/wizard-ci basic-integration/flask
/wizard-ci basic-integration/flutter
/wizard-ci basic-integration/javascript-node
/wizard-ci basic-integration/javascript-web
/wizard-ci basic-integration/laravel
/wizard-ci basic-integration/next-js
/wizard-ci basic-integration/nuxt
/wizard-ci basic-integration/python
/wizard-ci basic-integration/rails
/wizard-ci basic-integration/react-native
/wizard-ci basic-integration/react-router
/wizard-ci basic-integration/sveltekit
/wizard-ci basic-integration/swift
/wizard-ci basic-integration/tanstack-router
/wizard-ci basic-integration/tanstack-start
/wizard-ci basic-integration/vue
/wizard-ci mcp-analytics/custom-dispatcher
/wizard-ci mcp-analytics/typescript-sdk
/wizard-ci replay-vision/javascript-node
/wizard-ci replay-vision/next-js
/wizard-ci replay-vision/react-vite
/wizard-ci revenue/stripe
/wizard-ci self-driving/astro
/wizard-ci self-driving/fastapi
/wizard-ci self-driving/nuxt
/wizard-ci self-driving/react-router
/wizard-ci self-driving/sveltekit
/wizard-ci warehouse/monorepo-env
/wizard-ci warehouse/multi-source-next
/wizard-ci warehouse/stripe-node
/wizard-ci warehouse/zero-source
/wizard-ci warehouse-seeded/next-stripe
/wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

/wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

gewenyu99

Yah this is fine. I will approve top of stack once I run it e2e

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

gewenyu99 · 2026-09-09T01:42:09Z

Full Express flow completed; snapshot assertions FAIL due to a baseline harness bug.

basic-integration/javascript-node/express-todo — 73 observed model completions (73 successful), 73 model-turn PNGs, 164 PNGs total. Exit 1; phase completed.

Tested checkout SHAs: Wizard 6723957363f13ca5f886ba99e84702b2230a6a34, PostHog 90b30495bcedac556fe8f090eb5e26ec5022b911, Go gateway 9a2f445a93b7bdb36bf4332cbf32539378429730, context-mill f8f89ca9f2165ccc319212c9089b9cefefb7a148.

A full real Wizard Express integration run on the locally restacked model, effort, budget and backend chains, with no model-turn cutoff. Gateway system-prompt PRs #455/#464/#461 and companion Wizard #1227 were excluded.

Skills: Current full-run logs confirm http://localhost:8765 for skill menu and skill ZIP, http://localhost:8787/mcp for MCP, and http://localhost:8010 for PostHog.

Credential distinction: This run used a locally minted wizard-app OAuth fixture with the application’s full scope ceiling. The user’s two supplied PostHog keys were not exercised. A separate locally generated narrow control token returned a project-read 403; that result does not establish any scope defect in the user’s keys or explain an issue with them.

Every observed gateway model completion has a PNG below, including child-agent requests. These show the rendered TUI at capture time; concurrent requests can share the same visible state.

Integration completed through MCP setup, Slack skip and Keep Skills (frames 154–164); all 73 model completions succeeded. Snapshot assertions failed because the pre-existing resultWritten guard preserved the earlier outro state. The same guard exists on main. No cutoff was active. The final screen reports one conflict to review: npm installation succeeded but reported two moderate dependency audit vulnerabilities (frame 149).

Outcome and capture manifest · Source-stack provenance.

Source PR tips and local run adjustments

The tested SHAs above include local stack assemblies; source PR tips are recorded separately below.

PostHog/wizard: #1221 f6e1f89b, #1222 eb9e6050, #1224 388cd7d8, #1225 67239573
PostHog/ai-gateway: #435 a0ab2a5a, #462 14925b3a, #456 6c4c1433, #457 9b4b17eb, #459 3ea0a07b
PostHog/posthog: #96840 c26ac96c, #96846 74add69d, #96851 70efde1e, #96854 9bc50c09, #96861 a4540377
PostHog/charts: #15331 3719b27c
Snapshot harness captures observed gateway model completions and UI transitions, initializes local routing, and authenticates against the local API.
Local fixtures enabled gateway rollout, allowed 100 mints per day, and set a $30 team gateway overspend allowance.
The run used the chart-defined new-tier scoped-token cap of $6; the local team wallet was funded for provider dispatch.
System-prompt stack excluded for this run: Wizard fix: use gateway-owned file trust guidance #1227 and Go gateway chore(main): release 2.13.1 #455, Tasks stuck on analyzing project #464, chore: bikeshedding #461. Wizard uses the unchanged feat(ci): accept a wizard-app gateway token under --ci #1225 system prompt; the separate fix: use gateway-owned file trust guidance #1227 branch is preserved.
PostHog source tips were locally rebased onto master 1732fba27a5e7cfe9aa0d3391de5c26492bde759; all 23 replayed patches were unchanged.
Charts #15331 was locally rebased to e9fc004bf33ee1674d6fc25fa0d0d8c4564667d1 on main de54c2c75873ebf1624847867226d8849528f2fe; all six replayed patches were unchanged.
Go locally combines main → fix: request llm_gateway:read scope during signup provisioning #435 → fix: uninstall Claude Code plugin on removeServer, ignore missing legacy MCP entry #462 with main → fix(mcp): don't capture external Claude/Codex config errors as wizard exceptions #456 → chore(main): release 2.14.0 #457 → feat: User input overlay #459. Ancestry and source checks confirm chore(main): release 2.13.1 #455/Tasks stuck on analyzing project #464/chore: bikeshedding #461 are absent; no gateway-owned prompt policy code is present.
Wizard reached MCP setup, Slack skip and Keep Skills, then exited successfully. The existing resultWritten guard also present on main preserves the first-outro result.json, so raw wrapper assertions report a stale screenPath and skillsComplete=false. All raw results are preserved.

Full Express run — all 164 PNG snapshots

UI frame 001: 00001-intro.ans

UI frame 002: 00002-intro.ans

UI frame 003: 00003-before-action-confirm_setup.ans

UI frame 004: 00004-health-check.ans

UI frame 005: 00005-after-action-confirm_setup.ans

UI frame 006: 00006-auth.ans

UI frame 007: 00007-auth.ans

UI frame 008: 00008-run.ans

UI frame 009: 00009-run.ans

UI frame 010: 00010-run.ans

UI frame 011: 00011-run.ans

Model turn 1: openai/gpt-5.6-luna — HTTP 200, ok

UI frame 013: 00013-run.ans

Model turn 2: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 015: 00015-run.ans

Model turn 3: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 4: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 018: 00018-run.ans

UI frame 019: 00019-run.ans

Model turn 5: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 021: 00021-run.ans

Model turn 6: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 023: 00023-run.ans

Model turn 7: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 025: 00025-run.ans

UI frame 026: 00026-run.ans

UI frame 027: 00027-run.ans

UI frame 028: 00028-run.ans

Model turn 8: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 9: openai/gpt-5.6-luna — HTTP 200, ok

Model turn 10: openai/gpt-5.6-luna — HTTP 200, ok

Model turn 11: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 033: 00033-run.ans

UI frame 034: 00034-run.ans

Model turn 12: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 13: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 037: 00037-run.ans

Model turn 14: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 039: 00039-run.ans

Model turn 15: openai/gpt-5.6-luna — HTTP 200, ok

Model turn 16: openai/gpt-5.6-luna — HTTP 200, ok

UI frame 042: 00042-run.ans

Model turn 17: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 044: 00044-run.ans

Model turn 18: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 046: 00046-run.ans

Model turn 19: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 048: 00048-run.ans

Model turn 20: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 050: 00050-run.ans

Model turn 21: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 22: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 053: 00053-run.ans

Model turn 23: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 055: 00055-run.ans

Model turn 24: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 25: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 058: 00058-run.ans

UI frame 059: 00059-run.ans

UI frame 060: 00060-run.ans

Model turn 26: openai/gpt-5.6-luna — HTTP 200, ok

Model turn 27: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 063: 00063-run.ans

UI frame 064: 00064-run.ans

Model turn 28: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 066: 00066-run.ans

Model turn 29: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 30: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 069: 00069-run.ans

Model turn 31: openai/gpt-5.6-luna — HTTP 200, ok

Model turn 32: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 072: 00072-run.ans

UI frame 073: 00073-run.ans

Model turn 33: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 34: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 35: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 077: 00077-run.ans

Model turn 36: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 079: 00079-run.ans

Model turn 37: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 081: 00081-run.ans

UI frame 082: 00082-run.ans

Model turn 38: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 39: openai/gpt-5.6-luna — HTTP 200, ok

UI frame 085: 00085-run.ans

Model turn 40: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 087: 00087-run.ans

Model turn 41: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 089: 00089-run.ans

Model turn 42: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 091: 00091-run.ans

Model turn 43: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 093: 00093-run.ans

Model turn 44: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 095: 00095-run.ans

Model turn 45: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 46: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 098: 00098-run.ans

Model turn 47: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 100: 00100-run.ans

UI frame 101: 00101-run.ans

UI frame 102: 00102-run.ans

Model turn 48: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 49: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 105: 00105-run.ans

Model turn 50: openai/gpt-5.6-sol — HTTP 200, ok

UI frame 107: 00107-run.ans

UI frame 108: 00108-run.ans

Model turn 51: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 110: 00110-run.ans

Model turn 52: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 112: 00112-run.ans

Model turn 53: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 114: 00114-run.ans

Model turn 54: openai/gpt-5.6-sol — HTTP 200, ok

UI frame 116: 00116-run.ans

Model turn 55: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 118: 00118-run.ans

Model turn 56: openai/gpt-5.6-sol — HTTP 200, ok

Model turn 57: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 121: 00121-run.ans

Model turn 58: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 123: 00123-run.ans

Model turn 59: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 125: 00125-run.ans

Model turn 60: openai/gpt-5.6-sol — HTTP 200, ok

Model turn 61: openai/gpt-5.6-sol — HTTP 200, ok

UI frame 128: 00128-run.ans

Model turn 62: openai/gpt-5.6-sol — HTTP 200, ok

UI frame 130: 00130-run.ans

UI frame 131: 00131-run.ans

Model turn 63: openai/gpt-5.6-sol — HTTP 200, ok

Model turn 64: openai/gpt-5.6-sol — HTTP 200, ok

Model turn 65: openai/gpt-5.6-sol — HTTP 200, ok

UI frame 135: 00135-run.ans

Model turn 66: openai/gpt-5.6-sol — HTTP 200, ok

UI frame 137: 00137-run.ans

UI frame 138: 00138-run.ans

Model turn 67: openai/gpt-5.6-sol — HTTP 200, ok

UI frame 140: 00140-run.ans

Model turn 68: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 142: 00142-run.ans

Model turn 69: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 144: 00144-run.ans

Model turn 70: openai/gpt-5.6-terra — HTTP 200, ok

Model turn 71: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 147: 00147-run.ans

Model turn 72: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 149: 00149-run.ans

UI frame 150: 00150-run.ans

UI frame 151: 00151-outro.ans

Model turn 73: openai/gpt-5.6-terra — HTTP 200, ok

UI frame 153: 00153-before-action-dismiss_outro.ans

UI frame 154: 00154-mcp.ans

UI frame 155: 00155-after-action-dismiss_outro.ans

UI frame 156: 00156-before-action-set_mcp_outcome.ans

UI frame 157: 00157-slack-connect.ans

UI frame 158: 00158-after-action-set_mcp_outcome.ans

UI frame 159: 00159-before-action-dismiss_slack.ans

UI frame 160: 00160-keep-skills.ans

UI frame 161: 00161-after-action-dismiss_slack.ans

UI frame 162: 00162-before-action-keep_skills.ans

UI frame 163: 00163-after-action-keep_skills.ans

Final outcome: process exit — FAIL

brandonleung force-pushed the brandon/wizard-w1-4-ci-accepts-gateway-token branch from 4f68870 to c67f713 Compare September 8, 2026 19:52

brandonleung force-pushed the brandon/wizard-w1-4-ci-accepts-gateway-token branch from c67f713 to 02fd94d Compare September 8, 2026 20:20

brandonleung force-pushed the brandon/wizard-w1-4-ci-accepts-gateway-token branch from 02fd94d to 77aadc3 Compare September 8, 2026 20:45

brandonleung force-pushed the brandon/wizard-w1-4-ci-accepts-gateway-token branch from 77aadc3 to d3adb30 Compare September 8, 2026 20:57

brandonleung marked this pull request as ready for review September 8, 2026 21:17

brandonleung requested a review from a team as a code owner September 8, 2026 21:17

gewenyu99 reviewed Sep 8, 2026

View reviewed changes

feat(ci): accept a wizard-app gateway token under --ci

6723957

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

brandonleung force-pushed the brandon/wizard-w1-4-ci-accepts-gateway-token branch from d3adb30 to 6723957 Compare September 8, 2026 23:24

gewenyu99 mentioned this pull request Sep 9, 2026

fix: use gateway-owned file trust guidance #1227

Open

Navigation Menu

Uh oh!

feat(ci): accept a wizard-app gateway token under --ci - #1225

feat(ci): accept a wizard-app gateway token under --ci#1225
brandonleung wants to merge 1 commit into
brandon/wizard-w1-3-remint-on-401from
brandon/wizard-w1-4-ci-accepts-gateway-token

brandonleung commented Sep 8, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 8, 2026

Uh oh!

gewenyu99 left a comment

Uh oh!

gewenyu99 commented Sep 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

brandonleung commented Sep 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Changes

Test plan

Uh oh!

github-actions Bot commented Sep 8, 2026

🧙 Wizard CI

Uh oh!

gewenyu99 left a comment

Choose a reason for hiding this comment

Uh oh!

gewenyu99 commented Sep 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

brandonleung commented Sep 8, 2026 •

edited

Loading

gewenyu99 commented Sep 9, 2026 •

edited

Loading