Skip to content

chore(harness): WIP drive the e2e routes over the control socket #1278

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gewenyu99 wants to merge 10 commits into from
Closed

chore(harness): WIP drive the e2e routes over the control socket #1278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
535facf
chore(harness): drive the e2e routes over the control socket
gewenyu99 Sep 19, 2026
303fdfb
chore(cli): route every command through the surface dispatch
gewenyu99 Sep 19, 2026
fa416dd
chore(harness): exit a failed run from the handoff screen
gewenyu99 Sep 19, 2026
a50c7c5
Merge branch 'posthog/split-p4-control' into HEAD
gewenyu99 Sep 21, 2026
a1bc654
chore(harness): let the TUI spec ride out the readiness verdict
gewenyu99 Sep 19, 2026
d1f164a
chore(harness): document how to run every route
gewenyu99 Sep 19, 2026
67a7044
chore(harness): read the store mirror over the socket
gewenyu99 Sep 19, 2026
0be398b
chore(harness): derive launch commands, tighten the parity tests, cor…
gewenyu99 Sep 19, 2026
691aaeb
Merge branch 'posthog/split-p4-control' into posthog/split-p5-harness
gewenyu99 Sep 21, 2026
917199e
chore(build): refresh the chunk manifest fixtures
gewenyu99 Sep 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 45 additions & 38 deletions .claude/skills/exploring-the-wizard/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ compatibility:
wizard-ci MCP server.
metadata:
author: posthog
version: '5.0'
version: '6.0'
---

# Exploring the wizard as an agent
Expand All @@ -23,8 +23,8 @@ sequence, and gateway policy. For new exploration, launch the server with
`SNAP_HARNESS=pi`; prefer `SNAP_SEQUENCE=orchestrator` for the integration flow.
These are server environment variables, not MCP arguments. Restart an existing
server to change its environment. See the
[host architecture](../../../e2e-harness/ARCHITECTURE.md) for other programs,
overrides, and current limitations.
[harness architecture](../../../e2e-harness/ARCHITECTURE.md) for the control
API, other programs, and overrides.

## Prepare the run

Expand All @@ -35,16 +35,14 @@ wizard, so finish recording one app before opening another.
- **Detection only:** pass `appDir` and `projectId` (both required strings),
with no key. Stop at `auth` without calling `run_agent`.
- **Full integration:** reuse the authorized phx key file path, separate gateway
token file path, and project id;
ask only for missing inputs. Prefer `keyFile` so the key stays out of tool
arguments. Set `WIZARD_CI_GATEWAY_TOKEN_FILE` in the MCP server environment
before launch (restart an existing server); it is not an `open_app` argument.
The file must contain an already-issued gateway bearer, not the phx key.
CI does not mint or refresh it. Never print or commit either secret. See
[local credential setup](../../../docs/local-dev.md#credentials-for-local-ci-and-headless-runs). Read the
[credential and region limitations](../../../e2e-harness/ARCHITECTURE.md#current-host-limitations)
before starting: an inherited key can shadow `keyFile`, and the host currently
hardcodes the US region.
token file path, and project id; ask only for missing inputs. Prefer `keyFile`
so the key stays out of tool arguments. Set `WIZARD_CI_GATEWAY_TOKEN_FILE` in
the MCP server environment before launch (restart an existing server); it is
not an `open_app` argument. The file must contain an already-issued gateway
bearer, not the phx key. CI does not mint or refresh it. Never print or commit
either secret. See
[local credential setup](../../../docs/local-dev.md#credentials-for-local-ci-and-headless-runs).
`region` selects the PostHog region for auth and the gateway.
- **Questions during the run:** launch the server with `E2E_ASK=true` to keep
`wizard_ask` available in this CI session. Handle questions yourself through
the actions below; fixed-route answer profiles do not drive the MCP route.
Expand All @@ -64,11 +62,16 @@ It exposes exactly these tools:
| `run_agent` | None | Starts the real program in the background and returns immediately |

Use `read_state.actions` for action ids and parameters; there is no
`list_actions` MCP tool. Framework identity is `session.integration`, with
`list_actions` MCP tool. The state mirrors the wizard's store: `session` holds
the detection, setup, run, and follow-up fields (`session.runPhase`,
`session.pendingQuestion`, `session.taskNotice`, `session.outroData`, a redacted
`session.frameworkContext`), beside `tasks`, `setupQuestions`, and `actions`.
Framework identity is `session.integration`, with
`session.detectedFrameworkLabel` and `session.detectionComplete`. The separate
top-level `integration` field is the background status: `idle`, `running`,
`done`, or `failed`; `integrationError` holds a caught failure. Those two fields
are added by `read_state` and are absent from `perform_action` replies.
`done`, or `failed`, derived from `session.runPhase`; `integrationError` holds a
caught failure. Those two fields are added by `read_state` and are absent from
`perform_action` replies.

## Drive and record

Expand All @@ -79,31 +82,34 @@ own decisions through the same state and action contract.
judging detection. Inspect `session.integration` and `setupQuestions`.
2. Capture `render_screen` before each decision and during task or phase
changes. Save numbered frames such as `/tmp/wz-explore-snaps/01-intro.txt`.
3. Commit only actions currently offered. Common choices are below; the
[action registry](../../../e2e-harness/action-registry.ts) defines the full
set.
3. Commit only actions currently offered. Common choices are below; the generic
set lives in
[`src/store/control/actions.ts`](../../../src/store/control/actions.ts) and a
program adds its own through `controlActions` on its steps.
4. For a full run, confirm setup and call `run_agent` at `auth`. Continue
reading state and handling overlays while it runs; polling alone cannot
answer them.
5. Check `runPhase` (`idle`, `running`, `completed`, `error`), background
status, and the rendered outro. On error, capture the frame and reason before
dismissing it. An error outro can wait for dismissal while `integration`
still says `running`; a host exit can instead surface as a socket error.
5. Check `session.runPhase` (`idle`, `running`, `completed`, `error`),
background status, and the rendered outro. On error, capture the frame and
reason before dismissing it. An error outro can wait for dismissal while
`integration` still says `running`; a wizard exit can instead surface as a
socket error.
6. After successful agent completion, finish the offered outro and follow-up
actions. For the integration flow, `session.skillsComplete` marks the tail's
completion. Other programs can have a terminal outro or exit screen.

| Decision | Action and `params` |
| ---------------------------------------- | -------------------------------------------------------------------------------------------- |
| Confirm intro / dismiss blocking outage | `confirm_setup` / `dismiss_outage` |
| Answer setup question | `choose`, `{ key, value }` from `setupQuestions` |
| Answer every question in a pending batch | `answer_question`, `{ answers: { questionId: value } }`; values are strings or string arrays |
| Cancel a question batch | `cancel_question` |
| Accept or decline an optional task | `resolve_notice`, `{ keep: true }` or `{ keep: false }` |
| Finish outro | `dismiss_outro` |
| Record MCP outcome | `set_mcp_outcome`, `{ outcome: "skipped" }` or `{ outcome: "installed", clients: [...] }` |
| Dismiss suggested prompts / Slack step | `dismiss` / `dismiss_slack` |
| Record keep-skills choice | `keep_skills`, `{ kept: true }` or `{ kept: false }` |
| Decision | Action and `params` |
| ---------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Confirm intro / dismiss blocking outage | `confirm_setup` / `dismiss_outage` |
| Answer setup question | `choose`, `{ key, value }` from `setupQuestions` |
| Answer every question in a pending batch | `answer_question`, `{ answers: { questionId: value } }`; values are strings or string arrays |
| Cancel a question batch | `cancel_question` |
| Accept or decline an optional task | `resolve_notice`, `{ keep: true }` or `{ keep: false }` |
| Finish outro | `dismiss_outro` |
| Record MCP outcome | `set_mcp_outcome`, `{ outcome: "skipped" }` or `{ outcome: "installed", clients: [...] }` |
| Dismiss suggested prompts / Slack step | `dismiss` / `dismiss_slack` |
| Record keep-skills choice | `keep_skills`, `{ kept: true }` or `{ kept: false }` |
| Pick the project on a detect screen | `pick_integration_target`, `{ path, integration }`; source maps: `pick_source_maps_project`, `{ variant, path }` |

MCP and keep-skills actions commit store state; recording an outcome does not
perform the corresponding installation or cleanup. Report which outcomes were
Expand All @@ -127,7 +133,8 @@ copies after recording their results.

The shared log is `/tmp/posthog-wizard.log`. Record its byte count before a run
and read from that count plus one afterward. Run sweeps serially so their logs
remain attributable. `read_state` omits `frameworkContext`; an empty
`setupQuestions` list alone does not prove a router mode. When necessary,
inspect the detector under [`src/store/frameworks/`](../../../src/store/frameworks/) against
the same fixture.
remain attributable. `read_state` shows `session.frameworkContext` after
redaction; an empty `setupQuestions` list alone does not prove a router mode.
When necessary, inspect the detector under
[`src/store/frameworks/`](../../../src/store/frameworks/) against the same
fixture.
2 changes: 1 addition & 1 deletion .claude/skills/wizard-development/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ infrastructure should consume those boundaries.

| Concern | Owner |
| ----------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Framework detection, context, env conventions | [FrameworkConfig](../../../src/store/framework-config.ts) and [framework configs](../../../src/frameworks/) |
| Framework detection, context, env conventions | [FrameworkConfig](../../../src/store/framework-config.ts) and [framework configs](../../../src/store/frameworks/) |
| Integration instructions and orchestrator flows/tasks | [context-mill](https://github.com/PostHog/context-mill) |
| Programs, steps, prerequisites and outcomes | [programs](../../../src/store/programs/) |
| Sequence, harness, model and effort selection | [switchboard](../../../src/agent/runner/switchboard/) |
Expand Down
34 changes: 17 additions & 17 deletions .claude/skills/wizard-development/references/ARCHITECTURE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,7 @@ lifecycle in

[runner/index.ts](../../../../src/agent/runner/index.ts) resolves a program's
`run` definition, calls shared bootstrap, selects a binding, dispatches the
sequence, and flushes the scanner report on cleanup. The old
[agent-runner.ts](../../../../src/agent/runner/index.ts) is a compatibility
export.
sequence, and flushes the scanner report on cleanup.

| Layer | Source and responsibility |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
Expand Down Expand Up @@ -57,7 +55,8 @@ Do not migrate a linear program merely by changing its binding if it depends on
these hooks. Inspect the orchestrator's flow and completion path instead.
[Metrics](../../../../src/store/programs/metrics/) is a current Pi/orchestrator
example. Native command modules still need registration in
[bin.ts](../../../../bin.ts); screen sequences derive from the program registry.
[src/cli/main.ts](../../../../src/cli/main.ts); flows derive from the program
registry.

## Switchboard contract

Expand Down Expand Up @@ -138,7 +137,7 @@ durable credentials.
## UI state and agent output

Business logic uses [WizardUI](../../../../src/store/ui/wizard-ui.ts) through
`getUI()`. [InkUI](../../../../src/store/ui/store-ui.ts) updates the TUI store;
`getUI()`. [StoreUI](../../../../src/store/ui/store-ui.ts) updates the store;
[LoggingUI](../../../../src/tui/console/logging-ui.ts) is available for
noninteractive callers that select it. A missing TTY does not automatically mean
an arbitrary caller uses LoggingUI; snapshot CI drives Ink in a PTY.
Expand All @@ -152,11 +151,11 @@ session event handlers. Orchestrated tasks also have queue and handoff state. Do
not assume all harness output passes through `handleSDKMessage`.

Session changes go through explicit store setters. They emit updates,
re-evaluate gates, detect transitions, and refresh rendering. The
[router](../../../../src/tui/router.ts) resolves overlays first, then the first
visible incomplete screen from
[screen-sequences.ts](../../../../src/tui/screen-sequences.ts). Those sequences
are projected from registered program steps. Change the state/predicate that
re-evaluate gates, detect transitions, and refresh rendering. The store's
[flow resolution](../../../../src/store/state/flow-resolution.ts) resolves
interrupts first, then the first visible incomplete step of the program's flow
([flowFor](../../../../src/store/programs/flow-for.ts)). Those flows are
projected from registered program steps. Change the state/predicate that
represents progress rather than adding imperative navigation.

## MCP and instrumentation
Expand All @@ -167,9 +166,8 @@ them differently; inspect the selected harness rather than assuming identical
tool names or discovery. Context-mill supplies skills and flow/task prompts.

[Middleware](../../../../src/agent/middleware/) provides opt-in message/phase
instrumentation. The linear sequence creates the benchmark pipeline; there is no
pipeline construction in the compatibility `agent-runner.ts`. Inspect the actual
consumer before extending instrumentation to another sequence or harness.
instrumentation. The linear sequence creates the benchmark pipeline. Inspect the
actual consumer before extending instrumentation to another sequence or harness.

## Surfaces and the control API

Expand All @@ -184,9 +182,10 @@ lists what it owns and may import:
| `src/cli` | argv, command tree, runners that sequence runs and pass context, `ControlHooks` | every surface, through its public entries only |

Cross-surface imports go through `@store`, `@store/types`, `@store/programs`,
`@agent`, `@agent/types`, `@tui`, `@tui/types`, and `@tui/console`.
`src/__tests__/architecture` enforces the matrix and the public-entry rule;
`tsc -b tsconfig.solution.json` mirrors it with project references.
`@agent`, `@agent/types`, `@tui`, `@tui/types`, and `@tui/console`; the two cli
runners load `@store/control` lazily. `src/__tests__/architecture` enforces the
matrix and the public-entry rule; `tsc -b tsconfig.solution.json` mirrors it
with project references.

`--control-socket <path>` serves an HTTP/1.1 API over a unix socket from
`src/store/control`: state with long polling, actions that call one store setter
Expand All @@ -203,13 +202,14 @@ Run it:

```bash
# headless, every build; the key travels in the environment
mkdir -p /tmp/w
POSTHOG_WIZARD_API_KEY=phx_... WIZARD_CI_GATEWAY_TOKEN_FILE=/path/to/token \
npx tsx bin.ts --headless-DONOTUSE-EXPERIMENTAL --control-socket /tmp/w/w.sock \
--project-id <id> --region us --install-dir /tmp/app
curl -s --unix-socket /tmp/w/w.sock -X POST -H 'content-type: application/json' -d '{}' http://localhost/detect
curl -s --unix-socket /tmp/w/w.sock -X POST -H 'content-type: application/json' \
-d '{"programId":"posthog-integration"}' http://localhost/runs
curl -s --unix-socket /tmp/w/w.sock 'http://localhost/state?wait=60000&since=0' | jq '.state.run, .state.tasks'
curl -s --unix-socket /tmp/w/w.sock 'http://localhost/state?wait=60000&since=0' | jq '.state.session.runPhase, .state.tasks'
curl -s --unix-socket /tmp/w/w.sock http://localhost/runs
curl -s --unix-socket /tmp/w/w.sock -X POST http://localhost/shutdown

Expand Down
Loading
Loading