diff --git a/.claude/skills/wizard-development/SKILL.md b/.claude/skills/wizard-development/SKILL.md index 5fee5c7a5..3fa776651 100644 --- a/.claude/skills/wizard-development/SKILL.md +++ b/.claude/skills/wizard-development/SKILL.md @@ -43,8 +43,8 @@ infrastructure should consume those boundaries. new Anthropic models. Existing routing has not all migrated: -[DEFAULT_BINDING](../../../src/agent/runner/switchboard/index.ts) still -selects Anthropic + linear, with per-program and flag overrides. Set new +[DEFAULT_BINDING](../../../src/agent/runner/switchboard/index.ts) selects +Pi + linear, with per-program and flag overrides. Set new bindings explicitly. Migrating an existing program requires checking its flow, tasks, and lifecycle hooks; changing the default constant alone is insufficient. Both harnesses implement `run` and `runTask`. diff --git a/AGENTS.md b/AGENTS.md index 773faf9cf..0ecf21c7c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -34,9 +34,9 @@ Each domain has a dedicated boundary: `@agent/types` (types); see [src/agent/README.md](src/agent/README.md) - **Shared** → `src/shared/`, stateless library code with no upward imports; see [src/shared/README.md](src/shared/README.md) -- **Programs** → configs, detection, framework registry and task stream in - `src/programs/`; runtime and type entries are `@programs` and - `@programs/types` +- **Programs** → program configs and `runProgram` in `src/programs/`; see + [src/programs/README.md](src/programs/README.md) and the + [developer interfaces](docs/developer-interfaces.md) - **TUI** → screens, primitives and content decks in `src/ui/tui/` Adding a new concern means finding the narrowest existing surface, not adding @@ -76,7 +76,7 @@ Agent SDK is a supported legacy fallback, deprecated as the default; retain it for major Pi vulnerabilities or gaps in support for new Anthropic models. This is the contribution policy, not a claim that every existing binding has -migrated: `DEFAULT_BINDING` is still Anthropic + linear. Set new bindings +migrated: `DEFAULT_BINDING` is Pi + linear. Set new bindings explicitly and check sequence-specific hooks before migrating existing flows. See [execution policy and model admission](.claude/skills/wizard-development/SKILL.md#execution-policy-and-model-admission) @@ -202,7 +202,8 @@ wizard run points. Full catalog: [`docs/local-dev.md`](docs/local-dev.md). - TypeScript everywhere. Use `type` (not `interface`) for framework context types so they satisfy `Record`. - All UI calls go through `getUI()` (returns `WizardUI` interface). Never import - the store directly from business logic. + the store directly from business logic. A program's `run` and `ciPreRun` + use the runner context they receive, not `getUI()`. - Shared helpers never call `getUI()`; they take a sink or return data. `debug()` reaches the UI through the sink `src/ui/index.ts` installs. - Outside `src/agent`, import the agent through `@agent` or `@agent/types`. Add diff --git a/docs/developer-interfaces.md b/docs/developer-interfaces.md new file mode 100644 index 000000000..4f4247304 --- /dev/null +++ b/docs/developer-interfaces.md @@ -0,0 +1,193 @@ +# Developer interfaces + +There are four ways to run the wizard. Most end users run it from the TUI. The +headless runner runs the same flow without a terminal UI. `runProgram` runs just +one program, and `runAgent` runs only the agent. + +| Way | Entry | Use it for | Reference | +| ------------ | --------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ---------------------------------------------------------------------------- | +| TUI | `npx @posthog/wizard`, through [`run-wizard.ts`](../src/lib/runners/run-wizard.ts) | End users setting up PostHog in a terminal | [README](../README.md) | +| Headless | `npx @posthog/wizard --ci`, through [`run-non-interactive.ts`](../src/lib/runners/run-non-interactive.ts) | CI and scripts, with no prompts | [Local credentials](local-dev.md#credentials-for-local-ci-and-headless-runs) | +| `runProgram` | `runProgram(programId, input, options)` from `@programs` | One program, from code, with its policy and telemetry | [`runProgram`](#runprogram), [programs reference](../src/programs/README.md) | +| `runAgent` | `runAgent(config, input, options)` from `@agent` | Only the agent, from a resolved config | [`runAgent`](#runagent), [agent reference](../src/agent/README.md) | + +![The TUI and the headless runner go through their own state to runProgram. The workbench calls runProgram directly. runProgram calls runAgent, and detection calls runAgent directly](images/wizard-run-paths.svg) + +The dashed boxes are the state the TUI and the headless runner keep above +`runProgram`. The workbench and other integrations call `runProgram` directly. + +`runProgram` and `runAgent` live in this repository and import through its path +aliases. The `@posthog/wizard` npm package publishes the CLI, not these +functions. + +## `runProgram` + +### What `runProgram` is for + +`runProgram` is a clean interface between the TUI, which is the interface, and +the programmatic parts of what a wizard program does. + +> ⚠️ **Temporary adapter.** The TUI and the headless runner reach `runProgram` +> through `runProgramAgent` in +> [`run-agent-legacy.ts`](../src/programs/run-agent-legacy.ts). This is a +> temporary adapter, and we will remove it in the full program. + +Three things use it: + +1. **The TUI and the headless runner.** Both call `runProgram` to run a full + wizard program. +2. **Testing.** The test workbench calls `runProgram` directly, with no TUI, + from the `wizard-program` service in + [wizard-workbench](https://github.com/PostHog/wizard-workbench). +3. **Other integrations.** If you want to integrate with the wizard more + directly, without everything on top, call `runProgram` yourself. + +### Signatures + +```ts +import { runProgram } from '@programs'; +import type { + ProgramInput, + ProgramOptions, + ProgramRunOutcome, +} from '@programs/types'; + +// The shape `@programs` exports. +export const signature: ( + programId: string, + input: ProgramInput, + options?: ProgramOptions, +) => Promise = runProgram; +``` + +- **`programId`** says which program runs. +- **`input`** says what to run and where: the program's run definition built + from its `ProgramConfig`, the install directory and the login. +- **`options`** is how the caller plugs in: credentials, questions, progress, + the approval and gate waits, flags and cancellation. + +`runProgram` resolves to one `ProgramRunOutcome`. + +### Field definitions + +| Field | Type | What it's for | +| ------------------------------------------------------------ | ------------------- | -------------------------------------------------------------------------------- | +| `programId` | `string` | Which program runs. It names analytics, the route and the gateway spend. | +| [`input`](../src/programs/run-program.ts#L67) | `ProgramInput` | What to run and where. `installDir` and `run` are required. | +| [`input.program`](../src/programs/run-program.ts#L57) | `ProgramSettings` | The program's settings from its `ProgramConfig`. | +| [`options`](../src/programs/run-program.ts#L90) | `ProgramOptions` | The login, questions, approval and gate waits, flags, progress and cancellation. | +| [`options.onProgress`](../src/programs/program-store.ts#L17) | `ProgramProgress` | Agent events and program data snapshots. Never awaited. | +| [Outcome](../src/programs/run-program.ts#L107) | `ProgramRunOutcome` | How the run ended, the agent's result, the final data and the report path. | + +The outcome's `data` and the `kind: 'program'` progress snapshots hold PostHog +tokens, including the refresh token. Don't log or serialize them. + +`flags.ci` and `flags.signup` skip the AI-processing approval. Set them only +when consent is already settled. + +### Do a quack + +[`run-program-quack.ts`](examples/run-program-quack.ts) is the smallest +`runProgram` call. It runs one prompt that replies `quack`, logs status lines +and prints the outcome. Each step has a comment. + +Run it from the repository root against the [local stack](local-dev.md): + +```bash +npx tsx --tsconfig tsconfig.json docs/examples/run-program-quack.ts +``` + +It prints `reply: quack` and `outcome: success`. + +### Cancellation + +Pass a `signal` to cancel the run. The credentials, approval and gate waits +receive it, and each must settle when it aborts. A cancelled run resolves to +`aborted`, not a rejection. + +### Failures + +Most endings resolve to an outcome instead of throwing. Check `outcome` and read +`failure`. The promise rejects only when the call itself can't run, such as an +input field that can't be copied. The cases are in +[`run-program.ts`](../src/programs/run-program.ts#L164). + +### Program callbacks + +A program's `run` and `ciPreRun` receive a runner context, `RunnerContext` or +`CiRunnerContext`, instead of calling `getUI()`. Both types come from +`@programs/types` and are defined in +[`runner-context.ts`](../src/programs/runner-context.ts). Build it when you +build the run from a `ProgramConfig`, before you call `runProgram`. + +## `runAgent` + +### What `runAgent` is for + +`runAgent` runs only the agent. Call it when you have a resolved route and no +program policy to apply. It doesn't log in, ask for consent, load flags or +resolve a route. The caller does those. + +### Signatures + +```ts +import { runAgent } from '@agent'; +import type { + AgentInteraction, + AgentProgress, + RunConfig, + RunInput, + RunResult, +} from '@agent/types'; + +// The shape `@agent` exports. +export const signature: ( + config: RunConfig, + input: RunInput, + options?: { + onProgress?: (event: AgentProgress) => unknown; + interaction?: AgentInteraction; + signal?: AbortSignal; + }, +) => Promise = runAgent; +``` + +- **`config`** says what the agent runs: the run definition, the route and the + tools. +- **`input`** says where and as whom: the project, the login and the flags. +- **`options`** carries progress, questions and cancellation. + +`runAgent` resolves to one `RunResult`. It mints its own gateway token from +`input.credentials`. + +### Field definitions + +| Field | Type | What it's for | +| -------------------------------------------------------------------- | -------------------- | ------------------------------------------------------------------------------- | +| [`config`](../src/agent/runner/shared/types.ts#L151) | `RunConfig` | What the agent runs, with its route and tools. | +| [`config.run`](../src/agent/runner/shared/types.ts#L48) | `AgentRunDefinition` | The prompt and run options, such as `collectTranscript`. | +| [`config.allowedTools`](../src/agent/runner/shared/types.ts#L174) | `readonly string[]` | Tools added to the base tools. | +| [`config.disallowedTools`](../src/agent/runner/shared/types.ts#L176) | `readonly string[]` | Tools removed from the base tools. | +| [`input`](../src/agent/runner/shared/types.ts#L205) | `RunInput` | Where and as whom: the project, the login and the flags. | +| `options` | | `onProgress` for agent events, `interaction` for questions, `signal` to cancel. | +| [Result](../src/agent/runner/shared/types.ts#L307) | `RunResult` | How the run ended, with a snapshot of its tasks and transcript. | + +### Callers + +| Caller | What it runs | +| --------------------------------------------------------------------------------- | ----------------------------------------------- | +| `runProgram` | One program's agent run. | +| `detectProjectsWithAgent` in [`agentic.ts`](../src/programs/detection/agentic.ts) | The agentic project scan, one call per attempt. | +| [`a3-fault-probe.no-jest.ts`](../scripts/a3-fault-probe.no-jest.ts) | A fault probe against a local gateway. | + +### Do a quack + +[`run-agent-quack.ts`](examples/run-agent-quack.ts) is the smallest `runAgent` +call. It builds a `RunConfig` with one prompt and no Write, Edit or Bash, and +prints the transcript tail and the outcome. Each step has a comment. + +```bash +npx tsx --tsconfig tsconfig.json docs/examples/run-agent-quack.ts +``` + +It prints `transcriptTail: quack` and `outcome: success`. diff --git a/docs/examples/run-agent-quack.ts b/docs/examples/run-agent-quack.ts new file mode 100644 index 000000000..4d1f73aff --- /dev/null +++ b/docs/examples/run-agent-quack.ts @@ -0,0 +1,110 @@ +/* eslint-disable no-console -- the example prints to the terminal */ +// Run only the agent with runAgent, against a local PostHog stack. +// +// npx tsx --tsconfig tsconfig.json docs/examples/run-agent-quack.ts +// +// Needs local PostHog on :8010 (with its ai-gateway) and context-mill on :8765. +// POSTHOG_PERSONAL_API_KEY logs in. WIZARD_CI_GATEWAY_TOKEN_FILE holds the gateway token. +// QUACK_INSTALL_DIR sets the project the agent runs in (default: the current directory). +import { + configureGatewayFromCIEnvironment, + runAgent, + RunOutcome, +} from '@agent'; +import type { RunConfig, RunInput } from '@agent/types'; +import { + Harness, + HAIKU_MODEL, + Sequence, + getSkillsBaseUrl, +} from '@shared/constants'; +import { initLocalDev, POSTHOG_LOCAL_URL } from '@shared/local-dev'; +import { getOrAskForProjectData } from '@utils/setup-utils'; + +// Point PostHog, skills and MCP at the local stack, like --local-posthog --local-context-mill --local-mcp. +initLocalDev({ localPosthog: true, localContextMill: true, localMcp: true }); + +// Log in with keys instead of the browser, the same way --ci does. +const apiKey = process.env.POSTHOG_PERSONAL_API_KEY; +if (!apiKey) throw new Error('Set POSTHOG_PERSONAL_API_KEY'); +const programId = 'posthog-integration'; // a program the local gateway admits +const login = await getOrAskForProjectData({ + signup: false, + ci: true, // with apiKey, this skips OAuth + apiKey, + baseUrl: POSTHOG_LOCAL_URL, + localMcp: true, + programId, +}); +// Use the token in WIZARD_CI_GATEWAY_TOKEN_FILE at WIZARD_CI_GATEWAY_URL instead of minting one. +configureGatewayFromCIEnvironment(login.projectId, 'us'); + +// What the agent runs: one prompt, a small model, no Write, Edit or Bash. +const config: RunConfig = { + programId, // pins the gateway spend + run: { + integrationLabel: 'quack', + prompt: () => 'Reply with the single word quack. Use no tools.', + collectTranscript: true, // keep the agent's output for snapshot.transcriptTail + requestRemark: false, // no closing remark + spinnerMessage: 'Quacking...', + successMessage: 'Quacked', + estimatedDurationMinutes: 1, + reportFile: '', + docsUrl: 'https://posthog.com/docs', + }, + composed: true, // a sub-run: no terminal outro + // runAgent doesn't resolve a route. Linear on the Anthropic harness keeps the transcript. + binding: { + sequence: Sequence.linear, + harness: Harness.anthropic, + model: HAIKU_MODEL, + }, + switchboard: { program: programId, composed: true, flags: {} }, + skillsBaseUrl: getSkillsBaseUrl(), + wizardFlags: {}, + wizardFlagPayloads: {}, + wizardMetadata: {}, + disallowedTools: ['Write', 'Edit', 'Bash'], +}; + +// Where and as whom: the project, the login and the flags. +const input: RunInput = { + installDir: process.env.QUACK_INSTALL_DIR ?? process.cwd(), + credentials: { + accessToken: login.accessToken, + refreshToken: login.refreshToken, + expiresAt: login.expiresAt, + projectApiKey: login.projectApiKey, + host: login.host, + projectId: login.projectId, + missingScopes: login.missingScopes, + }, + project: login.project, + apiUser: login.user, + flags: { + ci: false, + signup: false, + debug: false, + e2eAsk: false, + localMcp: true, + captureAio: false, + benchmark: false, + yaraReport: false, + }, + host: { baseUrl: POSTHOG_LOCAL_URL }, +}; + +// Run it. Each step the agent takes arrives as one activity line. +const result = await runAgent(config, input, { + onProgress: (event) => { + if (event.kind === 'activity') console.log(`activity: ${event.line}`); + }, +}); + +// Every ending resolves to a result. Print the reply and the outcome. +console.log(`transcriptTail: ${result.snapshot.transcriptTail ?? ''}`); +console.log(`outcome: ${result.outcome}`); +if (result.outcome !== RunOutcome.Success) + console.log(`failure: ${result.failure.message}`); +process.exit(result.outcome === RunOutcome.Success ? 0 : 1); diff --git a/docs/examples/run-program-quack.ts b/docs/examples/run-program-quack.ts new file mode 100644 index 000000000..587ce3cea --- /dev/null +++ b/docs/examples/run-program-quack.ts @@ -0,0 +1,96 @@ +/* eslint-disable no-console -- the example prints to the terminal */ +// Run one program with runProgram, against a local PostHog stack. +// +// npx tsx --tsconfig tsconfig.json docs/examples/run-program-quack.ts +// +// Needs local PostHog on :8010 (with its ai-gateway) and context-mill on :8765. +// POSTHOG_PERSONAL_API_KEY logs in. WIZARD_CI_GATEWAY_TOKEN_FILE holds the gateway token. +// QUACK_INSTALL_DIR sets the project the agent runs in (default: the current directory). +import { configureGatewayFromCIEnvironment, RunOutcome } from '@agent'; +import { runProgram } from '@programs'; +import type { ProgramProgress } from '@programs/types'; +import { Harness, HAIKU_MODEL, Sequence } from '@shared/constants'; +import { initLocalDev, POSTHOG_LOCAL_URL } from '@shared/local-dev'; +import { getOrAskForProjectData } from '@utils/setup-utils'; + +// Point PostHog, skills and MCP at the local stack, like --local-posthog --local-context-mill --local-mcp. +initLocalDev({ localPosthog: true, localContextMill: true, localMcp: true }); + +// Log in with keys instead of the browser, the same way --ci does. +const apiKey = process.env.POSTHOG_PERSONAL_API_KEY; +if (!apiKey) throw new Error('Set POSTHOG_PERSONAL_API_KEY'); +const programId = 'posthog-integration'; // a program the local gateway admits +const login = await getOrAskForProjectData({ + signup: false, + ci: true, // with apiKey, this skips OAuth + apiKey, + baseUrl: POSTHOG_LOCAL_URL, + localMcp: true, + programId, +}); +// Use the token in WIZARD_CI_GATEWAY_TOKEN_FILE at WIZARD_CI_GATEWAY_URL instead of minting one. +configureGatewayFromCIEnvironment(login.projectId, 'us'); + +// Log status lines as the program reports them. runProgram never waits for this. +function logProgress(progress: ProgramProgress): void { + if (progress.kind === 'program') return; // a data snapshot; it holds tokens, don't log it + const { event } = progress; + if (event.kind === 'status') console.log(`status: ${event.message}`); + if (event.kind === 'lifecycle') console.log(`lifecycle: ${event.phase}`); +} + +const result = await runProgram( + programId, + { + installDir: process.env.QUACK_INSTALL_DIR ?? process.cwd(), + // A caller-built run in place of the program's own: one prompt, and keep the reply. + run: { + integrationLabel: 'quack', + prompt: () => 'Reply with the single word quack. Use no tools.', + collectTranscript: true, // keep the agent's output for the reply below + requestRemark: false, // no closing remark + spinnerMessage: 'Quacking...', + successMessage: 'Quacked', + estimatedDurationMinutes: 1, + reportFile: '', + docsUrl: 'https://posthog.com/docs', + }, + program: { disallowedTools: ['Write', 'Edit', 'Bash'] }, + // The login from above, so runProgram skips its own login step. + credentials: { + posthog: { + accessToken: login.accessToken, + refreshToken: login.refreshToken, + expiresAt: login.expiresAt, + projectApiKey: login.projectApiKey, + host: login.host, + projectId: login.projectId, + missingScopes: login.missingScopes, + }, + project: login.project, + apiUser: login.user, + }, + composed: true, // a sub-run: no terminal outro + // A small model on the linear Anthropic route, where the transcript is kept. + overrides: { + sequence: Sequence.linear, + harness: Harness.anthropic, + model: HAIKU_MODEL, + }, + flags: { localMcp: true }, + host: { baseUrl: POSTHOG_LOCAL_URL }, + wizardFlags: {}, // no flag snapshot to load + }, + { + // You approved AI data processing for this local test user. + awaitAiApproval: () => Promise.resolve(true), + onProgress: logProgress, + }, +); + +// Endings resolve to an outcome. The agent's reply is in its settled run's transcript. +const reply = result.settledRuns[0]?.result.snapshot.transcriptTail ?? ''; +console.log(`reply: ${reply}`); +console.log(`outcome: ${result.outcome}`); +if (result.failure) console.log(`failure: ${result.failure.message}`); +process.exit(result.outcome === RunOutcome.Success ? 0 : 1); diff --git a/docs/images/wizard-run-paths.svg b/docs/images/wizard-run-paths.svg new file mode 100644 index 000000000..9b2be6fbf --- /dev/null +++ b/docs/images/wizard-run-paths.svg @@ -0,0 +1,53 @@ + + How each way to run the wizard reaches runAgent + + + + + + + + + + TUI + + + Headless runner + + + + WizardStore + session and Ink screens + + + WizardSession + and LoggingUI + + + Workbench and + other integrations + + + Detection and + standalone callers + + + + runProgram + ProgramStore + + + + runAgent + + + + + + + + + + + + diff --git a/src/agent/README.md b/src/agent/README.md index 4116171cd..55281ce27 100644 --- a/src/agent/README.md +++ b/src/agent/README.md @@ -1,117 +1,33 @@ # Agent -The agent runs one program's AI pipeline against a project directory. It takes -resolved data in, reports through progress events, asks through an injected -answerer, and returns a result. It never reads a session, a store or a UI. +> ⚠️ **The bindings table will be gone.** The program bindings table still lives +> in the agent. By the end of this refactor it moves to programs, and the agent +> takes a resolved route only. -## Signatures +The agent runs one AI pipeline against a project. It takes resolved data in, +reports through progress events, asks through an answerer you pass, and returns +a result. It never reads a session, a store or a UI. -Import runtime values from `@agent` and types from `@agent/types`. Nothing -outside `src/agent` imports deeper; lint and the architecture test reject it. - -```ts -import { runAgent, RunOutcome } from '@agent'; -import type { RunConfig, RunInput, RunResult, AgentProgress } from '@agent/types'; - -runAgent(config: RunConfig, input: RunInput, options?: { - onProgress?: (event: AgentProgress) => void; - interaction?: AgentInteraction; - signal?: AbortSignal; -}): Promise -``` - -- `RunConfig`: the program id, its `AgentRunDefinition` (prompt, skill, tools, - copy), the resolved `binding` (sequence, harness, model), the switchboard - inputs, the skills origin, flag snapshot, trace tags, tool allow and deny - lists, seed tasks and bound completion `hooks`. -- `RunInput`: install directory, resolved credentials, project and user - payloads, skill id, detected integration, `flags` (`ci`, `signup`, `debug`, - `e2eAsk`, `localMcp`, `captureAio`, `benchmark`, `yaraReport`) and the host - the CLI was told. -- `RunResult`: `outcome` is `RunOutcome.Success | Aborted | Failed | Crashed`. - Success may carry an `outro`; the other three carry a `failure` - (`AgentFailure`: message, outro data, error, exit code, error code, detail). - Every result carries `skillId` and a `snapshot` of what the run reported: - tasks, status lines, stage, token usage totals, final cost, dashboard and - notebook URLs, handoff text. -- `AgentProgress`: one event per thing the run reports, in emission order. - Kinds: `lifecycle`, `spinner`, `log`, `status`, `tasks`, `stage`, `url`, - `usage`, `finalCost`, `authError`, `handoff`, `completion`. Payloads are - copies, never live objects. -- `AgentInteraction`: every member optional. `ask(question, { signal })` - resolves with answers, and `taskNotice(notice, { signal })` resolves with - whether to keep an optional task. Each request has its own signal, which - aborts when that request times out, the host aborts the run, or another task - fails the run; on abort the host dismisses that request alone, without - throwing. -- Errors: the agent does not exit the process and returns decided failures. A - caught coded error becomes `Failed`. An uncoded throw becomes `Crashed` with - the error attached. A gateway 401 returns an auth failure. The host decides - whether to show auth UI. `Aborted` means the host's signal cancelled the run; - an agent that stops itself with `[ABORT]` returns `Failed` with its abort - code. -- Analytics shutdown is host-owned: the agent never sends the terminal - `setup wizard finished` event. The host sends it from the outcome: `Success` - is `success`, `Aborted` is `cancelled`, `Failed` and `Crashed` are `error`. - -Other runtime exports: `resolveBinding`, `shouldDisableAsk`, `initializeAgent`, -`executeAgent`, `buildRunTags`, `AgentSignals`, -`configureGatewayFromCIEnvironment`, `downloadSkill`, `WIZARD_TOOL_NAMES`, -`LONGER_ASK_TIMEOUT_MS`, `flushScanReport`, and `runMcpPromptViaSdk`, which -loads the streaming module on first call. - -Minimal invocation: +To call it from code, use `runAgent`. The +[developer interfaces](../../docs/developer-interfaces.md#runagent) cover it. -```ts -const result = await runAgent(config, input, { - onProgress: (event) => { - if (event.kind === 'log') console.log(event.message); - }, - interaction: { - ask: async (question) => answersFor(question), - }, -}); -if (result.outcome !== RunOutcome.Success) { - process.exitCode = result.failure.exitCode ?? 1; -} -``` +## What goes in and out -`src/agent/__tests__/run-agent-standalone.test.ts` runs this with no UI, no -store and no registry. +- **In.** A `RunConfig`, a `RunInput`, and callbacks for progress and questions. +- **Out.** One `RunResult` at the end. +- **Never in.** A session, a store or the UI. -## Intent +## Where things live -Programs call the agent to do the work a skill describes. The TUI and the -headless runner observe the run through `onProgress` and answer it through -`interaction`; today `src/programs/run-agent-legacy.ts` does both on top of -the session. +| What | Where | +| ----------------------------------------- | ------------------------------------------------- | +| The entry points | [`index.ts`](index.ts) and [`types.ts`](types.ts) | +| The run types: config, input and result | [`shared/types.ts`](runner/shared/types.ts) | +| Progress events and the answerer | [`progress.ts`](progress.ts) | +| Sequences, harnesses and route resolution | [`runner`](runner/README.md) | +| The wizard tools both harnesses share | [`tools`](tools) | +| The benchmark pipeline | [`middleware`](middleware) | +| Security scans of what the run installs | [`yara-hooks.ts`](yara-hooks.ts) | -Without `onProgress` the run completes and its snapshot still comes back in the -result. Without `interaction` the agent installs no ask bridge: `wizard_ask` -returns its "not available" error and optional task notices are declined, which -is what a `--ci` run does. A throwing observer is logged and the run continues. - -## Architecture - -The agent owns run state for one invocation: the task queue, phase, status, -resolved skill, handoff text, usage and the final result. It depends on -`src/shared` and on `src/env.ts`, and on program types only until the bindings -table moves to programs. - -```text -caller ── RunConfig + RunInput ──▶ runAgent - │ prepareRun: gateway mint, triage provider - ▼ - sequence (linear | orchestrator) - │ - harness (anthropic | pi) ── tools (MCP or pi-native) - │ - onProgress ◀── events ───┤──── questions ──▶ interaction - ▼ - RunResult -``` - -`runner/` holds the dispatcher, sequences, harnesses and the switchboard. -`tools/` holds the wizard tools shared by both harnesses. `middleware/` holds -the benchmark pipeline. `progress.ts` defines the event and interaction -contracts; `yara-hooks.ts` scans what the run installs. +Import runtime values from `@agent` and types from `@agent/types`. Nothing +outside the agent imports deeper, and lint rejects it. diff --git a/src/agent/runner/README.md b/src/agent/runner/README.md index 480c1280b..03f572112 100644 --- a/src/agent/runner/README.md +++ b/src/agent/runner/README.md @@ -1,149 +1,31 @@ -# agent runner +# Agent runner -How an agent run is assembled. Everything under this directory is plumbing — the -pieces that decide _how_ a program runs (which query shape, which agent SDK, -which model) and the pieces that then actually run it. +> ⚠️ **The bindings table will be gone.** The switchboard still holds the +> program bindings table. By the end of this refactor it moves to programs, and +> the runner takes a resolved route only. -``` - ┌──────────────┐ ┌─────────────┐ ┌────────────────────────────┐ - │ │ │ │────▶│ sequence (query shape) │ - │ programs │────▶│ switchboard │ │ linear | orchestrator │ - │ │ │ │ └────────────────────────────┘ - │ integration │ │ binds each │ - │ audit │ │ program to │ ┌────────────────────────────┐ - │ migration │ │ a pair │────▶│ harness (SDK adapter) │ - │ ... │ │ │ │ anthropic | pi | ... │ - └──────────────┘ └─────────────┘ └────────────────────────────┘ -``` - -## Execution policy - -Use Pi for new work and prefer the orchestrator sequence. Linear execution is -retained for very simple tasks and legacy support. The Anthropic Agent SDK is a -supported legacy fallback, deprecated as the default, retained for major Pi -vulnerabilities or gaps in support for new Anthropic models. - -Existing `DEFAULT_BINDING` remains Anthropic + linear; explicit program bindings -and flags determine actual behavior. Both harnesses implement `run` and -`runTask`. Composed sub-runs are clamped to linear, and linear-only -post-run/outro hooks do not automatically transfer to an orchestrated flow. - -New models require Wizard capabilities **and** mint model/effort allowlists, -gateway provider/transport support, and compatibility with required Wizard and -security-triage prompt policies. Local model constants cannot bypass admission. -See the -[development guide](../../../.claude/skills/wizard-development/SKILL.md#execution-policy-and-model-admission) -for the coordinated change checklist. - -## The pieces - -Five layers, each with its own job. Nothing crosses layers unless it has to. - -**The entry point** (`index.ts`) is the front door: -`runAgent(config, input, {onProgress?, interaction?, signal?}) → RunResult`. It takes -resolved execution data and an invocation snapshot (`shared/types.ts`), reports -through `onProgress` and asks through `interaction` (`../progress.ts`), and -returns every ending as a result. It never renders, reads a session or exits. -The gates, OAuth, flags and binding lookup that used to run here live in -`src/programs/run-agent-legacy.ts`, which also maps progress back onto -`getUI()` for today's runners. - -**Prepare** (`shared/bootstrap.ts`) is the on-ramp inside the agent: logging -targets, the gateway mint and the scan-triage classifier. Whether the run turns -out to be linear or orchestrator, anthropic or pi, the setup is the same. - -**The switchboard** (`switchboard/`) is the router. Given a program id + the -fetched flags + any CLI overrides, it returns a `ProgramBinding` — which query -shape (sequence), which agent SDK (harness), which model. Two independent -middleware chains, one per axis, apply precedence rules (CLI > flag > program -config > default). This is the only layer that makes routing decisions. - -**Sequences** (`sequence/`) are LLM query shapes. Once the switchboard has -picked one, that sequence takes over the run and owns _how the LLM's work is -shaped_. See `sequence/README.md`. - -- **linear** — one long conversation with the model, start to finish. -- **orchestrator** — many focused conversations coordinated by a task queue, - each with its own prompt, tools, and model. - -**Harnesses** (`harness/`) are SDK adapters. Sequences don't call Anthropic's or -pi.dev's SDKs directly — they go through a harness, which knows how to translate -a run request into that SDK's shape. All harnesses drive the PostHog LLM +The runner decides how an agent run happens, then runs it. It picks a sequence +and a harness for the program, and drives the model through the PostHog LLM gateway. -- **anthropic** — wraps Anthropic's official Claude Agent SDK. See - `harness/anthropic/README.md`. -- **pi** — wraps pi.dev's coding-agent library. See `harness/pi/README.md`. - -## How they connect - -- Prepare mints the gateway token and builds triage for the resolved harness. -- The switchboard knows which sequences and harnesses exist (via its two - registries), but not what they do. -- A sequence knows how to shape a conversation, but delegates the actual model - call to a harness. -- A harness adapts its SDK, gateway transport, security hooks, and tool surface. +To call it, use `runAgent`. The +[developer interfaces](../../../docs/developer-interfaces.md#runagent) cover it. -Each layer is replaceable. - -## Ownership map - -```mermaid -%%{init: {"block": {"padding": 20}}}%% -block-beta - columns 11 - hostBand["Host: programs and UI"]:11 - runProgramAgent["runProgramAgent"]:3 space:1 wizardAbort["wizardAbort"]:3 space:4 - space:11 - runnerBand["Agent runner"]:11 - runAgent["runAgent"]:3 space:1 runResult["RunResult"]:3 space:4 - space:11 - sequenceBand["Orchestrator sequence"]:11 - runOrchestrator["runOrchestrator"]:3 space:1 sequenceResult["SequenceResult"]:3 space:4 - space:11 - drainQueue["drainQueue"]:3 space:5 runAbort["AbortController"]:3 - space:11 - harnessBand["Selected harness"]:11 - agentHarness["AgentHarness"]:3 space:1 agentResult["AgentResult"]:3 space:1 signal["TaskRunInputs.signal"]:3 - space:11 - sdkBand["External model SDK"]:11 - sdk["Selected SDK"]:3 space:8 - - runProgramAgent --> runAgent - runAgent --> runOrchestrator - runOrchestrator --> drainQueue - drainQueue --> agentHarness - agentHarness --> sdk - agentHarness --> agentResult - agentResult --> sequenceResult - sequenceResult --> runResult - runResult --> wizardAbort - drainQueue --> runAbort - runAbort --> signal +## The pieces - classDef owner fill:#9ca3af1f,stroke:#9ca3af,stroke-width:1.5px - classDef changed fill:#3b82f626,stroke:#3b82f6,stroke-width:2px - class hostBand,runnerBand,sequenceBand,harnessBand,sdkBand owner - class runResult,agentResult,runAbort changed -``` +| Piece | What it does | Where | +| ----------- | -------------------------------------------------------------------------------- | ------------------------------------- | +| Entry point | `runAgent` takes a run config and input, and returns one result. | [`index.ts`](index.ts) | +| Prepare | Sets up logging, the gateway token and the scan classifier. | [`bootstrap.ts`](shared/bootstrap.ts) | +| Switchboard | Picks the sequence, harness and model for a program. | [`switchboard`](switchboard/index.ts) | +| Sequence | Shapes the work: one conversation (linear), or many from a queue (orchestrator). | [`sequence`](sequence/README.md) | +| Harness | Adapts one SDK: the Anthropic Agent SDK or Pi. | [`harness`](harness/types.ts) | -Calls descend on the left, results return through the middle, and cancellation -moves down the right. Blue marks the result contracts and run-scoped abort. -On the first fatal task result, `drainQueue` stops scheduling, cancels -active work and pending asks, joins siblings, then preserves that failure for -the host to present. +## Execution policy -## Flow +Use Pi for new work, and prefer the orchestrator. Linear is for very simple +tasks. The Anthropic Agent SDK is a legacy fallback. `DEFAULT_BINDING` is Pi and +linear. -1. The caller runs its gates, authenticates, fetches PostHog flags and resolves - a `ProgramBinding { sequence, harness, model }`; analytics tags the run. -2. `runAgent(config, input, options)` prepares (mint, triage). -3. Sequence takes over — shapes the LLM's work into one conversation (linear) or - many (orchestrator), reporting through `onProgress`. -4. Harness drives each conversation through its SDK, using the bound model, on - the PostHog LLM gateway. -5. The scan report flushes; `runAgent` returns a `RunResult`. -6. The caller applies it: a decided failure goes to `wizardAbort` with the - terminal status its outcome names, a crash is rethrown for the runner's own - handling, and a non-composed success sends the terminal success analytics. - The agent sends no terminal analytics. +A new model needs gateway admission as well as wizard support. See the +[development guide](../../../.claude/skills/wizard-development/SKILL.md#execution-policy-and-model-admission). diff --git a/src/programs/README.md b/src/programs/README.md new file mode 100644 index 000000000..6a297ab32 --- /dev/null +++ b/src/programs/README.md @@ -0,0 +1,46 @@ +# Programs + +> ⚠️ **This changes by the end of this refactor.** +> +> - Each program moves into its own folder with everything it owns, so a team +> can own a full program through `CODEOWNERS`. +> - The temporary adapter `runProgramAgent` in +> [`run-agent-legacy.ts`](run-agent-legacy.ts) is removed. +> - Program configs stop taking a `WizardSession` and stop calling `getUI()`. + +A program is one thing the wizard does for a user, such as adding PostHog or +setting up error tracking. Each program is a `ProgramConfig`: the screens the +TUI walks, the agent run it performs, and its settings. + +To run a program from code, call `runProgram`. The +[developer interfaces](../../docs/developer-interfaces.md) cover it. + +## What goes in and out of `runProgram` + +- **In.** A `ProgramInput`, and callbacks for login, approval, questions and + progress. +- **Out.** One `ProgramRunOutcome` at the end. +- **Never in.** A session, a store or the UI. + +## Where things live + +| What | Where | +| --------------------------------------------- | --------------------------------------------------------------------------------- | +| One program's config, steps and prompt | Its own folder, such as [`metrics`](metrics/index.ts) | +| The list of every program | [`program-registry.ts`](program-registry.ts) | +| The `ProgramConfig` and step types | [`program-step.ts`](program-step.ts) | +| Running one program from explicit inputs | [`run-program.ts`](run-program.ts) | +| What a program's `run` receives from a runner | [`runner-context.ts`](runner-context.ts) | +| Framework detection and project scoping | [`detection`](detection/index.ts) | +| Framework integrations | [`frameworks`](frameworks) and [`frameworks/registry.ts`](frameworks/registry.ts) | +| Commands that pick a program by skill | [`dispatch-family.ts`](dispatch-family.ts) | + +Import runtime values from `@programs` and types from `@programs/types`. + +## Add a program + +Follow the +[adding-skill-program](../../.claude/skills/adding-skill-program/SKILL.md) +skill. A framework integration uses +[adding-framework-support](../../.claude/skills/adding-framework-support/SKILL.md) +instead. diff --git a/tsconfig.json b/tsconfig.json index b3b6d448f..9947307f6 100644 --- a/tsconfig.json +++ b/tsconfig.json @@ -19,6 +19,7 @@ "src/**/*", "test/**/*", "e2e-harness/**/*", + "docs/examples/**/*", "types/**/*" ], "exclude": [