Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude/skills/adding-skill-program/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,8 +62,8 @@ changing execution behavior.
3. Register the config in
[PROGRAM_REGISTRY](../../../src/programs/program-registry.ts) and add its
Pi/orchestrator entry to
[PROGRAM_BINDINGS](../../../src/agent/runner/switchboard/index.ts).
[Existing binding checks](../../../src/agent/runner/__tests__/switchboard.test.ts)
[PROGRAM_BINDINGS](../../../src/programs/binding.ts).
[Existing binding checks](../../../src/programs/__tests__/switchboard.test.ts)
enforce coverage; `ProgramId` currently widens to `string`.
4. For a standalone native command, create a command module with
[nativeCommandFactory](../../../src/commands/factories/native-command-factory.ts)
Expand Down
6 changes: 3 additions & 3 deletions .claude/skills/wizard-development/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ infrastructure should consume those boundaries.
| Framework detection, context, env conventions | [FrameworkConfig](../../../src/programs/framework-config.ts) and [framework configs](../../../src/programs/frameworks/) |
| Integration instructions and orchestrator flows/tasks | [context-mill](https://github.com/PostHog/context-mill) |
| Programs, steps, prerequisites and outcomes | [programs](../../../src/programs/) |
| Sequence, harness, model and effort selection | [switchboard](../../../src/agent/runner/switchboard/) |
| Sequence, harness, model and effort selection | [program bindings](../../../src/programs/binding.ts) and [agent clamps](../../../src/agent/runner/switchboard/) |
| Local tool permissions and scanner adapters | [agent-interface](../../../src/agent/agent-interface.ts), [YARA hooks](../../../src/agent/yara-hooks.ts), [Pi security](../../../src/agent/runner/harness/pi/security.ts) |
| Scanner rules | [warlock](https://github.com/PostHog/warlock) |
| Token admission and budgets | [PostHog mint endpoint](https://github.com/PostHog/posthog/blob/master/posthog/llm/wizard_gateway_token.py) and [ai-gateway](https://github.com/PostHog/ai-gateway) |
Expand All @@ -43,8 +43,8 @@ infrastructure should consume those boundaries.
new Anthropic models.

Existing routing has not all migrated:
[DEFAULT_BINDING](../../../src/agent/runner/switchboard/index.ts) still
selects Anthropic + linear, with per-program and flag overrides. Set new
[DEFAULT_AGENT_BINDING](../../../src/agent/default-binding.ts) selects Pi + linear
for standalone runs; programs apply their own binding and flag overrides. Set new
bindings explicitly. Migrating an existing program requires checking its flow,
tasks, and lifecycle hooks; changing the default constant alone is insufficient.
Both harnesses implement `run` and `runTask`.
Expand Down
18 changes: 10 additions & 8 deletions .claude/skills/wizard-development/references/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,10 +60,11 @@ example. Native command modules still need registration in

## Switchboard contract

`resolveBinding(ctx, role?)` is the routing seam. Read its exported types rather
than copying their fields into another document. It receives the program, flag
snapshot/payloads, composition state, and development overrides, returning the
binding and stamping a trace of the selected precedence rungs.
`resolveProgramBinding(ctx)` in [programs](../../../../src/programs/binding.ts)
is the routing seam. It receives the program, flag snapshot/payloads,
composition state and development overrides, returning a resolved binding and
stamping a trace of the selected precedence rungs. The agent receives this
binding and any pre-resolved task-role routes as data.

- Harness/model: development CLI override, declared flag route, per-program
binding, default.
Expand All @@ -72,7 +73,8 @@ binding and stamping a trace of the selected precedence rungs.

[Harness](../../../../src/agent/runner/switchboard/harness.ts) and
[sequence](../../../../src/agent/runner/switchboard/sequence.ts) contain the
exact chains. Published builds omit CLI overrides. `RUN_SURFACE` can disable
generic precedence and clamp chains over caller-supplied policy. Published
builds omit CLI overrides. `RUN_SURFACE` can disable
harness experiments; static bindings and harness capabilities also affect
resolution. Composed sub-runs remain linear even when a CLI override requests
orchestration.
Expand All @@ -84,11 +86,11 @@ admitted by the minted token and gateway. Local routing cannot bypass that
external policy; see the
[model admission checklist](../SKILL.md#execution-policy-and-model-admission).

Flags belong in
[switchboard/flags](../../../../src/agent/runner/switchboard/flags/).
Program flags belong in
[experiments](../../../../src/programs/experiments/).
Experiments declare their program scope; malformed payloads yield no experiment
route. Reuse the
[switchboard tests](../../../../src/agent/runner/__tests__/switchboard.test.ts)
[switchboard tests](../../../../src/programs/__tests__/switchboard.test.ts)
and experiment tests to check full bindings and isolation of unrelated programs.
Do not add a second flag-reading path inside a harness or sequence.

Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ Agent SDK is a supported legacy fallback, deprecated as the default; retain it
for major Pi vulnerabilities or gaps in support for new Anthropic models.

This is the contribution policy, not a claim that every existing binding has
migrated: `DEFAULT_BINDING` is still Anthropic + linear. Set new bindings
migrated. The default, `DEFAULT_AGENT_BINDING`, is Pi + linear. Set new bindings
explicitly and check sequence-specific hooks before migrating existing flows.
See
[execution policy and model admission](.claude/skills/wizard-development/SKILL.md#execution-policy-and-model-admission)
Expand Down
2 changes: 1 addition & 1 deletion docs/benchmarking.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ Everything below ships in this repo (`wizard/`) and its workbench
orchestrator on pi, per-task models from context-mill frontmatter; off → the
linear anthropic default). Per-stage variations ride
`wizard-orchestrator-override` payloads (`{stage: {model?, effort?}}`,
variant keys in `wizard/src/agent/runner/switchboard/flags/schemes.ts`).
variant keys in `wizard/src/programs/experiments/schemes.ts`).
The baseline is `{"wizard-orchestrator":"false"}` — never an empty override,
or live remote flags leak into the baseline.

Expand Down
5 changes: 0 additions & 5 deletions scripts/a3-fault-probe.no-jest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -66,11 +66,6 @@ const config: RunConfig = {
harness: harness as Harness,
model: DEFAULT_AGENT_MODEL,
},
switchboard: {
program: 'fault-probe',
flags: {},
cliHarness: harness as Harness,
},
skillsBaseUrl: 'http://127.0.0.1:1',
wizardFlags: {},
wizardFlagPayloads: {},
Expand Down
5 changes: 0 additions & 5 deletions src/__tests__/architecture/known-violations.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,5 @@
{
"violations": [
"src/agent/runner/switchboard/flags/index.ts -> src/programs/types.ts",
"src/agent/runner/switchboard/flags/schemes.ts -> src/programs/types.ts",
"src/agent/runner/switchboard/index.ts -> src/programs/types.ts",
"src/commands/ai-observability.ts -> src/programs/ai-observability/index.ts",
"src/commands/audit.ts -> src/programs/audit/index.ts",
"src/commands/basic-integration/ci-install.ts -> src/programs/posthog-integration/index.ts",
Expand Down Expand Up @@ -42,7 +39,6 @@
"src/programs/ai-opt-in-gate.ts -> src/lib/wizard-session.ts",
"src/programs/audit/index.ts -> src/lib/headless-mode.ts",
"src/programs/audit/index.ts -> src/lib/wizard-session.ts",
"src/programs/audit/ledger-watcher.ts -> src/lib/file-watcher.ts",
"src/programs/audit/ledger-watcher.ts -> src/ui/index.ts",
"src/programs/audit/types.ts -> src/lib/wizard-session.ts",
"src/programs/authenticate.ts -> src/lib/wizard-session.ts",
Expand Down Expand Up @@ -111,7 +107,6 @@
"src/programs/shared/posthog-cli-preinstall.ts -> src/steps/install-cli-steering/index.ts",
"src/programs/shared/posthog-cli-preinstall.ts -> src/ui/index.ts",
"src/programs/task-stream/destinations/posthog.ts -> src/lib/wizard-session.ts",
"src/programs/task-stream/event-plan-watcher.ts -> src/lib/file-watcher.ts",
"src/programs/task-stream/event-plan-watcher.ts -> src/ui/tui/store.ts",
"src/programs/task-stream/task-stream-push.ts -> src/lib/wizard-session.ts",
"src/programs/task-stream/task-stream-push.ts -> src/ui/tui/store.ts",
Expand Down
20 changes: 11 additions & 9 deletions src/agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,11 @@ runAgent(config: RunConfig, input: RunInput, options?: {
}): Promise<RunResult>
```

- `RunConfig`: the program id, its `AgentRunDefinition` (prompt, skill, tools,
copy), the resolved `binding` (sequence, harness, model), the switchboard
inputs, the skills origin, flag snapshot, trace tags, tool allow and deny
lists, seed tasks and bound completion `hooks`.
- `RunConfig`: the opaque program id, its `AgentRunDefinition` (prompt, skill,
tools, copy), the resolved `binding` (sequence, harness, model and task-role
routes), supplied program commandments and stage policy, the skills origin,
flag snapshot, trace tags, tool allow and deny lists, seed tasks and bound
completion `hooks`.
- `RunInput`: install directory, resolved credentials, project and user
payloads, skill id, detected integration, `flags` (`ci`, `signup`, `debug`,
`e2eAsk`, `localMcp`, `captureAio`, `benchmark`, `yaraReport`) and the host
Expand Down Expand Up @@ -54,11 +55,12 @@ runAgent(config: RunConfig, input: RunInput, options?: {
`setup wizard finished` event. The host sends it from the outcome: `Success`
is `success`, `Aborted` is `cancelled`, `Failed` and `Crashed` are `error`.

Other runtime exports: `resolveBinding`, `shouldDisableAsk`, `initializeAgent`,
`executeAgent`, `buildRunTags`, `AgentSignals`,
`configureGatewayFromCIEnvironment`, `downloadSkill`, `WIZARD_TOOL_NAMES`,
`LONGER_ASK_TIMEOUT_MS`, `flushScanReport`, and `runMcpPromptViaSdk`, which
loads the streaming module on first call.
Other runtime exports: `DEFAULT_AGENT_BINDING` for standalone callers,
`resolveHarness` and `harnessRunsTasks`, which programs resolve a binding with,
`shouldDisableAsk`, `initializeAgent`, `executeAgent`, `buildRunTags`,
`AgentSignals`, `configureGatewayFromCIEnvironment`, `downloadSkill`,
`WIZARD_TOOL_NAMES`, `LONGER_ASK_TIMEOUT_MS`, `flushScanReport`, and
`runMcpPromptViaSdk`, which loads the streaming module on first call.

Minimal invocation:

Expand Down
10 changes: 8 additions & 2 deletions src/agent/__tests__/commandments.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import { WIZARD_COMMANDMENTS } from '@agent/commandments';
import { assembleCommandments } from '@agent/runner/switchboard/commandments';
import { getProgramCommandments } from '@programs/commandments';
import { Harness, Sequence } from '@shared/constants';

const global = WIZARD_COMMANDMENTS.join('\n');
Expand All @@ -10,7 +11,13 @@ const prompt = (
harness: Harness,
sequence: Sequence,
program = 'posthog-integration',
) => assembleCommandments({ program, sequence, harness, caps: CAPS });
) =>
assembleCommandments({
programCommandments: getProgramCommandments(program),
sequence,
harness,
caps: CAPS,
});

const COMBOS = [
['anthropic', Harness.anthropic, Sequence.linear],
Expand Down Expand Up @@ -148,7 +155,6 @@ describe('commandments by axis', () => {
describe('runtime caps gate the pi runtime notes', () => {
const withCaps = (caps: { bash: boolean; posthogMcp: boolean }) =>
assembleCommandments({
program: 'warehouse-source',
sequence: Sequence.linear,
harness: Harness.pi,
caps,
Expand Down
16 changes: 16 additions & 0 deletions src/agent/__tests__/harness-capabilities.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
import { Harness } from '@shared/constants';
import { HARNESS_OPTIONS } from '@agent/runner/switchboard/harness';
import { HARNESS_RUNS_TASKS } from '@agent/runner/switchboard/resolve-harness';

describe('harness capabilities', () => {
it.each(Object.values(Harness))(
'records whether the %s backend implements runTask',
(harness) => {
const backend = HARNESS_OPTIONS[harness];
expect(backend).toBeDefined();
expect(HARNESS_RUNS_TASKS[harness]).toBe(
typeof backend?.runTask === 'function',
);
},
);
});
40 changes: 0 additions & 40 deletions src/agent/__tests__/run-agent-standalone.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -171,10 +171,6 @@ vi.mock('@agent/runner/switchboard/harness', () => {
harnessState.selected.push(name);
return { ...fake, name };
},
resolveHarness: (ctx: { cliHarness?: Harness }) => ({
harness: ctx.cliHarness ?? Harness.pi,
model: DEFAULT_AGENT_MODEL,
}),
};
});

Expand Down Expand Up @@ -237,7 +233,6 @@ const config = (over: Partial<RunConfig> = {}): RunConfig => ({
},
composed: false,
binding: { sequence: Sequence.linear, harness: Harness.pi, model: 'm' },
switchboard: { program: 'test-program', flags: {} },
skillsBaseUrl: 'https://skills.test',
wizardFlags: {},
wizardFlagPayloads: {},
Expand Down Expand Up @@ -316,11 +311,6 @@ describe('runAgent standalone', () => {
const running = runAgent(
config({
binding: { harness, sequence, model: DEFAULT_AGENT_MODEL },
switchboard: {
program: 'test-program',
flags: {},
cliHarness: harness,
},
}),
input(),
{ interaction: { ask }, onProgress: (event) => events.push(event) },
Expand Down Expand Up @@ -387,11 +377,6 @@ describe('runAgent standalone', () => {
const result = await runAgent(
config({
binding: { harness, sequence, model: DEFAULT_AGENT_MODEL },
switchboard: {
program: 'test-program',
flags: {},
cliHarness: harness,
},
}),
input(),
);
Expand Down Expand Up @@ -427,11 +412,6 @@ describe('runAgent standalone', () => {
sequence: Sequence.orchestrator,
model: DEFAULT_AGENT_MODEL,
},
switchboard: {
program: 'test-program',
flags: {},
cliHarness: Harness.anthropic,
},
}),
input(),
);
Expand All @@ -454,11 +434,6 @@ describe('runAgent standalone', () => {
sequence: Sequence.orchestrator,
model: DEFAULT_AGENT_MODEL,
},
switchboard: {
program: 'test-program',
flags: {},
cliHarness: Harness.anthropic,
},
}),
input(),
);
Expand All @@ -482,11 +457,6 @@ describe('runAgent standalone', () => {
sequence: Sequence.orchestrator,
model: DEFAULT_AGENT_MODEL,
},
switchboard: {
program: 'test-program',
flags: {},
cliHarness: Harness.pi,
},
}),
input(),
);
Expand Down Expand Up @@ -532,11 +502,6 @@ describe('runAgent standalone', () => {
sequence: Sequence.orchestrator,
model: DEFAULT_AGENT_MODEL,
},
switchboard: {
program: 'test-program',
flags: {},
cliHarness: Harness.pi,
},
}),
input(),
{
Expand Down Expand Up @@ -922,11 +887,6 @@ describe('runAgent standalone', () => {
sequence,
model: DEFAULT_AGENT_MODEL,
},
switchboard: {
program: 'test-program',
flags: {},
cliHarness: Harness.pi,
},
}),
input(),
{
Expand Down
35 changes: 7 additions & 28 deletions src/agent/agent-interface.ts
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,8 @@ export type AgentConfig = {
* another program's budget, so neither should depend on an optional string bag.
*/
programId: string;
/** Program-owned guidance supplied as data, never looked up here. */
programCommandments?: readonly string[];
/** Program identifier — selects the model for that program. */
integrationLabel?: string;
/**
Expand Down Expand Up @@ -355,8 +357,8 @@ type AgentRunConfig = {
* bearer.
*/
refreshGatewayAuth?: () => Promise<GatewayAuth>;
/** Program id, for the program-axis commandments. */
program?: string;
/** Program-owned guidance supplied as data. */
programCommandments?: readonly string[];
/** Resolved sequence, for the sequence-axis commandments. */
sequence: Sequence;
/** Where the run reports. A no-op when the caller passed none. */
Expand All @@ -365,30 +367,7 @@ type AgentRunConfig = {

const NO_PROGRESS: ProgressEmitter = () => undefined;

/**
* Global identifiers attached to every LLM gateway trace for a run. They ride on
* each `$ai_generation` the gateway emits (in the `X-PostHog-Properties` blob
* `buildAgentEnv` builds), so traces are filterable by program, framework, run,
* and build type for cost attribution and dashboards. `skill_id` is omitted when
* the run has none.
*/
export function buildRunTags(args: {
programId: string;
integration: string;
runId: string;
build: string;
skillId?: string;
}): Record<string, string> {
return {
program_id: args.programId,
integration: args.integration,
run_id: args.runId,
build: args.build,
// Triage and detection spread these tags and override this one.
call_type: CallType.agent,
...(args.skillId ? { skill_id: args.skillId } : {}),
};
}
export { buildRunTags } from '@shared/run-tags';

/**
* Whether Warlock/YARA scanning is disabled for this run. Off by default:
Expand Down Expand Up @@ -669,7 +648,7 @@ export async function initializeAgent(
triageProvider,
gatewayAuth: auth,
refreshGatewayAuth: currentGatewayAuth,
program: config.integrationLabel,
programCommandments: config.programCommandments,
// A queue context is present only on a task run; that is the sequence.
sequence: config.orchestrator ? Sequence.orchestrator : Sequence.linear,
emit,
Expand Down Expand Up @@ -1169,7 +1148,7 @@ export async function runAgent(
// we keep default Claude Code behaviors. An orchestrator context is
// present only on a task run — that is what picks the sequence.
append: assembleCommandments({
program: agentConfig.program,
programCommandments: agentConfig.programCommandments,
sequence: agentConfig.sequence,
harness: Harness.anthropic,
}),
Expand Down
10 changes: 10 additions & 0 deletions src/agent/default-binding.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import { GPT5_6_SOL_MODEL, Harness, Sequence } from '@shared/constants';
import type { ResolvedBinding } from './runner/shared/types';

/** Standalone callers can supply this resolved route without a program registry. */
export const DEFAULT_AGENT_BINDING: ResolvedBinding = {
sequence: Sequence.linear,
harness: Harness.pi,
model: GPT5_6_SOL_MODEL,
thinkingLevel: 'medium',
};
Loading
Loading