ai-observability on an openai v4 Node app ends with an unsupported or duplicate OpenAI SDK, and the outcome differs per run #1325

Problem

wizard ai-observability does not handle a Node app pinned to openai v4. The current @posthog/ai (8.13.1) declares openai ^6.48.0 as an optional peer dependency. npm still checks an optional peer when the package is present, so installing @posthog/ai next to openai ^4.77.0 fails with ERESOLVE.

Nothing tells the agent what to do next, so it improvises. Two sweep runs on the same fixture ended in two different states:

• One run installed @posthog/ai@^8.13.1 past the conflict. The lockfile shows @posthog/ai 8.13.1 next to the root openai 4.104.0, which npm only allows with --legacy-peer-deps or --force. The wrapper then extends an OpenAI SDK major it does not support. The report says so, but calls the setup done.
• The other run pinned @posthog/ai@^6.4.3. Before 8.0, @posthog/ai has openai as a hard dependency (^5.20.1 in 6.4.3). npm nests openai 5.23.2 under @posthog/ai. The wrapped client runs OpenAI SDK v5, and the app's own fallback client runs v4. The lockfile grows from 73 packages (8.13.1 run) to 162, because 6.4.3 also brings in @anthropic-ai/sdk, @google/genai, ai and langchain as hard dependencies. The report says the wrapper "installs alongside the unchanged root OpenAI v4 SDK". That is true of the manifest but not of what the wrapped calls run.

Both runs pass npm run build and report success. The skill's "Do not upgrade the vendor SDK" rule is kept in the manifest and broken at runtime in the second run.

The instructions also disagree about installing at all. The skill says to declare packages and not run the package manager. The wizard's program prompt and the pi harness tell the agent to run the install. The agent follows the wizard, hits ERESOLVE, and has no rule for it.

Evidence

Sweep program ai-observability, fixture ai-observability/openai/node-weather (wizard-workbench 04f55e0a9, apps/ai-observability/openai/node-weather/package.json:12, "openai": "^4.77.0", no lockfile).

• Release A top, 200961f (refactor(agent): WIP publish the agent through entry modules #1303): manifest ends with @posthog/ai ^8.13.1, openai ^4.77.0. Lockfile: 73 packages, node_modules/@posthog/ai 8.13.1 with peer openai ^6.48.0 (optional), node_modules/openai 4.104.0. Frames 15 and 16 show "Resolving the wrapper's peer-dependency constraint without changing the existing OpenAI SDK version."
• B2 head, d938a43 (feat(programs): B2 surface shell — runProgram types and stub signatures #1307): manifest ends with @posthog/ai ^6.4.3, openai ^4.77.0. Lockfile: 162 packages, node_modules/@posthog/ai 6.4.3 depending on openai ^5.20.1, node_modules/@posthog/ai/node_modules/openai 5.23.2, node_modules/openai 4.104.0, plus @langchain/openai with its own openai 5.12.2. src/index.ts builds the wrapped client from @posthog/ai/openai and a fallback client from openai. Frames 15 to 17 show "The current OpenAI dependency conflicts with the wrapper's required peer version, so configuration can proceed while compatibility remains unresolved." The agent added a "Resolve SDK compatibility" task and marked it completed.

The Release A run installed ai-observability-openai-node 1.54.1, and the B2 run installed 1.55.0. references/2-install.md did not change between v1.54.1 and v1.55.0, and it matches context-mill main. The wizard lines below are the same on the stack. So the install rule was the same in both runs, and this is not stack-only.

Skill, context-mill dc2898b:

• context/skills/ai-observability/references/2-install.md:7: "Declare the packages in the project manifest. Do not run the package manager."
• 2-install.md:22 and :46: do not add or upgrade the vendor SDK.
• 2-install.md:24: the pinned-major rule covers only pre-1.0 Python openai. It ends with "The same holds for any provider pinned below its current client interface", which the agent did not apply to openai v4 on Node.
• 2-install.md:44: "Do not run npm install or pip install."
• 4-verify.md:33: the import check passes in both runs, so it does not catch a mismatched SDK major.
• Compare context/agents/integration-v2/install.md:38, which names a pre-existing ERESOLVE as an environment failure with a defined fallback. The ai-observability skill has no such rule.

Wizard main, d8486dc1:

• src/lib/programs/ai-observability/index.ts:51-53: "The skill itself will install packages".
• src/agent/runner/harness/pi/tools.ts:390: detect_package_manager says to run the returned install command via bash.
• src/agent/runner/harness/pi/runtime-notes.ts:53: "Run installs synchronously and wait (e.g. npm install <pkg>)".

SDK, posthog-js f8eaea26:

• packages/ai/package.json:79: peer "openai": "^6.48.0", marked optional at :107. The optional peer arrived in 8.0.0 (#3739). Every earlier line bundles openai ^5 or ^6 as a hard dependency. No published @posthog/ai from 5.0 on supports openai v4.

Suggested fix

1. context-mill, context/skills/ai-observability/references/2-install.md: add a Node rule next to the pinned-major rule. Read the openai peer range of the current @posthog/ai. If the manifest's openai major is below it, do not install @posthog/ai with --legacy-peer-deps or --force, and do not pin a pre-8.0 @posthog/ai. Pick manual-capture instead, and say in the report that the wrapper needs openai 6 and that upgrading it is the user's call.
2. context-mill, 4-verify.md: add a check that the wrapper and the app load the same vendor SDK major (for example npm ls openai shows one major), and require the report to name the major the wrapped client runs.
3. Wizard: make the program prompt agree with the skill on who installs. Either drop "The skill itself will install packages" from src/lib/programs/ai-observability/index.ts:52 and scope the pi install guidance, or change the skill to allow one install attempt with an ERESOLVE fallback like integration-v2/install.md:38.
4. wizard-workbench: have the node-weather evaluator fail a run that ends with two openai majors in the lockfile or a @posthog/ai below 8.0.

Widening the @posthog/ai peer range to openai v4 (posthog-js packages/ai/package.json:79) is an option only if the wrapper is tested against v4. That call belongs to the posthog-js AI SDK owners.