From 44d87d4fa56dd7a0d69a2d0b76407634eb1de07d Mon Sep 17 00:00:00 2001 From: sarahxsanders Date: Wed, 9 Sep 2026 10:14:30 -0400 Subject: [PATCH] feat: allow AIO and Logs skills in default integrations --- .../__tests__/__snapshots__/commandments.test.ts.snap | 4 ++-- src/lib/agent/runner/harness/pi/README.md | 2 +- src/lib/agent/runner/harness/pi/runtime-notes.ts | 2 +- .../programs/__tests__/warehouse-suggestion.test.ts | 10 ++++++++++ src/lib/programs/posthog-integration/index.ts | 4 ++-- 5 files changed, 16 insertions(+), 6 deletions(-) diff --git a/src/lib/agent/__tests__/__snapshots__/commandments.test.ts.snap b/src/lib/agent/__tests__/__snapshots__/commandments.test.ts.snap index 9135f7aab..a1c395e11 100644 --- a/src/lib/agent/__tests__/__snapshots__/commandments.test.ts.snap +++ b/src/lib/agent/__tests__/__snapshots__/commandments.test.ts.snap @@ -93,7 +93,7 @@ Below are important guidance on the harness constraints you are bound to. Follow - If a \`bash\` command is blocked, do NOT retry it or a reworded variant — the fence is deterministic and will block it again. Change approach: inspect with \`read\`/\`grep\`, fix the \`edit\` and continue, or skip a step that is not essential. Retrying blocked commands only wastes turns. - If you get stuck on something outside your control — a package install that keeps failing, a command you are not permitted to run, or a fix outside the scope of this integration — do NOT spiral retrying it. Note it in the setup report for the user to resolve, and move on with the rest of the work. - A \`[YARA]\` block from the security scanner is on YOUR side — it caught a real problem in the edit you just tried (PII in a \`capture()\`, a hardcoded secret or host URL). Read the block reason, understand exactly what it flagged, and change the CODE to comply — e.g. a PII block means move that field off the event and onto the person via \`identify()\`/\`$set\`, keeping the event itself. Retrying the same edit will just block again, and dropping the step loses the instrumentation — so fix it to satisfy the scanner, then continue. -- Call \`load_skill_menu\` once to choose the skill, then \`install_skill\`. Do not call \`load_skill_menu\` again this session. +- Call \`load_skill_menu\` once per category needed by the workflow, then \`install_skill\` with the chosen ID. Reuse a category’s menu instead of loading it again. - Follow the skill's steps in order. Finish the SDK setup — install it, import it at the top of the module, and INITIALIZE it at the framework's entry point for every runtime the integration targets (typically both client and server) — BEFORE adding any event capture. A capture against an uninitialized SDK silently no-ops, so initialization comes first. If you're stuck and cannot install an SDK, add capture calls and add a clear note at the top of the integration report. Never guard a capture behind a runtime "if the SDK happens to be installed" check or a dynamic \`require\`; that ships an uninitialized SDK and no events fire. Do not jump ahead to the fix/revise step just to get a build passing. - Never write a PostHog URL or token as a literal in source (e.g. 'https://us.i.posthog.com') — it is blocked. Read them from environment variables (process.env.POSTHOG_HOST, os.environ['POSTHOG_HOST'], etc.). - To inspect or change a project's \`.env\` files, go straight to the wizard-tools MCP: \`check_env_keys\` to see which keys are present, \`set_env_values\` to write them. A plain \`read\`, \`edit\`, or \`write\` of any \`.env*\` file is blocked — reach for those tools first rather than discovering the block. @@ -192,7 +192,7 @@ Below are important guidance on the harness constraints you are bound to. Follow - If a \`bash\` command is blocked, do NOT retry it or a reworded variant — the fence is deterministic and will block it again. Change approach: inspect with \`read\`/\`grep\`, fix the \`edit\` and continue, or skip a step that is not essential. Retrying blocked commands only wastes turns. - If you get stuck on something outside your control — a package install that keeps failing, a command you are not permitted to run, or a fix outside the scope of this integration — do NOT spiral retrying it. Note it in the setup report for the user to resolve, and move on with the rest of the work. - A \`[YARA]\` block from the security scanner is on YOUR side — it caught a real problem in the edit you just tried (PII in a \`capture()\`, a hardcoded secret or host URL). Read the block reason, understand exactly what it flagged, and change the CODE to comply — e.g. a PII block means move that field off the event and onto the person via \`identify()\`/\`$set\`, keeping the event itself. Retrying the same edit will just block again, and dropping the step loses the instrumentation — so fix it to satisfy the scanner, then continue. -- Call \`load_skill_menu\` once to choose the skill, then \`install_skill\`. Do not call \`load_skill_menu\` again this session. +- Call \`load_skill_menu\` once per category needed by the workflow, then \`install_skill\` with the chosen ID. Reuse a category’s menu instead of loading it again. - Follow the skill's steps in order. Finish the SDK setup — install it, import it at the top of the module, and INITIALIZE it at the framework's entry point for every runtime the integration targets (typically both client and server) — BEFORE adding any event capture. A capture against an uninitialized SDK silently no-ops, so initialization comes first. If you're stuck and cannot install an SDK, add capture calls and add a clear note at the top of the integration report. Never guard a capture behind a runtime "if the SDK happens to be installed" check or a dynamic \`require\`; that ships an uninitialized SDK and no events fire. Do not jump ahead to the fix/revise step just to get a build passing. - Never write a PostHog URL or token as a literal in source (e.g. 'https://us.i.posthog.com') — it is blocked. Read them from environment variables (process.env.POSTHOG_HOST, os.environ['POSTHOG_HOST'], etc.). - To inspect or change a project's \`.env\` files, go straight to the wizard-tools MCP: \`check_env_keys\` to see which keys are present, \`set_env_values\` to write them. A plain \`read\`, \`edit\`, or \`write\` of any \`.env*\` file is blocked — reach for those tools first rather than discovering the block. diff --git a/src/lib/agent/runner/harness/pi/README.md b/src/lib/agent/runner/harness/pi/README.md index 0142954ef..bae0e2d17 100644 --- a/src/lib/agent/runner/harness/pi/README.md +++ b/src/lib/agent/runner/harness/pi/README.md @@ -76,7 +76,7 @@ extension: Because pi doesn't have Claude Code's built-in guidance, the wizard appends a long `PI_RUNTIME_NOTES` block to the shared commandments — batching rules, "use `ls`/`find`/`grep` not `bash ls`", "don't retry blocked commands", "call -`load_skill_menu` once", "no literal PostHog URLs in source", and the +`load_skill_menu` once per category", "no literal PostHog URLs in source", and the `posthog_exec` command grammar (`info` before `call`). These close the anti-spiral gaps that showed up in profiling before they became prompt engineering. diff --git a/src/lib/agent/runner/harness/pi/runtime-notes.ts b/src/lib/agent/runner/harness/pi/runtime-notes.ts index 0d9dabfa3..5350d72ab 100644 --- a/src/lib/agent/runner/harness/pi/runtime-notes.ts +++ b/src/lib/agent/runner/harness/pi/runtime-notes.ts @@ -69,7 +69,7 @@ const DONT_SPIRAL = '- If you get stuck on something outside your control — a package install that keeps failing, a command you are not permitted to run, or a fix outside the scope of this integration — do NOT spiral retrying it. Note it in the setup report for the user to resolve, and move on with the rest of the work.'; const SKILL_MENU = - '- Call `load_skill_menu` once to choose the skill, then `install_skill`. Do not call `load_skill_menu` again this session.'; + '- Call `load_skill_menu` once per category needed by the workflow, then `install_skill` with the chosen ID. Reuse a category’s menu instead of loading it again.'; const SKILL_STEPS = "- Follow the skill's steps in order. Finish the SDK setup — install it, import it at the top of the module, and INITIALIZE it at the framework's entry point for every runtime the integration targets (typically both client and server) — BEFORE adding any event capture. A capture against an uninitialized SDK silently no-ops, so initialization comes first. If you're stuck and cannot install an SDK, add capture calls and add a clear note at the top of the integration report. Never guard a capture behind a runtime \"if the SDK happens to be installed\" check or a dynamic `require`; that ships an uninitialized SDK and no events fire. Do not jump ahead to the fix/revise step just to get a build passing."; diff --git a/src/lib/programs/__tests__/warehouse-suggestion.test.ts b/src/lib/programs/__tests__/warehouse-suggestion.test.ts index 97257b228..945195ce2 100644 --- a/src/lib/programs/__tests__/warehouse-suggestion.test.ts +++ b/src/lib/programs/__tests__/warehouse-suggestion.test.ts @@ -139,6 +139,16 @@ const promptFor = async (sources: DetectedSource[]) => { }); }; +describe('default integration skill workflow', () => { + it('loads the framework category first and delegates observability to its workflow', async () => { + const prompt = await promptFor([]); + expect(prompt).toContain('category: "integration"'); + expect(prompt).toContain('AI Observability and Logs skills'); + expect(prompt).toContain('before verification and the setup report'); + expect(prompt).not.toContain('Do NOT pick skills from other categories'); + }); +}); + describe('report instruction', () => { it('asks the agent to note the sources in the report checklist', async () => { const prompt = await promptFor([POSTGRES]); diff --git a/src/lib/programs/posthog-integration/index.ts b/src/lib/programs/posthog-integration/index.ts index 7a90f313e..e60474ee9 100644 --- a/src/lib/programs/posthog-integration/index.ts +++ b/src/lib/programs/posthog-integration/index.ts @@ -325,12 +325,12 @@ Project context: Instructions (follow these steps IN ORDER - do not skip or reorder): -STEP 1: Call load_skill_menu (from the wizard-tools MCP server) to see available skills. +STEP 1: Call load_skill_menu (from the wizard-tools MCP server) with category: "integration" to see available framework skills. If the tool fails, emit: ${ AgentSignals.ERROR_MCP_MISSING } Could not load skill menu and halt. - Choose a skill from the \`integration\` category that matches this project's framework. Do NOT pick skills from other categories (llm-analytics, error-tracking, feature-flags, omnibus, etc.) — those are handled separately. + Choose a skill from the \`integration\` category that matches this project's framework. Start with this framework skill; load the AI Observability and Logs skills when its workflow calls for them, before verification and the setup report. Both are included by default where applicable; the skills define applicability and how to report skipped work. If no suitable integration skill is found, emit: ${ AgentSignals.ERROR_RESOURCE_MISSING } Could not find a suitable skill for this project.