-
Notifications
You must be signed in to change notification settings - Fork 17
feat(agents): add the error-tracking orchestrator flow #393
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
dbbb8ca
644a7a7
870f697
c59bda3
8be48c8
2b067cc
64955b1
31cc726
5d77c07
e79c007
9b14ab1
6223e9a
413c0ae
12bc01b
4de201c
7a4f082
779b999
9983432
c9f89e5
010ff33
ee6f8f2
a98931c
567bd04
a146290
3927aba
1700aae
488e759
136a9dd
811a5e9
843bd1e
476c40f
e98bb0e
fa69873
0886e3b
ba65456
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| --- | ||
| type: capture-exceptions | ||
| flow: error-tracking | ||
| label: Wire up exception capture | ||
| model_pi: openai/gpt-5.6-sol | ||
| effort_pi: medium | ||
| model_sdk: claude-sonnet-5 | ||
| effort_sdk: high | ||
| skills: [integration-v2-error-tracking-step, posthog-best-practices] | ||
| allowedTools: [Read, Write, Edit, Glob, Grep] | ||
| disallowedTools: [enqueue_task] | ||
| dependsOn: [install, init] | ||
| --- | ||
|
|
||
| ## Goal | ||
|
|
||
| Make the errors the app does not catch reach PostHog, by whatever means the | ||
| SDK offers for that. Which means depends on the SDK: some autocapture | ||
| exceptions once you enable it at init, some wire into the framework's own | ||
| error handler, some give you a boundary to mount at the app entry. Follow the | ||
| docs and the reference example for this one, and set it up in one place — | ||
| never manual capture calls sprinkled across files. | ||
|
|
||
| A page often shows two setups side by side: a global initializer, and an | ||
| explicit client you hold. Follow one of them the whole way through. Taking the | ||
| wiring from one and the calls from the other reads fine and compiles nowhere — | ||
| an initializer that hands back nothing leaves you nothing to call methods on. | ||
|
|
||
| The SDK is installed and initialized — either it already was, or the install | ||
| and init tasks before you did it (see their handoffs); build on that, do not | ||
| re-check it. | ||
|
|
||
| This is an instrument-only task. Do not install dependencies, run the build, | ||
| run tests, or start the app — the user-driven test-setup step at the end of | ||
| the flow verifies, when the user wants it. Do not touch the build config | ||
| either way; when the flow includes a configure task, it owns those files. | ||
| Stay inside this project's directory and set up that one place; that is the | ||
| whole job. | ||
|
|
||
| ## Do not copy the SDK's own listeners | ||
|
|
||
| Exception autocapture already registers the global handlers: `window.onerror` | ||
| and `unhandledrejection` in the browser, `uncaughtException` and | ||
| `unhandledRejection` in Node, `sys.excepthook` in Python, the panic hook in | ||
| Rust, and the uncaught-exception and signal handlers on iOS and Android. Turn | ||
| that on with the SDK's own option — `capture_exceptions`, | ||
| `enableExceptionAutocapture`, `enable_exception_autocapture`, `capture_panics`, | ||
| `errorTrackingConfig.autoCapture` — and register none of those handlers | ||
| yourself. A second listener on the same event sends every error twice, and it | ||
| stops matching the SDK's handling the moment the SDK changes. | ||
|
|
||
| What the SDK cannot see is yours to add: errors a framework catches before any | ||
| global handler fires. Express error middleware, Fastify `setErrorHandler`, Hono | ||
| `onError`, Vue `app.config.errorHandler`, Angular `ErrorHandler`, SvelteKit | ||
| `handleError`, a React error boundary, Next.js `global-error` — hook those, | ||
| because the framework swallows the error and the global listener never hears | ||
| of it. A platform with no autocapture at all, such as Go, is the other case: | ||
| there the capture boundary at the entry point is the mechanism, not a copy of | ||
| one. | ||
|
|
||
| ## How you know you succeeded | ||
|
|
||
| An error the app does not catch reaches PostHog, through the mechanism this | ||
| SDK gives you rather than one you invented, and no global error listener of | ||
| your own sits beside the SDK's autocapture. You did not install anything, run | ||
| a build, lint, or tests, search outside the project, or read through the whole | ||
| app or hand-wrap individual components or routes. Your handoff names the files | ||
| you changed and the capture mechanism, so the report can explain it to the | ||
| user. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| --- | ||
| type: configure | ||
| flow: error-tracking | ||
| label: Apply build-config changes | ||
| model_pi: openai/gpt-5.6-sol | ||
| effort_pi: medium | ||
| model_sdk: claude-sonnet-5 | ||
| effort_sdk: medium | ||
| skills: [] | ||
|
ablaszkiewicz marked this conversation as resolved.
|
||
| allowedTools: [Read, Write, Edit, Glob, Grep, Bash, load_skill_menu, install_skill, check_env_keys] | ||
| disallowedTools: [enqueue_task] | ||
| dependsOn: [capture-exceptions] | ||
| --- | ||
|
|
||
| ## Goal | ||
|
|
||
| Make this project's production build emit and upload source maps (or, for Go | ||
| and Rust, native debug symbols). Install the skill your task input names | ||
| (`install_skill` with the `skillId`) and read it — it is the source of truth | ||
| for the per-framework build-config and the uploader wiring. | ||
|
|
||
| Two of the skill's steps are yours: | ||
|
|
||
| - **"Apply build-config changes"** — make the bundler / build-config edits the | ||
| skill instructs for this platform, so the build produces and injects the | ||
| chunk IDs PostHog needs and runs the uploader. | ||
| - **"Make credentials available at build time"** — do the skill's step so the | ||
| build can read the upload credentials from the environment. If it calls for a | ||
| loader (e.g. `dotenv`), install it SILENTLY with the project's package | ||
| manager. Skip this step entirely when the platform already auto-loads `.env`. | ||
|
|
||
| Install every dependency with the project's own package manager: call | ||
| `detect_package_manager` before the first install and use its answer, | ||
| translating any `npm install` the skill or docs show (`pnpm add -D …`, | ||
| `yarn add -D …`). In a pnpm or yarn workspace, npm fails outright on | ||
| `workspace:*` dependencies (`EUNSUPPORTEDPROTOCOL`) — that error means the | ||
| wrong manager, never a flag to retry with. | ||
|
|
||
| When the upload step you wire runs a machine-global `posthog-cli` rather than | ||
| one from the project's own dependencies, check that it is on the `PATH` | ||
| (`command -v posthog-cli`). Do not try to install it: a global install is | ||
| blocked for you, and the wizard already tried before your task. When it is | ||
| missing, name it in your handoff as a manual follow-up — run | ||
| `npm install -g @posthog/cli@latest` before the next release build — because | ||
| without it the upload step cannot run. | ||
|
|
||
| When your changes make the build emit a bundle to a new directory (`dist/`, | ||
| `build/`), check that some script actually runs that output. A project whose | ||
| `build` writes `dist/index.js` while `start` still runs the original source | ||
| never executes the bundle the maps were uploaded for, so every uploaded map | ||
| goes unused. Add or fix the script that serves the built output. | ||
|
|
||
| That run script has its own environment to satisfy, and it is not the build's. | ||
| The uploader's credentials are handled — your build step passes them itself. The | ||
| separate question is the variables **the application reads when it starts**: the | ||
| project token and host its init looks up. Those live in the same gitignored env | ||
| file, and a compiled binary or a bare interpreter loads nothing on its own. | ||
|
|
||
| So open the init point, note the variables it reads, and make the run script | ||
| provide exactly those — export the env file ahead of the command, pass the | ||
| runtime's own flag for it, whatever that platform offers. Skip it and the | ||
| artifact starts, prints its own "variable missing" guard, and reports nothing: | ||
| the build is green, the symbols are uploaded, and the single command you hand | ||
| the user to verify with is the one command that cannot capture. | ||
|
|
||
| Put a multi-step build somewhere the tool actually runs it — a script file, a | ||
| make target, the manifest's own scripts — never an alias mechanism you are | ||
| assuming exists, borrowed from a neighbouring tool. | ||
|
|
||
| Match the bundle's module format to the package's type while you write the | ||
| command — you cannot run it, so it has to be right by construction. `esbuild | ||
| --platform=node` emits CommonJS unless you pass `--format=esm`, so in a package | ||
| whose `package.json` sets `"type": "module"` the bundle dies at boot with | ||
| `ReferenceError: module is not defined in ES module scope`. Read the `type` | ||
| field before writing the command. | ||
|
|
||
| Types are the same kind of trap. In a config the compiler checks — a `.ts` | ||
| config, or JS under `checkJs` — `process.env.ANYTHING` is `string | undefined`, | ||
| while a plugin's options usually require `string`. Where your skill's example | ||
| asserts or defaults that lookup, keep that part exactly: dropping a `!` or a | ||
| `?? ''` turns a working example into a build that fails type checking. The | ||
| upload step often runs before the type check, so the maps land and the build | ||
| still exits non-zero — a broken build that looks half-successful in the log. | ||
|
|
||
| ## The names are a contract | ||
|
|
||
| An env variable only works if the name the code reads is the name in the file. | ||
| You write the code that reads them; the `credentials` task writes the file, and | ||
| you two run in parallel. So never invent a name that already exists somewhere | ||
| else — look for the other half of the contract first, and adopt it: | ||
|
|
||
| - **Call `check_env_keys` before you write the config.** It returns names, never | ||
| values. If the PostHog upload variables are already there, make your config | ||
| read exactly those names, whatever they are, even when they are not the ones | ||
| your skill's example shows. | ||
| - **Only when they are absent yet** are you the one deciding. Use the names your | ||
| skill documents for the mechanism you are wiring, and name them in your | ||
| handoff in full so `credentials` and `wire-ci` can match them. | ||
|
|
||
| Names that merely look plausible are the failure here. A build reading | ||
| `POSTHOG_API_KEY` beside an env file holding `POSTHOG_CLI_API_KEY` throws no | ||
| error anywhere — the upload is skipped silently, the build looks clean, and | ||
| every stack trace stays minified. | ||
|
|
||
| Do not write any credential values and do not create env files — the | ||
| `credentials` task owns that, in parallel with you. Do not run the build. | ||
|
|
||
| ## How you know you succeeded | ||
|
|
||
| The build config carries the skill's source-map / debug-symbol changes and can | ||
| read its credentials from the environment at build time. Your handoff names | ||
| every file you changed and the exact build-config keys you added, so the CI | ||
| task can wire the same variables through the pipeline. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,80 @@ | ||
| --- | ||
| type: credentials | ||
| flow: error-tracking | ||
| label: Get and write the upload credentials | ||
| model_pi: openai/gpt-5.6-sol | ||
| effort_pi: medium | ||
| model_sdk: claude-sonnet-5 | ||
| effort_sdk: medium | ||
| skills: [] | ||
| allowedTools: [Read, Write, Edit, Glob, Grep, Bash, load_skill_menu, install_skill, wizard_ask] | ||
| disallowedTools: [enqueue_task] | ||
| dependsOn: [] | ||
| --- | ||
|
|
||
| ## Goal | ||
|
|
||
| Put the PostHog source-map upload credentials into this project's environment. | ||
| Install the skill your task input names (`install_skill` with the `skillId`) | ||
| and follow its **"Write credentials to the env file"** step for the variable | ||
| names and the env file to pick. | ||
|
|
||
| The upload needs a PostHog **personal API key** at build time. Only the user can | ||
| mint one — never call the PostHog API or any tool to create it. Get it and | ||
| write it in this one task (the key never survives across tasks): | ||
|
|
||
| 1. Ask with `wizard_ask`, exactly: | ||
| `{ id: "api-key", prompt: "Paste your PostHog personal API key below.\n\nDon't have one yet? Create one here:\n<SETTINGS_URL>\n\nWhen creating the key, choose the 'Source map upload' preset, then come back and paste it here.", kind: "text", sensitive: true }` | ||
| You receive `{ secretRef: "secret:..." }` — a vaulted reference, never the raw | ||
| value. If `wizard_ask` is unavailable (non-interactive run), report this task | ||
| with status `not needed` and say in your handoff that the user must create | ||
| the key and set the variables themselves; do not block. | ||
| 2. Pick the env file per the skill (reuse the one PostHog's SDK already writes | ||
| its `POSTHOG_*` vars to, when there is one). Call `check_env_keys` on it | ||
| first (it returns present/absent, never values — never read the file | ||
| directly). | ||
| 3. Settle the variable names before you write them — see "The names are a | ||
| contract" below. Then call `set_env_values`, passing the secretRef as a | ||
| value object, not a literal string — e.g. | ||
| `values: { "POSTHOG_CLI_API_KEY": { secretRef: "<the ref>" }, "POSTHOG_CLI_PROJECT_ID": "<PROJECT_ID>", "POSTHOG_CLI_HOST": "<UI_HOST>" }` | ||
| with whichever names you settled on. The wizard resolves the ref locally, so | ||
| you never see the key value. | ||
| 4. Document the same variable names for other developers: append them to | ||
| `.env.example` (create it if the project has none) with empty or | ||
| placeholder values — never a real value, and never the key itself. The | ||
| example file is committed and is the only `.env*` you may write directly; | ||
| it is how the next developer, and the next wizard run's `check_env_keys`, | ||
| learns the project expects these variables. | ||
|
|
||
| Replace `<SETTINGS_URL>`, `<PROJECT_ID>`, and `<UI_HOST>` from your project | ||
| context. Do not touch the build config — the `configure` task owns that. | ||
|
|
||
| ## The names are a contract | ||
|
|
||
| An env variable only works if the name in the file is the name the code reads. | ||
| You write the file; the `configure` task writes the code that reads it, and you | ||
| two run in parallel. So never invent a name that already exists somewhere else | ||
| — look for the other half of the contract first, and adopt it: | ||
|
|
||
| - **Read the build config before you choose.** If it already references PostHog | ||
| env variables, use exactly those names, whatever they are, even when they are | ||
| not the ones your skill would suggest. Reading config is safe — it holds | ||
| names, not secrets. | ||
| - **Only when nothing references them yet** are you the one deciding. Use the | ||
| names your skill documents for the mechanism that was actually wired. | ||
| - **Either way, say which set you chose in your handoff, in full.** That is how | ||
| `configure` and `wire-ci` learn what to match. A handoff that says "the usual | ||
| variables" hands the next task the same guess you just made. | ||
|
|
||
| Names that merely look plausible are the failure here. A build reading | ||
| `POSTHOG_API_KEY` beside an env file holding `POSTHOG_CLI_API_KEY` throws no | ||
| error anywhere — the upload simply never runs, and every stack trace stays | ||
| minified. | ||
|
|
||
| ## How you know you succeeded | ||
|
|
||
| The env file holds the upload variables (the key as a resolved secret, the | ||
| non-secret project id and host as literals), written through the wizard tools, | ||
| never hardcoded in source, and `.env.example` documents the same names with | ||
| placeholders. Your handoff names the env file and every variable name — never | ||
| a value — so the CI task carries the same names into the pipeline. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,117 @@ | ||
| --- | ||
| type: init | ||
| flow: error-tracking | ||
| label: Set up PostHog initialization | ||
| model_pi: openai/gpt-5.6-terra | ||
|
ablaszkiewicz marked this conversation as resolved.
|
||
| effort_pi: medium | ||
| model_sdk: claude-sonnet-5 | ||
| effort_sdk: medium | ||
| skills: [integration-v2-init, posthog-best-practices] | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's fucking go @edwinyjlim @sarahxsanders @daniloc Our dream of orchestrated, reusable skills is coming alive! |
||
| allowedTools: [Read, Write, Edit, Glob, Grep, Bash, check_env_keys, set_env_values] | ||
| disallowedTools: [enqueue_task] | ||
| dependsOn: [] | ||
| --- | ||
|
|
||
| ## Goal | ||
|
|
||
| Make sure PostHog is initialized. If the project already has a working | ||
| `posthog.init(...)` (or the framework's equivalent) with its env keys wired, | ||
| leave it alone and say so in your handoff. If it doesn't, create it following | ||
| your skill — it owns the how: the framework's init point, the env-var wiring | ||
| through the wizard tools, and `.env.example`. | ||
|
|
||
| You exist in this flow because the user asked for error tracking on a repo | ||
| whose PostHog init is missing or unproven. Initialize the SDK so exceptions can | ||
| flow and stop — no instrumentation, no extras. Don't set up exception capture | ||
| either way; the capture-exceptions task after you owns that. | ||
|
|
||
| ## An existing init still needs its variable defined | ||
|
|
||
| "Already initialised" is a property of the pair, not of the call. An init point | ||
| that reads `process.env.SOMETHING` only works when `SOMETHING` is defined in the | ||
| env file the project loads. A repo can carry an init that has never once run: | ||
| the call is there, the variable it names is nowhere, and the client is built | ||
| from an empty string. | ||
|
|
||
| So before you leave an existing init alone, call `check_env_keys` on that env | ||
| file and look for the exact name the init reads. It returns names, never values. | ||
|
|
||
| - **Present** — the pair is complete. Leave it alone and say so. | ||
| - **Absent** — the init is not wired yet, whoever wrote it. Write that variable | ||
| with `set_env_values` under the name the code already reads, and document it | ||
| in `.env.example`. Do not rename the code to match a name you would rather | ||
| have written; the code is the half that already exists. | ||
|
|
||
| An empty key is the quiet failure here. A client constructed from `''` throws | ||
| nothing and logs nothing. The build is clean, the app starts, every capture call | ||
| returns — and no event ever arrives. | ||
|
|
||
| ## Make the environment actually reachable | ||
|
|
||
| Writing the keys to `.env` is only half the job — something has to load that | ||
| file at runtime, or the app throws on boot and captures nothing. | ||
|
|
||
| Most frameworks do it for you: Next, Nuxt, Astro and SvelteKit auto-load `.env`, | ||
| and Vite auto-loads it for client code. Nothing to do there. | ||
|
|
||
| A plain Node backend does not — Express, Fastify, Hono, Koa, a raw `node:http` | ||
| server — and neither does a bare Rollup or webpack config. When your init point | ||
| reads `process.env` on one of those, wire the loading too, either way: | ||
|
|
||
| - install `dotenv` with the project's own package manager (detect it from the | ||
| lockfile) and import it above the PostHog init — `require('dotenv').config()`, | ||
| or `import 'dotenv/config'` for ESM; or | ||
| - add `--env-file=.env` to the `start` and `dev` scripts, when the project is on | ||
| Node 20.6+ and would rather not take a new dependency. | ||
|
|
||
| You cannot run the app, so this has to be right by construction. Writing the | ||
| guard without the loader is what produces | ||
| `POSTHOG_… variable required by PostHog is missing or un-configured` at module | ||
| load: the guard you wrote firing against an env file nothing reads. | ||
|
|
||
| Some platforms have no environment to read at all, and there the answer is not | ||
| a loader. Angular on the stock `@angular/build` builder is the common one: | ||
| nothing populates `process.env`, `import.meta.env`, or the project's own | ||
| `src/environments/.env.ts` with your keys. | ||
|
|
||
| The trap is that each of those *looks* like a mechanism. `import.meta.env` and | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is great, but I'd move all the .env handling and init out to the shared skill. Also that if a
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I suggest a separate PR as it affects other flows? You mean to do it such that replay vision also uses that? |
||
| the `NG_APP_` prefix only exist with `@ngx-env/builder` installed, and a | ||
| generated `.env.ts` usually carries one unrelated key such as | ||
| `npm_package_version` and nothing else. Read from either and your key is | ||
| `undefined`: in development the guard throws while the module evaluates and the | ||
| app renders a blank page, and in a production build the guard returns quietly, | ||
| so the app looks fine while PostHog never initialises at all. | ||
|
|
||
| So do not wire a lookup unless you have opened the thing it reads from and seen | ||
| your key defined there. When nothing populates it, write the real public project | ||
| token as a literal in the committed `src/environments/*` files. This is the | ||
| skill's "no valid environment to read from" case, and the public token is | ||
| publishable — it ships inside the browser bundle either way. | ||
|
|
||
| ## The init has to type-check | ||
|
|
||
| You cannot run the build either. So in a project the compiler checks — | ||
| TypeScript, or JavaScript under `checkJs` — the init has to compile by | ||
| construction. `process.env.ANYTHING` is `string | undefined`, and the SDK | ||
| constructor wants a `string`. | ||
|
|
||
| Pass the variable that you checked. `if (!key) { … } else { new PostHog(key, …) }` | ||
| narrows `key` to `string`. A check on a different variable does not: store the | ||
| missing name in `missing`, test `missing`, then pass `key`, and the compiler | ||
| still sees `string | undefined`. The build then stops with "Argument of type | ||
| 'string | undefined' is not assignable to parameter of type 'string'", emits | ||
| no bundle, and uploads no source maps. When the existing code already passes | ||
| `process.env.KEY ?? ''` or `process.env.KEY!`, keep that part as it is. | ||
|
|
||
| ## How you know you succeeded | ||
|
|
||
| An init point exists with the PostHog env keys present — whether it already | ||
| did or you just created it — keys in the env file and confirmed there with | ||
| `check_env_keys`, never hardcoded. On a platform that does not auto-load | ||
| `.env`, the loading is wired; on a platform with no environment at all, the | ||
| token is a literal rather than a lookup into something that never defines it. | ||
| In a type-checked project, the value you pass to the SDK is narrowed to a | ||
| `string`. | ||
| Your handoff names the files involved, how the client is constructed, and how | ||
| `.env` reaches it, so the capture-exceptions task can find the init options | ||
| without re-discovering them. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will be very fragile. I think this info should be maintained in docs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So our docs already explain how our auto capture works. But then agent during implementation really tried to to its best job and sometimes started hooking into things our auto capture already hooks into.
I don't think we should write in every technology doc "do not hook into x.y, it's already covered by auto capture". List here is not strict and doesn't really need to be updated. It's more of a general guidance to an agent so it can distinguish auto capture vs userful framework hooks