Skip to content

fix(analytics): stabilize the wizard's own exception stack frames - #1274

Draft
posthog[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixci-upload-wizard-source-maps-to-stop-d17b0f
Draft

posthog[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixci-upload-wizard-source-maps-to-stop-d17b0f

Conversation

@posthog

@posthog posthog Bot commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Problem

  • Wizard errors that recur for hundreds of users are filed as dozens of separate issues, so nobody can tell a genuinely new failure from an old one. Over 30 days one message spans 35 issues across 367 people; five more span 20–44 each.
  • The linked report's root cause does not hold. It attributes the split to build-hashed chunk names rotating per release. The data says otherwise — a single issue stays stable across 8 releases and 8 different chunk hashes, while one release/hash spans many issues.
  • What actually moves the grouping is the frame path. posthog-node re-anchors every frame to process.cwd(), but the wizard runs from the user's project while its bundle sits in an npx/dlx/bunx cache. Issues line up one-to-one with path shape:
Issue Frame path for dist/bin.js Releases in that issue
019e8427 .npm/_npx/H/…/dist/bin.js 3
019e6539 ../.npm/_npx/H/…/dist/bin.js 2
019e661d ../../.npm/_npx/H/…/dist/bin.js 8
019e2aa2 ../../../.npm/_npx/H/…/dist/bin.js 6
019e2ab0 ../../../../.npm/_npx/H/…/dist/bin.js 8
  • Same code, same error — one issue per ../ depth, home directory, drive letter and cache hash.
  • Separately, 94% of our own frames arrive in_app: false (23434 vs 1472), because the shipped bundle always lives under node_modules.

Note

The report's proposed fix — adding posthog-cli sourcemap upload to publish.yml — is not implemented here, because it cannot work. That flow matches symbol sets by an injected chunkId, and posthog-node emits no chunk_id on frames (verified in the SDK through the latest 5.52.4: the only frame modifiers shipped are context-lines, module and relative-path). Uploading maps would produce symbol sets nothing ever matches.

Changes

  • Re-anchor the wizard's own frames to the package root in before_send, so one line of code has one identity on every machine.
- ../../.npm/_npx/e5076a3e94cfa48e/node_modules/@posthog/wizard/dist/agent-runner-DVrkOFDA.js
+ dist/agent-runner.js
  • Mark those frames in_app: true — this bundle is the application.
  • Drop the rolldown content hash. It was not the observed driver, but once paths are stable it is the only remaining per-release variable in a frame.
  • Every frame outside the package (node: internals, ink, react-reconciler) is left untouched.

Test plan

  • New src/utils/__tests__/exception-frames.test.ts: path rewriting, hash stripping, cwd-independence, non-wizard frames untouched, frameless exceptions.
  • New case in analytics.test.ts covering the before_send wiring.
  • Full suite: 2842 tests / 178 files pass. eslint and prettier clean on touched files; pnpm typecheck reports 31 errors before and after this change (all pre-existing, in untouched files).
  • Verified the package-root assumption against a real build: import.meta.url survives bundling into dist/analytics-*.js, and resolves to the package root.
Agent context
  • The report is filed against posthog/ai-gateway, a Go service with no npm publish, no tsdown and no src/utils/analytics.ts. The described code is all in this repo, so the fix landed here.
  • Considered and rejected: adding the source-map upload step as specified — refuted above, no chunk_id in the SDK.
  • Grouping is computed server-side, so the exact fingerprint function was not read. The change is justified without it: removing machine-specific variance from frames and marking our own code in_app are correct regardless, and path shape is what the data correlates with. Expect the collapse to show up in new issues; existing ones are already minted.
  • Not addressed here (separate signals in the report, no customer impact): local developer test throws reaching production error tracking via ungated enableExceptionAutocapture, and the React is not defined startup crash already covered by a draft PR.

Created with PostHog Desktop from this inbox report.

posthog-node rewrites every stack frame path to be relative to process.cwd().
The wizard runs from the user's project directory while its bundle lives in an
npx / dlx / bunx cache, so a single frame reaches error tracking under a
different path for nearly every run, and the same failure is filed as a new
issue per path shape. Frames inside a package cache are also never marked
in_app, although the bundle is the application.

before_send now re-anchors our own frames to the package root, drops the
rolldown content hash, and marks them in_app. Every other frame is untouched.

Generated-By: PostHog Desktop
Task-Id: 45957cb9-8ed1-4349-858a-286e288fe5d5
@github-actions

Copy link
Copy Markdown

🧙 Wizard CI

Run the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands:

Test all apps:

  • /wizard-ci all

Test all apps in a directory:

  • /wizard-ci ai-observability
  • /wizard-ci basic-integration
  • /wizard-ci mcp-analytics
  • /wizard-ci replay-vision
  • /wizard-ci revenue
  • /wizard-ci self-driving
  • /wizard-ci warehouse
  • /wizard-ci warehouse-seeded

Test an individual app:

  • /wizard-ci ai-observability/anthropic
  • /wizard-ci ai-observability/google-adk
  • /wizard-ci ai-observability/groq
Show more apps
  • /wizard-ci ai-observability/manual-capture
  • /wizard-ci ai-observability/openai
  • /wizard-ci ai-observability/openai-agents
  • /wizard-ci ai-observability/opentelemetry
  • /wizard-ci ai-observability/vercel-ai
  • /wizard-ci basic-integration/android
  • /wizard-ci basic-integration/angular
  • /wizard-ci basic-integration/astro
  • /wizard-ci basic-integration/django
  • /wizard-ci basic-integration/fastapi
  • /wizard-ci basic-integration/flask
  • /wizard-ci basic-integration/flutter
  • /wizard-ci basic-integration/javascript-node
  • /wizard-ci basic-integration/javascript-web
  • /wizard-ci basic-integration/laravel
  • /wizard-ci basic-integration/next-js
  • /wizard-ci basic-integration/nuxt
  • /wizard-ci basic-integration/python
  • /wizard-ci basic-integration/rails
  • /wizard-ci basic-integration/react-native
  • /wizard-ci basic-integration/react-router
  • /wizard-ci basic-integration/sveltekit
  • /wizard-ci basic-integration/swift
  • /wizard-ci basic-integration/tanstack-router
  • /wizard-ci basic-integration/tanstack-start
  • /wizard-ci basic-integration/vue
  • /wizard-ci mcp-analytics/custom-dispatcher
  • /wizard-ci mcp-analytics/typescript-sdk
  • /wizard-ci replay-vision/javascript-node
  • /wizard-ci replay-vision/next-js
  • /wizard-ci replay-vision/react-vite
  • /wizard-ci revenue/stripe
  • /wizard-ci self-driving/astro
  • /wizard-ci self-driving/fastapi
  • /wizard-ci self-driving/nuxt
  • /wizard-ci self-driving/react-router
  • /wizard-ci self-driving/sveltekit
  • /wizard-ci warehouse/monorepo-env
  • /wizard-ci warehouse/multi-source-next
  • /wizard-ci warehouse/stripe-node
  • /wizard-ci warehouse/zero-source
  • /wizard-ci warehouse-seeded/next-stripe
  • /wizard-ci warehouse-seeded/next-stripe-declined

Test against a Context Mill branch:

  • /wizard-ci all context-mill:my-branch

Add context-mill:<branch> to any command above to pin the Context Mill branch. It defaults to main.

Results will be posted here when complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants