Skip to content
This repository was archived by the owner on Jul 27, 2026. It is now read-only.

fix: identify PostHog users with Cloud identity - #1948

Closed
neubig wants to merge 2 commits into
mainfrom
agent/posthog-cloud-user-identity
Closed

fix: identify PostHog users with Cloud identity#1948
neubig wants to merge 2 commits into
mainfrom
agent/posthog-cloud-user-identity

Conversation

@neubig

@neubig neubig commented Jul 26, 2026

Copy link
Copy Markdown
Member

HUMAN:

  • A human has tested these changes.

AGENT:


Why

Canvas currently attaches cloud_user_id as an event property but deliberately keeps a browser/install distinct_id. That makes person-level analysis depend on expensive coalesce(cloud_user_id, distinct_id) HogQL across every insight and leaves Cloud and local Canvas activity fragmented.

PostHog recommends calling identify with a unique, stable user ID when the user logs in so pre-login anonymous activity and later identified activity resolve to the same person: https://posthog.com/docs/product-analytics/identify

Fixes OpenHands/OpenHands#15417.

Summary

  • Reconcile the named Canvas PostHog client to the resolved Cloud user ID after telemetry consent.
  • Preserve anonymous install traffic and identity continuity across intentional local-backend switches.
  • Reset on logout, account change, consent denial, privacy clear, active Cloud disconnect, or Cloud credential replacement.
  • Revoke local Automation consent for the pre-reset actor, retaining a small retry marker until the backend confirms it.
  • Re-key Cloud organization and current-user queries when connection credentials change, without placing API keys in query keys.
  • Keep Cloud event context separate so local events do not carry stale Cloud properties.

Issue Number

OpenHands/OpenHands#15417

How to Test

Run:

npx vitest run __tests__/services/telemetry.test.ts __tests__/hooks/use-sync-automation-telemetry-consent.test.ts __tests__/hooks/use-telemetry-identity.test.ts __tests__/contexts/active-backend-context.test.tsx src/api/automation-service/automation-service.api.test.ts --reporter=dot
npx eslint <changed TypeScript files>
npm run typecheck
npm run build

Observed: 78 focused tests passed; ESLint, Prettier, and TypeScript completed without errors; the production client and SPA builds completed successfully.

The focused tests exercise anonymous-to-identified transition, account switching, logout, denial/re-grant, pre-reset consent revocation, privacy clear, credential replacement, Cloud disconnect, unresolved startup, and intentional local-backend identity preservation. Cloud OAuth was not exercised manually.

Video/Screenshots

Not applicable; this changes telemetry identity behavior without a visual UI change.

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

The companion Automations change carries the same actor identity into asynchronous local automation events and rejects browser identity at the Cloud trust boundary: OpenHands/automation#265

Existing dashboard queries can be simplified after both changes are deployed and historical data ages out.


🐳 Docker images for this PR

GHCR package: https://github.com/OpenHands/agent-canvas/pkgs/container/agent-canvas

Component Value
Image ghcr.io/openhands/agent-canvas
Architectures amd64, arm64
Agent Server ghcr.io/openhands/agent-server:1.37.0-python
Automation openhands-automation==1.3.1
Commit afeb4725847995d5ea99faf851047f858af97617

Pull (multi-arch manifest)

# Multi-arch manifest — Docker automatically pulls the correct architecture
docker pull ghcr.io/openhands/agent-canvas:sha-afeb472

Run

docker run -it --rm \
  -p 8000:8000 \
  ghcr.io/openhands/agent-canvas:sha-afeb472

All tags pushed for this build

ghcr.io/openhands/agent-canvas:sha-afeb472-amd64
ghcr.io/openhands/agent-canvas:agent-posthog-cloud-user-identity-amd64
ghcr.io/openhands/agent-canvas:pr-1948-amd64
ghcr.io/openhands/agent-canvas:sha-afeb472-arm64
ghcr.io/openhands/agent-canvas:agent-posthog-cloud-user-identity-arm64
ghcr.io/openhands/agent-canvas:pr-1948-arm64
ghcr.io/openhands/agent-canvas:sha-afeb472
ghcr.io/openhands/agent-canvas:agent-posthog-cloud-user-identity
ghcr.io/openhands/agent-canvas:pr-1948

About Multi-Architecture Support

  • Each tag (e.g., sha-afeb472) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., sha-afeb472-amd64) are also available if needed

@vercel

vercel Bot commented Jul 26, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
agent-canvas Ready Ready Preview, Comment Jul 27, 2026 12:36am

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

✅ Mock-LLM E2E Tests

60/60 passed

Commit: afeb4725 · Workflow run · Test artifacts

Details
Status Test Duration
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory 6.4s
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI 27.3s
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page 6.1s
automations/mock-llm-preset-automation.spec.ts › preset automation → slash command conversation › automation card sends the correct slash command to a conversation 15.2s
automations/mock-llm-preset-automation.spec.ts › preset automation → slash command conversation › direct slash command from home page triggers skill activation 12.5s
backends/mock-llm-auth-modes.spec.ts › auth mode: fresh install with runtime-injected key › reaches the onboarding modal without pre-seeded localStorage 1.3s
backends/mock-llm-auth-modes.spec.ts › auth mode: non-public key rotation › recovers when localStorage has a stale session API key 5.3s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › shows first-run onboarding before the auth screen when no key is configured 1.4s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › rejects an incorrect key with an inline error 1.6s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › allows access after pasting the correct key 1.7s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › skips auth screen for returning user with valid stored key 709ms
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › re-prompts when the server rotates its key (stale localStorage) 1.4s
backends/mock-llm-cross-connect.spec.ts › cross-connect: frontend-only → backend-only › frontend-only connects to a separate backend-only instance 14.7s
backends/mock-llm-cross-connect.spec.ts › cross-connect: frontend-only → multiple backends › connects to two separate backends and switches between them 19.2s
backends/mock-llm-partial-stack.spec.ts › partial stack: --frontend-only › serves the frontend but returns 503 for backend routes 7.3s
backends/mock-llm-partial-stack.spec.ts › partial stack: --backend-only › serves backend APIs but returns 503 for the frontend root 12.3s
backends/mock-llm-partial-stack.spec.ts › partial stack: port conflict › fails with a clear error when the ingress port is occupied 80ms
backends/mock-llm-partial-stack.spec.ts › partial stack: port conflict › starts successfully on a free port after a conflict 6.0s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server 6.1s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API 6.0s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM 6.4s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away 5.6s
conversations/mock-llm-image-upload.spec.ts › mock-LLM image upload › attaching an image embeds it as base64 in the LLM completion call 12.5s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 1: ensure mock LLM profile is configured 6.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 2: start conversation and attach workspace metadata 11.5s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 3: git control bar shows workspace pill and git actions 25.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 4: files tab can enable diff view for attached workspace 5.8s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 5: browser tab shows empty state 6.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 6: files tab defaults to file-tree view without attached workspace 7.3s
home/mock-llm-folder-workspace.spec.ts › mock-LLM folder browser → workspace → conversation › step 1: browse to a folder, add it as a workspace, and launch a conversation with the correct working_dir 7.6s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 1: GitHub card is visible on the MCP marketplace page 5.4s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 2: clicking GitHub add control opens the install modal with correct fields 5.6s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 3: full install flow — fill PAT, submit, verify installed 12.1s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 4: installed GitHub server can be deleted 5.7s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: invalid Slack credentials are blocked with a credential-check error 5.7s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: a valid token missing only a scope still installs (missing_scope is not a credential failure) 5.9s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: an older agent server that omits tool_result still installs (compat) 5.8s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › edit: Test Connection verifies the stored credentials and surfaces a credential failure 5.7s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › edit: Test Connection reports success for valid stored credentials 5.6s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › custom (non-catalog) server: Test Connection attaches no verification probe 5.7s
onboarding/mock-llm-onboarding-happy-path.spec.ts › onboarding happy path › completes the full onboarding flow and launches a conversation 3.6s
onboarding/mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › keeps the modal open on backdrop click and Escape 1.4s
onboarding/mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › defaults the LLM setup step to OpenAI GPT-5.5 1.4s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › scopes standalone styles to the agent-server-ui shell 1.3s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › renders critic results on agent messages and finish actions 1.4s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › loads older events when scrolling up 1.7s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › selected workspace persists after navigating away and returning 2.2s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › cleared sessionStorage yields empty workspace selection 880ms
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 1: configure ACP agent via Settings → Agent UI 13.1s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 2: reload and verify ACP settings are persisted in UI 5.6s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 3: start ACP conversation and verify agent reply 6.2s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 4: resume ACP conversation from sidebar after navigating away 5.6s
settings/mock-llm-model-switch.spec.ts › mock-LLM /model slash command › step 1: configure LLM, create switch-target profile, register trajectory 7.0s
settings/mock-llm-model-switch.spec.ts › mock-LLM /model slash command › step 2: start conversation, switch profile via /model, verify switch 6.7s
settings/mock-llm-profile-management.spec.ts › active profile deletion + reconciliation › active profile is deletable and reconciliation activates another profile 7.8s
settings/mock-llm-profile-management.spec.ts › same-model profile identity › chat header shows the correct profile when two profiles share the same model 14.1s
settings/mock-llm-profile-management.spec.ts › OpenHands provider hidden base_url preservation › re-saving an OpenHands profile from Basic view preserves hidden base_url 7.3s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › project skill in workspace/.agents/skills/ triggers on matching keyword 12.6s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › user skill in ~/.openhands/skills/ triggers on matching keyword 7.5s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › deleting a user skill removes it from subsequent conversations 7.5s

Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)

@github-actions

Copy link
Copy Markdown
Contributor

✅ Mock-LLM Docker E2E Test Results

60/60 passed

Commit: afeb4725 · Workflow run · Test artifacts

Details
Status Test Duration
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 1: setup LLM profile and register automation trajectory 6.2s
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 2: create automation and dispatch run via the UI 31.4s
automations/mock-llm-automation.spec.ts › mock-LLM automation lifecycle › step 3: verify automation and run on the automations page 5.8s
automations/mock-llm-preset-automation.spec.ts › preset automation → slash command conversation › automation card sends the correct slash command to a conversation 15.1s
automations/mock-llm-preset-automation.spec.ts › preset automation → slash command conversation › direct slash command from home page triggers skill activation 12.3s
backends/mock-llm-auth-modes.spec.ts › auth mode: fresh install with runtime-injected key › reaches the onboarding modal without pre-seeded localStorage 1.2s
backends/mock-llm-auth-modes.spec.ts › auth mode: non-public key rotation › recovers when localStorage has a stale session API key 5.2s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › shows first-run onboarding before the auth screen when no key is configured 1.4s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › rejects an incorrect key with an inline error 1.6s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › allows access after pasting the correct key 1.8s
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › skips auth screen for returning user with valid stored key 693ms
backends/mock-llm-auth-modes.spec.ts › auth mode: public gate › re-prompts when the server rotates its key (stale localStorage) 1.4s
backends/mock-llm-cross-connect.spec.ts › cross-connect: frontend-only → backend-only › frontend-only connects to a separate backend-only instance 19.6s
backends/mock-llm-cross-connect.spec.ts › cross-connect: frontend-only → multiple backends › connects to two separate backends and switches between them 18.4s
backends/mock-llm-partial-stack.spec.ts › partial stack: --frontend-only › serves the frontend but returns 503 for backend routes 7.3s
backends/mock-llm-partial-stack.spec.ts › partial stack: --backend-only › serves backend APIs but returns 503 for the frontend root 12.3s
backends/mock-llm-partial-stack.spec.ts › partial stack: port conflict › fails with a clear error when the ingress port is occupied 86ms
backends/mock-llm-partial-stack.spec.ts › partial stack: port conflict › starts successfully on a free port after a conflict 6.0s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 1: create an LLM profile pointing at the mock LLM server 6.1s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 2: activate the mock-llm profile and verify settings API 6.0s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 3: run a conversation with the mock LLM 6.2s
conversations/mock-llm-conversation.spec.ts › mock-LLM agent-server conversation › step 4: resume conversation from sidebar after navigating away 5.6s
conversations/mock-llm-image-upload.spec.ts › mock-LLM image upload › attaching an image embeds it as base64 in the LLM completion call 12.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 1: ensure mock LLM profile is configured 6.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 2: start conversation and attach workspace metadata 11.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 3: git control bar shows workspace pill and git actions 25.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 4: files tab can enable diff view for attached workspace 5.7s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 5: browser tab shows empty state 6.3s
files/mock-llm-files-and-git.spec.ts › files tab, git control bar, and browser tab › step 6: files tab defaults to file-tree view without attached workspace 7.1s
home/mock-llm-folder-workspace.spec.ts › mock-LLM folder browser → workspace → conversation › step 1: browse to a folder, add it as a workspace, and launch a conversation with the correct working_dir 7.1s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 1: GitHub card is visible on the MCP marketplace page 5.4s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 2: clicking GitHub add control opens the install modal with correct fields 5.5s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 3: full install flow — fill PAT, submit, verify installed 11.7s
mcp/mock-llm-mcp-github.spec.ts › MCP GitHub server install flow › step 4: installed GitHub server can be deleted 5.7s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: invalid Slack credentials are blocked with a credential-check error 5.6s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: a valid token missing only a scope still installs (missing_scope is not a credential failure) 5.8s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › install: an older agent server that omits tool_result still installs (compat) 5.6s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › edit: Test Connection verifies the stored credentials and surfaces a credential failure 5.6s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › edit: Test Connection reports success for valid stored credentials 5.6s
mcp/mock-llm-mcp-slack-credentials.spec.ts › MCP Test Connection credential verification (Slack) › custom (non-catalog) server: Test Connection attaches no verification probe 5.6s
onboarding/mock-llm-onboarding-happy-path.spec.ts › onboarding happy path › completes the full onboarding flow and launches a conversation 3.1s
onboarding/mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › keeps the modal open on backdrop click and Escape 1.3s
onboarding/mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › defaults the LLM setup step to OpenAI GPT-5.5 1.3s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › scopes standalone styles to the agent-server-ui shell 869ms
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › renders critic results on agent messages and finish actions 1.4s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › loads older events when scrolling up 1.4s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › selected workspace persists after navigating away and returning 1.9s
regressions/mock-llm-ui-regressions.spec.ts › UI regressions › cleared sessionStorage yields empty workspace selection 798ms
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 1: configure ACP agent via Settings → Agent UI 12.8s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 2: reload and verify ACP settings are persisted in UI 5.6s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 3: start ACP conversation and verify agent reply 6.6s
settings/mock-llm-acp-agent.spec.ts › mock-LLM ACP agent conversation › step 4: resume ACP conversation from sidebar after navigating away 5.6s
settings/mock-llm-model-switch.spec.ts › mock-LLM /model slash command › step 1: configure LLM, create switch-target profile, register trajectory 6.9s
settings/mock-llm-model-switch.spec.ts › mock-LLM /model slash command › step 2: start conversation, switch profile via /model, verify switch 6.5s
settings/mock-llm-profile-management.spec.ts › active profile deletion + reconciliation › active profile is deletable and reconciliation activates another profile 7.7s
settings/mock-llm-profile-management.spec.ts › same-model profile identity › chat header shows the correct profile when two profiles share the same model 13.9s
settings/mock-llm-profile-management.spec.ts › OpenHands provider hidden base_url preservation › re-saving an OpenHands profile from Basic view preserves hidden base_url 7.3s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › project skill in workspace/.agents/skills/ triggers on matching keyword 12.4s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › user skill in ~/.openhands/skills/ triggers on matching keyword 7.4s
skills/mock-llm-skills.spec.ts › skill loading: project, user, and deletion › deleting a user skill removes it from subsequent conversations 7.4s

Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)

@neubig
neubig marked this pull request as ready for review July 27, 2026 13:01
@neubig

neubig commented Jul 27, 2026

Copy link
Copy Markdown
Member Author

This repository has moved to https://github.com/OpenHands/OpenHands. We’d appreciate it if you re-opened this pull request there. Thank you!

@neubig neubig closed this Jul 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

type: fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use Cloud user IDs as PostHog identities

1 participant