Skip to content

feat(coding-agent): wire pi-langfuse agent tracing into MemSWE smoke runner - #18

Draft
LaloLalo1999 wants to merge 2 commits into
mainfrom
claude/langfuse-observability-integration-f9guj4
Draft

LaloLalo1999 wants to merge 2 commits into
mainfrom
claude/langfuse-observability-integration-f9guj4

Conversation

@LaloLalo1999

@LaloLalo1999 LaloLalo1999 commented Jun 25, 2026

Copy link
Copy Markdown

Summary

Adds the pi-langfuse extension (from https://pi.dev/packages/pi-langfuse) to the MemSWE harness so that runs emit complete observability to Langfuse for every action and datum on a run.

The harness already exported its own benchmark/memory/verifier/scoring spans over OTLP (memswe-trace-scaffold.ts), but the agent session itself loaded empty extensions (createMemSwerResourceLoader), so the agent-internal layer — prompts, LLM generations, tool calls, final response, usage, cost — was never captured. This PR closes that gap by loading pi-langfuse into the faux and minimax agent sessions, giving a single Langfuse view that combines:

  • Harness spans (existing): benchmark, memory, verifier, scoring + run metrics, over OTLP.
  • Agent traces (new): root agent, per-request generation, per-tool tool observations.

Changes

  • packages/coding-agent/package.json: pinned pi-langfuse@1.5.4 as a devDependency (harness scripts run via tsx; not shipped in dist/, and excluded from the published npm-shrinkwrap.json).
  • scripts/memswe-langfuse-extension.ts (new): isLangfuseAgentTracingConfigured() gate + loadLangfuseExtensions() that loads the extension from its package manifest path (so its @langfuse/* / @opentelemetry/* types stay out of the harness typecheck and a missing optional dep degrades to an inert run).
  • scripts/memswe-smoke-runner.ts: thread observabilityEnabled into the agent sessions; load the extension when configured; record langfuse_tracing in agent-result.json; log activation per session. Also imports the already-exported SessionSpec/VerifierSpec types the runner used but never imported.
  • README.md: new Observability (Langfuse) subsection documenting both layers and setup.

Activation & safety

  • Agent tracing activates only when pi-langfuse finds credentials: the LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY env pair, or a saved ~/.pi/agent/pi-langfuse/config.json. --no-otel-trace disables it.
  • Without credentials the extension is inert (it returns without prompting in headless mode), so the deterministic faux path is unchanged and default smokes never call Langfuse.
  • Langfuse credentials remain scrubbed from verifier/setup subprocess environments (existing SECRET_ENV_NAME_PATTERN), so traces never reach agent- or author-controlled commands.

Verification

  • npm run check (biome, pinned-deps, ts-imports, shrinkwrap, tsgo --noEmit, browser-smoke): passes. The published npm-shrinkwrap.json is unchanged (devDependencies are excluded from the shrinkwrap walk).
  • Wiring probe: loadLangfuseExtensions() loads the extension with 0 errors and registers its 3 commands and all 19 observability event handlers (before_provider_request, tool_call, tool_result, agent_end, …); isLangfuseAgentTracingConfigured() returns false/true for empty/keyed env.
  • Smoke-runner unit tests covering touched logic pass. The full memswe:smoke run and 5 suite tests require the sibling ../memswe benchmark repo, which is not cloned in this environment (pre-existing ENOENT, unrelated to this change).

🤖 Generated with Claude Code


Generated by Claude Code


View with Codesmith
Need help on this PR? Tag /codesmith with what you need. Autofix is enabled.

…runner

Add the pi-langfuse extension (pinned devDependency) and load it into the
faux and minimax agent sessions so harness runs emit complete Langfuse
observability: root agent, per-request generation, and per-tool
observations covering prompt, provider requests, tool calls, final
response, usage, and cost. This complements the existing OTLP harness
spans (benchmark/memory/verifier/scoring) for a single Langfuse view of
every action and datum on a run.

Agent tracing activates only when pi-langfuse finds credentials
(LANGFUSE_PUBLIC_KEY/SECRET_KEY env pair or a saved config.json) and is
disabled by --no-otel-trace, so the deterministic faux path stays inert
without credentials. The extension is loaded by package-manifest path to
keep its @langfuse/@opentelemetry types out of the harness typecheck and
to degrade gracefully if the optional dependency is absent.

Records langfuse_tracing in agent-result.json, logs activation per
session, documents setup in the README, and imports the already-exported
SessionSpec/VerifierSpec types the runner uses.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DVPsBgxwNigbDrdgHM3xNj
@blocksorg

blocksorg Bot commented Jun 25, 2026

Copy link
Copy Markdown

Mention Blocks like a regular teammate with your question or request:

@blocks review this pull request
@blocks make the following changes ...
@blocks create an issue from what was mentioned in the following comment ...
@blocks explain the following code ...
@blocks are there any security or performance concerns?

Run @blocks /help for more information.

Workspace settings | Disable this message

…angfuse

# Conflicts:
#	packages/coding-agent/scripts/memswe-smoke-runner.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants