You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
langchain: replace direct callback-handler unit tests with tests that drive the LangChain API #820
Many langchain instrumentation tests call OpenTelemetryLangChainCallbackHandler.on_* methods directly with hand-built arguments, instead of running real LangChain components:
File
Direct on_* calls
test_callback_handler.py
140
test_tools.py
67
test_conversation_id.py
28
Problems:
Bugs go unnoticed. LangChain's callback manager swallows handler exceptions and only logs a warning. Tests that assert an attribute is missing can pass even though the callback crashed. Example: test_prompt_template_omits_non_serializable_variable ([opentelemetry-instrumentation-genai-langchain] Capture prompt template name and variables on chat #730) passed only because on_llm_end raised TypeError, so the span never got its completion attributes or metrics.
The inputs are guesses. Hand-built serialized/inputs/output arguments don't necessarily match what LangChain sends. For example, on_tool_end gets either a ToolMessage or a raw value, depending on how the tool is called.
It doesn't match the other instrumentations. Other packages (openai, anthropic, llama-index, agno, ...) test through the library's public API, using VCR cassettes or fake models.
Proposal:
Rewrite these tests to use real LangChain primitives: fake chat models, @tool, RunnableSequence, prompt templates, and LangGraph agents. Use cassettes where a real provider is involved.
Make every test check that the span finished (e.g. response model, token usage), so a crashed callback can't pass silently.
Delete the direct-handler tests once the new tests cover the same behavior. Keep a handful only where no LangChain API reaches the code path.
Activity