Skip to content

langchain: replace direct callback-handler unit tests with tests that drive the LangChain API #820

Closed

Description

@lmolkova

Many langchain instrumentation tests call OpenTelemetryLangChainCallbackHandler.on_* methods directly with hand-built arguments, instead of running real LangChain components:

File Direct on_* calls
test_callback_handler.py 140
test_tools.py 67
test_conversation_id.py 28

Problems:

  • Bugs go unnoticed. LangChain's callback manager swallows handler exceptions and only logs a warning. Tests that assert an attribute is missing can pass even though the callback crashed. Example: test_prompt_template_omits_non_serializable_variable ([opentelemetry-instrumentation-genai-langchain] Capture prompt template name and variables on chat #730) passed only because on_llm_end raised TypeError, so the span never got its completion attributes or metrics.
  • The inputs are guesses. Hand-built serialized/inputs/output arguments don't necessarily match what LangChain sends. For example, on_tool_end gets either a ToolMessage or a raw value, depending on how the tool is called.
  • It doesn't match the other instrumentations. Other packages (openai, anthropic, llama-index, agno, ...) test through the library's public API, using VCR cassettes or fake models.

Proposal:

  • Rewrite these tests to use real LangChain primitives: fake chat models, @tool, RunnableSequence, prompt templates, and LangGraph agents. Use cassettes where a real provider is involved.
  • Make every test check that the span finished (e.g. response model, token usage), so a crashed callback can't pass silently.
  • Delete the direct-handler tests once the new tests cover the same behavior. Keep a handful only where no LangChain API reaches the code path.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions