feat(langchain): capture document relevance score on retrieval spans - #670
feat(langchain): capture document relevance score on retrieval spans#670dlowzzxx wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The implementation and tests currently don’t cover the documented metadata["relevance_score"] fallback, so behavior diverges from the PR description and can drop valid scores.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR enhances the LangChain GenAI instrumentation to include per-document relevance scores in retrieval span content, aligning emitted gen_ai.retrieval.documents data more closely with the GenAI semantic conventions.
Changes:
- Add polymorphic score extraction and include
scorein retrieval document serialization for spans when content capture is enabled. - Add/extend unit and integration test coverage to validate score extraction behavior and JSON safety (non-finite floats).
- Add a changelog fragment documenting the new behavior.
File summaries
| File | Description |
|---|---|
| instrumentation/opentelemetry-instrumentation-genai-langchain/src/opentelemetry/instrumentation/genai/langchain/callback_handler.py | Adds _extract_document_score / _document_to_dict and uses them to populate retrieval documents (including score) on retrieval spans. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/test_callback_handler.py | Adds unit tests around score extraction and document-to-dict conversion. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/tests/test_retriever.py | Adds end-to-end/integration assertions that retrieval spans include score where applicable and exclude invalid/non-finite scores. |
| instrumentation/opentelemetry-instrumentation-genai-langchain/.changelog/670.added | Changelog entry for capturing document relevance scores on retrieval spans. |
Review details
- Files reviewed: 4/4 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| Checks doc.score first, then falls back to doc.metadata['score']. | ||
| Also defensively supports Mapping/dict documents and duck-typed objects. | ||
| """ | ||
| score: Any = None | ||
| if isinstance(doc, Mapping): | ||
| doc_map = cast(Mapping[str, Any], doc) | ||
| score = doc_map.get("score") | ||
| if score is None: | ||
| metadata = doc_map.get("metadata") | ||
| if isinstance(metadata, Mapping): | ||
| meta_map = cast(Mapping[str, Any], metadata) | ||
| score = meta_map.get("score") | ||
| elif metadata is not None: | ||
| score = getattr(metadata, "score", None) | ||
| else: | ||
| score = getattr(doc, "score", None) | ||
| if score is None: | ||
| metadata = getattr(doc, "metadata", None) | ||
| if isinstance(metadata, Mapping): | ||
| meta_map = cast(Mapping[str, Any], metadata) | ||
| score = meta_map.get("score") | ||
| elif metadata is not None: | ||
| score = getattr(metadata, "score", None) |
| def test_metadata_score(self): | ||
| class Obj: | ||
| metadata = {"score": 0.75} | ||
|
|
||
| assert _extract_document_score(Obj()) == 0.75 |
Pull request dashboard statusWaiting on the author · refreshed 2026-09-11 06:26 UTC Wait for the required status checks to report; this pull request moves to reviewers once the results are clean. Status above doesn't look right?
|
|
Added explicit /dashboard route:reviewers |
| meta_map = cast(Mapping[str, Any], metadata) | ||
| score = meta_map.get("score") | ||
| if score is None: | ||
| score = meta_map.get("relevance_score") |
There was a problem hiding this comment.
question: Which real retrievers populate relevance_score or a top-level score attribute with a retrieval score? Bedrock supplies metadata["score"], but Cohere supplies relevance_score for reranking, which is a separate operation. The added tests supply these fields themselves, so they do not establish their retrieval semantics. Please ground the supported fields in real retrievers, add sync/async coverage through those integrations, and document when scores are available in the LangChain instrumentation’s instrumentation/opentelemetry-instrumentation-genai-langchain/README.rst.
Description
Fixes #584
Per the OpenTelemetry Semantic Conventions for Generative AI systems (
gen_ai.retrieval.documents), each retrieved document item should capture its relevance score under thescorekey when available from the retriever or vector search engine.This PR adds document relevance score extraction to
OpenTelemetryCallbackHandler._get_retrieval_documentsinopentelemetry-instrumentation-genai-langchain.Key Changes:
scoreby checking document attributes first (getattr(doc, "score", None)or dictionarydoc.get("score")), then falling back to document metadata (metadata.get("score")/metadata.get("relevance_score")).intorfloat.boolinstances (sinceisinstance(True, int)is True in Python).0and0.0scores (usingis not Noneand type checks instead of truthiness).NaN,+Inf,-Inf) viamath.isfinite.langchain_core.documents.Documentobjects and arbitrary duck-typed document objects/dictionaries withpage_contentorcontentandmetadata.scorekey alongsidecontentandidingen_ai.retrieval.documentsJSON serialization.Type of change
How has this been tested?
test_callback_handler.pycovering score extraction, attribute precedence, metadata fallbacks,0/0.0preservation, boolean exclusion,NaN/Infexclusion, invalid types, and duck-typed objects.test_retriever.pycovering synchronous and asynchronous retrievers (invoke,ainvoke,get_relevant_documents,aget_relevant_documents) with scores.instrumentation/opentelemetry-instrumentation-genai-langchain.pyright --level error(0 errors) andmypystrict type check clean.ruff check(0 errors) andruff format --checkclean across all files.Checklist