feat(passthrough): multi-provider (OpenAI + Gemini) capture to request_logs - #796
Open
sjawhar wants to merge 7 commits into
Open
Conversation
Member
|
Claude-generated merge-queue triage of all open Luthien PRs, requested by Scott (Jul 7, 2026). Advisory only; Scott has not yet acted on these recommendations. Recommendation: request changes (one substantive item), then merge. The item: the new Worth checking in the same pass, carried over from the review trail on #758 (the earlier passthrough attempt this PR is positioned to replace): Gemini Everything else looks right: self-contained, no migrations, credentials redacted at rest via |
This was referenced Jul 7, 2026
…ABLED and bound stream capture Addresses LuthienResearch#796 review: the passthrough routes forwarded client-supplied upstream credentials with no auth, so an always-on deployment acted as an open relay that anyone with network reach could use to write request_logs. The routes now mount only when PASSTHROUGH_ROUTES_ENABLED=true (else 404), and streamed-response capture is bounded by PASSTHROUGH_STREAM_CAPTURE_MAX_BYTES (default 10 MiB) so a large stream can't exhaust memory.
legion-implementer
Bot
force-pushed
the
feat/passthrough-multiprovider-capture
branch
from
July 11, 2026 23:33
210dec9 to
1eb8f18
Compare
…ation history + retrieval Normalize captured OpenAI (chat + Responses) and Gemini (generateContent + streamGenerateContent) passthrough payloads into canonical Anthropic-shaped conversation events while preserving exact provider-native request/response for faithful reprobe. Idempotent (advisory lock + existence guard, single txn, two summary updates); live post-commit RequestLogRecorder callback gated by PASSTHROUGH_MATERIALIZE_ENABLED plus a dashboard-only reconcile worker + one-shot backfill CLI gated by PASSTHROUGH_MATERIALIZE_BACKFILL_ENABLED. Passthrough user attribution mirrors the Anthropic trusted-header/JWT policy. Existing history/ debug/summary/FTS/export read paths light up with zero new migrations. fix(passthrough): SDK-typed normalizers + Gemini contents[].role defaulting + OpenAI reasoning tokens + content_filter mapping Four fixes to the passthrough materialization pipeline, all discovered while backfilling ~2600 historical prod rows against the initial hand-rolled normalizers. 1. Provider-SDK-based normalizers (openai + google-genai typed models): Replaces hand-rolled OpenAI/Gemini payload parsing with SDK validation via openai.types + google.genai.types. Lenient by design - skip unknown output/ input items, parts, tools; never fatal on a novel variant. Wraps OpenAI response parsers in try/except ValidationError -> typed MALFORMED_PAYLOAD so novel SDK-rejected values skip one transaction instead of wedging the backfill batch. Fixes reasoning-model support (gpt-5.6-sol) and Gemini serviceTier extra_forbidden. 2. Default missing Gemini contents[].role to 'user' per API spec: _content_message previously dropped role-less contents, then _request_messages reported the misleading MISSING_REQUIRED_FIELD:contents error. Fixed with 'role = optional_string(content, "role") or "user"'. Ref: https://ai.google.dev/api/generate-content#Content 3. Gemini stream ValidationError fallback (from oracle+fable code review): gemini_stream.py._add_chunk on SDK ValidationError keeps the raw chunk so downstream lenient mapping proceeds. Mirrors gemini_response.py's buffered raw fallback. Prevents streamed captures failing MALFORMED_PAYLOAD on the same usageMetadata.serviceTier field that buffered handles. 4. Delete validate_*_request no-op helpers + dead code (oracle+fable): validate_openai_chat_request / validate_openai_responses_request / validate_gemini_request all caught ValidationError and returned None - behavioral no-ops with ~90 lines of pydantic parse overhead and zero effect. Also deleted openai_common.object_field + strict text_content_from_openai (zero callers repo-wide). 5. OpenAI reasoning_tokens in canonical usage: canonical_usage now lifts reasoning_tokens from completion_tokens_details.reasoning_tokens (Chat) or output_tokens_details.reasoning_tokens (Responses) into the canonical usage dict, matching what gemini_common already does for thoughtsTokenCount. Non-reasoning models omit the field entirely at the API level; the SDK parser injects 0 as a default, so we treat 0 as absent to avoid polluting non-reasoning outputs with a spurious 0. 6. OpenAI content_filter -> canonical safety (not end_turn): stop_reason() previously collapsed content_filter into end_turn, making safety-policy blocks indistinguishable from natural stops. Now maps to 'safety' (matching the Gemini normalizer's SAFETY bucket) so downstream consumers can filter blocked completions. Verification: 84/84 passthrough_materialize unit tests pass (was 78/78 before + review regressions + reasoning_tokens + content_filter tests). ruff format + check clean. basedpyright clean. Real prod-data validation: chat 59/60, Responses 60/60, gpt-5.6-sol reasoning 60/60, Gemini 60/60. Prod backfill drain: 2618/2623 eligible materialized (99.8%).
legion-implementer
Bot
force-pushed
the
feat/passthrough-multiprovider-capture
branch
from
July 12, 2026 17:57
1eb8f18 to
c0fd9c7
Compare
…ization fragment
The materialize changelog fragment referenced an internal fork-only tool
name ('cybertasks') as a parenthetical example of downstream tooling that
could not read passthrough OpenAI/Gemini calls before this change. That
detail is meaningless to upstream readers and leaks an internal identifier
into public changelog prose. Drop the parenthetical; the sentence carries
the same meaning without it.
red: grep -n -i cybertasks changelog.d/passthrough-materialization.md (at c0fd9c7)
-> 9: unreadable by downstream tooling (cybertasks).
green: grep -n -i cybertasks changelog.d/passthrough-materialization.md (this commit)
-> no match
Omp-Session: 01a081d4-2514-7000-b8e2-ff8548776146
Running the repo's declared gate (scripts/dev_checks.sh, which runs 'ruff format' as its Phase 1 fix step) found this test file out of sync with the project's formatting rules: missing blank lines between top-level test functions, and one call left unwrapped past the configured line length. Applying the formatter's own output; no behavior change. red: uv run ruff format --check test_gemini_normalizers.py (pre-fix copy, at c0fd9c7) -> Would reformat: ...; 1 file would be reformatted green: uv run ruff format --check test_gemini_normalizers.py (this commit) -> 1 file already formatted Omp-Session: 01a081d4-2514-7000-b8e2-ff8548776146
…emaining budget The capture loop checked `captured_bytes < max_capture` and then unconditionally appended the whole chunk before re-checking the total. A chunk that started under the cap but was itself larger than the remaining budget (e.g. one big final SSE chunk) got fully retained and persisted, letting the actually-captured/stored bytes exceed PASSTHROUGH_STREAM_CAPTURE_MAX_BYTES by an unbounded amount — defeating the whole point of the bound scottwofford asked for in the Jul 7 review comment (also reachable via a271fe0's own capture_truncated flag, which set the *flag* correctly while the underlying byte count still overran). Now slices an oversized chunk down to exactly the remaining budget before appending, so captured_bytes never exceeds max_capture. Added a regression test (test_streaming_passthrough_caps_captured_bytes_when_chunk_exceeds_remaining_budget) that streams a 5-byte chunk then a 100-byte chunk through a 10-byte cap and asserts the captured total stays <= 10. red: uv run -m pytest tests/luthien_proxy/unit_tests/test_passthrough_routes.py::test_streaming_passthrough_caps_captured_bytes_when_chunk_exceeds_remaining_budget --no-cov -> AssertionError: assert 105 <= 10 (against the pre-fix loop) green: same command (against this commit) -> 1 passed Also: ruff format/check clean, pyright 0 errors/0 warnings/0 informations on both changed files; full test_passthrough_routes.py file: 11 passed. Omp-Session: 01a081d4-2514-7000-b8e2-ff8548776146
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds passthrough capture for OpenAI (
/openai/*) and Gemini (/gemini/*) that proxies to the upstream provider while recording the full request/response (streaming included) intorequest_logs— mirroring the existing Anthropic/v1/*capture. Sessions group via the same header/metadata contract already used for Anthropic.Why
Enables full-session transcript capture for red-team / eval traffic on non-Anthropic providers. Today only Anthropic traffic is captured; OpenAI- and Gemini-based harnesses bypass the transcript lake.
Changes
src/luthien_proxy/passthrough_routes.py—/openai/*+/gemini/*passthrough routes (FastAPIAPIRouter), streaming + non-streaming.src/luthien_proxy/passthrough_capture.py— capture helper that records request + response intorequest_logs.src/luthien_proxy/request_log/recorder.py,sanitize.py— record + redact multi-provider payloads.src/luthien_proxy/main.py,dependencies.py— wire the router; addfastapidep.Response to review feedback
Addressed the items from scottwofford's Jul 7 merge-queue triage comment:
PASSTHROUGH_ROUTES_ENABLED(default off) —a271fe09passthrough_stream_capture_max_bytes+capture_truncated) —a271fe09, with a follow-up fix (817e1ac5) for an edge case where a single chunk larger than the remaining budget could still push the persisted bytes past the cap?key=query param is redacted in stored logs (sanitize_url) —c0fd9c7a_HOP_BY_HOP_HEADERS) —c0fd9c7aTesting
0 errors, 0 warnings, 0 informations).scripts/dev_checks.sh's pytest phase showed 3 failures in one local run; all three traced to a stray local.env(GATEWAY_PORT/POSTGRES_PORT/REDIS_PORT) picked up bytests/luthien_proxy/e2e_tests/conftest.py's collection-timeload_dotenvfallback, unrelated to this diff — the full suite is green on a checkout of this same code with no local.envpresent.migration-check.ymlhas run against this PR's head; it fails on an unrelated pre-existing workflow-file issue and doesn't exercise this change (the PR adds no migrations).build-and-push/dev-checks/parityhave not run yet.request_logs.Related
Distinct from (and complementary to) the earlier
track-apassthrough drafts (#757 foundation, #758 routes, #759 plugin): those add proxy routes; this focuses on capture torequest_logswith session grouping across providers.