Skip to content

test(worker): update the receiver and ClickHouse reader tests the request limits change left red - #80

Merged
moe-berri merged 2 commits into
BerriAI:mainfrom
mateo-berri:litellm_lens_receiver_oversized_test
Oct 30, 2026
Merged

moe-berri merged 2 commits into
BerriAI:mainfrom
mateo-berri:litellm_lens_receiver_oversized_test

Conversation

@mateo-berri

@mateo-berri mateo-berri commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

The rust job on main has been red since #78 added request limits. With --no-fail-fast from #79, main's run shows 8 failing tests in three test binaries, and each one still expects the behavior from before #78. This PR only changes tests

malformed_uploads_never_reach_storage sends a 16 MiB body of spaces and expects 400. With no content type the receiver decodes it as protobuf, and its roughly 8 million fields go past the 1 million node budget, so the receiver now answers 413. The other 7 tests run typed queries through the scoped query reader, which #78 pinned to max_result_rows = 50000 as a CONST setting. Typed reads send max_result_rows=0, so ClickHouse refuses them with code 164 and the tests get a 500. In the app only ad hoc SQL and query help use the scoped reader, and they send the same limits it pins, so no route is broken

Original

TLDR

The receiver test drops its 16 MiB upload case, and the typed query and cost lookup tests read through the configured reader that the app uses for typed reads, so the rust job can pass on main again

User Flow

Before

  1. Push to Lens main or open a PR against it
  2. The rust job in lens-server fails at Run tests. 1 test fails in litellm-lens --test receiver, 2 in litellm-traces-clickhouse --test costs, and 5 in litellm-traces-clickhouse --test queries

After

  1. Push to Lens main or open a PR against it
  2. The rust job passes

Changes

malformed_uploads_never_reach_storage in src/worker/crates/lens/tests/receiver.rs keeps its malformed JSON case and drops the oversized case. oversized_uploads_are_rejected_before_storage already checks the 32 MiB byte limit on /v1/traces and /v1/logs, plain and gzip, and the decode budgets have their own tests in traces/tests/otlp.rs

In src/worker/crates/traces-clickhouse/tests/queries.rs, typed_queries_read_normalized_spans_and_keep_trace_identities_separate, typed_trace_cursor_returns_the_next_fixture_trace, and captured_sdk_exports_round_trip_through_clickhouse now read with Connection::reader instead of a scoped connection. In tests/costs.rs, canonical_updates_fence_stale_aliases_and_owners drops its scoped parameter and reads with Connection::reader too

Decisions

Why the tests change and the limits stay

I kept #78's limits and changed the tests, because the failing cases checked a path the app does not use. Every typed read in the app goes through config.storage().reader(). The scoped reader only serves ad hoc SQL and query help, which send the same 50,000 rows and 64 MiB that the scoped user pins, so they keep working. Scope checks stay covered by the ScopeCase tests in queries.rs, which run ad hoc SQL through the scoped reader

I rejected two other fixes. Setting the scoped user's limits back to 0 would let typed reads run on it, but it would drop the 50,000 row cap that ad hoc SQL relies on. Letting each connection carry its own read limits would add an API that no route needs today

For the receiver, I dropped the oversized case instead of changing it to expect 413. A 16 MiB body is under the 32 MiB byte limit, so its 413 comes from the protobuf decode budget. traces/tests/otlp.rs already covers that budget, and the receiver test does not need to pin it a second time

Tests

No new tests. The 8 failing tests now pass, and the byte limit and decode budget tests that cover what the dropped case checked still pass

Local and CI runs

On Rust 1.99.0 with the ClickHouse 26.9.6.6 state fixture from the workflow, cargo fmt --check, cargo clippy --locked --workspace --all-targets -- -D warnings, and cargo test --locked --workspace --no-fail-fast -- --test-threads=2 all passed at 5377647. After merging main at 36ce51d, fmt, clippy, and the three changed test binaries passed again

Screenshots / Proof of Fix

The rust job passed at 36ce51d with the whole workspace suite and --no-fail-fast. The two standalone-container reds fail at the same image scan step as main at ad31c17, where the findings are Node 24.20.0 CVEs, and this PR does not touch the image

The request limits change made the receiver answer 413 for a 16 MiB body of spaces, which the decode budget now
rejects, so the stale 400 half of malformed_uploads_never_reach_storage goes. oversized_uploads_are_rejected_before_storage
already covers the 32 MiB byte limit on both upload routes.

It also pinned the scoped query reader to max_result_rows = 50000, so a typed fetch through that reader, which sends
max_result_rows=0, now gets a ClickHouse 500. Production typed reads use the configured reader and the scoped reader
only serves ad hoc SQL and query help, so the typed-query tests in queries.rs and the costs fence test read through
the configured reader too.
@mateo-berri

Copy link
Copy Markdown
Collaborator Author

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 36ce51d. Configure here.

@mateo-berri

Copy link
Copy Markdown
Collaborator Author

@moe-berri the rust job is green here, and the container scan red is on main too. Could you merge this when you get a chance?

@moe-berri
moe-berri merged commit ccd5d66 into BerriAI:main Oct 30, 2026
4 of 6 checks passed
@mateo-berri
mateo-berri deleted the litellm_lens_receiver_oversized_test branch October 30, 2026 01:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants