Repository navigation
test(worker): update the receiver and ClickHouse reader tests the request limits change left red - #80
Merged
moe-berri merged 2 commits intoOct 30, 2026
Conversation
The request limits change made the receiver answer 413 for a 16 MiB body of spaces, which the decode budget now rejects, so the stale 400 half of malformed_uploads_never_reach_storage goes. oversized_uploads_are_rejected_before_storage already covers the 32 MiB byte limit on both upload routes. It also pinned the scoped query reader to max_result_rows = 50000, so a typed fetch through that reader, which sends max_result_rows=0, now gets a ClickHouse 500. Production typed reads use the configured reader and the scoped reader only serves ad hoc SQL and query help, so the typed-query tests in queries.rs and the costs fence test read through the configured reader too.
Collaborator
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 36ce51d. Configure here.
Collaborator
Author
|
@moe-berri the rust job is green here, and the container scan red is on main too. Could you merge this when you get a chance? |
moe-berri
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
rustjob on main has been red since #78 added request limits. With--no-fail-fastfrom #79, main's run shows 8 failing tests in three test binaries, and each one still expects the behavior from before #78. This PR only changes testsmalformed_uploads_never_reach_storagesends a 16 MiB body of spaces and expects 400. With no content type the receiver decodes it as protobuf, and its roughly 8 million fields go past the 1 million node budget, so the receiver now answers 413. The other 7 tests run typed queries through the scoped query reader, which #78 pinned tomax_result_rows = 50000as aCONSTsetting. Typed reads sendmax_result_rows=0, so ClickHouse refuses them with code 164 and the tests get a 500. In the app only ad hoc SQL and query help use the scoped reader, and they send the same limits it pins, so no route is brokenOriginal
TLDR
The receiver test drops its 16 MiB upload case, and the typed query and cost lookup tests read through the configured reader that the app uses for typed reads, so the
rustjob can pass on main againUser Flow
Before
rustjob in lens-server fails at Run tests. 1 test fails inlitellm-lens --test receiver, 2 inlitellm-traces-clickhouse --test costs, and 5 inlitellm-traces-clickhouse --test queriesAfter
rustjob passesChanges
malformed_uploads_never_reach_storageinsrc/worker/crates/lens/tests/receiver.rskeeps its malformed JSON case and drops the oversized case.oversized_uploads_are_rejected_before_storagealready checks the 32 MiB byte limit on/v1/tracesand/v1/logs, plain and gzip, and the decode budgets have their own tests intraces/tests/otlp.rsIn
src/worker/crates/traces-clickhouse/tests/queries.rs,typed_queries_read_normalized_spans_and_keep_trace_identities_separate,typed_trace_cursor_returns_the_next_fixture_trace, andcaptured_sdk_exports_round_trip_through_clickhousenow read withConnection::readerinstead of a scoped connection. Intests/costs.rs,canonical_updates_fence_stale_aliases_and_ownersdrops itsscopedparameter and reads withConnection::readertooDecisions
Why the tests change and the limits stay
I kept #78's limits and changed the tests, because the failing cases checked a path the app does not use. Every typed read in the app goes through
config.storage().reader(). The scoped reader only serves ad hoc SQL and query help, which send the same 50,000 rows and 64 MiB that the scoped user pins, so they keep working. Scope checks stay covered by theScopeCasetests inqueries.rs, which run ad hoc SQL through the scoped readerI rejected two other fixes. Setting the scoped user's limits back to 0 would let typed reads run on it, but it would drop the 50,000 row cap that ad hoc SQL relies on. Letting each connection carry its own read limits would add an API that no route needs today
For the receiver, I dropped the oversized case instead of changing it to expect 413. A 16 MiB body is under the 32 MiB byte limit, so its 413 comes from the protobuf decode budget.
traces/tests/otlp.rsalready covers that budget, and the receiver test does not need to pin it a second timeTests
No new tests. The 8 failing tests now pass, and the byte limit and decode budget tests that cover what the dropped case checked still pass
Local and CI runs
On Rust 1.99.0 with the ClickHouse 26.9.6.6 state fixture from the workflow,
cargo fmt --check,cargo clippy --locked --workspace --all-targets -- -D warnings, andcargo test --locked --workspace --no-fail-fast -- --test-threads=2all passed at5377647. After merging main at36ce51d, fmt, clippy, and the three changed test binaries passed againScreenshots / Proof of Fix
The
rustjob passed at36ce51dwith the whole workspace suite and--no-fail-fast. The twostandalone-containerreds fail at the same image scan step as main atad31c17, where the findings are Node 24.20.0 CVEs, and this PR does not touch the image