Benchmark methodology: throughput bypasses the carrier path (stale QRY032 claim), cold-start measures warm caches, PreparedQuery/streaming/large-result/concurrency unmeasured

[DJGosnell]

Combined finding from the 2026-07-07 multi-agent deep review (benchmarks perspective, grade B+). The suite is stronger than most (five-way comparisons, _RawFallback fairness conventions, MemoryDiagnoser, CI dashboard with 500-run history) — but the two most marketing-visible benchmarks were confirmed to not measure what they claim, and coverage gaps sit exactly on the paths the project sells.

1 (high, CONFIRMED incl. git-history check): ThroughputBenchmarks measures the RawSqlAsync escape hatch, not the carrier path

• The suite's only sustained-load benchmark runs Quarry via QuarryDb.RawSqlAsync<User>(...).ToListAsync() in a 1000-iteration loop (ThroughputBenchmarks.cs:75-76). docs/articles/benchmarks.md:136-138 justifies this with "the source generator cannot analyze query chains inside loop bodies (QRY032)" — provably stale: ChainAnalyzer.CheckDisqualifiers (ChainAnalyzer.cs:2611-2633) explicitly allows chains fully inside a loop, and CarrierGenerationTests.ChainFullyInsideLoop_DoesNotTriggerQRY032 (CarrierGenerationTests.cs:2107) locks it in. Git history: doc + benchmark authored 2026-03-27 (Add DocFX documentation site and enhance benchmark suite #90); the loop allowance landed two days later (Fix QRY032 false positives and FirstOrDefault nullable return type mismatch #131); the benchmark was touched twice since without updating either.
• The comparison is also skewed: Raw uses SingleResult|SingleRow, Dapper QueryFirstOrDefaultAsync, EF FirstOrDefaultAsync, SqlKata reads first row — while Quarry buffers all rows into a List with no SingleRow behavior, charging Quarry per-iteration allocations no competitor pays.
• Work: rewrite Quarry_Throughput to use the chain API (db.Users().Where(u => u.UserId == id).ExecuteFetchFirstOrDefaultAsync() — the single-shot equivalent already exists in FilterWhereByIdBenchmarks.cs:63-81); keep a RawSqlAsync variant as a secondary row; fix the stale QRY032 claim in the doc.

2 (high, CONFIRMED): ColdStartBenchmarks measures warm state, docs cite it as cold-start evidence

• Under Job.Default (BDN warmup), EF Core's compiled model + internal service provider are cached statically per context type (the connection instance is not in the provider hash — OnModelCreating runs once per process), and Dapper's IL materializers live in the static SqlMapper cache. EfCore_ColdStart's line-73 comment ("New context each time forces model compilation") is false after the first invocation; Dapper_ColdStart constructs nothing per invocation. The benchmark measures context instantiation + warm query.
• Meanwhile docs/articles/benchmarks.md:91-96 claims it "isolates the one-time startup cost: EF Core's model compilation, Dapper's first-run reflection/IL emit", and benchmarks.md:182-189 + why-quarry.md:132-137 present cold start as an advantage "visible in the benchmarks". The advantage is real but unevidenced.
• Work: use RunStrategy.ColdStart with process-per-invocation isolation to actually measure first-query cost — or rename the class to context-instantiation overhead and soften the doc claims. The real zero-warmup advantage deserves real evidence.

3 (verifier: PARTIAL — gap real, severity reduced): SQLite-only coverage; server dialects unmeasured

• Everything runs on in-memory SQLite (BenchmarkBase.cs:19; csproj references only Microsoft.Data.Sqlite). Verifier corrections: README.md:79 explicitly discloses the methodology; the MySQL positional ? binding misalignment in DistinctOrderBy wrap (and audit other surfaces) #303 MySQL bind-order machinery is compile-time (generated binding code is byte-identical in the common case — no runtime regression to guard); most dialect differences are pre-built SQL strings; and Patch SET assembly (the genuinely per-execute path) IS benchmarked. Remaining real gap: no measured baseline for server-dialect binding/pagination paths; server benchmarks would mostly measure network/server latency, so scope accordingly (e.g. allocation-focused, localhost containers, excluded from default CI filter).
• Work (low-moderate priority): add PostgreSQL + MySQL variants of 3–4 core classes (SelectAll, FilterWhereById, InsertSingle, PatchUpdate) using the existing Testcontainers infra.

4 (medium): Unmeasured paths and scale

• Largest materialized result set is ~250 rows; nothing at 10k–1M rows where materialization differences compound (List growth, per-row reader overhead).
• ToAsyncEnumerable() and PreparedQuery (.Prepare()) are never benchmarked (grep: zero uses) — the claimed "zero overhead single-terminal elision via Unsafe.As" has no perf evidence.
• No concurrency/parallel-throughput benchmark (TFB-style): per-task contexts over pooled connections, where allocation-per-op compounds into GC pressure. The single-shared-connection design precludes it — needs its own benchmark class.
• GeneratorHarness measures only cold generator runs — no incremental (edit + re-run) benchmark (coordinates with the generator-caching issue).
• No AOT-published benchmark run despite AOT being a headline feature.

5 (medium): Methodology nits

• [IterationSetup]/[IterationCleanup] on all mutation benchmarks forces UnrollFactor=1 / one invocation per iteration — single ~30–100µs measurements subject to timer quantization; small regressions hide in the widened confidence interval. Consider restructuring (e.g. batch-of-N inside the benchmark with amortized reset).
• CI merge step drops failed benchmarks silently (benchmark.yml:93 — select(.Statistics != null)); a broken benchmark vanishes from the dashboard (already happened: AggregateSumBenchmarks Raw_Sum = NA). Fail the workflow on missing expected series. (Also referenced in the test-guardrails issue with the perf gate.)
• Committed BenchmarkDotNet.Artifacts are stale/partially broken; Quarry.DapperIlProbe is an empty husk — clean up or regenerate.

Acceptance criteria

• Throughput benchmark exercises the carrier chain path; docs QRY032 claim corrected.
• Cold-start either truly cold (process isolation) or renamed + docs softened.
• PreparedQuery, ToAsyncEnumerable, and a 100k-row materialization benchmark exist.
• A parallel-throughput benchmark with per-task contexts exists.
• Benchmark CI fails on missing series (gate itself tracked in the test-guardrails issue).