[moe-berri]

Concurrent investigation tasks can fail with HTTP 409 after ClickHouse commits a progress update but its acknowledgment arrives after the writer lease expires. Retry uncertain progress writes only when they contain no review, using the existing retry window and reacquiring ownership before each attempt

Implements the narrow no-review retry proposed by the reporter in #74. Review-bearing writes and arbitrary mutations retain their existing no-replay behavior, so an uncertain write cannot append the same review or charge usage twice

Validation uses real ClickHouse 26.9.6.6 and Keeper with a forwarding proxy that delays the acknowledgment after publication. On base e320cab, four concurrent worker progress calls reproduce the reported 409. On this branch all four succeed and retain their activities. Storage tests also cover one stage transition, no duplicate reviews, and persistent delays exhausting the retry window with WriteUnconfirmed

Reproduction and regression checks:

cargo test --locked --manifest-path src/worker/Cargo.toml -p litellm-lens --lib concurrent_worker_progress -- --test-threads=1
cargo test --locked --manifest-path src/worker/Cargo.toml -p litellm-storage-clickhouse --test investigations unconfirmed_progress_replays -- --test-threads=2

This fixes transient progress-write uncertainty. It does not replay model-budget mutations or guarantee completion through a prolonged storage outage. Preserving finished candidate results is addressed separately for #75

Formatting and workspace Clippy pass. The full workspace test run passed all targets except a Docker startup timeout in the existing trace-query fixture; all 21 tests in that target passed on a serial rerun. An earlier unrelated OAuth fixture port-discovery failure also passed on rerun. The new regression tests passed in the workspace run

The bundled agent dependencies now use patched zod, body-parser and undici releases, including the required qs update. Both agent suites pass (96 tests), type/style checks pass, and the production agent bundle has zero Grype findings with the October 26 vulnerability database

The existing live-writer test now verifies the lease through the active repository connection. Its separate fixture connection was idle for the full 30-second retry window, matching ClickHouse’s HTTP keep-alive timeout. The unchanged base reproduced the same post-timeout Transport error in one of six repeated runs. The corrected test passed all six equivalent repeats, and all 146 investigation tests pass on this branch. Formatting and workspace Clippy pass. The assertion still checks the exact lease and unchanged investigation, and the deadline assertions are retained

Fixes #74

[moe-berri]

bugbot run