[moe-berri]

An investigation that fails during Checking currently loses its grouping and completed candidate results. Persist the exact grouping output and each successfully completed check, including empty results, then reuse them when the same evidence is investigated again

Checkpoints live in separate ClickHouse state records. Writes verify the active job, attempt, worker and lease, preserve the first published value, and recover an uncertain acknowledgment by reading the committed checkpoint. Candidate results are saved before the final activity update. Final findings remain unpublished until consolidation

Reuse includes the full set of sampled review content versions and extractions, analysis settings, previous findings and prompts. Changing concurrency preserves reusable work; changed context or trace evidence forces fresh analysis. Failed checks and results produced after evidence-read errors are not cached

The HTTP integration regression ingests synthetic OTLP traces, creates an investigation, injects a late provider 409 after one candidate completes, restarts the service, and submits another run. Base e320cab repeats grouping and completed checks. The fix reuses grouping and the completed candidate, makes only the outstanding check call, and retains the completed finding. Cases also cover successful empty results and invalidation

cargo test --locked --manifest-path src/worker/Cargo.toml -p litellm-lens --test local interrupted_investigation -- --test-threads=1
cargo test --locked --manifest-path src/worker/Cargo.toml -p litellm-storage-clickhouse --test investigations checkpoints -- --test-threads=2

Storage tests cover database restart, stale workers, changed attempts, expired leases, cancellation, investigation isolation, first-publication preservation and recovery of an uncertain checkpoint acknowledgment. Verification uses disposable test credentials, a controlled OpenAI-compatible provider and real ClickHouse/Keeper; it does not claim live-model quality or latency improvements

In-flight candidate work may still need to run again

Formatting and workspace Clippy pass. All five HTTP recovery/invalidation cases and seven checkpoint storage cases pass. The full workspace run encountered Docker startup timeouts in two existing credential fixtures; all 17 tests in their target passed on a serial rerun

The bundled agent dependencies now use patched zod, body-parser and undici releases, including the required qs update. Both agent suites pass (96 tests), type/style checks pass, and the production agent bundle has zero Grype findings with the October 26 vulnerability database

The existing live-writer test now verifies the lease through the active repository connection. Its separate fixture connection was idle for the full 30-second retry window, matching ClickHouse’s HTTP keep-alive timeout. The unchanged base reproduced the same post-timeout Transport error in one of six repeated runs. The corrected test passed all six equivalent repeats. All 150 investigation tests pass in 121.02 seconds, compared with 125.57 seconds before the one-line test change. Formatting and workspace Clippy pass. The assertion still checks the exact lease and unchanged investigation, and the deadline assertions are retained

Fixes #75

[moe-berri]

bugbot run