Skip to content

fix: keep scheduled findings running and prevent storage log exhaustion - #73

Merged
moe-berri merged 7 commits into
mainfrom
litellm_docs_feedback_recovery
Oct 17, 2026
Merged

moe-berri merged 7 commits into
mainfrom
litellm_docs_feedback_recovery

Conversation

@moe-berri

@moe-berri moe-berri commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Scheduled investigations could finish their model analysis after the four-minute deadline or five-minute ownership expiry and never publish a finding. Remove the investigation deadline and forced twenty-turn synthesis, renew the durable ownership claim while work continues, and cancel on shutdown or ownership loss. Resume a crashed review after its claim expires, while preserving the full interval after an attempted Slack delivery. Model requests have no client or transport response timeout; interactive Slack chat retains its existing limits

ClickHouse also exhausted its disk with unbounded diagnostic tables and a large Keeper replay window. Add one-day diagnostic retention, matching 10,000-entry log rotation and snapshot intervals, and a 1,000-entry retained replay window. Keep logs uncompressed because the pinned ClickHouse version cannot replay compressed entries after cache eviction. The container smoke test now forces cache eviction and kills the storage process before verifying recovery

Keep native finding persistence, evidence validation, daily investigation budgets and durable Slack deduplication. Update the operator documentation to explain the new behavior and spending controls

Validation: 50 agent tests and 46 Slack connector tests pass, including a fifteen-minute scheduled investigation, renewal during persistence, ownership loss, explicit request cancellation and completion after twenty-four tool turns. Both packages pass type and formatting checks. A real Responses request from Render through the configured external development gateway returned HTTP 200; the same credential returned HTTP 401 on the standard gateway

A real HTTP response delayed for six minutes completed after 360,023 ms without cancellation. An OpenAI SDK Responses request using the deployed Render environment and exact configured base URL also succeeded in 1,673 ms. The revised transport passed a separate Render job using the matching Undici fetch implementation and dispatcher

Production ClickHouse is healthy again with about 31 GiB free after clearing a snapshot-backed query-log segment and compacting Keeper logs. Existing Lens traces, findings and durable state remain available. A real production review completed in 4 minutes 12 seconds, persisted a verified native finding and delivered its Slack report. The report and evidence reply were verified in Slack

A separate Render job using the pinned ClickHouse binary reproduced the compressed-log replay crash after forced termination with a one-entry cache. The same test retained the original value with uncompressed logs. Production then completed a normal restart using the default cache limits, with healthy replicas, retained findings and delivery state, and about 30 GiB free

Review automation: no Greptile, Veria, Bugbot or coverage result is available on this PR. All five GitHub Actions checks passed on e1882d1, including Rust, agent and UI checks, and the complete AMD64 and ARM64 container qualification

Existing CI annotations concern unchanged action runtime deprecations and the upcoming Ubuntu runner-image migration. They do not report failures in the changed code

@moe-berri moe-berri changed the title fix(agent): let scheduled investigations finish without deadlines fix: keep scheduled findings running and prevent storage log exhaustion Oct 17, 2026
@moe-berri
moe-berri merged commit e320cab into main Oct 17, 2026
5 checks passed
@moe-berri
moe-berri deleted the litellm_docs_feedback_recovery branch October 17, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant