Repository navigation
fix: keep scheduled findings running and prevent storage log exhaustion - #73
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scheduled investigations could finish their model analysis after the four-minute deadline or five-minute ownership expiry and never publish a finding. Remove the investigation deadline and forced twenty-turn synthesis, renew the durable ownership claim while work continues, and cancel on shutdown or ownership loss. Resume a crashed review after its claim expires, while preserving the full interval after an attempted Slack delivery. Model requests have no client or transport response timeout; interactive Slack chat retains its existing limits
ClickHouse also exhausted its disk with unbounded diagnostic tables and a large Keeper replay window. Add one-day diagnostic retention, matching 10,000-entry log rotation and snapshot intervals, and a 1,000-entry retained replay window. Keep logs uncompressed because the pinned ClickHouse version cannot replay compressed entries after cache eviction. The container smoke test now forces cache eviction and kills the storage process before verifying recovery
Keep native finding persistence, evidence validation, daily investigation budgets and durable Slack deduplication. Update the operator documentation to explain the new behavior and spending controls
Validation: 50 agent tests and 46 Slack connector tests pass, including a fifteen-minute scheduled investigation, renewal during persistence, ownership loss, explicit request cancellation and completion after twenty-four tool turns. Both packages pass type and formatting checks. A real Responses request from Render through the configured external development gateway returned HTTP 200; the same credential returned HTTP 401 on the standard gateway
A real HTTP response delayed for six minutes completed after 360,023 ms without cancellation. An OpenAI SDK Responses request using the deployed Render environment and exact configured base URL also succeeded in 1,673 ms. The revised transport passed a separate Render job using the matching Undici fetch implementation and dispatcher
Production ClickHouse is healthy again with about 31 GiB free after clearing a snapshot-backed query-log segment and compacting Keeper logs. Existing Lens traces, findings and durable state remain available. A real production review completed in 4 minutes 12 seconds, persisted a verified native finding and delivered its Slack report. The report and evidence reply were verified in Slack
A separate Render job using the pinned ClickHouse binary reproduced the compressed-log replay crash after forced termination with a one-entry cache. The same test retained the original value with uncompressed logs. Production then completed a normal restart using the default cache limits, with healthy replicas, retained findings and delivery state, and about 30 GiB free
Review automation: no Greptile, Veria, Bugbot or coverage result is available on this PR. All five GitHub Actions checks passed on e1882d1, including Rust, agent and UI checks, and the complete AMD64 and ARM64 container qualification
Existing CI annotations concern unchanged action runtime deprecations and the upcoming Ubuntu runner-image migration. They do not report failures in the changed code