test: fix intermittent test_sample e2e race (poll on sandbox+report postconditions)#312
Open
sbneto wants to merge 1 commit into
Open
test: fix intermittent test_sample e2e race (poll on sandbox+report postconditions)#312sbneto wants to merge 1 commit into
sbneto wants to merge 1 commit into
Conversation
… race test_sample (sync + async) polled the aggregated sample until the LLM report left its pre-trigger states, then asserted the sandbox tasks read COMPLETED. The report auto-triggers on any one completed dependency, and the sample's per-task requested_status fields don't update atomically, so the report could read triggered in the same response where sandbox_cape still projected NOT_TRIGGERED — an intermittent AssertionError against a live stack. Poll on the exact postconditions asserted instead: break once both sandbox deps read COMPLETED and the report is triggered. The test already drives both sandboxes to SUCCEEDED, so this only closes the projection-lag window; the committed cassettes replay unchanged.
|
Review — clean. Test-only change, correctly scoped.
No action needed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
test_sample(syncScanTestCaseV2+ asyncTestAsyncScanCase.test_async_sample) is intermittently flaky against a live stack:AssertionError: assert 'NOT_TRIGGERED' == 'COMPLETED'onresult.tasks['sandbox_cape']['requested_status'].llm_reportleft its pre-trigger states, then asserted a different task (sandbox_cape/sandbox_triage) wasCOMPLETED.requested_statusfields don't update atomically — so the report can read triggered in the same response wheresandbox_capestill projectsNOT_TRIGGERED.COMPLETEDand the report is triggered. Comment-only + loop-condition change; assertions unchanged.What changed
Both the sync and async
test_sampleswap this proxy exit condition:for one that waits on what the test actually asserts:
The test already drives both sandboxes to
SUCCEEDEDbefore polling, so this only closes the projection-lag window; it never waits on anything the old loop didn't already require.Cassette safety
No re-recording needed. At the interaction where the old loop broke, both sandboxes already project
COMPLETEDin the committed cassettes, so the new combined condition breaks at the identical interaction. Verified in replay: the twotest_samplecases pass, and the fullclient_scan_test.py+async_client_test.pyreplay is green (77 passed). The live (TESTS_VCR=off) path is the one this actually hardens.