Skip to content

test: fix intermittent test_sample e2e race (poll on sandbox+report postconditions)#312

Open
sbneto wants to merge 1 commit into
developfrom
fix-test-sample-sandbox-status-race
Open

test: fix intermittent test_sample e2e race (poll on sandbox+report postconditions)#312
sbneto wants to merge 1 commit into
developfrom
fix-test-sample-sandbox-status-race

Conversation

@sbneto

@sbneto sbneto commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

TL;DR

  • test_sample (sync ScanTestCaseV2 + async TestAsyncScanCase.test_async_sample) is intermittently flaky against a live stack: AssertionError: assert 'NOT_TRIGGERED' == 'COMPLETED' on result.tasks['sandbox_cape']['requested_status'].
  • Root cause is test logic, not the SDK or server: the poll loop broke as soon as llm_report left its pre-trigger states, then asserted a different task (sandbox_cape/sandbox_triage) was COMPLETED.
  • The report auto-triggers on any one completed dependency, and the sample's per-task requested_status fields don't update atomically — so the report can read triggered in the same response where sandbox_cape still projects NOT_TRIGGERED.
  • Fix: poll on the exact postconditions asserted — break once both sandbox deps read COMPLETED and the report is triggered. Comment-only + loop-condition change; assertions unchanged.

What changed

Both the sync and async test_sample swap this proxy exit condition:

if result.tasks.get('llm_report', {}).get('requested_status') not in _PRE_TRIGGER:
    break

for one that waits on what the test actually asserts:

tasks = result.tasks or {}
sandboxes_completed = all(
    tasks.get(f'sandbox_{s}', {}).get('requested_status') == 'COMPLETED'
    for s in ('cape', 'triage')
)
llm_triggered = tasks.get('llm_report', {}).get('requested_status') not in _PRE_TRIGGER
if sandboxes_completed and llm_triggered:
    break

The test already drives both sandboxes to SUCCEEDED before polling, so this only closes the projection-lag window; it never waits on anything the old loop didn't already require.

Cassette safety

No re-recording needed. At the interaction where the old loop broke, both sandboxes already project COMPLETED in the committed cassettes, so the new combined condition breaks at the identical interaction. Verified in replay: the two test_sample cases pass, and the full client_scan_test.py + async_client_test.py replay is green (77 passed). The live (TESTS_VCR=off) path is the one this actually hardens.

… race

test_sample (sync + async) polled the aggregated sample until the LLM
report left its pre-trigger states, then asserted the sandbox tasks read
COMPLETED. The report auto-triggers on any one completed dependency, and
the sample's per-task requested_status fields don't update atomically, so
the report could read triggered in the same response where sandbox_cape
still projected NOT_TRIGGERED — an intermittent AssertionError against a
live stack.

Poll on the exact postconditions asserted instead: break once both sandbox
deps read COMPLETED and the report is triggered. The test already drives
both sandboxes to SUCCEEDED, so this only closes the projection-lag window;
the committed cassettes replay unchanged.
@claude

claude Bot commented Jul 10, 2026

Copy link
Copy Markdown

Review — clean. Test-only change, correctly scoped.

  • Correctness: The new exit condition (sandboxes_completed and llm_triggered) is a superset of the exact postconditions asserted immediately after the loop (sandbox_cape/sandbox_triage == COMPLETED, llm_report not in _PRE_TRIGGER) in both the sync and async copies. Waiting on the asserted state rather than the llm_report proxy is the right fix for the projection-lag race; on timeout the assertions still fire with a clear message. result.tasks or {} also guards the previously-unguarded .get on tasks.
  • Spec / architecture: No SDK surface touched (core.py, resources.py, session/api layers untouched), so no unasync regen and no specs/ update needed. Consistent with the e2e-first/VCR-cassette convention in specs/04-testing.md; PR notes replay stays green (77 passed) without re-recording, and the fix hardens the TESTS_VCR=off live path.
  • Downstream contract: No public surface change, so no version bump, per AGENTS.md. Correct.
  • Gitflow: Base is develop, no pyproject.toml bump. Correct.

No action needed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant