Summary
The released v3.0.1 scorer and the published redteamsubnet61/ada_detection:3.0.1 image contain an inverted fail-fast predicate in PayloadManager.submit_task.
For every non-human task, _human_failed starts as True and is never reset, so submit_task returns True even when both product and headless predictions are correct. For a correct human task, _is_detected is True and the code assigns _human_failed = _is_detected, so the correct human result also returns True. Conversely, an incorrect human result returns False.
service.submit_payload copies this return value into payload_manager.failed_fast. The next loop iteration therefore exits with score 0.0. With the current schedule, no correct submission can proceed past its first task.
Reproduction against the released image
Image: redteamsubnet61/ada_detection:3.0.1
Manifest digest: sha256:7e3dda3c41cc1182a2049c2fd987dd9ae1bfb1c59f8f1593a51ad3ce58319e0f
Direct calls to the released submit_task implementation produce:
correct_driver_returns_failed: True
correct_human_returns_failed: True
incorrect_human_returns_failed: False
Expected correction
Initialize _human_failed to False. In the human branch, assign it to not _is_detected (or equivalently to len(framework_names) > 0). Keep _headless_failed responsible only for driver runs.
Tests should cover all three cases above and assert that only the incorrect human case returns True. A correct and an incorrect headless driver case should also be covered.
This regression is separate from issue #5: even a perfect detector vector is forced to zero by v3.0.1 before the normal all-or-nothing gates can complete.
Summary
The released v3.0.1 scorer and the published redteamsubnet61/ada_detection:3.0.1 image contain an inverted fail-fast predicate in PayloadManager.submit_task.
For every non-human task, _human_failed starts as True and is never reset, so submit_task returns True even when both product and headless predictions are correct. For a correct human task, _is_detected is True and the code assigns _human_failed = _is_detected, so the correct human result also returns True. Conversely, an incorrect human result returns False.
service.submit_payload copies this return value into payload_manager.failed_fast. The next loop iteration therefore exits with score 0.0. With the current schedule, no correct submission can proceed past its first task.
Reproduction against the released image
Image: redteamsubnet61/ada_detection:3.0.1
Manifest digest: sha256:7e3dda3c41cc1182a2049c2fd987dd9ae1bfb1c59f8f1593a51ad3ce58319e0f
Direct calls to the released submit_task implementation produce:
Expected correction
Initialize _human_failed to False. In the human branch, assign it to not _is_detected (or equivalently to len(framework_names) > 0). Keep _headless_failed responsible only for driver runs.
Tests should cover all three cases above and assert that only the incorrect human case returns True. A correct and an incorrect headless driver case should also be covered.
This regression is separate from issue #5: even a perfect detector vector is forced to zero by v3.0.1 before the normal all-or-nothing gates can complete.