[test optimization] Fix cypress attempt to fix #7767
[test optimization] Fix cypress attempt to fix #7767juan-fernandez wants to merge 2 commits intomasterfrom
Conversation
…with code 0
When a test is marked as attempt-to-fix by test management and fails,
the spec requires exit code 0 (failures should be ignored for session
outcome). This extends the existing quarantine error suppression pattern
to also cover attempt-to-fix tests: errors are caught in Cypress.on('fail')
and not re-thrown, while the actual failure status is still reported to
Datadog via the afterEach hook.
Also removes stale TODO comments for quarantine/disabled handling that
were fixed in PR #7442.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Overall package sizeSelf size: 4.97 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.0 | 81.15 kB | 815.98 kB | | dc-polyfill | 0.1.10 | 26.73 kB | 26.73 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #7767 +/- ##
==========================================
- Coverage 80.43% 80.42% -0.02%
==========================================
Files 741 741
Lines 32173 32182 +9
==========================================
+ Hits 25879 25882 +3
- Misses 6294 6300 +6 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This comment has been minimized.
This comment has been minimized.
BenchmarksBenchmark execution time: 2026-03-13 11:19:44 Comparing candidate commit 6fb9668 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 233 metrics, 27 unstable metrics. |
… changes Adding lines to support.js shifts line numbers in Cypress's bundled output. The invocationDetails test hardcodes the expected source line, which moved from 246 to 255. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
anmarchenko
left a comment
anmarchenko
left a comment
There was a problem hiding this comment.
In spec we ignore the failure for attempt to fix only if test is quarantined or disbaled:
If test is tagged with test.test_management.is_test_disabled or test.test_management.is_quarantined tag, then the test results are ignored (reported to the test framework as skipped, passed, or quarantined, whatever is easier to implement and available in the test framework)
2 participants
What does this PR do?
Suppresses attempt-to-fix test failures in Cypress so the process exits with code 0, matching the test management spec. Also removes stale TODO comments for quarantine/disabled handling fixed in #7442.
Motivation
The spec requires that when a test is marked as "attempt to fix" and fails, the exit code should remain 0 — failures are ignored for session outcome. Quarantined tests already achieved this by catching errors in
Cypress.on('fail')and not re-throwing. Attempt-to-fix tests were not getting the same treatment, causing Cypress to exit with code 1.Additional Notes
The fix mirrors the existing quarantine pattern:
support.js: Catch errors for attempt-to-fix tests inCypress.on('fail'), store them, and reportstate: 'failed'to Datadog inafterEach(so the actual result is preserved)cypress-plugin.js: PreventafterSpecfrom overwriting the reportedfailstatus back topassfor attempt-to-fix testscypress.spec.js: Update exit code assertion to expect 0 whenisAttemptToFixis true