Skip to content

[stable33] test(cypress): Improve stability of cypress test suite by fixing several issues - #63097

Open
DerDreschner wants to merge 3 commits into
stable33from
fix/fix-cypress-stability
Open

[stable33] test(cypress): Improve stability of cypress test suite by fixing several issues#63097
DerDreschner wants to merge 3 commits into
stable33from
fix/fix-cypress-stability

Conversation

@DerDreschner

@DerDreschner DerDreschner commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

While stable34 upwards is being moved to Playwright, the older branches still rely on the Cypress end-to-end test suite. As we still need to support those versions, it's important to have a more reliable CI pipeline there as well (IMHO). For myself, the long runtime for all tests isn't a problem per-se - it's the unreliable test results, which makes auto-merge impossible and needs a lot of re-runs and manual force-merges... Which slows down the general development speed a lot.

To achieve that, I've used Claude Fable 5 with xhigh to find and fix the issues that make our cypress tests so unreliable under limited CPU resources (simulated that by reducing the available container resources to --cpus="0.2"). This PR is the result of all runs.

The following issues were fixed:

  • Changed the defaultCommandTimeout from 4 to 10 seconds, as all 32 timeout failures across two analyzed CI runs hit exactly the 4000ms default, while none of the 28 timeouts already raised to ≥10s in the suite ever fired
  • Raised the timeout for the cron.php call in files-external-failed.cy.ts, as the first cron run on a fresh instance takes ~71 seconds (measured — every later run needs ~250ms) and therefore blew cypress' 60 seconds execTimeout default; it runs in a before all hook, which isn't retried, so the whole suite got skipped
  • Explicitly wait for long-running requests to benefit from the way higher request timeout instead of the defaultCommandTimeout
  • Moved the testapp to apps-writable, as the changes to the e2e-test-server broke the custom apps-cypress folder since this PR - auto-detect the folder as it was apps_writable with this PR and changed to apps-writable with another PR later
  • The checks for the Download button in the files-download.cy.ts and files_trashbin/files.cy.ts queried the exact match, which failed as it's actually Download: <filename>
  • The headline check in the admin-settings_branding.cy.ts is ambiguous, as there is a hidden headline with the same text
  • The default-app combobox check searched for the app names inside the <input role="combobox"> itself - impossible, since Vue 3's vue-select moved that role from the wrapper onto the input; it now asserts the deselect buttons instead
  • Made systemtags/admin-settings.cy.ts retry-safe and self-contained: randomized tag names, each describe block creates its own tag via occ instead of inheriting it from the previous one, and the create test reloads per attempt - previously one slow-runner hiccup left the form dirty (testIsolation: false means retries don't reset state) and cascaded into six failures
  • Hardened the file picker confirmation in FilesUtils.ts, as the confirm button is rendered disabled while the picker (re)loads its directory listing and clicking into that transition swallowed the click, so no COPY/MOVE request was sent at all; when navigating to the home root we additionally wait for the picker's own PROPFIND, as the button label switches to the root label before the picker's current directory follows (which copied into the wrong folder, deduplicated as … (1))
  • Hardened action menu opening mechanism in FilesUtils.ts and filesVersionsUtils.ts, as the previous logic was prone to fail under different circumstances (the click firing before Vue attached the handler to the button, the DOM switched to aria-expanded="true" while the menu is still opening and the file list being re-rendered when a thumbnail/preview request is being finished)
  • Versions test mechanism in filesVersionsUtils.ts is being hardened by setting explicit creation times via X-OC-MTime, as the versions could be created within the same second on slow runners, which means one of the versions will be ignored due to our internal mechanisms
  • Versions test mechanism in filesVersionsUtils.ts is being hardened by reloading the page if the new share isn't visible (that can take some time under high CPU load)
  • Fixed a wrong assertion in view_file-drop.cy.ts, as there could be multiple progress bars, some are present in the DOM but not visible (the upload picker and the file-drop view created one)
  • Expect a picture in router-query.cy.ts, as the preview generation could fail or take too much time under high CPU load and there is a fallback mechanism to get the picture itself instead which the test doesn't honor
  • Explicitly set the necessary state in admin-settings_default-app.cy.ts instead of implicitly expect that state to be there with the previous test finished (which isn't the case on re-tries or failed tests)
  • Fixed the trashbin row assertions in files_trashbin/files.cy.ts, as fileId is assigned inside a .then() while the row selectors were built synchronously and therefore queried [data-cy-files-list-row-fileid="undefined"] whenever the upload chain hadn't resolved yet - reading it inside a cy.then() defers that until the value exists
  • Made cy.mkdir idempotent by accepting the 405 that MKCOL answers for an already existing collection, as a retry re-runs the test body but not the data it created, so every attempt after the first failed during setup - and included the HTTP status in the error message, which previously hid the cause entirely.

The following issue is being fixed in this PR, but should be removed once the root cause is being fixed in other repositories:

  • This PR contains a hardened login mechanism to ensure the login actually happened and there were no silent errors (e.g., a failed login POST, or a dead cached session that still passes the package's redirect-following validation) - remove once included upstream
  • This PR contains a change to skip tests that might fail due to a known race-condition in the FilePicker module in nextcloud-dialogs which needs a fix there

Follow-up tasks

Checklist

AI (if applicable)

  • The content of this PR was partly or fully generated using AI

Since @nextcloud/e2e-test-server 0.5.0, the `apps-cypress` directory is being overwritten by the directory structure from the test server suite. This is being fixed by putting them into `apps_writable`, which is compliant with what's being used by default.

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
@DerDreschner DerDreschner added this to the Nextcloud 33.0.8 milestone Aug 9, 2026
@DerDreschner
DerDreschner requested review from a team, ChristophWurst, artonge and susnux August 9, 2026 18:41
@DerDreschner DerDreschner self-assigned this Aug 9, 2026
@DerDreschner
DerDreschner requested a review from a team as a code owner August 9, 2026 18:41
@DerDreschner DerDreschner added 3. to review Waiting for reviews tests Related to tests CI labels Aug 9, 2026
@DerDreschner
DerDreschner requested review from leftybournes, provokateurin and salmart-dev and removed request for a team August 9, 2026 18:41
@DerDreschner
DerDreschner force-pushed the fix/fix-cypress-stability branch 3 times, most recently from 6b2e19a to 7a3f3b3 Compare August 9, 2026 22:14
@DerDreschner
DerDreschner requested a review from a team as a code owner August 9, 2026 22:14
@DerDreschner
DerDreschner force-pushed the fix/fix-cypress-stability branch 3 times, most recently from 6e944c0 to 71f399c Compare August 9, 2026 22:50
@DerDreschner

This comment was marked as outdated.

@susnux susnux left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First look seems to be fine - waiting for CI.
But why so many unrelated changes? I mean all those:

- ..., () => {
+ ..., function () {

@DerDreschner

DerDreschner commented Aug 9, 2026

Copy link
Copy Markdown
Contributor Author

But why so many unrelated changes? I mean all those:

- ..., () => {
+ ..., function () {

Those are necessary when using skipOnKnownFilePickerRace(this). The arrow functions used before doesn't have their own this context. But that's necessary so the Mocha.Context for the currently running test is available and passed to the skipOnKnownFilePickerRace function. See https://mochajs.org/features/arrow-functions/ for example:

If you do not need to use Mocha's context, lambdas should work.

but it isn't recommended for exactly the use-case we have here:

Passing arrow functions (aka “lambdas”) to Mocha is discouraged. Lambdas lexically bind this and cannot access the Mocha context.

@DerDreschner
DerDreschner requested a review from susnux August 9, 2026 23:26
Some assertions target the wrong text or be ambigious. This is being fixed by this PR.

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
@DerDreschner
DerDreschner force-pushed the fix/fix-cypress-stability branch from 71f399c to e39fe2e Compare August 10, 2026 00:12
This PR fixes a bunch of flaky tests by fixing the root cause for the flakyness.

Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
@DerDreschner
DerDreschner force-pushed the fix/fix-cypress-stability branch from e39fe2e to fd326e9 Compare August 10, 2026 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3. to review Waiting for reviews AI assisted CI tests Related to tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants