[stable33] test(cypress): Improve stability of cypress test suite by fixing several issues - #63097
[stable33] test(cypress): Improve stability of cypress test suite by fixing several issues#63097DerDreschner wants to merge 3 commits into
Conversation
Since @nextcloud/e2e-test-server 0.5.0, the `apps-cypress` directory is being overwritten by the directory structure from the test server suite. This is being fixed by putting them into `apps_writable`, which is compliant with what's being used by default. Assisted-by: ClaudeCode:claude-fable-5 Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
6b2e19a to
7a3f3b3
Compare
6e944c0 to
71f399c
Compare
This comment was marked as outdated.
This comment was marked as outdated.
susnux
left a comment
There was a problem hiding this comment.
First look seems to be fine - waiting for CI.
But why so many unrelated changes? I mean all those:
- ..., () => {
+ ..., function () {
Those are necessary when using
but it isn't recommended for exactly the use-case we have here:
|
Some assertions target the wrong text or be ambigious. This is being fixed by this PR. Assisted-by: ClaudeCode:claude-fable-5 Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
71f399c to
e39fe2e
Compare
This PR fixes a bunch of flaky tests by fixing the root cause for the flakyness. Assisted-by: ClaudeCode:claude-fable-5 Signed-off-by: David Dreschner <david.dreschner@nextcloud.com>
e39fe2e to
fd326e9
Compare
Summary
While
stable34upwards is being moved to Playwright, the older branches still rely on the Cypress end-to-end test suite. As we still need to support those versions, it's important to have a more reliable CI pipeline there as well (IMHO). For myself, the long runtime for all tests isn't a problem per-se - it's the unreliable test results, which makes auto-merge impossible and needs a lot of re-runs and manual force-merges... Which slows down the general development speed a lot.To achieve that, I've used Claude Fable 5 with
xhighto find and fix the issues that make our cypress tests so unreliable under limited CPU resources (simulated that by reducing the available container resources to--cpus="0.2"). This PR is the result of all runs.The following issues were fixed:
defaultCommandTimeoutfrom 4 to 10 seconds, as all 32 timeout failures across two analyzed CI runs hit exactly the 4000ms default, while none of the 28 timeouts already raised to ≥10s in the suite ever firedcron.phpcall infiles-external-failed.cy.ts, as the first cron run on a fresh instance takes ~71 seconds (measured — every later run needs ~250ms) and therefore blew cypress' 60 secondsexecTimeoutdefault; it runs in abefore allhook, which isn't retried, so the whole suite got skippeddefaultCommandTimeouttestapptoapps-writable, as the changes to thee2e-test-serverbroke the customapps-cypressfolder since this PR - auto-detect the folder as it wasapps_writablewith this PR and changed toapps-writablewith another PR laterDownloadbutton in thefiles-download.cy.tsandfiles_trashbin/files.cy.tsqueried the exact match, which failed as it's actuallyDownload: <filename>admin-settings_branding.cy.tsis ambiguous, as there is a hidden headline with the same text<input role="combobox">itself - impossible, since Vue 3's vue-select moved that role from the wrapper onto the input; it now asserts the deselect buttons insteadsystemtags/admin-settings.cy.tsretry-safe and self-contained: randomized tag names, each describe block creates its own tag viaoccinstead of inheriting it from the previous one, and the create test reloads per attempt - previously one slow-runner hiccup left the form dirty (testIsolation: falsemeans retries don't reset state) and cascaded into six failuresFilesUtils.ts, as the confirm button is rendered disabled while the picker (re)loads its directory listing and clicking into that transition swallowed the click, so no COPY/MOVE request was sent at all; when navigating to the home root we additionally wait for the picker's own PROPFIND, as the button label switches to the root label before the picker's current directory follows (which copied into the wrong folder, deduplicated as… (1))FilesUtils.tsandfilesVersionsUtils.ts, as the previous logic was prone to fail under different circumstances (the click firing before Vue attached the handler to the button, the DOM switched toaria-expanded="true"while the menu is still opening and the file list being re-rendered when a thumbnail/preview request is being finished)filesVersionsUtils.tsis being hardened by setting explicit creation times viaX-OC-MTime, as the versions could be created within the same second on slow runners, which means one of the versions will be ignored due to our internal mechanismsfilesVersionsUtils.tsis being hardened by reloading the page if the new share isn't visible (that can take some time under high CPU load)view_file-drop.cy.ts, as there could be multiple progress bars, some are present in the DOM but not visible (the upload picker and the file-drop view created one)router-query.cy.ts, as the preview generation could fail or take too much time under high CPU load and there is a fallback mechanism to get the picture itself instead which the test doesn't honoradmin-settings_default-app.cy.tsinstead of implicitly expect that state to be there with the previous test finished (which isn't the case on re-tries or failed tests)files_trashbin/files.cy.ts, asfileIdis assigned inside a.then()while the row selectors were built synchronously and therefore queried[data-cy-files-list-row-fileid="undefined"]whenever the upload chain hadn't resolved yet - reading it inside acy.then()defers that until the value existscy.mkdiridempotent by accepting the405that MKCOL answers for an already existing collection, as a retry re-runs the test body but not the data it created, so every attempt after the first failed during setup - and included the HTTP status in the error message, which previously hid the cause entirely.The following issue is being fixed in this PR, but should be removed once the root cause is being fixed in other repositories:
FilePickermodule in nextcloud-dialogs which needs a fix thereFollow-up tasks
e2e-test-serveritselfnextcloud-dialogsfix is backportedChecklist
3. to review, feature component)stable32)AI (if applicable)