Vscode e2e test very flaky

[timotheeguerin]

Started yersteday, the e2e test have been failing quite regularly in the CI

packages/typespec-vscode test:e2e:   run `npm fund` for details
packages/typespec-vscode test:e2e: found 0 vulnerabilities
packages/typespec-vscode test:e2e:  ✓ preview-typespec.test.ts (1 test) 57932ms
packages/typespec-vscode test:e2e:      ✓ PreviewTypespecProject Trigger CommandPalette  9851ms
packages/typespec-vscode test:e2e: ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
packages/typespec-vscode test:e2e:  FAIL  create-typespec.test.ts > CreateTypespecProject > CreateTypespecProject GenericRestAPI Trigger CommandPalette EmptyFolder
packages/typespec-vscode test:e2e: Error: electronApplication.evaluate: Execution context was destroyed, most likely because of a navigation.
packages/typespec-vscode test:e2e:  ❯ stubMultipleDialogs common/mock-dialogs.ts:56:14
packages/typespec-vscode test:e2e:      54| 
packages/typespec-vscode test:e2e:      55|   // [https://github.com/microsoft/playwright/issues/8278#issuecomment-…](https://github.com/microsoft/playwright/issues/8278#issuecomment-%E2%80%A6)
packages/typespec-vscode test:e2e:      56|   return app.evaluate(({ dialog }, mocks) => {
packages/typespec-vscode test:e2e:        |              ^
packages/typespec-vscode test:e2e:      57|     mocks.forEach((mock) => {
packages/typespec-vscode test:e2e:      58|       const thisDialog = dialog[mock.method];
packages/typespec-vscode test:e2e:  ❯ stubDialog common/mock-dialogs.ts:37:10
packages/typespec-vscode test:e2e:  ❯ mockShowOpenDialog common/mock-dialogs.ts:79:10
packages/typespec-vscode test:e2e:  ❯ create-typespec.test.ts:72:11
packages/typespec-vscode test:e2e: ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯
packages/typespec-vscode test:e2e:  FAIL  create-typespec.test.ts > CreateTypespecProject > CreateTypespecProject GenericRestAPI Trigger CommandPalette EmptyFolder
packages/typespec-vscode test:e2e: Error: electronApplication.firstWindow: Target page, context or browser has been closed
packages/typespec-vscode test:e2e:  ❯ common/utils.ts:62:30
packages/typespec-vscode test:e2e:      60|         ].filter((v): v is string => !!v),
packages/typespec-vscode test:e2e:      61|       });
packages/typespec-vscode test:e2e:      62|       const page = await app.firstWindow();
packages/typespec-vscode test:e2e:        |                              ^
packages/typespec-vscode test:e2e:      63|       const tracePath = join(projectRoot, "test-results", task.name, "…
packages/typespec-vscode test:e2e:      64|       const artifactsDir = join(tempDir, "playwright-artifacts");
packages/typespec-vscode test:e2e:  ❯ create-typespec.test.ts:68:27
packages/typespec-vscode test:e2e: ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/2]⎯
packages/typespec-vscode test:e2e:  Test Files  1 failed | 3 passed (4)
packages/typespec-vscode test:e2e:       Tests  1 failed | 4 passed (5)
packages/typespec-vscode test:e2e:    Start at  20:53:40
packages/typespec-vscode test:e2e:    Duration  217.17s (transform 123ms, setup 0ms, import 1.92s, tests 210.49s, environment 1ms)
packages/typespec-vscode test:e2e:  ELIFECYCLE  Command failed with exit code 1.
packages/typespec-vscode test:e2e: Failed
/mnt/vss/_work/1/s/packages/typespec-vscode:
 ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  typespec-vscode@1.9.0 test:e2e: `pnpm test:web && pnpm test:extension`
Exit status 1
 ELIFECYCLE  Command failed with exit code 1.

Example pipelines

• https://dev.azure.com/azure-sdk/public/_build/results?buildId=5968283&view=logs&jobId=9ea70591-ad2c-549f-9859-1ba11dbbd52c&j=9ea70591-ad2c-549f-9859-1ba11dbbd52c&t=10398ed4-c55a-5196-35ee-bfe9b8fba852

Full log

``` packages/typespec-vscode test:e2e: packages/typespec-vscode test:e2e: > npx -y -p /mnt/vss/_work/1/s/temp/artifacts/typespec-compiler-1.9.0.tgz tsp install packages/typespec-vscode test:e2e: npm warn Unknown env config "auto-install-peers". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown env config "store-dir". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown env config "manage-package-manager-versions". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown env config "verify-deps-before-run". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown env config "minimum-release-age". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown env config "npm-globalconfig". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown env config "overrides". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown env config "recursive". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown env config "_jsr-registry". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: npm warn Unknown global config "store-dir". This will stop working in the next major version of npm. See `npm help npmrc` for supported config options. packages/typespec-vscode test:e2e: added 136 packages, and audited 137 packages in 5s packages/typespec-vscode test:e2e: 25 packages are looking for funding packages/typespec-vscode test:e2e: run `npm fund` for details packages/typespec-vscode test:e2e: found 0 vulnerabilities packages/typespec-vscode test:e2e: ✓ preview-typespec.test.ts (1 test) 57932ms packages/typespec-vscode test:e2e: ✓ PreviewTypespecProject Trigger CommandPalette 9851ms packages/typespec-vscode test:e2e: ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯ packages/typespec-vscode test:e2e: FAIL create-typespec.test.ts > CreateTypespecProject > CreateTypespecProject GenericRestAPI Trigger CommandPalette EmptyFolder packages/typespec-vscode test:e2e: Error: electronApplication.evaluate: Execution context was destroyed, most likely because of a navigation. packages/typespec-vscode test:e2e: ❯ stubMultipleDialogs common/mock-dialogs.ts:56:14 packages/typespec-vscode test:e2e: 54| packages/typespec-vscode test:e2e: 55| // [https://github.com/microsoft/playwright/issues/8278#issuecomment-…](https://github.com/microsoft/playwright/issues/8278#issuecomment-%E2%80%A6) packages/typespec-vscode test:e2e: 56| return app.evaluate(({ dialog }, mocks) => { packages/typespec-vscode test:e2e: | ^ packages/typespec-vscode test:e2e: 57| mocks.forEach((mock) => { packages/typespec-vscode test:e2e: 58| const thisDialog = dialog[mock.method]; packages/typespec-vscode test:e2e: ❯ stubDialog common/mock-dialogs.ts:37:10 packages/typespec-vscode test:e2e: ❯ mockShowOpenDialog common/mock-dialogs.ts:79:10 packages/typespec-vscode test:e2e: ❯ create-typespec.test.ts:72:11 packages/typespec-vscode test:e2e: ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯ packages/typespec-vscode test:e2e: FAIL create-typespec.test.ts > CreateTypespecProject > CreateTypespecProject GenericRestAPI Trigger CommandPalette EmptyFolder packages/typespec-vscode test:e2e: Error: electronApplication.firstWindow: Target page, context or browser has been closed packages/typespec-vscode test:e2e: ❯ common/utils.ts:62:30 packages/typespec-vscode test:e2e: 60| ].filter((v): v is string => !!v), packages/typespec-vscode test:e2e: 61| }); packages/typespec-vscode test:e2e: 62| const page = await app.firstWindow(); packages/typespec-vscode test:e2e: | ^ packages/typespec-vscode test:e2e: 63| const tracePath = join(projectRoot, "test-results", task.name, "… packages/typespec-vscode test:e2e: 64| const artifactsDir = join(tempDir, "playwright-artifacts"); packages/typespec-vscode test:e2e: ❯ create-typespec.test.ts:68:27 packages/typespec-vscode test:e2e: ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/2]⎯ packages/typespec-vscode test:e2e: Test Files 1 failed | 3 passed (4) packages/typespec-vscode test:e2e: Tests 1 failed | 4 passed (5) packages/typespec-vscode test:e2e: Start at 20:53:40 packages/typespec-vscode test:e2e: Duration 217.17s (transform 123ms, setup 0ms, import 1.92s, tests 210.49s, environment 1ms) packages/typespec-vscode test:e2e:  ELIFECYCLE  Command failed with exit code 1. packages/typespec-vscode test:e2e: Failed /mnt/vss/_work/1/s/packages/typespec-vscode:  ERR_PNPM_RECURSIVE_RUN_FIRST_FAIL  typespec-vscode@1.9.0 test:e2e: `pnpm test:web && pnpm test:extension` Exit status 1  ELIFECYCLE  Command failed with exit code 1. ```

[timotheeguerin]

Some of logs before, seems related to a vscode crash

CodeExpectedError
packages/typespec-vscode test:e2e:     at ene.withProvider (http://localhost:3000/static/build/out/vs/workbench/workbench.web.main.internal.js:4943:134359)
packages/typespec-vscode test:e2e:     at async ene.doWatch (http://localhost:3000/static/build/out/vs/workbench/workbench.web.main.internal.js:4943:147217)
packages/typespec-vscode test:e2e:     at async http://localhost:3000/static/build/out/vs/workbench/workbench.web.main.internal.js:4943:146907
packages/typespec-vscode test:e2e: Settings Sync: Account status changed from uninitialized to unavailable
packages/typespec-vscode test:e2e: ENOPRO: No file system provider found for resource 'vscode-test-web://mount/.vscode': CodeExpectedError
packages/typespec-vscode test:e2e:     at ene.withProvider (http://localhost:3000/static/build/out/vs/workbench/workbench.web.main.internal.js:4943:134359)
packages/typespec-vscode test:e2e:     at async ene.exists (http://localhost:3000/static/build/out/vs/workbench/workbench.web.main.internal.js:4943:136678)
packages/typespec-vscode test:e2e:     at async http://localhost:3000/static/build/out/vs/workbench/workbench.web.main.internal.js:1009:24660
packages/typespec-vscode test:e2e:     at async Promise.all (index 0)
packages/typespec-vscode test:e2e:     at async sEe._initWorkspaceFolderSnippets (http://localhost:3000/static/build/out/vs/workbench/workbench.web.main.internal.js:1009:24825)
packages/typespec-vscode test:e2e: GET /extensions/groovy/snippets/groovy.code-snippets - - ms - -
packages/typespec-vscode test:e2e:   Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
packages/typespec-vscode test:e2e:       at ServerResponse.<anonymous> (node:internal/streams/pipeline:415:29)
packages/typespec-vscode test:e2e:       at ServerResponse.emit (node:events:531:35)
packages/typespec-vscode test:e2e:       at emitCloseNT (node:_http_server:1066:10)
packages/typespec-vscode test:e2e:       at Socket.onServerResponseClose (node:_http_server:285:5)
packages/typespec-vscode test:e2e:       at Socket.emit (node:events:531:35)
packages/typespec-vscode test:e2e:       at TCP.<anonymous> (node:net:346:12)
packages/typespec-vscode test:e2e:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
packages/typespec-vscode test:e2e:   Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
packages/typespec-vscode test:e2e:       at ServerResponse.<anonymous> (node:internal/streams/pipeline:415:29)
packages/typespec-vscode test:e2e:       at ServerResponse.emit (node:events:531:35)
packages/typespec-vscode test:e2e:       at emitCloseNT (node:_http_server:1066:10)
packages/typespec-vscode test:e2e:       at Socket.onServerResponseClose (node:_http_server:285:5)
packages/typespec-vscode test:e2e:       at Socket.emit (node:events:531:35)
packages/typespec-vscode test:e2e:       at TCP.<anonymous> (node:net:346:12)
packages/typespec-vscode test:e2e:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
packages/typespec-vscode test:e2e:   Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
packages/typespec-vscode test:e2e:       at ServerResponse.<anonymous> (node:internal/streams/pipeline:415:29)
packages/typespec-vscode test:e2e:       at ServerResponse.emit (node:events:531:35)
packages/typespec-vscode test:e2e:       at emitCloseNT (node:_http_server:1066:10)
packages/typespec-vscode test:e2e:       at Socket.onServerResponseClose (node:_http_server:285:5)
packages/typespec-vscode test:e2e:       at Socket.emit (node:events:531:35)
packages/typespec-vscode test:e2e:       at TCP.<anonymous> (node:net:346:12)
packages/typespec-vscode test:e2e:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
packages/typespec-vscode test:e2e:   Error [ERR_STREAM_PREMATURE_CLOSE]: Premature close
packages/typespec-vscode test:e2e:       at ServerResponse.<anonymous> (node:internal/streams/pipeline:415:29)
packages/typespec-vscode test:e2e:       at ServerResponse.emit (node:events:531:35)
packages/typespec-vscode test:e2e:       at emitCloseNT (node:_http_server:1066:10)
packages/typespec-vscode test:e2e:       at Socket.onServerResponseClose (node:_http_server:285:5)
packages/typespec-vscode test:e2e:       at Socket.emit (node:events:531:35)
packages/typespec-vscode test:e2e:       at TCP.<anonymous> (node:net:346:12)
packages/typespec-vscode test:e2e:       at TCP.callbackTrampoline (node:internal/async_hooks:130:17)
packages/typespec-vscode test:e2e: > typespec-vscode@1.9.0 test:extension /mnt/vss/_work/1/s/packages/typespec-vscode

[RodgeFu]

@v-xuto , please help to have a look, thx

[timotheeguerin]

juste to note the last error actually isn't related, it is thrown in the vscode web e2e test which always pass.

[zedy-wj]

Has this intermittent problem occurred recently? The pipeline link you provided seems to have expired, so I’m unable to access it or check the latest status. I’ve run many tests locally but haven’t been able to reproduce the issue.

[timotheeguerin]

Fails pretty regularly, its definitely not consitent

Last publish pipeline that failed. Also fail in the typespec ci from time to time, though having a harder time finding which ones as it runs a lot more across standalone emitters too

• https://dev.azure.com/azure-sdk/internal/_build/results?buildId=6005089&view=logs&j=ea30bdc8-161d-5396-05e1-aa7d235d8149&t=8bc488e4-922e-5081-707e-9e43737d812d

• typespec ci https://dev.azure.com/azure-sdk/public/_build/results?buildId=6016396&view=logs&j=9ea70591-ad2c-549f-9859-1ba11dbbd52c&t=10398ed4-c55a-5196-35ee-bfe9b8fba852

[zedy-wj]

@RodgeFu, @timotheeguerin We raised a PR #10106 and added some optimizations. Please help review.