-
Notifications
You must be signed in to change notification settings - Fork 0
feat(PINE-27): Setup Playwright E2E testing infrastructure #25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
ce9ccbf
feat: add multi-database abstraction layer
stepandel f48e9b6
feat(PINE-27): Setup Playwright E2E testing infrastructure
stepandel 572ad74
Revert "feat: add multi-database abstraction layer"
stepandel aa6db14
refactor(e2e): remove multi-db helpers from test setup
stepandel 9c874a7
refactor(e2e): remove Qdrant/Weaviate tests
stepandel 31434ce
chore: remove docker test infrastructure
stepandel 2786f36
fix(e2e): clear encrypted stores before test runs
stepandel 5bf7944
fix(e2e): add required packages field to pnpm-workspace.yaml
github-actions[bot] 8455b42
chore: remove e2e GitHub Actions workflow
stepandel 369bdd5
fix(e2e): add null checks in afterAll hook
stepandel ec42692
fix(e2e): isolate test data to prevent deletion of real user settings
github-actions[bot] b811898
fix(e2e): remove unnecessary 1s sleep in connectToProfile
github-actions[bot] 9ac319d
fix(e2e): guard against empty E2E_USER_DATA_DIR to prevent CWD deletion
github-actions[bot] File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,220 @@ | ||
| # E2E Testing Guide | ||
|
|
||
| This document provides guidance for running and developing end-to-end (E2E) tests for Pinecone Explorer using Playwright. | ||
|
|
||
| ## Overview | ||
|
|
||
| The E2E testing infrastructure uses **Playwright** for Electron app automation. | ||
|
|
||
| > **Note**: E2E tests run locally only. Electron cannot run in CI sandbox environments like GitHub Actions. | ||
|
|
||
| ## Architecture | ||
|
|
||
| ### Test Strategy | ||
| Tests create database connection profiles programmatically via the `window.electronAPI` exposed by the preload script. This approach: | ||
| - Tests the actual IPC communication path users take | ||
| - Avoids modifying production code for test purposes | ||
| - Validates the full integration flow | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - Node.js 22+ | ||
| - pnpm 9+ | ||
|
|
||
| ## Quick Start | ||
|
|
||
| ### Run Tests | ||
|
|
||
| ```bash | ||
| # Run all tests | ||
| pnpm run test:e2e | ||
|
|
||
| # Run with UI mode (interactive) | ||
| pnpm run test:e2e:ui | ||
|
|
||
| # Run with debugger | ||
| pnpm run test:e2e:debug | ||
| ``` | ||
|
|
||
| ### View Test Results | ||
|
|
||
| After tests complete: | ||
| ```bash | ||
| pnpm exec playwright show-report | ||
| ``` | ||
|
|
||
| ## Available NPM Scripts | ||
|
|
||
| | Script | Description | | ||
| |--------|-------------| | ||
| | `test:build` | Build Electron app for testing | | ||
| | `test:e2e` | Build and run E2E tests | | ||
| | `test:e2e:ui` | Run tests in interactive UI mode | | ||
| | `test:e2e:debug` | Run tests with debugger | | ||
|
|
||
| ## Project Structure | ||
|
|
||
| ``` | ||
| pinecone-explorer/ | ||
| ├── e2e/ | ||
| │ ├── electron.setup.ts # Electron launcher and profile utilities | ||
| │ └── example.spec.ts # Example test suite | ||
| └── playwright.config.ts # Playwright configuration | ||
| ``` | ||
|
|
||
| ## Writing Tests | ||
|
|
||
| ### Basic Test Structure | ||
|
|
||
| ```typescript | ||
| import { test, expect } from '@playwright/test' | ||
| import { | ||
| launchElectronApp, | ||
| closeElectronApp, | ||
| cleanupTestProfiles, | ||
| createPineconeTestProfile, | ||
| connectToProfile, | ||
| type ElectronTestContext, | ||
| } from './electron.setup' | ||
|
|
||
| let electronContext: ElectronTestContext | ||
|
|
||
| test.beforeAll(async () => { | ||
| electronContext = await launchElectronApp() | ||
| }) | ||
|
|
||
| test.afterAll(async () => { | ||
| await cleanupTestProfiles(electronContext.page) | ||
| await closeElectronApp(electronContext.app) | ||
| }) | ||
|
|
||
| test('should create Pinecone profile', async () => { | ||
| const { page } = electronContext | ||
|
|
||
| // Create profile programmatically | ||
| const profileId = await createPineconeTestProfile(page) | ||
|
|
||
| // Connect to profile | ||
| await connectToProfile(page, profileId) | ||
|
|
||
| // Use electronAPI to interact with the database | ||
| const indexes = await page.evaluate(async (id) => { | ||
| return await (window as any).electronAPI.pinecone.listIndexes(id) | ||
| }, profileId) | ||
|
|
||
| expect(Array.isArray(indexes)).toBe(true) | ||
| }) | ||
| ``` | ||
|
|
||
| ### Available Setup Utilities | ||
|
|
||
| #### `launchElectronApp(): Promise<ElectronTestContext>` | ||
| Launches the Electron app with test environment variables. | ||
|
|
||
| #### `createPineconeTestProfile(page, name?, apiKey?): Promise<string>` | ||
| Creates a Pinecone profile and returns the profile ID. | ||
|
|
||
| #### `connectToProfile(page, profileId): Promise<void>` | ||
| Connects to a profile via IPC. | ||
|
|
||
| #### `cleanupTestProfiles(page): Promise<void>` | ||
| Deletes all test profiles (IDs starting with 'test-'). | ||
|
|
||
| #### `closeElectronApp(app): Promise<void>` | ||
| Gracefully closes the Electron app. | ||
|
|
||
| ### Test Isolation | ||
|
|
||
| Each test should: | ||
| 1. Create its own unique profile using timestamp-based IDs | ||
| 2. Clean up indexes/data created during the test | ||
| 3. Use the `afterAll` hook to delete test profiles | ||
|
|
||
| ## Environment Variables | ||
|
|
||
| The following environment variables are automatically set during tests: | ||
|
|
||
| ```bash | ||
| NODE_ENV=test | ||
| DISABLE_ANALYTICS=true | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Electron won't launch | ||
|
|
||
| **Problem**: Tests fail with "Could not find Electron app" | ||
|
|
||
| **Solution**: | ||
| ```bash | ||
| # Ensure the app is built | ||
| pnpm run test:build | ||
|
|
||
| # Verify main.js exists | ||
| ls -la dist-electron/main.js | ||
| ``` | ||
|
|
||
| ### Profiles not created | ||
|
|
||
| **Problem**: Tests fail with "electronAPI is not defined" | ||
|
|
||
| **Solution**: | ||
| - Verify the preload script is loaded | ||
| - Check that `window.electronAPI` is exposed | ||
| - Enable debug mode: | ||
| ```bash | ||
| DEBUG=pw:api pnpm run test:e2e:debug | ||
| ``` | ||
|
|
||
| ### Tests timeout | ||
|
|
||
| **Problem**: Tests hang or timeout after 60 seconds | ||
|
|
||
| **Solutions**: | ||
| - Check if the app is launching properly | ||
| - Increase timeout in `playwright.config.ts` if needed | ||
| - Run in headed mode to see what's happening: | ||
| ```bash | ||
| pnpm run test:e2e:debug | ||
| ``` | ||
|
|
||
| ## Pinecone Testing | ||
|
|
||
| Since Pinecone requires a cloud connection, testing against real Pinecone requires: | ||
|
|
||
| 1. Set the `PINECONE_API_KEY` environment variable: | ||
| ```bash | ||
| PINECONE_API_KEY=your-key-here pnpm run test:e2e | ||
| ``` | ||
|
|
||
| 2. Use the free tier for testing (avoid costs) | ||
|
|
||
| 3. Consider skipping Pinecone API tests in local development: | ||
| ```typescript | ||
| test.skip(!process.env.PINECONE_API_KEY, 'should work with Pinecone', async () => { | ||
| // Pinecone test | ||
| }) | ||
| ``` | ||
|
|
||
| ## Best Practices | ||
|
|
||
| 1. **Always build before testing**: The `test:e2e` script does this automatically | ||
| 2. **Use unique IDs**: All test profiles use timestamp-based IDs to avoid conflicts | ||
| 3. **Clean up resources**: Delete test indexes and profiles in `afterAll` hooks | ||
| 4. **Serial execution**: Tests run serially (workers: 1) to avoid conflicts | ||
| 5. **Capture artifacts**: Screenshots, videos, and traces are captured on failure | ||
|
|
||
| ## Performance | ||
|
|
||
| - **Build time**: ~10-15 seconds | ||
| - **Test execution**: ~5-10 seconds per test | ||
| - **Total workflow**: ~30 seconds - 1 minute | ||
|
|
||
| ## Future Improvements | ||
|
|
||
| - [ ] Add visual regression testing | ||
| - [ ] Add performance benchmarks | ||
| - [ ] Test more complex workflows (multi-step operations) | ||
| - [ ] Add accessibility testing | ||
| - [ ] Test offline/error scenarios | ||
| - [ ] Investigate CI options for Electron testing (self-hosted runners, etc.) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.