Skip to content

playwright.md

Latest commit

 

History

History
449 lines (299 loc) · 12.8 KB

playwright.md

File metadata and controls

449 lines (299 loc) · 12.8 KB

Playwright in Element Web

Contents

Overview

Element Web contains two sets of Playwright tests:

  1. Element Web E2E Tests (playwright/e2e/) - Full end-to-end tests of the Element Web application with real homeserver instances
  2. Shared Components Tests (packages/shared-components/) - Visual regression tests for the shared component library using Storybook

Both test suites run automatically in CI on every pull request and on every merge to develop & master.

Prerequisites

Before running Playwright tests, ensure you have the following set up:

1. Install Playwright Browsers and System Dependencies

Follow the Playwright installation instructions:

pnpm playwright install --with-deps

2. Container Runtime

See Supported Container Runtimes for details on supported container runtimes (Docker, Podman, Colima).

3. Element Web Server (for E2E tests)

Element Web E2E tests require an instance running on http://localhost:8080 (configured in playwright.config.ts).

You can either:

  • Run manually: pnpm start in a separate terminal (not working for screenshot tests running in a docker environment).
  • Auto-start: Playwright will start the webserver automatically if it's not already running

Running the Tests

Element Web E2E Tests

Our main Playwright tests run against a full Element Web instance with Synapse/Dendrite homeservers.

Run all E2E tests:

pnpm run test:playwright

Run a specific test file:

pnpm run test:playwright playwright/e2e/register/register.spec.ts

Run tests interactively with Playwright UI:

pnpm run test:playwright:open

Run screenshot tests only:

Warning

This command run the playwright tests in a docker environment.

pnpm run test:playwright:screenshots

For more information about visual testing, see Visual Testing.

Additional command line options: https://playwright.dev/docs/test-cli

Shared Components Tests

See the Shared Components README for instructions on running the shared components Playwright tests.

Projects

By default, Playwright runs tests against all "Projects": Chrome, Firefox, "Safari" (Webkit), Dendrite and Picone.

  • Chrome, Firefox, Safari run against Synapse
  • Dendrite and Picone run against Chrome

Misc:

  • Pull Request CI: Tests run only against Chrome
  • Merge Queue: Tests run against all projects
  • Some tests are excluded from certain browsers due to incompatibilities (see Test Tags)

How the Tests Work

Test Structure

Element Web tests are located in the playwright/ subdirectory:

  • playwright/e2e/ - E2E test files
  • playwright/testcontainers/ - Testcontainers for Synapse/Dendrite instances
  • playwright/snapshots/ - Visual regression test screenshots
  • playwright/pages/ - Page object models
  • playwright/plugins/ - Custom Playwright plugins

Homeserver Setup

Homeservers (Synapse or Dendrite) are launched by Playwright workers and reused for all tests matching the worker configuration.

Configure Synapse options:

test.use({
    synapseConfig: {
        // Configuration options for the Synapse instance
    },
});

Important notes:

  • Homeservers are reused between tests for efficiency
  • Please use unique names for any rooms put into the room directory as they may be visible from other tests, the suggested approach is to use testInfo.testId within the name or lodash's uniqueId.
  • We remove public rooms from the room directory between tests but deleting users doesn't have a homeserver agnostic solution.
  • Homeserver logs are attached to Playwright test reports

Fixtures

We heavily leverage Playwright fixtures to provide:

  • Homeserver instances (homeserver)
  • Logged-in users (user)
  • Bot users (bot)
  • Application state (app)

See Writing Tests for usage examples.

Writing Tests

For general Playwright best practices, see:

Getting a Homeserver

Use the homeserver fixture to acquire a Homeserver instance:

test("should do something", async ({ homeserver }) => {
    // homeserver is a ready-to-use Synapse/Dendrite instance
});

The fixture provides:

  • Server port information
  • Instance ID for shutdown
  • Registration shared secret (registrationSecret) for registering users via REST API

Homeserver instances are:

  • Reasonably cheap to start (first run may be slow while pulling Docker image)
  • Automatically cleaned up by the fixture

Logging In

Use the user fixture to get a logged-in user:

test("should do something", async ({ user }) => {
    // user is logged in and ready to use
});

Customize the user:

test.use({
    displayName: "Alice",
});

test("should do something", async ({ user }) => {
    // user is logged in as "Alice"
});

What the fixture does:

  • Registers a random userId with the registrationSecret
  • Generates a random password (or uses specified display name)
  • Seeds localStorage with credentials
  • Loads the app at path /
  • Provides user details for User-Interactive Auth if needed

Joining a Room

To start with a user in a room:

test("should send a message", async ({ user, app, bot }) => {
    // Use the bot client to create a room
    const roomId = await bot.createRoom({
        name: "Test Room",
        invite: [user.userId],
    });

    // Accept the invite using the app client
    await app.client.joinRoom(roomId);

    // Now ready to test messaging
});

Best practice: Use the REST API (via bot or app.client) to set up room state rather than driving the UI.

Using matrix-js-sdk

Due to CI constraints, use the matrix-js-sdk module exposed on window.matrixcs:

const matrixcs = window.matrixcs;

Limitation: Only accessible when the app is loaded. This may be revisited in the future.

Best Practices

For more guidance, see the Playwright best practices guide.

1. Test from the User's Perspective

Work with roles, labels, and accessible elements rather than CSS selectors:

// Good
await page.getByRole("button", { name: "Send" }).click();

// Avoid
await page.locator(".mx_MessageComposer_sendButton").click();

See https://playwright.dev/docs/locators for more guidance.

2. Test Well-Isolated Functionality

  • Focus on specific, well-defined units of functionality
  • Easier to debug when tests fail
  • More maintainable over time

3. Maintain Test Independence

  • Each test should run successfully in isolation
  • Don't depend on state from other tests
  • Clean up after your test if needed

4. Minimize UI Driving for Setup

  • Use REST APIs to set up test state when possible
  • Only drive the UI for the functionality you're actually testing

Example:

// Testing reactions - good approach
test("should react to a message", async ({ page, app, bot }) => {
    // Send message via API
    const eventId = await bot.sendMessage(roomId, "Hello");

    // Test the reaction UI
    await page.getByText("Hello").hover();
    await page.getByRole("button", { name: "React" }).click();
    await page.getByLabel("😀").click();

    // Verify reaction was sent
    await expect(page.getByLabel("😀 1")).toBeVisible();
});

5. Avoid Explicit Waits

Playwright locators and assertions automatically wait and retry:

// Good - implicit waiting
await expect(page.getByText("Message sent")).toBeVisible();

// Avoid - explicit waits
await page.waitForTimeout(1000);

For dynamic content:

// Assert on the final state - Playwright will wait for it
await expect(page.getByRole("textbox")).toHaveValue("Edited message");
await expect(page.getByText("edited")).toBeVisible();

When you do need to wait:

// Wait for network requests
await page.waitForResponse("**/messages");

// Wait for specific conditions
await page.waitForFunction(() => window.matrixcs !== undefined);

Visual Testing

Playwright has built-in support for visual comparison testing.

Screenshot location: playwright/snapshots/

Rendering environment: Linux Docker (for consistency across environments)

Test Tag for Screenshots

All screenshot tests must use the @screenshot tag:

test("should render message list", { tag: "@screenshot" }, async ({ page }) => {
    await expect(page).toMatchScreenshot("message-list.png");
});

Purpose of @screenshot tag:

  • Allows running only screenshot tests via test:playwright:screenshots
  • Speeds up screenshot test runs and updates

Taking Screenshots

Use the custom toMatchScreenshot assertion (not the native toHaveScreenshot):

await expect(page).toMatchScreenshot("my-screenshot.png");

Why a custom assertion? We inject custom CSS to stabilize dynamic UI elements (e.g., BaseAvatar color selection based on Matrix ID hash).

Masking Dynamic Content

Always mask dynamic content that changes between runs:

await expect(page).toMatchScreenshot("chat.png", {
    mask: [page.locator(".mx_MessageTimestamp"), page.locator(".mx_BaseAvatar")],
});

Common elements to mask:

  • Timestamps
  • Avatars (when dynamic)
  • Animated elements
  • User-generated IDs

See Playwright masking docs for more details.

Updating Screenshots

This command runs only tests tagged with @screenshot in the Docker environment. When you need to update screenshot baselines (e.g., after intentional UI changes):

pnpm run test:playwright:screenshots

Important: Always use this command to update screenshots rather than running tests locally with --update-snapshots.

Why? Screenshots must be rendered in a consistent Linux Docker environment because:

  • Font rendering differs between operating systems (macOS, Windows, Linux)
  • Subpixel rendering varies across systems
  • Browser rendering engines have platform-specific differences

Using test:playwright:screenshots ensures screenshots are generated in the same Docker environment used in CI, preventing false failures due to rendering differences.

Test Tags

Test tags categorize tests for efficient subset execution.

Available Tags

  • @mergequeue: Slow or flaky tests covering rarely-updated app areas

    • Not run on every PR commit
    • Run in the Merge Queue

  • @screenshot: Tests using toMatchScreenshot for visual regression testing

  • @no-firefox: Tests unsupported in Firefox

    • Automatically skipped in Firefox project
    • Common reason: Service worker required (disabled in Playwright Firefox for routing)

  • @no-webkit: Tests unsupported in Webkit

    • Automatically skipped in Webkit project
    • Common reasons: Service worker required, microphone functionality unavailable

Running All Tests in a PR

Add the X-Run-All-Tests label to your pull request to run all tests, including @mergequeue tests.

Supported Container Runtimes

We use testcontainers to manage Synapse, Matrix Authentication Service, and other service instances.

Supported runtimes:

Platform-Specific Configuration

Colima users:

If using Colima, you may need to set the TMPDIR environment variable to allow bind mounting temporary directories:

export TMPDIR=/tmp/colima
# or
export TMPDIR=$HOME/tmp