diff --git a/README.md b/README.md index b0a111d..e90a4ce 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,7 @@ Every pull request will be checked against WPCS through GitHub Actions. ## Development and releases See [GitHub Actions: tests and publishing](docs/workflows.md) for workflow selection, reports, and release instructions. +See [Testing the plugin](tests/README.md) for local setup and test coverage. ## Version History ### 2.1.4 diff --git a/RELEASE_GUIDE.md b/RELEASE_GUIDE.md deleted file mode 100644 index d887215..0000000 --- a/RELEASE_GUIDE.md +++ /dev/null @@ -1,154 +0,0 @@ -# WordPress Plugin Release Guide - -This guide explains how to release your WordPress plugin using different deployment modes. - -## 🏷️ **Tag Patterns for Different Modes** - -The release workflow automatically detects the deployment mode based on your git tag: - -### **1. Dry Run Mode** (Safe Testing) -```bash -git tag v2.0.8-dry-run -git push origin v2.0.8-dry-run -``` -**What happens:** -- ✅ Creates GitHub release with zip file -- ✅ Runs deployment in dry-run mode -- ✅ Shows what would be deployed -- ❌ No actual SVN operations -- ❌ No changes made to any repository - -### **2. Test Mode** (Test Repository) -```bash -git tag v2.0.8-test -git push origin v2.0.8-test -``` -**What happens:** -- ✅ Creates GitHub release with zip file -- ✅ Deploys to test SVN repository -- ✅ Uses test credentials -- ✅ Marks commits as "TEST:" -- ❌ No changes to production WordPress.org - -### **3. Production Mode** (Live Deployment) -```bash -git tag v2.0.8 -git push origin v2.0.8 -``` -**What happens:** -- ✅ Creates GitHub release with zip file -- ✅ Deploys to WordPress.org marketplace -- ✅ Uses production credentials -- ✅ Live deployment to users - -## 🔧 **Prerequisites** - -### **Required Secrets** -- `WP_SVN_USERNAME`: Your WordPress.org SVN username -- `WP_SVN_PASSWORD`: Your WordPress.org SVN password -- `TEST_SVN_USERNAME`: Your test SVN username (for test mode) -- `TEST_SVN_PASSWORD`: Your test SVN password (for test mode) - -### **Required Variables** -- `WP_PLUGIN_SLUG`: Your WordPress.org plugin slug -- `WP_SVN_URL`: Your WordPress.org SVN repository URL -- `TEST_SVN_URL`: Your test SVN repository URL -- `WP_ASSETS_DIR`: Directory containing plugin assets - -## 📋 **Release Process** - -### **Step 1: Prepare Your Release** -1. Update version in `siteimprove/siteimprove.php` -2. Update changelog in `siteimprove/readme.txt` -3. Commit and push your changes -4. Test locally with `./scripts/test-deploy.sh` - -### **Step 2: Choose Deployment Mode** - -#### **For Testing (Recommended First)** -```bash -# Dry run to validate everything -git tag v2.0.8-dry-run -git push origin v2.0.8-dry-run - -# Test with actual SVN operations -git tag v2.0.8-test -git push origin v2.0.8-test -``` - -#### **For Production** -```bash -# Live deployment to WordPress.org -git tag v2.0.8 -git push origin v2.0.8 -``` - -### **Step 3: Monitor the Workflow** -1. Go to **Actions** tab in your repository -2. Watch the workflow run -3. Check the logs for any issues -4. Verify the deployment was successful - -## 🎯 **Recommended Workflow** - -### **For New Versions:** -1. **Dry Run**: `v2.0.8-dry-run` - Validate files and process -2. **Test**: `v2.0.8-test` - Test with real SVN operations -3. **Production**: `v2.0.8` - Deploy to WordPress.org - -### **For Minor Updates:** -1. **Test**: `v2.0.8-test` - Quick validation -2. **Production**: `v2.0.8` - Deploy to WordPress.org - -## 🔍 **What to Check After Each Release** - -### **Dry Run Mode:** -- ✅ Files are prepared correctly -- ✅ Version is updated -- ✅ No errors in the process -- ✅ Deployment preview looks correct - -### **Test Mode:** -- ✅ Files are committed to test repository -- ✅ Tag is created in test repository -- ✅ Commit message starts with "TEST:" -- ✅ All files are present - -### **Production Mode:** -- ✅ Files are committed to WordPress.org -- ✅ Tag is created on WordPress.org -- ✅ Plugin appears on WordPress.org -- ✅ Version is available for download - -## 🚨 **Troubleshooting** - -### **Common Issues:** - -**"Workflow not triggered"** -- Ensure tag matches pattern: `v*`, `v*-test`, or `v*-dry-run` -- Check that tag was pushed to the repository - -**"SVN authentication failed"** -- Verify credentials are set correctly -- Check that credentials have write access -- For test mode, ensure test repository exists - -**"Files not found"** -- Ensure `siteimprove/` directory exists -- Check that main plugin file is present -- Verify file paths in the action - -**"Version already exists"** -- Use a different version number -- Delete existing tag if needed -- Check WordPress.org for existing versions - -## 📊 **Workflow Summary** - -| Tag Pattern | Mode | SVN Operations | Credentials | Purpose | -|-------------|------|----------------|-------------|---------| -| `v2.0.8-dry-run` | Dry Run | ❌ None | Production | Validate files | -| `v2.0.8-test` | Test | ✅ Test repo | Test | Test deployment | -| `v2.0.8` | Production | ✅ WordPress.org | Production | Live release | - -This approach ensures safe, controlled releases with multiple validation steps before going live. \ No newline at end of file diff --git a/docs/workflows.md b/docs/workflows.md index 6975bd9..f988d96 100644 --- a/docs/workflows.md +++ b/docs/workflows.md @@ -38,7 +38,7 @@ No real Siteimprove credentials are used. Open the failed step first, then download **pr-browser-test-report** or **pr-wordpress-test-reports** from the run's Artifacts. Reports are retained for 14 days and contain individual assertions and synthetic failure screenshots. -A startup failure may have no browser report. See [testing details](../TESTING.md) +A startup failure may have no browser report. See [testing details](../tests/README.md) and the [regression catalog](../tests/integration-regressions/COVERAGE.md). ## Check the real Siteimprove connection @@ -59,7 +59,8 @@ highlighting. See [live test coverage and setup](../tests/live/README.md). ## Create and deploy a release 1. Merge the reviewed change after all PR checks pass. Update the plugin version - and release notes before tagging. + in `siteimprove/siteimprove.php` and the changelog in `siteimprove/readme.txt` + before tagging. 2. Use a version tag such as `v2.1.5`. Pushing that tag automatically starts production validation and publishing. Do not push a production tag merely to try a dry run. @@ -85,6 +86,15 @@ Production uses `WP_SVN_USERNAME`, `WP_SVN_PASSWORD`, and optional `WP_SVN_URL` `TEST_SVN_USERNAME`, `TEST_SVN_PASSWORD`, and `TEST_SVN_URL`, which must differ from production. Shared optional variables are `WP_PLUGIN_SLUG` and `WP_ASSETS_DIR`. +### Troubleshooting + +- If no run starts, check that the version tag was pushed and contains the release workflow. +- For SVN authentication failures, check the credentials for the selected mode and + their write access to the destination repository. +- For missing-file errors, confirm the tagged commit contains `siteimprove/siteimprove.php`. +- For an existing-version error, inspect the GitHub release and SVN tag before retrying; + do not delete published version tags to make a retry succeed. + ## Cleanup map The separate `prepublish-test.yml`, `prepublish-wordpress.yml`, and diff --git a/package.json b/package.json index 298c199..89e8547 100644 --- a/package.json +++ b/package.json @@ -2,19 +2,19 @@ "name": "siteimprove-wordpress-tests", "private": true, "scripts": { - "test": "playwright test", + "test": "playwright test --config=tests/config/playwright.config.js", "test:report": "playwright show-report", "env:start": "node scripts/wordpress-env.js start --runtime=playground", "env:stop": "node scripts/wordpress-env.js stop", "env:status": "node scripts/wordpress-env.js status", - "test:wordpress": "playwright test --config=playwright.wordpress.config.js", + "test:wordpress": "playwright test --config=tests/config/playwright.wordpress.config.js", "test:wordpress:report": "playwright show-report playwright-wordpress-report", "test:live": "node tests/live/run.js", "test:live:contracts": "node --test tests/live/*.test.js", - "test:live:fixture": "playwright test --config=playwright.live-fixture.config.js", + "test:live:fixture": "playwright test --config=tests/config/playwright.live-fixture.config.js", "env:regressions:start": "node scripts/integration-regressions-env.js start", "env:regressions:stop": "node scripts/integration-regressions-env.js stop", - "test:regressions": "playwright test --config=playwright.integration-regressions.config.js", + "test:regressions": "playwright test --config=tests/config/playwright.integration-regressions.config.js", "test:regressions:catalog": "node scripts/regression-coverage.js", "test:regressions:report": "playwright show-report playwright-integration-regressions-report" }, diff --git a/phpcs.xml b/phpcs.xml index 236b9fb..ddf68a7 100644 --- a/phpcs.xml +++ b/phpcs.xml @@ -11,7 +11,6 @@ /tests/* - /playwright*.config.js /scripts/* /node_modules/* diff --git a/TESTING.md b/tests/README.md similarity index 96% rename from TESTING.md rename to tests/README.md index dad7269..7e98c62 100644 --- a/TESTING.md +++ b/tests/README.md @@ -1,10 +1,13 @@ # Testing the plugin +Run the commands below from the repository root. Playwright configurations live in +`tests/config/`; reports and test results remain in their existing root directories. + ## WordPress integration regressions -The [integration regression suite](tests/integration-regressions/README.md) adds real +The [integration regression suite](integration-regressions/README.md) adds real WordPress multisite, capability, URL mapping and settings checks, plus browser -recheck contracts. Its [coverage catalog](tests/integration-regressions/COVERAGE.md) +recheck contracts. Its [coverage catalog](integration-regressions/COVERAGE.md) describes behavior and distinguishes automated coverage from planned scenarios. All fixtures must use synthetic identities, content, tokens and reserved domains. @@ -13,7 +16,7 @@ All fixtures must use synthetic identities, content, tokens and reserved domains These tests cover the fixture browser layer for prepublish flow and related plugin UI/command handoff contracts. The deployment checks further down this document only test packaging/deployment, not this bug. -For a plain-English checklist, read [Prepublish test scenarios](tests/SCENARIOS.md). +For a plain-English checklist, read [Prepublish test scenarios](SCENARIOS.md). The automated tests use named Given / When / Then steps, shown in the console, GitHub Actions logs and the browser suite's HTML report. The checklist clearly separates existing automated coverage from planned authenticated checks. @@ -108,7 +111,7 @@ by its command queue and callback contract; its actual UI is not exercised. ### GitHub Actions Use **Plugin tests** for automatic PR checks or manual plugin/runtime selection. -See [the workflow guide](docs/workflows.md) for inputs, tested commits, reports, +See [the workflow guide](../docs/workflows.md) for inputs, tested commits, reports, release procedures and credentials. The manual workflows have been consolidated; all browser, single-site and multisite suites remain. @@ -223,7 +226,7 @@ the missing-title result. This concerns page-content fidelity, not SDK UI stylin ### What remains for an actual Siteimprove end-to-end test The live workflow and runner are documented in -[tests/live/README.md](tests/live/README.md). GitHub runs against PR64 verified +[tests/live/README.md](live/README.md). GitHub runs against PR64 verified login, Live page data, draft handoff, and loading-state exit. The missing-title result assertion is explicitly skipped. The following account and network requirements still apply; local fixture suites use no secrets. @@ -319,7 +322,7 @@ References: [Playwright CI setup](https://playwright.dev/docs/ci-intro) and ## Deployment validation -See [Create and deploy a release](docs/workflows.md#create-and-deploy-a-release) +See [Create and deploy a release](../docs/workflows.md#create-and-deploy-a-release) for the supported tag formats, manual modes, destination settings, release checks, and dry-run limitations. The separate Marketplace deployment workflow is retired. @@ -343,7 +346,7 @@ unchanged; the check names remain **Browser tests**, **WordPress tests**, and ** **Create and deploy release** calls the shared `release-checks.yml` before any publishing. Every check must succeed, including the protected live test and ZIP -lifecycle check. See [the workflow guide](docs/workflows.md) for the exact sequence. +lifecycle check. See [the workflow guide](../docs/workflows.md) for the exact sequence. ### Live result assertion temporarily skipped diff --git a/playwright.config.js b/tests/config/playwright.config.js similarity index 71% rename from playwright.config.js rename to tests/config/playwright.config.js index 48e7f29..27a19b7 100644 --- a/playwright.config.js +++ b/tests/config/playwright.config.js @@ -1,16 +1,18 @@ +const path = require('node:path'); +const root = path.resolve(__dirname, '../..'); const { defineConfig } = require('@playwright/test'); module.exports = defineConfig({ // CI reports must not collect contributor identities or source diffs. captureGitInfo: { commit: false, diff: false }, - testDir: './tests/browser', + testDir: path.join(root, './tests/browser'), timeout: 45000, - outputDir: './test-results/browser', + outputDir: path.join(root, './test-results/browser'), expect: { timeout: 5000 }, forbidOnly: Boolean(process.env.CI), retries: 0, workers: process.env.CI ? 2 : undefined, - reporter: [['list', { printSteps: true }], ['html', { open: 'never' }]], + reporter: [['list', { printSteps: true }], ['html', { open: 'never', outputFolder: path.join(root, 'playwright-report') }]], use: { trace: 'retain-on-failure', screenshot: 'only-on-failure' }, projects: [ { name: 'chromium', use: { browserName: 'chromium' } }, diff --git a/playwright.integration-regressions.config.js b/tests/config/playwright.integration-regressions.config.js similarity index 59% rename from playwright.integration-regressions.config.js rename to tests/config/playwright.integration-regressions.config.js index ba67636..4f03add 100644 --- a/playwright.integration-regressions.config.js +++ b/tests/config/playwright.integration-regressions.config.js @@ -1,19 +1,21 @@ +const path = require('node:path'); +const root = path.resolve(__dirname, '../..'); const { defineConfig } = require('@playwright/test'); module.exports = defineConfig({ // CI reports must not collect contributor identities or source diffs. captureGitInfo: { commit: false, diff: false }, - testDir: './tests/integration-regressions', + testDir: path.join(root, './tests/integration-regressions'), timeout: 60000, expect: { timeout: 5000 }, workers: 1, // Tests reset options in one disposable multisite installation. retries: 0, forbidOnly: Boolean(process.env.CI), - outputDir: './test-results/integration-regressions', + outputDir: path.join(root, './test-results/integration-regressions'), reporter: [ ['list'], - ['html', { open: 'never', outputFolder: 'playwright-integration-regressions-report' }], - ['json', { outputFile: 'test-results/integration-regressions-results.json' }], + ['html', { open: 'never', outputFolder: path.join(root, 'playwright-integration-regressions-report') }], + ['json', { outputFile: path.join(root, 'test-results/integration-regressions-results.json') }], ], projects: ['chromium', 'firefox'].map(browserName => ({ name: browserName, use: { browserName } })), use: { baseURL: 'http://127.0.0.1', trace: 'off', screenshot: 'only-on-failure', video: 'off' }, diff --git a/playwright.live-fixture.config.js b/tests/config/playwright.live-fixture.config.js similarity index 53% rename from playwright.live-fixture.config.js rename to tests/config/playwright.live-fixture.config.js index 56fbe76..2147acb 100644 --- a/playwright.live-fixture.config.js +++ b/tests/config/playwright.live-fixture.config.js @@ -1,7 +1,9 @@ +const path = require('node:path'); +const root = path.resolve(__dirname, '../..'); const { defineConfig } = require('@playwright/test'); module.exports = defineConfig({ - testDir:'./tests/live', testMatch:['fixture.spec.js','observer.spec.js'], timeout:60000, workers:1, - outputDir:'./test-results/live-fixture', retries:0, forbidOnly:Boolean(process.env.CI), + testDir:path.join(root, './tests/live'), testMatch:['fixture.spec.js','observer.spec.js'], timeout:60000, workers:1, + outputDir:path.join(root, './test-results/live-fixture'), retries:0, forbidOnly:Boolean(process.env.CI), reporter:[['list',{printSteps:true}]], projects:[{name:'chromium',use:{browserName:'chromium'}},{name:'firefox',testIgnore:'observer.spec.js',use:{browserName:'firefox'}}], use:{baseURL:'http://localhost:8888',trace:'off',screenshot:'off',video:'off'}, diff --git a/playwright.wordpress.config.js b/tests/config/playwright.wordpress.config.js similarity index 73% rename from playwright.wordpress.config.js rename to tests/config/playwright.wordpress.config.js index af90ea7..7672d45 100644 --- a/playwright.wordpress.config.js +++ b/tests/config/playwright.wordpress.config.js @@ -1,17 +1,19 @@ +const path = require('node:path'); +const root = path.resolve(__dirname, '../..'); const { defineConfig } = require('@playwright/test'); module.exports = defineConfig({ // Keep contributor identities and source diffs out of reports. captureGitInfo: { commit: false, diff: false }, - testDir: './tests/wordpress', + testDir: path.join(root, './tests/wordpress'), timeout: 60000, - outputDir: './test-results/wordpress', + outputDir: path.join(root, './test-results/wordpress'), workers: 1, forbidOnly: Boolean(process.env.CI), retries: 0, reporter: [ ['list', { printSteps: true }], - ['html', { open: 'never', outputFolder: 'playwright-wordpress-report' }], + ['html', { open: 'never', outputFolder: path.join(root, 'playwright-wordpress-report') }], ], projects: [ { name: 'chromium', use: { browserName: 'chromium' } }, diff --git a/tests/live/safety.test.js b/tests/live/safety.test.js index 3cb8120..2ec0923 100644 --- a/tests/live/safety.test.js +++ b/tests/live/safety.test.js @@ -42,7 +42,7 @@ test('The live runner and workflow do not capture or upload account artifacts', assert.doesNotMatch(workflow, /uses:\s*[^\n]*(?:upload-artifact|upload-pages-artifact)/); assert.match(workflow, /run: npm run test:live\s/); assert.doesNotMatch(workflow, /run:\s*(?:npx playwright test|npm run test:live:fixture)/); - const fixtureConfig = require('../../playwright.live-fixture.config'); + const fixtureConfig = require('../config/playwright.live-fixture.config'); assert.equal(fixtureConfig.use.screenshot, 'off'); assert.equal(fixtureConfig.use.trace, 'off'); assert.equal(fixtureConfig.use.video, 'off');