diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..72fc798 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,37 @@ +version: 2 +updates: + - package-ecosystem: pip + # requirements.txt pins the dev tools; pyproject.toml keeps ranges, raised only when a fix needs it + directory: "/" + schedule: + interval: weekly + open-pull-requests-limit: 5 + versioning-strategy: increase-if-necessary + groups: + # one PR for all minor/patch bumps + pip-minor-patch: + patterns: + - "*" + update-types: + - minor + - patch + # majors usually need code changes, so they come apart from minor/patch + pip-major: + patterns: + - "*" + update-types: + - major + # security fixes in a single PR too + pip-security: + applies-to: security-updates + patterns: + - "*" + + - package-ecosystem: github-actions + directory: "/" + schedule: + interval: weekly + groups: + github-actions: + patterns: + - "*" diff --git a/.github/workflows/workflow.yml b/.github/workflows/workflow.yml index fe0f948..40f260c 100644 --- a/.github/workflows/workflow.yml +++ b/.github/workflows/workflow.yml @@ -27,10 +27,3 @@ jobs: run: pip install playwright && python -m playwright install-deps chromium - name: Run Tox run: tox - - name: Codacy Coverage Reporter - # secrets are not available to pull requests from forks - if: github.event_name == 'push' - uses: codacy/codacy-coverage-reporter-action@v1 - with: - project-token: ${{ secrets.CODACY_TOKEN }} - coverage-reports: coverage.xml diff --git a/.gitignore b/.gitignore index f7359d4..e73dbde 100644 --- a/.gitignore +++ b/.gitignore @@ -2,7 +2,6 @@ junit visual_regression_tracker.egg-info htmlcov -coverage.xml build .idea dist diff --git a/README.md b/README.md index 8d8b5df..384d9b9 100644 --- a/README.md +++ b/README.md @@ -1,283 +1,163 @@ # Python SDK for [Visual Regression Tracker](https://github.com/Visual-Regression-Tracker/Visual-Regression-Tracker) -[![Codacy Badge](https://app.codacy.com/project/badge/Grade/0b98190cea064d6f9f1210da653ba37b)](https://www.codacy.com/gh/Visual-Regression-Tracker/sdk-python?utm_source=github.com&utm_medium=referral&utm_content=Visual-Regression-Tracker/sdk-python&utm_campaign=Badge_Grade) -[![Codacy Badge](https://app.codacy.com/project/badge/Coverage/0b98190cea064d6f9f1210da653ba37b)](https://www.codacy.com/gh/Visual-Regression-Tracker/sdk-python?utm_source=github.com&utm_medium=referral&utm_content=Visual-Regression-Tracker/sdk-python&utm_campaign=Badge_Coverage) +[![CI](https://github.com/Visual-Regression-Tracker/sdk-python/actions/workflows/workflow.yml/badge.svg)](https://github.com/Visual-Regression-Tracker/sdk-python/actions/workflows/workflow.yml) +[![PyPI](https://img.shields.io/pypi/v/visual-regression-tracker)](https://pypi.org/project/visual-regression-tracker/) -Requires Python 3.10 or later. Image URLs in results need VRT server 5.0.10 or later; use 5.x of this SDK with VRT 6.0. +Requires Python 3.10 or later. Use 5.x of this SDK with VRT 6.0; image URLs in results need VRT server 5.0.10 or later. ## Install ```sh pip install visual-regression-tracker -# or, with playwright integration +# or, with the Playwright integration pip install "visual-regression-tracker[playwright]" python -m playwright install ``` -## Usage +## Configuration -### Import +| Option | Environment variable | Default | Meaning | +|---|---|---|---| +| `apiUrl` | `VRT_APIURL` | `http://localhost:4200` | URL of the VRT API | +| `apiKey` | `VRT_APIKEY` | required | API key of a VRT user | +| `project` | `VRT_PROJECT` | required | Project name or ID | +| `branchName` | `VRT_BRANCHNAME` | required | Current git branch | +| `ciBuildId` | `VRT_CIBUILDID` | none | CI build ID, such as the commit SHA; runs with the same ID share a build | +| `enableSoftAssert` | `VRT_ENABLESOFTASSERT` | `false` | Log a failed check instead of raising `TestRunError` | +| `httpTimeoutInSeconds` | `VRT_HTTPTIMEOUTINSECONDS` | `360` | How long to wait for the server's answer, `0` for no limit. Connecting is limited to 10 s. | -```python -from visual_regression_tracker import VisualRegressionTracker, Config, TestRun, IgnoreArea -``` +The server compares images while it answers `track()`. With VLM checks or a busy comparison queue that can +take minutes, so keep `httpTimeoutInSeconds` above 300. -### Configure connection +A `Config` passed to the constructor is used as it is: -#### As python ```python -config = Config( - # apiUrl - URL where backend is running - apiUrl='http://localhost:4200', +from visual_regression_tracker import VisualRegressionTracker, Config - # project - Project name or ID - project='Default project', - - # apiKey - User apiKey +vrt = VisualRegressionTracker(Config( + apiUrl='http://localhost:4200', apiKey='tXZVHX0EA4YQM1MGDD', - - # ciBuildId - Current git commit SHA - ciBuildId='commit_sha', - - # branch - Current git branch + project='Default project', branchName='develop', - - # enableSoftAssert - Log errors instead of exceptions - enableSoftAssert=False, - - # httpTimeoutInSeconds - How long to wait for the server's answer, 0 for no limit. - # The server compares images while answering track(); with VLM checks or a busy - # comparison queue that can take minutes, so keep it above 300. - httpTimeoutInSeconds=360, -) - -vrt = VisualRegressionTracker(config) + ciBuildId='commit_sha', +)) ``` -#### Or, as JSON config file `vrt.json` +Without one, the SDK reads `vrt.json` from the current working directory, then lets environment variables +override it. `Config.default(path)` reads another file. + ```json { - "apiUrl":"http://localhost:4200", - "project":"Default project", - "apiKey":"tXZVHX0EA4YQM1MGDD", - "ciBuildId":"commit_sha", - "branchName":"develop", - "enableSoftAssert":false, - "httpTimeoutInSeconds":360 + "apiUrl": "http://localhost:4200", + "apiKey": "tXZVHX0EA4YQM1MGDD", + "project": "Default project", + "branchName": "develop", + "ciBuildId": "commit_sha", + "enableSoftAssert": false, + "httpTimeoutInSeconds": 360 } ``` -`vrt.json` is read from the current working directory. -```python -vrt = VisualRegressionTracker() -``` - -#### Or, as environment variables ```sh VRT_APIURL="http://localhost:4200" \ -VRT_PROJECT="Default project" \ VRT_APIKEY="tXZVHX0EA4YQM1MGDD" \ -VRT_CIBUILDID="commit_sha" \ +VRT_PROJECT="Default project" \ VRT_BRANCHNAME="develop" \ +VRT_CIBUILDID="commit_sha" \ VRT_ENABLESOFTASSERT=true \ VRT_HTTPTIMEOUTINSECONDS=360 \ - python -``` -Empty variables are ignored. -```python -vrt = VisualRegressionTracker() + python -m pytest ``` -### Setup / Teardown +Empty variables are ignored. `VRT_ENABLESOFTASSERT` is true for `true` or `1`. -As context manager: -```python -with vrt: - ... - # track test runs - ... -``` +## Usage -Without context manager: ```python -build = vrt.start() # Build(id, projectId) -... -# track test runs -... -vrt.stop() -``` - -### Assert +from visual_regression_tracker import VisualRegressionTracker, TestRun, IgnoreArea -```python -result = vrt.track(TestRun( - # Name to be displayed - # Required - name='Image name', - - # Base64 encoded string - # Required - imageBase64=image, - - # Allowed mismatch tollerance in % - # Optional - # Default: 0% - diffTollerancePercent=1, - - # Optional - os='Mac', - - # Optional - browser='Chrome', - - # Optional - viewport='800x600', - - # Optional - device='PC', - - # Array of areas to be ignored - ignoreAreas=[ - IgnoreArea( - # X-coordinate relative of left upper corner - # Required - x=10, - # Y-coordinate relative of left upper corner - # Required - y=20, - # Area width in px - # Required - width=300, - # Height width in px - # Required - height=400 - ) - ], -)) +with VisualRegressionTracker() as vrt: # start() on enter, stop() on exit + result = vrt.track(TestRun( + name='Home page', + imageBase64=image_base64, + )) ``` -`track` returns a `TestRunResult`: `testRunResponse` (with `status` and the `url` of the test run in the -VRT app) and `imageUrl`, `diffUrl`, `baselineUrl`. +Without the context manager, call `build = vrt.start()` (returns `Build(id, projectId)`) and `vrt.stop()`. -`track` raises `TestRunError` when the status is `new` (no baseline yet) or `unresolved` (difference found); -with `enableSoftAssert` it logs the error instead. `ok`, `approved` and `autoApproved` pass; `failed` passes -too, as in the JavaScript SDK. A status this SDK does not know becomes `TestRunStatus.UNKNOWN`. +`TestRun` fields: -An HTTP error from the server raises `ServerError` with `status_code` and the server's `message`. -A request that gets no answer (connection refused, connect timeout after 10 s, read timeout after -`httpTimeoutInSeconds`) raises `RequestError` with the `url`. A read timeout on `track` does not mean -the server dropped the test run: it may still be stored. +| Field | Meaning | +|---|---| +| `name` | Required. Name of the image in VRT | +| `imageBase64` | Required. The PNG image, base64 encoded | +| `os`, `browser`, `viewport`, `device` | Optional. Together with `name` they pick the baseline | +| `diffTollerancePercent` | Optional. Allowed difference in %, default 0. The spelling is the API's. | +| `ignoreAreas` | Optional. List of `IgnoreArea(x, y, width, height)` in pixels | -### Integration with Microsoft Playwright +### Playwright -#### Imports ```python from playwright.sync_api import sync_playwright -from visual_regression_tracker import Config -from visual_regression_tracker.playwright import PlaywrightVisualRegressionTracker -``` - -#### Start playwright and navigate to page -```python -playwright = sync_playwright().start() -browserType = playwright.chromium -browser = browserType.launch(headless=False) -page = browser.new_page() -page.goto('https://www.python.org/') -``` +from visual_regression_tracker.playwright import PlaywrightVisualRegressionTracker, PageTrackOptions -#### Configure connection to VRT server -```python -vrt = PlaywrightVisualRegressionTracker(browserType, config) -``` - -#### Setup / Tear down - -As context manager: -```python -with vrt: - ... - # track test runs - ... -``` - -Without context manager: -```python -vrt.start() -... -# track test runs -... -vrt.stop() -``` +with sync_playwright() as playwright: + browser_type = playwright.chromium + page = browser_type.launch().new_page() + page.goto('https://www.python.org/') -#### Track page -```python -result = vrt.trackPage(page, imageName[, options]) + with PlaywrightVisualRegressionTracker(browser_type) as vrt: + vrt.trackPage(page, 'Home page') + vrt.trackElementHandle(page.query_selector('#touchnav-wrapper'), 'Navigation') ``` -- `page: Page` [Playwright page](https://playwright.dev/python/docs/api/class-page) - -- `imageName: str` name for the taken screenshot image - -- `options: PageTrackOptions` optional configuration with: - - `diffTollerancePercent: float` specify acceptable difference from baseline, between `0-100`. - - - `ignoreAreas: List[IgnoreArea]` - - `x: int` X-coordinate relative of left upper corner - - `y: int` Y-coordinate relative of left upper corner - - `width: int` area width in px - - `height: int` area height in px +`browser` is the Playwright browser name and `viewport` the page's viewport size. The async API has +`trackPageAsync` and `trackElementHandleAsync`. - - `screenshotOptions: PageScreenshotOptions` configuration for Playwrights `screenshot` method - - `full_page: bool` When true, takes a screenshot of the full scrollable page, instead of the currently visible viewport. Defaults to `false`. +The last argument is optional: `PageTrackOptions` for `trackPage`, `ElementHandleTrackOptions` for +`trackElementHandle`. Both take: - - `omit_background: bool` Hides default white background and allows capturing screenshots with transparency. Defaults to `false`. +- `diffTollerancePercent` and `ignoreAreas`, as in `TestRun`; +- `agent: Agent(os, device, viewport)`: baseline fields Playwright doesn't know. `viewport` is used only for + element handles; +- `screenshotOptions`: arguments for Playwright's `screenshot()`. `PageScreenshotOptions` takes + `full_page`, `clip`, `omit_background`, `timeout`, `type`, `quality` and `path`. + `ElementHandleScreenshotOptions` takes the same, without `full_page` and `clip`. - - `clip: FloatRect` An object which specifies clipping of the resulting image. Should have the following fields: +## Result and statuses - - `x: float` x-coordinate of top-left corner of clip area - - `y: float` y-coordinate of top-left corner of clip area - - `width: float` width of clipping area - - `height: float` height of clipping area +`track` and the Playwright methods return a `TestRunResult`: - - `timeout: float` Maximum time in milliseconds, defaults to 30 seconds, pass 0 to disable timeout. - - - `agent: Agent` Additional information to mark baseline across agents that have different: - - `os: str` operating system name, like Windows, Mac, etc. - - `device: str` device name, PC identifier, mobile identifier etc. - - `viewport: str` viewport size. - -#### Track element handle -```python -result = vrt.trackElementHandle(elementHandle, imageName[, options]) -``` +- `testRunResponse`: `id`, `status`, `url` (the test run in the VRT app), `diffPercent`, + `diffTollerancePercent`, `pixelMisMatchCount`, `merge`; +- `imageUrl`, `diffUrl` and `baselineUrl`: the images, under `{apiUrl}/images/`. `diffUrl` and + `baselineUrl` are `None` when there is no diff or baseline. -The async API has `trackPageAsync` and `trackElementHandleAsync`. All four return the same `TestRunResult` as `track`. +| `TestRunStatus` | Server status | `track` | +|---|---|---| +| `OK` | `ok` | passes | +| `APPROVED` | `approved` | passes | +| `AUTO_APPROVED` | `autoApproved` | passes | +| `FAILED` | `failed` | passes, as in the other SDKs | +| `NEW` | `new` (no baseline yet) | raises `TestRunError` | +| `UNRESOLVED` | `unresolved` (difference found) | raises `TestRunError` | +| `UNKNOWN` | any other | passes, logs a warning | -- `elementHandle: ElementHandle` [Playwright ElementHandle](https://playwright.dev/python/docs/api/class-elementhandle) +With `enableSoftAssert`, `track` logs `new` and `unresolved` as errors instead of raising. -- `imageName: str` name for the taken screenshot image +## Errors -- `options: ElementHandleTrackOptions` optional configuration with: - - `diffTollerancePercent: float` specify acceptable difference from baseline, between `0-100`. +All errors derive from `VisualRegressionTrackerError`. - - `ignoreAreas: List[IgnoreArea]` - - `x: int` X-coordinate relative of left upper corner - - `y: int` Y-coordinate relative of left upper corner - - `width: int` area width in px - - `height: int` area height in px - - - `screenshotOptions: ElementHandleScreenshotOptions` configuration for Playwrights `screenshot` method - - - `omit_background: bool` Hides default white background and allows capturing screenshots with transparency. Defaults to `false`. - - - `timeout: float` Maximum time in milliseconds, defaults to 30 seconds, pass 0 to disable timeout. - - - `agent: Agent` Additional information to mark baseline across agents that have different: - - `os: str` operating system name, like Windows, Mac, etc. - - `device: str` device name, PC identifier, mobile identifier etc. - - `viewport: str` viewport size. +| Error | When | +|---|---| +| `ServerError` | The server answered with an HTTP error. `status_code` and the server's `message`; 413 and 422 add a hint about the server's size limits. | +| `RequestError` | No answer: connection refused, connect timeout, or read timeout after `httpTimeoutInSeconds`. Has the `url`. A read timeout on `track` doesn't mean the server dropped the test run: it may still be stored. | +| `TestRunError` | `track` got `new` or `unresolved`. Has the `status`. | +| `MissingConfigurationError` | A required option is not set. Has the `field`. | +| `InvalidConfigurationError` | An option has an invalid value. Has the `field`. | +| `VisualRegressionTrackerError` | `track` or `stop` before `start`. | ## Development @@ -287,6 +167,8 @@ python -m playwright install chromium python -m pytest ``` +`tox` runs the tests on every supported Python. + ## Releasing Merging to `master` releases. [semantic-release](https://semantic-release.gitbook.io/) reads the @@ -301,3 +183,10 @@ publishes the GitHub release with the generated notes. The package version comes If publishing fails, the tag and a draft release already exist: re-run the failed jobs of that run (not the whole run) before anything else is merged. + +## Links + +- [Visual Regression Tracker](https://github.com/Visual-Regression-Tracker/Visual-Regression-Tracker): the server and how to run it +- [Python examples](https://github.com/Visual-Regression-Tracker/examples-python) +- [Other SDKs and agents](https://github.com/Visual-Regression-Tracker) +- [Licence: Apache-2.0](LICENSE) diff --git a/requirements.txt b/requirements.txt index fc898b9..ef99718 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,5 +1,5 @@ -pytest >= 9.2 -pytest-cov >= 7.1 -pytest-mock >= 3.17 -pytest-asyncio >= 1.4 -pylint >= 4.1 +pytest==9.2.0 +pytest-cov==7.1.0 +pytest-mock==3.17.0 +pytest-asyncio==1.4.1.post1 +pylint==4.1.3 diff --git a/tox.ini b/tox.ini index 94016ad..006b983 100644 --- a/tox.ini +++ b/tox.ini @@ -14,7 +14,6 @@ commands = --doctest-modules \ --junitxml=junit/test-results.xml \ --cov=visual_regression_tracker \ - --cov-report=xml \ --cov-report=html \ {posargs:tests} extras = playwright