diff --git a/docs/skills/test-authoring/gnome/SKILL.md b/docs/skills/test-authoring/gnome/SKILL.md index eeeae1e60..036802e61 100644 --- a/docs/skills/test-authoring/gnome/SKILL.md +++ b/docs/skills/test-authoring/gnome/SKILL.md @@ -49,13 +49,13 @@ metadata: ## Stack -| Layer | Component | Install | -|---|---|---| -| BDD runner | behave | pip | -| Session bridge | qecore-headless | pip | -| GUI automation | dogtail (AT-SPI) | pip | -| Wayland coord bridge | gnome-ponytail-daemon | `sudo dnf install gnome-ponytail-daemon` inside VM | -| Shell bridge | `org.gnome.Shell.Eval` | built-in (requires `unsafe_mode=true`) | +| Layer | Component | Install | +| -------------------- | ---------------------- | -------------------------------------------------- | +| BDD runner | behave | pip | +| Session bridge | qecore-headless | pip | +| GUI automation | dogtail (AT-SPI) | pip | +| Wayland coord bridge | gnome-ponytail-daemon | `sudo dnf install gnome-ponytail-daemon` inside VM | +| Shell bridge | `org.gnome.Shell.Eval` | built-in (requires `unsafe_mode=true`) | ## Screen Lock/Unlock D-Bus calls (GNOME 50) @@ -91,9 +91,18 @@ test manual session locking via `loginctl lock-session` / `loginctl unlock-sessi and logind `LockedHint`, which remain fully operational even when automatic idle lock is disabled. -## GNOME 51 workarounds audit (issue #827) +## GNOME 51 (issue #827) -GNOME 51 (GA 2026-09-16) is Wayland-only and freezes the extension API. Keep every GNOME 50 workaround until a `gnomeos-51` run proves it obsolete — inventory, ground rules and validation procedure in [`references/gnome-51-audit.md`](references/gnome-51-audit.md). +`dakota:testing` and `utah:testing` run GNOME Shell 51.1 (qecore 4.19.3 works). +Each GNOME 50 workaround's GNOME 51 verdict, with probe evidence, is in +[`references/gnome-51-audit.md`](references/gnome-51-audit.md). Still needed on +51: the Files AT-SPI name `org.gnome.Nautilus`, and Ptyxis's `Terminal` title +where Ptyxis ships. The rest are obsolete on 51; keep their GNOME 50 paths while +any tested image runs GNOME 50. Notification dismissal checks that the test +notification is gone, not that no banner shows: Shell holds earlier banners on +screen while the session is idle, as a headless VM always is. Calculator and +Text Editor divergences on plain GNOME OS 51: +[`references/gnome-51-app-divergences.md`](references/gnome-51-app-divergences.md). ## Remote session commands from the runner container @@ -474,15 +483,6 @@ helper; key contract points: Reuse this helper rather than a new `gdbus`-poll loop. See `docs/skills/ci-ops/ops/references/qecore-headless-restarts-gdm-bus-socket-churn.md`. -## GNOME 51 workarounds audit - -`dakota:testing` runs GNOME 51.1; its e2e smoke runs exercise workarounds on -GNOME 51. Keep GNOME 50 workarounds while any tested image still needs them. -See `references/gnome-51-audit.md` for run evidence and per-workaround verdicts. -Calculator display lookup and Text Editor new-document/quit diverged on plain -GNOME OS 51. When the PR validation plan selects VM-covered suites, the check -runs `dakota:testing`; it cannot capture AT-SPI trees for apps that image lacks. - ## On-demand references Load these when you hit the specific topic: @@ -491,7 +491,8 @@ Load these when you hit the specific topic: - [Top-bar interactions and Shell.Eval parsing on GNOME 50+.](references/top-bar.md) - [MIME, display, and session configuration in containerized tests.](references/display-config.md) - [Deep dive: Preinstalled Flatpak desktop app launch checks](references/preinstalled-flatpak-desktop-app-launch-checks.md) -- [GNOME 50 → 51 workaround audit, live run evidence, and the Calculator / Text Editor divergences.](references/gnome-51-audit.md) +- [GNOME 50 → 51 workaround audit and its targeted probe evidence.](references/gnome-51-audit.md) +- [Calculator and Text Editor divergences on plain GNOME OS 51.](references/gnome-51-app-divergences.md) ## Overview, clone paint, and helper fault injection (issue #919) diff --git a/docs/skills/test-authoring/gnome/references/gnome-51-app-divergences.md b/docs/skills/test-authoring/gnome/references/gnome-51-app-divergences.md new file mode 100644 index 000000000..e74c2f438 --- /dev/null +++ b/docs/skills/test-authoring/gnome/references/gnome-51-app-divergences.md @@ -0,0 +1,55 @@ +--- +name: gnome-51-app-divergences +description: "Calculator display lookup and Text Editor new-document/quit failures found in the plain gnomeos-51 run; documented, not fixed. Not among the twelve audited GNOME 50 workarounds." +metadata: + type: reference + audience: agents + maturity: stable +--- +# GNOME 51 app divergences: Calculator and Text Editor + +These are **not** among the twelve workarounds above. In the plain `gnomeos-51` +run ([gnome-51-audit.md](gnome-51-audit.md) §Run evidence), Calculator and Text Editor scenarios diverged. In the `dakota:testing` +v1 gate run (same section), both apps were absent and their scenarios skipped; that run +does not show them passing or failing. When the PR validation plan selects +VM-covered suites, the check runs `dakota:testing`; it cannot capture AT-SPI +trees for apps that image does not ship. +The findings below are **documented here, not fixed**. + +### Calculator display lookup — broken on 51 + +- **Failing:** `basic_multiplication` ("display shows 42"), + `subtracting_to_a_negative_result` ("display shows -5"). +- **Passing (decoys):** `basic_addition` (3), `basic_division` (3), + `basic_subtraction` (5), `clear_button_resets_display` (0), + `decimal_arithmetic` (3), `calculator_launches`. +- **Root cause:** `click_calculator_button` / `Calculator display shows` in + `tests/smoke/features/steps/gnome_calculator_steps.py` resolve the display via + `_display_candidates()`, which walks the window for nodes with role in + `{text, entry, label, static}` and a non-empty value. On GNOME 51 the display + node is **not** located, so the step falls back to the keypad **buttons** + (7, 8, 9, `×`, `÷`, `−`, `+`, `=`, …). A single positive digit that happens to + equal a button label (`3`, `5`, `0`) matches by accident and passes; `42` and + `-5` match no button and fail. The screenshot + `screenshot_smoke_failed_basic_multiplication.png` shows the calculator with + no display value matching. +- **What the run proves:** the step did not locate the display node on this + `gnomeos-51` image. Capture the app's GNOME 51 AT-SPI tree to identify the + real display node or distinguish it from the keypad buttons. +- **Also failing:** `calculator_closes_cleanly_via_ctrl_q` — `Ctrl+Q` did not + quit (see Text Editor below; shared key-combo path). + +### Text Editor new-document / quit — broken on 51 + +- **Failing:** `text_editor_creates_new_empty_document_buffer` (`Ctrl+N`), + `text_editor_save_dialog_is_open` (`Ctrl+S`), `text_editor_discard_dialog_is_open` + (`Ctrl+Shift+S`), and the Text Editor coredump/quited scenarios. +- **Passing:** `text_editor` `@input` ("Type text") — plain typing works. +- **Root cause:** after `Ctrl+N` the buffer still shows the previous scenario's + content (`"Bluefin smoke testPrevious document content"`) instead of a new + empty buffer. `Ctrl+N` is not creating a new document on GNOME 51. Plain + character entry via uinput works, so the input layer is fine — the failure is + specific to the `Key combo` path (`Ctrl+N`/`Ctrl+S`/`Ctrl+Shift+S`). +- **What the run shows:** the `Key combo` step did not drive the expected + accelerator for Text Editor on this `gnomeos-51` image. A run with `qecore` + debug logging is needed to establish whether the combo is dropped or mis-routed. diff --git a/docs/skills/test-authoring/gnome/references/gnome-51-audit.md b/docs/skills/test-authoring/gnome/references/gnome-51-audit.md index 04279931c..6c5b14512 100644 --- a/docs/skills/test-authoring/gnome/references/gnome-51-audit.md +++ b/docs/skills/test-authoring/gnome/references/gnome-51-audit.md @@ -1,6 +1,6 @@ --- name: gnome-51-audit -description: "Audit of GNOME 50-specific workarounds against GNOME 51 — per-workaround verdict, the live gnomeos-51 run evidence, and the Calculator / Text Editor divergences found in that run." +description: "Audit of GNOME 50-specific workarounds against GNOME 51 — per-workaround verdict, the live gnomeos-51 run evidence, the targeted probes behind each verdict, and qecore on GNOME 51." metadata: type: reference audience: agents @@ -36,28 +36,26 @@ Version-branch or fix a workaround only after a run proves behavior diverges. ## The twelve GNOME 50 workarounds -| # | Workaround | Status (GNOME 51) | Where | -|---|---|---|---| -| 1 | Re-prepend `global.context.unsafe_mode = true` before every `Shell.Eval` (GNOME 50 reset it) | **OBSOLETE ON 51** — §Targeted probes (round 2): `global.context.unsafe_mode`, read by a raw `Shell.Eval` that does not re-enable it, stayed `true` after a 3 s idle, an overview open and close over D-Bus, a notification, a GNOME Text Editor window opening and closing, and both screen-lock paths with their unlocks, on `dakota:testing` and `utah:testing`. GNOME 50's reset did not happen. Keep the prepend while any tested image runs GNOME 50; it is harmless. Modal dialogs other than the lock screen were not exercised | `tests/smoke/features/steps/steps.py`; documented in `SKILL.md` "Screen Lock/Unlock" / "Overview search entry" | -| 2 | Double-quoted `Shell.Eval` results `(true, '"true"')` | **OBSOLETE ON 51 as a version quirk** — §Targeted probes (round 2): `Shell.Eval` returns the result JSON-encoded on both images: `true` → `(true, 'true')`, `'abc'` → `(true, '"abc"')`, objects and arrays as JSON, `undefined` → `''`. A quoted `'"true"'` only appears when the expression evaluates to the string `"true"`. Keep `_eval_bool`'s optional quotes for string results | `tests/smoke/features/steps/steps.py` | -| 3 | Extension `state=6` via `Shell.Eval` → use `GetExtensionInfo` D-Bus | **OBSOLETE ON 51** — §Targeted probes (round 2): for the 12 extensions sampled on each image (all `ACTIVE`), `Main.extensionManager.lookup(uuid).state` via `Shell.Eval` returned `1`, matching `GetExtensionInfo` (`1`) and `gnome-extensions show` (`ACTIVE`); GNOME 50's constant `6` did not occur. Keep `GetExtensionInfo`: it is the public API and needs no unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Extension state via D-Bus" | -| 4 | `filler`/`frame` toplevel-role fallback + require populated subtree | **WORKS ON 51** — Files, Calculator, Text Editor, and fractional/integer scale windows resolved their toplevels; the run does not prove which fallback was needed | `tests/shared/gnome_shell_steps.py`; `SKILL.md` "Window-role checks must prove the subtree is populated" | -| 5 | Nautilus sidebar role (`home_folder_is_in_the_sidebar`) | **WORKS ON 51** — `home_folder_is_in_the_sidebar` passed; necessity remains unproven | `tests/smoke/features/steps/gnome_files_steps.py` | -| 6 | Nautilus AT-SPI app-name handling (`FILES_APP_NAMES`, `_nautilus_app`) | **WORKS ON 51** — Files launched, navigated, and closed; necessity remains unproven | `tests/smoke/features/steps/gnome_files_steps.py` | -| 7 | Ptyxis window title (`user@host:~`) | **STILL REQUIRED on `utah:testing`; not testable on `dakota:testing`** — §Targeted probes (round 2): on utah (`/usr/bin/ptyxis`) a new window was titled `Terminal` (WM class `org.gnome.Ptyxis`) and its AT-SPI frame was named `Terminal`, so a check for a `Ptyxis` title would fail. `dakota:testing` ships no Ptyxis (its terminal is Ghostty), which is why smoke-a skipped the scenario | `tests/smoke/features/steps/gnome_apps_steps.py:34-38` | -| 8 | ScreenSaver D-Bus `Lock` (not `screenShield.lock(true)`) | **OBSOLETE ON 51** — §Targeted probes (round 2): `Main.screenShield.lock(true)` via `Shell.Eval` locked the session on both images (`org.gnome.ScreenSaver.GetActive` → `true`), and `SetActive false` unlocked it; `org.gnome.ScreenSaver.Lock` behaved the same. GNOME 50's "deprecated and fails" no longer holds. Keep the D-Bus `Lock`: it is the public API and needs no unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Screen Lock/Unlock D-Bus calls" | -| 9 | `Main.overview.visible` reads `false` in QEMU; do not assert it | **OBSOLETE ON 51** — the targeted probe (§Targeted probes) read `Main.overview.visible` as `false`, `true`, `false` across an overview open and close in QEMU on GNOME Shell 51.1, matching `OverviewActive` each time, on `dakota:testing` and `utah:testing`. Keep checking `OverviewActive`: it needs no `Shell.Eval` or unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Activities overview (GNOME 50 QEMU)" | -| 10 | Notification banner dismissal check | **WORKS ON 51** — `toast_notification_banner_can_be_dismissed` passed; necessity remains unproven | `tests/smoke/features/steps/gnome_notifications_steps.py` | -| 11 | Panel buttons `showing=False` before overview toggle | **OBSOLETE ON 51** — the first probe read the Activities toggle as `showing=True`; §Targeted probes (round 2) found the clock (a `menu` with no accessible name, holding the date and time label) and the `System` menu in the same top-bar panel, and both read `showing=True` before, during and after an overview toggle on both images. GNOME 50's "every panel button reports `showing=False`" holds for none of the three | `tests/shared/gnome_shell_steps.py` | -| 12 | keyring skip on GDM restart | **WORKS ON 51** — §Targeted probes (round 2): with qecore's keyring setup skipped, every smoke-b scenario passed on both images. The login keyring is unusable without it: on dakota the `login` collection exists but is locked; on utah `gnome-keyring-daemon` runs with no `login` collection (default alias `/`); a `secret-tool store` blocked until a 15 s timeout on both. No smoke scenario stores a secret. Whether qecore's setup, which restarts GDM, would still break the run is not measured: it would restart the session under test | `tests/smoke/features/environment.py` | - -Rows 4, 5, 6, and 10 worked in the run in §Run evidence; a passing scenario -does not prove that the workaround is required. Rows 1, 2, 3, 7, 8, 9, 11 and -12 were measured by targeted probes (§Targeted probes and §Targeted probes -(round 2)). Row 7 is still required where Ptyxis ships (`utah:testing`); -`dakota:testing` has no Ptyxis. **This audit is incomplete**: issue #827 still -needs targeted comparisons for rows 4, 5, 6 and 10, which passed but have not -been run without their workarounds. +| # | Workaround | Status (GNOME 51) | Where | +| --- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | +| 1 | Re-prepend `global.context.unsafe_mode = true` before every `Shell.Eval` (GNOME 50 reset it) | **OBSOLETE ON 51** — §Targeted probes (round 2): `global.context.unsafe_mode`, read by a raw `Shell.Eval` that does not re-enable it, stayed `true` after a 3 s idle, an overview open and close over D-Bus, a notification, a GNOME Text Editor window opening and closing, and both screen-lock paths with their unlocks, on `dakota:testing` and `utah:testing`. GNOME 50's reset did not happen. Keep the prepend while any tested image runs GNOME 50; it is harmless. Modal dialogs other than the lock screen were not exercised | `tests/smoke/features/steps/steps.py`; documented in `SKILL.md` "Screen Lock/Unlock" / "Overview search entry" | +| 2 | Double-quoted `Shell.Eval` results `(true, '"true"')` | **OBSOLETE ON 51 as a version quirk** — §Targeted probes (round 2): `Shell.Eval` returns the result JSON-encoded on both images: `true` → `(true, 'true')`, `'abc'` → `(true, '"abc"')`, objects and arrays as JSON, `undefined` → `''`. A quoted `'"true"'` only appears when the expression evaluates to the string `"true"`. Keep `_eval_bool`'s optional quotes for string results | `tests/smoke/features/steps/steps.py` | +| 3 | Extension `state=6` via `Shell.Eval` → use `GetExtensionInfo` D-Bus | **OBSOLETE ON 51** — §Targeted probes (round 2): for the 12 extensions sampled on each image (all `ACTIVE`), `Main.extensionManager.lookup(uuid).state` via `Shell.Eval` returned `1`, matching `GetExtensionInfo` (`1`) and `gnome-extensions show` (`ACTIVE`); GNOME 50's constant `6` did not occur. Keep `GetExtensionInfo`: it is the public API and needs no unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Extension state via D-Bus" | +| 4 | `filler`/`frame` toplevel-role fallback + require populated subtree | **OBSOLETE ON 51 for every app measured** — §Targeted probes (round 3): on both images the toplevel of Files (`org.gnome.Nautilus`, 172 descendants) and Settings (`gnome-control-center`, 150), and on utah Firefox (29 children), was a populated `frame`; no `filler` toplevel and no bare childless `filler` occurred. Calculator, Text Editor and Clocks ship on neither image, so were not measured. Keep `{frame, filler}` and the populated-subtree check while any tested image runs GNOME 50 | `tests/shared/gnome_shell_steps.py`; `SKILL.md` "Window-role checks must prove the subtree is populated" | +| 5 | Nautilus sidebar role (`home_folder_is_in_the_sidebar`) | **OBSOLETE ON 51** — §Targeted probes (round 3): replaying the step's own search, its only match on both images was a `list item` named `Open Personal Folder` in the unnamed sidebar `list`: the pre-GNOME 50 role. GNOME 50's `button` did not occur in the sidebar (the `Home` `button` is the path bar's). The `Personal Folder` name match is what finds it. Keep accepting `button` while any tested image runs GNOME 50 | `tests/smoke/features/steps/gnome_files_steps.py` | +| 6 | Nautilus AT-SPI app-name handling (`FILES_APP_NAMES`, `_nautilus_app`) | **STILL REQUIRED** — §Targeted probes (round 3): Files registers in AT-SPI as `org.gnome.Nautilus` on both images, not the pre-GNOME 50 `nautilus`, so a lookup by `nautilus` alone finds nothing | `tests/smoke/features/steps/gnome_files_steps.py` | +| 7 | Ptyxis window title (`user@host:~`) | **STILL REQUIRED on `utah:testing`; not testable on `dakota:testing`** — §Targeted probes (round 2): on utah (`/usr/bin/ptyxis`) a new window was titled `Terminal` (WM class `org.gnome.Ptyxis`) and its AT-SPI frame was named `Terminal`, so a check for a `Ptyxis` title would fail. `dakota:testing` ships no Ptyxis (its terminal is Ghostty), which is why smoke-a skipped the scenario | `tests/smoke/features/steps/gnome_apps_steps.py:34-38` | +| 8 | ScreenSaver D-Bus `Lock` (not `screenShield.lock(true)`) | **OBSOLETE ON 51** — §Targeted probes (round 2): `Main.screenShield.lock(true)` via `Shell.Eval` locked the session on both images (`org.gnome.ScreenSaver.GetActive` → `true`), and `SetActive false` unlocked it; `org.gnome.ScreenSaver.Lock` behaved the same. GNOME 50's "deprecated and fails" no longer holds. Keep the D-Bus `Lock`: it is the public API and needs no unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Screen Lock/Unlock D-Bus calls" | +| 9 | `Main.overview.visible` reads `false` in QEMU; do not assert it | **OBSOLETE ON 51** — the targeted probe (§Targeted probes) read `Main.overview.visible` as `false`, `true`, `false` across an overview open and close in QEMU on GNOME Shell 51.1, matching `OverviewActive` each time, on `dakota:testing` and `utah:testing`. Keep checking `OverviewActive`: it needs no `Shell.Eval` or unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Activities overview (GNOME 50 QEMU)" | +| 10 | Notification banner dismissal check | **BROKEN ON 51 — fixed** — §Targeted probes (round 3): the GNOME 50 check also required `Main.messageTray._bannerBin` hidden, which GNOME 51 never does, and treated any visible banner as ours, so it always fell through to a force-dismiss and a warning and could not fail. After `CloseNotification` this scenario's notification leaves the tray; the banner still on screen is an earlier one (other scenarios and retries send without closing) that Shell holds while the session is idle. Each sent notification now gets a unique title; the close step first requires Shell to hold it, and the check then requires it to be neither the visible banner nor in the tray (a tray read error never counts as gone). It passes on both images ([43556476178](https://github.com/projectbluefin/testsuite/actions/runs/43556476178), [43556485608](https://github.com/projectbluefin/testsuite/actions/runs/43556485608)) | `tests/smoke/features/steps/gnome_notifications_steps.py` | +| 11 | Panel buttons `showing=False` before overview toggle | **OBSOLETE ON 51** — the first probe read the Activities toggle as `showing=True`; §Targeted probes (round 2) found the clock (a `menu` with no accessible name, holding the date and time label) and the `System` menu in the same top-bar panel, and both read `showing=True` before, during and after an overview toggle on both images. GNOME 50's "every panel button reports `showing=False`" holds for none of the three | `tests/shared/gnome_shell_steps.py` | +| 12 | keyring skip on GDM restart | **WORKS ON 51** — §Targeted probes (round 2): with qecore's keyring setup skipped, every smoke-b scenario passed on both images. The login keyring is unusable without it: on dakota the `login` collection exists but is locked; on utah `gnome-keyring-daemon` runs with no `login` collection (default alias `/`); a `secret-tool store` blocked until a 15 s timeout on both. No smoke scenario stores a secret. Whether qecore's setup, which restarts GDM, would still break the run is not measured: it would restart the session under test | `tests/smoke/features/environment.py` | + +Every row now has a GNOME 51 verdict from a targeted probe (§Targeted probes, +rounds 1–3). Still required on 51: row 6, and row 7 where Ptyxis ships +(`utah:testing`). Row 10 was broken and is fixed. The rest are obsolete on 51 +or, for row 12, work without the workaround; keep their GNOME 50 paths while +any tested image runs GNOME 50. ## Run target @@ -139,53 +137,43 @@ What each reading did: Every probe scenario passed. Both utah runs failed only at the SELinux denial gate, on utah#1198's `chcon` `mac_admin` denials. +### Targeted probes (round 3) + +A third temporary smoke scenario (commits `ea968538` to `e4aa74e5`, removed +before merge) read, in `smoke-b` on GNOME Shell 51.1 with qecore 4.19.3: each +app's AT-SPI toplevels (row 4), the sidebar matches of the step's own search +(row 5), the application names (row 6), and for row 10 the message tray, the +visible banner's notification and the session idle time around +`CloseNotification`, before and after one uinput keypress. + +| Image | Run | +| ---------------- | ----------------------------------------------------------------------------------- | +| `dakota:testing` | [43549588532](https://github.com/projectbluefin/testsuite/actions/runs/43549588532) | +| `utah:testing` | [43549598764](https://github.com/projectbluefin/testsuite/actions/runs/43549598764) | + +Row 10 in numbers: the visible banner was never this scenario's notification +but an earlier one (`Settings`, or another test notification left open); this +scenario's notification left the tray within 1 s of `CloseNotification`; the +session had been idle 34–282 s; one `` reset the idle time and the held +banner cleared within 2.3 s. The same hold showed with the close sent at once or +after 1 s (runs 43540402711, 43544509681, 43546740107, 43554149449, +43554157456). The fixed step passed in smoke-a on both images (runs above). + +### qecore on GNOME 51 + +Every GNOME 51 run above used qecore 4.19.3 (the pin in +`container/Containerfile.runner`, `container/Containerfile.gnomeos-harness` +and the gnome-e2e action), with dogtail 2.0.3 and behave 1.3.3, read from the +test process on both images. qecore-headless and its common steps work there: +smoke-a, smoke-b and the GNOME OS canary pass. qecore's releases since (4.20 to +4.22.1) name no GNOME 51 change; 4.22 adds handling for a missing keyring, +which matches row 12 on utah. + ## Run findings — Calculator and Text Editor divergences -These are **not** among the twelve workarounds above. In the plain `gnomeos-51` -run above, Calculator and Text Editor scenarios diverged. In the `dakota:testing` -v1 gate run above, both apps were absent and their scenarios skipped; that run -does not show them passing or failing. When the PR validation plan selects -VM-covered suites, the check runs `dakota:testing`; it cannot capture AT-SPI -trees for apps that image does not ship. -The findings below are **documented here, not fixed**. - -### Calculator display lookup — broken on 51 - -- **Failing:** `basic_multiplication` ("display shows 42"), - `subtracting_to_a_negative_result` ("display shows -5"). -- **Passing (decoys):** `basic_addition` (3), `basic_division` (3), - `basic_subtraction` (5), `clear_button_resets_display` (0), - `decimal_arithmetic` (3), `calculator_launches`. -- **Root cause:** `click_calculator_button` / `Calculator display shows` in - `tests/smoke/features/steps/gnome_calculator_steps.py` resolve the display via - `_display_candidates()`, which walks the window for nodes with role in - `{text, entry, label, static}` and a non-empty value. On GNOME 51 the display - node is **not** located, so the step falls back to the keypad **buttons** - (7, 8, 9, `×`, `÷`, `−`, `+`, `=`, …). A single positive digit that happens to - equal a button label (`3`, `5`, `0`) matches by accident and passes; `42` and - `-5` match no button and fail. The screenshot - `screenshot_smoke_failed_basic_multiplication.png` shows the calculator with - no display value matching. -- **What the run proves:** the step did not locate the display node on this - `gnomeos-51` image. Capture the app's GNOME 51 AT-SPI tree to identify the - real display node or distinguish it from the keypad buttons. -- **Also failing:** `calculator_closes_cleanly_via_ctrl_q` — `Ctrl+Q` did not - quit (see Text Editor below; shared key-combo path). - -### Text Editor new-document / quit — broken on 51 - -- **Failing:** `text_editor_creates_new_empty_document_buffer` (`Ctrl+N`), - `text_editor_save_dialog_is_open` (`Ctrl+S`), `text_editor_discard_dialog_is_open` - (`Ctrl+Shift+S`), and the Text Editor coredump/quited scenarios. -- **Passing:** `text_editor` `@input` ("Type text") — plain typing works. -- **Root cause:** after `Ctrl+N` the buffer still shows the previous scenario's - content (`"Bluefin smoke testPrevious document content"`) instead of a new - empty buffer. `Ctrl+N` is not creating a new document on GNOME 51. Plain - character entry via uinput works, so the input layer is fine — the failure is - specific to the `Key combo` path (`Ctrl+N`/`Ctrl+S`/`Ctrl+Shift+S`). -- **What the run shows:** the `Key combo` step did not drive the expected - accelerator for Text Editor on this `gnomeos-51` image. A run with `qecore` - debug logging is needed to establish whether the combo is dropped or mis-routed. +These are not among the twelve workarounds. The plain `gnomeos-51` run's +Calculator display lookup and Text Editor `Ctrl+N`/`Ctrl+Q` failures are in +[gnome-51-app-divergences.md](gnome-51-app-divergences.md) (documented, not fixed). ## Bluefin-only failures (expected on gnomeos-51 — not divergences) diff --git a/tests/smoke/features/steps/gnome_notifications_steps.py b/tests/smoke/features/steps/gnome_notifications_steps.py index 6fcb66e50..4cd229f6e 100644 --- a/tests/smoke/features/steps/gnome_notifications_steps.py +++ b/tests/smoke/features/steps/gnome_notifications_steps.py @@ -1,11 +1,20 @@ """Custom step definitions for GNOME notification smoke tests.""" +import json import os import re import subprocess +import uuid from time import sleep from behave import step from tests.shared.ssh_config import ssh_argv + +# Summary prefix of the notifications the smoke scenarios send. Each send adds +# a unique suffix: scenarios that send without closing (and retries) leave +# same-titled notifications in the tray, so a bare title cannot say whether +# this scenario's notification was dismissed. +TEST_NOTIFICATION_TITLE = "Test Notification" + try: from qecore.common_steps import * # noqa: F401,F403 except Exception: # noqa: BLE001 @@ -45,12 +54,14 @@ def _parse_notification_id(output: str) -> int: @step("A test desktop notification is sent via gdbus") def send_test_desktop_notification(context) -> None: + title = f"{TEST_NOTIFICATION_TITLE} {uuid.uuid4().hex[:8]}" + context.test_notification_title = title output, returncode, stderr = _run( "gdbus call --session " "--dest org.freedesktop.Notifications " "--object-path /org/freedesktop/Notifications " "--method org.freedesktop.Notifications.Notify " - "'' 0 '' 'Test Notification' 'Testsuite notification body' '[]' '{}' 3000" + f"'' 0 '' '{title}' 'Testsuite notification body' '[]' '{{}}' 3000" ) if returncode != 0: combined = (stderr + output).lower() @@ -81,11 +92,28 @@ def notification_request_returns_positive_id(context) -> None: @step("Dismiss the notification via gdbus CloseNotification") -def dismiss_notification_via_gdbus(context) -> None: +def dismiss_notification_via_gdbus(context, run=subprocess.run, pause=sleep) -> None: notification_id = getattr(context, "notification_id", None) assert notification_id is not None, ( "No notification ID captured — run 'A test desktop notification is sent via gdbus' first" ) + # The dismissal check can only prove CloseNotification removed the + # notification if Shell was seen holding it first; otherwise a tray API + # change (or a notification that never arrived) would read as "gone". + title = _sent_title(context) + state = {} + for _ in range(NOTIFICATION_SEEN_TIMEOUT_S * 2): + state = _tray_state(title, run) + if not state.get("error") and (state.get("queued") or state.get("shown")): + break + pause(0.5) + else: + raise AssertionError( + f"'{title}' was never seen in the message tray or as the banner within " + f"{NOTIFICATION_SEEN_TIMEOUT_S}s of sending, so its dismissal cannot be " + f"checked; last tray state: {state}" + ) + context.test_notification_seen = True _, returncode, stderr = _run( f"gdbus call --session " f"--dest org.freedesktop.Notifications " @@ -96,78 +124,108 @@ def dismiss_notification_via_gdbus(context) -> None: assert returncode == 0, f"CloseNotification failed for id={notification_id}: {stderr}" -@step("Notification banner is no longer showing via Shell.Eval") -def notification_banner_no_longer_showing(context) -> None: - import re - # Wait up to 20 s for the banner to be dismissed from the message tray. - # The banner ref becomes null once Shell finishes the dismiss animation. - # GNOME 50 changed the banner API — check both _banner property and banner visibility. - # Use a compound expression that returns 'false' when no banner is showing. - banner_js = ( - "((Main.messageTray._banner === null || " - "Main.messageTray._banner === undefined || " - "!Main.messageTray._banner.visible) && " - "(typeof Main.messageTray._bannerBin === 'undefined' || " - "Main.messageTray._bannerBin === null || " - "!Main.messageTray._bannerBin.visible)).toString()" +# True once the test notification is gone: not the visible banner and not in +# the message tray. Do not test "no banner at all": GNOME Shell holds a banner +# on screen while the session is idle, and a headless VM is always idle, so an +# unrelated earlier banner ("Settings") stayed up for minutes after our +# notification was already closed. The GNOME 50 version also required +# Main.messageTray._bannerBin hidden, which GNOME 51 never does with no banner +# in it, so it only ever "passed" through a force-dismiss and a warning, closed +# or not (#827 audit row 10, probe 3: our notification leaves the tray at once; +# the held banner clears ~2 s after any input). +def banner_gone_js(title: str) -> str: + """Shell.Eval JS: where is the notification titled ``title``? + + Returns a JSON object: ``shown`` (it is the visible banner), ``queued`` (it + is in the message tray) and ``error`` (the tray could not be read). An + error is never reported as gone. + """ + return ( + "(() => { const title = " + json.dumps(title) + ";" + " let shown = false, queued = false, error = null, banner = null, idle = null;" + " try { const t = Main.messageTray; const b = t._banner;" + " shown = !!(b && b.visible && b.notification && b.notification.title === title);" + " banner = b ? ((b.notification && b.notification.title) || '?') : null;" + " queued = t.getSources().some(s => s.notifications.some(n => n.title === title)); }" + " catch (e) { error = String(e); }" + " try { idle = global.backend.get_core_idle_monitor().get_idletime(); } catch (e) {}" + " return JSON.stringify({ shown: shown, queued: queued, error: error," + " banner: banner, idle_ms: idle }); })()" ) - last_out = "" - for _ in range(40): # 20s — GNOME 50 banners can linger longer in QEMU - result = subprocess.run( + + +BANNER_DISMISS_TIMEOUT_S = 20 # GNOME 50 banners can linger in QEMU +NOTIFICATION_SEEN_TIMEOUT_S = 5 + + +def _parse_eval_json(out: str): + """The object a JSON-returning Shell.Eval call produced, or None. + + Shell.Eval answers ``(true, '""')``: a GVariant string holding + the JSON encoding of the string the expression returned. + """ + m = re.match(r"\(\s*(true|false)\s*,\s*'(.*)'\s*\)\s*$", out.strip(), re.S) + if not m or m.group(1) != "true": + return None + literal = m.group(2).replace("\\\\", "\x00").replace("\\'", "'").replace("\x00", "\\") + try: + value = json.loads(literal) + return json.loads(value) if isinstance(value, str) else None + except ValueError: + return None + + +def _tray_state(title: str, run=subprocess.run) -> dict: + """Shell's view of the notification titled ``title``; ``error`` set on any failure.""" + try: + result = run( ["gdbus", "call", "--session", "--dest", "org.gnome.Shell", "--object-path", "/org/gnome/Shell", "--method", "org.gnome.Shell.Eval", - banner_js], + banner_gone_js(title)], capture_output=True, text=True, timeout=5, ) - last_out = result.stdout - m = re.search(r",\s*'\"?(true|false)\"?'\s*\)", last_out, re.IGNORECASE) - if m and m.group(1).lower() == "true": - return - sleep(0.5) - - # Banner still showing after 20s — force-dismiss via Shell.Eval. - dismiss_js = ( - "try {" - " if (Main.messageTray._banner && Main.messageTray._banner.destroy) {" - " Main.messageTray._banner.destroy();" - " } else if (Main.messageTray._bannerBin) {" - " Main.messageTray._bannerBin.hide();" - " }" - "} catch(e) {}; 'dismissed'" + except (OSError, subprocess.SubprocessError) as exc: + return {"error": f"gdbus failed: {exc}"} + state = _parse_eval_json(result.stdout) + if not isinstance(state, dict): + return {"error": f"unusable Shell.Eval reply: {result.stdout!r}"} + return state + + +def _sent_title(context) -> str: + title = getattr(context, "test_notification_title", None) + assert title, ( + "No test notification was sent in this scenario — run 'A test desktop " + "notification is sent via gdbus' first" ) - subprocess.run( - ["gdbus", "call", "--session", - "--dest", "org.gnome.Shell", - "--object-path", "/org/gnome/Shell", - "--method", "org.gnome.Shell.Eval", - dismiss_js], - capture_output=True, text=True, timeout=5, - ) - sleep(1) - # Final check after explicit dismiss - result = subprocess.run( - ["gdbus", "call", "--session", - "--dest", "org.gnome.Shell", - "--object-path", "/org/gnome/Shell", - "--method", "org.gnome.Shell.Eval", - banner_js], - capture_output=True, text=True, timeout=5, + return title + + +def _banner_gone(title: str, run=subprocess.run) -> tuple[bool, dict]: + """(test notification gone?, tray state) for one read; an error is not gone.""" + state = _tray_state(title, run) + gone = not state.get("error") and not state.get("shown") and not state.get("queued") + return gone, state + + +@step("Notification banner is no longer showing via Shell.Eval") +def notification_banner_no_longer_showing(context, run=subprocess.run, pause=sleep) -> None: + title = _sent_title(context) + assert getattr(context, "test_notification_seen", False), ( + f"'{title}' was not seen before CloseNotification, so its dismissal cannot be " + "checked — run 'Dismiss the notification via gdbus CloseNotification' first" ) - last_out = result.stdout - m = re.search(r",\s*'\"?(true|false)\"?'\s*\)", last_out, re.IGNORECASE) - if m and m.group(1).lower() == "true": - return - # In headless GNOME 50 QEMU, the banner visibility JS check can return - # "false" (still visible) even after a successful CloseNotification call. - # This is a known environment limitation — not a functional regression. - # Treat as a warning rather than a hard failure to avoid blocking the suite. - print( - f"WARNING: Notification banner still showing after explicit dismiss " - f"(headless GNOME 50 QEMU limitation) — Shell.Eval returned: {last_out!r}. " - "Skipping hard assertion.", - flush=True, + state = {} + for _ in range(BANNER_DISMISS_TIMEOUT_S * 2): + gone, state = _banner_gone(title, run) + if gone: + return + pause(0.5) + raise AssertionError( + f"'{title}' is still shown or queued, or the tray could not be read, " + f"{BANNER_DISMISS_TIMEOUT_S}s after CloseNotification; last tray state: {state}" ) diff --git a/tests/unit/test_gnome_notifications_steps.py b/tests/unit/test_gnome_notifications_steps.py index 958bdc9ac..3ebbc99d7 100644 --- a/tests/unit/test_gnome_notifications_steps.py +++ b/tests/unit/test_gnome_notifications_steps.py @@ -46,3 +46,142 @@ def test_parse_notification_id_raises_on_unparseable_output(output): with pytest.raises(AssertionError, match="Could not parse"): module._parse_notification_id(output) + + + +class _Done: + def __init__(self, stdout): + self.stdout = stdout + + +def _reply(state: dict) -> str: + """A Shell.Eval reply carrying ``state`` the way GNOME Shell encodes it.""" + import json + + literal = json.dumps(json.dumps(state)).replace("\\", "\\\\").replace("'", "\\'") + return f"(true, '{literal}')\n" + + +def _runner(outputs): + calls = [] + + def run(cmd, **kwargs): + calls.append(cmd) + return _Done(outputs[min(len(calls) - 1, len(outputs) - 1)]) + + return run, calls + + +def _context(**attrs): + return type("C", (), dict({"test_notification_title": "Test Notification ab12cd34", + "test_notification_seen": True, "notification_id": 7}, **attrs))() + + +GONE = {"shown": False, "queued": False, "error": None} +SHOWN = {"shown": True, "queued": True, "error": None} +QUEUED = {"shown": False, "queued": True, "error": None} +BROKEN = {"shown": False, "queued": False, "error": "TypeError: t.getSources is not a function"} + + +def test_parse_reads_the_reply_shell_sends(): + module = _import_gnome_notifications_steps() + assert module._parse_eval_json(_reply(QUEUED)) == QUEUED + assert module._parse_eval_json("(false, '')") is None + assert module._parse_eval_json("(true, '\"not json\"')") is None + + +def test_banner_check_tracks_the_test_notification_not_any_banner(): + # GNOME Shell holds an unrelated banner on screen while the session is + # idle, and GNOME 51 keeps _bannerBin visible with no banner (#827 audit + # row 10), so the check is about this scenario's notification only. + module = _import_gnome_notifications_steps() + js = module.banner_gone_js("Test Notification ab12cd34") + assert "_bannerBin" not in js + assert 'const title = "Test Notification ab12cd34"' in js + assert "catch (e) { error = String(e); }" in js + + +def test_each_sent_notification_gets_a_unique_checked_title(monkeypatch): + # Other scenarios (and retries) leave same-titled notifications in the + # tray, so the dismissal check needs this send's own title. + module = _import_gnome_notifications_steps() + sent = [] + monkeypatch.setattr(module, "_run", lambda cmd: (sent.append(cmd) or "(uint32 7,)", 0, "")) + monkeypatch.setattr(module, "sleep", lambda s: None) + first, second = type("C", (), {})(), type("C", (), {})() + module.send_test_desktop_notification(first) + module.send_test_desktop_notification(second) + assert first.test_notification_title != second.test_notification_title + assert first.test_notification_title.startswith(module.TEST_NOTIFICATION_TITLE + " ") + assert f"'{first.test_notification_title}'" in sent[0] + + +def test_check_passes_once_the_notification_is_gone(): + module = _import_gnome_notifications_steps() + run, calls = _runner([_reply(SHOWN), _reply(GONE)]) + module.notification_banner_no_longer_showing(_context(), run=run, pause=lambda s: None) + assert len(calls) == 2 + + +def test_check_fails_instead_of_force_dismissing(): + module = _import_gnome_notifications_steps() + run, calls = _runner([_reply(SHOWN)]) + with pytest.raises(AssertionError, match="still shown or queued"): + module.notification_banner_no_longer_showing(_context(), run=run, pause=lambda s: None) + assert all("destroy()" not in " ".join(c) and "hide()" not in " ".join(c) for c in calls) + assert len(calls) == module.BANNER_DISMISS_TIMEOUT_S * 2 + + +def test_a_tray_read_error_is_never_gone(): + # The JS reports the error rather than shown/queued false: an API change in + # Shell must not read as "dismissed" (fresh-context review of #1209). + module = _import_gnome_notifications_steps() + run, _ = _runner([_reply(BROKEN)]) + with pytest.raises(AssertionError, match="getSources is not a function"): + module.notification_banner_no_longer_showing(_context(), run=run, pause=lambda s: None) + + +def test_a_failed_shell_eval_is_never_gone(): + module = _import_gnome_notifications_steps() + run, _ = _runner(["(false, '')\n"]) + with pytest.raises(AssertionError, match="unusable Shell.Eval reply"): + module.notification_banner_no_longer_showing(_context(), run=run, pause=lambda s: None) + + +def test_check_without_a_sent_notification_fails(): + # Nothing sends the bare prefix any more, so defaulting to it always passed. + module = _import_gnome_notifications_steps() + run, calls = _runner([_reply(GONE)]) + with pytest.raises(AssertionError, match="No test notification was sent"): + module.notification_banner_no_longer_showing(type("C", (), {})(), run=run, pause=lambda s: None) + assert calls == [] + + +def test_check_without_seeing_the_notification_first_fails(): + module = _import_gnome_notifications_steps() + run, _ = _runner([_reply(GONE)]) + with pytest.raises(AssertionError, match="not seen before CloseNotification"): + module.notification_banner_no_longer_showing( + _context(test_notification_seen=False), run=run, pause=lambda s: None + ) + + +def test_dismiss_closes_only_after_seeing_the_notification(monkeypatch): + module = _import_gnome_notifications_steps() + closed = [] + monkeypatch.setattr(module, "_run", lambda cmd: (closed.append(cmd) or "", 0, "")) + run, calls = _runner([_reply(GONE), _reply(QUEUED)]) + context = _context(test_notification_seen=False) + module.dismiss_notification_via_gdbus(context, run=run, pause=lambda s: None) + assert len(calls) == 2 and context.test_notification_seen is True + assert closed and "CloseNotification 7" in closed[0] + + +def test_dismiss_refuses_when_the_tray_cannot_be_read(monkeypatch): + module = _import_gnome_notifications_steps() + closed = [] + monkeypatch.setattr(module, "_run", lambda cmd: (closed.append(cmd) or "", 0, "")) + run, _ = _runner([_reply(BROKEN)]) + with pytest.raises(AssertionError, match="was never seen"): + module.dismiss_notification_via_gdbus(_context(test_notification_seen=False), run=run, pause=lambda s: None) + assert closed == []