From 993d7a5d716de490179e1f917da8956903354e3e Mon Sep 17 00:00:00 2001 From: Danathar Date: Sat, 14 Nov 2026 03:31:06 +0000 Subject: [PATCH 1/2] test(smoke): log which GNOME 50 workaround branch fires for audit rows 4, 5, 6 and 10 The GNOME 51 audit could not tell "still required" from "obsolete" for these four rows: their scenarios pass, but nothing recorded which branch of each workaround matched. Each step now prints one `gnome-51-audit row N: ` line, so every smoke run on a GNOME 51 image (the PR VM check boots dakota:testing, GNOME 51.1) carries the evidence without removing a path GNOME 50 images still use. Row 10 is reclassified LIKELY-KEEP: its banner step ends on a warning instead of an assertion, so a pass never proved anything. Refs #827 Hive-Run: projectbluefin/testsuite#827 Hive-Plan: gnome-51-audit remaining rows 4, 5, 6, 10 Hive-Spec: issue-827#tasks-validate-each-row Assisted-by: Claude Opus 5.5 via Claude Code Co-Authored-By: Claude Opus 5.5 Signed-off-by: Danathar --- docs/skills/test-authoring/gnome/SKILL.md | 2 + .../gnome/references/gnome-51-audit.md | 16 +++---- .../smoke/features/steps/gnome_files_steps.py | 27 +++++++++--- .../steps/gnome_notifications_steps.py | 5 ++- tests/unit/test_gnome_files_steps.py | 44 +++++++++++++++++++ tests/unit/test_gnome_notifications_steps.py | 38 ++++++++++++++++ 6 files changed, 118 insertions(+), 14 deletions(-) diff --git a/docs/skills/test-authoring/gnome/SKILL.md b/docs/skills/test-authoring/gnome/SKILL.md index eeeae1e60..056640ec2 100644 --- a/docs/skills/test-authoring/gnome/SKILL.md +++ b/docs/skills/test-authoring/gnome/SKILL.md @@ -95,6 +95,8 @@ lock is disabled. GNOME 51 (GA 2026-09-16) is Wayland-only and freezes the extension API. Keep every GNOME 50 workaround until a `gnomeos-51` run proves it obsolete — inventory, ground rules and validation procedure in [`references/gnome-51-audit.md`](references/gnome-51-audit.md). +Rows still being measured print a `gnome-51-audit row N: ` line from the step that holds the workaround. Keep the line when you edit that step; it is the evidence the audit reads from a smoke run's behave log. + ## Remote session commands from the runner container Commands that access the GNOME user session, including `gsettings`, `gdbus diff --git a/docs/skills/test-authoring/gnome/references/gnome-51-audit.md b/docs/skills/test-authoring/gnome/references/gnome-51-audit.md index 04279931c..ad938a03e 100644 --- a/docs/skills/test-authoring/gnome/references/gnome-51-audit.md +++ b/docs/skills/test-authoring/gnome/references/gnome-51-audit.md @@ -41,23 +41,23 @@ Version-branch or fix a workaround only after a run proves behavior diverges. | 1 | Re-prepend `global.context.unsafe_mode = true` before every `Shell.Eval` (GNOME 50 reset it) | **OBSOLETE ON 51** — §Targeted probes (round 2): `global.context.unsafe_mode`, read by a raw `Shell.Eval` that does not re-enable it, stayed `true` after a 3 s idle, an overview open and close over D-Bus, a notification, a GNOME Text Editor window opening and closing, and both screen-lock paths with their unlocks, on `dakota:testing` and `utah:testing`. GNOME 50's reset did not happen. Keep the prepend while any tested image runs GNOME 50; it is harmless. Modal dialogs other than the lock screen were not exercised | `tests/smoke/features/steps/steps.py`; documented in `SKILL.md` "Screen Lock/Unlock" / "Overview search entry" | | 2 | Double-quoted `Shell.Eval` results `(true, '"true"')` | **OBSOLETE ON 51 as a version quirk** — §Targeted probes (round 2): `Shell.Eval` returns the result JSON-encoded on both images: `true` → `(true, 'true')`, `'abc'` → `(true, '"abc"')`, objects and arrays as JSON, `undefined` → `''`. A quoted `'"true"'` only appears when the expression evaluates to the string `"true"`. Keep `_eval_bool`'s optional quotes for string results | `tests/smoke/features/steps/steps.py` | | 3 | Extension `state=6` via `Shell.Eval` → use `GetExtensionInfo` D-Bus | **OBSOLETE ON 51** — §Targeted probes (round 2): for the 12 extensions sampled on each image (all `ACTIVE`), `Main.extensionManager.lookup(uuid).state` via `Shell.Eval` returned `1`, matching `GetExtensionInfo` (`1`) and `gnome-extensions show` (`ACTIVE`); GNOME 50's constant `6` did not occur. Keep `GetExtensionInfo`: it is the public API and needs no unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Extension state via D-Bus" | -| 4 | `filler`/`frame` toplevel-role fallback + require populated subtree | **WORKS ON 51** — Files, Calculator, Text Editor, and fractional/integer scale windows resolved their toplevels; the run does not prove which fallback was needed | `tests/shared/gnome_shell_steps.py`; `SKILL.md` "Window-role checks must prove the subtree is populated" | -| 5 | Nautilus sidebar role (`home_folder_is_in_the_sidebar`) | **WORKS ON 51** — `home_folder_is_in_the_sidebar` passed; necessity remains unproven | `tests/smoke/features/steps/gnome_files_steps.py` | -| 6 | Nautilus AT-SPI app-name handling (`FILES_APP_NAMES`, `_nautilus_app`) | **WORKS ON 51** — Files launched, navigated, and closed; necessity remains unproven | `tests/smoke/features/steps/gnome_files_steps.py` | +| 4 | `filler`/`frame` toplevel-role fallback + require populated subtree | **WORKS ON 51** — Files, Calculator, Text Editor, and fractional/integer scale windows resolved their toplevels; the run does not prove which fallback was needed. Files logs its role as `gnome-51-audit row 4:` | `tests/shared/gnome_shell_steps.py`; `SKILL.md` "Window-role checks must prove the subtree is populated" | +| 5 | Nautilus sidebar role (`home_folder_is_in_the_sidebar`) | **WORKS ON 51** — `home_folder_is_in_the_sidebar` passed; the matched role is logged as `gnome-51-audit row 5:` | `tests/smoke/features/steps/gnome_files_steps.py` | +| 6 | Nautilus AT-SPI app-name handling (`FILES_APP_NAMES`, `_nautilus_app`) | **WORKS ON 51** — Files launched, navigated, and closed; the matched name is logged as `gnome-51-audit row 6:` | `tests/smoke/features/steps/gnome_files_steps.py` | | 7 | Ptyxis window title (`user@host:~`) | **STILL REQUIRED on `utah:testing`; not testable on `dakota:testing`** — §Targeted probes (round 2): on utah (`/usr/bin/ptyxis`) a new window was titled `Terminal` (WM class `org.gnome.Ptyxis`) and its AT-SPI frame was named `Terminal`, so a check for a `Ptyxis` title would fail. `dakota:testing` ships no Ptyxis (its terminal is Ghostty), which is why smoke-a skipped the scenario | `tests/smoke/features/steps/gnome_apps_steps.py:34-38` | | 8 | ScreenSaver D-Bus `Lock` (not `screenShield.lock(true)`) | **OBSOLETE ON 51** — §Targeted probes (round 2): `Main.screenShield.lock(true)` via `Shell.Eval` locked the session on both images (`org.gnome.ScreenSaver.GetActive` → `true`), and `SetActive false` unlocked it; `org.gnome.ScreenSaver.Lock` behaved the same. GNOME 50's "deprecated and fails" no longer holds. Keep the D-Bus `Lock`: it is the public API and needs no unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Screen Lock/Unlock D-Bus calls" | | 9 | `Main.overview.visible` reads `false` in QEMU; do not assert it | **OBSOLETE ON 51** — the targeted probe (§Targeted probes) read `Main.overview.visible` as `false`, `true`, `false` across an overview open and close in QEMU on GNOME Shell 51.1, matching `OverviewActive` each time, on `dakota:testing` and `utah:testing`. Keep checking `OverviewActive`: it needs no `Shell.Eval` or unsafe mode | `tests/smoke/features/steps/steps.py`; `SKILL.md` "Activities overview (GNOME 50 QEMU)" | -| 10 | Notification banner dismissal check | **WORKS ON 51** — `toast_notification_banner_can_be_dismissed` passed; necessity remains unproven | `tests/smoke/features/steps/gnome_notifications_steps.py` | +| 10 | Notification banner dismissal check | **LIKELY-KEEP** — `toast_notification_banner_can_be_dismissed` passed, but that is not evidence: its last step cannot fail; a banner still showing after the 20 s wait and a `Shell.Eval` force-dismiss only prints a warning. Which path ran is logged as `gnome-51-audit row 10:` | `tests/smoke/features/steps/gnome_notifications_steps.py` | | 11 | Panel buttons `showing=False` before overview toggle | **OBSOLETE ON 51** — the first probe read the Activities toggle as `showing=True`; §Targeted probes (round 2) found the clock (a `menu` with no accessible name, holding the date and time label) and the `System` menu in the same top-bar panel, and both read `showing=True` before, during and after an overview toggle on both images. GNOME 50's "every panel button reports `showing=False`" holds for none of the three | `tests/shared/gnome_shell_steps.py` | | 12 | keyring skip on GDM restart | **WORKS ON 51** — §Targeted probes (round 2): with qecore's keyring setup skipped, every smoke-b scenario passed on both images. The login keyring is unusable without it: on dakota the `login` collection exists but is locked; on utah `gnome-keyring-daemon` runs with no `login` collection (default alias `/`); a `secret-tool store` blocked until a 15 s timeout on both. No smoke scenario stores a secret. Whether qecore's setup, which restarts GDM, would still break the run is not measured: it would restart the session under test | `tests/smoke/features/environment.py` | -Rows 4, 5, 6, and 10 worked in the run in §Run evidence; a passing scenario +Rows 4, 5 and 6 worked in the run in §Run evidence (row 10 cannot fail); a passing scenario does not prove that the workaround is required. Rows 1, 2, 3, 7, 8, 9, 11 and 12 were measured by targeted probes (§Targeted probes and §Targeted probes (round 2)). Row 7 is still required where Ptyxis ships (`utah:testing`); -`dakota:testing` has no Ptyxis. **This audit is incomplete**: issue #827 still -needs targeted comparisons for rows 4, 5, 6 and 10, which passed but have not -been run without their workarounds. +`dakota:testing` has no Ptyxis. **This audit is incomplete**: rows 4, 5, 6 and +10 print a `gnome-51-audit row N:` line naming the branch that matched in every +smoke run (search the behave log); record a `dakota:testing` run's lines here. ## Run target diff --git a/tests/smoke/features/steps/gnome_files_steps.py b/tests/smoke/features/steps/gnome_files_steps.py index 13a8e6815..4e9e8efac 100644 --- a/tests/smoke/features/steps/gnome_files_steps.py +++ b/tests/smoke/features/steps/gnome_files_steps.py @@ -45,15 +45,29 @@ def _skip_if_no_atspi(context) -> bool: } +_AUDIT_READINGS: set = set() + + +def _audit_reading(row: int, reading: str) -> None: + """Print a GNOME 51 audit reading once per run; it records which branch matched.""" + line = f"gnome-51-audit row {row}: {reading}" + if line not in _AUDIT_READINGS: + _AUDIT_READINGS.add(line) + print(line, flush=True) + + +def _app_name(app) -> str: + name = getattr(app, "name", None) or (app.get_name() if hasattr(app, "get_name") else None) + return name.strip("'\" ") if isinstance(name, str) else "" + + def _files_apps() -> list: """Take one ``applications()`` snapshot; dogtail's per-name lookup retries internally.""" matches = [] for app in tree.root.applications(): - name = getattr(app, "name", None) or (app.get_name() if hasattr(app, "get_name") else None) - if isinstance(name, str): - clean = name.strip("'\" ") - if clean in FILES_APP_NAMES or "nautilus" in clean.lower(): - matches.append(app) + clean = _app_name(app) + if clean in FILES_APP_NAMES or "nautilus" in clean.lower(): + matches.append(app) return matches @@ -64,6 +78,7 @@ def _nautilus_app(timeout: int = 15): while True: try: for app in _files_apps(): + _audit_reading(6, f"Files AT-SPI app name={_app_name(app)!r}") return app last_error = f"none of {FILES_APP_NAMES} registered" except Exception as exc: # noqa: BLE001 @@ -96,6 +111,7 @@ def _is_frame(n): for _ in range(timeout * 2): frames = app.findChildren(_is_frame) if frames: + _audit_reading(4, f"Files toplevel role={getattr(frames[0], 'roleName', '')!r}") return frames[0] last_children = [(getattr(child, "roleName", getattr(child, "get_role_name", lambda: "")()), getattr(child, "name", getattr(child, "get_name", lambda: "")())) for child in getattr(app, "children", [])[:10]] sleep(0.5) @@ -174,6 +190,7 @@ def home_folder_is_in_the_sidebar(context) -> None: ) assert home_items, "Home list item not found in Nautilus sidebar" + _audit_reading(5, f"Files sidebar Home item roles={sorted({n.roleName for n in home_items})}") @step('Navigate to "{name}" in Files sidebar') diff --git a/tests/smoke/features/steps/gnome_notifications_steps.py b/tests/smoke/features/steps/gnome_notifications_steps.py index 6fcb66e50..cb22b0762 100644 --- a/tests/smoke/features/steps/gnome_notifications_steps.py +++ b/tests/smoke/features/steps/gnome_notifications_steps.py @@ -112,7 +112,7 @@ def notification_banner_no_longer_showing(context) -> None: "!Main.messageTray._bannerBin.visible)).toString()" ) last_out = "" - for _ in range(40): # 20s — GNOME 50 banners can linger longer in QEMU + for attempt in range(40): # 20s — GNOME 50 banners can linger longer in QEMU result = subprocess.run( ["gdbus", "call", "--session", "--dest", "org.gnome.Shell", @@ -124,6 +124,7 @@ def notification_banner_no_longer_showing(context) -> None: last_out = result.stdout m = re.search(r",\s*'\"?(true|false)\"?'\s*\)", last_out, re.IGNORECASE) if m and m.group(1).lower() == "true": + print(f"gnome-51-audit row 10: banner closed on its own within {attempt * 0.5:.1f}s", flush=True) return sleep(0.5) @@ -158,7 +159,9 @@ def notification_banner_no_longer_showing(context) -> None: last_out = result.stdout m = re.search(r",\s*'\"?(true|false)\"?'\s*\)", last_out, re.IGNORECASE) if m and m.group(1).lower() == "true": + print("gnome-51-audit row 10: banner closed only after the Shell.Eval force-dismiss", flush=True) return + print("gnome-51-audit row 10: banner still showing after the force-dismiss; step passes on a warning", flush=True) # In headless GNOME 50 QEMU, the banner visibility JS check can return # "false" (still visible) even after a successful CloseNotification call. # This is a known environment limitation — not a functional regression. diff --git a/tests/unit/test_gnome_files_steps.py b/tests/unit/test_gnome_files_steps.py index 953707797..d1385b4b1 100644 --- a/tests/unit/test_gnome_files_steps.py +++ b/tests/unit/test_gnome_files_steps.py @@ -556,3 +556,47 @@ def find(pred): patch("subprocess.run"): with pytest.raises(AssertionError, match="GNOME Files is still visible"): self.m.files_is_no_longer_running(MagicMock()) + + +# --------------------------------------------------------------------------- +# GNOME 51 audit readings (rows 4, 5 and 6 of references/gnome-51-audit.md) +# --------------------------------------------------------------------------- + +class TestGnome51AuditReadings: + def setup_method(self): + self.m = _import_files_steps() + + def _window_app(self, role): + frame = MagicMock() + frame.roleName = role + frame.name = "Files" + frame.showing = True + app = MagicMock() + app.findChildren.return_value = [frame] + return app + + def test_window_reports_toplevel_role(self, capsys): + with patch.object(self.m, "tree", _files_tree(self._window_app("filler"))): + self.m._nautilus_window(timeout=1) + assert "gnome-51-audit row 4: Files toplevel role='filler'" in capsys.readouterr().out + + def test_reading_is_printed_once_per_run(self, capsys): + with patch.object(self.m, "tree", _files_tree(self._window_app("frame"))): + self.m._nautilus_window(timeout=1) + self.m._nautilus_window(timeout=1) + assert capsys.readouterr().out.count("gnome-51-audit row 4:") == 1 + + def test_app_reports_matched_name(self, capsys): + files = MagicMock() + files.name = "Files" + tree_mock = MagicMock() + tree_mock.root.applications.return_value = [files] + with patch.object(self.m, "tree", tree_mock): + self.m._nautilus_app(timeout=1) + assert "gnome-51-audit row 6: Files AT-SPI app name='Files'" in capsys.readouterr().out + + def test_sidebar_reports_home_item_role(self, capsys): + window = TestHomeFolderIsInTheSidebar()._make_window_with_sidebar(item_role="button") + with patch.object(self.m, "_nautilus_window", return_value=window): + self.m.home_folder_is_in_the_sidebar(MagicMock()) + assert "gnome-51-audit row 5: Files sidebar Home item roles=['button']" in capsys.readouterr().out diff --git a/tests/unit/test_gnome_notifications_steps.py b/tests/unit/test_gnome_notifications_steps.py index 958bdc9ac..7fab80aba 100644 --- a/tests/unit/test_gnome_notifications_steps.py +++ b/tests/unit/test_gnome_notifications_steps.py @@ -46,3 +46,41 @@ def test_parse_notification_id_raises_on_unparseable_output(output): with pytest.raises(AssertionError, match="Could not parse"): module._parse_notification_id(output) + + +def _eval_result(value: str): + return types.SimpleNamespace(stdout=f"(true, '{value}')", returncode=0) + + +def _run_dismiss_check(module, outputs): + from unittest.mock import patch + + with patch.object(module.subprocess, "run", side_effect=outputs), \ + patch.object(module, "sleep"): + module.notification_banner_no_longer_showing(None) + + +def test_banner_check_reports_banner_closing_on_its_own(capsys): + module = _import_gnome_notifications_steps() + + _run_dismiss_check(module, [_eval_result("false"), _eval_result("true")]) + + assert "gnome-51-audit row 10: banner closed on its own within 0.5s" in capsys.readouterr().out + + +def test_banner_check_reports_force_dismiss(capsys): + module = _import_gnome_notifications_steps() + + outputs = [_eval_result("false")] * 40 + [_eval_result("dismissed"), _eval_result("true")] + _run_dismiss_check(module, outputs) + + assert "gnome-51-audit row 10: banner closed only after the Shell.Eval force-dismiss" in capsys.readouterr().out + + +def test_banner_check_reports_soft_pass_when_banner_stays(capsys): + module = _import_gnome_notifications_steps() + + _run_dismiss_check(module, [_eval_result("false")] * 42) + + out = capsys.readouterr().out + assert "gnome-51-audit row 10: banner still showing after the force-dismiss; step passes on a warning" in out From be83d38c2ac07fd1c7afb06a17081fed1a9305e6 Mon Sep 17 00:00:00 2001 From: Danathar Date: Sat, 14 Nov 2026 05:30:52 +0000 Subject: [PATCH 2/2] docs(gnome): keep the GNOME 51 skill under its 500-line cap Co-Authored-By: Claude Opus 5.5 Signed-off-by: Danathar --- docs/skills/test-authoring/gnome/SKILL.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/skills/test-authoring/gnome/SKILL.md b/docs/skills/test-authoring/gnome/SKILL.md index 45f4d2639..6ae1d7641 100644 --- a/docs/skills/test-authoring/gnome/SKILL.md +++ b/docs/skills/test-authoring/gnome/SKILL.md @@ -103,8 +103,7 @@ notification is gone, not that no banner shows: Shell holds earlier banners on screen while the session is idle, as a headless VM always is. Calculator and Text Editor divergences on plain GNOME OS 51: [`references/gnome-51-app-divergences.md`](references/gnome-51-app-divergences.md). - -The Files steps for rows 4, 5 and 6 print a `gnome-51-audit row N: ` line naming the branch that matched, so each smoke run's behave log re-checks those verdicts. Keep the line when you edit the step. +The Files steps for rows 4-6 print `gnome-51-audit row N: ` (the branch that matched) in each smoke log; keep it when editing them. ## Remote session commands from the runner container