Skip to content

fix(smoke): GNOME 51 audit verdicts for rows 4, 5, 6 and 10; notification dismissal check could not fail - #1209

Merged
Danathar merged 11 commits into
mainfrom
test/gnome51-audit-necessity
Nov 14, 2026
Merged

Danathar merged 11 commits into
mainfrom
test/gnome51-audit-necessity

Conversation

@Danathar

@Danathar Danathar commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary: Every GNOME 50 workaround in the #827 audit now has a GNOME 51 verdict from a targeted probe, and the one that was broken is fixed. The notification-dismissal check could never fail. On GNOME 51 it required a widget to hide that never hides, and it treated any visible banner as ours, so it always ended in a forced dismissal and a warning. It now checks that this scenario's own notification is gone. qecore 4.19.3 is confirmed working on GNOME 51, and the GNOME skill doc gets one GNOME 51 section.

Refs #827. What's left is below.

Verdicts (GNOME Shell 51.1, dakota:testing and utah:testing)

Row Verdict
4 filler/frame toplevels Obsolete on 51 for every app measured. Files, Settings and Firefox expose a populated frame, never filler. Calculator, Text Editor and Clocks ship on neither image
5 sidebar button role Obsolete on 51. The sidebar step's only match is a list item "Open Personal Folder", the pre-GNOME 50 role
6 Files AT-SPI name Still required: Files registers as org.gnome.Nautilus, not nautilus
10 notification dismissal Broken on 51, fixed (below)

Rows 1–3, 7–9, 11 and 12 were settled in #1199 and #1206.

The notification fix

The probe showed what the old check was measuring:

  • After CloseNotification, this scenario's notification leaves the tray.
  • The banner still on screen is an earlier notification. Other scenarios and retries send notifications they never close, and Shell holds that banner while the session is idle (34–282 s idle in CI). One <Shift> press clears it within 2.3 s.
  • _bannerBin stays visible on 51 even with no banner in it.

The step now gives each sent notification a unique title. It passes when that notification is neither the visible banner nor in the tray, and it fails with the details instead of force-dismissing.

After a fresh-context review (b21b5bb), two further gaps are closed:

  • Tray read errors: an error reading the tray, a failed Shell.Eval or an unparseable reply never counts as gone. The JS reports the error instead of treating it as "not queued".
  • Proving it was there: the close step first requires Shell to hold the notification (queued or shown), and the check fails if no send step recorded a title. So a notification that never arrived can't count as dismissed.

The fixed scenario passes every step on both images at b21b5bb: 43559017193 (dakota) and 43559025260 (utah: 0 scenarios failed, and the job fails only at the SELinux gate on utah#1198).

Not tested on GNOME 50: no tested image runs GNOME 50, so CI can't exercise the GNOME 50 path. The check needs only Main.messageTray._banner, getSources() and each notification's title.

qecore on GNOME 51

The GNOME 51 runs use qecore 4.19.3, dogtail 2.0.3 and behave 1.3.3, read from the test process, and smoke-a, smoke-b and the GNOME OS canary pass. qecore's later releases (4.20–4.22.1) name no GNOME 51 change. 4.22's missing-keyring handling matches row 12 on utah.

Docs

  • gnome-51-audit.md: rows 4, 5, 6 and 10; a "round 3" probe section; a qecore note; and a summary saying every row now has a verdict.
  • The Calculator and Text Editor divergence write-up moves to gnome-51-app-divergences.md, which keeps the audit under 200 lines.
  • SKILL.md: its two GNOME 51 audit sections become one GNOME 51 section, at 499 lines.

Tests

  • New unit tests for the notification step drive it with Shell.Eval replies encoded as Shell sends them. They cover: dismissed, still shown, a tray read error, a (false, '') reply, no send, not seen before close, a close refused when the tray can't be read, and unique titles per send.
  • Local unit suite: all pass, except the 9 test_quarantine_age_git.py failures that also fail on main. Docs validation and the behave dry run pass.
  • All probe code is removed: the net diff is the notification step, its tests and docs.

Left on #827

  • The Calculator display lookup and Text Editor Ctrl+N/Ctrl+Q divergences seen on plain GNOME OS 51 are documented, not fixed. Neither app ships on dakota:testing or utah:testing.

🤖 Generated with Claude Code

https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn

Danathar and others added 10 commits November 13, 2026 22:27
…nd 10

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…ell's

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
"Notification banner is no longer showing via Shell.Eval" also required
Main.messageTray._bannerBin to be hidden. On GNOME 51 the bin stays
visible with no banner in it (#827 probe 3: _banner is null 0.2 s after
CloseNotification while _bannerBin.visible stays true for 20 s), so the
check never passed; the step then force-destroyed the banner and, failing
that, only printed a warning. It could not fail, on GNOME 50 or 51,
whether or not CloseNotification worked.

Check only the banner (null, undefined or hidden, which also covers
GNOME 50 keeping a hidden ref), keep the 20 s allowance for slow QEMU
banners, and fail with the last Shell.Eval output instead of
force-dismissing.

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…e smoke scenario does

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…rents

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…r step's search

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…ner shows

Probe 3 (#827 row 10) showed why the stricter check failed: after
CloseNotification our notification leaves the message tray at once, but
the visible banner is an earlier, unrelated one ("Settings") that GNOME
Shell holds on screen while the session is idle; a headless VM is always
idle (idle time ~4 min), and one keypress clears it in ~2 s. The old
"no banner at all" check therefore tracked the wrong notification.

Check that "Test Notification" is neither the visible banner nor queued
in the tray, using one shared title constant for the send step.

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…smoke-a

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
…sal check

Smoke-a runs 43554149449/43554157456 showed the tray holding several
"Test Notification" entries: other scenarios and retries send the same
title and never close theirs, so a title match could not tell whether this
scenario's notification was dismissed (CloseNotification did remove one).
Each send now adds a short unique suffix, stored on the context, and the
dismissal check looks for that title only.

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
… GNOME 51 skill section; drop probe 3

Probe 3 (removed here) measured each workaround's guarded property on the
pre-workaround path, on dakota:testing and utah:testing (GNOME Shell 51.1):
- Row 4: Files, Settings and Firefox expose a populated `frame` toplevel;
  no `filler` occurred. Obsolete on 51 for every app measured.
- Row 5: the sidebar step's only match is a `list item` "Open Personal
  Folder" (the pre-GNOME 50 role). Obsolete on 51.
- Row 6: Files registers as `org.gnome.Nautilus`. Still required.
- Row 10: broken on 51 and fixed earlier in this branch.
- qecore 4.19.3 (with dogtail 2.0.3, behave 1.3.3) is what the GNOME 51 runs
  use, and works there; releases since name no GNOME 51 change.

The Calculator and Text Editor divergence write-up moves to its own
reference (gnome-51-app-divergences.md) to keep the audit under 200 lines.
SKILL.md's two GNOME 51 audit sections become one GNOME 51 section.

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@codecov-commenter

codecov-commenter commented Nov 14, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 98.64865% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
.../smoke/features/steps/gnome_notifications_steps.py 96.49% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

@hivecommons-hive hivecommons-hive Bot added author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 14, 2026
…as dismissed

Fresh-context review of #1209 found two ways the dismissal check could
still pass without proving anything:
- banner_gone_js caught a tray-read error as `queued = false`, so an API
  change in GNOME Shell would read as "gone". The JS now reports the error,
  and any error, failed Shell.Eval or unparseable reply counts as not gone.
- With no send step the title defaulted to the bare "Test Notification",
  which nothing sends any more, so the check always passed. It now fails
  unless a send recorded a title.
The close step also requires Shell to hold the notification (queued or
shown) before closing it, so the check can only credit CloseNotification
with removing something that was there.

Unit tests now drive the step with Shell.Eval replies encoded as Shell
sends them: an error reply, a `(false, '')` reply, no send, not seen, and
a close refused when the tray cannot be read.

Assisted-by: Claude Opus 5.5 via Claude Code
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WcWm7qQa538Zjcuvv8jBtn
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
@hivecommons-hive hivecommons-hive Bot added stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act and removed stage/050-review-pending Review swarm requested; awaiting a verdict for the current head labels Nov 14, 2026
@hivecommons-hive

Copy link
Copy Markdown
Contributor

@Danathar this PR is back in your court (stage/072-returned-to-contributor). The hive will not merge it until the head changes.

Failing checks:

  • test / GNOME — smoke-a

Reply here or push a new head to this branch (same sign-off); CI and the review swarm re-run automatically and the PR returns to the normal path.

Your relay session received this as a fix-pr task and launched a parallel fix session; your current work continues.

@Danathar
Danathar merged commit 51a3941 into main Nov 14, 2026
19 of 20 checks passed
@Danathar

Danathar commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

The failing test / GNOME — smoke-a check is the manual run on utah:testing, 43559025260. It isn't caused by this PR, so there's no new head:

I didn't add an allowlist entry here. utah#1198 marks that as a separate testsuite decision, and selinux_allowlist.yaml says to fix the denial in the image repo instead. This check turns green once utah#1203 merges and utah:testing is rebuilt.


🐝 Hive Agent: contributor | SHA: unknown


🐝 Hive Agent: contributor | SHA: unknown

Danathar added a commit that referenced this pull request Nov 14, 2026
main's #1209 measured the verdicts for rows 4, 5, 6 and 10 and rewrote
the notification dismissal check, so this merge takes main's
gnome-51-audit.md, gnome_notifications_steps.py and its unit tests.
The row 10 log lines are dropped because the branches they reported no
longer exist. The rows 4, 5 and 6 readings in gnome_files_steps.py stay,
and the SKILL.md note now says what they re-check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Danathar <Danathar@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author/clanker A contributor's own hive authored this PR under their login dco All commits are DCO signed-off role/trusted Author is trusted for unattended merge (trusted-author role or registry/claim tier in require_tiers) stage/072-returned-to-contributor Contributor PR: CI red, changes requested or conflicting; diagnosis posted, author must act

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants