Skip to content

Add listbox-picker test case (custom dropdown ARIA listbox contract)#21

Open
manichandra wants to merge 1 commit into
microsoft:mainfrom
manichandra:add-listbox-picker-test-case
Open

Add listbox-picker test case (custom dropdown ARIA listbox contract)#21
manichandra wants to merge 1 commit into
microsoft:mainfrom
manichandra:add-listbox-picker-test-case

Conversation

@manichandra

@manichandra manichandra commented Jun 19, 2026

Copy link
Copy Markdown

Summary

Adds test_cases/listbox-picker/, a test case that evaluates a custom dropdown "picker" against the ARIA listbox contract — semantics that a static axe-core pass does not catch on a div-based dropdown.

Motivation

There is currently no listbox/combobox test case. Custom dropdown menus are a pattern models frequently generate without accessibility semantics:

  • the trigger has no aria-haspopup / aria-expanded, so assistive technology cannot announce that it opens a popup or whether it is open;
  • the popup is built from divs with no role="listbox" / role="option", so it is not exposed as a selectable list;
  • options carry no aria-selected, so the current choice is not announced;
  • there is no keyboard support (arrow keys, Escape).

axe-core does not flag a div dropdown that merely looks interactive, so these gaps go unmeasured. The case maps to WAI-ARIA 1.2, the WAI-ARIA APG listbox pattern, and WCAG 2.1 SC 4.1.2 (Name, Role, Value).

What this adds

  • prompt.yaml — a neutral product spec for a text-style picker that explicitly forbids a native <select>. It does not state the ARIA answer, so the eval measures whether the model gets it right.
  • test.js — six Requirement assertions, following the existing module.exports.run = async ({ page, assert, utils }) convention with { pass, message } results:
    1. Picker trigger exposes popup state (aria-haspopup and aria-expanded)
    2. Open menu uses role="listbox" with role="option" children
    3. Each option has an accessible name
    4. Options expose selection state with aria-selected (exactly one selected)
    5. Arrow keys move the active option
    6. Escape closes the listbox
  • examples/ — two example files with embedded a11y-assertions expectations: a conformant ARIA listbox and a div-based dropdown anti-pattern.

DOM discovery is self-contained (it tags candidate options and inspects roles/attributes), so the case does not depend on widget-specific utils; the only utils call (reload) is guarded.

Validation

Ran test.js against both examples with node_runner:

  • the conformant listbox passes all six assertions;
  • the div-based dropdown fails the five role / selection / keyboard assertions (only "Each option has an accessible name" passes, since the items have text).

Checklist

  • I have signed the Microsoft CLA
  • I ran the repository's evaluation harness locally
  • The new case follows the existing test_cases/ conventions

Covers a custom dropdown picker and the ARIA listbox contract that a static axe-core pass does not catch: the trigger must expose aria-haspopup and aria-expanded, the popup must be role=listbox with role=option children, options must carry aria-selected, and the menu must support arrow-key navigation and close on Escape (WAI-ARIA 1.2 / APG listbox pattern; WCAG 2.1 SC 4.1.2).

Includes prompt.yaml, test.js (six Requirement assertions), and two examples with embedded a11y-assertions expectations (a conformant listbox and a div-based dropdown anti-pattern). Verified with node_runner: the conformant example passes all six; the anti-pattern fails the five role/selection/keyboard assertions.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant