Skip to content

feat(audit): report advisory accessibility findings from the current screen - #4592

Open
thymikee wants to merge 1 commit into
mainfrom
feat/static-accessibility-audit-v2
Open

thymikee wants to merge 1 commit into
mainfrom
feat/static-accessibility-audit-v2

Conversation

@thymikee

@thymikee thymikee commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Stacked on #4594.

Summary

Adds agent-device audit: one fresh capture of the current screen, then advisory accessibility findings with usable refs. 44 files, 1258 insertions. Gross diff is above the 1,000-line target but is one coherent change (it includes ~500 lines of tests and fixtures).

  • small-target: enabled control under 44pt (iOS) or 48dp (Android, captured density). Unknown units report a limitation.
  • missing-name: control with no accessible name. Identifiers, resource ids and typed values are not names.
  • covered-control: controls and rows the existing occlusion pass already marks covered.
  • Decorative images and structural wrappers are excluded. Only proven duplicates collapse (a sole same-role child, or a sole child of a covered parent), never by matching rects.
  • Pure auditSnapshot in capture-kit, types in contracts. Registered through the descriptor registry. Text, JSON and MCP output. Not recorded in replay, not batchable. No audit --platform, no help dogfood change.
$ agent-device audit        # iOS Settings
agent-device audit
3 warnings, 1 observation — current screen
@e48  covered-control  Accessible action is unavailable to touch
@e55  small-target     336×38pt; review touch area
@e56  small-target     17.3×22pt; review touch area
@e58  small-target     17.3×22pt; review touch area

snapshot is unchanged. The only edit in the snapshot path is an audit command added to runtime/snapshot.ts; no existing snapshot, settle, stabilization or replay code changed. The audit module loads lazily, so no other command's import closure grows (eager-closure budgets pass). A live snapshot before and after the audit differed in nothing.

Validation

  • Restacked onto refactor(capture-kit): share the snapshot tree helpers outside Android presentation #4594. pnpm check:affected --run --head d2c380da13 passed, exactCommit: true. git range-diff shows the only change versus the previous head is the tree-helper import.
  • At the previous head: pnpm check:affected --run --head 998e9cee0d passed, exactCommit: true. It covers unit, provider-integration, integration-node, layering and fallow.
  • Tests: 22 pure cases with nearest negatives, formatter test, iOS provider scenario (a ref from the audit is accepted by the next press), Android provider scenario (px to dp).
  • Device runs with the repo CLI: iOS rn-matrix-iphone17 (iOS 26.2) as above. Android Pixel_9_Pro_XL_API_37 Settings returned no findings with density captured. Both devices closed with shutdown.
  • Risks: iOS hittable stands in for visibility, and generic clickable Android containers are not evaluated. Not byte-compared against a separate main build.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

View guided diff Turn on auto-fix

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Size Report

Metric Base Current Diff
Installed (including dependencies) 6.32 MB 6.33 MB +10.1 kB
Package (unpacked) 6.32 MB 6.33 MB +10.1 kB
Package (download) 1.92 MB 1.92 MB +3.3 kB

Startup median (7 runs, lower is better):

Scenario Base Current Diff
CLI --version 33.4 ms 29.7 ms -3.8 ms
CLI --help 77.7 ms 51.1 ms -26.6 ms

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 44 files

You’re at about 96% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread packages/contracts/src/audit.ts Resolved
Comment thread website/docs/docs/commands.md Resolved
Comment thread website/docs/docs/commands.md Resolved
Comment thread website/docs/docs/commands.md Resolved
@apex-sentinel-staging

apex-sentinel-staging Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 0 no longer raised

No open findings.

Reviewed with this repository's own Sentinel configuration (b3c1a3c8abd9).

💬 Talking to Sentinel

In a finding's thread, reply with:

Command What it does
@apex-sentinel-staging note <context> Adds context; Sentinel re-checks the cited code
@apex-sentinel-staging pivot <why> Says the finding is wrong; Sentinel re-checks the cited code
@apex-sentinel-staging like Marks the finding as useful
@apex-sentinel-staging dislike <why> Marks it as not useful, so it is not raised again

On this conversation, comment @apex-sentinel-staging review to run a full review once again.

Earlier reviews

Review of 521a20b4 · 16.11.2026 16:20 UTC

Apex Sentinel · ✅ Approve

Since the last review: 0 new · 0 still open · 2 no longer raised (2 marked fixed)

No open findings.

Reviewed with this repository's own Sentinel configuration (244d1ff68344).

Review of d2c380da · 16.11.2026 15:04 UTC

Apex Sentinel · ✅ Approve

Since the last review: 1 new · 1 still open · 1 no longer raised

2 finding(s): 2 minor.

🟡 Minor (2)

Where Finding
packages/capture-kit/src/snapshot/snapshot-audit.ts:131 🆕 Root-level link treated as inline against every root text node
packages/capture-kit/src/snapshot/snapshot-audit.ts:211 Covered control under a covered plain container produces no finding at all

Reviewed with this repository's own Sentinel configuration (cfe08502f0f6).

Review of 998e9cee · 16.11.2026 14:42 UTC

Apex Sentinel · ✅ Approve

2 finding(s): 2 minor.

🟡 Minor (2)

Where Finding
packages/capture-kit/src/snapshot/snapshot-audit.ts:145 Small-target check ignores Android density for non-mobile targets, so tvOS skips every check
packages/capture-kit/src/snapshot/snapshot-audit.ts:211 Covered control under a covered plain container produces no finding at all

1 finding below minor hidden (user_settings.min_severity).

Reviewed with this repository's own Sentinel configuration (cfe08502f0f6).

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 2 new findings on 998e9ce. The verdict and the full list are in the summary comment.

@thymikee
thymikee force-pushed the feat/static-accessibility-audit-v2 branch from 998e9ce to d2c380d Compare November 16, 2026 14:56
@thymikee
thymikee changed the base branch from main to refactor/capture-kit-shared-snapshot-tree November 16, 2026 14:56

@apex-sentinel-staging apex-sentinel-staging Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apex Sentinel: 1 new finding on d2c380d. The verdict and the full list are in the summary comment.

@thymikee

thymikee commented Nov 16, 2026 •

Copy link
Copy Markdown
Member Author

I reviewed d2c380d and found two problems that need fixes before merge. CI is green (25 checks, 0 failing) and there are no conflicts.

First, audit activates a complete ref frame, but it returns only the finding refs. issuesCompleteRefFrame returns true for audit, which ADR 0014 (lines 156-170, 401-428) treats as a partial publication. ref-frame.ts:135-143 reserves the complete transition for snapshot. Sequence: snapshot (gen 5), press @e3 (frame expires), the screen navigates, audit (even with 0 findings), then press @e10 using a plain ref remembered from gen 5. Before this PR the press fails with ref_frame_expired. At this head it is admitted and taps whatever @e10 is in a tree the caller never saw. The rule: only a response that returns the complete ref namespace may activate a complete frame. Every other ref-returning response publishes through markSessionPartialRefsIssued (session-snapshot.ts:95), with no frame change for empty findings. That also means pinned @eN~s<refsGeneration> refs in CLI text (ADR line 427), and audit merged into MCP REF_ISSUING_TOOLS (tool-ref-pins.ts:79) the way find is. Please update help, docs and the ios-audit test to press the pinned ref, and add a test that a plain pre-audit ref is still refused after an expired frame.

Second, snapshot.ts:202 uses snapshotTruncationForResult(...) === true, which turns the documented unknown truncation (Appium page source, Limrun, unlisted XCTest producers) into false. snapshot-audit.ts:67 then reports checks.snapshot: "complete" with no limitation. On those provider sessions, "no findings / complete" claims coverage the command cannot know. Please keep the tri-state through AuditContext, report unknown as unknown or partial with a limitation line, and extend the output schema enum.

Not blocking, take or leave: both collapse rules in snapshot-audit.ts (isNestedDuplicate around line 124, isOnlyChildOfCovered at 226-229) drop a child without checking that the parent yields the same finding, so a screen can get zero findings; collapse only into a parent that itself reports, and add the two nearest negative tests. The web test at runtime.test.ts:627 duplicates the diff test above it and never calls audit, yet is cited as web evidence for audit; delete it or declare web a gap. checks.platform and source in contracts/audit.ts are single-valued constants, and the detail size fields are all optional although small-target always sets them. audit is added to the timeout-evidence command union in android-snapshot-timeout-evidence.ts but line 44 still returns undefined for it, and ios-audit.test.ts:96 does not assert which node press @e2 tapped. The docs (commands.md:433) omit the 0.5pt/dp tolerance, covered rows, role-based control detection and the inline-link exemption, the capability-matrix label still says "snapshot and snapshot diff", and the inline-link exemption should require a text-container parent rather than any text sibling.

Could we keep the command and the pure module, publish the finding refs through markSessionPartialRefsIssued instead of widening issuesCompleteRefFrame, and drop the speculative constant fields and the optional-everything detail type? That removes contract and schema lines and puts the ref behavior under the existing ADR 0014 owner. I found no smaller way to deliver this: a snapshot --audit flag would change snapshot output. If you want audit to be a complete publication instead, ADR 0014 needs an amendment first.

The open inline threads that still apply: stale refs in tap test, Android timeout evidence, parent actionable check, covered parent suppression, optional detail fields, unknown becomes complete, inline link exemption, root link text, and covered-parent duplicate. Five lower-priority threads also still apply. One thread no longer applies and can be resolved: target checked before platform, which is no longer raised on d2c380d.

Before merge, please fix the ref frame so stale plain refs stay refused after audit and keep unknown truncation unknown. Then run this on the new head, on an iOS simulator and an Android emulator: snapshot, a press (frame expires), then audit. The output should show pinned @eN~sG refs, press <pinned finding ref> succeeding, and a plain pre-audit ref refused with ref_frame_expired or plain_ref_requires_complete_frame. The device runs reported earlier were from the previous head. The first finding comes from reading the code and ADR.

@thymikee

Copy link
Copy Markdown
Member Author

Thanks for the review. All of it is addressed at 521a20b (force-pushed; base stays #4594). pnpm check:affected --run --head 521a20b43b passed with exactCommit: true.

1. Ref frame. issuesCompleteRefFrame is back to snapshot only. audit now publishes its finding refs through markSessionPartialRefsIssued; with no findings it issues nothing and the frame is unchanged, and refsGeneration is present only with findings. CLI text prints pinned @eN~s<refsGeneration> refs, and audit is in MCP REF_ISSUING_TOOLS (merge-only, no generation means no change). Help, docs and ADR 0014 (partial-publication list) say so. Tests: ios-audit presses the pinned ref and asserts the tap lands on the Menu button; a plain finding ref and an unissued pinned ref are refused; an audit with no findings leaves an expired frame expired (ref_frame_expired). I confirmed that widening issuesCompleteRefFrame for audit fails that last test. Pin-store tests cover both audit cases.

2. Truncation. AuditContext.truncated stays boolean | undefined. Unknown reports checks.snapshot: "unknown" with a limitation line; the output schema enum includes it.

Take-or-leave items, all taken.

  • Dropped checks.platform and source; the finding type is a union with a required small-target detail.
  • Both collapse rules collapse only into a parent that reports the same kind (covered: a covered control or row). Four nearest-negative tests added.
  • The inline-link exemption requires a text-container parent. This also fixes the root-level link thread.
  • Deleted the web test; web evidence is a pure test of web ARIA roles.
  • Android timeout evidence now covers audit (the guard excludes only diff).
  • ios-audit asserts the tapped coordinates.
  • Docs: tolerance, covered rows, role detection, coverage limits and the inline-link exemption. Capability-matrix label now includes audit.

All 15 open threads have a reply and are resolved. Per the maintainer note, the "target checked before platform" thread is resolved without a change.

Device evidence (this worktree's build, new head). snapshot, a press (frame expires), audit, press <pinned ref>, then a plain pre-audit ref.

  • iOS e2e-apex (iOS 26.2), Settings. rn-matrix-iphone17 is held by a live session in another worktree, so I used another simulator.
    • press @e7 (General) → audit printed @e57~s406993 small-target 88×28pt; review touch area and 13 more findings, all pinned.
    • press @e8 (plain, pre-audit) → "needs a complete snapshot — the current frame only authorizes its emitted refs" (plain_ref_requires_complete_frame).
    • press @e57~s406993 → Tapped @e57 (78, 415).
    • press @e42~s406993, a covered finding, passed ref admission and was refused as covered.
    • After the press, press @e8 → "belongs to an expired ref frame" (ref_frame_expired).
  • Android Pixel_9_Pro_XL_API_37, Agentdevicelab Settings tab (the system Settings screens had no findings).
    • press @e31 (Settings tab) → audit printed @e36~s345028 small-target 379.3×47dp; review touch area and three more, all pinned.
    • press @e36~s345028 → Tapped @e36 (672, 931).
    • plain pre-audit press @e28 → "belongs to an expired ref frame" (ref_frame_expired).

Both devices are closed with shutdown. Remaining risk: I did not rerun the earlier snapshot before/after comparison.

Base automatically changed from refactor/capture-kit-shared-snapshot-tree to main November 16, 2026 16:42
@thymikee
thymikee force-pushed the feat/static-accessibility-audit-v2 branch from 521a20b to 2e90b57 Compare November 16, 2026 16:42
@thymikee

thymikee commented Nov 16, 2026 •

Copy link
Copy Markdown
Member Author

Most of the earlier findings (d2c380d) are fixed at 2e90b57, and one docs mismatch remains. The user doc at https://github.com/callstack/agent-device/blob/2e90b57/website/docs/docs/commands.md#L437 says a control that is the only child of a same-role control reports only through the parent. The code does not do that: collapseIntoParent drops a child finding only when the parent reports the same kind, so a sole child of a disabled, large, or ref-less same-role parent still reports, and the PR's own "keeps the child when the parent does not report that kind" test asserts it. An agent reading the doc would expect no findings on nested same-role children and misread the ones it gets. Please state the rule the code has: a sole child of a same-role control drops a finding only when the parent reports the same kind.

Not blocking: the ref paragraph at commands.md#L439 covers only the expired-frame case, while an audit with findings also ends the snapshot's complete frame, so after snapshot then audit a ref outside the findings is refused (plain_ref_requires_complete_frame or ref_not_issued); one clause saying "take a new snapshot before using other refs" would cover it, and the stale platform: 'not-requested' in the daemon test fake can also be taken or left.

Of the open inline threads, the threads from the earlier round are fixed at this head and can be resolved, and none still applies. This also covers the root-level link thread (#4592 (comment)) and the covered-child thread (#4592 (comment)). The target-before-platform thread (#4592 (comment)) does not apply, because the tv and desktop limitation is the intended result.

CI is green with 25 checks and none failing, and there are no conflicts. Device evidence was taken on 521a20b, and the range-diff shows 2e90b57 is the identical patch on the moved base, so that evidence still covers this head. snapshotTruncationForResult returns false for non-xctest backends with absent truncation, such as an Android Appium-sourced tree; that is the existing helper's rule, shared with snapshot, so it is out of scope here. Before merge, the commands.md sentence needs to match collapseIntoParent.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant