iOS adapter

[DataTriny]

This project was funded through the NGI0 Commons Fund, a fund established by NLnet. See the project page for more information.

Abstract

The aim of this project is to implement UIKit accessibility. UIKit is Apple's user interface framework powering iOS, iPadOS, tvOS, and visionOS. Once complete, this new backend will allow any user of the AccessKit library to get UI accessibility on iPhone, iPad, Apple TV, and the Apple Vision Pro without substantial change to their current implementation. The finished product will contain a generic component that can be plugged into any iOS application as well as a specialized component to support the popular winit windowing library. An existing UI toolkit written in Rust will be modified to showcase the feature.

Tasks

• Phase 1: Core iOS Adapter
  • Setup the low-level adapter skeleton following existing adapters structure
  • Traits, properties and states mapping for nodes
  • Post simple notifications to report UI events
  • Create the subclassing adapter to inject into a UIKit view
  • Write unit tests and update the CI to have them run
  • Document how to run the winit examples
  • Publish first version of accesskit_ios on crates.IO
• Phase 2: Integrate into the Slint UI toolkit
  • Validate the subclassing adapter through accesskit_winit with the Slint winit backend and make adjustments
  • Implement text editing capability using parley and validate on desktop platforms, iOS will be validated in the next milestone
  • Modify the Slint CI to ensure the accessibility tree can be seen in the simulator
• Phase 3: Integration Testing & Editable Text
  • Setup a test harness
  • Configure iOS simulator on GitHub CI
  • Write some integration tests to cover the key features (traits, properties and states exposed, changes detected)
  • Implement methods to support editable text (characters, selection), validate with a Slint application
  • Post required notifications for editable text (caret/selection move, text insertion/deletion), validate with a Slint application
  • Publish second version of accesskit_ios on crates.IO
• Phase 4: Finishing touches
  • Process feedback from security audit
  • Expose both low-level and subclassing adapters API in the C bindings
  • Update the Meson build script
  • Update the CI to generate pre-built packages
  • Integrate into the existing SDL example to demonstrate how to use
  • Publish new version of the C bindings with the new binaries for iOS
  • Write a blog post about the work done

[MarcelStruckWO]

In case anyone wants to continue work on this, I have created an MVP version here: https://github.com/MarcelStruckWO/accesskit/tree/ios-support

It only covers the most basic steps for integrating with iOS and doesn't support any special traits etc. but at the very least it works for labels.

Unfortunately, I don't have the capacity to build a fully fledged integration at the moment. But I'm happy to assist anyone with technical input and testing.

[DataTriny]

Hello @MarcelStruckWO, thank you for your interest in this.

I will be working on the iOS adapter in the coming months thanks to a grant received from the NLnet foundation, as announced yesterday.

Any help testing or reviewing code will be welcome. I will let you know once I start working on it.

[iMacHumphries]

Hi @DataTriny, really glad to see this feature is on the way! I wanted to introduce myself as a downstream user in case it's useful for prioritization or testing.

I'm developing a 2D mobile game in the Godot Engine, which already integrates AccessKit on macOS, Windows, and Linux. Once accesskit_ios is available, I'd like to wire it into Godot's iOS DisplayServer and try it out in my game. My game is essentially all 2D menus, so it would be a near-ideal validation target for the iOS adapter once it lands. You can check it out here: https://heroiclegends.net

A couple questions to help me plan, and see if I can help out.

1. Timeline. I saw the commit in January and the 4 phase plan in this issue. Do you have a rough sense of when Phase 1 might be testable / when phase 4 would follow? I'm trying to decide if I should wait for upstream or build a temp UIKit accessibility backend for my Godot fork.
2. Where can I help? You mentioned testing and code review would be appreciated. I'd be glad to test pre-release builds in Godot. I'm also interested in contributing if that can help speed this adaptor along.

No rush on a reply, but I just wanted to put a real downstream product on your radar to see if that helps with prioritization, and so I can plan our roadmap. Thanks for taking this on!

[DataTriny]

Hi @iMacHumphries, thanks for your message.

Phase 1 is ~80% complete and I plan to open the pull request in the coming days. Integration in our C bindings was planned in Phase 4 as I didn't expect non-Rust users to take advantage of it so soon. Your game could be a very nice showcase though, I can easily add the 0.1 version of the iOS adapter there. The path for Phase 3 is a bit uncertain at the moment and might require additions to the C API to support editable text. Is this something you would need right away?

[iMacHumphries]

@DataTriny This is so awesome, thank you! A 0.1 version in the C bindings would be exactly what I need to start adding this to Godot / Heroic Legends.

I will eventually need editable text, but I’d be more than happy to get started with basic accessibility functions. My game has a character name input during onboarding and in-game live chat. If you ship the initial adapter without editable text, then I can integrate and test everything else. I can probably even do a temporary workaround while we wait for phase 3.

I’ll keep an eye out for the phase 1 PR. Again, no rush here. I don't have any hard deadlines for this feature, but I'm super excited that this work is moving forward. Let me know if you need anything.

[iMacHumphries]

cc: @bruvzg who has been doing amazing work adding AccessKit to Godot!

[DataTriny]

Hi all, the pull request for phase 1 of this project is in #710. Feedback welcome!

[lukewarlow]

Out of curiosity, have you looked much into the additional accessibility properties and capabilities provided as part of BrowserEngineKit? I'm curious if they provide any otherwise missing pieces and whether this adapter can use them without actually being used in a browser?

E.g. https://developer.apple.com/documentation/browserenginekit/beaccessibility

There's no priority attached to this and we don't need it for servo (there's currently no concrete plans for an iOS port), just wondered if it would be useful.

[mwcampbell]

My preference would be to avoid using BrowserEngineKit by default. My understanding is that Apple is still only allowing third-party browser engines in specific jurisdictions. So using BrowserEngineKit would probably bring extra scrutiny to downstream apps when they're submitted to the App Store. But if it would provide some practical benefit to the Servo project, we could support it as a Cargo feature, disabled by default.

[DataTriny]

@lukewarlow the UIAccessibility protocol is not rich enough to expose a web view that could compete with Safari in terms of user experience. Supporting this additional API is a priority for me but it won't happen now and will likely have to be feature-gated so only "entitled" apps can enable it.