Add support for tvOS input - #65
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d1c59d721b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if now >= state.deadline { | ||
| keyboardShortcutStates[slot] = nil |
There was a problem hiding this comment.
When a user holds only one button from the text-input chord, such as the default Y button for an in-game hold action, the state is cleared as soon as the delay expires. On the next sample the still-held button starts a new shortcut state, so the button is suppressed again; the stream sees brief repeated taps instead of a continuous hold. After the timeout for an incomplete chord, keep that partial press in a pass-through state until all involved buttons are released.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Indeed, Tested in FH6. Could you make sure that if only part of the shortcut is held, the button continues behaving like a normal held button until it’s released? At the moment, it might be sent as repeated taps.
| if shortcutButtons == textInputTriggerSequence.asSet() { | ||
| notifyControllerKeyboardShortcut() |
There was a problem hiding this comment.
For the common case where the full chord is pressed together, this branch calls notifyControllerKeyboardShortcut() immediately instead of waiting for textInputTriggerDelayMs. That makes the new hold-delay setting ineffective and still lets quick accidental chord presses open the Apple TV keyboard, contrary to the setting description that the buttons must be held for the configured duration.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
We should apply the configured hold delay after the full shortcut is pressed? A quick press of the shortcut shouldn’t open the keyboard.
|
Hi @omgitsalexl Could you please run the SwiftLint, swiftformat on the branch while I test. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: dad7c125ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let shortcutGraceWindow = UInt64(textInputTriggerDelayMs) * 1_000_000 | ||
| let shortcutButtons = physicalKeyboardShortcutButtons(on: controller) | ||
| if var state = keyboardShortcutStates[slot] { | ||
| state.pendingButtons.formUnion(shortcutButtons) |
There was a problem hiding this comment.
When a user taps one configured shortcut button, releases it, and then taps another before the delay expires, this line keeps the released button in pendingButtons. At the timeout, keyboardShortcutReplayButtons(for: state.pendingButtons) replays every accumulated button as one chord, so with the default Options+Y shortcut the stream receives a synthetic Options+Y tap instead of the two separate game inputs. Clear or replay the released partial before unioning later presses.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
We should preserve separate button presses here. If someone taps Y and then Options, the game should receive two individual taps rather than one combined shortcut.
| guard let (usage, requiresShift) = Self.usageForCharacter(character) else { | ||
| inputLog.info("Skipping unsupported replay character: \(String(character), privacy: .public)") | ||
| continue |
There was a problem hiding this comment.
When the Apple TV keyboard returns characters outside this hard-coded HID map (for example accented letters, CJK IME output, or emoji), this path silently skips them; replaySubmittedText still appends Return afterward, so the remote game/chat can receive a truncated or empty submission. Either reject/disable sending when any character is unmapped, or use an input path that preserves the full String.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
This I couldn't test yet but maybe you can validate the text before sending it? If it contains unsupported characters, keep the keyboard open and show an error instead of sending incomplete text.
Also, please avoid writing the character to logs since it could be part of a password.
|
@omgitsalexl I didn't get the chance to test in a game where I need to type which I will do in sometime, need to look for a game where I can input. But did high level testing and noticed that the review comments from the bots are valid. Also, maybe the UI overlay for the text input could be styled a bit better, like centre the text, etc. Also, after entering the text, and clicking on "done" closes the entire overlay, and I was never able to "send" it. |
|
Codex: |
|
I really appreciate the thorough review! I had only tested it myself in a very specific manner, inputting server details mostly. Regarding the done button vs a send button: pressing done closes the window and sends input at the same time. I could definitely make this more clear to the user. The Codex bot does indeed have some good points as well, and yours regarding the potential logging issues. I'll take all these into account and try to get it cleaned up! |
|
Both |
|
@omgitsalexl By chance did you check the failing logs? Running SwiftFormat... Let me test it once |
|
here's the fixed file @omgitsalexl Since I can edit the PR, you can extract and push it. |
|
I think I did a poor job getting across what I had meant in my comment regarding the failure; my apologies. I had seen that the logs pointed towards Sendable being the culprit. My confusion, or questioning, was more along the lines of why the CI would give different output than the same rules file ran from my machine. Hope that makes sense! |
Yes indeed and most probably that is happening because you might be using different version of swiftlint and Swift-Format locally as compared to CI, you should check the documentation. I think somewhere it's mentioned which exact versions to use. Hope this helps. |
|
These commands require the exact tool versions pinned by CI: SwiftFormat 0.61.1 and SwiftLint 0.63.3. See Linting for installation and version details. From Readme.md |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f69c24e9aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let shouldPause = state == .textEntry | ||
| || (state == .pauseMenu && streamController.remoteMode != .gamepadMouse) |
There was a problem hiding this comment.
When Remote Mode is gamepadMouse, opening the pause menu leaves shouldPause false, but InputSender.tick() still forwards every extended-controller snapshot regardless of remote mode while VideoSurfaceViewRepresentable also enables controller focus for the overlay. In that mode, using the controller D-pad/A to navigate or select pause-menu buttons also sends those buttons to the game underneath; only the Siri Remote mouse is suppressed by overlayVisible. The pause menu should still pause controller/gamepad forwarding in this mode.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8db6d89909
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| .onExitCommand { | ||
| dismiss() | ||
| } |
There was a problem hiding this comment.
When the capture screen is active, pressing any controller button that tvOS maps to Exit/Back (for example B/Circle, and often Menu) invokes this handler and dismisses the screen before monitorControllerButtons() can save the sequence. The streaming path explicitly disables tvOS system gestures for these same buttons in InputSender.claimControllerInput, but this settings flow never does, so some advertised shortcut buttons cannot actually be configured. Gate this exit handler while isCapturing or claim/release controller gestures for the capture session.
Useful? React with 👍 / 👎.
|
The latest commit fixes the narrow Back/Menu capture issue, but 2 P1 and 5 P2 issues remain. 🔴 [P1] Preserve button history while resolving the shortcut chordLocation: The shortcut state machine still does not fully satisfy the requested input behavior:
This interrupts the original Required change: Buffer ordered shortcut-button transitions while the gesture is unresolved. If the chord succeeds, consume them. If it fails or releases early, replay the original down/up edges in order and preserve any currently held button continuously. 🔴 [P1] Make submitted text respect the configured keyboard layoutLocation:
This silently produces incorrect text for non-US layouts, including:
This can corrupt usernames or passwords and still submit them with Enter. Required change: Pass the selected layout into Focused tests should cover at least 🟡 [P2] Provide a way to cancel controller captureLocation: The latest
Required change: Add a cancellation/timeout path and explicit no-controller handling. Prefer temporarily claiming the selected controller’s system gestures while leaving a Siri Remote or UI cancellation route available. 🟡 [P2] Revalidate the shortcut when related settings changeLocations:
Shortcut conflicts are validated only during capture. A valid single-button shortcut can become invalid later when the user:
That can make the text shortcut permanently consume the configured overlay or Steam button. Required change: Revalidate or reset the sequence whenever either related setting changes. Conflicting persisted settings should also be normalized during decoding. 🟡 [P2] Capture from the controller that actually produces inputLocation: Capture always polls the first extended controller. Controllers 2–4 are ignored despite CloudNow supporting four simultaneous controllers. Pressing buttons on another connected controller also leads directly into the non-cancellable capture state described above. Required change: Detect and pin whichever connected controller first produces input for the capture session, or explicitly let the user select a controller. 🟡 [P2] Add the new localization keys to every localeLocation: All 40 new keys exist only in English. Each of the 33 non-English locale tables is missing all 40 keys. This produces mixed-English UI and violates the repository requirement in
Required change: Add all new keys to every locale table. A parity check comparing each table against 🟡 [P2] Disable automatic text transformationsLocation: The generic remote text field uses the default autocorrection and capitalization behavior. These can modify usernames, passwords, commands, or other exact game input before replay. Required change: TextField(
L10n.text("controller_text_entry_placeholder"),
text: $textEntryText
)
.autocorrectionDisabled()
.textInputAutocapitalization(.never)🟢 [P3] Bound or make long text replay cancellableLocation: Replay length is unlimited. Each character generates 2–4 events at 18 ms intervals while all game input remains paused. A 1,000-character lowercase paste takes roughly 36 seconds. The overlay closes immediately, and the user has no progress indicator or cancellation path while controller input remains blocked. Recommended change: Apply a character/event limit, or keep replay visible with progress and cancellation support. Previous feedback statusConfirmed fixed
Still incomplete
|
2 participants
When streaming certain games, they require text input from the user to continue (e.g. Inputting a password to join a dedicated server in Enshrouded, Palworld, or other games). This pull request adds the ability to pull up the tvOS native keyboard, receive input, and then play this input back to the stream as a sequence of characters.
There are settings added to allow customizing the shortcut keys to pull up the native keyboard, and to adjust the hold time needed to display the keyboard; to help prevent crossover between games that may have the same button sequence.