Adaptive mode

[mondalaci]

Adaptive mode will enable Agent to detect the actual foreground window and change the keymap of the UHK accordingly.

First up, we'll have to query the current window of the operating system. A long time ago, I was searching for such node modules, and couldn't find any, so I started writing current-window-linux, but it's not complete, may not be very stable, and it must be wrapped into a node module.

In the spirit of modularity and code sharing, I propose creating a global npm module called current-window which will depend on the OS-specific current-window-linux, current-window-macintosh, and current-window-windows packages, all yet to be created.

The interface of these modules should be a simple getCurrentWindow() function which returns a object containing all the properties of the current window that can be extracted from the OS. The properties of the toplevel current-window module should be the union of the OS-specific modules.

Actually, I'm wondering whether there's a callback mechanism in various OS-es to detect the change window event. That'd be more efficient and faster than polling. If there's such a mechanism, it should be exposed via an API.

There'll be configuration options under a dedicated menu item for this feature named "Adaptive mode" under the Agent menu. The above will take quite some work, so I'll specify the UI when we'll get there.

We will also have to update the Adaptive mode (triangle) icon of the LED display based on whether adaptive mode is enabled.

[laxu]

This will need OS-specific implementations because the methods to get the current active window are different.

For Windows you might be able to use the GetActiveWindow or subscribe a global hook to monitor the WM_ACTIVATE event. I don't know anything really about Windows programming so there might be better and more modern ways to do it.

For OSX the simplest way seems to be to just use AppleScript to figure out what the frontmost app window is. There are probably ways to detect the window with Objective-C as well.

How should adaptive mode set which keymap to use for each window? It might be easiest to build a map of application and keymap that could be managed in Agent.

I use a mouse gesture app called StrokesPlus on Windows that has a feature that allows detecting a window by dragging a reticle over it. It gets several identifiers like executable name, window title etc that can be used to fine tune if the gestures work on the whole app or just one of its windows. Something similar might be good for adaptive mode.

[mondalaci]

Thanks for the references! They seem very useful. The simplest solution may be to look into the source of existing open source applications that offer such functionality. I know that there are a lot of such apps on Linux and I can name a few. Not sure about other OSes.

Yes, Agent will allow for the user to create applications to keymap mappings, and then the firmware will expose a switch keymap command via USB which Agent will call upon window switching.

StrokesPlus probably uses relevant APIs. Like I said, it'd be useful to extract code from relevant apps. Not sure whether this one is open source and has a compatible license.

[Hurricaaane]

If I were to do some research based on open source software, I'd look at Open Broadcaster Software or Open Broadcaster Studio, which is a popular tool used by online streamers (i.e. Twitch). If it matters they are using GPLv2.

• Their method to find the process name on Windows, and a handle to the foreground process.

  • It's interesting to see that their call to OpenProcess is obfuscated. According to comments I've found, it's to deal with antivirus software, but there may be more to it.

• If you search for find_window there's some interesting information for platform-specific implementations.

• For some features, they are going through the trouble of doing window title matching.

---

Just as an anecdote, on October 2015 one of the programs I have created was wrongly detected by Valve Anti Cheat (VAC) because it was reading the name of the foreground process. Thankfully it was confirmed as a false positive and I was unbanned in less than a week. I probably won't use Adaptive mode due to paranoia...

The implementation that I used was mostly copied from this StackOverflow answer. The OpenProcess with the PROCESS_VM_READ flag (that OBS does not use) was likely a contributing factor to the red flag.

Nowadays I'm using the following instead, which is bloody expensive but I'm not willing to take any risks:

    WinDef.HWND hwnd = User32.INSTANCE.GetForegroundWindow();
    User32.INSTANCE.GetWindowThreadProcessId(hwnd, pointer);
    int pid = pointer.getValue()
    // when pid changes, call this:
    Process pr = Runtime.getRuntime().exec("tasklist.exe /fo csv /nh /fi \"PID eq " + pid + "\"");

[mondalaci]

@Hurricaaane Thank you, this is gonna be tremendously useful when implementing the current-window-windows module.

[mmroczka]

What is the status of this item? This functionality (application-specific keymaps) is listed on the home page of the UHK website. Is this finished and I just don't see it? How can I get my keyboard to recognize a specific application (like Intelli-J in the example in the video on the landing page) and automatically switch to that keymap?

[mondalaci]

@mmroczka We haven't started to implement this feature yet, and given its complexity it will take quite some time. We're doing our best, but we have to deal with a lot of other, higher priority issues. We'll definitely make it happen, but not ETA yet.

[mmroczka]

Thanks for working diligently towards making this an awesome product! Do you have any sort of public roadmap so we can get visibility into higher priority issues that are planned to be fixed?

[mondalaci]

Thanks for your understanding! We don't have an official roadmap, but recently, we've been focusing on implementing various recovery modes. In the near future, add-on specific features will take priority, and finally, extra features like this one.

[cboebel]

Just checking in since it's been awhile. Any progress on this?

[mondalaci]

Nope. If we'll make progress on this, we'll update the issue.

[mondalaci]

@cboebel Please thumb up the first comment of an issue if want to sign your interest. Repeated queries tend to me slow down things and increase noise.

[cboebel]

I've been giving this some thought and would like to try to hack on it a bit. Please forgive a potentially naive question, but.. if you suppose that active window information is available, how will the keyboard obtain this information? Do you envision agent notifying the UHK when the active window changes?

Thanks!

[mondalaci]

Yes, the idea is that Agent will detect active window changes, match window properties, and map the actual window to the preferred keymap. Then Agent will send the UHK a keymap change command featuring the relevant keymap.

It'd probably be worth to store the mapping information in the EEPROM of the UHK for portability purposes. It's doable, but there are some blocking issues, so we should implement this part later.

As for the next step, we could badly use the aforementioned current-window-{linux,windows,macintosh} npm packages. Ideally, these packages should be as efficient as possible by exposing a callback based API (whenever possible based on native system APIs) that doesn't utilize polling, and provide as many window related properties as possible for the most accurate matching.